Tests the invoke-custom bytecode.