Phase 3: INVOKEDYNAMIC handler isolation

[jbachorik]

Summary

Replace INVOKESTATIC handler copying with INVOKEDYNAMIC dispatch. Probe handler methods now stay in the probe class (bootstrap CL) and are called via ConstantCallSite, eliminating bytecode copying into target classes.

Architecture change

	Old (INVOKESTATIC + CopyingVisitor)	New (INVOKEDYNAMIC + ConstantCallSite)
Isolation	Probe bytecode embedded in target class	Handlers stay in probe class
Dispatch	Direct INVOKESTATIC to copied handler	INVOKEDYNAMIC → cached ConstantCallSite
Unload	Incomplete (stale embedded code)	Complete (cache cleared on unregister)
Bootstrap	Indy.class (Java 15-specific, reflection)	IndyDispatcher + HandlerRepository interface
Golden files	3× per test (static/dynamic/leveled)	1× per test (unified)
Overhead	Direct call	~1.2 ns (ConstantCallSite inlines after JIT)

Dispatch chain

Instrumented target method
  → INVOKEDYNAMIC (first call triggers bootstrap)
    → IndyDispatcher.bootstrap()
      → HandlerRepositoryImpl.resolveHandler()
        → MethodHandles.publicLookup().findStatic(probeClass, handlerName, type)
          → ConstantCallSite (cached — subsequent calls go direct)

Integration test fixes

1. AnyType descriptor transformation — Probe methods using @AnyType get Lorg/openjdk/btrace/core/types/AnyType; → Ljava/lang/Object; in descriptors so INVOKEDYNAMIC call site types match. Applied in both BTraceProbeNode.getBytecode() and BTraceProbePersisted.register().

2. StackWalker auxiliary frame skipping — getCallerClassLoader() and getCallerClass() in BTraceRuntimeImpl_9 and _11 now skip org.openjdk.btrace.runtime.auxiliary.* frames so probe handler frames are transparent to classloader resolution.

3. HandlerRepositoryImpl cleanup — Removed fragile asType() fallback; handler resolution is now a clean findStatic lookup with warn-on-failure.

Review fixes

• ConcurrentHashMap.computeIfAbsent null NPE — ConcurrentHashMap disallows null values; replaced with get() + put() pattern so failed resolutions aren't cached and can retry.
• Symmetric probe lifecycle — Added HandlerRepositoryImpl.unregisterProbe() to both BTraceProbeNode.unregister() and BTraceProbePersisted.unregister(). Removed premature registerProbe from BTraceProbeFactory.createProbe() — registration now happens only in probe.register() after definedClass is set.
• COMPUTE_FRAMES → 0 in BTraceProbePersisted.transformAnyTypeDescriptors() — only descriptors change (not control flow), so existing frames are preserved as-is. COMPUTE_FRAMES was unnecessary and risked ClassNotFoundException during hierarchy resolution.
• Redundant unregisterProbe removed from Client.onExit() (now handled by probe lifecycle).

Benchmark results

DispatchBenchmark uses the real BTraceTransformer pipeline to instrument a target class with a compiled BTrace probe, then measures dispatch overhead.

Benchmark                                  Mode  Cnt   Score   Error  Units
DispatchBenchmark.baseline_noArgs          avgt   10   0.527 ± 0.016  ns/op
DispatchBenchmark.baseline_withReturn      avgt   10   0.527 ± 0.015  ns/op
DispatchBenchmark.instrumented_noArgs      avgt   10   1.529 ± 0.033  ns/op
DispatchBenchmark.instrumented_withReturn  avgt   10  20.223 ± 0.309  ns/op

• Entry handler: ~1.0 ns overhead — pure INVOKEDYNAMIC dispatch cost
• Return handler with @Duration: ~19.7 ns overhead — dominated by two System.nanoTime() calls (~11 ns each on macOS), not dispatch

Files

New: IndyDispatcher, HandlerRepository, DispatchBenchmark, DispatchTarget, Workload, DispatchScript
Deleted: CopyingVisitor, Indy, ~400 redundant golden files (static/dynamic/leveled → unified)
Refactored: HandlerRepositoryImpl, Instrumentor, Assembler, BTraceProbeNode, BTraceProbePersisted, BTraceRuntimeImpl_9/_11

Test plan

• ./gradlew :btrace-instr:test — all instrumentor tests pass
• ./gradlew :integration-tests:test -Pintegration — all 22 integration tests pass (including Docker)
• ./gradlew :benchmarks:runtime-benchmarks:jmh -PjmhInclude='DispatchBenchmark' — benchmarks run and produce stable results
• Manual: verify attach/detach cycle cleans up handler cache entries

🤖 Generated with Claude Code

---

This change is