[not-matthias]

TLDR: Simulation and memory runs measured every benchmark in one shared process, so allocator, JIT and heap state from earlier benchmarks leaked into later ones. Each analysis benchmark now runs in its own fresh process, reusing the walltime fork machinery (tinybench and Vitest 5).

• core: analysisIsolation() reads the same env as walltimeIsolation(), but always uses 1 fork. CODSPEED_WALLTIME_FORKS and forks options don't apply. CODSPEED_PROCESS_ISOLATION=false restores the shared process.
• tinybench: the parent runs each task via runForks(uri, 1); the child measures only that task.
• vitest (v5): workers queue analysis benchmarks; the main process runs one vitest bench per benchmark after the run. Vitest 3/4 and benchmark.js are unchanged.
• Under valgrind --tool=callgrind --trace-children=yes (runner args), examples/with-typescript-simple-esm and examples/with-vitest-v5 produce one dump per benchmark, each from a distinct PID, none from the parent. switch 1/switch 2 measure 4,625/4,683 Ir (shared process: 4,625/3,152).

Review notes

• Stacked on fix: keep one-time setup costs out of the first measured benchmark #123. Without it, every isolated benchmark pays the first-benchmark setup costs (about +230–300 µs per tinybench benchmark on CI).
• Analysis forks write empty samples. They only tell the parent that the child measured its benchmark, so a URI mismatch fails loudly instead of silently measuring nothing.
• Each benchmark now pays a full Node (or Vitest) startup under Valgrind: on CI, tinybench simulation went 5m47 → 9m41 and vitest simulation 6m22 → 14m35.

Closes COD-4156

[codspeed]

Merging this PR will degrade performance by 86.96%

3 warnings, 1 note

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 34 benchmarks are dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚠️ 12 benchmarks with a significant change ran on different hardware

Their base and head results were measured on machines with a different CPU, core count or memory, so the changes might not come from the code. Run base and head on the same runner type.

Learn more about Macro Runners

See affected benchmarks

ℹ️ 28 benchmarks have no flamegraph

The flamegraph wasn't generated: most of the time was spent in runtime internals, which would leave nothing useful to analyze.

See affected benchmarks

⚡ 6 improved benchmarks
❌ 61 regressed benchmarks
✅ 203 untouched benchmarks
⏩ 56 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Memory	wait 1ms	656 B	34,392 B	-98.09%
❌	Memory	short body	656 B	33,864 B	-98.06%
❌	Memory	long body	656 B	33,864 B	-98.06%
❌	Memory	switch 2	656 B	33,864 B	-98.06%
❌	Memory	test_iterative_fibo_100	784 B	33,904 B	-97.69%
❌	Memory	test_iterative_fibo_10	784 B	33,904 B	-97.69%
❌	Memory	test_recursive_cached_fibo_20	784 B	33,904 B	-97.69%
❌	Memory	test sync baz 10	784 B	33,904 B	-97.69%
❌	Memory	test_recursive_cached_fibo_30	784 B	33,904 B	-97.69%
❌	Memory	test_recursive_fibo_20	784 B	33,904 B	-97.69%
❌	Memory	test_recursive_cached_fibo_10	784 B	33,904 B	-97.69%
❌	Memory	test sync baz 100	784 B	33,904 B	-97.69%
❌	CPU Simulation	switch 2	23.5 µs	661.5 µs	-96.44%
❌	Memory	test_recursive_cached_fibo_20	1.3 KB	33.6 KB	-96.19%
❌	Memory	switch 2	1.3 KB	33.6 KB	-96.19%
❌	Memory	switch 2	1.3 KB	33.6 KB	-96.19%
❌	Memory	test_iterative_fibo_100	1.3 KB	33.6 KB	-96.19%
❌	Memory	test_iterative_fibo_10	1.3 KB	33.6 KB	-96.19%
❌	Memory	test_recursive_cached_fibo_10	1.3 KB	33.6 KB	-96.19%
❌	Memory	test_recursive_cached_fibo_30	1.3 KB	33.6 KB	-96.19%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing cod-4156-support-process-isolation-in-simulation (7d16c57) with cod-4179-fix-nondeterministic-gc-and-async-bench-handling-in (16decfc)}

Footnotes

1. 56 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The changes appear safe to merge; no blocking issue was found.

Summary

Analysis benchmarks now run in separate processes in tinybench and Vitest 5.

• Tinybench analysis benchmarks run in separate fresh processes.
• Vitest 5 analysis benchmarks run in separate fresh processes.

No actionable issues were found in the changes since the previous review or the full-PR rule check. No previous review threads were supplied. This review used source checks; no runtime tests were run.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Benchmark registration] --> B{Isolation setting}
  B -->|Disabled| C[Measure in shared process]
  B -->|Enabled| D[Parent selects benchmark URI]
  D --> E[Start fresh child process]
  E --> F[Measure only matching benchmark]
  F --> G[Instruments keep analysis measurement]
  F --> H[Write completion file]
  H --> I[Parent checks child completed]

Loading

_{Reviews (2) · Last reviewed commit: "feat(vitest): run analysis benchmarks in..." · Reviewed by Greptile}