IBenchmark

[karimsayedre]

Refactor GPU benchmarking with unified helper framework

Refactored the GPU benchmarking system to use new helper classes (IBenchmark, GPUBenchmarkHelper, and related types), replacing manual pipeline
nuffer, and query management with a unified, extensible framework. Benchmarks now use a common Aggregator for session orchestration, CLI parsing, baseline comparison, and reporting. Added support for hierarchical variant names, focus/median-of-K runs, JSON output, and improved console reporting. Updated main application and all benchmarks to use the new structure, improving maintainability and extensibility.

here's what -h tells you can do:

  --output PATH              write this run's report to PATH (default: SamplerBench.json)
  --baseline [LABEL=]PATH    load PATH as a baseline; LABEL becomes the column header ('vs LABEL').
                             repeatable. If LABEL= is omitted, the file's stem is used
                             (e.g. main.json -> 'main'). '=' is used instead of ':' so Windows
                             drive letters in paths don't collide with the separator.
  --no-baseline              skip the default auto-load of the output path
  --no-color                 disable ANSI color in the live table (also honored: NO_COLOR=1 env var)
  --focus NAME               print a focused baseline-comparison table for NAME before the run.
                             NAME is the hierarchical name with '>' between segments (whitespace
                             around '>' is optional). Repeatable; one row per --focus. The first
                             loaded baseline is the reference for inline deltas in this table.
                             Example: --focus "Linear > Linear > 1:1"
  --focus-samples N          run each focused row N times (median + outlier rejection) for
                             jitter-robust comparisons. Default 3; clamped to [1, 32]. N=1
                             matches the rest-phase single-shot path. Wall time per focused
                             row scales linearly with N.
  --help, -h                 print this help

A screenshot can tell more:

A simple json result for example 64: