Skip to content

Report failed performance scenarios separately from timings #112

Description

Context

scripts/Measure-InstallPerformance.ps1 records elapsed time even when a measured installation action fails. The console summary omits the recorded Error property, allowing partial work to look like a successful benchmark result.

Observed behavior

The action exception is caught and stored, but a normal result object is still appended to JSONL and displayed with DurationS. The -All measurements used for #100 stopped on font-file collisions yet appeared in the timing table.

Expected behavior

Failed scenarios are visibly reported as failures and are excluded from successful timing comparisons.

Reproduction

  1. Run the harness with an action that throws, such as a known colliding -All installation.
  2. Observe that a duration is written and displayed as though the scenario completed.
  3. Inspect the JSONL manually to discover the hidden Error value.

Environment

PowerShell 7 on Windows. The behavior is platform-independent.

Regression

Introduced with the performance harness in #100.

Workaround

Inspect the Error field in every JSONL record before comparing durations.

Acceptance criteria

  • Failed scenarios are unmistakably marked as failed in console and persisted output.
  • Failed or partial runs are excluded from successful timing comparisons.
  • Successful scenario output retains elapsed time and module-version metadata.
  • The harness exits nonzero, or otherwise exposes a machine-readable failure state, when any requested scenario fails.

Technical decisions

Preserve failure evidence and elapsed diagnostics, but do not classify failed work as a successful benchmark sample.

Implementation plan

  • Add a failing-action regression test or repeatable harness verification.
  • Separate successful timing records from failed scenario records.
  • Include status and error information in the console summary.
  • Verify successful and failed JSONL output shapes.

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions