docs(design): benchmarking system by michael-johnston · Pull Request #51 · IBM/algorithm-nexus

michael-johnston · 2026-04-28T09:56:25Z

No description provided.

initial commit

Co-authored-by: Alessandro Pomponio <10339005+AlessandroPomponio@users.noreply.github.com> Co-authored-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com> Signed-off-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com>

Changes addressed

Co-authored-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com> Signed-off-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com>

michael-johnston · 2026-04-29T11:36:14Z

@christian-pinto any comments?

michael-johnston · 2026-04-29T15:36:32Z

I may modify 2.1 solution as maybe we shouldn’t require resolving all the benchmark packages together as the only reason to do so seems to be for the proposed 2.1 solution and nothing else.

edit: 3.3 solution also relies on this global resolution

Co-authored-by: Christian Pinto <55737893+christian-pinto@users.noreply.github.com> Signed-off-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com>

DRL-NextGen · 2026-04-29T15:55:07Z

No vulnerabilities found.

Co-authored-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com> Signed-off-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com>

michael-johnston · 2026-04-29T16:39:31Z

@christian-pinto @AlessandroPomponio What about this? The aim is to not require resolving all benchmark packages against each other.

for REQ 2.1 - in the model.yaml users specify the path to the benchmark packages the model needs to execute the experiments given in the yaml. We can have just package or package+experiments used.
- Change: No addition of benchmark package via uv, no deps req, no install.
for REQ 2.2. - We have two levels. The nexus cliI can list all experiments registered and all the packages. It can't list experiments not used or the details of the experiments as this requires ado+ installing the package. However at any time user can install a package and get more details
- Change: We use nexus cli not ado for listing - we can't get same detail and validation, this is the price.
for REQ 3.3 - same approach as REQ2.1 works as user just gives a path to a benchmark package in a different nexus package

Choice is then between

Users add benchmark package to uv group.
- Pro: ado takes care of everything. No additional files need to be edited or code written.
- Con: All benchmark packages have to resolve together
Users add benchmark package info to nexus package metadata.
- Pro: No (or much less) dependency issues as for benchmarking only the packages for a single model/algo need to be installed
- Con: Users need to add metadata to a file. Nexus CLI requires additional code. You can't get same detail as with ado CLI.

Co-authored-by: Christian Pinto <55737893+christian-pinto@users.noreply.github.com> Signed-off-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com>

Changes addressed

Co-authored-by: Alessandro Pomponio <10339005+AlessandroPomponio@users.noreply.github.com> Signed-off-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com>

christian-pinto · 2026-05-05T14:08:34Z

+this to the model directory of the relevant nexus package (REQ 2.3). The
+benchmark configuration can reference any benchmark experiment registered by the
+Nexus package. If the benchmark experiment they need is not registered by the
+nexus package [they can add it.](#defining-a-benchmark-to-use-for-a-model). The


This is referencing the paragraph it belongs to. I assume it is like the below?

Suggested change

nexus package [they can add it.](#defining-a-benchmark-to-use-for-a-model). The

nexus package [they can add it.](#defining-the-benchmark-experiment-packages-used-by-a-nexus-package). The

@michael-johnston this one is still unresolved.

Co-authored-by: Christian Pinto <55737893+christian-pinto@users.noreply.github.com> Signed-off-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com>

michael-johnston added 4 commits April 27, 2026 10:01

docs(design): benchmarking system

c567a7e

initial commit

docs(design): updates

b3ff3c1

Merge remote-tracking branch 'origin/main' into maj_benchmark_design

decab6a

docs(design): updates

ed0baa5

michael-johnston requested review from AlessandroPomponio and christian-pinto April 28, 2026 09:56

michael-johnston linked an issue Apr 28, 2026 that may be closed by this pull request

docs (benchmarks): benchmark system design #43

Open

AlessandroPomponio previously requested changes Apr 28, 2026

View reviewed changes

michael-johnston commented Apr 28, 2026

View reviewed changes

Comment thread docs/design/benchmark_system.md Outdated

michael-johnston requested a review from AlessandroPomponio April 28, 2026 12:15

michael-johnston commented Apr 28, 2026

View reviewed changes

Comment thread docs/design/benchmark_system.md Outdated

michael-johnston commented Apr 28, 2026

View reviewed changes

Comment thread docs/design/benchmark_system.md Outdated

Apply suggestions from code review

3c386aa

Co-authored-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com> Signed-off-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com>

AlessandroPomponio previously approved these changes Apr 29, 2026

View reviewed changes

Merge branch 'main' into maj_benchmark_design

f21be10

christian-pinto requested changes Apr 29, 2026

View reviewed changes

Comment thread docs/design/benchmark_system.md Outdated

Comment thread docs/design/benchmark_system.md Outdated

Comment thread docs/design/benchmark_system.md Outdated

Comment thread docs/design/benchmark_system.md Outdated

Comment thread docs/design/benchmark_system.md Outdated

Apply suggestions from code review

5b764a1

Co-authored-by: Christian Pinto <55737893+christian-pinto@users.noreply.github.com> Signed-off-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com>

michael-johnston dismissed AlessandroPomponio’s stale review via 5b764a1 April 29, 2026 15:36

michael-johnston commented Apr 29, 2026

View reviewed changes

Comment thread docs/design/benchmark_system.md Outdated

michael-johnston commented Apr 29, 2026

View reviewed changes

Comment thread docs/design/benchmark_system.md Outdated

michael-johnston commented Apr 29, 2026

View reviewed changes

Comment thread docs/design/benchmark_system.md Outdated

michael-johnston commented Apr 29, 2026

View reviewed changes

Comment thread docs/design/benchmark_system.md Outdated

Apply suggestions from code review

9e2cc47

Co-authored-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com> Signed-off-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com>

michael-johnston requested a review from AlessandroPomponio April 30, 2026 12:45

michael-johnston requested a review from christian-pinto April 30, 2026 12:45

christian-pinto previously requested changes Apr 30, 2026

View reviewed changes

michael-johnston and others added 2 commits April 30, 2026 15:40

Apply suggestions from code review

8e7b1f5

Co-authored-by: Christian Pinto <55737893+christian-pinto@users.noreply.github.com> Signed-off-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com>

docs(design): updates

b5f50d2

michael-johnston requested a review from christian-pinto May 1, 2026 15:03

michael-johnston enabled auto-merge (squash) May 2, 2026 13:57

AlessandroPomponio reviewed May 4, 2026

View reviewed changes

Comment thread docs/design/benchmark_system.md

Comment thread docs/design/benchmark_system.md Outdated

Comment thread docs/design/benchmark_system.md Outdated

AlessandroPomponio mentioned this pull request May 4, 2026

feat: implement execution of benchmark jobs on GitHub trigger #69

Open

Apply suggestions from code review

76dee92

Co-authored-by: Alessandro Pomponio <10339005+AlessandroPomponio@users.noreply.github.com> Signed-off-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com>

christian-pinto requested changes May 5, 2026

View reviewed changes

Apply suggestions from code review

509aa2a

Co-authored-by: Christian Pinto <55737893+christian-pinto@users.noreply.github.com> Signed-off-by: Michael Johnston <66301584+michael-johnston@users.noreply.github.com>

	nexus package [they can add it.](#defining-a-benchmark-to-use-for-a-model). The
	nexus package [they can add it.](#defining-the-benchmark-experiment-packages-used-by-a-nexus-package). The

Conversation

michael-johnston commented Apr 28, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michael-johnston commented Apr 29, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michael-johnston commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DRL-NextGen commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michael-johnston commented Apr 29, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

christian-pinto May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

christian-pinto May 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

michael-johnston commented Apr 29, 2026 •

edited

Loading

DRL-NextGen commented Apr 29, 2026 •

edited

Loading

christian-pinto May 5, 2026 •

edited

Loading