Description
On a GitHub event, associated with a PR, we want benchmarks associated with the package in that PR to be executed on our infrastructure
Motivation
This featured is defined in the benchmark requirements doc (docs/requirements/benchmark.md)
Proposed Solution
- User opens PR
- At a point in PR reviewers request the benchmark via GitHub command e.g. /run-benchmark
- This triggers a GitHub action which creates a benchmark "job" specification, which then is pending
- A Github Runner, executing on our infrastructure, on a schedule, retrieves pending benchmark jobs
- The runner pulls the relevant PR and constructs using the job spec the ado explore operation and runs it
What needs to be placed in the job specification and what can be inferred at ado job creation time TBD
Alternatives Considered
An alternative which
- PR is merged
- PR merge GitHub action triggers - on of these is run-benchmark, which creates the task
- the GitHub runner on our arch is performing a long-polling on the benchmark tasks, it sees the task and pulls it
- Tasks are queued so the runner will pull max one at each poll
- (As step 5 above)
The event could also be on deployment event, release event etc.
Description
On a GitHub event, associated with a PR, we want benchmarks associated with the package in that PR to be executed on our infrastructure
Motivation
This featured is defined in the benchmark requirements doc (docs/requirements/benchmark.md)
Proposed Solution
What needs to be placed in the job specification and what can be inferred at ado job creation time TBD
Alternatives Considered
An alternative which
The event could also be on deployment event, release event etc.