Enable AL-LSP for evaluation

[haoranpb]

Enable latest AL-LSP for evaluation in BC-Bench, it is also shipped from altool but doesn't require a running container.

Also introduce a scripts/Download-BCSymbols.ps1 script to download all the artifacts using BCContainerHelper, this is mostly used for local development.

Use the plugin pattern for Claude Code, it is also supported by GitHub Copilot.

Currently, only enabled integration with Copilot.

Local Development

First follow CONTRIBUTING.md

# Only need if altool is not installed
dotnet tool install -g Microsoft.Dynamics.BusinessCentral.Development.Tools --version 18.0.36.64936-beta

# Only need to run once per instance/task
.\scripts\Download-BCSymbols.ps1 -Category bug-fix -InstanceId microsoft__BCApps-5633

uv run bcbench run copilot microsoft__BCApps-5633 --category bug-fix --al-lsp

TODO

• Integration with Claude Code

[Copilot]

Pull request overview

This PR adds AL Language Server Protocol (AL-LSP) support to BC-Bench evaluations (currently wired for GitHub Copilot CLI) and factors shared AL tool path/symbol-resolution logic into reusable helpers, including a new PowerShell script to prefetch BC symbols into the BCContainerHelper artifact cache for local development.

Changes:

• Add --al-lsp flag to Copilot run/evaluate flows and record al_lsp_enabled in ExperimentConfiguration + GitHub job summary output.
• Introduce project-level Copilot LSP config generation (.github/lsp.json) using either container compiler symbols or the BC artifact cache (container-free).
• Add shared altool_paths helpers and a scripts/Download-BCSymbols.ps1 helper script to download the required BC artifacts/symbols.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/test_mcp_config.py	Updates imports to use the new shared AL tool helper module.
tests/test_lsp_config.py	Adds unit coverage for LSP config generation and symbol-source selection behavior.
tests/test_experiment_configuration.py	Extends experiment configuration tests for the new al_lsp_enabled flag.
src/bcbench/types.py	Adds al_lsp_enabled to ExperimentConfiguration and includes it in is_empty().
src/bcbench/results/display.py	Surfaces AL-LSP enabled/disabled state in GitHub job summaries.
src/bcbench/commands/run.py	Adds --al-lsp CLI option for Copilot “run” command.
src/bcbench/commands/evaluate.py	Adds --al-lsp CLI option for Copilot “evaluate” command.
src/bcbench/agent/shared/mcp.py	Moves AL compiler path/runtime helpers into altool_paths and reuses them.
src/bcbench/agent/shared/lsp.py	New: writes Copilot project-level LSP config and resolves symbols from container/artifact cache.
src/bcbench/agent/shared/altool_paths.py	New: shared helpers for runtime version, assembly probing paths, and artifact-cache symbol resolution.
src/bcbench/agent/shared/init.py	Exports build_lsp_config alongside existing shared agent helpers.
src/bcbench/agent/copilot/agent.py	Wires AL-LSP setup into Copilot agent execution and records it in experiment metadata.
scripts/Download-BCSymbols.ps1	New: downloads artifacts/symbols for a dataset entry into the BCContainerHelper cache.
.github/workflows/copilot-evaluation.yml	Adds al-lsp workflow input, installs AL tool when LSP/MCP enabled, and passes --al-lsp.

[kalberes]

Should probably be configurable and I would add the latest tool released >= 2026.05.27 as there were some serious refactorings done