[DRAFT] # PR: Support Repository-based MCP Plugins & Options in Codex CLI Generator

[silviojr]

This PR extends the Codex CLI generator to support cloning plugin repositories containing bundled MCP servers (such as the Data Agent Kit), auto-enabling them in config.toml, and passing configuration parameters (userConfig) to them.

1. Refactored Configuration Serialization

• Problem: Previously, config.toml was written immediately on initialization with only inline mcp_servers configuration. This prevented adding plugins cloned dynamically during setup.
• Solution: Refactored CodexCliGenerator to collect configurations (inline servers, repository-based servers, and cloned skills/plugins) into internal state dictionaries (self.inline_mcp_servers and self.enabled_plugins) first, and deferred writing the config.toml file to the end of the _setup method.

2. Supported List-Based mcp_servers Repositories

• Decision: Adopted the same configuration schema as ClaudeCodeGenerator to maintain cross-agent testing consistency.
• Implementation: Implemented _install_mcp_servers_from_repo which clones the plugin repository, registers it in marketplace.json, and records its ID and parameters in the enabled plugins list.

3. Enabled Plugins in config.toml

• Decision: Registered plugins in marketplace.json must be explicitly marked as enabled in config.toml to be loaded by Codex CLI.
• Implementation: Wrote [plugins."<plugin-id>"] sections with enabled = true and options inside config.toml. The plugin_id is constructed using the plugin-name@marketplace-name pattern.

4. Supported Plugin Options & Variables (userConfig)

• Problem: DAK's MCP servers rely on GCP project/location parameters to run queries (e.g. ${user_config.PROJECT_ID}).
• Solution: Extracted the config blocks from the YAML settings, passed them to self.enabled_plugins, and serialized them as inline TOML tables under options = { ... }.

Testing & Verification

1. Test Scenario Setup

We executed the evaluation pipeline using the Bullseye_PCA scenario:

• Run Config: Bullseye_PCA_codex_config.yaml (local)
• Model Config: codex_cli_dak_model.yaml (local)
  Specifies cloning and loading the dak plugin from its repository:

  setup:
    mcp_servers:
      - action: install_from_repo
        url: "https://github.com/gemini-cli-extensions/data-agent-kit-starter-pack.git"
        plugin: "dak"
        config:
          PROJECT_ID: "datagravity-demo"
          GCP_REGION: "us-central1"
          BIGQUERY_LOCATION: "US"

• Dataset Config: Bullseye_PCA.json (local)
  Requests creating a notebook and running a sample BigQuery query.

2. Sandbox Verification

After executing the setup sequence, we inspected the sandboxed directory .venv/fake_home_codex and verified:

1. Marketplace Configuration: The marketplace manifest file at .agents/plugins/marketplace.json was correctly created, pointing to the local cloned repository with the relative path starting with ./ (e.g. "path": "./.codex/plugins/data-agent-kit-starter-pack").
2. Plugin Registration & Installation: Running codex plugin list in the sandbox confirmed the plugin status was successfully transitioned to:
   dak@evalbench-local-marketplace | installed, enabled
3. Skills Installation: The 19 skills bundled inside the starter pack repository (such as developing-with-bigquery, notebook-guidance, etc.) were successfully copied to .codex/skills/.

3. Execution Verification

We executed the run and examined the outputs in results/aadf29ef-3145-4d8e-aca1-9e9bd09b58a9:

• Goal Completion Evaluation: PASS (100.0%)
• Actual Tool Execution: The agent successfully executed actual tool calls to the MCP server rather than falling back to text generation:
  • notebook__create_notebook (succeeded)
  • notebook__list_cells (succeeded)
  • notebook__insert_cell (succeeded, twice)
• Stats and Logs: Verified that all tool calls resolved and completed successfully with no goal/thread error logs in Codex CLI.

[IsmailMehdi]

Leaving the three blocker-level items inline. Full review posted separately (subprocess env / TOML escaping / implicit skills-config behavior change). Non-blocking items are in that comment, not repeated here.

[IsmailMehdi]

Blocker. env=self.env will make this fail to find npm.

subprocess.run(env=...) replaces the process environment; it does not merge. self.env is only populated with HOME, CODEX_HOME, and (optionally) OPENAI_API_KEY — no PATH. So npm isn't resolvable and this exits 127 / FileNotFoundError, and the plugin never transitions to installed, enabled (which is the whole point of adding this subprocess).

The other subprocess call sites in this file already do the right thing — _setup_skills and _install_mcp_servers_from_repo both build:

setup_env = os.environ.copy()
setup_env.update(self.env)

and pass that in. Use the same pattern here (either recompute locally or take setup_env as a parameter from the callers that already have it).

[IsmailMehdi]

Should fix before merge. plugin_id isn't TOML-escaped, which produces invalid TOML if plugin_name or marketplace_name ever contains a " or \. It also opens a small TOML-injection surface on user-authored configs (e.g. a plugin_name of foo"] evil="bar would break out of the section).

_toml_key (line 622) already handles this exact case — it json.dumps keys that aren't bare-key-safe, and @ isn't bare-key-safe, so it round-trips dak@evalbench-local-marketplace to "dak@evalbench-local-marketplace" unchanged. Just reuse it:

lines.append(f"[plugins.{self._toml_key(plugin_id)}]")

[IsmailMehdi]

Should fix before merge — or at minimum verify. This is a silent behavior change for every existing skills: config that uses action: install_from_repo.

Before this PR, cloning a skill repo only registered it in the marketplace and copied SKILL folders — nothing was ever added to config.toml. After this PR, every such repo also gets a [plugins."<name>@evalbench-local-marketplace"] section with enabled = true written to config.toml (via self.enabled_plugins).

Whether that breaks anything depends on how Codex CLI handles an enabled = true on a plugin with no runtime component. Two ways forward:

1. Confirm against an existing skills-only config that Codex is tolerant, and add a one-line comment here explaining why the skills and MCP paths deliberately share the enable-in-config behavior.
2. If it's not safe / not intended, scope the enabled_plugins.setdefault(...) call to _install_mcp_servers_from_repo only and leave _setup_skills doing what it did before.

Either is fine; the current state (silent change with no comment) is the risky option.