Skip to content

refactor(providers)!: make provider profiles import-only #3299

Description

@feloy

Sub-issue of #3171 (step 3 of its Suggested Work Breakdown). Coordinated pre-0.1.0
breaking change under #2565.

User Story

As an OpenShell platform operator, I want the gateway's provider-profile catalog to
contain only the profiles I explicitly imported or configured, so that the profiles
my sandboxes can use match the images I actually run instead of a catalog compiled
into the release binary.

As a product builder, I want the profile YAML in providers/ to be reviewable,
copyable examples rather than platform defaults, so that I can adapt binary
selectors and endpoints to my own image without weakening least privilege or
waiting on a core release.

Problem Statement

OpenShell compiles fifteen provider profile YAML files into every release binary and
selects them by default. BUILT_IN_PROFILE_YAMLS in
crates/openshell-providers/src/profiles.rs:28 embeds providers/*.yaml with
include_str!, builtin_profiles() parses them into a process-wide catalog, and
GatewayConfig::default() selects [{ type = "builtin" }, { type = "user" }]
(crates/openshell-core/src/config.rs:793). A fresh gateway therefore always
publishes a catalog it was never configured with.

Several consumers depend on that compiled catalog rather than on the gateway's
active one:

  • BuiltinProviderProfileSource declares allow_empty() == false, so the built-in
    source is structurally required to be non-empty.
  • The CLI's credential-environment suggestions scan builtin_profiles() directly in
    the client process (crates/openshell-cli/src/commands/common.rs:783), so
    suggestions come from the binary, not from the gateway the CLI is talking to.
  • Command-to-provider inference resolves through detect_provider_from_command()
    normalize_provider_type(), a hardcoded alias table
    (crates/openshell-providers/src/lib.rs:140) that is a second, independent copy
    of the built-in catalog's identifiers and aliases.

The profiles themselves are not image-neutral. Their binary selectors name paths
that exist in the OpenShell Community image — /sandbox/.venv, /app/.venv,
/usr/lib/node_modules, /bin/bash — so changing the sandbox image can silently
make a profile inert while the catalog still advertises it as available.

The import-first foundation already exists: user-managed platform- and
workspace-scoped profile storage, provider list-profiles, provider import-profiles, export, lint, update, revisioned sources, configurable source
composition, and interceptor-vended catalogs. What is missing is making the
explicitly configured catalog the only catalog.

Impact / Why This Matters

Today an operator who runs a non-community image sees a full catalog of profiles,
attaches one, and gets a provider whose binary selectors match nothing in their
image. The failure surfaces later as a denied connection or an uninjected
credential rather than as a configuration error, and the catalog gives no signal
that the profile was authored for a different filesystem layout.

The workaround is to broaden the shipped profiles' binary paths until they match
every image in use. That trades away binary-scoped least privilege, which is the
control that makes credential injection safe, and it creates an unbounded
compatibility promise across arbitrary third-party images.

Built-in identifiers also behave as a reserved namespace with unusual semantics: a
user-managed profile named github does not replace the built-in one but shadows it
into static_fallback (crates/openshell-server/src/provider_profile_sources.rs:669),
so an operator who imports a corrected github profile still has the shipped
definition resident and reachable on delete. Removing the embedded catalog is the
only change that makes the catalog match what the operator configured.

Beyond correctness, the compiled catalog is a release compatibility surface. Every
endpoint, credential name, and binary path in providers/ is effectively part of the
0.1.0 public contract while the project's direction (#3171) is to keep the core
runtime neutral.

Proposed Design

Catalog composition

The gateway's default provider_profile_sources becomes [{ type = "user" }]. No
profile YAML is compiled into a release binary, and the builtin source type is
removed from the configuration schema. A gateway that starts with no imported
profiles serves an empty catalog successfully; an empty catalog is a valid state,
not a startup failure.

Operators obtain profiles by importing them:

openshell provider import-profiles --platform ./providers/github.yaml

Because nothing reserves the identifiers any more, github, pypi, anthropic
and the rest import at their canonical IDs at platform or workspace scope, and an
imported profile is the profile — no shadowed static fallback behind it.

Resolution and diagnostics

Provider creation, sandbox attachment, credential-discovery suggestions, and
command auto-inference resolve only against the gateway's active catalog. The CLI
stops consulting any local profile table: credential suggestions come from the
catalog the connected gateway publishes, and command inference either resolves
through the catalog or is dropped in favor of an explicit --provider.

When a required profile is absent, the operation fails with a bounded, actionable
message naming the missing profile ID and the import command, rather than silently
falling back to a compiled definition or an alias guess.

Upgrade behavior

Existing provider instances whose profile is absent after upgrade must reach a
documented, diagnosable state rather than failing opaquely at sandbox creation. The
upgrade path is: export the profiles the deployment relies on before upgrading, or
copy them from the providers/ directory of the matching release tag, then import
them at the scope the providers use.

Scope boundary

This issue removes embedded profile data. It does not remove provider plugin
implementations under crates/openshell-providers/src/providers/ — Google Cloud
metadata emulation, Vertex project/region projection, and similar type-specific
behavior stay implemented, but their activation requires an imported profile whose
ID selects them. Inventorying those adapters and deciding which become expressible
in the profile contract is #3171 step 5 and belongs in a separate issue.

Retired-alias plugins that no longer have a corresponding example profile
(gitlab.rs, opencode.rs, outlook.rs, generic.rs) should be confirmed dead
and removed or left for the step 5 inventory, not silently retained.

Acceptance Criteria

  • No provider profile YAML is compiled into any OpenShell release binary;
    include_str! of providers/*.yaml is gone and builtin_profiles() is
    removed from the public API of openshell-providers.
  • The default provider_profile_sources is [{ type = "user" }], and the
    builtin source type is rejected by gateway configuration with a message
    pointing at profile import.
  • A gateway started with no configuration and no imported profiles reaches
    ready and serves an empty profile catalog; provider list-profiles returns an
    empty list rather than an error.
  • Importing providers/github.yaml, providers/pypi.yaml, and the remaining
    example files at platform scope succeeds using their canonical IDs, and the
    imported profile is the only definition for that ID — no static fallback
    remains behind it.
  • Provider creation, sandbox provider attachment, credential-discovery
    suggestions, and command auto-inference resolve exclusively against the
    gateway's active catalog.
  • A provider or sandbox referencing an absent profile fails with a bounded
    diagnostic that names the profile ID and the import command; no compiled
    definition or hardcoded alias satisfies the reference.
  • The CLI contains no local provider profile table; normalize_provider_type's
    hardcoded alias map is removed or reduced to catalog-driven resolution.
  • Provider plugin implementations activate only for an imported profile whose ID
    selects them, and no plugin activates from a compiled profile definition.
  • Every file under providers/ documents its expected client binary identities,
    compatible reference image or installation layout, credential scope, endpoint
    access, and a smoke test; all files still pass provider lint-profiles.
  • docs/reference/gateway-config.mdx documents the new default and the removal
    of the builtin source type; an upgrade note covers export-before-upgrade,
    importing from the release tag's providers/ directory, and the behavior of
    existing provider instances whose profile is missing.
  • Tests prove a fresh gateway has an empty catalog, that import at canonical IDs
    works, that absent-profile references fail closed with the documented
    diagnostic, and that no test fixture depends on a compiled catalog.

Alternatives Considered

Keep the built-ins but deselect them by default. This removes automatic
selection while still embedding the catalog in the binary, reserving its
identifiers, driving CLI inference, and implying a release compatibility surface for
every endpoint and binary path. It leaves the shadowing static_fallback semantics
in place. Import-only is the cleaner platform boundary.

Broaden the built-in profiles' binary selectors to match more images. This
creates an unbounded compatibility promise across arbitrary third-party images and
encourages wildcard binary patterns, weakening the binary-scoped least privilege
that makes credential injection safe.

Ship the profiles as a bundled-but-importable seed that auto-imports on first
start.
Auto-import reproduces the current problem with extra steps: a fresh
gateway still has a catalog nobody configured, and the profiles still carry image
assumptions the operator did not review. It also introduces first-start ordering and
idempotence questions across HA replicas.

Remove the provider plugin implementations along with the data. Profile data and
protocol behavior have different lifetimes. Refresh strategies, credential brokers,
inference backends, and metadata emulators can be genuine platform capabilities; the
correct treatment is explicit activation plus a separate classification pass
(#3171 step 5), not deletion in the same change.

Agent Investigation

Explored the current provider profile stack on main (5b9daab):

  • Embedded catalog: crates/openshell-providers/src/profiles.rs:28
    (BUILT_IN_PROFILE_YAMLS, 15 include_str! entries, lines 29–43) parsed by
    builtin_profiles() at profiles.rs:3236 behind a OnceLock.
  • Default selection: crates/openshell-core/src/config.rs:793 sets
    provider_profile_sources = [Builtin, User]; the variants are defined at
    config.rs:454.
  • Source registry: crates/openshell-server/src/provider_profile_sources.rs.
    BuiltinProviderProfileSource (line 70) is user_managed() == false and
    allow_empty() == false; UserProviderProfileSource (line 108) is already
    allow_empty() == true, and it already reads platform- and workspace-scoped
    stored profiles. A user-only source list therefore looks structurally viable today
    — the work is changing the default and removing the builtin variant, not building
    new storage.
  • Shadowing, not reservation: provider_profile_sources.rs:669 moves a built-in
    entry into static_fallback when a user-managed profile shares its ID, so
    canonical IDs are importable today but the compiled definition stays resident.
    handle_update_provider_profiles still rejects updates targeting a built-in
    (crates/openshell-server/src/grpc/provider.rs, test at line 5408).
  • CLI coupling: crates/openshell-cli/src/commands/common.rs:783 scans
    builtin_profiles() in-process for credential suggestions;
    crates/openshell-providers/src/lib.rs:140 normalize_provider_type() is a
    hardcoded alias→ID map (ghgithub, claudeclaude-code, vertex
    google-vertex-ai, …) feeding detect_provider_from_command() at line 161, used
    by crates/openshell-cli/src/run.rs:562.
  • Test coupling: builtin_profiles() is called from
    openshell-cli/tests/{sandbox_create_lifecycle,provider_commands,ensure_providers}_integration.rs
    and openshell-server/src/grpc/sandbox.rs:3092; these fixtures need to import
    example YAML instead.
  • Plugins are separable: crates/openshell-providers/src/providers/ holds 14
    plugin modules keyed by profile ID through inject_env_for_profile_id
    (lib.rs:127), which already resolves without alias normalization. Four of them
    (gitlab, opencode, outlook, generic) correspond to IDs
    normalize_provider_type explicitly retired and have no YAML in providers/.
  • Docs: docs/reference/gateway-config.mdx:169 and :399 document the
    builtin + user default and the fail-closed rules for the source list.

Not investigated: the behavior of existing persisted provider instances when their
profile disappears, which needs a decision on whether the gateway rejects at
attach time or marks the provider degraded at read time.

Checklist

  • I've reviewed existing issues and the architecture docs
  • This is a design proposal, not a "please build this" request

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions