feat: Lakeflow Jobs Plugin

[atilafassina]

Important

to maintain contribution history this PR should be rebased when merged
(don't merge it if it has more than 2 commits)

Summary

Resource-scoped jobs plugin following the files plugin pattern. Jobs are configured as named resources discovered from environment variables at startup.

Design

• Resource-scoped: Only configured jobs are accessible — this is not an open SDK wrapper
• Env-var discovery: Jobs are discovered from DATABRICKS_JOB_<KEY> env vars (e.g. DATABRICKS_JOB_ETL=123)
• Single-job shorthand: DATABRICKS_JOB_ID maps to the "default" key
• Manifest declares job resources with CAN_MANAGE_RUN permission
• Works with databricks apps init --features jobs

API

// Trigger a configured job
const { run_id } = await appkit.jobs("etl").runNow();

// Trigger and wait for completion
const run = await appkit.jobs("etl").runNowAndWait();

// OBO access
await appkit.jobs("etl").asUser(req).runNow();

// List recent runs
const runs = await appkit.jobs("etl").listRuns({ limit: 10 });

// Single-job shorthand
await appkit.jobs("default").runNow();

Files changed

• plugins/jobs/manifest.json — declares job resource with CAN_MANAGE_RUN permission
• plugins/jobs/types.ts — JobAPI, JobHandle, JobsExport, IJobsConfig types
• plugins/jobs/plugin.ts — JobsPlugin with discoverJobs(), getResourceRequirements(), resource-scoped createJobAPI()
• plugins/jobs/index.ts — barrel exports
• connectors/jobs/client.ts — listRuns now respects limit parameter
• plugins/jobs/tests/plugin.test.ts — 32 tests covering discovery, resource requirements, exports, OBO, multi-job, and auto-fill

Documentation safety checklist

• Examples use least-privilege permissions
• Sensitive values are obfuscated
• No insecure patterns introduced

Reopened from #223 — moved head branch from fork to upstream to fix CI secrets access.

demo

lakeflow-jobs.mp4

[Copilot]

Pull request overview

Adds a new resource-scoped “jobs” plugin to @databricks/appkit, following the existing “files” plugin pattern: jobs are discovered from environment variables at startup and exposed via a keyed accessor API, with HTTP routes for triggering and monitoring runs.

Changes:

• Introduces plugins/jobs (manifest, defaults, params mapping, types, plugin implementation, and extensive tests).
• Adds connectors/jobs with telemetry + cancellation support and updates exports/docs to surface the new plugin and types.
• Updates templates and generated API docs/sidebars to include the new Jobs plugin.

Reviewed changes

Copilot reviewed 24 out of 25 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
template/appkit.plugins.json	Adds “jobs” to plugin template metadata + resource description.
pnpm-lock.yaml	Locks new dependency (zod@4.3.6) and related lockfile updates.
packages/appkit/src/plugins/jobs/types.ts	Public types for Jobs plugin API/config (JobAPI/JobHandle/IJobsConfig).
packages/appkit/src/plugins/jobs/plugin.ts	Core JobsPlugin: env discovery, dynamic resource requirements, API methods, HTTP routes.
packages/appkit/src/plugins/jobs/params.ts	TaskType-based param mapping into SDK request fields.
packages/appkit/src/plugins/jobs/defaults.ts	Execution defaults for read/write/stream operations.
packages/appkit/src/plugins/jobs/manifest.json	Plugin manifest + config schema + baseline resource definition.
packages/appkit/src/plugins/jobs/index.ts	Barrel exports for Jobs plugin/types.
packages/appkit/src/plugins/jobs/tests/plugin.test.ts	Comprehensive unit tests for discovery, API, routes, and validation.
packages/appkit/src/plugins/index.ts	Exposes jobs from the plugins barrel.
packages/appkit/src/index.ts	Exposes jobs and related public types/configs from package root.
packages/appkit/src/connectors/jobs/client.ts	JobsConnector SDK wrapper with telemetry instrumentation + limit handling.
packages/appkit/src/connectors/jobs/types.ts	Connector config type (timeout/telemetry).
packages/appkit/src/connectors/jobs/index.ts	Connector barrel exports.
packages/appkit/src/connectors/index.ts	Exposes jobs connector from connectors barrel.
packages/appkit/package.json	Adds zod dependency for runtime param schemas + JSON schema generation.
docs/docs/api/appkit/typedoc-sidebar.ts	Adds new Jobs docs entries to sidebar.
docs/docs/api/appkit/*.md	Adds generated API docs pages for Jobs plugin types.
docs/docs/api/appkit/index.md	Adds Jobs-related types to API index list.
docs/docs/api/appkit/Interface.BasePluginConfig.md	Updates “Extended by” list to include IJobsConfig.

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[pkosiec]

How it is different from @keugenek PR here: #221?

If you based on top of Evgenii's work, maybe it's worth to recognize his contribution? What I'm thinking is either:

• have 2 commits on this PR: Evgenii's base one and yours (with all changes, PR review fixes etc.), and temporarily enable the "Rebase & Merge" option to have 2 commits on main
• or, merge Evgenii's PR as is, right before this PR (so we'd need to wait for this PR to be approved first)

WDYT?

[atilafassina]

@pkosiec

How it is different from @keugenek PR here: #221?
If you based on top of Evgenii's work, maybe it's worth to recognize his contribution? What I'm thinking is either:

• have 2 commits on this PR: Evgenii's base one and yours (with all changes, PR review fixes etc.), and temporarily enable the "Rebase & Merge" option to have 2 commits on main
• or, merge Evgenii's PR as is, right before this PR (so we'd need to wait for this PR to be approved first)

WDYT?

I got stuck trying to update his PR because it was opened from a fork, it's noted in the PR description:

Reopened from #223 — moved head branch from fork to upstream to fix CI secrets access.

So, I couldn't get a clean CI to merge it, and if I merged to main, it would trigger a release.
You'll notice that in the commit history the very first commit is his, that's his work that I iterated on top of.

It's also marked as a comment on the original PR

#223 (comment)

[pkosiec]

So, I couldn't get a clean CI to merge it, and if I merged to main, it would trigger a release.

It wouldn't trigger a release currently (finalizing releases is manual) so there shouldn't be a problem with that 👍 but I'd wait until your PR (on top of this one) is ready to merge, and I'd merge both one after another. What do you think?