fix(activation): always INSTALL iceberg even when bundled#646
Merged
Conversation
LoadExtensions skips INSTALL whenever a .duckdb_extension binary is sitting in /app/extensions (preseeded from the Dockerfile). That works for the PostHog-fork extensions (httpfs, ducklake) and the stable stock ones (json, postgres_scanner): LOAD against the seeded extension_directory file just works. It does NOT work for iceberg. After #645 bundled iceberg, the bundle-skip turned db.Exec("LOAD iceberg") into the same ~60s silent hang the bundle was meant to fix: the bundled binary is in the cache, but without a prior INSTALL DuckDB's internal extension metadata leaves LOAD blocked indefinitely (observed live in mw-dev — worker logs the load-iceberg-extension step start and then no further activity until the activate-tenant deadline kills the worker). Special-case iceberg in shouldInstallExtension so INSTALL runs even when bundled. INSTALL with the binary already in extension_directory is a cheap no-op — DuckDB sees the cached file and skips the CDN download — so the bundle benefit is preserved while LOAD now finds the extension already installed. The upstream-overwrite risk that motivates skipping INSTALL for the PostHog forks doesn't apply: we bundle the same stock iceberg build the DuckDB repository would serve.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
After #645 bundled the iceberg extension into the worker image, iceberg-only tenant activation still hangs past the activate-tenant deadline. The bundle is correct —
/data/extensions/v1.5.3/linux_arm64/iceberg.duckdb_extensionis in place (the bootstrap seeded it from/app/extensions/...) — butdb.Exec("LOAD iceberg")blocks for the full ~60s with no progress and no DuckDB-level log, until the CP cancels and the worker is retired.Why:
LoadExtensionsskipsINSTALLwheneverhasBundledExtensionBinaryreturns true:That works for
httpfs,ducklake,json,postgres_scanner— LOAD against the seededextension_directoryfile just works. It does not work for iceberg: without a priorINSTALL, DuckDB's internal extension metadata leavesLOAD icebergblocked indefinitely, even with the binary already in the cache. So the veryINSTALLthe bundle was meant to spare us turns into the same silent stall when the bundle deletes the call.Live evidence on mw-dev (
ben-ext-ice, build2f2f38d= #645):Fix
Special-case
iceberginshouldInstallExtensionsoINSTALLruns even when the binary is bundled. With the file already inextension_directory,INSTALLis a cheap no-op — DuckDB sees the cached file and skips the CDN download — so the bundle benefit is preserved (no first-load fetch from the CDN) whileLOADnow finds the extension already installed.The upstream-overwrite risk that motivates skipping
INSTALLfor the PostHog forks (httpfs,ducklake) doesn't apply: we bundle the same stock iceberg build the DuckDB repository would serve.Test plan (mw-dev after deploy)
ben-ext-iceactivates within the deadline.SELECT 1succeeds;CREATE TABLE iceberg.public.<t> / INSERT / SELECTround-trips.🤖 Generated with Claude Code