Skip to content

feat(python-notebook-migration): add database tables for Notebook Migration tool #5055

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
zyratlo wants to merge 10 commits into
base: main
Choose a base branch
from
Open

feat(python-notebook-migration): add database tables for Notebook Migration tool #5055

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
e60310e
added notebook and workflow_notebook_mapping tables
zyratlo May 13, 2026
c8fa486
Merge branch 'main' into migration-tool-database-tables
mengw15 May 18, 2026
986bd38
Merge branch 'main' into migration-tool-database-tables
zyratlo May 18, 2026
9c63915
removed "\c texera_db" in sql file and added changeSet element in acc…
zyratlo May 18, 2026
6deb604
added newline and changed changeSet id to match update SQL file
zyratlo May 18, 2026
4387ad0
Merge branch 'main' into migration-tool-database-tables
zyratlo May 20, 2026
914f6b5
Merge branch 'main' into migration-tool-database-tables
mengw15 May 25, 2026
f4e019a
Merge branch 'main' into migration-tool-database-tables
zyratlo May 26, 2026
bc8381c
remove DROP TABLE statements from sql/updates/23.sql and switch to CR…
zyratlo May 26, 2026
89015d3
Merge branch 'main' into migration-tool-database-tables
zyratlo May 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions sql/changelog.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,11 @@
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.9.xsd">

<!-- Add notebook and workflow_notebook_mapping tables -->
<changeSet id="23" author="zyratlo">
<sqlFile path="sql/updates/23.sql"/>
</changeSet>

<!-- example changeSet
<changeSet id="1" author="author">
<sqlFile path="sql/updates/1.sql"/>
Expand Down
24 changes: 24 additions & 0 deletions sql/texera_ddl.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,8 @@ DROP TABLE IF EXISTS dataset_user_likes CASCADE;
DROP TABLE IF EXISTS dataset_view_count CASCADE;
DROP TABLE IF EXISTS site_settings CASCADE;
DROP TABLE IF EXISTS computing_unit_user_access CASCADE;
DROP TABLE IF EXISTS notebook CASCADE;
DROP TABLE IF EXISTS workflow_notebook_mapping CASCADE;

-- ============================================
-- 4. Create PostgreSQL enum types
Expand Down Expand Up @@ -435,6 +437,28 @@ CREATE TABLE IF NOT EXISTS computing_unit_user_access
FOREIGN KEY (uid) REFERENCES "user"(uid) ON DELETE CASCADE
);

-- notebook table
CREATE TABLE IF NOT EXISTS notebook
(
nid SERIAL NOT NULL PRIMARY KEY,
wid INT NOT NULL,
notebook JSONB NOT NULL,
FOREIGN KEY (wid) REFERENCES workflow(wid) ON DELETE CASCADE
);
Comment on lines +441 to +447
Copy link
Copy Markdown
Contributor

@mengw15 mengw15 May 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cardinality question: nothing in the schema prevents inserting two notebook rows for the same wid, but the parent issue (#4301, demo #5 "when the user reopens a workflow that was generated from a notebook, it will also reopen the notebook") reads like a 1:1 relationship. If a workflow is supposed to have at most one notebook, would a UNIQUE (wid) on notebook (or making wid the PK in place of nid) be safer than relying on application code to enforce it?

Copy link
Copy Markdown
Contributor Author

@zyratlo zyratlo May 26, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the current prototype implementation, you are correct in that a workflow will only ever have one notebook (a 1:1 relationship). However, the reason the schema is designed this way is because we wanted to allow future work to make the notebook editable and savable, which would create the situation where multiple notebooks (or versions of the same notebook) are linked to the same workflow. This is why we didn't make wid UNIQUE here.

Alternatively, another option is that we can make wid UNIQUE in this PR, and when the aforementioned future work is done then we can make the schema change to allow multiple notebooks for a workflow. What do you think?

Comment on lines +441 to +447
Copy link
Copy Markdown
Contributor

@mengw15 mengw15 May 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main read pattern for notebook seems to be "given a workflow, find its notebook" (e.g., reopening a workflow → load its notebook). Postgres doesn't auto-create an index on FK columns, so this lookup would currently sequential-scan the table. Worth a CREATE INDEX ON notebook(wid)? (If a UNIQUE(wid) is added per the previous comment, that already creates an index and this is moot.)

Copy link
Copy Markdown
Contributor Author

@zyratlo zyratlo May 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that indexing wid would help here. Since this is tied to the discussion on whether to make wid UNIQUE, I will wait for our decision on that before making changes. If we decide to make wid UNIQUE, then no further work needs to be done here. If we keep wid non-UNIQUE, I will add the CREATE INDEX.


-- workflow_notebook_mapping table
CREATE TABLE IF NOT EXISTS workflow_notebook_mapping
(
wid INT NOT NULL,
vid INT NOT NULL,
nid INT NOT NULL,
mapping JSONB NOT NULL,
PRIMARY KEY (wid, vid, nid),
FOREIGN KEY (wid) REFERENCES workflow(wid) ON DELETE CASCADE,
FOREIGN KEY (vid) REFERENCES workflow_version(vid) ON DELETE CASCADE,
FOREIGN KEY (nid) REFERENCES notebook(nid) ON DELETE CASCADE
);

-- START Fulltext search index creation (DO NOT EDIT THIS LINE)
CREATE EXTENSION IF NOT EXISTS pgroonga;

Expand Down
49 changes: 49 additions & 0 deletions sql/updates/23.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

-- ============================================
-- 1. Connect to the texera_db database
-- ============================================
SET search_path TO texera_db;

-- ============================================
-- 2. Create the tables to store notebook and mapping
-- ============================================

BEGIN;

CREATE TABLE IF NOT EXISTS notebook (
nid SERIAL NOT NULL PRIMARY KEY,
wid INT NOT NULL,
notebook JSONB NOT NULL,
FOREIGN KEY (wid) REFERENCES workflow(wid) ON DELETE CASCADE
);

CREATE TABLE IF NOT EXISTS workflow_notebook_mapping (
wid INT NOT NULL,
vid INT NOT NULL,
nid INT NOT NULL,
mapping JSONB NOT NULL,
PRIMARY KEY (wid, vid, nid),
FOREIGN KEY (wid) REFERENCES workflow(wid) ON DELETE CASCADE,
FOREIGN KEY (vid) REFERENCES workflow_version(vid) ON DELETE CASCADE,
FOREIGN KEY (nid) REFERENCES notebook(nid) ON DELETE CASCADE
);

COMMIT;
Loading