-
Notifications
You must be signed in to change notification settings - Fork 3
Develop #80
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Develop #80
Changes from all commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
508357c
change all template urls to use url_from helper
jirivrany 016038a
version 1.2.2 for develop
jirivrany 9f09b78
Replace db-init.py with baseline Alembic migration, track migrations …
jirivrany 385434f
Baseline migration (001_baseline.py) — idempotent, handles any ExaFS …
jirivrany 49b8e8c
Move migrations/ inside flowapp package so they ship with pip install
jirivrany d3689b4
Baseline migration: fix column/data checks for pre-v0.5 databases
jirivrany cdaaf81
version 1.2.2 release candidate, fixes #73
jirivrany f42c480
Update docs/DB_MIGRATIONS.md
jirivrany 9f66cb9
Update scripts/migrate_v0x_to_v1.py
jirivrany 6cf92c1
Update flowapp/migrations/versions/001_baseline.py
jirivrany f70a357
Update flowapp/migrations/versions/001_baseline.py
jirivrany 33d68ed
Update flowapp/migrations/versions/001_baseline.py
jirivrany 5c16cc3
add test for base migration
jirivrany 9d7a954
add create-admin script, update docs
jirivrany 2033d8d
Add exafs-db-init and exafs-create-admin console scripts
jirivrany 4a1c750
Fix org_id inconsistency in baseline migration
jirivrany cd8dc5d
black formating, typos
jirivrany File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,6 @@ | ||
| config.py | ||
| instance_config_override.py | ||
| run.py | ||
| migrations/ | ||
|
|
||
| # PyPi | ||
| .pypirc | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,35 +1,115 @@ | ||
| # How to Upgrade the Database | ||
| # Database Migrations | ||
|
|
||
| ## General Guidelines | ||
| Migrations can be inconsistent. To avoid issues, we removed migrations from git repostory. To start the migration on your server, it is recomended reset the migration state on the server and run the migration based on the updated database models when switching application versions via Git. | ||
| ExaFS uses [Flask-Migrate](https://flask-migrate.readthedocs.io/) (Alembic) for database schema management. Migration files are shipped inside the `flowapp` package (`flowapp/migrations/`) and are found automatically — no `flask db init` is needed. | ||
|
|
||
| ## New Installation | ||
|
|
||
| For a fresh database, run the migrations to create all tables and seed data: | ||
|
|
||
| ```bash | ||
| flask db upgrade | ||
| ``` | ||
|
|
||
| Or use the init script (source install): | ||
|
|
||
| ```bash | ||
| python scripts/db-init.py | ||
| ``` | ||
|
|
||
| Or the installed command (PyPI install): | ||
|
|
||
| ```bash | ||
| rm -rf migrations/ | ||
| exafs-db-init | ||
| ``` | ||
|
|
||
| ```SQL | ||
| DROP TABLE alembic_version; | ||
| ## Upgrading Between Versions | ||
|
|
||
| When upgrading ExaFS to a new version, apply any new migrations: | ||
|
|
||
| ```bash | ||
| flask db upgrade | ||
| ``` | ||
|
|
||
| This will apply only the migrations that haven't been applied yet. | ||
|
|
||
| ## Existing Installation (One-Time Setup) | ||
|
|
||
| If you already have a running ExaFS database from any previous version, the baseline migration is idempotent — it will create missing tables, add missing columns, and skip anything that already exists. | ||
|
|
||
| ### Deployments that used `flask db init` (self-managed migrations) | ||
|
|
||
| Some deployments previously ran `flask db init` to create a local `migrations/` directory and auto-generated migration files. Starting with v1.2.2, migration files are tracked in git and shipped with the project. To switch to the official migrations: | ||
|
|
||
| 1. **Delete the local migrations directory** created by `flask db init`: | ||
| ```bash | ||
| rm -rf migrations/ | ||
| ``` | ||
| Migrations are now bundled inside the `flowapp` pip package — no local directory needed. | ||
|
|
||
| 2. **Clear the old alembic_version** and **stamp the baseline** to register with the official migration track (your schema is already up to date): | ||
| ```sql | ||
| DELETE FROM alembic_version; | ||
| ``` | ||
| ```bash | ||
| flask db stamp 001_baseline | ||
| ``` | ||
|
|
||
| 3. From now on, just run `flask db upgrade` when updating ExaFS. | ||
|
|
||
| ### Deployments without any migration tracking | ||
|
|
||
| If your database has an `alembic_version` table from a previous migration setup but no local `migrations/` directory, clear it first: | ||
|
|
||
| ```sql | ||
| DELETE FROM alembic_version; | ||
| ``` | ||
|
|
||
| Then run the upgrade: | ||
|
|
||
| ```bash | ||
| flask db init | ||
| flask db migrate -m "Initial migration based on current DB state" | ||
| flask db upgrade | ||
| ``` | ||
|
|
||
| ## Steps for Upgrading to v1.0.x | ||
| Limits for number of rules were introduced. Some database engines (Mariadb 10.x for example) have issue to set Non Null foreigin key to 0 and automatic migrations fail. The solution may be in diferent version (Mariadb 11.x works fine), or to set limits in db manually later. | ||
| The baseline migration will inspect your database and bring it up to the current schema without affecting existing data. | ||
|
|
||
| ## Upgrading from v0.x to v1.0+ | ||
|
|
||
| To set the limit to 0 for existing organizations run | ||
| If you are upgrading from a pre-1.0 version, the baseline migration will add the missing `org_id` columns and organization limit columns automatically. However, existing rules still need to be linked to organizations. An optional helper script is provided for this: | ||
|
|
||
| ```SQL | ||
| UPDATE organization | ||
| SET limit_flowspec4 = 0, limit_flowspec6 = 0, limit_rtbh = 0 | ||
| WHERE limit_flowspec4 IS NULL OR limit_flowspec6 IS NULL OR limit_rtbh IS NULL; | ||
| ```bash | ||
| python scripts/migrate_v0x_to_v1.py | ||
| ``` | ||
|
|
||
| In all cases we need later assign rules to organizations. There's an admin endpoint for this: | ||
| This script: | ||
| 1. Sets NULL organization limits to 0 | ||
| 2. Helps assign existing rules to organizations based on users' organizations | ||
| 3. Reports users with multiple organizations or ambiguous rule ownership that need manual assignment | ||
|
|
||
| Feel free to contact jiri.vrany@cesnet.cz if you need help with the migration. | ||
|
|
||
| ## Creating New Migrations | ||
|
|
||
| `https://yourexafs.url/admin/set-org-if-zero` | ||
| When you modify a database model, create a new migration: | ||
|
|
||
| ```bash | ||
| flask db migrate -m "Description of changes" | ||
| ``` | ||
|
|
||
| Review the generated file in `flowapp/migrations/versions/`, then apply it: | ||
|
|
||
| ```bash | ||
| flask db upgrade | ||
| ``` | ||
|
|
||
| Commit the migration file to git so other deployments can apply it. | ||
|
|
||
| ## Development Reset | ||
|
|
||
| To completely reset the database during development: | ||
|
|
||
| ```bash | ||
| python scripts/db-init.py --reset # source install | ||
| exafs-db-init --reset # PyPI install | ||
| ``` | ||
|
|
||
| Or you can start with clean database and manually migrate data by SQL dump later. Feel free to contact jiri.vrany@cesnet.cz if you need help with the DB migration to 1.0.x. | ||
| This drops all tables and recreates them from scratch. **Do not use in production.** | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The documentation states that migrate_v0x_to_v1.py is "optional" for upgrading from v0.x to v1.0+, but based on the migration code, this script appears to be required for databases with existing rules. Without running it, rules will have org_id=0 or NULL, which will:
Consider either: