Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
config.py
instance_config_override.py
run.py
migrations/

# PyPi
.pypirc
Expand Down
20 changes: 20 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,24 @@ All notable changes to ExaFS will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [1.2.2] - 2026-02-19

### Changed
- **Database migrations now tracked in git** — `migrations/` removed from `.gitignore`
- Replaced `db-init.py` with migration-based initialization (`flask db upgrade`)
- Removed one-time `/admin/set-org-if-zero` endpoint, replaced with standalone `scripts/migrate_v0x_to_v1.py`
- Fixed Flask-SQLAlchemy deprecation warning in Alembic `env.py`
- Template URLs changed to use `url_for` helper, removed unused `rule.html` template
- **`db-init.py` and `create-admin.py` moved to `scripts/`** — all setup scripts now live under `scripts/`

### Added
- Idempotent baseline migration (`001_baseline`) that brings any ExaFS database (from v0.4+ to current) to the v1.2.2 schema
- Optional `scripts/migrate_v0x_to_v1.py` helper for v0.x to v1.0+ data migration (org_id backfill)
- `scripts/create-admin.py` — interactive script to create the first admin user and organization (replaces manual SQL inserts)
- `scripts/db-init.py --reset` flag for development database reset
- Migration test suite (`tests/test_migration.py`) — 46 tests covering fresh install, idempotency, upgrade from v0.4/v0.8/v1.0 schemas, and real 2019 production backup upgrade
- `PYTHONPATH` set in Docker dev container for easier development

## [1.2.1] - 2026-01-30

### Fixed
Expand Down Expand Up @@ -286,6 +304,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Route Distinguisher for VRF now supported
- See config example and update your `config.py`

[1.2.2]: https://github.com/CESNET/exafs/compare/v1.2.1...v1.2.2
[1.2.1]: https://github.com/CESNET/exafs/compare/v1.2.0...v1.2.1
[1.2.0]: https://github.com/CESNET/exafs/compare/v1.1.9...v1.2.0
[1.1.9]: https://github.com/CESNET/exafs/compare/v1.1.8...v1.1.9
[1.1.8]: https://github.com/CESNET/exafs/compare/v1.1.7...v1.1.8
Expand Down
34 changes: 27 additions & 7 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,10 @@ exafs/
├── config.example.py # Configuration template
├── instance_config_override.example.py # Dashboard override template
├── run.example.py # Application run script template
├── db-init.py # Database initialization script
├── scripts/
│ ├── db-init.py # Database initialization (runs flask db upgrade)
│ ├── create-admin.py # Interactive first admin user setup
│ └── migrate_v0x_to_v1.py # Optional v0.x to v1.0+ migration helper
├── pyproject.toml # Project metadata and dependencies
├── setup.cfg # Setup configuration
├── CHANGELOG.md # Version history
Expand Down Expand Up @@ -283,8 +286,11 @@ cp run.example.py run.py

# Edit config.py with database credentials and settings

# Initialize database
python db-init.py
# Initialize database (runs flask db upgrade)
python scripts/db-init.py

# Create the first admin user and organization
python scripts/create-admin.py

# Run tests
pytest
Expand All @@ -295,15 +301,20 @@ python run.py

### Database Migrations

Migration files are tracked in `migrations/versions/` and committed to git.

```bash
# Create a new migration
# Create a new migration after model changes
flask db migrate -m "Description of changes"

# Apply migrations
flask db upgrade

# Rollback migration
flask db downgrade

# For existing databases adopting migrations for the first time
flask db stamp 001_baseline
```

### Running Tests
Expand Down Expand Up @@ -787,8 +798,17 @@ flask db migrate -m "message" # Create migration
flask db upgrade # Apply migrations
flake8 . # Lint code

# Database
python db-init.py # Initialize database
# Database (source install)
python scripts/db-init.py # Initialize database (runs migrations)
python scripts/db-init.py --reset # Drop all tables and recreate (dev only)
python scripts/create-admin.py # Create first admin user interactively

# Database (PyPI install — run from directory containing config.py)
exafs-db-init # Initialize database (runs migrations)
exafs-db-init --reset # Drop all tables and recreate (dev only)
exafs-create-admin # Create first admin user interactively

flask db stamp 001_baseline # Mark existing DB as baseline
flask db current # Show current migration
flask db history # Show migration history

Expand All @@ -804,7 +824,7 @@ supervisorctl status # Check status
When working with this codebase:

1. **Always run tests** after making changes: `pytest`
2. **Create migrations** for model changes: `flask db migrate`
2. **Create migrations** for model changes: `flask db migrate` — commit migration files to git
3. **Follow the service layer pattern** - business logic goes in services, not views
4. **Use existing validators** in `flowapp/validators.py` for validation
5. **Check authentication** - most routes need `@auth_required` decorator
Expand Down
39 changes: 0 additions & 39 deletions db-init.py

This file was deleted.

116 changes: 98 additions & 18 deletions docs/DB_MIGRATIONS.md
Original file line number Diff line number Diff line change
@@ -1,35 +1,115 @@
# How to Upgrade the Database
# Database Migrations

## General Guidelines
Migrations can be inconsistent. To avoid issues, we removed migrations from git repostory. To start the migration on your server, it is recomended reset the migration state on the server and run the migration based on the updated database models when switching application versions via Git.
ExaFS uses [Flask-Migrate](https://flask-migrate.readthedocs.io/) (Alembic) for database schema management. Migration files are shipped inside the `flowapp` package (`flowapp/migrations/`) and are found automatically — no `flask db init` is needed.

## New Installation

For a fresh database, run the migrations to create all tables and seed data:

```bash
flask db upgrade
```

Or use the init script (source install):

```bash
python scripts/db-init.py
```

Or the installed command (PyPI install):

```bash
rm -rf migrations/
exafs-db-init
```

```SQL
DROP TABLE alembic_version;
## Upgrading Between Versions

When upgrading ExaFS to a new version, apply any new migrations:

```bash
flask db upgrade
```

This will apply only the migrations that haven't been applied yet.

## Existing Installation (One-Time Setup)

If you already have a running ExaFS database from any previous version, the baseline migration is idempotent — it will create missing tables, add missing columns, and skip anything that already exists.

### Deployments that used `flask db init` (self-managed migrations)

Some deployments previously ran `flask db init` to create a local `migrations/` directory and auto-generated migration files. Starting with v1.2.2, migration files are tracked in git and shipped with the project. To switch to the official migrations:

1. **Delete the local migrations directory** created by `flask db init`:
```bash
rm -rf migrations/
```
Migrations are now bundled inside the `flowapp` pip package — no local directory needed.

2. **Clear the old alembic_version** and **stamp the baseline** to register with the official migration track (your schema is already up to date):
```sql
DELETE FROM alembic_version;
```
```bash
flask db stamp 001_baseline
```

3. From now on, just run `flask db upgrade` when updating ExaFS.

### Deployments without any migration tracking

If your database has an `alembic_version` table from a previous migration setup but no local `migrations/` directory, clear it first:

```sql
DELETE FROM alembic_version;
```

Then run the upgrade:

```bash
flask db init
flask db migrate -m "Initial migration based on current DB state"
flask db upgrade
```

## Steps for Upgrading to v1.0.x
Limits for number of rules were introduced. Some database engines (Mariadb 10.x for example) have issue to set Non Null foreigin key to 0 and automatic migrations fail. The solution may be in diferent version (Mariadb 11.x works fine), or to set limits in db manually later.
The baseline migration will inspect your database and bring it up to the current schema without affecting existing data.

## Upgrading from v0.x to v1.0+

To set the limit to 0 for existing organizations run
If you are upgrading from a pre-1.0 version, the baseline migration will add the missing `org_id` columns and organization limit columns automatically. However, existing rules still need to be linked to organizations. An optional helper script is provided for this:

```SQL
UPDATE organization
SET limit_flowspec4 = 0, limit_flowspec6 = 0, limit_rtbh = 0
WHERE limit_flowspec4 IS NULL OR limit_flowspec6 IS NULL OR limit_rtbh IS NULL;
```bash
python scripts/migrate_v0x_to_v1.py
```

In all cases we need later assign rules to organizations. There's an admin endpoint for this:
This script:
1. Sets NULL organization limits to 0
2. Helps assign existing rules to organizations based on users' organizations
3. Reports users with multiple organizations or ambiguous rule ownership that need manual assignment

Feel free to contact jiri.vrany@cesnet.cz if you need help with the migration.
Comment on lines +75 to +88
Copy link

Copilot AI Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documentation states that migrate_v0x_to_v1.py is "optional" for upgrading from v0.x to v1.0+, but based on the migration code, this script appears to be required for databases with existing rules. Without running it, rules will have org_id=0 or NULL, which will:

  1. Not satisfy the foreign key constraints expected by the models (all define org_id as nullable=False)
  2. Cause application errors when trying to create or query rules

Consider either:

  • Changing "optional" to "required" in the documentation
  • Making the migration script part of the automated migration process
  • Or clearly documenting that the application WILL NOT WORK until this script is run

Copilot uses AI. Check for mistakes.

## Creating New Migrations

`https://yourexafs.url/admin/set-org-if-zero`
When you modify a database model, create a new migration:

```bash
flask db migrate -m "Description of changes"
```

Review the generated file in `flowapp/migrations/versions/`, then apply it:

```bash
flask db upgrade
```

Commit the migration file to git so other deployments can apply it.

## Development Reset

To completely reset the database during development:

```bash
python scripts/db-init.py --reset # source install
exafs-db-init --reset # PyPI install
```

Or you can start with clean database and manually migrate data by SQL dump later. Feel free to contact jiri.vrany@cesnet.cz if you need help with the DB migration to 1.0.x.
This drops all tables and recreates them from scratch. **Do not use in production.**
Loading