diff --git a/.github/workflows/deploy_docs.yml b/.github/workflows/deploy_docs.yml new file mode 100644 index 0000000..672de1c --- /dev/null +++ b/.github/workflows/deploy_docs.yml @@ -0,0 +1,54 @@ +name: Deploy Documentation Site + +on: + push: + branches: [ main ] + paths: + - "docs/**" + - "mkdocs.yml" + workflow_dispatch: + +permissions: + contents: read + pages: write + id-token: write + +concurrency: + group: "pages" + cancel-in-progress: false + +jobs: + build: + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Set up Python + uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 + with: + python-version: '3.11' + + - name: Install MkDocs + run: | + python -m pip install --upgrade pip + pip install mkdocs mkdocs-mermaid2-plugin + + - name: Build site + run: mkdocs build --strict + + - name: Upload Pages artifact + uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0 + with: + path: site + + deploy: + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + runs-on: ubuntu-latest + needs: build + steps: + - name: Deploy to GitHub Pages + id: deployment + uses: actions/deploy-pages@cd2ce8fcbc39b97be8ca5fce6e763baed58fa128 # v5.0.0 diff --git a/docs/api.md b/docs/api.md new file mode 100644 index 0000000..8b41fb7 --- /dev/null +++ b/docs/api.md @@ -0,0 +1,120 @@ +# API Reference + +The examples below use the Compose stack's local address, `http://localhost:5001`. Note that the service serves its own OpenAPI specification at `/docs`. + +!!! note + `POST /v1/ro_crates/validate_metadata` is always available, but the **storage-backed endpoints** are only available when the service runs with `STORAGE_ENABLED=true` (see [Installation & Setup](installation.md#enabling-object-storage)); without this set, `POST /v1/ro_crates/{crate_id}/validation` or `GET /v1/ro_crates/{crate_id}/validation` requests will return a `404` result. + +## Validate metadata + +`POST /v1/ro_crates/validate_metadata` + +This validates the contents of an `ro-crate-metadata.json` document and returns the result in the response. + +| Field | Required | Description | +|-------|----------|-------------| +| `crate_json` | yes | The metadata document, as a JSON string | +| `profile_name` | no | Profile to validate against, e.g. `ro-crate-1.2`. The validator will default to `ro-crate-1.1` when this is omitted | + +!!! warning + Currently, the validation profile is not detected from the RO-Crate. In other words, a `conformsTo` declaration in the metadata does not influence which validation profile is used by the validator, and the validation always runs against `profile_name`, or `ro-crate-1.1` when it is omitted. + +To validate an ro-crate metadata file: + +```bash +jq -Rs '{crate_json: .}' ro-crate-metadata.json | curl -X POST http://localhost:5001/v1/ro_crates/validate_metadata -H 'Content-Type: application/json' -d @- +``` + +To choose a profile, add it to the json object as a `profile_name` entry: + +```bash +jq -Rs '{crate_json: ., profile_name: "ro-crate-1.2"}' ro-crate-metadata.json | curl -X POST http://localhost:5001/v1/ro_crates/validate_metadata -H 'Content-Type: application/json' -d @- +``` + +| Code | Meaning | +|------|---------| +| `200` | Validated; the result has a `status` of `valid` or `invalid` | +| `422` | `crate_json` is either missing, empty or invalid, or the validation could not run (an `error` result) | + +## Validate a stored RO-Crate + +`POST /v1/ro_crates/{crate_id}/validation` + +This queues validation of an RO-Crate held in the object store. The RO-Crate is resolved first, so a missing or ambiguous crate ID may be reported immediately; the validation process itself runs asynchronously on a worker. + +!!! note + See [Crate IDs](#crate-ids) for how `{crate_id}` maps to objects in the bucket. + +| Field | Required | Description | +|-------|----------|-------------| +| `profile_name` | no | Profile to validate against; defaults to `ro-crate-1.1` when omitted | +| `webhook_url` | no | URL that receives the result when validation finishes | + +```bash +curl -X POST http://localhost:5001/v1/ro_crates/my-dataset-2026/validation -H 'Content-Type: application/json' -d '{"profile_name": "ro-crate-1.2"}' +``` + +| Code | Meaning | +|------|---------| +| `202` | Queued; the body is `{"message": "Validation in progress"}` | +| `400` | Invalid Crate ID | +| `404` | Either storage mode is not enabled, or there is no RO-Crate at the location defined by the given Crate ID | +| `409` | Both a zip and a directory exist for this Crate ID | +| `422` | Request body invalid | +| `503` | Object store unreachable | + +## Fetch a validation result + +`GET /v1/ro_crates/{crate_id}/validation` + +This returns the stored result for an RO-Crate. + +```bash +curl http://localhost:5001/v1/ro_crates/my-dataset-2026/validation +``` + +| Code | Meaning | +|------|---------| +| `200` | The stored result, including persisted `error` results | +| `400` | Invalid Crate ID | +| `404` | No result stored for this Crate ID | + +## Validation results + +Every validation produces a result object: + +```json +{ + "status": "invalid", + "profile": "ro-crate-1.2", + "created_at": "2026-07-22T10:30:00+00:00", + "detail": {} +} +``` + +An RO-Crate's `status` can be: + +| `status` | Meaning | +|----------|---------| +| `valid` | The RO-Crate conforms to the profile | +| `invalid` | The RO-Crate does not conform to the profile, issues listed in `detail` field | +| `error` | The validation could not run; the reason is in an `error` field. No `detail` field is provided | + +!!! note + `detail` contains the complete validation report. `created_at` is the UTC time of a stored-crate validation, and `null` for metadata-only validation, which does not set it. `profile` is the requested profile name, or `null` when the default (`ro-crate-1.1`) was used. + +For stored RO-Crates the same object is saved to `{S3_RESULTS_PREFIX}/.json` and returned by the GET endpoint. + +## Webhooks + +If `webhook_url` was given, the worker POSTs the result object to it as JSON once validation finishes. The result is saved to the store first and the webhook sent after, so a notification is never sent for a result that was not stored. + +Note that delivery is attempted three times, waiting `0.5s` then `1s` between attempts, with a `10s` timeout per attempt. + +## Crate IDs + +A Crate ID is the label in the URL path that identifies an RO-Crate in the object store: the service looks for `{S3_CRATE_PREFIX}/.zip` (zip) or `{S3_CRATE_PREFIX}//` (directory). Crate IDs must match `^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$`: they start with a letter or digit, may contain letters, digits, `.`, `_` and `-`, and are at most 128 characters long. Anything else is rejected with `400`. + +## Health + +`GET /healthz` reports that the process is up, and always returns `200 {"status": "ok"}`. `GET /readyz` checks the object store and Celery broker, returning `200` when ready and `503` otherwise, with the individual checks in the body. When storage is off, both checks report `disabled`. diff --git a/docs/contribution.md b/docs/contribution.md new file mode 100644 index 0000000..c1cd5ab --- /dev/null +++ b/docs/contribution.md @@ -0,0 +1,121 @@ +# Development and Contributions + +The [RO-Crate Validation Service](https://github.com/eScienceLab/RO-Crate-Validation-Service) is an open source project and welcomes contributions of all kinds: bug reports, code or documentation changes, and reviews of proposed changes. The underlying [rocrate-validator tool](https://github.com/crs4/rocrate-validator) is also open source, and is a separate project maintained by CRS4. + +This service is written with Python 3.11, built on Flask/APIFlask and Celery, and wraps the [`rocrate-validator`](https://rocrate-validator.readthedocs.io/) in a REST API. The RO-Crate Validation Service enables pipelines, other services, and Trusted Research Environments (TREs) to validate an RO-Crate over HTTP without running the validator themselves. + +## Contributing + +The easiest way to start contributing is to create an issue, either to let us know of a bug or error, or to propose a piece of work you want to do. For the RO-Crate Validation Service (the API service, Docker image, and Compose stack) use the [RO-Crate Validation Service issues](https://github.com/eScienceLab/RO-Crate-Validation-Service/issues) page. Issues with the validation checks themselves belong to the underlying tool rather than this service: report those on the [rocrate-validator issues](https://github.com/crs4/rocrate-validator/issues) page, and follow that project's own contribution guidance. + +### Code contributions + +If you want to contribute code changes via GitHub then you may want to read ['How to Contribute to an Open Source Project on GitHub'](https://egghead.io/courses/how-to-contribute-to-an-open-source-project-on-github). We use [GitHub flow](https://docs.github.com/en/get-started/using-github/github-flow) to manage changes: + +1. Create a new branch in your local clone of this repository for each significant change. +2. Commit the change in that branch. +3. Push that branch to your fork of this repository on GitHub. +4. Submit a pull request from that branch to the [upstream repository](https://github.com/eScienceLab/RO-Crate-Validation-Service). +5. If you receive feedback, make the changes in your local clone and push to your branch on GitHub: the pull request will update automatically. + +!!! warning + Note that we use the `develop` branch for development work, and this is where your PR should be aimed. The `main` branch is used for releases, and only pull requests from the `develop` branch are accepted to this. + +## Development stack + +The development Compose file builds the image from the local `Dockerfile` and mounts the repository's test profiles into both the API and worker containers: + +```bash +docker compose -f docker-compose-develop.yml up --build +``` + +Here `--build` matters: without it, Compose reuses the previously built image and local code changes are not picked up. Add `--profile objectstore` to start the bundled RustFS store for storage-backed work; configuration is the same as in [Installation & Setup](installation.md#configuration-reference). + +## Tests + +Install the development dependencies, then run the unit tests, which do not use Docker Engine: + +```bash +pip install -r requirements-dev.txt +``` + +```bash +pytest --ignore=tests/test_integration.py +``` + +The integration tests bring up the full Compose stack (including the object store) and seed crates with `boto3`, for which they need Docker Engine to be running: + +```bash +pytest tests/test_integration.py +``` + +`tests/` mirrors the layout of the `app/` package, so the tests for a module are in the matching directory. + +## Linting + +The project uses [Ruff](https://docs.astral.sh/ruff/) for linting and formatting, configured in `pyproject.toml`: + +```bash +ruff check . && ruff format --check . +``` + +## Dependencies + +Direct dependencies are declared in `pyproject.toml`; while the `requirements*.txt` files are locks generated using `pip-compile`: + +```bash +pip-compile pyproject.toml -o requirements.txt +``` + +```bash +pip-compile --extra dev pyproject.toml -o requirements-dev.txt +``` + +## Continuous Integration + +Pull requests to `develop` will trigger three workflows: unit tests, integration tests (which start the Compose stack), and lint (`ruff check` and `ruff format --check`). + +## How the API works + +The API server handles HTTP and runs metadata-only validation inline. Object storage-backed validation is queued through Redis to a Celery worker, which reads the crate from the S3-compatible store, validates it, and writes the result back: + +```mermaid +flowchart LR + Client([Client]) + API["Flask API"] + Broker[("Redis")] + Worker["Celery worker"] + Validator["rocrate-validator"] + Store[("S3-compatible store")] + + Client --> API + API -->|metadata-only: inline| Validator + API --> Broker --> Worker --> Validator + Worker <--> Store +``` + +The worker runs its stages strictly in order: fetch, validate, persist, webhook; so a storage write failure can never be followed by a success notification, and every outcome (including `error` outcomes) is persisted so a later `GET` reflects what happened. + +## Project structure + +``` +app/ +├── __init__.py # app factory: config, blueprints, error handlers, request IDs +├── health.py # /healthz and /readyz +├── storage/ # object-storage abstraction +│ ├── base.py # StorageBackend protocol +│ ├── s3.py # boto3 implementation (any S3-compatible store) +│ ├── memory.py # in-memory backend (tests / local) +│ └── errors.py # StorageError, ObjectNotFound +├── crates/ # crate identity, layout, resolution +│ ├── ids.py # Crate ID validation +│ ├── layout.py # object keys +│ └── resolver.py # deterministic zip/directory resolution +├── validation/ # validation boundary +│ ├── results.py # ValidationOutcome (valid/invalid/error) +│ └── runner.py # wraps rocrate-validator +├── ro_crates/routes/ # HTTP endpoints (metadata + ID-based) +├── services/ # request handling and logging +├── tasks/validation_tasks.py # Celery task: fetch, validate, persist, webhook +└── utils/ # validated settings, webhook delivery +``` diff --git a/docs/five-safes.md b/docs/five-safes.md new file mode 100644 index 0000000..e998c96 --- /dev/null +++ b/docs/five-safes.md @@ -0,0 +1,98 @@ +# Five Safes RO-Crate validation + +The [Five Safes RO-Crate 0.4 profile](https://trefx.uk/5s-crate/) describes an RO-Crate used to request and record workflow runs on sensitive data in Trusted Research Environments (TREs), supporting the Five Safes framework. The RO-Crate Validation Service validates against this profile when `profile_name` is set to `five-safes-crate`. + +!!! warning + Note that the Five Safes RO-Crate 0.4 profile is not bundled with the base validator, so the service needs the profile to be made available. There are two ways to do this, described below. + +## Getting a service with the profile + +The prebuilt `ghcr.io/esciencelab/ro-crate-validation-service-fivesafes-profile` image packages the `five-safes-crate` profile with the standard RO-Crate Validation Service. The image also carries a pre-warmed validation cache, so it supports offline validation (`VALIDATION_OFFLINE=true`) inside restricted networks. The profile version is fixed when the image is built, and is recorded in the image label `org.ro-crate-validation-service.five-safes-profile-version`. + +Alternatively, you may run the standard service image with the profile directory mounted and `EXTRA_PROFILES_PATH` set, [as described in custom profiles](installation.md#custom-profiles). + +The `five-safes-crate` profile itself is defined in the [eScienceLab rocrate-validator fork](https://github.com/eScienceLab/rocrate-validator). + +## Validating a Five Safes RO-Crate + +A [complete example crate](https://github.com/eScienceLab/rocrate-validator/blob/five-safes-0.7.4-beta/tests/data/crates/valid/five-safes-crate-result/ro-crate-metadata.json) is available in the fork's test data. + +!!! note + The current `-fivesafes-profile` image pairs the profile with a base profile for RO-Crate 1.1, whilst the profile itself expects RO-Crate 1.2. The walkthrough below mounts the matched profile set instead. + +For this walkthrough, run the service from a checkout of this repository, with the repository's profile set mounted in place of the bundled profiles: + +```bash +docker run --rm -p 5001:5000 \ + -e FLASK_APP=wsgi.py \ + -e PROFILES_PATH=/app/profiles \ + -v "$PWD/tests/data/rocrate_validator_profiles:/app/profiles:ro" \ + ghcr.io/esciencelab/ro-crate-validation-service-fivesafes-profile:latest +``` + +Download the example `ro-crate-metadata.json`: + +```bash +curl -sO https://raw.githubusercontent.com/eScienceLab/rocrate-validator/five-safes-0.7.4-beta/tests/data/crates/valid/five-safes-crate-result/ro-crate-metadata.json +``` + +and validate it with `profile_name` set to `five-safes-crate`: + +```bash +jq -Rs '{crate_json: ., profile_name: "five-safes-crate"}' ro-crate-metadata.json | curl -X POST http://localhost:5001/v1/ro_crates/validate_metadata -H 'Content-Type: application/json' -d @- +``` + +The crate conforms, so the response (abridged) is: + +```json +{ + "status": "valid", + "profile": "five-safes-crate", + "created_at": null, + "detail": { + "issues": [], + "passed": true + } +} +``` + +To see how conformance issues are reported, remove something the profile requires, such as the `CreateAction` recording the requested workflow run. Validate again: + +```bash +jq '."@graph" |= map(select(."@type" != "CreateAction"))' ro-crate-metadata.json > broken.json +``` + +```bash +jq -Rs '{crate_json: ., profile_name: "five-safes-crate"}' broken.json | curl -X POST http://localhost:5001/v1/ro_crates/validate_metadata -H 'Content-Type: application/json' -d @- +``` + +The result will now show `invalid`, and each entry in `detail.issues` identifies the failed check, its severity, and the entity at fault (abridged): + +```json +{ + "status": "invalid", + "profile": "five-safes-crate", + "detail": { + "issues": [ + { + "check": { + "identifier": "five-safes-crate-0.4_25.1", + "name": "mentions" + }, + "severity": "REQUIRED", + "message": "`RootDataEntity` MUST reference at least one `CreateAction` through `mentions`", + "violatingEntity": "./" + } + ], + "passed": false + } +} +``` + +Complete RO-Crates work the same way through the storage-backed endpoints: upload the crate as `crates/.zip` (or a directory under `crates//`), then queue validation with the profile: + +```bash +curl -X POST http://localhost:5001/v1/ro_crates/my-5s-crate/validation -H 'Content-Type: application/json' -d '{"profile_name": "five-safes-crate"}' +``` + +The [API reference](api.md) covers the endpoints, results and webhooks in full. diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..a6b635b --- /dev/null +++ b/docs/index.md @@ -0,0 +1,41 @@ +# RO-Crate Validation Service + +The RO-Crate Validation Service provides a REST API for evaluating whether an [RO-Crate](https://www.researchobject.org/ro-crate/) conforms to a given RO-Crate profile. The service wraps the [`rocrate-validator`](https://rocrate-validator.readthedocs.io/) library, enabling pipelines, other services, and Trusted Research Environments (TREs) to validate an RO-Crate over HTTP without having to install the validator themselves. + +## RO-Crates and profiles in brief + +An [RO-Crate](https://www.researchobject.org/ro-crate/) packages research data together with structured, machine-readable metadata: a JSON-LD file named `ro-crate-metadata.json`. Validating an RO-Crate involves evaluating that metadata against a given **profile**. A profile is a set of requirements the RO-Crate must satisfy, either the base requirements of the [RO-Crate specification](https://www.researchobject.org/ro-crate/1.2/) itself, or a [community profile](https://www.researchobject.org/ro-crate/profiles) that adds domain-specific rules to the base specification. An example of a community profile is the [Five Safes RO-Crate profile](https://trefx.uk/5s-crate/), designed for researchers working with sensitive data in Trusted Research Environments (TREs). + +Whilst the validation checks themselves are performed by the [`rocrate-validator`](https://rocrate-validator.readthedocs.io/), this service is a deployable HTTP wrapper around that tool: it adds a web API, asynchronous processing, and object-storage integration. The base RO-Crate specification, as well as several [community profiles](https://rocrate-validator.readthedocs.io/en/latest/#features), are provided with the validator itself. We additionally package our own [Five Safes RO-Crate profile](five-safes.md) rules with the service for working in TREs. + +## Validation methods + +1. **Metadata-only**. Send the contents of an `ro-crate-metadata.json` file and receive the validation result in the response. This is synchronous and stateless, so nothing is stored, and no object store or worker is required. This approach is simpler to use, and is intended for quick evaluations whilst metadata is being written, or before a full crate has been assembled. + +2. **Storage-backed**. The service reads complete RO-Crates (zip or directory) from an S3-compatible object store, such as RustFS, AWS S3, MinIO, and others. Validation runs asynchronously on a worker process; the result is stored for later retrieval and can optionally be delivered to a webhook. This is more suited to pipelines and workflows. + +```mermaid +flowchart LR + Client([Your application]) + API["REST API"] + Validator["rocrate-validator"] + Worker["Celery worker"] + Store[("S3-compatible store")] + + Client -->|HTTP| API + API -->|metadata-only: validated inline| Validator + API -->|storage-backed: queued| Worker + Worker --> Validator + Worker <--> Store +``` + +## Documentation + +- To run the service yourself, start with [Installation & Setup](installation.md), or with the [Upgrade Guide](upgrading.md) if you already run a 1.x version. +- The [API Reference](api.md) documents the endpoints for anyone integrating against a running instance. +- The [Five Safes RO-Crate](five-safes.md) page walks through validating RO-Crates in a TRE. +- To help contribute to the service, please see the [Contribution Guide](contribution.md). + +## About + +The RO-Crate Validation Service is developed by the [eScience Lab](https://esciencelab.org.uk/) at The University of Manchester, and available on [GitHub](https://github.com/eScienceLab/RO-Crate-Validation-Service) under the MIT licence. diff --git a/docs/installation.md b/docs/installation.md new file mode 100644 index 0000000..bcbb0c3 --- /dev/null +++ b/docs/installation.md @@ -0,0 +1,114 @@ +# Installation & Setup + +The RO-Crate Validation Service works in two ways: a metadata-only mode, in which the contents of an `ro-crate-metadata.json` file are assessed, and storage-backed validation, where complete RO-Crates (zip or directory) are evaluated. + +## Quick start + +You will need Docker with Docker Compose. + +To start, clone the repository, copy the example environment file (`example.env`), and start the stack: + +```bash +git clone https://github.com/eScienceLab/RO-Crate-Validation-Service.git +cd RO-Crate-Validation-Service +cp example.env .env +docker compose up +``` + +!!! warning + Remember to update the default `.env` values when running the object store in production. + +The API is served at `http://localhost:5001`. Redis and a Celery worker are also started, but these are only used once storage is enabled. + +To check the service is up, run: + +```bash +curl http://localhost:5001/healthz +``` + +This returns `{"status": "ok"}`. + +To validate the contents of an `ro-crate-metadata.json` file, post to the metadata endpoint. The [running example](https://www.researchobject.org/ro-crate/specification/1.2/introduction.html#running-example) from the RO-Crate specification is a good test document. Create an `ro-crate-metadata.json` file and copy the text from the link above into this, for use in the example below. + +The file contents need to be sent to the API as an escaped JSON string, identified using the `crate_json` tag, within a JSON object. The command-line JSON processor, [`jq`](https://jqlang.org/) can be used to do this, as shown below. + +```bash +jq -Rs '{crate_json: .}' ro-crate-metadata.json | curl -X POST http://localhost:5001/v1/ro_crates/validate_metadata -H 'Content-Type: application/json' -d @- +``` + +The returned response will contain a `status` of `valid`, `invalid` or `error`, along with the detailed validation. For more information, the [API reference](api.md) describes the endpoints and result format in full. + +## Enabling object storage + +To enable the validation of complete RO-Crates (zip or directory) that are held in an object store, set `STORAGE_ENABLED=true` in `.env`. + +The storage-backed validation mode requires six environmental variables to be set in the `.env` file, [described below](#configuration-reference): `S3_ENDPOINT`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`, `S3_BUCKET`, `CELERY_BROKER_URL` and `CELERY_RESULT_BACKEND`. The service will fail at startup if any are missing. + +The `CELERY_BROKER_URL` and `CELERY_RESULT_BACKEND` are already configured to the bundled Redis service within the docker compose stack, so the values for these within the `example.env` file can be left as they are. For any purpose other than an initial demonstration of the service the S3 settings given in the `example.env` file, `S3_ENDPOINT`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`, and `S3_BUCKET`, should be changed for both security reasons and to match your own local setup. + +To start the stack with the bundled development object store (RustFS), run: + +```bash +docker compose --profile objectstore up +``` + +RustFS serves the S3 API on port 9000 and a web console at `http://localhost:9001`. + +!!! warning + The service does not create the bucket itself. Create the bucket in the console or with an S3 client. The bucket name needs to match the value in the `S3_BUCKET` environmental variable (`ro-crates` by default). + +Upload an RO-Crate to this S3 bucket, using the prefix `crates`. This should give the uploaded RO-Crate a path of either `crates/.zip` for a zipped RO-Crate, or `crates//` for a directory. Note that for a zipped RO-Crate, `ro-crate-metadata.json` must be at the root of the archive. + +The readiness endpoint can be used to check the object store and broker connections: + +```bash +curl http://localhost:5001/readyz +``` + +## Using your own object store + +Any S3-compatible store can be used in place of RustFS, including AWS S3, MinIO and Ceph. To do this set the `S3_ENDPOINT` and `S3_BUCKET` environment variables to match the location of your store, and provide your store credentials in the `S3_ACCESS_KEY` and `S3_SECRET_KEY` environment variables. Make sure that the `STORAGE_ENABLED` environment variable is still set to `true`, but do not include the `objectstore` profile flag in your docker compose command: + +```bash +docker compose up +``` + +If you already use a 1.x release against MinIO, the [upgrade guide](upgrading.md) maps the old settings to the new ones. + +## Configuration reference + +The following can all be set as environment variables for the service using an `.env` file. + +| Variable | Default | Description | +|----------|---------|-------------| +| `STORAGE_ENABLED` | `false` | Enable the stored-crate endpoints and storage checks | +| `S3_ENDPOINT` | — | Object store endpoint, e.g. `objectstore:9000` (required in storage mode) | +| `S3_ACCESS_KEY` | — | Object store access key (required in storage mode) | +| `S3_SECRET_KEY` | — | Object store secret key (required in storage mode) | +| `S3_BUCKET` | — | Bucket holding RO-Crates and results (required in storage mode) | +| `S3_USE_SSL` | `false` | Use HTTPS for connecting to the object store | +| `S3_REGION` | — | Region; needed when using AWS S3 object stores | +| `S3_CRATE_PREFIX` | `crates` | Prefix key from which RO-Crates are read | +| `S3_RESULTS_PREFIX` | `validation-results` | Prefix key to which results are written | +| `CELERY_BROKER_URL` | — | Redis broker URL (required in storage mode; preset in the Compose stack) | +| `CELERY_RESULT_BACKEND` | — | Celery result backend URL (required in storage mode; preset in the Compose stack) | +| `PROFILES_PATH` | — | Profiles directory for replacing the bundled profiles | +| `EXTRA_PROFILES_PATH` | — | Profiles directory for adding extra profiles | +| `CACHE_PATH` | `/app/.rocrate-cache` | Validator HTTP cache location | +| `VALIDATION_OFFLINE` | `false` | Validate using only the cache, with no network access | +| `FLASK_ENV` | `development` | Set to `production` to disable debug behaviour | + +## Custom profiles + +The validator comes with several RO-Crate profiles, and for the Five Safes RO-Crate, the prebuilt `ghcr.io/esciencelab/ro-crate-validation-service-fivesafes-profile` image has the `five-safes-crate` profile already included; see [Five Safes validation](five-safes.md). + +Other profiles can be provided by mounting the directory containing these profiles as a volume for the `flask` container. Mount the same directory as a volume for the `celery_worker` container as well if you have enabled stored-crate validation. Then set either the `EXTRA_PROFILES_PATH` or `PROFILES_PATH` environment variable to match the volume path. There is a working example in `docker-compose-develop.yml`. + +!!! note + `EXTRA_PROFILES_PATH` adds the directory to the bundled profiles, whereas `PROFILES_PATH` replaces them entirely. The two can be set together, in which case the validator takes profiles from both locations. + +## Offline validation + +The validator fetches profile and context resources over HTTP and caches them. The published v2.* images pre-populate this cache at build time, so setting `VALIDATION_OFFLINE=true` runs validation entirely from the cache, with no network access at runtime. This is useful inside TREs and other networks with restricted internet access. + +Online validation (the default) also uses and refreshes the same cache. Offline validation requires `rocrate-validator` at 0.10.0 or later, which the published v2.* ro-crate validation service images include. diff --git a/docs/upgrading.md b/docs/upgrading.md new file mode 100644 index 0000000..6588248 --- /dev/null +++ b/docs/upgrading.md @@ -0,0 +1,57 @@ +# Upgrading from 1.x + +The RO-Crate Validation Service 2.0 release replaced the MinIO-specific storage layer with a general S3-compatible one. + +!!! note + If you only use metadata validation (`POST /v1/ro_crates/validate_metadata`), nothing changes and the endpoint, request fields and responses are the same as in 1.\*. The rest of this page concerns storage-backed validation. + +## Server settings + +The old `MINIO_*` variables are replaced by equivalent `S3_*` variables, and storage is now switched on explicitly: + +| 1.\* | 2.\* | +|------|-----| +| `MINIO_ENDPOINT` | `S3_ENDPOINT` | +| `MINIO_ROOT_USER` | `S3_ACCESS_KEY` | +| `MINIO_ROOT_PASSWORD` | `S3_SECRET_KEY` | +| `MINIO_BUCKET_NAME` | `S3_BUCKET` | +| `ssl` field in each request | `S3_USE_SSL` | +| — | `STORAGE_ENABLED` (new; must be `true` for the stored-crate endpoints to exist) | +| — | `S3_CRATE_PREFIX`, `S3_RESULTS_PREFIX` (new; default `crates` and `validation-results`) | +| `FLASK_APP=cratey.py` | `FLASK_APP=wsgi.py` | + +The published image is also renamed: `ghcr.io/esciencelab/cratey-validator` is now `ghcr.io/esciencelab/ro-crate-validation-service` (with a `-fivesafes-profile` variant that has the Five Safes profile included). The [configuration reference](installation.md#configuration-reference) lists all the settings. + +## Keeping your existing MinIO + +You do not need to change object store as MinIO is S3-compatible. Set `S3_ENDPOINT` to your existing MinIO endpoint, `S3_ACCESS_KEY` and `S3_SECRET_KEY` to your MinIO credentials, and `S3_BUCKET` to your bucket. + +## API changes + +The service connects to the object store defined in the server-side configuration. Requests carry only the Crate ID and validation options. The body of `POST /v1/ro_crates/{crate_id}/validation` contains the optional `profile_name` and `webhook_url`. `GET /v1/ro_crates/{crate_id}/validation` takes no body. + +!!! warning + Ensure that you update any existing request bodies before sending requests to the new RO-Crate Validation Service API. Incorrect request bodies will receive `422` validation errors. + +### Crate IDs + +A Crate ID is the label that addresses an RO-Crate in the API path, for example `my-dataset-2026` in `POST /v1/ro_crates/my-dataset-2026/validation`. It is chosen by whoever uploads the RO-Crate, and the service composes the object keys from it: `{S3_CRATE_PREFIX}/.zip` for a zipped RO-Crate, or `{S3_CRATE_PREFIX}//` for a directory. The Crate ID itself is not a filename, path or URL. + +Crate IDs are now validated strictly: they must match `^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$`. This does not allow slashes or path segments. Paths inside the bucket are handled by the prefix settings. + +Response codes are more specific than the 1.\* `400`/`500` pattern: + +| Situation | 1.\* | 2.\* | +|-----------|------|-----| +| Crate not found in the store | `400` | `404` | +| No validation result stored yet | `400` | `404` | +| Invalid Crate ID | — | `400` | +| Both zip and directory exist for one Crate ID | — | `409` | +| Request body invalid (e.g., contains removed 1.\* fields) | — | `422` | +| Object store unreachable | `500` | `503` | + +Validation results are saved to `{S3_RESULTS_PREFIX}/{id}.json` (by default `validation-results/.json`) instead of `{crate_id}_validation/validation_status.txt`. The [API reference](api.md) documents the current endpoints in full. + +## Existing RO-Crates and results + +The service now finds an RO-Crate at a fixed key rather than by prefix search: a zipped RO-Crate must be at `{S3_CRATE_PREFIX}/.zip` and a directory RO-Crate under `{S3_CRATE_PREFIX}//`, so an existing RO-Crate may need moving into the RO-Crate prefix. Results stored by 1.\* are not read by the new service, so you will need to re-validate an RO-Crate whose result you still need. diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 0000000..a1bb182 --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,15 @@ +site_name: RO-Crate Validation Service +site_url: https://esciencelab.org.uk/RO-Crate-Validation-Service/ +nav: + - Home: index.md + - Installation: installation.md + - Upgrading: upgrading.md + - API: api.md + - Five Safes RO-Crate: five-safes.md + - Contributions: contribution.md +theme: readthedocs +plugins: + - search + - mermaid2 +markdown_extensions: + - admonition