122 documentation website mkdocs

.github/workflows/deploy_docs.yml

@@ -0,0 +1,54 @@
+name: Deploy Documentation Site
+
+on:
+  push:
+    branches: [ main ]
+    paths:
+      - "docs/**"
+      - "mkdocs.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+
+concurrency:
+  group: "pages"
+  cancel-in-progress: false
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
+
+      - name: Set up Python
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: '3.11'
+
+      - name: Install MkDocs
+        run: |
+          python -m pip install --upgrade pip
+          pip install mkdocs mkdocs-mermaid2-plugin
+
+      - name: Build site
+        run: mkdocs build --strict
+
+      - name: Upload Pages artifact
+        uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0
+        with:
+          path: site
+
+  deploy:
+    environment:
+      name: github-pages
+      url: ${{ steps.deployment.outputs.page_url }}
+    runs-on: ubuntu-latest
+    needs: build
+    steps:
+      - name: Deploy to GitHub Pages
+        id: deployment
+        uses: actions/deploy-pages@cd2ce8fcbc39b97be8ca5fce6e763baed58fa128 # v5.0.0

docs/api.md

@@ -0,0 +1,120 @@
+# API Reference
+
+The examples below use the Compose stack's local address, `http://localhost:5001`. Note that the service serves its own OpenAPI specification at `/docs`.
+
+!!! note
+    `POST /v1/ro_crates/validate_metadata` is always available, but the **storage-backed endpoints** are only available when the service runs with `STORAGE_ENABLED=true` (see [Installation & Setup](installation.md#enabling-object-storage)); without this set, `POST /v1/ro_crates/{crate_id}/validation` or `GET /v1/ro_crates/{crate_id}/validation` requests will return a `404` result. 
+
+## Validate metadata
+
+`POST /v1/ro_crates/validate_metadata`
+
+This validates the contents of an `ro-crate-metadata.json` document and returns the result in the response.
+
+| Field | Required | Description |
+|-------|----------|-------------|
+| `crate_json` | yes | The metadata document, as a JSON string |
+| `profile_name` | no | Profile to validate against, e.g. `ro-crate-1.2`. The validator will default to `ro-crate-1.1` when this is omitted |
+
+!!! warning
+    Currently, the validation profile is not detected from the RO-Crate. In other words, a `conformsTo` declaration in the metadata does not influence which validation profile is used by the validator, and the validation always runs against `profile_name`, or `ro-crate-1.1` when it is omitted.
+
+To validate an ro-crate metadata file:
+
+```bash
+jq -Rs '{crate_json: .}' ro-crate-metadata.json | curl -X POST http://localhost:5001/v1/ro_crates/validate_metadata -H 'Content-Type: application/json' -d @-
+```
+
+To choose a profile, add it to the json object as a `profile_name` entry:
+
+```bash
+jq -Rs '{crate_json: ., profile_name: "ro-crate-1.2"}' ro-crate-metadata.json | curl -X POST http://localhost:5001/v1/ro_crates/validate_metadata -H 'Content-Type: application/json' -d @-
+```
+
+| Code | Meaning |
+|------|---------|
+| `200` | Validated; the result has a `status` of `valid` or `invalid` |
+| `422` | `crate_json` is either missing, empty or invalid, or the validation could not run (an `error` result) |
+
+## Validate a stored RO-Crate
+
+`POST /v1/ro_crates/{crate_id}/validation`
+
+This queues validation of an RO-Crate held in the object store. The RO-Crate is resolved first, so a missing or ambiguous crate ID may be reported immediately; the validation process itself runs asynchronously on a worker. 
+
+!!! note
+    See [Crate IDs](#crate-ids) for how `{crate_id}` maps to objects in the bucket.
+
+| Field | Required | Description |
+|-------|----------|-------------|
+| `profile_name` | no | Profile to validate against; defaults to `ro-crate-1.1` when omitted |
+| `webhook_url` | no | URL that receives the result when validation finishes |
+
+```bash
+curl -X POST http://localhost:5001/v1/ro_crates/my-dataset-2026/validation -H 'Content-Type: application/json' -d '{"profile_name": "ro-crate-1.2"}'
+```
+
+| Code | Meaning |
+|------|---------|
+| `202` | Queued; the body is `{"message": "Validation in progress"}` |
+| `400` | Invalid Crate ID |
+| `404` | Either storage mode is not enabled, or there is no RO-Crate at the location defined by the given Crate ID |
+| `409` | Both a zip and a directory exist for this Crate ID |
+| `422` | Request body invalid |
+| `503` | Object store unreachable |
+
+## Fetch a validation result
+
+`GET /v1/ro_crates/{crate_id}/validation`
+
+This returns the stored result for an RO-Crate.
+
+```bash
+curl http://localhost:5001/v1/ro_crates/my-dataset-2026/validation
+```
+
+| Code | Meaning |
+|------|---------|
+| `200` | The stored result, including persisted `error` results |
+| `400` | Invalid Crate ID |
+| `404` | No result stored for this Crate ID |
+
+## Validation results
+
+Every validation produces a result object:
+
+```json
+{
+  "status": "invalid",
+  "profile": "ro-crate-1.2",
+  "created_at": "2026-07-22T10:30:00+00:00",
+  "detail": {}
+}
+```
+
+An RO-Crate's `status` can be:
+
+| `status` | Meaning |
+|----------|---------|
+| `valid` | The RO-Crate conforms to the profile |
+| `invalid` | The RO-Crate does not conform to the profile, issues listed in `detail` field |
+| `error` | The validation could not run; the reason is in an `error` field. No `detail` field is provided |
+
+!!! note
+    `detail` contains the complete validation report. `created_at` is the UTC time of a stored-crate validation, and `null` for metadata-only validation, which does not set it. `profile` is the requested profile name, or `null` when the default (`ro-crate-1.1`) was used.
+
+For stored RO-Crates the same object is saved to `{S3_RESULTS_PREFIX}/<id>.json` and returned by the GET endpoint.
+
+## Webhooks
+
+If `webhook_url` was given, the worker POSTs the result object to it as JSON once validation finishes. The result is saved to the store first and the webhook sent after, so a notification is never sent for a result that was not stored. 
+
+Note that delivery is attempted three times, waiting `0.5s` then `1s` between attempts, with a `10s` timeout per attempt.
+
+## Crate IDs
+
+A Crate ID is the label in the URL path that identifies an RO-Crate in the object store: the service looks for `{S3_CRATE_PREFIX}/<id>.zip` (zip) or `{S3_CRATE_PREFIX}/<id>/` (directory). Crate IDs must match `^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$`: they start with a letter or digit, may contain letters, digits, `.`, `_` and `-`, and are at most 128 characters long. Anything else is rejected with `400`.
+
+## Health
+
+`GET /healthz` reports that the process is up, and always returns `200 {"status": "ok"}`. `GET /readyz` checks the object store and Celery broker, returning `200` when ready and `503` otherwise, with the individual checks in the body. When storage is off, both checks report `disabled`.

docs/contribution.md

@@ -0,0 +1,121 @@
+# Development and Contributions
+
+The [RO-Crate Validation Service](https://github.com/eScienceLab/RO-Crate-Validation-Service) is an open source project and welcomes contributions of all kinds: bug reports, code or documentation changes, and reviews of proposed changes. The underlying [rocrate-validator tool](https://github.com/crs4/rocrate-validator) is also open source, and is a separate project maintained by CRS4. 
+
+This service is written with Python 3.11, built on Flask/APIFlask and Celery, and wraps the [`rocrate-validator`](https://rocrate-validator.readthedocs.io/) in a REST API. The RO-Crate Validation Service enables pipelines, other services, and Trusted Research Environments (TREs) to validate an RO-Crate over HTTP without running the validator themselves.
+
+## Contributing
+
+The easiest way to start contributing is to create an issue, either to let us know of a bug or error, or to propose a piece of work you want to do. For the RO-Crate Validation Service (the API service, Docker image, and Compose stack) use the [RO-Crate Validation Service issues](https://github.com/eScienceLab/RO-Crate-Validation-Service/issues) page. Issues with the validation checks themselves belong to the underlying tool rather than this service: report those on the [rocrate-validator issues](https://github.com/crs4/rocrate-validator/issues) page, and follow that project's own contribution guidance.
+
+### Code contributions
+
+If you want to contribute code changes via GitHub then you may want to read ['How to Contribute to an Open Source Project on GitHub'](https://egghead.io/courses/how-to-contribute-to-an-open-source-project-on-github). We use [GitHub flow](https://docs.github.com/en/get-started/using-github/github-flow) to manage changes:
+
+1. Create a new branch in your local clone of this repository for each significant change.
+2. Commit the change in that branch.
+3. Push that branch to your fork of this repository on GitHub.
+4. Submit a pull request from that branch to the [upstream repository](https://github.com/eScienceLab/RO-Crate-Validation-Service).
+5. If you receive feedback, make the changes in your local clone and push to your branch on GitHub: the pull request will update automatically.
+
+!!! warning
+    Note that we use the `develop` branch for development work, and this is where your PR should be aimed. The `main` branch is used for releases, and only pull requests from the `develop` branch are accepted to this.
+
+## Development stack
+
+The development Compose file builds the image from the local `Dockerfile` and mounts the repository's test profiles into both the API and worker containers:
+
+```bash
+docker compose -f docker-compose-develop.yml up --build
+```
+
+Here `--build` matters: without it, Compose reuses the previously built image and local code changes are not picked up. Add `--profile objectstore` to start the bundled RustFS store for storage-backed work; configuration is the same as in [Installation & Setup](installation.md#configuration-reference).
+
+## Tests
+
+Install the development dependencies, then run the unit tests, which do not use Docker Engine:
+
+```bash
+pip install -r requirements-dev.txt
+```
+
+```bash
+pytest --ignore=tests/test_integration.py
+```
+
+The integration tests bring up the full Compose stack (including the object store) and seed crates with `boto3`, for which they need Docker Engine to be running:
+
+```bash
+pytest tests/test_integration.py
+```
+
+`tests/` mirrors the layout of the `app/` package, so the tests for a module are in the matching directory.
+
+## Linting
+
+The project uses [Ruff](https://docs.astral.sh/ruff/) for linting and formatting, configured in `pyproject.toml`:
+
+```bash
+ruff check . && ruff format --check .
+```
+
+## Dependencies
+
+Direct dependencies are declared in `pyproject.toml`; while the `requirements*.txt` files are locks generated using `pip-compile`:
+
+```bash
+pip-compile pyproject.toml -o requirements.txt
+```
+
+```bash
+pip-compile --extra dev pyproject.toml -o requirements-dev.txt
+```
+
+## Continuous Integration
+
+Pull requests to `develop` will trigger three workflows: unit tests, integration tests (which start the Compose stack), and lint (`ruff check` and `ruff format --check`).
+
+## How the API works
+
+The API server handles HTTP and runs metadata-only validation inline. Object storage-backed validation is queued through Redis to a Celery worker, which reads the crate from the S3-compatible store, validates it, and writes the result back:
+
+```mermaid
+flowchart LR
+    Client([Client])
+    API["Flask API"]
+    Broker[("Redis")]
+    Worker["Celery worker"]
+    Validator["rocrate-validator"]
+    Store[("S3-compatible store")]
+
+    Client --> API
+    API -->|metadata-only: inline| Validator
+    API --> Broker --> Worker --> Validator
+    Worker <--> Store
+```
+
+The worker runs its stages strictly in order: fetch, validate, persist, webhook; so a storage write failure can never be followed by a success notification, and every outcome (including `error` outcomes) is persisted so a later `GET` reflects what happened.
+
+## Project structure
+
+```
+app/
+├── __init__.py                 # app factory: config, blueprints, error handlers, request IDs
+├── health.py                   # /healthz and /readyz
+├── storage/                    # object-storage abstraction
+│   ├── base.py                 #   StorageBackend protocol
+│   ├── s3.py                   #   boto3 implementation (any S3-compatible store)
+│   ├── memory.py               #   in-memory backend (tests / local)
+│   └── errors.py               #   StorageError, ObjectNotFound
+├── crates/                     # crate identity, layout, resolution
+│   ├── ids.py                  #   Crate ID validation
+│   ├── layout.py               #   object keys
+│   └── resolver.py             #   deterministic zip/directory resolution
+├── validation/                 # validation boundary
+│   ├── results.py              #   ValidationOutcome (valid/invalid/error)
+│   └── runner.py               #   wraps rocrate-validator
+├── ro_crates/routes/           # HTTP endpoints (metadata + ID-based)
+├── services/                   # request handling and logging
+├── tasks/validation_tasks.py   # Celery task: fetch, validate, persist, webhook
+└── utils/                      # validated settings, webhook delivery
+```

docs/five-safes.md

@@ -0,0 +1,98 @@
+# Five Safes RO-Crate validation
+
+The [Five Safes RO-Crate 0.4 profile](https://trefx.uk/5s-crate/) describes an RO-Crate used to request and record workflow runs on sensitive data in Trusted Research Environments (TREs), supporting the Five Safes framework. The RO-Crate Validation Service validates against this profile when `profile_name` is set to `five-safes-crate`.
+
+!!! warning
+    Note that the Five Safes RO-Crate 0.4 profile is not bundled with the base validator, so the service needs the profile to be made available. There are two ways to do this, described below.
+
+## Getting a service with the profile
+
+The prebuilt `ghcr.io/esciencelab/ro-crate-validation-service-fivesafes-profile` image packages the `five-safes-crate` profile with the standard RO-Crate Validation Service. The image also carries a pre-warmed validation cache, so it supports offline validation (`VALIDATION_OFFLINE=true`) inside restricted networks. The profile version is fixed when the image is built, and is recorded in the image label `org.ro-crate-validation-service.five-safes-profile-version`.
+
+Alternatively, you may run the standard service image with the profile directory mounted and `EXTRA_PROFILES_PATH` set, [as described in custom profiles](installation.md#custom-profiles). 
+
+The `five-safes-crate` profile itself is defined in the [eScienceLab rocrate-validator fork](https://github.com/eScienceLab/rocrate-validator).
+
+## Validating a Five Safes RO-Crate
+
+A [complete example crate](https://github.com/eScienceLab/rocrate-validator/blob/five-safes-0.7.4-beta/tests/data/crates/valid/five-safes-crate-result/ro-crate-metadata.json) is available in the fork's test data.
+
+!!! note
+    The current `-fivesafes-profile` image pairs the profile with a base profile for RO-Crate 1.1, whilst the profile itself expects RO-Crate 1.2. The walkthrough below mounts the matched profile set instead.
+
+For this walkthrough, run the service from a checkout of this repository, with the repository's profile set mounted in place of the bundled profiles:
+
+```bash
+docker run --rm -p 5001:5000 \
+  -e FLASK_APP=wsgi.py \
+  -e PROFILES_PATH=/app/profiles \
+  -v "$PWD/tests/data/rocrate_validator_profiles:/app/profiles:ro" \
+  ghcr.io/esciencelab/ro-crate-validation-service-fivesafes-profile:latest
+```
+
+Download the example `ro-crate-metadata.json`:
+
+```bash
+curl -sO https://raw.githubusercontent.com/eScienceLab/rocrate-validator/five-safes-0.7.4-beta/tests/data/crates/valid/five-safes-crate-result/ro-crate-metadata.json
+```
+
+and validate it with `profile_name` set to `five-safes-crate`:
+
+```bash
+jq -Rs '{crate_json: ., profile_name: "five-safes-crate"}' ro-crate-metadata.json | curl -X POST http://localhost:5001/v1/ro_crates/validate_metadata -H 'Content-Type: application/json' -d @-
+```
+
+The crate conforms, so the response (abridged) is:
+
+```json
+{
+  "status": "valid",
+  "profile": "five-safes-crate",
+  "created_at": null,
+  "detail": {
+    "issues": [],
+    "passed": true
+  }
+}
+```
+
+To see how conformance issues are reported, remove something the profile requires, such as the `CreateAction` recording the requested workflow run. Validate again:
+
+```bash
+jq '."@graph" |= map(select(."@type" != "CreateAction"))' ro-crate-metadata.json > broken.json
+```
+
+```bash
+jq -Rs '{crate_json: ., profile_name: "five-safes-crate"}' broken.json | curl -X POST http://localhost:5001/v1/ro_crates/validate_metadata -H 'Content-Type: application/json' -d @-
+```
+
+The result will now show `invalid`, and each entry in `detail.issues` identifies the failed check, its severity, and the entity at fault (abridged):
+
+```json
+{
+  "status": "invalid",
+  "profile": "five-safes-crate",
+  "detail": {
+    "issues": [
+      {
+        "check": {
+          "identifier": "five-safes-crate-0.4_25.1",
+          "name": "mentions"
+        },
+        "severity": "REQUIRED",
+        "message": "`RootDataEntity` MUST reference at least one `CreateAction` through `mentions`",
+        "violatingEntity": "./"
+      }
+    ],
+    "passed": false
+  }
+}
+```
+
+Complete RO-Crates work the same way through the storage-backed endpoints: upload the crate as `crates/<id>.zip` (or a directory under `crates/<id>/`), then queue validation with the profile:
+
+```bash
+curl -X POST http://localhost:5001/v1/ro_crates/my-5s-crate/validation -H 'Content-Type: application/json' -d '{"profile_name": "five-safes-crate"}'
+```
+
+The [API reference](api.md) covers the endpoints, results and webhooks in full.