From 1c098adcfc218de5410ebc7a2f5fd165e5ec36e2 Mon Sep 17 00:00:00 2001 From: Douglas Lowe <10961945+douglowe@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:04:21 +0100 Subject: [PATCH 01/19] mkdocs config --- mkdocs.yml | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 mkdocs.yml diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 0000000..7ad4a73 --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,2 @@ +site_name: Crate Validation Service +theme: readthedocs From 3dd53919d1cd44c8289e05b7d32bc075837174ec Mon Sep 17 00:00:00 2001 From: Douglas Lowe <10961945+douglowe@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:04:39 +0100 Subject: [PATCH 02/19] docs index page --- docs/index.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 docs/index.md diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..59246bc --- /dev/null +++ b/docs/index.md @@ -0,0 +1,21 @@ +# RO-Crate Validation Service + +[RO-Crates](https://www.researchobject.org/ro-crate/) are a lightweight approach to packaging research data and metadata. +They are designed to be easy to create and consume, and can be used to share research data in a way that is both human and machine-readable. + +To ensure that RO-Crates are valid and can be consumed by others, we need to validate them against a set of standards. +The base RO-Crate standards are defined in the [RO-Crate Metadata Specification](https://www.researchobject.org/ro-crate/1.2/metadata.html), +and include requirements for the structure of the RO-Crate, the metadata it contains, and the files it includes. +On top of this, there are also community-specific extensions to the RO-Crate standards, +such as the [Five Safes RO-Crate extension](https://trefx.uk/5s-crate/), for working with sensitive data within Trusted Research Environments (TREs). + +This validation process can be complex, requiring both structural and semantic checks to ensure that the RO-Crate is compliant with the relevant standards. +The [rocrate-validator](https://rocrate-validator.readthedocs.io/en/latest/) provides a means for the structural validation of RO-Crates, +checking that they conform to the required structure and contain the necessary metadata. +These checks are carried out using a combination of [SHACL](https://www.w3.org/TR/shacl/) shapes, a language for validating RDF graphs against a set of constraints, and python functions. +It is available on [PyPI](https://pypi.org/project/roc-validator/) and can be installed using pip. + +This RO-Crate Validation Service provides a REST API for validating RO-Crates using the rocrate-validator. +It is built in python using flask, and is provided as a docker image for ease of deployment. +Several base profiles are included, while more can be added as needed when the service is deployed. +The service accepts RO-Crates as zip files, and returns a validation report in JSON format, detailing any issues found with the RO-Crate. From d90ab08e31cd79ea81be58dc9f620a4056b9b67d Mon Sep 17 00:00:00 2001 From: Douglas Lowe <10961945+douglowe@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:08:35 +0100 Subject: [PATCH 03/19] docs index list --- mkdocs.yml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/mkdocs.yml b/mkdocs.yml index 7ad4a73..278d0ff 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,2 +1,8 @@ site_name: Crate Validation Service +nav: + - Home: index.md + - Installation: installation.md + - API: api.md + - ENTRUST: entrust.md + - Contributions: contribution.md theme: readthedocs From 07e3f73c81af0ca989a13e4b4b9b4eb426aa150a Mon Sep 17 00:00:00 2001 From: Douglas Lowe <10961945+douglowe@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:11:53 +0100 Subject: [PATCH 04/19] original api description --- docs/api.md | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 116 insertions(+) create mode 100644 docs/api.md diff --git a/docs/api.md b/docs/api.md new file mode 100644 index 0000000..7c7c8a0 --- /dev/null +++ b/docs/api.md @@ -0,0 +1,116 @@ +# Crate Validator REST API + +## Overview + +The Crate Validator REST API exposes operations for validating stored +RO-Crates, validating submitted RO-Crate metadata, and retrieving saved +validation results. + +Stored RO-Crate validation uses MinIO-compatible object storage. Requests that +operate on stored crates include `minio_config`, which tells the service where +the crate is stored and which bucket to use. + + +## Endpoints + +### `POST /v1/ro_crates/{crate_id}/validation` + +Queues validation for an RO-Crate stored in MinIO-compatible object storage. +The `crate_id` path parameter is the name used to find the crate object in the +configured bucket. + +Request body: + +```jsonc +{ + "minio_config": { + "endpoint": "string", // required, e.g. "localhost:9000" or "minio:9000" + "accesskey": "string", // required, MinIO access key or username + "secret": "string", // required, MinIO secret key or password + "ssl": false, // required, true when the MinIO endpoint uses HTTPS + "bucket": "string" // required, bucket containing the RO-Crate + }, + "root_path": "string", // optional folder/path inside the bucket + "profile_name": "string" // optional validation profile name +} +``` + +Expected responses: + +- `202`: validation queued. +- `400`: RO-Crate does not exist or validation request cannot be satisfied. +- `500`: internal service, MinIO, Celery, or validation error. + + +### `GET /v1/ro_crates/{crate_id}/validation` + +Fetches the latest validation result from MinIO-compatible object storage. + +Request body: + +```jsonc +{ + "minio_config": { + "endpoint": "string", // required, e.g. "localhost:9000" or "minio:9000" + "accesskey": "string", // required, MinIO access key or username + "secret": "string", // required, MinIO secret key or password + "ssl": false, // required, true when the MinIO endpoint uses HTTPS + "bucket": "string" // required, bucket containing the RO-Crate + }, + "root_path": "string" // optional folder/path inside the bucket +} +``` + +Expected responses: + +- `200`: validation result JSON returned. +- `400`: RO-Crate or validation result is missing. +- `500`: MinIO or internal retrieval error. + + +### `POST /v1/ro_crates/validate_metadata` + +Validates a submitted RO-Crate metadata JSON string. + +Request body: + +```jsonc +{ + "crate_json": "string", // required, stringified content of ro-crate-metadata.json + "profile_name": "string" // optional validation profile name +} +``` + +Expected responses: + +- `200`: validation result returned. +- `422`: missing, malformed, or empty metadata JSON. +- `500`: internal validation error. + + +## Validation Profiles + +The optional `profile_name` field selects a specific RO-Crate validation +profile. If omitted, the service uses the validator default. + +Custom profile definitions can be made available to the service through the +`PROFILES_PATH` environment variable. + + +## Result Storage + +Validation results for stored RO-Crates are saved back to MinIO-compatible +object storage. + +Without `root_path`, results are stored at: + +```text +{crate_id}_validation/validation_status.txt +``` + +With `root_path`, results are stored at: + +```text +{root_path}/{crate_id}_validation/validation_status.txt +``` + From fda67c2365dfc1d58e55a256374e7d3b50370dd1 Mon Sep 17 00:00:00 2001 From: Douglas Lowe <10961945+douglowe@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:34:51 +0100 Subject: [PATCH 05/19] initial installation page --- docs/installation.md | 106 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 106 insertions(+) create mode 100644 docs/installation.md diff --git a/docs/installation.md b/docs/installation.md new file mode 100644 index 0000000..c8f3f52 --- /dev/null +++ b/docs/installation.md @@ -0,0 +1,106 @@ +# Installation + +## Service Snapshot + +Cratey Validator is a web service that checks whether RO-Crates follow the +expected structure and metadata rules. It can validate a complete RO-Crate +stored in MinIO-compatible object storage, or it can validate the contents +of an `ro-crate-metadata.json` file directly. + +At a high level, a client sends a validation request, the service runs the +RO-Crate validation checks, and the result is either returned directly or +saved so it can be retrieved later. + +## What You Can Validate + +### Stored RO-Crates + +Use this option when the RO-Crate already exists in MinIO-compatible object +storage. The service reads the crate from the configured bucket, runs the +validation checks, and saves the validation result back to object storage. + +This is useful for validating complete crates as part of an upload, review, +or publication workflow. + + +### Metadata Files + +Use this option when you only need to check the contents of +`ro-crate-metadata.json`. Instead of asking the service to download a full +crate, you submit the metadata JSON directly and receive the validation result +in the response. + +This is useful for quick checks while editing metadata or before a complete +crate has been assembled. + + +### Saved Validation Results + +After a stored RO-Crate has been validated, the saved validation result can be +retrieved later. This lets another application or user interface show the most +recent validation status without rerunning the checks. + + +## Before You Start + +For stored RO-Crate validation, you need: + +- A MinIO-compatible object store. +- A bucket containing the RO-Crate. +- Access credentials for that bucket. +- The crate identifier used by the object store. + +For metadata-only validation, you only need the contents of +`ro-crate-metadata.json`. + + +## Running The Service + +The service can be started with Docker Compose: + +```bash +docker compose up --build +``` + +For local container development, use: + +```bash +docker compose --file docker-compose-develop.yml up --build +``` + +Expected local services: + +- Flask API: `http://localhost:5001` +- MinIO API: `http://localhost:9000` +- MinIO console: `http://localhost:9001` +- Redis: `localhost:6379` + +MinIO needs a bucket for RO-Crates, normally `ro-crates`. Bucket versioning +should be enabled so uploaded crate objects can be tracked reliably. + + +## Configuration + +The main environment variables are: + +- `FLASK_APP`: Flask entrypoint, normally `cratey.py`. +- `FLASK_ENV`: selects development or production config. +- `CELERY_BROKER_URL`: Redis broker URL. +- `CELERY_RESULT_BACKEND`: Redis result backend URL. +- `PROFILES_PATH`: optional path to custom RO-Crate validator profile +definitions. +- `MINIO_ENDPOINT`: default MinIO endpoint used by Docker examples. +- `MINIO_ROOT_USER`: MinIO root username for local development. +- `MINIO_ROOT_PASSWORD`: MinIO root password. +- `MINIO_BUCKET_NAME`: default bucket name used by local setup. + +API calls also pass MinIO access details in `minio_config`, so the service can validate crates in a specified object store and bucket. + +## More Information + +- For endpoint paths, request bodies, response codes, validation profiles, and +result storage paths, see the [REST API documentation](./rest_api). +- For implementation details, service components, runtime flow, and test +coverage, see the [Architecture documentation](./architecture). +- For deployment context and the architecture diagram, see +[Deployment](./deployment). \ No newline at end of file From f24a31d58f14e2b989add48f128061a376e5edd2 Mon Sep 17 00:00:00 2001 From: Douglas Lowe <10961945+douglowe@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:35:19 +0100 Subject: [PATCH 06/19] initial entrust page --- docs/entrust.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/entrust.md diff --git a/docs/entrust.md b/docs/entrust.md new file mode 100644 index 0000000..489edda --- /dev/null +++ b/docs/entrust.md @@ -0,0 +1 @@ +# ENTRUST \ No newline at end of file From ed1cfa5f25524b796f0240b615e3d3bb575eee17 Mon Sep 17 00:00:00 2001 From: Douglas Lowe <10961945+douglowe@users.noreply.github.com> Date: Wed, 22 Jul 2026 14:35:44 +0100 Subject: [PATCH 07/19] initial contributing page --- docs/contribution.md | 1 + 1 file changed, 1 insertion(+) create mode 100644 docs/contribution.md diff --git a/docs/contribution.md b/docs/contribution.md new file mode 100644 index 0000000..d292e89 --- /dev/null +++ b/docs/contribution.md @@ -0,0 +1 @@ +# Development / Contributions \ No newline at end of file From 2902f629bf534dff4220babf402cea7a729b15cc Mon Sep 17 00:00:00 2001 From: Alex Hambley <33315205+alexhambley@users.noreply.github.com> Date: Wed, 22 Jul 2026 16:17:33 +0100 Subject: [PATCH 08/19] chore(docs): rewrite index page for the current service --- docs/index.md | 58 ++++++++++++++++++++++++++++++++++----------------- 1 file changed, 39 insertions(+), 19 deletions(-) diff --git a/docs/index.md b/docs/index.md index 59246bc..254e9d2 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,21 +1,41 @@ # RO-Crate Validation Service -[RO-Crates](https://www.researchobject.org/ro-crate/) are a lightweight approach to packaging research data and metadata. -They are designed to be easy to create and consume, and can be used to share research data in a way that is both human and machine-readable. - -To ensure that RO-Crates are valid and can be consumed by others, we need to validate them against a set of standards. -The base RO-Crate standards are defined in the [RO-Crate Metadata Specification](https://www.researchobject.org/ro-crate/1.2/metadata.html), -and include requirements for the structure of the RO-Crate, the metadata it contains, and the files it includes. -On top of this, there are also community-specific extensions to the RO-Crate standards, -such as the [Five Safes RO-Crate extension](https://trefx.uk/5s-crate/), for working with sensitive data within Trusted Research Environments (TREs). - -This validation process can be complex, requiring both structural and semantic checks to ensure that the RO-Crate is compliant with the relevant standards. -The [rocrate-validator](https://rocrate-validator.readthedocs.io/en/latest/) provides a means for the structural validation of RO-Crates, -checking that they conform to the required structure and contain the necessary metadata. -These checks are carried out using a combination of [SHACL](https://www.w3.org/TR/shacl/) shapes, a language for validating RDF graphs against a set of constraints, and python functions. -It is available on [PyPI](https://pypi.org/project/roc-validator/) and can be installed using pip. - -This RO-Crate Validation Service provides a REST API for validating RO-Crates using the rocrate-validator. -It is built in python using flask, and is provided as a docker image for ease of deployment. -Several base profiles are included, while more can be added as needed when the service is deployed. -The service accepts RO-Crates as zip files, and returns a validation report in JSON format, detailing any issues found with the RO-Crate. +The RO-Crate Validation Service evaluates whether [RO-Crates](https://www.researchobject.org/ro-crate/) conform to the RO-Crate specification and to community profiles. The service wraps the [`rocrate-validator`](https://rocrate-validator.readthedocs.io/) library in a REST API, and is deployed as a Docker image. The RO-Crate Validation Service enables pipelines, other services, and Trusted Research Environments (TREs) to validate crates over HTTP without running the validator themselves. + +## RO-Crates and profiles in brief + +An [RO-Crate](https://www.researchobject.org/ro-crate/) packages research data together with structured, machine-readable metadata: a JSON-LD file named `ro-crate-metadata.json`. Validating an RO-Crate evaluates that metadata against a **profile**. A profile is a set of requirements the RO-Crate must satisfy. That can be the base requirements of the [RO-Crate specification](https://www.researchobject.org/ro-crate/1.2/) itself, or a [community profile](https://www.researchobject.org/ro-crate/profiles) that adds domain-specific rules, such as the [Five Safes RO-Crate profile](https://trefx.uk/5s-crate/) for working with sensitive data in TREs. + +Whilst the validation checks themselves are performed by [`rocrate-validator`](https://rocrate-validator.readthedocs.io/), this service is a deployable HTTP wrapper around that tool: it adds a web API, asynchronous processing, and object-storage integration. The base validation rules come from the validator itself; we additionally package our own [Five Safes RO-Crate profile](five-safes.md) rules with the service for working in TREs. + +## Validation methods + +1. **Metadata-only**. Send the contents of an `ro-crate-metadata.json` file and receive the validation result in the response. This is synchronous and stateless, so nothing is stored, and no object store or worker is required. This approach is simpler to use, and is intended for quick evaluations whilst metadata is being written, or before a full crate has been assembled. + +2. **Storage-backed**. The service reads complete RO-Crates (zip or directory) from an S3-compatible object store, such as RustFS, AWS S3, MinIO, and others. Validation runs asynchronously on a worker process; the result is stored for later retrieval and can optionally be delivered to a webhook. This is more suited to pipelines and workflows. + +```mermaid +flowchart LR + Client([Your application]) + API["REST API"] + Validator["rocrate-validator"] + Worker["Celery worker"] + Store[("S3-compatible store")] + + Client -->|HTTP| API + API -->|metadata-only: validated inline| Validator + API -->|storage-backed: queued| Worker + Worker --> Validator + Worker <--> Store +``` + +## Documentation + +- To run the service yourself, start with [Installation & Setup](installation.md), or with the [Upgrade Guide](upgrading.md) if you already run a 1.\* version. +- The [API Reference](api.md) documents the endpoints for anyone integrating against a running instance. +- The [Five Safes RO-Crate](five-safes.md) page walks through validating RO-Crates in a TRE. +- To help contribute to the service, see [Development](development.md). + +## About + +The RO-Crate Validation Service is developed by the [eScience Lab](https://esciencelab.org.uk/) at The University of Manchester, and available on [GitHub](https://github.com/eScienceLab/RO-Crate-Validation-Service) under the MIT licence. From b7d26598d6ff1c311c8d2e8b3f20681645827b19 Mon Sep 17 00:00:00 2001 From: Alex Hambley <33315205+alexhambley@users.noreply.github.com> Date: Thu, 23 Jul 2026 10:02:40 +0100 Subject: [PATCH 09/19] chore(docs): rewrite installation page; set mermaid and markdown_extensions plugins --- docs/installation.md | 135 ++++++++++++++++++++----------------------- mkdocs.yml | 5 ++ 2 files changed, 68 insertions(+), 72 deletions(-) diff --git a/docs/installation.md b/docs/installation.md index c8f3f52..3eab1f2 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,106 +1,97 @@ -# Installation +# Installation & Setup -## Service Snapshot +The service runs with Docker Compose. The default setup validates metadata documents only; enabling object storage adds validation of complete stored crates. If you are upgrading from a 1.\* release, see the [upgrade guide](upgrading.md) first. -Cratey Validator is a web service that checks whether RO-Crates follow the -expected structure and metadata rules. It can validate a complete RO-Crate -stored in MinIO-compatible object storage, or it can validate the contents -of an `ro-crate-metadata.json` file directly. +## Quick start -At a high level, a client sends a validation request, the service runs the -RO-Crate validation checks, and the result is either returned directly or -saved so it can be retrieved later. +You will need Docker with Docker Compose. -## What You Can Validate +First, clone the repository, copy the example environment file, and start the stack: -### Stored RO-Crates - -Use this option when the RO-Crate already exists in MinIO-compatible object -storage. The service reads the crate from the configured bucket, runs the -validation checks, and saves the validation result back to object storage. - -This is useful for validating complete crates as part of an upload, review, -or publication workflow. +```bash +git clone https://github.com/eScienceLab/RO-Crate-Validation-Service.git +cd RO-Crate-Validation-Service +cp example.env .env +docker compose up --build +``` +!!! warning + You should update the default `.env` values when running the object store in production. -### Metadata Files +The API is served at `http://localhost:5001`. Redis and a Celery worker are also started, but are only used once storage is enabled. To check the service is up: -Use this option when you only need to check the contents of -`ro-crate-metadata.json`. Instead of asking the service to download a full -crate, you submit the metadata JSON directly and receive the validation result -in the response. +```bash +curl http://localhost:5001/healthz +``` -This is useful for quick checks while editing metadata or before a complete -crate has been assembled. +This returns `{"status": "ok"}`. +To validate some metadata, post the contents of an `ro-crate-metadata.json` file to the metadata endpoint. The [running example](https://www.researchobject.org/ro-crate/specification/1.2/introduction.html#running-example) from the RO-Crate specification is a good test document. The file needs to be sent as an escaped JSON string, which `jq` can do: -### Saved Validation Results +```bash +jq -Rs '{crate_json: .}' ro-crate-metadata.json | curl -X POST http://localhost:5001/v1/ro_crates/validate_metadata -H 'Content-Type: application/json' -d @- +``` -After a stored RO-Crate has been validated, the saved validation result can be -retrieved later. This lets another application or user interface show the most -recent validation status without rerunning the checks. +The response contains a `status` of `valid`, `invalid` or `error`, along with the detailed findings. The [API reference](api.md) describes the endpoints and result format in full. +## Enabling object storage -## Before You Start +To validate a complete RO-Crate (zip or directory) held in an object store, set `STORAGE_ENABLED=true` in `.env`. Storage mode requires six settings, [described below](#configuration-reference): `S3_ENDPOINT`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`, `S3_BUCKET`, `CELERY_BROKER_URL` and `CELERY_RESULT_BACKEND`. The service will fail at startup if any are missing. The Compose stack already sets the Celery variables to the bundled Redis, so in practice only the S3 settings in `.env` matter here. -For stored RO-Crate validation, you need: +Then start the stack with the bundled development object store (RustFS): -- A MinIO-compatible object store. -- A bucket containing the RO-Crate. -- Access credentials for that bucket. -- The crate identifier used by the object store. +```bash +docker compose --profile objectstore up --build +``` -For metadata-only validation, you only need the contents of -`ro-crate-metadata.json`. +RustFS serves the S3 API on port 9000 and a web console at `http://localhost:9001`. Development credentials are set in `example.env`. +!!! warning + The service does not create the bucket itself. Create the bucket in the console or with any S3 client. The bucket name needs to match `S3_BUCKET` (`ro-crates` by default). -## Running The Service +Upload an RO-Crate under the crate prefix: `crates/.zip` for a zipped RO-Crate, or `crates//` for a directory. Note that for a zipped RO-Crate, `ro-crate-metadata.json` must be at the root of the archive. -The service can be started with Docker Compose: +The readiness endpoint checks the object store and broker connections: ```bash -docker compose up --build +curl http://localhost:5001/readyz ``` -For local container development, use: - -```bash -docker compose --file docker-compose-develop.yml up --build -``` +## Using your own object store -Expected local services: +Any S3-compatible store can be used in place of RustFS, including AWS S3, MinIO and Ceph: set `S3_ENDPOINT`, the credentials and `S3_BUCKET` for your store, and skip the `objectstore` profile. If you already run a 1.\* release against MinIO, the [upgrade guide](upgrading.md) maps the old settings to the new ones. -- Flask API: `http://localhost:5001` -- MinIO API: `http://localhost:9000` -- MinIO console: `http://localhost:9001` -- Redis: `localhost:6379` +## Configuration reference -MinIO needs a bucket for RO-Crates, normally `ro-crates`. Bucket versioning -should be enabled so uploaded crate objects can be tracked reliably. +| Variable | Default | Description | +|----------|---------|-------------| +| `STORAGE_ENABLED` | `false` | Enable the stored-crate endpoints and storage checks | +| `S3_ENDPOINT` | — | Object store endpoint, e.g. `objectstore:9000` (required in storage mode) | +| `S3_ACCESS_KEY` | — | Object store access key (required in storage mode) | +| `S3_SECRET_KEY` | — | Object store secret key (required in storage mode) | +| `S3_BUCKET` | — | Bucket holding crates and results (required in storage mode) | +| `S3_USE_SSL` | `false` | Use HTTPS to the object store | +| `S3_REGION` | — | Region; needed for AWS S3 | +| `S3_CRATE_PREFIX` | `crates` | Key prefix crates are read from | +| `S3_RESULTS_PREFIX` | `validation-results` | Key prefix results are written to | +| `CELERY_BROKER_URL` | — | Redis broker URL (required in storage mode; preset in the Compose stack) | +| `CELERY_RESULT_BACKEND` | — | Celery result backend URL (required in storage mode; preset in the Compose stack) | +| `PROFILES_PATH` | — | Profiles directory that replaces the bundled profiles | +| `EXTRA_PROFILES_PATH` | — | Profiles directory added to the bundled profiles | +| `CACHE_PATH` | `/app/.rocrate-cache` in the published image | Validator HTTP cache location | +| `VALIDATION_OFFLINE` | `false` | Validate using only the cache, with no network access | +| `FLASK_ENV` | `development` | Set to `production` to disable debug behaviour | +## Custom profiles -## Configuration +The validator comes with the base RO-Crate profiles. For Five Safes validation, the prebuilt `ghcr.io/esciencelab/ro-crate-validation-service-fivesafes-profile` image has the `five-safes-crate` profile already included; see [Five Safes validation](five-safes.md). -The main environment variables are: +Adding other profiles works by mounting a directory of profile definitions into both the `flask` and `celery_worker` containers. You set `EXTRA_PROFILES_PATH` to the mounted path. Both containers need the mount as metadata-only validation runs in the API process and stored-crate validation runs in the worker. There is a working example in `docker-compose-develop.yml`. -- `FLASK_APP`: Flask entrypoint, normally `cratey.py`. -- `FLASK_ENV`: selects development or production config. -- `CELERY_BROKER_URL`: Redis broker URL. -- `CELERY_RESULT_BACKEND`: Redis result backend URL. -- `PROFILES_PATH`: optional path to custom RO-Crate validator profile -definitions. -- `MINIO_ENDPOINT`: default MinIO endpoint used by Docker examples. -- `MINIO_ROOT_USER`: MinIO root username for local development. -- `MINIO_ROOT_PASSWORD`: MinIO root password. -- `MINIO_BUCKET_NAME`: default bucket name used by local setup. +`EXTRA_PROFILES_PATH` adds the directory to the bundled profiles, whereas `PROFILES_PATH` replaces them entirely. -API calls also pass MinIO access details in `minio_config`, so the service can validate crates in a specified object store and bucket. +## Offline validation -## More Information +The validator fetches profile and context resources over HTTP and caches them. The published images pre-populate this cache at build time, so setting `VALIDATION_OFFLINE=true` runs validation entirely from the cache, with no network access at runtime. This is useful inside TREs and other restricted networks. -- For endpoint paths, request bodies, response codes, validation profiles, and -result storage paths, see the [REST API documentation](./rest_api). -- For implementation details, service components, runtime flow, and test -coverage, see the [Architecture documentation](./architecture). -- For deployment context and the architecture diagram, see -[Deployment](./deployment). \ No newline at end of file +Online validation (the default) also uses and refreshes the same cache. Offline validation requires `rocrate-validator` at 0.10.0 or later, which the published images include. diff --git a/mkdocs.yml b/mkdocs.yml index 278d0ff..a1ebf4d 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -6,3 +6,8 @@ nav: - ENTRUST: entrust.md - Contributions: contribution.md theme: readthedocs +plugins: + - search + - mermaid2 +markdown_extensions: + - admonition From f055816406d9fb24c2405ab6573a1fc2193cfc47 Mon Sep 17 00:00:00 2001 From: Alex Hambley <33315205+alexhambley@users.noreply.github.com> Date: Thu, 23 Jul 2026 10:21:18 +0100 Subject: [PATCH 10/19] chore(docs): improve clarity for installation.md --- docs/installation.md | 39 +++++++++++++++++++++++++-------------- 1 file changed, 25 insertions(+), 14 deletions(-) diff --git a/docs/installation.md b/docs/installation.md index 3eab1f2..fc43b30 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -1,12 +1,12 @@ # Installation & Setup -The service runs with Docker Compose. The default setup validates metadata documents only; enabling object storage adds validation of complete stored crates. If you are upgrading from a 1.\* release, see the [upgrade guide](upgrading.md) first. +The RO-Crate Validation Service works in two ways: a metadata-only mode, in which the contents of an `ro-crate-metadata.json` file are assessed, and storage-backed validation, where complete RO-Crates (zip or directory) are evaluated. ## Quick start You will need Docker with Docker Compose. -First, clone the repository, copy the example environment file, and start the stack: +To start, clone the repository, copy the example environment file, and start the stack: ```bash git clone https://github.com/eScienceLab/RO-Crate-Validation-Service.git @@ -16,9 +16,11 @@ docker compose up --build ``` !!! warning - You should update the default `.env` values when running the object store in production. + Remember to update the default `.env` values when running the object store in production. -The API is served at `http://localhost:5001`. Redis and a Celery worker are also started, but are only used once storage is enabled. To check the service is up: +The API is served at `http://localhost:5001`. Redis and a Celery worker are also started, but these are only used once storage is enabled. + +To check the service is up, run: ```bash curl http://localhost:5001/healthz @@ -26,19 +28,25 @@ curl http://localhost:5001/healthz This returns `{"status": "ok"}`. -To validate some metadata, post the contents of an `ro-crate-metadata.json` file to the metadata endpoint. The [running example](https://www.researchobject.org/ro-crate/specification/1.2/introduction.html#running-example) from the RO-Crate specification is a good test document. The file needs to be sent as an escaped JSON string, which `jq` can do: +To validate the contents of an `ro-crate-metadata.json` file, post to the metadata endpoint. The [running example](https://www.researchobject.org/ro-crate/specification/1.2/introduction.html#running-example) from the RO-Crate specification is a good test document. + +The file needs to be sent as an escaped JSON string, which `jq` can do: ```bash jq -Rs '{crate_json: .}' ro-crate-metadata.json | curl -X POST http://localhost:5001/v1/ro_crates/validate_metadata -H 'Content-Type: application/json' -d @- ``` -The response contains a `status` of `valid`, `invalid` or `error`, along with the detailed findings. The [API reference](api.md) describes the endpoints and result format in full. +The response contains a `status` of `valid`, `invalid` or `error`, along with the detailed validation. For more information, the [API reference](api.md) describes the endpoints and result format in full. ## Enabling object storage -To validate a complete RO-Crate (zip or directory) held in an object store, set `STORAGE_ENABLED=true` in `.env`. Storage mode requires six settings, [described below](#configuration-reference): `S3_ENDPOINT`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`, `S3_BUCKET`, `CELERY_BROKER_URL` and `CELERY_RESULT_BACKEND`. The service will fail at startup if any are missing. The Compose stack already sets the Celery variables to the bundled Redis, so in practice only the S3 settings in `.env` matter here. +To validate a complete RO-Crate (zip or directory) held in an object store, set `STORAGE_ENABLED=true` in `.env`. + +The storage-backed validation mode requires six settings, [described below](#configuration-reference): `S3_ENDPOINT`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`, `S3_BUCKET`, `CELERY_BROKER_URL` and `CELERY_RESULT_BACKEND`. -Then start the stack with the bundled development object store (RustFS): +The service will fail at startup if any are missing. The Compose stack already sets the Celery variables to the bundled Redis, so in practice only the S3 settings in `.env` matter here. + +Then start the stack with the bundled development object store (RustFS), run: ```bash docker compose --profile objectstore up --build @@ -47,9 +55,9 @@ docker compose --profile objectstore up --build RustFS serves the S3 API on port 9000 and a web console at `http://localhost:9001`. Development credentials are set in `example.env`. !!! warning - The service does not create the bucket itself. Create the bucket in the console or with any S3 client. The bucket name needs to match `S3_BUCKET` (`ro-crates` by default). + The service does not create the bucket itself. Create the bucket in the console or with an S3 client. The bucket name needs to match `S3_BUCKET` (`ro-crates` by default). -Upload an RO-Crate under the crate prefix: `crates/.zip` for a zipped RO-Crate, or `crates//` for a directory. Note that for a zipped RO-Crate, `ro-crate-metadata.json` must be at the root of the archive. +Upload an RO-Crate under the prefix: `crates/.zip` for a zipped RO-Crate, or `crates//` for a directory. Note that for a zipped RO-Crate, `ro-crate-metadata.json` must be at the root of the archive. The readiness endpoint checks the object store and broker connections: @@ -59,7 +67,9 @@ curl http://localhost:5001/readyz ## Using your own object store -Any S3-compatible store can be used in place of RustFS, including AWS S3, MinIO and Ceph: set `S3_ENDPOINT`, the credentials and `S3_BUCKET` for your store, and skip the `objectstore` profile. If you already run a 1.\* release against MinIO, the [upgrade guide](upgrading.md) maps the old settings to the new ones. +Any S3-compatible store can be used in place of RustFS, including AWS S3, MinIO and Ceph: set `S3_ENDPOINT`, the credentials and `S3_BUCKET` for your store. In this case, do not run the `objectstore` profile. + +If you already use a 1.\* release against MinIO, the [upgrade guide](upgrading.md) maps the old settings to the new ones. ## Configuration reference @@ -84,11 +94,12 @@ Any S3-compatible store can be used in place of RustFS, including AWS S3, MinIO ## Custom profiles -The validator comes with the base RO-Crate profiles. For Five Safes validation, the prebuilt `ghcr.io/esciencelab/ro-crate-validation-service-fivesafes-profile` image has the `five-safes-crate` profile already included; see [Five Safes validation](five-safes.md). +The validator comes with several RO-Crate profiles, and for the Five Safes RO-Crate, the prebuilt `ghcr.io/esciencelab/ro-crate-validation-service-fivesafes-profile` image has the `five-safes-crate` profile already included; see [Five Safes validation](five-safes.md). -Adding other profiles works by mounting a directory of profile definitions into both the `flask` and `celery_worker` containers. You set `EXTRA_PROFILES_PATH` to the mounted path. Both containers need the mount as metadata-only validation runs in the API process and stored-crate validation runs in the worker. There is a working example in `docker-compose-develop.yml`. +To add other profiles, mount a directory into both the `flask` and `celery_worker` containers, and set `EXTRA_PROFILES_PATH` to the mounted path. Note that both containers need the mount as metadata-only validation runs in the API process and stored-crate validation runs in the worker. There is a working example in `docker-compose-develop.yml`. -`EXTRA_PROFILES_PATH` adds the directory to the bundled profiles, whereas `PROFILES_PATH` replaces them entirely. +!!! note + `EXTRA_PROFILES_PATH` adds the directory to the bundled profiles, whereas `PROFILES_PATH` replaces them entirely. ## Offline validation From 8f33980019b54f62ee47609d06d316874b210061 Mon Sep 17 00:00:00 2001 From: Alex Hambley <33315205+alexhambley@users.noreply.github.com> Date: Fri, 24 Jul 2026 11:27:41 +0100 Subject: [PATCH 11/19] chore(docs): add 1.x to 2.0 upgrade docs --- docs/index.md | 6 ++--- docs/installation.md | 8 +++---- docs/upgrading.md | 57 ++++++++++++++++++++++++++++++++++++++++++++ mkdocs.yml | 3 ++- 4 files changed, 66 insertions(+), 8 deletions(-) create mode 100644 docs/upgrading.md diff --git a/docs/index.md b/docs/index.md index 254e9d2..5092a03 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,6 +1,6 @@ # RO-Crate Validation Service -The RO-Crate Validation Service evaluates whether [RO-Crates](https://www.researchobject.org/ro-crate/) conform to the RO-Crate specification and to community profiles. The service wraps the [`rocrate-validator`](https://rocrate-validator.readthedocs.io/) library in a REST API, and is deployed as a Docker image. The RO-Crate Validation Service enables pipelines, other services, and Trusted Research Environments (TREs) to validate crates over HTTP without running the validator themselves. +The RO-Crate Validation Service evaluates whether [RO-Crates](https://www.researchobject.org/ro-crate/) conform to the RO-Crate specification and to community profiles. The service wraps the [`rocrate-validator`](https://rocrate-validator.readthedocs.io/) library in a REST API, and is deployed as a Docker image. The RO-Crate Validation Service enables pipelines, other services, and Trusted Research Environments (TREs) to validate an RO-Crate over HTTP without running the validator themselves. ## RO-Crates and profiles in brief @@ -31,10 +31,10 @@ flowchart LR ## Documentation -- To run the service yourself, start with [Installation & Setup](installation.md), or with the [Upgrade Guide](upgrading.md) if you already run a 1.\* version. +- To run the service yourself, start with [Installation & Setup](installation.md), or with the [Upgrade Guide](upgrading.md) if you already run a 1.x version. - The [API Reference](api.md) documents the endpoints for anyone integrating against a running instance. - The [Five Safes RO-Crate](five-safes.md) page walks through validating RO-Crates in a TRE. -- To help contribute to the service, see [Development](development.md). +- To help contribute to the service, please see the [Contribution Guide](contribution.md). ## About diff --git a/docs/installation.md b/docs/installation.md index fc43b30..f5be8e2 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -69,7 +69,7 @@ curl http://localhost:5001/readyz Any S3-compatible store can be used in place of RustFS, including AWS S3, MinIO and Ceph: set `S3_ENDPOINT`, the credentials and `S3_BUCKET` for your store. In this case, do not run the `objectstore` profile. -If you already use a 1.\* release against MinIO, the [upgrade guide](upgrading.md) maps the old settings to the new ones. +If you already use a 1.x release against MinIO, the [upgrade guide](upgrading.md) maps the old settings to the new ones. ## Configuration reference @@ -79,16 +79,16 @@ If you already use a 1.\* release against MinIO, the [upgrade guide](upgrading.m | `S3_ENDPOINT` | — | Object store endpoint, e.g. `objectstore:9000` (required in storage mode) | | `S3_ACCESS_KEY` | — | Object store access key (required in storage mode) | | `S3_SECRET_KEY` | — | Object store secret key (required in storage mode) | -| `S3_BUCKET` | — | Bucket holding crates and results (required in storage mode) | +| `S3_BUCKET` | — | Bucket holding RO-Crates and results (required in storage mode) | | `S3_USE_SSL` | `false` | Use HTTPS to the object store | | `S3_REGION` | — | Region; needed for AWS S3 | -| `S3_CRATE_PREFIX` | `crates` | Key prefix crates are read from | +| `S3_CRATE_PREFIX` | `crates` | Key prefix RO-Crates are read from | | `S3_RESULTS_PREFIX` | `validation-results` | Key prefix results are written to | | `CELERY_BROKER_URL` | — | Redis broker URL (required in storage mode; preset in the Compose stack) | | `CELERY_RESULT_BACKEND` | — | Celery result backend URL (required in storage mode; preset in the Compose stack) | | `PROFILES_PATH` | — | Profiles directory that replaces the bundled profiles | | `EXTRA_PROFILES_PATH` | — | Profiles directory added to the bundled profiles | -| `CACHE_PATH` | `/app/.rocrate-cache` in the published image | Validator HTTP cache location | +| `CACHE_PATH` | `/app/.rocrate-cache` | Validator HTTP cache location | | `VALIDATION_OFFLINE` | `false` | Validate using only the cache, with no network access | | `FLASK_ENV` | `development` | Set to `production` to disable debug behaviour | diff --git a/docs/upgrading.md b/docs/upgrading.md new file mode 100644 index 0000000..cea302f --- /dev/null +++ b/docs/upgrading.md @@ -0,0 +1,57 @@ +# Upgrading from 1.x + +The RO-Crate Validation Service 2.0 release replaced the MinIO-specific storage layer with a general S3-compatible one. + +!!! note + If you only use metadata validation (`POST /v1/ro_crates/validate_metadata`), nothing changes and the endpoint, request fields and responses are the same as in 1.\*. The rest of this page concerns storage-backed validation. + +## Server settings + +The old `MINIO_*` variables are replaced by equivalent `S3_*` variables, and storage is now switched on explicitly: + +| 1.\* | Now | +|------|-----| +| `MINIO_ENDPOINT` | `S3_ENDPOINT` | +| `MINIO_ROOT_USER` | `S3_ACCESS_KEY` | +| `MINIO_ROOT_PASSWORD` | `S3_SECRET_KEY` | +| `MINIO_BUCKET_NAME` | `S3_BUCKET` | +| `ssl` field in each request | `S3_USE_SSL` | +| — | `STORAGE_ENABLED` (new; must be `true` for the stored-crate endpoints to exist) | +| — | `S3_CRATE_PREFIX`, `S3_RESULTS_PREFIX` (new; default `crates` and `validation-results`) | +| `FLASK_APP=cratey.py` | `FLASK_APP=wsgi.py` | + +The published image is now `ghcr.io/esciencelab/ro-crate-validation-service` (with a `-fivesafes-profile` variant that has the Five Safes profile included). The [configuration reference](installation.md#configuration-reference) lists all the settings. + +## Keeping your existing MinIO + +You do not need to change object store as MinIO is S3-compatible. Set `S3_ENDPOINT` to your existing MinIO endpoint, `S3_ACCESS_KEY` and `S3_SECRET_KEY` to your MinIO credentials, and `S3_BUCKET` to your bucket. + +## API changes + +The service connects to the object store defined in the server-side configuration. Requests carry only the crate ID and validation options. The body of `POST /v1/ro_crates/{crate_id}/validation` contains the optional `profile_name` and `webhook_url`. `GET /v1/ro_crates/{crate_id}/validation` takes no body. + +!!! warning + Ensure that you update any existing request bodies before sending requests to the new RO-Crate Validation Service API. Incorrect request bodies will receive `422` validation errors. + +### Crate IDs + +A crate ID is the short label that addresses an RO-Crate in the API path, for example `my-dataset-2026` in `POST /v1/ro_crates/my-dataset-2026/validation`. It is chosen by whoever uploads the RO-Crate, and the service composes the object keys from it: `{S3_CRATE_PREFIX}/.zip` for a zipped RO-Crate, or `{S3_CRATE_PREFIX}//` for a directory. The ID itself is not a filename, path or URL. + +Crate IDs are now validated strictly: they must match `^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$`. This does not allow slashes or path segments. Paths inside the bucket are handled by the prefix settings. + +Response codes are more specific than the 1.\* `400`/`500` pattern: + +| Situation | 1.\* | Now | +|-----------|------|-----| +| Crate not found in the store | `400` | `404` | +| No validation result stored yet | `400` | `404` | +| Invalid crate ID | — | `400` | +| Both zip and directory exist for one ID | — | `409` | +| Request body invalid (e.g., contains removed 1.\* fields) | — | `422` | +| Object store unreachable | `500` | `503` | + +Validation results are saved to `{S3_RESULTS_PREFIX}/{id}.json` (by default `validation-results/.json`) instead of `{crate_id}_validation/validation_status.txt`. The [API reference](api.md) documents the current endpoints in full. + +## Existing RO-Crates and results + +The service now finds an RO-Crate at a fixed key rather than by prefix search: a zipped RO-Crate must be at `{S3_CRATE_PREFIX}/.zip` and a directory RO-Crate under `{S3_CRATE_PREFIX}//`, so an existing RO-Crate may need moving into the RO-Crate prefix. Results stored by 1.\* are not read by the new service, so you will need to re-validate an RO-Crate whose result you still need. diff --git a/mkdocs.yml b/mkdocs.yml index a1ebf4d..e14d2fb 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -2,8 +2,9 @@ site_name: Crate Validation Service nav: - Home: index.md - Installation: installation.md + - Upgrading: upgrading.md - API: api.md - - ENTRUST: entrust.md + - Five Safes RO-Crate: five-safes.md - Contributions: contribution.md theme: readthedocs plugins: From d81a75d1cd7cf3fa3e9066440832fdc15ffec24a Mon Sep 17 00:00:00 2001 From: Alex Hambley <33315205+alexhambley@users.noreply.github.com> Date: Fri, 24 Jul 2026 12:14:53 +0100 Subject: [PATCH 12/19] chore(docs): API docs --- docs/api.md | 154 +++++++++++++++++++++++----------------------- docs/upgrading.md | 8 +-- mkdocs.yml | 2 +- 3 files changed, 82 insertions(+), 82 deletions(-) diff --git a/docs/api.md b/docs/api.md index 7c7c8a0..c03add7 100644 --- a/docs/api.md +++ b/docs/api.md @@ -1,116 +1,116 @@ -# Crate Validator REST API +# API Reference -## Overview +The examples below use the Compose stack's local address, `http://localhost:5001`. Note that the service serves its own OpenAPI specification at `/docs`. -The Crate Validator REST API exposes operations for validating stored -RO-Crates, validating submitted RO-Crate metadata, and retrieving saved -validation results. +!!! note + `POST /v1/ro_crates/validate_metadata` is always available, but the **storage-backed endpoints** are only available when the service runs with `STORAGE_ENABLED=true` (see [Installation & Setup](installation.md#enabling-object-storage)); without this set, requests return `404`. -Stored RO-Crate validation uses MinIO-compatible object storage. Requests that -operate on stored crates include `minio_config`, which tells the service where -the crate is stored and which bucket to use. +## Validate metadata +`POST /v1/ro_crates/validate_metadata` -## Endpoints +This validates the contents of an `ro-crate-metadata.json` document and returns the result in the response. -### `POST /v1/ro_crates/{crate_id}/validation` +| Field | Required | Description | +|-------|----------|-------------| +| `crate_json` | yes | The metadata document, as a JSON string | +| `profile_name` | no | Profile to validate against, e.g. `ro-crate-1.2`; defaults to `ro-crate-1.1` when omitted | -Queues validation for an RO-Crate stored in MinIO-compatible object storage. -The `crate_id` path parameter is the name used to find the crate object in the -configured bucket. +!!! warning + Currently, the validation profile is not detected from the RO-Crate. In other words, a `conformsTo` declaration in the metadata does not influence which validation profile is used by the validator, and the validation always runs against `profile_name`, or `ro-crate-1.1` when it is omitted. -Request body: +To validate a file: -```jsonc -{ - "minio_config": { - "endpoint": "string", // required, e.g. "localhost:9000" or "minio:9000" - "accesskey": "string", // required, MinIO access key or username - "secret": "string", // required, MinIO secret key or password - "ssl": false, // required, true when the MinIO endpoint uses HTTPS - "bucket": "string" // required, bucket containing the RO-Crate - }, - "root_path": "string", // optional folder/path inside the bucket - "profile_name": "string" // optional validation profile name -} +```bash +jq -Rs '{crate_json: .}' ro-crate-metadata.json | curl -X POST http://localhost:5001/v1/ro_crates/validate_metadata -H 'Content-Type: application/json' -d @- ``` -Expected responses: +To choose a profile, add it to the `jq` object: `jq -Rs '{crate_json: ., profile_name: "ro-crate-1.2"}' ro-crate-metadata.json`. -- `202`: validation queued. -- `400`: RO-Crate does not exist or validation request cannot be satisfied. -- `500`: internal service, MinIO, Celery, or validation error. +| Code | Meaning | +|------|---------| +| `200` | Validated; the result has a `status` of `valid` or `invalid` | +| `422` | `crate_json` is either missing, empty or invalid, or the validation could not run (an `error` result) | +## Validate a stored RO-Crate -### `GET /v1/ro_crates/{crate_id}/validation` +`POST /v1/ro_crates/{crate_id}/validation` -Fetches the latest validation result from MinIO-compatible object storage. +This queues validation of an RO-Crate held in the object store. The RO-Crate is resolved first, so a missing or ambiguous crate ID may be reported; the validation itself runs on a worker thread. -Request body: +!!! note + See [Crate IDs](#crate-ids) for how `{crate_id}` maps to objects in the bucket. -```jsonc -{ - "minio_config": { - "endpoint": "string", // required, e.g. "localhost:9000" or "minio:9000" - "accesskey": "string", // required, MinIO access key or username - "secret": "string", // required, MinIO secret key or password - "ssl": false, // required, true when the MinIO endpoint uses HTTPS - "bucket": "string" // required, bucket containing the RO-Crate - }, - "root_path": "string" // optional folder/path inside the bucket -} +| Field | Required | Description | +|-------|----------|-------------| +| `profile_name` | no | Profile to validate against; defaults to `ro-crate-1.1` when omitted | +| `webhook_url` | no | URL that receives the result when validation finishes | + +```bash +curl -X POST http://localhost:5001/v1/ro_crates/my-dataset-2026/validation -H 'Content-Type: application/json' -d '{"profile_name": "ro-crate-1.2"}' ``` -Expected responses: +| Code | Meaning | +|------|---------| +| `202` | Queued; the body is `{"message": "Validation in progress"}` | +| `400` | Invalid Crate ID | +| `404` | No RO-Crate at the expected keys, or storage mode is not enabled | +| `409` | Both a zip and a directory exist for this Crate ID | +| `422` | Request body invalid | +| `503` | Object store unreachable | + +## Fetch a validation result -- `200`: validation result JSON returned. -- `400`: RO-Crate or validation result is missing. -- `500`: MinIO or internal retrieval error. +`GET /v1/ro_crates/{crate_id}/validation` +This returns the stored result for an RO-Crate. -### `POST /v1/ro_crates/validate_metadata` +```bash +curl http://localhost:5001/v1/ro_crates/my-dataset-2026/validation +``` -Validates a submitted RO-Crate metadata JSON string. +| Code | Meaning | +|------|---------| +| `200` | The stored result, including persisted `error` results | +| `400` | Invalid Crate ID | +| `404` | No result stored for this Crate ID yet | -Request body: +## Validation results -```jsonc +Every validation produces a result object: + +```json { - "crate_json": "string", // required, stringified content of ro-crate-metadata.json - "profile_name": "string" // optional validation profile name + "status": "invalid", + "profile": "ro-crate-1.2", + "created_at": "2026-07-22T10:30:00+00:00", + "detail": {} } ``` -Expected responses: - -- `200`: validation result returned. -- `422`: missing, malformed, or empty metadata JSON. -- `500`: internal validation error. - +An RO-Crate's `status` can be: -## Validation Profiles +| `status` | Meaning | +|----------|---------| +| `valid` | The RO-Crate conforms to the profile | +| `invalid` | Validated, but with conformance issues listed in `detail` | +| `error` | The validation could not run; the reason is in an `error` field instead of `detail` | -The optional `profile_name` field selects a specific RO-Crate validation -profile. If omitted, the service uses the validator default. +!!! note + `detail` contains the complete validation report. `created_at` is the UTC time of the validation, and `profile` is the requested profile name, or `null` when the default (`ro-crate-1.1`) was used. -Custom profile definitions can be made available to the service through the -`PROFILES_PATH` environment variable. +For stored RO-Crates the same object is saved to `{S3_RESULTS_PREFIX}/.json` and returned by the GET endpoint. +## Webhooks -## Result Storage +If `webhook_url` was given, the worker POSTs the result object to it as JSON once validation finishes. The result is saved to the store first and the webhook sent after, so a notification is never sent for a result that was not stored. -Validation results for stored RO-Crates are saved back to MinIO-compatible -object storage. +Note that delivery is attempted three times, waiting `0.5s` then `1s` between attempts, with a `10s` timeout per attempt. -Without `root_path`, results are stored at: +## Crate IDs -```text -{crate_id}_validation/validation_status.txt -``` - -With `root_path`, results are stored at: +A Crate ID is the label in the URL path that identifies an RO-Crate in the object store: the service looks for `{S3_CRATE_PREFIX}/.zip` (zip) or `{S3_CRATE_PREFIX}//` (directory). Crate IDs must match `^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$`: they start with a letter or digit, may contain letters, digits, `.`, `_` and `-`, and are at most 128 characters long. Anything else is rejected with `400`. -```text -{root_path}/{crate_id}_validation/validation_status.txt -``` +## Health +`GET /healthz` reports that the process is up, and always returns `200 {"status": "ok"}`. `GET /readyz` checks the object store and Celery broker, returning `200` when ready and `503` otherwise, with the individual checks in the body. When storage is off, both checks report `disabled`. diff --git a/docs/upgrading.md b/docs/upgrading.md index cea302f..cdbeb45 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -28,14 +28,14 @@ You do not need to change object store as MinIO is S3-compatible. Set `S3_ENDPOI ## API changes -The service connects to the object store defined in the server-side configuration. Requests carry only the crate ID and validation options. The body of `POST /v1/ro_crates/{crate_id}/validation` contains the optional `profile_name` and `webhook_url`. `GET /v1/ro_crates/{crate_id}/validation` takes no body. +The service connects to the object store defined in the server-side configuration. Requests carry only the Crate ID and validation options. The body of `POST /v1/ro_crates/{crate_id}/validation` contains the optional `profile_name` and `webhook_url`. `GET /v1/ro_crates/{crate_id}/validation` takes no body. !!! warning Ensure that you update any existing request bodies before sending requests to the new RO-Crate Validation Service API. Incorrect request bodies will receive `422` validation errors. ### Crate IDs -A crate ID is the short label that addresses an RO-Crate in the API path, for example `my-dataset-2026` in `POST /v1/ro_crates/my-dataset-2026/validation`. It is chosen by whoever uploads the RO-Crate, and the service composes the object keys from it: `{S3_CRATE_PREFIX}/.zip` for a zipped RO-Crate, or `{S3_CRATE_PREFIX}//` for a directory. The ID itself is not a filename, path or URL. +A Crate ID is the short label that addresses an RO-Crate in the API path, for example `my-dataset-2026` in `POST /v1/ro_crates/my-dataset-2026/validation`. It is chosen by whoever uploads the RO-Crate, and the service composes the object keys from it: `{S3_CRATE_PREFIX}/.zip` for a zipped RO-Crate, or `{S3_CRATE_PREFIX}//` for a directory. The Crate ID itself is not a filename, path or URL. Crate IDs are now validated strictly: they must match `^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$`. This does not allow slashes or path segments. Paths inside the bucket are handled by the prefix settings. @@ -45,8 +45,8 @@ Response codes are more specific than the 1.\* `400`/`500` pattern: |-----------|------|-----| | Crate not found in the store | `400` | `404` | | No validation result stored yet | `400` | `404` | -| Invalid crate ID | — | `400` | -| Both zip and directory exist for one ID | — | `409` | +| Invalid Crate ID | — | `400` | +| Both zip and directory exist for one Crate ID | — | `409` | | Request body invalid (e.g., contains removed 1.\* fields) | — | `422` | | Object store unreachable | `500` | `503` | diff --git a/mkdocs.yml b/mkdocs.yml index e14d2fb..1fae747 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,4 +1,4 @@ -site_name: Crate Validation Service +site_name: RO-Crate Validation Service nav: - Home: index.md - Installation: installation.md From ad847b2d33a608340c6c96d7085292f1144cf18c Mon Sep 17 00:00:00 2001 From: Alex Hambley <33315205+alexhambley@users.noreply.github.com> Date: Mon, 27 Jul 2026 09:56:58 +0100 Subject: [PATCH 13/19] chore(docs): Five Safes RO-Crate --- docs/api.md | 4 +- docs/entrust.md | 1 - docs/five-safes.md | 98 ++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 100 insertions(+), 3 deletions(-) delete mode 100644 docs/entrust.md create mode 100644 docs/five-safes.md diff --git a/docs/api.md b/docs/api.md index c03add7..23ddf61 100644 --- a/docs/api.md +++ b/docs/api.md @@ -36,7 +36,7 @@ To choose a profile, add it to the `jq` object: `jq -Rs '{crate_json: ., profile `POST /v1/ro_crates/{crate_id}/validation` -This queues validation of an RO-Crate held in the object store. The RO-Crate is resolved first, so a missing or ambiguous crate ID may be reported; the validation itself runs on a worker thread. +This queues validation of an RO-Crate held in the object store. The RO-Crate is resolved first, so a missing or ambiguous crate ID may be reported; the validation itself runs on a worker. !!! note See [Crate IDs](#crate-ids) for how `{crate_id}` maps to objects in the bucket. @@ -97,7 +97,7 @@ An RO-Crate's `status` can be: | `error` | The validation could not run; the reason is in an `error` field instead of `detail` | !!! note - `detail` contains the complete validation report. `created_at` is the UTC time of the validation, and `profile` is the requested profile name, or `null` when the default (`ro-crate-1.1`) was used. + `detail` contains the complete validation report. `created_at` is the UTC time of a stored-crate validation, and `null` for metadata-only validation, which does not set it. `profile` is the requested profile name, or `null` when the default (`ro-crate-1.1`) was used. For stored RO-Crates the same object is saved to `{S3_RESULTS_PREFIX}/.json` and returned by the GET endpoint. diff --git a/docs/entrust.md b/docs/entrust.md deleted file mode 100644 index 489edda..0000000 --- a/docs/entrust.md +++ /dev/null @@ -1 +0,0 @@ -# ENTRUST \ No newline at end of file diff --git a/docs/five-safes.md b/docs/five-safes.md new file mode 100644 index 0000000..e998c96 --- /dev/null +++ b/docs/five-safes.md @@ -0,0 +1,98 @@ +# Five Safes RO-Crate validation + +The [Five Safes RO-Crate 0.4 profile](https://trefx.uk/5s-crate/) describes an RO-Crate used to request and record workflow runs on sensitive data in Trusted Research Environments (TREs), supporting the Five Safes framework. The RO-Crate Validation Service validates against this profile when `profile_name` is set to `five-safes-crate`. + +!!! warning + Note that the Five Safes RO-Crate 0.4 profile is not bundled with the base validator, so the service needs the profile to be made available. There are two ways to do this, described below. + +## Getting a service with the profile + +The prebuilt `ghcr.io/esciencelab/ro-crate-validation-service-fivesafes-profile` image packages the `five-safes-crate` profile with the standard RO-Crate Validation Service. The image also carries a pre-warmed validation cache, so it supports offline validation (`VALIDATION_OFFLINE=true`) inside restricted networks. The profile version is fixed when the image is built, and is recorded in the image label `org.ro-crate-validation-service.five-safes-profile-version`. + +Alternatively, you may run the standard service image with the profile directory mounted and `EXTRA_PROFILES_PATH` set, [as described in custom profiles](installation.md#custom-profiles). + +The `five-safes-crate` profile itself is defined in the [eScienceLab rocrate-validator fork](https://github.com/eScienceLab/rocrate-validator). + +## Validating a Five Safes RO-Crate + +A [complete example crate](https://github.com/eScienceLab/rocrate-validator/blob/five-safes-0.7.4-beta/tests/data/crates/valid/five-safes-crate-result/ro-crate-metadata.json) is available in the fork's test data. + +!!! note + The current `-fivesafes-profile` image pairs the profile with a base profile for RO-Crate 1.1, whilst the profile itself expects RO-Crate 1.2. The walkthrough below mounts the matched profile set instead. + +For this walkthrough, run the service from a checkout of this repository, with the repository's profile set mounted in place of the bundled profiles: + +```bash +docker run --rm -p 5001:5000 \ + -e FLASK_APP=wsgi.py \ + -e PROFILES_PATH=/app/profiles \ + -v "$PWD/tests/data/rocrate_validator_profiles:/app/profiles:ro" \ + ghcr.io/esciencelab/ro-crate-validation-service-fivesafes-profile:latest +``` + +Download the example `ro-crate-metadata.json`: + +```bash +curl -sO https://raw.githubusercontent.com/eScienceLab/rocrate-validator/five-safes-0.7.4-beta/tests/data/crates/valid/five-safes-crate-result/ro-crate-metadata.json +``` + +and validate it with `profile_name` set to `five-safes-crate`: + +```bash +jq -Rs '{crate_json: ., profile_name: "five-safes-crate"}' ro-crate-metadata.json | curl -X POST http://localhost:5001/v1/ro_crates/validate_metadata -H 'Content-Type: application/json' -d @- +``` + +The crate conforms, so the response (abridged) is: + +```json +{ + "status": "valid", + "profile": "five-safes-crate", + "created_at": null, + "detail": { + "issues": [], + "passed": true + } +} +``` + +To see how conformance issues are reported, remove something the profile requires, such as the `CreateAction` recording the requested workflow run. Validate again: + +```bash +jq '."@graph" |= map(select(."@type" != "CreateAction"))' ro-crate-metadata.json > broken.json +``` + +```bash +jq -Rs '{crate_json: ., profile_name: "five-safes-crate"}' broken.json | curl -X POST http://localhost:5001/v1/ro_crates/validate_metadata -H 'Content-Type: application/json' -d @- +``` + +The result will now show `invalid`, and each entry in `detail.issues` identifies the failed check, its severity, and the entity at fault (abridged): + +```json +{ + "status": "invalid", + "profile": "five-safes-crate", + "detail": { + "issues": [ + { + "check": { + "identifier": "five-safes-crate-0.4_25.1", + "name": "mentions" + }, + "severity": "REQUIRED", + "message": "`RootDataEntity` MUST reference at least one `CreateAction` through `mentions`", + "violatingEntity": "./" + } + ], + "passed": false + } +} +``` + +Complete RO-Crates work the same way through the storage-backed endpoints: upload the crate as `crates/.zip` (or a directory under `crates//`), then queue validation with the profile: + +```bash +curl -X POST http://localhost:5001/v1/ro_crates/my-5s-crate/validation -H 'Content-Type: application/json' -d '{"profile_name": "five-safes-crate"}' +``` + +The [API reference](api.md) covers the endpoints, results and webhooks in full. From 55f45d2d15952675011e0391d403b6925edb2e53 Mon Sep 17 00:00:00 2001 From: Douglas Lowe <10961945+douglowe@users.noreply.github.com> Date: Mon, 27 Jul 2026 10:42:16 +0100 Subject: [PATCH 14/19] contribution guide, focused on open source procedures --- docs/contribution.md | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/docs/contribution.md b/docs/contribution.md index d292e89..46863b1 100644 --- a/docs/contribution.md +++ b/docs/contribution.md @@ -1 +1,25 @@ -# Development / Contributions \ No newline at end of file +# Development / Contributions + +Both the [RO-Crate Validation Service](https://github.com/eScienceLab/RO-Crate-Validation-Service) and the [roc-validator tool](https://github.com/crs4/rocrate-validator) are open source projects and welcome contributions of all kinds: bug reports, code or documentation changes, and reviews of proposed changes. + +## How to Contribute + +The easiest way to start contributing is to create an issue on the relevant GitHub repository to either let us know of a bug or error, or to propose a piece of work you want to do. For the RO-Crate validation tool (the underlying tool which generates the validation reports) use the [roc-validator Issues](https://github.com/crs4/rocrate-validator/issues) page. For RO-Crate validation service (the docker container, API service, and docker compose service stack) use the [RO-Crate Validation Service Issues](https://github.com/eScienceLab/RO-Crate-Validation-Service/issues) page. + +### Code Contributions + +If you want to contribute some code changes via GitHub then you may want to read [How to Contribute to an Open Source Project on GitHub](https://egghead.io/courses/how-to-contribute-to-an-open-source-project-on-github). We use [GitHub flow](https://docs.github.com/en/get-started/using-github/github-flow) to manage changes: + +1. Create a new branch in your desktop copy of this repository for each + significant change. +2. Commit the change in that branch. +3. Push that branch to your fork of this repository on GitHub. +4. Submit a pull request from that branch to the [upstream repository][repo]. +5. If you receive feedback, make changes on your desktop and push to your + branch on GitHub: the pull request will update automatically. + +Note that we use the `develop` branch for development work, and this is where your PR should be aimed. The `main` branch is used for releases, and only pull requests from the `develop` branch are accepted to this. + +## Service Stack Architecture + +## Development Setup \ No newline at end of file From 57c7b41f102c58fad9b1670167d89ecca6f16e52 Mon Sep 17 00:00:00 2001 From: Alex Hambley <33315205+alexhambley@users.noreply.github.com> Date: Mon, 27 Jul 2026 11:18:58 +0100 Subject: [PATCH 15/19] chore(docs): development and contributions guide --- docs/contribution.md | 124 ++++++++++++++++++++++++++++++++++++++----- 1 file changed, 110 insertions(+), 14 deletions(-) diff --git a/docs/contribution.md b/docs/contribution.md index 46863b1..be9b376 100644 --- a/docs/contribution.md +++ b/docs/contribution.md @@ -1,25 +1,121 @@ -# Development / Contributions +# Development and Contributions -Both the [RO-Crate Validation Service](https://github.com/eScienceLab/RO-Crate-Validation-Service) and the [roc-validator tool](https://github.com/crs4/rocrate-validator) are open source projects and welcome contributions of all kinds: bug reports, code or documentation changes, and reviews of proposed changes. +The [RO-Crate Validation Service](https://github.com/eScienceLab/RO-Crate-Validation-Service) is an open source project and welcomes contributions of all kinds: bug reports, code or documentation changes, and reviews of proposed changes. The underlying [rocrate-validator tool](https://github.com/crs4/rocrate-validator) is also open source, and is a separate project maintained by CRS4. -## How to Contribute +This service is written with Python 3.11, built on Flask/APIFlask and Celery, and wraps the [`rocrate-validator`](https://rocrate-validator.readthedocs.io/) in a REST API. The RO-Crate Validation Service enables pipelines, other services, and Trusted Research Environments (TREs) to validate an RO-Crate over HTTP without running the validator themselves. -The easiest way to start contributing is to create an issue on the relevant GitHub repository to either let us know of a bug or error, or to propose a piece of work you want to do. For the RO-Crate validation tool (the underlying tool which generates the validation reports) use the [roc-validator Issues](https://github.com/crs4/rocrate-validator/issues) page. For RO-Crate validation service (the docker container, API service, and docker compose service stack) use the [RO-Crate Validation Service Issues](https://github.com/eScienceLab/RO-Crate-Validation-Service/issues) page. +## Contributing -### Code Contributions +The easiest way to start contributing is to create an issue, either to let us know of a bug or error, or to propose a piece of work you want to do. For the RO-Crate Validation Service (the API service, Docker image, and Compose stack) use the [RO-Crate Validation Service issues](https://github.com/eScienceLab/RO-Crate-Validation-Service/issues) page. Issues with the validation checks themselves belong to the underlying tool rather than this service: report those on the [rocrate-validator issues](https://github.com/crs4/rocrate-validator/issues) page, and follow that project's own contribution guidance. -If you want to contribute some code changes via GitHub then you may want to read [How to Contribute to an Open Source Project on GitHub](https://egghead.io/courses/how-to-contribute-to-an-open-source-project-on-github). We use [GitHub flow](https://docs.github.com/en/get-started/using-github/github-flow) to manage changes: +### Code contributions -1. Create a new branch in your desktop copy of this repository for each - significant change. +If you want to contribute code changes via GitHub then you may want to read ['How to Contribute to an Open Source Project on GitHub'](https://egghead.io/courses/how-to-contribute-to-an-open-source-project-on-github). We use [GitHub flow](https://docs.github.com/en/get-started/using-github/github-flow) to manage changes: + +1. Create a new branch in your local clone of this repository for each significant change. 2. Commit the change in that branch. 3. Push that branch to your fork of this repository on GitHub. -4. Submit a pull request from that branch to the [upstream repository][repo]. -5. If you receive feedback, make changes on your desktop and push to your - branch on GitHub: the pull request will update automatically. +4. Submit a pull request from that branch to the [upstream repository](https://github.com/eScienceLab/RO-Crate-Validation-Service). +5. If you receive feedback, make the changes in your local clone and push to your branch on GitHub: the pull request will update automatically. + +!!! warning + Note that we use the `develop` branch for development work, and this is where your PR should be aimed. The `main` branch is used for releases, and only pull requests from the `develop` branch are accepted to this. + +## Development stack + +The development Compose file builds the image from the local `Dockerfile` and mounts the repository's test profiles into both the API and worker containers: + +```bash +docker compose -f docker-compose-develop.yml up --build +``` + +Here `--build` matters: without it, Compose reuses the previously built image and local code changes are not picked up. Add `--profile objectstore` to start the bundled RustFS store for storage-backed work; configuration is the same as in [Installation & Setup](installation.md#configuration-reference). + +## Tests + +Install the development dependencies, then run the unit tests, which need no Docker: + +```bash +pip install -r requirements-dev.txt +``` + +```bash +pytest --ignore=tests/test_integration.py +``` + +The integration tests bring up the full Compose stack (including the object store) and seed crates with `boto3`, so they need Docker running: + +```bash +pytest tests/test_integration.py +``` + +`tests/` mirrors the layout of the `app/` package, so the tests for a module are in the matching directory. + +## Linting + +The project uses [Ruff](https://docs.astral.sh/ruff/) for linting and formatting, configured in `pyproject.toml`: + +```bash +ruff check . && ruff format --check . +``` + +## Dependencies + +Direct dependencies are declared in `pyproject.toml`; the `requirements*.txt` files are locks generated with pip-compile: + +```bash +pip-compile pyproject.toml -o requirements.txt +``` + +```bash +pip-compile --extra dev pyproject.toml -o requirements-dev.txt +``` + +## Continuous Integration + +Pull requests to `develop` run three workflows: unit tests, integration tests (which start the Compose stack), and lint (`ruff check` and `ruff format --check`). + +## How the API works + +The API process handles HTTP and runs metadata-only validation inline. Object storage-backed validation is queued through Redis to a Celery worker, which reads the crate from the S3-compatible store, validates it, and writes the result back: + +```mermaid +flowchart LR + Client([Client]) + API["Flask API"] + Broker[("Redis")] + Worker["Celery worker"] + Validator["rocrate-validator"] + Store[("S3-compatible store")] + + Client --> API + API -->|metadata-only: inline| Validator + API --> Broker --> Worker --> Validator + Worker <--> Store +``` -Note that we use the `develop` branch for development work, and this is where your PR should be aimed. The `main` branch is used for releases, and only pull requests from the `develop` branch are accepted to this. +The worker runs its stages strictly in order: fetch, validate, persist, webhook; so a storage write failure can never be followed by a success notification, and every outcome (including `error` outcomes) is persisted so a later `GET` reflects what happened. -## Service Stack Architecture +## Project structure -## Development Setup \ No newline at end of file +``` +app/ +├── __init__.py # app factory: config, blueprints, error handlers, request IDs +├── health.py # /healthz and /readyz +├── storage/ # object-storage abstraction +│ ├── base.py # StorageBackend protocol +│ ├── s3.py # boto3 implementation (any S3-compatible store) +│ ├── memory.py # in-memory backend (tests / local) +│ └── errors.py # StorageError, ObjectNotFound +├── crates/ # crate identity, layout, resolution +│ ├── ids.py # Crate ID validation +│ ├── layout.py # object keys +│ └── resolver.py # deterministic zip/directory resolution +├── validation/ # validation boundary +│ ├── results.py # ValidationOutcome (valid/invalid/error) +│ └── runner.py # wraps rocrate-validator +├── ro_crates/routes/ # HTTP endpoints (metadata + ID-based) +├── services/ # request handling and logging +├── tasks/validation_tasks.py # Celery task: fetch, validate, persist, webhook +└── utils/ # validated settings, webhook delivery +``` From c70d5e677555c3c529245d0c45916d9a220a318f Mon Sep 17 00:00:00 2001 From: Alex Hambley <33315205+alexhambley@users.noreply.github.com> Date: Mon, 27 Jul 2026 11:30:42 +0100 Subject: [PATCH 16/19] chore(docs): clarify installation and upgrading docs --- docs/installation.md | 6 +++--- docs/upgrading.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/installation.md b/docs/installation.md index f5be8e2..08a4425 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -12,7 +12,7 @@ To start, clone the repository, copy the example environment file, and start the git clone https://github.com/eScienceLab/RO-Crate-Validation-Service.git cd RO-Crate-Validation-Service cp example.env .env -docker compose up --build +docker compose up ``` !!! warning @@ -49,7 +49,7 @@ The service will fail at startup if any are missing. The Compose stack already s Then start the stack with the bundled development object store (RustFS), run: ```bash -docker compose --profile objectstore up --build +docker compose --profile objectstore up ``` RustFS serves the S3 API on port 9000 and a web console at `http://localhost:9001`. Development credentials are set in `example.env`. @@ -99,7 +99,7 @@ The validator comes with several RO-Crate profiles, and for the Five Safes RO-Cr To add other profiles, mount a directory into both the `flask` and `celery_worker` containers, and set `EXTRA_PROFILES_PATH` to the mounted path. Note that both containers need the mount as metadata-only validation runs in the API process and stored-crate validation runs in the worker. There is a working example in `docker-compose-develop.yml`. !!! note - `EXTRA_PROFILES_PATH` adds the directory to the bundled profiles, whereas `PROFILES_PATH` replaces them entirely. + `EXTRA_PROFILES_PATH` adds the directory to the bundled profiles, whereas `PROFILES_PATH` replaces them entirely. The two can be set together, in which case the validator takes profiles from both locations. ## Offline validation diff --git a/docs/upgrading.md b/docs/upgrading.md index cdbeb45..02b9bf4 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -20,7 +20,7 @@ The old `MINIO_*` variables are replaced by equivalent `S3_*` variables, and sto | — | `S3_CRATE_PREFIX`, `S3_RESULTS_PREFIX` (new; default `crates` and `validation-results`) | | `FLASK_APP=cratey.py` | `FLASK_APP=wsgi.py` | -The published image is now `ghcr.io/esciencelab/ro-crate-validation-service` (with a `-fivesafes-profile` variant that has the Five Safes profile included). The [configuration reference](installation.md#configuration-reference) lists all the settings. +The published image is also renamed: `ghcr.io/esciencelab/cratey-validator` is now `ghcr.io/esciencelab/ro-crate-validation-service` (with a `-fivesafes-profile` variant that has the Five Safes profile included). The [configuration reference](installation.md#configuration-reference) lists all the settings. ## Keeping your existing MinIO From 776c92fdbdd2d9e9e61886e7af2050d5e8b70d78 Mon Sep 17 00:00:00 2001 From: Alex Hambley <33315205+alexhambley@users.noreply.github.com> Date: Mon, 27 Jul 2026 11:40:09 +0100 Subject: [PATCH 17/19] chore: add GitHub Pages docs deployment workflow --- .github/workflows/deploy_docs.yml | 54 +++++++++++++++++++++++++++++++ mkdocs.yml | 1 + 2 files changed, 55 insertions(+) create mode 100644 .github/workflows/deploy_docs.yml diff --git a/.github/workflows/deploy_docs.yml b/.github/workflows/deploy_docs.yml new file mode 100644 index 0000000..7e780a2 --- /dev/null +++ b/.github/workflows/deploy_docs.yml @@ -0,0 +1,54 @@ +name: Deploy Documentation Site + +on: + push: + branches: [ main ] + paths: + - "docs/**" + - "mkdocs.yml" + workflow_dispatch: + +permissions: + contents: read + pages: write + id-token: write + +concurrency: + group: "pages" + cancel-in-progress: false + +jobs: + build: + runs-on: ubuntu-latest + steps: + - name: Checkout code + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 + + - name: Set up Python + uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0 + with: + python-version: '3.11' + + - name: Install MkDocs + run: | + python -m pip install --upgrade pip + pip install mkdocs mkdocs-mermaid2-plugin + + - name: Build site + run: mkdocs build --strict + + - name: Upload Pages artifact + uses: actions/upload-pages-artifact@v3 + with: + path: site + + deploy: + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + runs-on: ubuntu-latest + needs: build + steps: + - name: Deploy to GitHub Pages + id: deployment + uses: actions/deploy-pages@v4 diff --git a/mkdocs.yml b/mkdocs.yml index 1fae747..a1bb182 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -1,4 +1,5 @@ site_name: RO-Crate Validation Service +site_url: https://esciencelab.org.uk/RO-Crate-Validation-Service/ nav: - Home: index.md - Installation: installation.md From a2a7b915a52fa607e9fc159773ac5b365ecafe38 Mon Sep 17 00:00:00 2001 From: Alex Hambley <33315205+alexhambley@users.noreply.github.com> Date: Mon, 27 Jul 2026 12:03:20 +0100 Subject: [PATCH 18/19] ci: pages workflow actions to v5 SHAs --- .github/workflows/deploy_docs.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/deploy_docs.yml b/.github/workflows/deploy_docs.yml index 7e780a2..672de1c 100644 --- a/.github/workflows/deploy_docs.yml +++ b/.github/workflows/deploy_docs.yml @@ -38,7 +38,7 @@ jobs: run: mkdocs build --strict - name: Upload Pages artifact - uses: actions/upload-pages-artifact@v3 + uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0 with: path: site @@ -51,4 +51,4 @@ jobs: steps: - name: Deploy to GitHub Pages id: deployment - uses: actions/deploy-pages@v4 + uses: actions/deploy-pages@cd2ce8fcbc39b97be8ca5fce6e763baed58fa128 # v5.0.0 From 6351b210f8805c26a7f9d8a502a241dd754c144c Mon Sep 17 00:00:00 2001 From: Alex Hambley <33315205+alexhambley@users.noreply.github.com> Date: Tue, 28 Jul 2026 13:01:05 +0100 Subject: [PATCH 19/19] chore(docs): improve docs clarity and wording --- docs/api.md | 22 +++++++++++-------- docs/contribution.md | 10 ++++----- docs/index.md | 6 +++--- docs/installation.md | 50 +++++++++++++++++++++++++------------------- docs/upgrading.md | 6 +++--- 5 files changed, 52 insertions(+), 42 deletions(-) diff --git a/docs/api.md b/docs/api.md index 23ddf61..8b41fb7 100644 --- a/docs/api.md +++ b/docs/api.md @@ -3,7 +3,7 @@ The examples below use the Compose stack's local address, `http://localhost:5001`. Note that the service serves its own OpenAPI specification at `/docs`. !!! note - `POST /v1/ro_crates/validate_metadata` is always available, but the **storage-backed endpoints** are only available when the service runs with `STORAGE_ENABLED=true` (see [Installation & Setup](installation.md#enabling-object-storage)); without this set, requests return `404`. + `POST /v1/ro_crates/validate_metadata` is always available, but the **storage-backed endpoints** are only available when the service runs with `STORAGE_ENABLED=true` (see [Installation & Setup](installation.md#enabling-object-storage)); without this set, `POST /v1/ro_crates/{crate_id}/validation` or `GET /v1/ro_crates/{crate_id}/validation` requests will return a `404` result. ## Validate metadata @@ -14,18 +14,22 @@ This validates the contents of an `ro-crate-metadata.json` document and returns | Field | Required | Description | |-------|----------|-------------| | `crate_json` | yes | The metadata document, as a JSON string | -| `profile_name` | no | Profile to validate against, e.g. `ro-crate-1.2`; defaults to `ro-crate-1.1` when omitted | +| `profile_name` | no | Profile to validate against, e.g. `ro-crate-1.2`. The validator will default to `ro-crate-1.1` when this is omitted | !!! warning Currently, the validation profile is not detected from the RO-Crate. In other words, a `conformsTo` declaration in the metadata does not influence which validation profile is used by the validator, and the validation always runs against `profile_name`, or `ro-crate-1.1` when it is omitted. -To validate a file: +To validate an ro-crate metadata file: ```bash jq -Rs '{crate_json: .}' ro-crate-metadata.json | curl -X POST http://localhost:5001/v1/ro_crates/validate_metadata -H 'Content-Type: application/json' -d @- ``` -To choose a profile, add it to the `jq` object: `jq -Rs '{crate_json: ., profile_name: "ro-crate-1.2"}' ro-crate-metadata.json`. +To choose a profile, add it to the json object as a `profile_name` entry: + +```bash +jq -Rs '{crate_json: ., profile_name: "ro-crate-1.2"}' ro-crate-metadata.json | curl -X POST http://localhost:5001/v1/ro_crates/validate_metadata -H 'Content-Type: application/json' -d @- +``` | Code | Meaning | |------|---------| @@ -36,7 +40,7 @@ To choose a profile, add it to the `jq` object: `jq -Rs '{crate_json: ., profile `POST /v1/ro_crates/{crate_id}/validation` -This queues validation of an RO-Crate held in the object store. The RO-Crate is resolved first, so a missing or ambiguous crate ID may be reported; the validation itself runs on a worker. +This queues validation of an RO-Crate held in the object store. The RO-Crate is resolved first, so a missing or ambiguous crate ID may be reported immediately; the validation process itself runs asynchronously on a worker. !!! note See [Crate IDs](#crate-ids) for how `{crate_id}` maps to objects in the bucket. @@ -54,7 +58,7 @@ curl -X POST http://localhost:5001/v1/ro_crates/my-dataset-2026/validation -H 'C |------|---------| | `202` | Queued; the body is `{"message": "Validation in progress"}` | | `400` | Invalid Crate ID | -| `404` | No RO-Crate at the expected keys, or storage mode is not enabled | +| `404` | Either storage mode is not enabled, or there is no RO-Crate at the location defined by the given Crate ID | | `409` | Both a zip and a directory exist for this Crate ID | | `422` | Request body invalid | | `503` | Object store unreachable | @@ -73,7 +77,7 @@ curl http://localhost:5001/v1/ro_crates/my-dataset-2026/validation |------|---------| | `200` | The stored result, including persisted `error` results | | `400` | Invalid Crate ID | -| `404` | No result stored for this Crate ID yet | +| `404` | No result stored for this Crate ID | ## Validation results @@ -93,8 +97,8 @@ An RO-Crate's `status` can be: | `status` | Meaning | |----------|---------| | `valid` | The RO-Crate conforms to the profile | -| `invalid` | Validated, but with conformance issues listed in `detail` | -| `error` | The validation could not run; the reason is in an `error` field instead of `detail` | +| `invalid` | The RO-Crate does not conform to the profile, issues listed in `detail` field | +| `error` | The validation could not run; the reason is in an `error` field. No `detail` field is provided | !!! note `detail` contains the complete validation report. `created_at` is the UTC time of a stored-crate validation, and `null` for metadata-only validation, which does not set it. `profile` is the requested profile name, or `null` when the default (`ro-crate-1.1`) was used. diff --git a/docs/contribution.md b/docs/contribution.md index be9b376..c1cd5ab 100644 --- a/docs/contribution.md +++ b/docs/contribution.md @@ -33,7 +33,7 @@ Here `--build` matters: without it, Compose reuses the previously built image an ## Tests -Install the development dependencies, then run the unit tests, which need no Docker: +Install the development dependencies, then run the unit tests, which do not use Docker Engine: ```bash pip install -r requirements-dev.txt @@ -43,7 +43,7 @@ pip install -r requirements-dev.txt pytest --ignore=tests/test_integration.py ``` -The integration tests bring up the full Compose stack (including the object store) and seed crates with `boto3`, so they need Docker running: +The integration tests bring up the full Compose stack (including the object store) and seed crates with `boto3`, for which they need Docker Engine to be running: ```bash pytest tests/test_integration.py @@ -61,7 +61,7 @@ ruff check . && ruff format --check . ## Dependencies -Direct dependencies are declared in `pyproject.toml`; the `requirements*.txt` files are locks generated with pip-compile: +Direct dependencies are declared in `pyproject.toml`; while the `requirements*.txt` files are locks generated using `pip-compile`: ```bash pip-compile pyproject.toml -o requirements.txt @@ -73,11 +73,11 @@ pip-compile --extra dev pyproject.toml -o requirements-dev.txt ## Continuous Integration -Pull requests to `develop` run three workflows: unit tests, integration tests (which start the Compose stack), and lint (`ruff check` and `ruff format --check`). +Pull requests to `develop` will trigger three workflows: unit tests, integration tests (which start the Compose stack), and lint (`ruff check` and `ruff format --check`). ## How the API works -The API process handles HTTP and runs metadata-only validation inline. Object storage-backed validation is queued through Redis to a Celery worker, which reads the crate from the S3-compatible store, validates it, and writes the result back: +The API server handles HTTP and runs metadata-only validation inline. Object storage-backed validation is queued through Redis to a Celery worker, which reads the crate from the S3-compatible store, validates it, and writes the result back: ```mermaid flowchart LR diff --git a/docs/index.md b/docs/index.md index 5092a03..a6b635b 100644 --- a/docs/index.md +++ b/docs/index.md @@ -1,12 +1,12 @@ # RO-Crate Validation Service -The RO-Crate Validation Service evaluates whether [RO-Crates](https://www.researchobject.org/ro-crate/) conform to the RO-Crate specification and to community profiles. The service wraps the [`rocrate-validator`](https://rocrate-validator.readthedocs.io/) library in a REST API, and is deployed as a Docker image. The RO-Crate Validation Service enables pipelines, other services, and Trusted Research Environments (TREs) to validate an RO-Crate over HTTP without running the validator themselves. +The RO-Crate Validation Service provides a REST API for evaluating whether an [RO-Crate](https://www.researchobject.org/ro-crate/) conforms to a given RO-Crate profile. The service wraps the [`rocrate-validator`](https://rocrate-validator.readthedocs.io/) library, enabling pipelines, other services, and Trusted Research Environments (TREs) to validate an RO-Crate over HTTP without having to install the validator themselves. ## RO-Crates and profiles in brief -An [RO-Crate](https://www.researchobject.org/ro-crate/) packages research data together with structured, machine-readable metadata: a JSON-LD file named `ro-crate-metadata.json`. Validating an RO-Crate evaluates that metadata against a **profile**. A profile is a set of requirements the RO-Crate must satisfy. That can be the base requirements of the [RO-Crate specification](https://www.researchobject.org/ro-crate/1.2/) itself, or a [community profile](https://www.researchobject.org/ro-crate/profiles) that adds domain-specific rules, such as the [Five Safes RO-Crate profile](https://trefx.uk/5s-crate/) for working with sensitive data in TREs. +An [RO-Crate](https://www.researchobject.org/ro-crate/) packages research data together with structured, machine-readable metadata: a JSON-LD file named `ro-crate-metadata.json`. Validating an RO-Crate involves evaluating that metadata against a given **profile**. A profile is a set of requirements the RO-Crate must satisfy, either the base requirements of the [RO-Crate specification](https://www.researchobject.org/ro-crate/1.2/) itself, or a [community profile](https://www.researchobject.org/ro-crate/profiles) that adds domain-specific rules to the base specification. An example of a community profile is the [Five Safes RO-Crate profile](https://trefx.uk/5s-crate/), designed for researchers working with sensitive data in Trusted Research Environments (TREs). -Whilst the validation checks themselves are performed by [`rocrate-validator`](https://rocrate-validator.readthedocs.io/), this service is a deployable HTTP wrapper around that tool: it adds a web API, asynchronous processing, and object-storage integration. The base validation rules come from the validator itself; we additionally package our own [Five Safes RO-Crate profile](five-safes.md) rules with the service for working in TREs. +Whilst the validation checks themselves are performed by the [`rocrate-validator`](https://rocrate-validator.readthedocs.io/), this service is a deployable HTTP wrapper around that tool: it adds a web API, asynchronous processing, and object-storage integration. The base RO-Crate specification, as well as several [community profiles](https://rocrate-validator.readthedocs.io/en/latest/#features), are provided with the validator itself. We additionally package our own [Five Safes RO-Crate profile](five-safes.md) rules with the service for working in TREs. ## Validation methods diff --git a/docs/installation.md b/docs/installation.md index 08a4425..bcbb0c3 100644 --- a/docs/installation.md +++ b/docs/installation.md @@ -6,7 +6,7 @@ The RO-Crate Validation Service works in two ways: a metadata-only mode, in whic You will need Docker with Docker Compose. -To start, clone the repository, copy the example environment file, and start the stack: +To start, clone the repository, copy the example environment file (`example.env`), and start the stack: ```bash git clone https://github.com/eScienceLab/RO-Crate-Validation-Service.git @@ -28,38 +28,38 @@ curl http://localhost:5001/healthz This returns `{"status": "ok"}`. -To validate the contents of an `ro-crate-metadata.json` file, post to the metadata endpoint. The [running example](https://www.researchobject.org/ro-crate/specification/1.2/introduction.html#running-example) from the RO-Crate specification is a good test document. +To validate the contents of an `ro-crate-metadata.json` file, post to the metadata endpoint. The [running example](https://www.researchobject.org/ro-crate/specification/1.2/introduction.html#running-example) from the RO-Crate specification is a good test document. Create an `ro-crate-metadata.json` file and copy the text from the link above into this, for use in the example below. -The file needs to be sent as an escaped JSON string, which `jq` can do: +The file contents need to be sent to the API as an escaped JSON string, identified using the `crate_json` tag, within a JSON object. The command-line JSON processor, [`jq`](https://jqlang.org/) can be used to do this, as shown below. ```bash jq -Rs '{crate_json: .}' ro-crate-metadata.json | curl -X POST http://localhost:5001/v1/ro_crates/validate_metadata -H 'Content-Type: application/json' -d @- ``` -The response contains a `status` of `valid`, `invalid` or `error`, along with the detailed validation. For more information, the [API reference](api.md) describes the endpoints and result format in full. +The returned response will contain a `status` of `valid`, `invalid` or `error`, along with the detailed validation. For more information, the [API reference](api.md) describes the endpoints and result format in full. ## Enabling object storage -To validate a complete RO-Crate (zip or directory) held in an object store, set `STORAGE_ENABLED=true` in `.env`. +To enable the validation of complete RO-Crates (zip or directory) that are held in an object store, set `STORAGE_ENABLED=true` in `.env`. -The storage-backed validation mode requires six settings, [described below](#configuration-reference): `S3_ENDPOINT`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`, `S3_BUCKET`, `CELERY_BROKER_URL` and `CELERY_RESULT_BACKEND`. +The storage-backed validation mode requires six environmental variables to be set in the `.env` file, [described below](#configuration-reference): `S3_ENDPOINT`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`, `S3_BUCKET`, `CELERY_BROKER_URL` and `CELERY_RESULT_BACKEND`. The service will fail at startup if any are missing. -The service will fail at startup if any are missing. The Compose stack already sets the Celery variables to the bundled Redis, so in practice only the S3 settings in `.env` matter here. +The `CELERY_BROKER_URL` and `CELERY_RESULT_BACKEND` are already configured to the bundled Redis service within the docker compose stack, so the values for these within the `example.env` file can be left as they are. For any purpose other than an initial demonstration of the service the S3 settings given in the `example.env` file, `S3_ENDPOINT`, `S3_ACCESS_KEY`, `S3_SECRET_KEY`, and `S3_BUCKET`, should be changed for both security reasons and to match your own local setup. -Then start the stack with the bundled development object store (RustFS), run: +To start the stack with the bundled development object store (RustFS), run: ```bash docker compose --profile objectstore up ``` -RustFS serves the S3 API on port 9000 and a web console at `http://localhost:9001`. Development credentials are set in `example.env`. +RustFS serves the S3 API on port 9000 and a web console at `http://localhost:9001`. !!! warning - The service does not create the bucket itself. Create the bucket in the console or with an S3 client. The bucket name needs to match `S3_BUCKET` (`ro-crates` by default). + The service does not create the bucket itself. Create the bucket in the console or with an S3 client. The bucket name needs to match the value in the `S3_BUCKET` environmental variable (`ro-crates` by default). -Upload an RO-Crate under the prefix: `crates/.zip` for a zipped RO-Crate, or `crates//` for a directory. Note that for a zipped RO-Crate, `ro-crate-metadata.json` must be at the root of the archive. +Upload an RO-Crate to this S3 bucket, using the prefix `crates`. This should give the uploaded RO-Crate a path of either `crates/.zip` for a zipped RO-Crate, or `crates//` for a directory. Note that for a zipped RO-Crate, `ro-crate-metadata.json` must be at the root of the archive. -The readiness endpoint checks the object store and broker connections: +The readiness endpoint can be used to check the object store and broker connections: ```bash curl http://localhost:5001/readyz @@ -67,12 +67,18 @@ curl http://localhost:5001/readyz ## Using your own object store -Any S3-compatible store can be used in place of RustFS, including AWS S3, MinIO and Ceph: set `S3_ENDPOINT`, the credentials and `S3_BUCKET` for your store. In this case, do not run the `objectstore` profile. +Any S3-compatible store can be used in place of RustFS, including AWS S3, MinIO and Ceph. To do this set the `S3_ENDPOINT` and `S3_BUCKET` environment variables to match the location of your store, and provide your store credentials in the `S3_ACCESS_KEY` and `S3_SECRET_KEY` environment variables. Make sure that the `STORAGE_ENABLED` environment variable is still set to `true`, but do not include the `objectstore` profile flag in your docker compose command: + +```bash +docker compose up +``` If you already use a 1.x release against MinIO, the [upgrade guide](upgrading.md) maps the old settings to the new ones. ## Configuration reference +The following can all be set as environment variables for the service using an `.env` file. + | Variable | Default | Description | |----------|---------|-------------| | `STORAGE_ENABLED` | `false` | Enable the stored-crate endpoints and storage checks | @@ -80,14 +86,14 @@ If you already use a 1.x release against MinIO, the [upgrade guide](upgrading.md | `S3_ACCESS_KEY` | — | Object store access key (required in storage mode) | | `S3_SECRET_KEY` | — | Object store secret key (required in storage mode) | | `S3_BUCKET` | — | Bucket holding RO-Crates and results (required in storage mode) | -| `S3_USE_SSL` | `false` | Use HTTPS to the object store | -| `S3_REGION` | — | Region; needed for AWS S3 | -| `S3_CRATE_PREFIX` | `crates` | Key prefix RO-Crates are read from | -| `S3_RESULTS_PREFIX` | `validation-results` | Key prefix results are written to | +| `S3_USE_SSL` | `false` | Use HTTPS for connecting to the object store | +| `S3_REGION` | — | Region; needed when using AWS S3 object stores | +| `S3_CRATE_PREFIX` | `crates` | Prefix key from which RO-Crates are read | +| `S3_RESULTS_PREFIX` | `validation-results` | Prefix key to which results are written | | `CELERY_BROKER_URL` | — | Redis broker URL (required in storage mode; preset in the Compose stack) | | `CELERY_RESULT_BACKEND` | — | Celery result backend URL (required in storage mode; preset in the Compose stack) | -| `PROFILES_PATH` | — | Profiles directory that replaces the bundled profiles | -| `EXTRA_PROFILES_PATH` | — | Profiles directory added to the bundled profiles | +| `PROFILES_PATH` | — | Profiles directory for replacing the bundled profiles | +| `EXTRA_PROFILES_PATH` | — | Profiles directory for adding extra profiles | | `CACHE_PATH` | `/app/.rocrate-cache` | Validator HTTP cache location | | `VALIDATION_OFFLINE` | `false` | Validate using only the cache, with no network access | | `FLASK_ENV` | `development` | Set to `production` to disable debug behaviour | @@ -96,13 +102,13 @@ If you already use a 1.x release against MinIO, the [upgrade guide](upgrading.md The validator comes with several RO-Crate profiles, and for the Five Safes RO-Crate, the prebuilt `ghcr.io/esciencelab/ro-crate-validation-service-fivesafes-profile` image has the `five-safes-crate` profile already included; see [Five Safes validation](five-safes.md). -To add other profiles, mount a directory into both the `flask` and `celery_worker` containers, and set `EXTRA_PROFILES_PATH` to the mounted path. Note that both containers need the mount as metadata-only validation runs in the API process and stored-crate validation runs in the worker. There is a working example in `docker-compose-develop.yml`. +Other profiles can be provided by mounting the directory containing these profiles as a volume for the `flask` container. Mount the same directory as a volume for the `celery_worker` container as well if you have enabled stored-crate validation. Then set either the `EXTRA_PROFILES_PATH` or `PROFILES_PATH` environment variable to match the volume path. There is a working example in `docker-compose-develop.yml`. !!! note `EXTRA_PROFILES_PATH` adds the directory to the bundled profiles, whereas `PROFILES_PATH` replaces them entirely. The two can be set together, in which case the validator takes profiles from both locations. ## Offline validation -The validator fetches profile and context resources over HTTP and caches them. The published images pre-populate this cache at build time, so setting `VALIDATION_OFFLINE=true` runs validation entirely from the cache, with no network access at runtime. This is useful inside TREs and other restricted networks. +The validator fetches profile and context resources over HTTP and caches them. The published v2.* images pre-populate this cache at build time, so setting `VALIDATION_OFFLINE=true` runs validation entirely from the cache, with no network access at runtime. This is useful inside TREs and other networks with restricted internet access. -Online validation (the default) also uses and refreshes the same cache. Offline validation requires `rocrate-validator` at 0.10.0 or later, which the published images include. +Online validation (the default) also uses and refreshes the same cache. Offline validation requires `rocrate-validator` at 0.10.0 or later, which the published v2.* ro-crate validation service images include. diff --git a/docs/upgrading.md b/docs/upgrading.md index 02b9bf4..6588248 100644 --- a/docs/upgrading.md +++ b/docs/upgrading.md @@ -9,7 +9,7 @@ The RO-Crate Validation Service 2.0 release replaced the MinIO-specific storage The old `MINIO_*` variables are replaced by equivalent `S3_*` variables, and storage is now switched on explicitly: -| 1.\* | Now | +| 1.\* | 2.\* | |------|-----| | `MINIO_ENDPOINT` | `S3_ENDPOINT` | | `MINIO_ROOT_USER` | `S3_ACCESS_KEY` | @@ -35,13 +35,13 @@ The service connects to the object store defined in the server-side configuratio ### Crate IDs -A Crate ID is the short label that addresses an RO-Crate in the API path, for example `my-dataset-2026` in `POST /v1/ro_crates/my-dataset-2026/validation`. It is chosen by whoever uploads the RO-Crate, and the service composes the object keys from it: `{S3_CRATE_PREFIX}/.zip` for a zipped RO-Crate, or `{S3_CRATE_PREFIX}//` for a directory. The Crate ID itself is not a filename, path or URL. +A Crate ID is the label that addresses an RO-Crate in the API path, for example `my-dataset-2026` in `POST /v1/ro_crates/my-dataset-2026/validation`. It is chosen by whoever uploads the RO-Crate, and the service composes the object keys from it: `{S3_CRATE_PREFIX}/.zip` for a zipped RO-Crate, or `{S3_CRATE_PREFIX}//` for a directory. The Crate ID itself is not a filename, path or URL. Crate IDs are now validated strictly: they must match `^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$`. This does not allow slashes or path segments. Paths inside the bucket are handled by the prefix settings. Response codes are more specific than the 1.\* `400`/`500` pattern: -| Situation | 1.\* | Now | +| Situation | 1.\* | 2.\* | |-----------|------|-----| | Crate not found in the store | `400` | `404` | | No validation result stored yet | `400` | `404` |