Skip to content

Upload SAF files in bounded chunks with progress before creating the import process - #6247

Open
bram-atmire wants to merge 1 commit into
DSpace:mainfrom
bram-atmire:feature/saf-chunked-uploads
Open

Upload SAF files in bounded chunks with progress before creating the import process#6247
bram-atmire wants to merge 1 commit into
DSpace:mainfrom
bram-atmire:feature/saf-chunked-uploads

Conversation

@bram-atmire

@bram-atmire bram-atmire commented Sep 12, 2026

Copy link
Copy Markdown
Member

References

Description

The Import Batch page now uploads the SAF ZIP in bounded chunks through the REST API's new staging endpoints, shows an accessible progress bar with an approximate upload speed, and navigates to the existing process page once the import process has been created. The URL-based import is unchanged.

Instructions for Reviewers

List of changes in this PR:

  • StagedUploadService (new, core/data/staged-upload.service.ts), written against the generic /api/core/uploads resource so the submission form can reuse it (Generic resumable uploads: stage large files in bounded chunks for scripts, submissions and other upload targets DSpace#13106, phase 2). A metadata preflight runs first, so an oversized file is rejected with 413 before any file data is sent. Chunks are built with File.slice() and sent sequentially as PUT requests through the normal JWT and CSRF interceptors. After a transient failure or an in-app navigation (for example a re-login), the upload resumes from the server's committed offset. Retries are bounded and limited to network errors, 408, 429, 502, 503, 504, and a 401 once a renewed token is available. The caller supplies the handoff, which runs once after the last chunk is acknowledged; a repeated handoff after a lost response is answered by the REST API with the same result.
  • ScriptDataService.invokeWithUploads: the JSON form of POST /api/system/scripts/{name}/processes that attaches staged uploads as input files.
  • BatchImportPageComponent. While a file is retained or uploading, the selection form is replaced by a confirmation of the file name and the target collection (or a note that each item's SAF collections file decides). A <progress> element with a role="status" label and aria-busy, a five-second rolling upload speed, a distinct "Upload complete. Starting import" phase after the last chunk is acknowledged, and navigation to /processes/{id} only once the process ID is known. While the upload runs the only action is a red Cancel button in the button row (disabled during the brief handoff); Back and Proceed return once the upload has stopped. A resume notice appears when an unfinished upload is retained. Notifications cover success, expired login, oversized file and generic failure.
  • New i18n keys admin.batch-import.upload.* in en.json5.
  • Specs: 10 for the service (preflight rejection, chunk sequencing, speed averaging and stalls, retransmission accounting, resume, handoff retry, an already consumed upload, cancel, current JWT per chunk, expiry between chunks with the retained settings), one for the JSON script invocation, and the page specs.

Points for reviewers:

  • This requires the backend PR: the service resolves the uploads link from the API root. Against a REST API without that link the page reports an error instead of falling back to the multipart request. If a silent fallback is preferred for mixed deployments, that is a small addition.
  • Upload progress is not import progress. 100% is reserved for the server's acknowledgement of the final chunk, and the process page owns execution status and logs.
  • The selected File and the upload receipt survive navigation within the app but not a browser reload.

How to test:

  1. Deploy Stage large uploads in bounded chunks and start scripts from staged uploads (SAF import first) DSpace#13104, log in as an administrator, open Import Batch, choose a collection and a SAF ZIP, and click Proceed. Watch the progress bar and speed, then the "Starting import" phase, then the process page.
  2. Pick a ZIP larger than the REST API's spring.servlet.multipart.max-file-size: the error shows immediately and the network tab contains only the metadata POST.
  3. Throttle the network in the browser's developer tools and cancel mid-upload, then start again with the same file to see the upload resume from the committed offset.
  4. Set a short jwt.login.token.expiration on the backend and upload a large file with a small chunk size: later chunk requests carry the renewed token.

Validation: npm run lint (0 errors), npm run check-circ-deps, and the focused specs saf-upload.service.spec.ts and batch-import-page.component.spec.ts pass.

Checklist

  • My PR is created against the main branch of code (unless it is a backport or is fixing an issue specific to an older branch).
  • My PR is small in size (e.g. less than 1,000 lines of code, not including comments & specs/tests), or I have provided reasons as to why that's not possible.
  • My PR follows all coding best practices based on the Code Conventions Guide
  • My PR passes ESLint validation using npm run lint
  • My PR doesn't introduce circular dependencies (verified via npm run check-circ-deps)
  • My PR includes TypeDoc comments for all new (or modified) public methods and classes. It also includes TypeDoc for large or complex private methods.
  • My PR passes all specs/tests and includes new/updated specs or tests based on the Code Testing Guide.
  • My PR aligns with Accessibility guidelines if it makes changes to the user interface.
  • My PR uses i18n (internationalization) keys instead of hardcoded English text, to allow for translations.
  • My PR includes details on how to test it. I've provided clear instructions to reviewers on how to successfully test this fix or feature.
  • If my PR includes new libraries/dependencies (in package.json), I've made sure their licenses align with the DSpace BSD License based on the Licensing of Contributions documentation. (No new dependencies.)
  • If my PR includes new features or configurations, I've provided basic technical documentation in the PR itself.
  • If my PR fixes an issue ticket, I've linked them together.

https://claude.ai/code/session_01HtPYmqfFzg7fmZgovGWP5A

…import process

The Import Batch page now stages the SAF ZIP through the REST API's
generic staged-upload resource instead of one multipart request, and
starts the import with the JSON script invocation that references the
staged upload. Each chunk carries the current JWT and CSRF token, a lost
response is retried against the server's committed offset, and the
upload can resume after signing in again with its original settings.
While a file is retained or uploading the page confirms the file and
target collection, shows an accessible progress bar with approximate
upload speed and a distinct "starting import" phase, offers only
cancellation, and navigates to the existing process page once the
process exists. The URL-based import is unchanged.

Relates to DSpace/DSpace#13103 and DSpace/DSpace#13106.

Claude-Session: https://claude.ai/code/session_01HtPYmqfFzg7fmZgovGWP5A
@bram-atmire
bram-atmire force-pushed the feature/saf-chunked-uploads branch from e3dba83 to 4da84d0 Compare September 12, 2026 10:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant