Skip to content

Improve performance: memory footprint and speed - #385

Draft
j-atkins wants to merge 12 commits into
mainfrom
performance
Draft

j-atkins wants to merge 12 commits into
mainfrom
performance

Conversation

@j-atkins

@j-atkins j-atkins commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

This PR makes changes to fieldset ingestion to improve performance, especially so in RAM-limited workspaces.

  • Adds more FetchSpec detail to different instruments (principally for the depth dimension), to minimise redundant data retrieval.
  • Move to using Parcels to_chunk_cached_arrays() instead of to_windowed_arrays(), which is more suitable and performant for VirtualShip simulations.
    • Also sets the max_cache_bytes to 300_000_000 (rather than the Parcels default 600_000_000).
    • This appears to strike a balance of speed vs. memory footprint, which I have found to be especially important when using VirtualShip on RAM-limited virtual machines, e.g. the VirtualShip GitHub Codespaces implementation.
  • Improved performance with to_chunk_cached_arrays() means the tmp ds writing step (_via_tmp_ds) can be removed in VirtuaShip. Therefore, wrapping instrument simulations in a context manager for proper tmp dir handling is also removed.

N.B.

There is a temporary fix in argo_floats.py to handle an indexing bug in Parcels (i.e. #2906). I will keep this PR as a draft until fixed upstream.


  • Update tests

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant