Reduce peak memory use in destination choice and large trip models

[jpn--]

Summary

This PR reduces peak and retained memory in large ActivitySim runs, particularly during location choice, tour/trip destination choice, trip scheduling, and trip mode choice.

The main change is to move chunk boundaries outward so that entire high-memory pipelines—not only utility evaluation—operate on bounded chooser subsets. Large sampled-alternative joins, preprocessors, TAZ-to-MAZ expansion, logsum calculations, and final simulation are completed and released one chunk at a time.

It also adds a cross-platform, best-effort mechanism for returning unused native allocator pages to the operating system between large model segments.

Motivation

Existing evaluator chunking limits the size of individual utility calculations, but several model components still constructed complete merged chooser or sampled-alternative tables before entering those evaluators. At production scale, these intermediate tables can contain millions of rows and many derived columns per worker.
This caused two problems:

• Peak memory could exceed the available RAM even when explicit chunking was enabled.
• Long-lived multiprocess workers retained allocator pages after temporary NumPy and pandas objects were deleted, allowing RSS to remain high across successive model segments.

The in-development Boston full scale model exposed these issues with approximately 3.6 million households, 8.7 million persons, and more than 30 million generated trips.

Changes

Bound location and tour-destination logsums

Workplace, school, and tour-destination logsum calculations now:

• Chunk choosers and sampled alternatives before joining person attributes.
• Run the logsum preprocessor and evaluator within each outer chunk.
• Concatenate results in the original sampled-alternative order.
• Disable inner evaluator chunking because the outer loop owns the chunk lifecycle.
  This prevents a complete sampled-alternative/person merge and its derived columns from remaining live throughout logsum evaluation.

The logsum utility accepts an optional explicit chunk-size override so callers can distinguish configured outer chunking from disabled inner chunking.

Bound the complete trip-destination pipeline

When explicit_chunk is configured, trip destination now streams each chooser chunk through:

1. Destination sampling.
2. TAZ-to-MAZ expansion.
3. Origin-to-destination and destination-to-primary-destination logsums.
4. Alternative preprocessing.
5. Final destination simulation.

Only the final one-row-per-trip choices are retained between chunks.

Additional changes include:

• Bounding TAZ presampling and MAZ selection together, since MAZ expansion was a significant source of peak memory.
• Moving sampled-alternative/trip merges inside logsum chunks.
• Running alternative preprocessing inside the same chunk as final simulation.
• Preventing nested adaptive chunking by passing zero chunk sizes to inner evaluators.
• Releasing temporary arrays after each chunk and purpose segment.
  Estimation and destination-sample-table workflows retain their original whole-segment lifecycle because they intentionally consume the complete sampled table.

If explicit chunking is not configured, trip-destination sampling follows the legacy unchunked path. This preserves compatibility with existing models and lightweight test settings objects.

Reduce full trip-table copies

Trip scheduling now reuses its existing working trip frame instead of creating a second complete copy near the end of the component. Temporary scheduling columns are removed, and any pre-existing columns used as scratch space are restored.
For school-escort runs, the first working frame is released before reconstructing the complete trip table.

Trip mode choice now:

• Groups the original trips table by primary purpose.
• Merges each purpose segment with only the required tour columns.
• Releases each merged segment after its choices have been collected.
• Avoids retaining a complete trips/tours merged table.

Post-choice annotations temporarily receive trip_period for the complete trips table so three-dimensional skim expressions remain correctly aligned. The temporary column is removed afterward, preserving the original table schema.

Return unused allocator memory

A new mem.release_memory() helper:

• Runs Python garbage collection.
• Calls malloc_trim on Linux when available.
• Calls malloc_zone_pressure_relief on macOS when available.
• Requests working-set trimming on Windows.
• Degrades safely to garbage collection if the platform-specific operation is unavailable.

These calls are advisory and do not affect model behavior if the allocator declines to release memory.

Supporting production-run fixes

Two supporting fixes were needed for programmatic and containerized benchmarking:

• Programmatic multiprocess runs now forward the tracing run ID in the same way as the CLI.
• Example resources used by activitysim create are explicitly included in built packages, avoiding incomplete wheels in Docker or source-only build contexts.

These changes are logically separate from model chunking but support running and validating large production models outside the CLI.

Compatibility and model results

The changes do not require updates to existing model specifications or prototype regression targets.

• Legacy behavior is preserved when explicit chunking is not configured.
• Estimation and sample-table generation retain their original processing paths.
• Outer chunkers preserve the configured chunk size; inner chunking is disabled to prevent illegal nested adaptive chunking.
• Chunk results are concatenated and checked against the original index order.

Both Sharrow and non-Sharrow execution paths remain supported.

Note: This PR is a result of work for CTPS (Boston MPO) on model optimization, not a consortium funded task.

[Copilot]

🟡 Changes recommended

Critical nested-chunking behavior and unresolved memory/chunk-boundary issues must be fixed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Reduces peak and retained memory in large ActivitySim runs by widening chunk boundaries, avoiding full-table copies, releasing native allocator memory, and improving production-run support.

Changes:

• Streams destination-choice and logsum pipelines through bounded chooser chunks.
• Reuses trip scheduling and mode-choice working frames.
• Adds cross-platform allocator memory release.
• Includes run-ID forwarding, packaging updates, and regression tests.

File summaries

File	Description
pyproject.toml	Packages example resources.
activitysim/core/workflow/runner.py	Forwards tracing run IDs.
activitysim/core/test/test_workflow_runner.py	Tests run-ID forwarding.
activitysim/core/test/test_mem.py	Adds basic memory-release coverage; cross-platform and fallback branches remain untested.
activitysim/core/mem.py	Adds cross-platform allocator memory release.
activitysim/abm/test/trip_dest/test_trip_destination.py	Compares chunked destination results.
activitysim/abm/test/test_trip_mode_choice.py	Tests temporary trip-period handling.
activitysim/abm/test/test_misc/test_tour_destination_sampling.py	Tests chunked tour-destination processing.
activitysim/abm/test/test_location_choice.py	Tests chunk-local person joins.
activitysim/abm/models/util/tour_destination.py	Chunks tour-destination logsums.
activitysim/abm/models/util/logsums.py	Supports explicit chunk-size overrides.
activitysim/abm/models/trip_scheduling.py	Reuses scheduling frames.
activitysim/abm/models/trip_mode_choice.py	Processes purpose-sized segments, but retained skim-wrapper targets can prevent timely memory release.
activitysim/abm/models/trip_destination.py	Streams destination stages, but incorrectly enables outer chunking outside explicit mode and reapplies chunking internally.
activitysim/abm/models/location_choice.py	Chunks location-choice logsums.

Review details

Suppressed comments (1)

activitysim/core/test/test_mem.py:7

• This test only exercises whichever native branch happens to match the test host, so the macOS and Windows ctypes calls—and the promised fallback when symbols or calls are unavailable—remain untested. Mock sys.platform and the ctypes loaders/functions to cover successful and failing calls on all three branches, including restoration of a previously disabled garbage collector.

def test_release_memory_is_advisory():
    assert isinstance(mem.release_memory(), bool)

• Files reviewed: 15/15 changed files
• Comments generated: 3
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.