feat: allow creating drop message for accumulator

[vaibhavtiwari33]

Closes #357

Testing

Tested with and without message drop semantics.

Disk Usage

When using the drop method the size of the folder where WALs are stored remains roughly constant across time:

/workspace/cws (master)$ kexec accumulator-blackhole-blackhole-0-gjb7u -c numa -- /bin/bash
root@accumulator-blackhole-blackhole-0:/# du -sh /var/numaflow/pbq
2.8M    /var/numaflow/pbq
root@accumulator-blackhole-blackhole-0:/# du -sh /var/numaflow/pbq
2.0M    /var/numaflow/pbq
root@accumulator-blackhole-blackhole-0:/# du -sh /var/numaflow/pbq
2.3M    /var/numaflow/pbq

When not using the drop method and running the loop without awaiting any result, the size steadily increases:

/workspace/cws (master)$ kexec accumulator-empty-blackhole-0-sn8dg -c numa -- /bin/bash
root@accumulator-empty-blackhole-0:/# du -sh /var/numaflow/pbq
7.4M    /var/numaflow/pbq
root@accumulator-empty-blackhole-0:/# du -sh /var/numaflow/pbq
16M     /var/numaflow/pbq
root@accumulator-empty-blackhole-0:/# du -sh /var/numaflow/pbq
33M     /var/numaflow/pbq

Memory Usage

When using the drop method, the memory usage for the numa container in accumulator vertex remains mostly steady

root@accumulator-blackhole-blackhole-0:/# cat /sys/fs/cgroup/memory.current
12353536
root@accumulator-blackhole-blackhole-0:/# cat /sys/fs/cgroup/memory.current
12410880
root@accumulator-blackhole-blackhole-0:/# cat /sys/fs/cgroup/memory.current
12832768
root@accumulator-blackhole-blackhole-0:/# cat /sys/fs/cgroup/memory.current
12922880
root@accumulator-blackhole-blackhole-0:/# cat /sys/fs/cgroup/memory.current
12718080
root@accumulator-blackhole-blackhole-0:/# cat /sys/fs/cgroup/memory.current
12152832

Without the drop method the memory usage increases quickly

/workspace/cws (master)$ kexec accumulator-empty-blackhole-0-sn8dg -c numa -- /bin/bash
root@accumulator-empty-blackhole-0:/# cat /sys/fs/cgroup/memory.current
186560512
root@accumulator-empty-blackhole-0:/# cat /sys/fs/cgroup/memory.current
201605120
root@accumulator-empty-blackhole-0:/# cat /sys/fs/cgroup/memory.current
205459456
root@accumulator-empty-blackhole-0:/# cat /sys/fs/cgroup/memory.current
220364800
root@accumulator-empty-blackhole-0:/# cat /sys/fs/cgroup/memory.current
283959296
root@accumulator-empty-blackhole-0:/# cat /sys/fs/cgroup/memory.current
462209024

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.99%. Comparing base (5fd1823) to head (7195e76).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #358   +/-   ##
=======================================
  Coverage   92.99%   92.99%           
=======================================
  Files          68       68           
  Lines        3656     3656           
  Branches      236      236           
=======================================
  Hits         3400     3400           
  Misses        190      190           
  Partials       66       66           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[vaibhavtiwari33]

Not required anymore: numaproj/numaflow-java#218 (comment)

[vaibhavtiwari33]

numaproj/numaflow#3645 (comment)

[vigith]

can you create an issue to track for pynumaflow-lite too?

[BulkBeing]

May be we should use https://github.com/astral-sh/uv-docker-example/blob/main/standalone.Dockerfile as the reference for all Dockerfiles. It runs final image without uv, no dev dependencies, runs as non-root user, and also sets PYTHONUNBUFFERED.

[vaibhavtiwari33]

May be we should use https://github.com/astral-sh/uv-docker-example/blob/main/standalone.Dockerfile as the reference for all Dockerfiles. It runs final image without uv, no dev dependencies, runs as non-root user, and also sets PYTHONUNBUFFERED.

Created a separate issue