Skip to content

IOBuffer channels#425

Open
arthurpaulino wants to merge 2 commits into
mainfrom
ap/io-buffer-channels
Open

IOBuffer channels#425
arthurpaulino wants to merge 2 commits into
mainfrom
ap/io-buffer-channels

Conversation

@arthurpaulino
Copy link
Copy Markdown
Member

Adds a per-channel dimension to Aiur's IOBuffer and uses it to clean
up IxVM's key-suffix workaround for distinguishing constants, blobs,
and reducibility hints.

Motivation

Today IOBuffer is a single keyed store: any two pieces of
non-deterministic data that happen to share a key collide. IxVM works
around this by appending a discriminator byte to keys (addr ++ [0]
for blobs, addr ++ [1] for hints). The workaround scales poorly,
clutters every reader, and forces all subsystems into one global
namespace.

A channel : G argument on every IO op gives each subsystem its own
key space and its own data arena, with no semantic cost — IOBuffer
is unconstrained, so channels are pure prover-side namespacing.

What changed

Aiur core

IOBuffer is now two-axis:

structure IOBuffer where
  data : Std.HashMap G (Array G)            -- per-channel arenas
  map  : Std.HashMap (G × Array G) IOKeyInfo -- channel-keyed info map

Each of the four ops takes a leading channel : G:

io_get_info(channel, key)
io_set_info(channel, key, idx, len); cont
io_read(channel, idx, len)
io_write(channel, data); cont

Idx in IOKeyInfo is an offset into the channel's arena, not a
global one — two channels can both io_set_info(ch, key, idx=5, len=3)
without aliasing each other's bytes.

The channel threads through every IR stage
(Source/Typed/Simple/Concrete/Bytecode), every compiler pass
(Check/Match/Simple/Concretize/Lower/Layout), every evaluator
(BytecodeEval/SourceEval/Interpret), the parser (Meta.lean), and the
Rust executor + FFI wire format.

IxVM cleanup

Existing IxVM code is migrated off the suffix-byte hack onto dedicated
channels:

channel content
0 constants
1 blobs
2 reducibility hints

Keys are now always the bare 32-G blake3 hash — no addr ++ [0] /
addr ++ [1] suffixes anywhere.

read_byte_stream gains a leading channel argument so callers can
read from the arena they previously queried with io_get_info.

Also drops the dead len == 0 fallback in load_constant_hint:
io_get_info aborts on missing keys, so the harness must seed channel
2 for every defn — there is no absence-handling path to take, and the
previous match len { 0 => 1, _ => ... } was unreachable.

Tests

The existing read_write_io test (in both Tests/Aiur/Aiur.lean and
Tests/Aiur/Cross.lean) is enhanced to exercise channel
disambiguation directly: reads identical key [0] from two channels
backed by distinct arenas, then writes the concatenation to a third
channel.

Test plan

  • lake build
  • lake test -- aiur-cross
  • lake test -- aiur
  • lake test -- --ignored ixvm
  • cargo clippy --all-targets

Commits

  1. Add channel arg to IOBuffer ops for key disambiguation
    pipeline-wide plumbing for the new channel dimension. Existing
    IxVM Aiur sources are migrated to pass channel 0 so nothing
    changes semantically yet.
  2. Use channels to separate consts/blobs/hints in IxVM — drops
    the suffix-byte hack in favour of channels 0/1/2; tightens
    load_constant_hint to remove the dead absence-handling branch.

@arthurpaulino arthurpaulino force-pushed the ap/io-buffer-channels branch 2 times, most recently from 9fa2c8e to 4f386c2 Compare May 22, 2026 19:26
@arthurpaulino arthurpaulino changed the title # IOBuffer channels IOBuffer channels May 22, 2026
`IOBuffer` gains a per-channel dimension. Each of `io_get_info`,
`io_set_info`, `io_read`, `io_write` takes a leading `channel : G`.
The same key on different channels resolves to distinct `(idx, len)`,
and each channel owns its own data arena.

Data model: `data : HashMap G (Array G)` (per-channel arenas),
`map : HashMap (G × Array G) IOKeyInfo` (channel-keyed info map).
Idx in `IOKeyInfo` is an offset into the channel's arena.

All four ops thread the channel through every IR stage
(Source/Typed/Simple/Concrete/Bytecode), every pass
(Check/Match/Simple/Concretize/Lower/Layout), every evaluator
(BytecodeEval/SourceEval/Interpret), the parser (`Meta.lean`), and
the Rust executor + FFI wire format.

In-repo Aiur sources migrated to pass an explicit channel: existing
IxVM kernel/blake3/sha256/ingress/claim code stays on channel 0;
`ClaimHarness.addEntries` and `buildSerdeIOBuffer` seed channel-0
entries. The `read_write_io` test now exercises channel
disambiguation across channels 0/1/2: reads identical key `[0]` on
two channels with distinct arenas, writes the concatenation to a
third.

`lake test -- aiur-cross`, `lake test -- aiur`, and
`lake test -- --ignored ixvm` all pass. `cargo clippy --all-targets`
is clean.
Replaces the suffix-byte hack (`addr ++ [0]` for blobs, `addr ++ [1]`
for hints) with dedicated IOBuffer channels. The key is always the
bare 32-G blake3 hash now.

- channel 0: constants
- channel 1: blobs
- channel 2: reducibility hints

`read_byte_stream` gains a leading `channel` argument so callers can
read from the arena they queried with `io_get_info`. The blob-vs-
constant detection in `load_with_deps` still relies on the empty
marker on channel 0 for blob addrs (so a single channel-0 query
distinguishes the two cases).

Also drops the dead `len == 0` fallback in `load_constant_hint`:
IOBuffer aborts on missing keys, so the harness must seed channel 2
for every defn — there is no absence-handling path to take.
@arthurpaulino arthurpaulino force-pushed the ap/io-buffer-channels branch from 4f386c2 to 20bb200 Compare May 22, 2026 19:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant