Support LEANN for RAG pipelines

[zxqfd555]

Is your feature request related to a problem? Please describe.
Pathway already provides capabilities for various RAG scenarios, including a flexible choice for the vector index storage. LEANN is a robust and popular choice for a vector index; it makes sense to support it. Reference repository here.

Describe the solution you'd like
LEANN is supported and available in RAG pipelines.

[iamonuwa]

Hi! I’d like to work on this issue.
I’ve reviewed both codebases and have a few questions to ensure I take the right approach. Looking at pathway.stdlib.indexing, the current vector index uses usearch via default_usearch_knn_document_index. Would the LEANN integration be a new index backend option alongside usearch (something like default_leann_knn_document_index), or an alternative to the entire DataIndex abstraction?
One thing I want to flag: LEANN achieves its storage savings by storing a pruned graph and recomputing embeddings on-demand at query time rather than storing them upfront. This is different from traditional vector stores. In Pathway’s streaming context where data changes trigger re-indexing, I’m curious whether LEANN should recompute embeddings at query time (maximizing storage savings), cache embeddings during indexing (trading storage for query latency), or be configurable for both.
Is there an existing PR or design doc I should reference? Happy to start with a minimal integration and iterate.​​​​​​​​​​​​​​​​

[zxqfd555]

Hey @iamonuwa,

Amazing, it will be appreciated!

Implementation-wise, there are several options. I wouldn't say that there is a need for an alternative to the whole DataIndex structure, and you can use default_leann_knn_document_index as an example: it reuses DataIndex by providing an instance of USearchKnn class, wrapping USearch. Please pay attention that it uses some of the Rust wrappers inside, so it may be problematic if the interface of LEANN is entirely in Python (it appears it is). On the other hand, if there is a way to add Rust bindings, the index can be supported by implementing an interface as follows: #130.

If there's only a Python interface, I think that the best way to proceed is by implementing a Python output connector that indexes data in LEANN, accessible as, for example, pw.io.leann.write. This is in line with our roadmap, since we want the external indexes to be usable as output connectors as well: #167. This way would be the easiest, and perhaps, the most productive for proceeding with this task.

Please also note that the LEANN library is currently under active development, and it's impossible to easily install it via pip. There are some efforts to facilitate it, notably in yichuan-w/LEANN#225, but still, I would suggest adding an optional dependency for LEANN and providing instructions on how to get the runtime running in the connector docs.

Speaking of computation/memory tradeoffs, I would propose to make it configurable in the parameters of an index or an output connector. If we start with the connector option, we don't have to deal with the queries for a while, but some flexibility for the writer would be beneficial.

Please let me know if you'd like to discuss anything in more detail.

[iamonuwa]

Thanks for the detailed guidance! The output connector approach (pw.io.leann.write) makes sense. I'll follow the pattern in #167.
I'll handle LEANN as an optional dependency with setup instructions given the pip situation (tracking yichuan-w/LEANN#225). For the compute/memory tradeoffs, I'll expose LEANN's configuration parameters in the connector.
I'll start by studying the existing output connector implementations. Will open a draft PR once I have something to show.

[zxqfd555]

The abandoned PR has been integrated into Pathway framework.
The feature will be released with the next Pathway release.