feat: add 6 vector-search backend plugins#320
Open
hgaol wants to merge 6 commits into
Open
Conversation
…figuration and sync functionality
There was a problem hiding this comment.
Pull request overview
This PR adds six new vector-search backend plugins to the Apache Answer plugins repo to address Issue #319, enabling multiple external vector stores to be used for semantic search.
Changes:
- Introduced 6 new vector-search backend plugins: pgvector, qdrant, milvus, weaviate, elasticsearch, chromadb (each as its own Go module under
vector-search-*). - Implemented common capabilities across backends: config-driven initialization, embedding generation via Answer’s embedding API integration, search-by-vector, upsert/delete, and background full-sync via
VectorSearchSyncer. - Added per-plugin documentation and i18n strings (en_US / zh_CN) for Admin UI configuration.
Reviewed changes
Copilot reviewed 48 out of 53 changed files in this pull request and generated 10 comments.
Show a summary per file
| File | Description |
|---|---|
| vector-search-weaviate/weaviate.go | Weaviate backend implementation: config, client init, schema/class creation, search/upsert/delete. |
| vector-search-weaviate/sync.go | Background full-sync and bulk indexing for Weaviate backend. |
| vector-search-weaviate/README.md | Usage and configuration guide for Weaviate plugin. |
| vector-search-weaviate/info.yaml | Plugin metadata (slug/type/version/author/link) for Weaviate plugin. |
| vector-search-weaviate/i18n/zh_CN.yaml | Chinese translations for Weaviate plugin name/description/config fields. |
| vector-search-weaviate/i18n/en_US.yaml | English translations for Weaviate plugin name/description/config fields. |
| vector-search-weaviate/i18n/translation.go | Translation keys/constants for Weaviate plugin. |
| vector-search-weaviate/go.mod | Go module definition and dependencies for Weaviate plugin. |
| vector-search-qdrant/qdrant.go | Qdrant backend implementation: config, gRPC client, collection management, search/upsert/delete. |
| vector-search-qdrant/sync.go | Background full-sync and bulk indexing for Qdrant backend. |
| vector-search-qdrant/README.md | Usage and configuration guide for Qdrant plugin. |
| vector-search-qdrant/info.yaml | Plugin metadata for Qdrant plugin. |
| vector-search-qdrant/i18n/zh_CN.yaml | Chinese translations for Qdrant plugin. |
| vector-search-qdrant/i18n/en_US.yaml | English translations for Qdrant plugin. |
| vector-search-qdrant/i18n/translation.go | Translation keys/constants for Qdrant plugin. |
| vector-search-qdrant/go.mod | Go module definition and dependencies for Qdrant plugin. |
| vector-search-pgvector/pgvector.go | PostgreSQL+pgvector backend implementation: config, schema/table/index management, search/upsert/delete. |
| vector-search-pgvector/sync.go | Background full-sync and bulk indexing for pgvector backend. |
| vector-search-pgvector/README.md | Usage and configuration guide for pgvector plugin. |
| vector-search-pgvector/info.yaml | Plugin metadata for pgvector plugin. |
| vector-search-pgvector/i18n/zh_CN.yaml | Chinese translations for pgvector plugin. |
| vector-search-pgvector/i18n/en_US.yaml | English translations for pgvector plugin. |
| vector-search-pgvector/i18n/translation.go | Translation keys/constants for pgvector plugin. |
| vector-search-pgvector/go.mod | Go module definition and dependencies for pgvector plugin. |
| vector-search-milvus/milvus.go | Milvus backend implementation: config, collection/index management, search/upsert/delete. |
| vector-search-milvus/sync.go | Background full-sync and bulk indexing for Milvus backend. |
| vector-search-milvus/README.md | Usage and configuration guide for Milvus plugin. |
| vector-search-milvus/info.yaml | Plugin metadata for Milvus plugin. |
| vector-search-milvus/i18n/zh_CN.yaml | Chinese translations for Milvus plugin. |
| vector-search-milvus/i18n/en_US.yaml | English translations for Milvus plugin. |
| vector-search-milvus/i18n/translation.go | Translation keys/constants for Milvus plugin. |
| vector-search-milvus/go.mod | Go module definition and dependencies for Milvus plugin. |
| vector-search-elasticsearch/es.go | Elasticsearch backend implementation: config, index creation, kNN search, upsert/delete. |
| vector-search-elasticsearch/sync.go | Background full-sync and bulk indexing for Elasticsearch backend. |
| vector-search-elasticsearch/README.md | Usage and configuration guide for Elasticsearch plugin. |
| vector-search-elasticsearch/info.yaml | Plugin metadata for Elasticsearch plugin. |
| vector-search-elasticsearch/i18n/zh_CN.yaml | Chinese translations for Elasticsearch plugin. |
| vector-search-elasticsearch/i18n/en_US.yaml | English translations for Elasticsearch plugin. |
| vector-search-elasticsearch/i18n/translation.go | Translation keys/constants for Elasticsearch plugin. |
| vector-search-elasticsearch/go.mod | Go module definition and dependencies for Elasticsearch plugin. |
| vector-search-chromadb/chromadb.go | ChromaDB backend implementation via REST: collection ensure, search, upsert/delete. |
| vector-search-chromadb/sync.go | Background full-sync and bulk indexing for ChromaDB backend. |
| vector-search-chromadb/README.md | Usage and configuration guide for ChromaDB plugin. |
| vector-search-chromadb/info.yaml | Plugin metadata for ChromaDB plugin. |
| vector-search-chromadb/i18n/zh_CN.yaml | Chinese translations for ChromaDB plugin. |
| vector-search-chromadb/i18n/en_US.yaml | English translations for ChromaDB plugin. |
| vector-search-chromadb/i18n/translation.go | Translation keys/constants for ChromaDB plugin. |
| vector-search-chromadb/go.mod | Go module definition and dependencies for ChromaDB plugin. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
|
||
| // ConfigReceiver applies configuration from the admin UI. | ||
| func (e *VectorSearchEngine) ConfigReceiver(config []byte) error { | ||
| log.Debugf("pgvector: ConfigReceiver called, raw config=%s", string(config)) |
Comment on lines
+300
to
+301
| log.Debugf("pgvector: config parsed: dsn=%s model=%s level=%s threshold=%.2f", | ||
| conf.DSN, conf.EmbeddingModel, conf.EmbeddingLevel, conf.SimilarityThreshold) |
Comment on lines
+31
to
+49
| func (e *VectorSearchEngine) sync() { | ||
| if e.syncer == nil { | ||
| log.Warn("weaviate: syncer not registered, skip sync") | ||
| return | ||
| } | ||
| if e.syncing { | ||
| log.Warn("weaviate: sync already running, skip") | ||
| return | ||
| } | ||
|
|
||
| go func() { | ||
| e.lock.Lock() | ||
| defer e.lock.Unlock() | ||
| if e.syncing { | ||
| return | ||
| } | ||
| e.syncing = true | ||
| defer func() { e.syncing = false }() | ||
|
|
Comment on lines
+29
to
+47
| func (e *VectorSearchEngine) sync() { | ||
| if e.syncer == nil { | ||
| log.Warn("qdrant: syncer not registered, skip sync") | ||
| return | ||
| } | ||
| if e.syncing { | ||
| log.Warn("qdrant: sync already running, skip") | ||
| return | ||
| } | ||
|
|
||
| go func() { | ||
| e.lock.Lock() | ||
| defer e.lock.Unlock() | ||
| if e.syncing { | ||
| return | ||
| } | ||
| e.syncing = true | ||
| defer func() { e.syncing = false }() | ||
|
|
Comment on lines
+31
to
+49
| func (e *VectorSearchEngine) sync() { | ||
| if e.syncer == nil { | ||
| log.Warn("pgvector: syncer not registered, skip sync") | ||
| return | ||
| } | ||
| if e.syncing { | ||
| log.Warn("pgvector: sync already running, skip") | ||
| return | ||
| } | ||
|
|
||
| go func() { | ||
| e.lock.Lock() | ||
| defer e.lock.Unlock() | ||
| if e.syncing { | ||
| return | ||
| } | ||
| e.syncing = true | ||
| defer func() { e.syncing = false }() | ||
|
|
Comment on lines
+29
to
+46
| func (e *VectorSearchEngine) sync() { | ||
| if e.syncer == nil { | ||
| log.Warn("milvus: syncer not registered, skip sync") | ||
| return | ||
| } | ||
| if e.syncing { | ||
| log.Warn("milvus: sync already running, skip") | ||
| return | ||
| } | ||
|
|
||
| go func() { | ||
| e.lock.Lock() | ||
| defer e.lock.Unlock() | ||
| if e.syncing { | ||
| return | ||
| } | ||
| e.syncing = true | ||
| defer func() { e.syncing = false }() |
Comment on lines
+29
to
+46
| func (e *VectorSearchEngine) sync() { | ||
| if e.syncer == nil { | ||
| log.Warn("chromadb: syncer not registered, skip sync") | ||
| return | ||
| } | ||
| if e.syncing { | ||
| log.Warn("chromadb: sync already running, skip") | ||
| return | ||
| } | ||
|
|
||
| go func() { | ||
| e.lock.Lock() | ||
| defer e.lock.Unlock() | ||
| if e.syncing { | ||
| return | ||
| } | ||
| e.syncing = true | ||
| defer func() { e.syncing = false }() |
Comment on lines
+31
to
+48
| func (e *VectorSearchEngine) sync() { | ||
| if e.syncer == nil { | ||
| log.Warn("es-vector: syncer not registered, skip sync") | ||
| return | ||
| } | ||
| if e.syncing { | ||
| log.Warn("es-vector: sync already running, skip") | ||
| return | ||
| } | ||
|
|
||
| go func() { | ||
| e.lock.Lock() | ||
| defer e.lock.Unlock() | ||
| if e.syncing { | ||
| return | ||
| } | ||
| e.syncing = true | ||
| defer func() { e.syncing = false }() |
Comment on lines
+443
to
+458
| if err := e.ensureClass(context.Background()); err != nil { | ||
| if conf.APIKey != "" { | ||
| log.Debugf("weaviate: ensureClass failed with API key auth, retrying without auth (anonymous access)") | ||
| cfg.AuthConfig = nil | ||
| client, err = weaviate.NewClient(cfg) | ||
| if err != nil { | ||
| return fmt.Errorf("create weaviate client: %w", err) | ||
| } | ||
| e.client = client | ||
| if err := e.ensureClass(context.Background()); err != nil { | ||
| return fmt.Errorf("ensure class: %w", err) | ||
| } | ||
| } else { | ||
| return fmt.Errorf("ensure class: %w", err) | ||
| } | ||
| } |
Comment on lines
+35
to
+38
| - **Embedding dimensions** are auto-detected from the configured model. No manual dimension configuration is needed. | ||
| - On first configuration, the plugin creates an index `answer_vector` with a `dense_vector` field matching the detected dimensions and cosine similarity. | ||
| - If the embedding model changes and produces different dimensions, the index is automatically deleted and recreated. | ||
| - Uses Elasticsearch kNN search for vector similarity queries. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Close #319