feat: add multimodal embed() activity and provider adapters

.changeset/embed-activity-multimodal.md

@@ -0,0 +1,14 @@
+---
+'@tanstack/ai': minor
+'@tanstack/ai-event-client': minor
+'@tanstack/ai-openai': minor
+'@tanstack/ai-gemini': minor
+'@tanstack/ai-mistral': minor
+'@tanstack/ai-bedrock': minor
+'@tanstack/ai-ollama': minor
+'@tanstack/ai-cohere': minor
+---
+
+Add a multimodal `embed()` activity. A single primitive covers one input or a batch — `input` accepts a string, a text part, an image part, or a fused text+image item written as a nested `Array<ContentPart>` (`[textPart, imagePart]`, the same shape chat messages use), one vector per item, with the accepted item types narrowed per model at compile time. Top-level `dimensions` requests Matryoshka output sizes where supported. Results carry `embeddings: [{ vector, index }]` plus `usage` when the provider reports it, and `embed()` participates in generation middleware, debug logging, OTel (`gen_ai.operation.name: embeddings`), and devtools events like every other activity.
+
+Provider adapters: `openaiEmbedding` (text-embedding-3-small/large), `geminiEmbedding` (gemini-embedding-001), `mistralEmbedding` (mistral-embed, codestral-embed), `ollamaEmbedding` (nomic-embed-text and any local model), `bedrockEmbedding` (Titan Text V2, Titan Multimodal G1 with fused text+image, Cohere Embed v3 on Bedrock), and `@tanstack/ai-cohere`'s `cohereEmbedding` (embed-v4.0, multimodal text+image with required `inputType`).

docs/adapters/bedrock.md

@@ -204,6 +204,69 @@ for await (const chunk of chat({
 }
 ```
 
+## Embeddings
+
+Generate embedding vectors with Titan or Cohere embedding models via `InvokeModel`:
+
+```typescript
+import { embed } from "@tanstack/ai";
+import { bedrockEmbedding } from "@tanstack/ai-bedrock";
+
+const result = await embed({
+  adapter: bedrockEmbedding("amazon.titan-embed-text-v2:0"),
+  input: ["a red guitar", "a blue drum kit"],
+  dimensions: 512, // 256 | 512 | 1024
+});
+
+console.log(result.embeddings[0]?.vector);
+```
+
+Titan Multimodal embeds text and images — alone or fused into a single vector. Fuse parts by nesting them in an array (the same `Array<ContentPart>` shape a chat message's `content` uses); the outer array is the item list, so this embeds one fused item:
+
+```typescript
+import { embed } from "@tanstack/ai";
+import { bedrockEmbedding } from "@tanstack/ai-bedrock";
+
+const productPhoto = "iVBORw0KGgo..."; // base64 image data
+
+const result = await embed({
+  adapter: bedrockEmbedding("amazon.titan-embed-image-v1"),
+  input: [
+    [
+      { type: "text", content: "a red guitar" },
+      {
+        type: "image",
+        source: {
+          type: "data",
+          value: productPhoto,
+          mimeType: "image/png",
+        },
+      },
+    ],
+  ],
+  dimensions: 1024, // 256 | 384 | 1024
+});
+```
+
+Cohere Embed v3 on Bedrock is also supported (text-only, batched, requires `inputType`):
+
+```typescript
+import { embed } from "@tanstack/ai";
+import { bedrockEmbedding } from "@tanstack/ai-bedrock";
+
+const result = await embed({
+  adapter: bedrockEmbedding("cohere.embed-english-v3"),
+  input: ["a red guitar", "a blue drum kit"],
+  modelOptions: { inputType: "search_document" },
+});
+```
+
+> Titan models have no batch API — a batch of N items runs as N `InvokeModel`
+> calls under a small concurrency cap. Titan Multimodal does not fetch remote
+> image URLs; pass base64 data (or a `data:` URI).
+
+See the [Embeddings guide](../embeddings.md) for the full API.
+
 ## Model Availability
 
 The adapter ships with a hand-seeded snapshot catalog (`src/model-catalog.generated.ts`) of confirmed model IDs. This catalog can be refreshed by the maintainer script `scripts/fetch-bedrock-models.ts`, which calls `ListFoundationModels` with AWS credentials.

docs/adapters/cohere.md

@@ -2,105 +2,173 @@
 title: Cohere
 id: cohere-adapter
 order: 11
-description: "Rerank documents by relevance to a query with Cohere's rerank models in TanStack AI via the @tanstack/ai-cohere adapter."
+description: "Use Cohere with TanStack AI via @tanstack/ai-cohere: embed-v4.0 multimodal embeddings for semantic search and RAG, plus rerank models for relevance."
 keywords:
   - tanstack ai
   - cohere
+  - embed-v4
+  - embeddings
+  - multimodal embeddings
+  - semantic search
   - rerank
   - reranking
-  - relevance
   - retrieval
   - adapter
 ---
 
-The Cohere adapter is **rerank-focused**. It exposes one capability:
+The Cohere adapter covers the two retrieval steps of a RAG pipeline:
 
-- **Reranking** (`cohereRerank`) — reorder documents by relevance to a query via `rerank()`.
+- **Embeddings** (`cohereEmbedding`): turn text, images, and fused text+image inputs into vectors with `embed()`.
+- **Reranking** (`cohereRerank`): reorder candidate documents by relevance to a query with `rerank()`.
 
-It does not support text `chat()`, `summarize()`, embeddings, or media — use
-OpenAI, Anthropic, or Gemini for those. The adapter talks to Cohere's
-`/v2/rerank` endpoint directly over `fetch` (no SDK dependency).
+It does not support `chat()`, `summarize()`, or media generation. Use OpenAI, Anthropic, or Gemini for those. The adapter talks to Cohere's HTTP API directly over `fetch`, with no SDK dependency.
 
 ## Installation
 
 ```bash
-npm install @tanstack/ai-cohere
+npm install @tanstack/ai @tanstack/ai-cohere
 ```
 
-Peer dependency:
+## Embeddings
 
-```bash
-npm install @tanstack/ai
+```typescript
+import { embed } from "@tanstack/ai";
+import { cohereEmbedding } from "@tanstack/ai-cohere";
+
+const result = await embed({
+  adapter: cohereEmbedding("embed-v4.0"),
+  input: ["a red guitar", "a blue drum kit"],
+  modelOptions: { inputType: "search_document" },
+});
+
+console.log(result.embeddings[0]?.vector);
+console.log(result.usage?.promptTokens);
 ```
 
-## Basic Usage
+`inputType` is required by Cohere's API. Use `search_document` at index time and `search_query` at query time (or `classification` / `clustering` for those workloads). TanStack AI enforces this at the type level, so `modelOptions` is required for Cohere embedding calls.
+
+### Multimodal Embeddings
+
+embed-v4.0 embeds images alongside text. An image part produces an image vector. A nested array of parts (`[textPart, imagePart]`) fuses text and image into one vector, which suits product catalogs and screenshot search. The outer array is the item list, so nest to fuse:
 
 ```typescript
-import { rerank } from '@tanstack/ai'
-import { cohereRerank } from '@tanstack/ai-cohere'
+import { embed } from "@tanstack/ai";
+import { cohereEmbedding } from "@tanstack/ai-cohere";
+
+const productPhoto = "iVBORw0KGgo..."; // base64 image data
+
+const result = await embed({
+  adapter: cohereEmbedding("embed-v4.0"),
+  input: [
+    {
+      type: "image",
+      source: {
+        type: "data",
+        value: productPhoto,
+        mimeType: "image/png",
+      },
+    },
+    // A nested array fuses its parts into a single vector.
+    [
+      { type: "text", content: "Fender Stratocaster, sunburst finish" },
+      {
+        type: "image",
+        source: {
+          type: "data",
+          value: productPhoto,
+          mimeType: "image/png",
+        },
+      },
+    ],
+  ],
+  modelOptions: { inputType: "search_document" },
+});
+
+console.log(result.embeddings.length); // 2
+```
 
-const { rerankedDocuments } = await rerank({
-  adapter: cohereRerank('rerank-v3.5'),
-  query: 'talk about rain',
-  documents: ['sunny day at the beach', 'rainy afternoon in the city'],
-})
+Cohere's API does not fetch remote image URLs. Pass base64 data (or a `data:` URI), or opt into adapter-side downloading:
+
+```typescript
+import { embed } from "@tanstack/ai";
+import { cohereEmbedding } from "@tanstack/ai-cohere";
 
-console.log(rerankedDocuments[0]) // 'rainy afternoon in the city'
+const adapter = cohereEmbedding("embed-v4.0", { allowUrlFetch: true });
+
+const result = await embed({
+  adapter,
+  input: {
+    type: "image",
+    source: { type: "url", value: "https://example.com/guitar.png" },
+  },
+  modelOptions: { inputType: "search_document" },
+});
 ```
 
-For the full reranking guide — object documents, RAG pipelines, options, and
-the result shape — see [Reranking](../rerank/rerank).
+### Requesting Dimensions
 
-## Models
+embed-v4.0 supports Matryoshka output dimensions via the top-level `dimensions` option:
+
+```typescript
+import { embed } from "@tanstack/ai";
+import { cohereEmbedding } from "@tanstack/ai-cohere";
+
+const result = await embed({
+  adapter: cohereEmbedding("embed-v4.0"),
+  input: "a red guitar",
+  dimensions: 1024, // 256 | 512 | 1024 | 1536
+  modelOptions: { inputType: "search_document" },
+});
+```
 
-| Model                      | Description                              |
-| -------------------------- | ---------------------------------------- |
-| `rerank-v3.5`              | Latest multilingual reranker (recommended) |
-| `rerank-english-v3.0`      | English-optimized reranker               |
-| `rerank-multilingual-v3.0` | Multilingual reranker                    |
+## Reranking
 
-## Configuration
+```typescript
+import { rerank } from "@tanstack/ai";
+import { cohereRerank } from "@tanstack/ai-cohere";
 
-`cohereRerank(model, config?)` reads `COHERE_API_KEY` from the environment.
-`config` accepts:
+const { rerankedDocuments } = await rerank({
+  adapter: cohereRerank("rerank-v3.5"),
+  query: "talk about rain",
+  documents: ["sunny day at the beach", "rainy afternoon in the city"],
+});
 
-| Option    | Type                       | Default                    | Description                          |
-| --------- | -------------------------- | -------------------------- | ------------------------------------ |
-| `baseUrl` | `string`                   | `https://api.cohere.com`   | Override the API base URL            |
-| `headers` | `Record<string, string>`  | —                          | Extra headers merged into requests   |
+console.log(rerankedDocuments[0]); // 'rainy afternoon in the city'
+```
 
-### Provider Options
+For the full reranking guide, with object documents, RAG pipelines, options, and the result shape, see [Reranking](../rerank/rerank).
 
-Per-request options are passed via `modelOptions` on `rerank()`:
+Per-request rerank options go on `modelOptions`:
 
 ```typescript
-import { rerank } from '@tanstack/ai'
-import { cohereRerank } from '@tanstack/ai-cohere'
+import { rerank } from "@tanstack/ai";
+import { cohereRerank } from "@tanstack/ai-cohere";
 
 const { ranking } = await rerank({
-  adapter: cohereRerank('rerank-v3.5'),
-  query: 'refund policy',
-  documents: ['Returns accepted within 30 days.', 'Free shipping over $50.'],
+  adapter: cohereRerank("rerank-v3.5"),
+  query: "refund policy",
+  documents: ["Returns accepted within 30 days.", "Free shipping over $50."],
   modelOptions: {
     maxTokensPerDoc: 512, // Cap tokens kept per document (Cohere default: 4096)
   },
-})
+});
 
-console.log(ranking)
+console.log(ranking);
 ```
 
-## Explicit API Keys
-
-To pass an API key directly instead of reading the environment:
-
-```typescript
-import { createCohereRerank } from '@tanstack/ai-cohere'
+## Models
 
-const adapter = createCohereRerank('rerank-v3.5', 'your-cohere-api-key')
-```
+| Model                      | Capability | Description                                                       |
+| -------------------------- | ---------- | ----------------------------------------------------------------- |
+| `embed-v4.0`               | Embeddings | Multimodal (text + images), Matryoshka `dimensions` support        |
+| `rerank-v3.5`              | Reranking  | Latest multilingual reranker (recommended)                         |
+| `rerank-english-v3.0`      | Reranking  | English-optimized reranker                                         |
+| `rerank-multilingual-v3.0` | Reranking  | Multilingual reranker                                              |
 
 ## Environment Variables
 
+Both adapters read your API key from the environment:
+
 ```bash
 COHERE_API_KEY=your-cohere-api-key
 ```
@@ -109,24 +177,54 @@ COHERE_API_KEY=your-cohere-api-key
 | ---------------- | -------- | ------------------- |
 | `COHERE_API_KEY` | Yes      | Your Cohere API key |
 
-Get your API key from the [Cohere dashboard](https://dashboard.cohere.com/).
+Get a key from the [Cohere dashboard](https://dashboard.cohere.com/api-keys).
+
+## Explicit API Keys
+
+To pass a key directly instead of reading the environment, use the `create*` factories:
+
+```typescript
+import {
+  createCohereEmbedding,
+  createCohereRerank,
+} from "@tanstack/ai-cohere";
+
+const embedAdapter = createCohereEmbedding(
+  "embed-v4.0",
+  process.env.MY_COHERE_KEY!,
+);
+const rerankAdapter = createCohereRerank("rerank-v3.5", "your-cohere-api-key");
+```
 
 ## API Reference
 
-### `cohereRerank(model, config?)`
+### `cohereEmbedding(model, config?)`
 
-Creates a Cohere rerank adapter for use with `rerank()`, reading
-`COHERE_API_KEY` from the environment.
+Creates an embedding adapter using `COHERE_API_KEY` from the environment.
 
-### `createCohereRerank(model, apiKey, config?)`
+- `model`: `"embed-v4.0"`
+- `config.baseUrl`: override the API base URL (default `https://api.cohere.com`)
+- `config.headers`: extra request headers
+- `config.allowUrlFetch`: download `http(s)` image URLs and inline them as base64 (default `false`)
+
+### `createCohereEmbedding(model, apiKey, config?)`
 
-Same as `cohereRerank`, but takes an explicit API key.
+Same as `cohereEmbedding` with an explicit API key.
 
-## Limitations
+### `cohereRerank(model, config?)`
+
+Creates a rerank adapter using `COHERE_API_KEY` from the environment.
+
+- `model`: one of the rerank models above
+- `config.baseUrl`: override the API base URL (default `https://api.cohere.com`)
+- `config.headers`: extra request headers
+
+### `createCohereRerank(model, apiKey, config?)`
 
-- **Rerank only** — Use OpenAI, Anthropic, or Gemini for `chat()`, `summarize()`, embeddings, or media generation.
+Same as `cohereRerank` with an explicit API key.
 
 ## Next Steps
 
-- [Reranking Guide](../rerank/rerank) — full walkthrough including RAG pipelines
-- [OpenAI Adapter](./openai) — text, embeddings, and media
+- [Embeddings guide](../embeddings.md): the full `embed()` API
+- [Reranking guide](../rerank/rerank): full walkthrough including RAG pipelines
+- [Generation Hooks](../media/generation-hooks.md): usage and lifecycle middleware