When searching in FTS-only mode (no embedding provider), extract meaningful
keywords from conversational queries using LLM to improve search results.
Changes:
- New query-expansion module with keyword extraction
- Supports English and Chinese stop word filtering
- Null safety guards for FTS-only mode (provider can be null)
- Lint compliance fixes for string iteration
This helps users find relevant memory entries even with vague queries.
When no embedding provider is available (e.g., OAuth mode without API keys),
memory_search now falls back to FTS-only mode instead of returning disabled: true.
Changes:
- embeddings.ts: return null provider with reason instead of throwing
- manager.ts: handle null provider, use FTS-only search mode
- manager-search.ts: allow searching all models when provider is undefined
- memory-tool.ts: expose search mode in results
The search results now include a 'mode' field indicating 'hybrid' or 'fts-only'.
* fix: enforce embedding model token limit to prevent 8192 overflow
- Replace EMBEDDING_APPROX_CHARS_PER_TOKEN=1 with UTF-8 byte length
estimation (safe upper bound for tokenizer output)
- Add EMBEDDING_MODEL_MAX_TOKENS=8192 hard cap
- Add splitChunkToTokenLimit() that binary-searches for the largest
safe split point, with surrogate pair handling
- Add enforceChunkTokenLimit() wrapper called in indexFile() after
chunkMarkdown(), before any embedding API call
- Fixes: session files with large JSONL entries could produce chunks
exceeding text-embedding-3-small's 8192 token limit
Tests: 2 new colocated tests in manager.embedding-token-limit.test.ts
- Verifies oversized ASCII chunks are split to <=8192 bytes each
- Verifies multibyte (emoji) content batching respects byte limits
* fix: make embedding token limit provider-aware
- Add optional maxInputTokens to EmbeddingProvider interface
- Each provider (openai, gemini, voyage) reports its own limit
- Known-limits map as fallback: openai 8192, gemini 2048, voyage 32K
- Resolution: provider field > known map > default 8192
- Backward compatible: local/llama uses fallback
* fix: enforce embedding input size limits (#13455) (thanks @rodrigouroz)
---------
Co-authored-by: Tak Hoffman <781889+Takhoffman@users.noreply.github.com>
* fix: remap session JSONL chunk line numbers to original source positions
buildSessionEntry() flattens JSONL messages into plain text before
chunkMarkdown() assigns line numbers. The stored startLine/endLine
values therefore reference positions in the flattened text, not the
original JSONL file.
- Add lineMap to SessionFileEntry tracking which JSONL line each
extracted message came from
- Add remapChunkLines() to translate chunk positions back to original
JSONL lines after chunking
- Guard remap with source === "sessions" to prevent misapplication
- Include lineMap in content hash so existing sessions get re-indexed
Fixes#12044
* memory: dedupe session JSONL parsing
---------
Co-authored-by: Tak Hoffman <781889+Takhoffman@users.noreply.github.com>
* feat(memory): add native Voyage AI embedding support with batching
Cherry-picked from PR #2519, resolved conflict in memory-search.ts
(hasRemote -> hasRemoteConfig rename + added voyage provider)
* fix(memory): optimize voyage batch memory usage with streaming and deduplicate code
Cherry-picked from PR #2519. Fixed lint error: changed this.runWithConcurrency
to use imported runWithConcurrency function after extraction to internal.ts
Add a `paths` option to `memorySearch` config, allowing users to
explicitly specify additional directories or files to include in
memory search.
Follow-up to #2961 as suggested by @gumadeiras — instead of auto-following
symlinks (which has security implications), users can now explicitly
declare additional search paths.
- Add `memorySearch.paths` config option (array of strings)
- Paths can be absolute or relative (resolved from workspace)
- Directories are recursively scanned for `.md` files
- Single `.md` files can also be specified
- Paths from defaults and agent overrides are merged
- Added 4 test cases for listMemoryFiles
