* fix: enforce embedding model token limit to prevent 8192 overflow
- Replace EMBEDDING_APPROX_CHARS_PER_TOKEN=1 with UTF-8 byte length
estimation (safe upper bound for tokenizer output)
- Add EMBEDDING_MODEL_MAX_TOKENS=8192 hard cap
- Add splitChunkToTokenLimit() that binary-searches for the largest
safe split point, with surrogate pair handling
- Add enforceChunkTokenLimit() wrapper called in indexFile() after
chunkMarkdown(), before any embedding API call
- Fixes: session files with large JSONL entries could produce chunks
exceeding text-embedding-3-small's 8192 token limit
Tests: 2 new colocated tests in manager.embedding-token-limit.test.ts
- Verifies oversized ASCII chunks are split to <=8192 bytes each
- Verifies multibyte (emoji) content batching respects byte limits
* fix: make embedding token limit provider-aware
- Add optional maxInputTokens to EmbeddingProvider interface
- Each provider (openai, gemini, voyage) reports its own limit
- Known-limits map as fallback: openai 8192, gemini 2048, voyage 32K
- Resolution: provider field > known map > default 8192
- Backward compatible: local/llama uses fallback
* fix: enforce embedding input size limits (#13455) (thanks @rodrigouroz)
---------
Co-authored-by: Tak Hoffman <781889+Takhoffman@users.noreply.github.com>
* fix: remap session JSONL chunk line numbers to original source positions
buildSessionEntry() flattens JSONL messages into plain text before
chunkMarkdown() assigns line numbers. The stored startLine/endLine
values therefore reference positions in the flattened text, not the
original JSONL file.
- Add lineMap to SessionFileEntry tracking which JSONL line each
extracted message came from
- Add remapChunkLines() to translate chunk positions back to original
JSONL lines after chunking
- Guard remap with source === "sessions" to prevent misapplication
- Include lineMap in content hash so existing sessions get re-indexed
Fixes#12044
* memory: dedupe session JSONL parsing
---------
Co-authored-by: Tak Hoffman <781889+Takhoffman@users.noreply.github.com>
* refactor: consolidate duplicate utility functions
- Add escapeRegExp to src/utils.ts and remove 10 local duplicates
- Rename bash-tools clampNumber to clampWithDefault (different signature)
- Centralize formatError calls to use formatErrorMessage from infra/errors.ts
- Re-export formatErrorMessage from cli/cli-utils.ts to preserve API
* refactor: consolidate remaining escapeRegExp duplicates
* refactor: consolidate sleep, stripAnsi, and clamp duplicates
* Memory/QMD: symlink default model cache into custom XDG_CACHE_HOME
QmdMemoryManager overrides XDG_CACHE_HOME to isolate the qmd index
per-agent, but this also moves where qmd looks for its ML models
(~2.1GB). Since models are installed at the default location
(~/.cache/qmd/models/), every qmd invocation would attempt to
re-download them from HuggingFace and time out.
Fix: on initialization, symlink ~/.cache/qmd/models/ into the custom
XDG_CACHE_HOME path so the index stays isolated per-agent while the
shared models are reused. The symlink is only created when the default
models directory exists and the target path does not already exist.
Includes tests for the three key scenarios: symlink creation, existing
directory preservation, and graceful skip when no default models exist.
* Memory/QMD: skip model symlink warning on ENOENT
* test: stabilize warning-filter visibility assertion (#12114) (thanks @tyler6204)
* fix: add changelog entry for QMD cache reuse (#12114) (thanks @tyler6204)
* fix: handle plain context-overflow strings in compaction detection (#12114) (thanks @tyler6204)
* fix: use .js extension for ESM imports of RoutePeerKind
The imports incorrectly used .ts extension which doesn't resolve
with moduleResolution: NodeNext. Changed to .js and added 'type'
import modifier.
* fix tsconfig
* refactor: unify peer kind to ChatType, rename dm to direct
- Replace RoutePeerKind with ChatType throughout codebase
- Change 'dm' literal values to 'direct' in routing/session keys
- Keep backward compat: normalizeChatType accepts 'dm' -> 'direct'
- Add ChatType export to plugin-sdk, deprecate RoutePeerKind
- Update session key parsing to accept both 'dm' and 'direct' markers
- Update all channel monitors and extensions to use ChatType
BREAKING CHANGE: Session keys now use 'direct' instead of 'dm'.
Existing 'dm' keys still work via backward compat layer.
* fix tests
* test: update session key expectations for dmdirect migration
- Fix test expectations to expect :direct: in generated output
- Add explicit backward compat test for normalizeChatType('dm')
- Keep input test data with :dm: keys to verify backward compat
* fix: accept legacy 'dm' in session key parsing for backward compat
getDmHistoryLimitFromSessionKey now accepts both :dm: and :direct:
to ensure old session keys continue to work correctly.
* test: add explicit backward compat tests for dmdirect migration
- session-key.test.ts: verify both :dm: and :direct: keys are valid
- getDmHistoryLimitFromSessionKey: verify both formats work
* feat: backward compat for resetByType.dm config key
* test: skip unix-path Nix tests on Windows
* Tests: harden flake hotspots and consolidate provider-auth suites
* Tests: restore env vars by deleting missing snapshot values
* Tests: use real newline in memory summary filter case
* Tests(memory): use fake timers for qmd timeout coverage
* Changelog: add tests hardening entry for #11598
* fix(memory): add input_type to Voyage AI embeddings for improved retrieval
Voyage AI recommends passing input_type='document' when indexing and
input_type='query' when searching. This improves retrieval quality by
optimising the embedding space for each direction.
Changes:
- embedQuery now passes input_type: 'query'
- embedBatch now passes input_type: 'document'
- Batch API request_params includes input_type: 'document'
- Tests updated to verify input_type is passed correctly
* Changelog: note Voyage embeddings input_type fix (#10818) (thanks @mcinteerj)
---------
Co-authored-by: Tak Hoffman <781889+Takhoffman@users.noreply.github.com>
* feat(memory): add native Voyage AI embedding support with batching
Cherry-picked from PR #2519, resolved conflict in memory-search.ts
(hasRemote -> hasRemoteConfig rename + added voyage provider)
* fix(memory): optimize voyage batch memory usage with streaming and deduplicate code
Cherry-picked from PR #2519. Fixed lint error: changed this.runWithConcurrency
to use imported runWithConcurrency function after extraction to internal.ts
