9c916d12f4
Major release with significant performance improvements and new processing strategy. ## Core Changes - Implemented simple_full_document processing strategy (default) - Full document → LLM approach: 1-2 passes, ~5-6 minutes processing time - Achieved 100% completeness with 2 API calls (down from 5+) - Removed redundant Document AI passes for faster processing ## Financial Data Extraction - Enhanced deterministic financial table parser - Improved FY3/FY2/FY1/LTM identification from varying CIM formats - Automatic merging of parser results with LLM extraction ## Code Quality & Infrastructure - Cleaned up debug logging (removed emoji markers from production code) - Fixed Firebase Secrets configuration (using modern defineSecret approach) - Updated OpenAI API key - Resolved deployment conflicts (secrets vs environment variables) - Added .env files to Firebase ignore list ## Deployment - Firebase Functions v2 deployment successful - All 7 required secrets verified and configured - Function URL: https://api-y56ccs6wva-uc.a.run.app ## Performance Improvements - Processing time: ~5-6 minutes (down from 23+ minutes) - API calls: 1-2 (down from 5+) - Completeness: 100% achievable - LLM Model: claude-3-7-sonnet-latest ## Breaking Changes - Default processing strategy changed to 'simple_full_document' - RAG processor available as alternative strategy 'document_ai_agentic_rag' ## Files Changed - 36 files changed, 5642 insertions(+), 4451 deletions(-) - Removed deprecated documentation files - Cleaned up unused services and models This release represents a major refactoring focused on speed, accuracy, and maintainability.
80 lines
3.1 KiB
Markdown
80 lines
3.1 KiB
Markdown
# Quick Fix Implementation Summary
|
|
|
|
## Problem
|
|
List fields (keyAttractions, potentialRisks, valueCreationLevers, criticalQuestions, missingInformation) were not consistently generating 5-8 numbered items, causing test failures.
|
|
|
|
## Solution Implemented (Phase 1: Quick Fix)
|
|
|
|
### Files Modified
|
|
|
|
1. **backend/src/services/llmService.ts**
|
|
- Added `generateText()` method for simple text completion tasks
|
|
- Line 105-121: New public method wrapping callLLM for quick repairs
|
|
|
|
2. **backend/src/services/optimizedAgenticRAGProcessor.ts**
|
|
- Line 1299-1320: Added list field validation call before returning results
|
|
- Line 2136-2307: Added 3 new methods:
|
|
- `validateAndRepairListFields()` - Validates all list fields have 5-8 items
|
|
- `repairListField()` - Uses LLM to fix lists with wrong item count
|
|
- `getNestedField()` / `setNestedField()` - Utility methods for nested object access
|
|
|
|
### How It Works
|
|
|
|
1. **After multi-pass extraction completes**, the code now validates each list field
|
|
2. **If a list has < 5 or > 8 items**, it automatically repairs it:
|
|
- For lists < 5 items: Asks LLM to expand to 6 items
|
|
- For lists > 8 items: Asks LLM to consolidate to 7 items
|
|
3. **Uses document context** to ensure new items are relevant
|
|
4. **Lower temperature** (0.3) for more consistent output
|
|
5. **Tracks repair API calls** separately
|
|
|
|
### Test Status
|
|
- ✅ Build successful
|
|
- 🔄 Running pipeline test to validate fix
|
|
- Expected: All tests should pass with list validation
|
|
|
|
## Next Steps (Phase 2: Proper Fix - This Week)
|
|
|
|
### Implement Tool Use API (Proper Solution)
|
|
|
|
Create `/backend/src/services/llmStructuredExtraction.ts`:
|
|
- Use Anthropic's tool use API with JSON schema
|
|
- Define strict schemas with minItems/maxItems constraints
|
|
- Claude will internally retry until schema compliance
|
|
- More reliable than post-processing repair
|
|
|
|
**Benefits:**
|
|
- 100% schema compliance (Claude retries internally)
|
|
- No post-processing repair needed
|
|
- Lower overall API costs (fewer retry attempts)
|
|
- Better architectural pattern
|
|
|
|
**Timeline:**
|
|
- Phase 1 (Quick Fix): ✅ Complete (2 hours)
|
|
- Phase 2 (Tool Use): 📅 Implement this week (6 hours)
|
|
- Total investment: 8 hours
|
|
|
|
## Additional Improvements for Later
|
|
|
|
### 1. Semantic Chunking (Week 2)
|
|
- Replace fixed 4000-char chunks with semantic chunking
|
|
- Respect document structure (don't break tables/sections)
|
|
- Use 800-char chunks with 200-char overlap
|
|
- **Expected improvement**: 12-30% better retrieval accuracy
|
|
|
|
### 2. Hybrid Retrieval (Week 3)
|
|
- Add BM25/keyword search alongside vector similarity
|
|
- Implement cross-encoder reranking
|
|
- Consider HyDE (Hypothetical Document Embeddings)
|
|
- **Expected improvement**: 15-25% better retrieval accuracy
|
|
|
|
### 3. Fix RAG Search Issue
|
|
- Current logs show `avgSimilarity: 0`
|
|
- Implement HyDE or improve query embedding strategy
|
|
- **Problem**: Query embeddings don't match document embeddings well
|
|
|
|
## References
|
|
- Claude Tool Use: https://docs.claude.com/en/docs/agents-and-tools/tool-use
|
|
- RAG Chunking: https://community.databricks.com/t5/technical-blog/the-ultimate-guide-to-chunking-strategies
|
|
- Structured Output: https://dev.to/heuperman/how-to-get-consistent-structured-output-from-claude-20o5
|