634 lines
17 KiB
Markdown
634 lines
17 KiB
Markdown
# LLM Agent Documentation Guide
|
|
## Best Practices for Code Documentation Optimized for AI Coding Assistants
|
|
|
|
### 🎯 Purpose
|
|
This guide outlines best practices for documenting code in a way that maximizes LLM coding agent understanding, evaluation accuracy, and development efficiency.
|
|
|
|
---
|
|
|
|
## 📋 Documentation Structure for LLM Agents
|
|
|
|
### 1. **Hierarchical Information Architecture**
|
|
|
|
#### Level 1: Project Overview (README.md)
|
|
- **Purpose**: High-level system understanding
|
|
- **Content**: What the system does, core technologies, architecture diagram
|
|
- **LLM Benefits**: Quick context establishment, technology stack identification
|
|
|
|
#### Level 2: Architecture Documentation
|
|
- **Purpose**: System design and component relationships
|
|
- **Content**: Detailed architecture, data flow, service interactions
|
|
- **LLM Benefits**: Understanding component dependencies and integration points
|
|
|
|
#### Level 3: Service-Level Documentation
|
|
- **Purpose**: Individual service functionality and APIs
|
|
- **Content**: Service purpose, methods, interfaces, error handling
|
|
- **LLM Benefits**: Precise understanding of service capabilities and constraints
|
|
|
|
#### Level 4: Code-Level Documentation
|
|
- **Purpose**: Implementation details and business logic
|
|
- **Content**: Function documentation, type definitions, algorithm explanations
|
|
- **LLM Benefits**: Detailed implementation understanding for modifications
|
|
|
|
---
|
|
|
|
## 🔧 Best Practices for LLM-Optimized Documentation
|
|
|
|
### 1. **Clear Information Hierarchy**
|
|
|
|
#### Use Consistent Section Headers
|
|
```markdown
|
|
## 🎯 Purpose
|
|
## 🏗️ Architecture
|
|
## 🔧 Implementation
|
|
## 📊 Data Flow
|
|
## 🚨 Error Handling
|
|
## 🧪 Testing
|
|
## 📚 References
|
|
```
|
|
|
|
#### Emoji-Based Visual Organization
|
|
- 🎯 Purpose/Goals
|
|
- 🏗️ Architecture/Structure
|
|
- 🔧 Implementation/Code
|
|
- 📊 Data/Flow
|
|
- 🚨 Errors/Issues
|
|
- 🧪 Testing/Validation
|
|
- 📚 References/Links
|
|
|
|
### 2. **Structured Code Comments**
|
|
|
|
#### Function Documentation Template
|
|
```typescript
|
|
/**
|
|
* @purpose Brief description of what this function does
|
|
* @context When/why this function is called
|
|
* @inputs What parameters it expects and their types
|
|
* @outputs What it returns and the format
|
|
* @dependencies What other services/functions it depends on
|
|
* @errors What errors it can throw and when
|
|
* @example Usage example with sample data
|
|
* @complexity Time/space complexity if relevant
|
|
*/
|
|
```
|
|
|
|
#### Service Documentation Template
|
|
```typescript
|
|
/**
|
|
* @service ServiceName
|
|
* @purpose High-level purpose of this service
|
|
* @responsibilities List of main responsibilities
|
|
* @dependencies External services and internal dependencies
|
|
* @interfaces Main public methods and their purposes
|
|
* @configuration Environment variables and settings
|
|
* @errorHandling How errors are handled and reported
|
|
* @performance Expected performance characteristics
|
|
*/
|
|
```
|
|
|
|
### 3. **Context-Rich Descriptions**
|
|
|
|
#### Instead of:
|
|
```typescript
|
|
// Process document
|
|
function processDocument(doc) { ... }
|
|
```
|
|
|
|
#### Use:
|
|
```typescript
|
|
/**
|
|
* @purpose Processes CIM documents through the AI analysis pipeline
|
|
* @context Called when a user uploads a PDF document for analysis
|
|
* @workflow 1. Extract text via Document AI, 2. Chunk content, 3. Generate embeddings, 4. Run LLM analysis, 5. Create PDF report
|
|
* @inputs Document object with file metadata and user context
|
|
* @outputs Structured analysis data and PDF report URL
|
|
* @dependencies Google Document AI, Claude AI, Supabase, Google Cloud Storage
|
|
*/
|
|
function processDocument(doc: DocumentInput): Promise<ProcessingResult> { ... }
|
|
```
|
|
|
|
---
|
|
|
|
## 📊 Data Flow Documentation
|
|
|
|
### 1. **Visual Flow Diagrams**
|
|
```mermaid
|
|
graph TD
|
|
A[User Upload] --> B[Get Signed URL]
|
|
B --> C[Upload to GCS]
|
|
C --> D[Confirm Upload]
|
|
D --> E[Start Processing]
|
|
E --> F[Document AI Extraction]
|
|
F --> G[Semantic Chunking]
|
|
G --> H[Vector Embedding]
|
|
H --> I[LLM Analysis]
|
|
I --> J[PDF Generation]
|
|
J --> K[Store Results]
|
|
K --> L[Notify User]
|
|
```
|
|
|
|
### 2. **Step-by-Step Process Documentation**
|
|
```markdown
|
|
## Document Processing Pipeline
|
|
|
|
### Step 1: File Upload
|
|
- **Trigger**: User selects PDF file
|
|
- **Action**: Generate signed URL from Google Cloud Storage
|
|
- **Output**: Secure upload URL with expiration
|
|
- **Error Handling**: Retry on URL generation failure
|
|
|
|
### Step 2: Text Extraction
|
|
- **Trigger**: File upload confirmation
|
|
- **Action**: Send PDF to Google Document AI
|
|
- **Output**: Extracted text with confidence scores
|
|
- **Error Handling**: Fallback to OCR if extraction fails
|
|
```
|
|
|
|
---
|
|
|
|
## 🔍 Error Handling Documentation
|
|
|
|
### 1. **Error Classification System**
|
|
```typescript
|
|
/**
|
|
* @errorType VALIDATION_ERROR
|
|
* @description Input validation failures
|
|
* @recoverable true
|
|
* @retryStrategy none
|
|
* @userMessage "Please check your input and try again"
|
|
*/
|
|
|
|
/**
|
|
* @errorType PROCESSING_ERROR
|
|
* @description AI processing failures
|
|
* @recoverable true
|
|
* @retryStrategy exponential_backoff
|
|
* @userMessage "Processing failed, please try again"
|
|
*/
|
|
|
|
/**
|
|
* @errorType SYSTEM_ERROR
|
|
* @description Infrastructure failures
|
|
* @recoverable false
|
|
* @retryStrategy none
|
|
* @userMessage "System temporarily unavailable"
|
|
*/
|
|
```
|
|
|
|
### 2. **Error Recovery Documentation**
|
|
```markdown
|
|
## Error Recovery Strategies
|
|
|
|
### LLM API Failures
|
|
1. **Retry Logic**: Up to 3 attempts with exponential backoff
|
|
2. **Model Fallback**: Switch from Claude to GPT-4 if available
|
|
3. **Graceful Degradation**: Return partial results if possible
|
|
4. **User Notification**: Clear error messages with retry options
|
|
|
|
### Database Connection Failures
|
|
1. **Connection Pooling**: Automatic retry with connection pool
|
|
2. **Circuit Breaker**: Prevent cascade failures
|
|
3. **Read Replicas**: Fallback to read replicas for queries
|
|
4. **Caching**: Serve cached data during outages
|
|
```
|
|
|
|
---
|
|
|
|
## 🧪 Testing Documentation
|
|
|
|
### 1. **Test Strategy Documentation**
|
|
```markdown
|
|
## Testing Strategy
|
|
|
|
### Unit Tests
|
|
- **Coverage Target**: >90% for business logic
|
|
- **Focus Areas**: Service methods, utility functions, data transformations
|
|
- **Mock Strategy**: External dependencies (APIs, databases)
|
|
- **Assertion Style**: Behavior-driven assertions
|
|
|
|
### Integration Tests
|
|
- **Coverage Target**: All API endpoints
|
|
- **Focus Areas**: End-to-end workflows, data persistence, external integrations
|
|
- **Test Data**: Realistic CIM documents with known characteristics
|
|
- **Environment**: Isolated test database and storage
|
|
|
|
### Performance Tests
|
|
- **Load Testing**: 10+ concurrent document processing
|
|
- **Memory Testing**: Large document handling (50MB+)
|
|
- **API Testing**: Rate limit compliance and optimization
|
|
- **Cost Testing**: API usage optimization and monitoring
|
|
```
|
|
|
|
### 2. **Test Data Documentation**
|
|
```typescript
|
|
/**
|
|
* @testData sample_cim_document.pdf
|
|
* @description Standard CIM document with typical structure
|
|
* @size 2.5MB
|
|
* @pages 15
|
|
* @sections Financial, Market, Management, Operations
|
|
* @expectedOutput Complete analysis with all sections populated
|
|
*/
|
|
|
|
/**
|
|
* @testData large_cim_document.pdf
|
|
* @description Large CIM document for performance testing
|
|
* @size 25MB
|
|
* @pages 150
|
|
* @sections Comprehensive business analysis
|
|
* @expectedOutput Analysis within 5-minute time limit
|
|
*/
|
|
```
|
|
|
|
---
|
|
|
|
## 📚 API Documentation
|
|
|
|
### 1. **Endpoint Documentation Template**
|
|
```markdown
|
|
## POST /documents/upload-url
|
|
|
|
### Purpose
|
|
Generate a signed URL for secure file upload to Google Cloud Storage.
|
|
|
|
### Request
|
|
```json
|
|
{
|
|
"fileName": "string",
|
|
"fileSize": "number",
|
|
"contentType": "application/pdf"
|
|
}
|
|
```
|
|
|
|
### Response
|
|
```json
|
|
{
|
|
"uploadUrl": "string",
|
|
"expiresAt": "ISO8601",
|
|
"fileId": "UUID"
|
|
}
|
|
```
|
|
|
|
### Error Responses
|
|
- `400 Bad Request`: Invalid file type or size
|
|
- `401 Unauthorized`: Missing or invalid authentication
|
|
- `500 Internal Server Error`: Storage service unavailable
|
|
|
|
### Dependencies
|
|
- Google Cloud Storage
|
|
- Firebase Authentication
|
|
- File validation service
|
|
|
|
### Rate Limits
|
|
- 100 requests per minute per user
|
|
- 1000 requests per hour per user
|
|
```
|
|
|
|
### 2. **Request/Response Examples**
|
|
```typescript
|
|
/**
|
|
* @example Successful Upload URL Generation
|
|
* @request {
|
|
* "fileName": "sample_cim.pdf",
|
|
* "fileSize": 2500000,
|
|
* "contentType": "application/pdf"
|
|
* }
|
|
* @response {
|
|
* "uploadUrl": "https://storage.googleapis.com/...",
|
|
* "expiresAt": "2024-12-20T15:30:00Z",
|
|
* "fileId": "550e8400-e29b-41d4-a716-446655440000"
|
|
* }
|
|
*/
|
|
```
|
|
|
|
---
|
|
|
|
## 🔧 Configuration Documentation
|
|
|
|
### 1. **Environment Variables**
|
|
```markdown
|
|
## Environment Configuration
|
|
|
|
### Required Variables
|
|
- `GOOGLE_CLOUD_PROJECT_ID`: Google Cloud project identifier
|
|
- `GOOGLE_CLOUD_STORAGE_BUCKET`: Storage bucket for documents
|
|
- `ANTHROPIC_API_KEY`: Claude AI API key for document analysis
|
|
- `DATABASE_URL`: Supabase database connection string
|
|
|
|
### Optional Variables
|
|
- `AGENTIC_RAG_ENABLED`: Enable AI processing (default: true)
|
|
- `PROCESSING_STRATEGY`: Processing method (default: optimized_agentic_rag)
|
|
- `LLM_MODEL`: AI model selection (default: claude-3-opus-20240229)
|
|
- `MAX_FILE_SIZE`: Maximum file size in bytes (default: 52428800)
|
|
|
|
### Development Variables
|
|
- `NODE_ENV`: Environment mode (development/production)
|
|
- `LOG_LEVEL`: Logging verbosity (debug/info/warn/error)
|
|
- `ENABLE_METRICS`: Enable performance monitoring (default: true)
|
|
```
|
|
|
|
### 2. **Service Configuration**
|
|
```typescript
|
|
/**
|
|
* @configuration LLM Service Configuration
|
|
* @purpose Configure AI model behavior and performance
|
|
* @settings {
|
|
* "model": "claude-3-opus-20240229",
|
|
* "maxTokens": 4000,
|
|
* "temperature": 0.1,
|
|
* "timeoutMs": 60000,
|
|
* "retryAttempts": 3,
|
|
* "retryDelayMs": 1000
|
|
* }
|
|
* @constraints {
|
|
* "maxTokens": "1000-8000",
|
|
* "temperature": "0.0-1.0",
|
|
* "timeoutMs": "30000-300000"
|
|
* }
|
|
*/
|
|
```
|
|
|
|
---
|
|
|
|
## 📊 Performance Documentation
|
|
|
|
### 1. **Performance Characteristics**
|
|
```markdown
|
|
## Performance Benchmarks
|
|
|
|
### Document Processing Times
|
|
- **Small Documents** (<5MB): 30-60 seconds
|
|
- **Medium Documents** (5-15MB): 1-3 minutes
|
|
- **Large Documents** (15-50MB): 3-5 minutes
|
|
|
|
### Resource Usage
|
|
- **Memory**: 50-150MB per processing session
|
|
- **CPU**: Moderate usage during AI processing
|
|
- **Network**: 10-50 API calls per document
|
|
- **Storage**: Temporary files cleaned up automatically
|
|
|
|
### Scalability Limits
|
|
- **Concurrent Processing**: 5 documents simultaneously
|
|
- **Daily Volume**: 1000 documents per day
|
|
- **File Size Limit**: 50MB per document
|
|
- **API Rate Limits**: 1000 requests per 15 minutes
|
|
```
|
|
|
|
### 2. **Optimization Strategies**
|
|
```markdown
|
|
## Performance Optimizations
|
|
|
|
### Memory Management
|
|
1. **Batch Processing**: Process chunks in batches of 10
|
|
2. **Garbage Collection**: Automatic cleanup of temporary data
|
|
3. **Connection Pooling**: Reuse database connections
|
|
4. **Streaming**: Stream large files instead of loading entirely
|
|
|
|
### API Optimization
|
|
1. **Rate Limiting**: Respect API quotas and limits
|
|
2. **Caching**: Cache frequently accessed data
|
|
3. **Model Selection**: Use appropriate models for task complexity
|
|
4. **Parallel Processing**: Execute independent operations concurrently
|
|
```
|
|
|
|
---
|
|
|
|
## 🔍 Debugging Documentation
|
|
|
|
### 1. **Logging Strategy**
|
|
```typescript
|
|
/**
|
|
* @logging Structured Logging Configuration
|
|
* @levels {
|
|
* "debug": "Detailed execution flow",
|
|
* "info": "Important business events",
|
|
* "warn": "Potential issues",
|
|
* "error": "System failures"
|
|
* }
|
|
* @correlation Correlation IDs for request tracking
|
|
* @context User ID, session ID, document ID
|
|
* @format JSON structured logging
|
|
*/
|
|
```
|
|
|
|
### 2. **Debug Tools and Commands**
|
|
```markdown
|
|
## Debugging Tools
|
|
|
|
### Log Analysis
|
|
```bash
|
|
# View recent errors
|
|
grep "ERROR" logs/app.log | tail -20
|
|
|
|
# Track specific request
|
|
grep "correlation_id:abc123" logs/app.log
|
|
|
|
# Monitor processing times
|
|
grep "processing_time" logs/app.log | jq '.processing_time'
|
|
```
|
|
|
|
### Health Checks
|
|
```bash
|
|
# Check service health
|
|
curl http://localhost:5001/health
|
|
|
|
# Check database connectivity
|
|
curl http://localhost:5001/health/database
|
|
|
|
# Check external services
|
|
curl http://localhost:5001/health/external
|
|
```
|
|
```
|
|
|
|
---
|
|
|
|
## 📈 Monitoring Documentation
|
|
|
|
### 1. **Key Metrics**
|
|
```markdown
|
|
## Monitoring Metrics
|
|
|
|
### Business Metrics
|
|
- **Documents Processed**: Total documents processed per day
|
|
- **Success Rate**: Percentage of successful processing
|
|
- **Processing Time**: Average time per document
|
|
- **User Activity**: Active users and session duration
|
|
|
|
### Technical Metrics
|
|
- **API Response Time**: Endpoint response times
|
|
- **Error Rate**: Percentage of failed requests
|
|
- **Memory Usage**: Application memory consumption
|
|
- **Database Performance**: Query times and connection usage
|
|
|
|
### Cost Metrics
|
|
- **API Costs**: LLM API usage costs
|
|
- **Storage Costs**: Google Cloud Storage usage
|
|
- **Compute Costs**: Server resource usage
|
|
- **Bandwidth Costs**: Data transfer costs
|
|
```
|
|
|
|
### 2. **Alert Configuration**
|
|
```markdown
|
|
## Alert Rules
|
|
|
|
### Critical Alerts
|
|
- **High Error Rate**: >5% error rate for 5 minutes
|
|
- **Service Down**: Health check failures
|
|
- **High Latency**: >30 second response times
|
|
- **Memory Issues**: >80% memory usage
|
|
|
|
### Warning Alerts
|
|
- **Increased Error Rate**: >2% error rate for 10 minutes
|
|
- **Performance Degradation**: >15 second response times
|
|
- **High API Usage**: >80% of rate limits
|
|
- **Storage Issues**: >90% storage usage
|
|
```
|
|
|
|
---
|
|
|
|
## 🚀 Deployment Documentation
|
|
|
|
### 1. **Deployment Process**
|
|
```markdown
|
|
## Deployment Process
|
|
|
|
### Pre-deployment Checklist
|
|
- [ ] All tests passing
|
|
- [ ] Documentation updated
|
|
- [ ] Environment variables configured
|
|
- [ ] Database migrations ready
|
|
- [ ] External services configured
|
|
|
|
### Deployment Steps
|
|
1. **Build**: Create production build
|
|
2. **Test**: Run integration tests
|
|
3. **Deploy**: Deploy to staging environment
|
|
4. **Validate**: Verify functionality
|
|
5. **Promote**: Deploy to production
|
|
6. **Monitor**: Watch for issues
|
|
|
|
### Rollback Plan
|
|
1. **Detect Issue**: Monitor error rates and performance
|
|
2. **Assess Impact**: Determine severity and scope
|
|
3. **Execute Rollback**: Revert to previous version
|
|
4. **Verify Recovery**: Confirm system stability
|
|
5. **Investigate**: Root cause analysis
|
|
```
|
|
|
|
### 2. **Environment Management**
|
|
```markdown
|
|
## Environment Configuration
|
|
|
|
### Development Environment
|
|
- **Purpose**: Local development and testing
|
|
- **Database**: Local Supabase instance
|
|
- **Storage**: Development GCS bucket
|
|
- **AI Services**: Test API keys with limits
|
|
|
|
### Staging Environment
|
|
- **Purpose**: Pre-production testing
|
|
- **Database**: Staging Supabase instance
|
|
- **Storage**: Staging GCS bucket
|
|
- **AI Services**: Production API keys with monitoring
|
|
|
|
### Production Environment
|
|
- **Purpose**: Live user service
|
|
- **Database**: Production Supabase instance
|
|
- **Storage**: Production GCS bucket
|
|
- **AI Services**: Production API keys with full monitoring
|
|
```
|
|
|
|
---
|
|
|
|
## 📚 Documentation Maintenance
|
|
|
|
### 1. **Documentation Review Process**
|
|
```markdown
|
|
## Documentation Maintenance
|
|
|
|
### Review Schedule
|
|
- **Weekly**: Update API documentation for new endpoints
|
|
- **Monthly**: Review and update architecture documentation
|
|
- **Quarterly**: Comprehensive documentation audit
|
|
- **Release**: Update all documentation for new features
|
|
|
|
### Quality Checklist
|
|
- [ ] All code examples are current and working
|
|
- [ ] API documentation matches implementation
|
|
- [ ] Configuration examples are accurate
|
|
- [ ] Error handling documentation is complete
|
|
- [ ] Performance metrics are up-to-date
|
|
- [ ] Links and references are valid
|
|
```
|
|
|
|
### 2. **Version Control for Documentation**
|
|
```markdown
|
|
## Documentation Version Control
|
|
|
|
### Branch Strategy
|
|
- **main**: Current production documentation
|
|
- **develop**: Latest development documentation
|
|
- **feature/***: Documentation for new features
|
|
- **release/***: Documentation for specific releases
|
|
|
|
### Change Management
|
|
1. **Propose Changes**: Create documentation issue
|
|
2. **Review Changes**: Peer review of documentation updates
|
|
3. **Test Examples**: Verify all code examples work
|
|
4. **Update References**: Update all related documentation
|
|
5. **Merge Changes**: Merge with approval
|
|
```
|
|
|
|
---
|
|
|
|
## 🎯 LLM Agent Optimization Tips
|
|
|
|
### 1. **Context Provision**
|
|
- Provide complete context for each code section
|
|
- Include business rules and constraints
|
|
- Document assumptions and limitations
|
|
- Explain why certain approaches were chosen
|
|
|
|
### 2. **Example-Rich Documentation**
|
|
- Include realistic examples for all functions
|
|
- Provide before/after examples for complex operations
|
|
- Show error scenarios and recovery
|
|
- Include performance examples
|
|
|
|
### 3. **Structured Information**
|
|
- Use consistent formatting and organization
|
|
- Provide clear hierarchies of information
|
|
- Include cross-references between related sections
|
|
- Use standardized templates for similar content
|
|
|
|
### 4. **Error Scenario Documentation**
|
|
- Document all possible error conditions
|
|
- Provide specific error messages and codes
|
|
- Include recovery procedures for each error type
|
|
- Show debugging steps for common issues
|
|
|
|
---
|
|
|
|
## 📋 Documentation Checklist
|
|
|
|
### For Each New Feature
|
|
- [ ] Update README.md with feature overview
|
|
- [ ] Document API endpoints and examples
|
|
- [ ] Update architecture diagrams if needed
|
|
- [ ] Add configuration documentation
|
|
- [ ] Include error handling scenarios
|
|
- [ ] Add test examples and strategies
|
|
- [ ] Update deployment documentation
|
|
- [ ] Review and update related documentation
|
|
|
|
### For Each Code Change
|
|
- [ ] Update function documentation
|
|
- [ ] Add inline comments for complex logic
|
|
- [ ] Update type definitions if changed
|
|
- [ ] Add examples for new functionality
|
|
- [ ] Update error handling documentation
|
|
- [ ] Verify all links and references
|
|
|
|
---
|
|
|
|
This guide ensures that your documentation is optimized for LLM coding agents, providing them with the context, structure, and examples they need to understand and work with your codebase effectively. |