Commit Graph

96 Commits

Author SHA1 Message Date
admin
400342456f docs(phase-03): complete phase execution and verification 2026-02-24 15:49:41 -05:00
admin
c9edaec8d6 docs(03-01): complete admin API endpoints plan
- 03-01-SUMMARY.md: requireAdminEmail middleware + four admin endpoints
- STATE.md: decisions, session continuity updated
- ROADMAP.md: phase 3 progress updated
- REQUIREMENTS.md: INFR-02, HLTH-01 marked complete

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 15:46:19 -05:00
admin
081c5357c1 docs(03-02): complete analytics instrumentation plan
- Create 03-02-SUMMARY.md with task outcomes and decisions
- Update STATE.md: advance to Phase 3 Plan 2 complete
- Update ROADMAP.md: mark plan progress for phase 3
- Mark ANLY-02 requirement complete in REQUIREMENTS.md

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 15:45:59 -05:00
admin
4169a3731f feat(03-01): create admin routes and mount at /admin
- admin.ts: four endpoints (GET /health, GET /analytics, GET /alerts, POST /alerts/:id/acknowledge)
- Auth chain: addCorrelationId + verifyFirebaseToken + requireAdminEmail (router-level)
- Health: queries all four service names matching healthProbeService output
- Analytics: validates range format (/^\d+[hd]$/) then delegates to getAnalyticsSummary()
- Alerts: findActive() returns all active alerts; acknowledge returns 404 on not-found
- Response envelope: { success, data, correlationId } matching codebase pattern
- index.ts: mounts admin router at /admin alongside existing routes
2026-02-24 15:44:41 -05:00
admin
dabd4a5ecf feat(03-02): instrument processJob with fire-and-forget analytics events
- Add import for recordProcessingEvent from analyticsService
- Emit upload_started after markAsProcessing (job processing start)
- Emit completed with duration_ms after markAsCompleted (job success)
- Emit failed with duration_ms and error_message in catch block (job failure)
- All calls are void/fire-and-forget (no await) per PITFALL-6 constraint
- Null-guard on job in catch block prevents runtime errors
2026-02-24 15:44:12 -05:00
admin
301d0bf159 feat(03-01): add requireAdminEmail middleware and getAnalyticsSummary function
- requireAdmin.ts: returns 404 (not 403) for non-admin users per locked decision
- Reads env vars inside function body to avoid Firebase Secrets timing issue
- Fails closed if neither ADMIN_EMAIL nor EMAIL_WEEKLY_RECIPIENT configured
- analyticsService.ts: adds AnalyticsSummary interface and getAnalyticsSummary()
- Uses getPostgresPool() for aggregate SQL (COUNT/AVG not supported by Supabase JS)
- Parameterized interval with $1::interval cast for PostgreSQL compatibility
2026-02-24 15:43:53 -05:00
admin
ad464cb633 docs(03): create phase plan 2026-02-24 15:37:15 -05:00
admin
4807e85610 docs(03): research phase domain 2026-02-24 15:33:12 -05:00
admin
6c8af6d35f docs(03): capture phase context 2026-02-24 15:27:36 -05:00
admin
a29d449e58 docs(phase-02): complete phase execution and verification 2026-02-24 14:40:39 -05:00
admin
e4a7699938 docs(02-04): complete runHealthProbes + runRetentionCleanup plan
- Phase 2 plan 4 complete — two scheduled Cloud Function exports added
- SUMMARY.md created with decisions, deviations, and phase readiness notes
- STATE.md updated: phase 2 complete, plan counter at 4/4
- ROADMAP.md updated: phase 2 all 4 plans complete
- Requirements HLTH-03 and INFR-03 marked complete

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-24 14:37:00 -05:00
admin
1f9df623b4 feat(02-04): add runHealthProbes scheduled Cloud Function export
- Runs every 5 minutes, separate from processDocumentJobs (PITFALL-2, HLTH-03)
- Calls healthProbeService.runAllProbes() then alertService.evaluateAndAlert()
- Uses dynamic import() pattern matching processDocumentJobs
- retryCount: 0 — probes re-run in 5 minutes, no retry needed
- Lists all required secrets: anthropicApiKey, openaiApiKey, databaseUrl, supabaseServiceKey, supabaseAnonKey
2026-02-24 14:35:01 -05:00
admin
0acacd1269 docs(02-03): complete alertService plan
- SUMMARY.md with deduplication, lazy transporter, and email decisions
- STATE.md: plan 3/4, 50% progress, decisions recorded
- ROADMAP.md: phase 02 updated (3/4 summaries)
- REQUIREMENTS.md: ALRT-01, ALRT-02, ALRT-04 marked complete
2026-02-24 14:33:05 -05:00
admin
4b5afe2132 test(02-03): add alertService unit tests (8 passing)
- healthy probes trigger no alert logic
- down probe creates alert_events row and sends email
- degraded probe uses alert_type service_degraded
- deduplication suppresses row creation and email within cooldown
- recipient read from process.env.EMAIL_WEEKLY_RECIPIENT
- missing recipient skips email but still creates alert row
- email failure does not throw (non-throwing pipeline)
- multiple probes processed independently
- vi.mock factories use inline vi.fn() only (no TDZ hoisting errors)
2026-02-24 14:30:16 -05:00
admin
91f609cf92 feat(02-03): create alertService with deduplication and email
- evaluateAndAlert() iterates ProbeResults and skips healthy probes
- Maps 'down' -> 'service_down', 'degraded' -> 'service_degraded'
- Deduplication via AlertEventModel.findRecentByService with configurable cooldown
- Creates alert_events row before sending email (suppression skips both)
- Recipient read from process.env.EMAIL_WEEKLY_RECIPIENT (never hardcoded)
- createTransporter() called inside function scope (Firebase Secret timing fix)
- Email failures caught and logged, never re-thrown
2026-02-24 14:28:20 -05:00
admin
520b6b1fe2 docs(02-01): complete analytics service plan
- Created 02-01-SUMMARY.md with full execution documentation
- Updated ROADMAP.md with phase 2 plan progress (2 of 4 plans with summaries)
- Marked requirements ANLY-01 and ANLY-03 complete in REQUIREMENTS.md
- Added 02-01 key decisions to STATE.md
2026-02-24 14:26:41 -05:00
admin
018fb7a24c docs(02-02): complete health probe service plan
- SUMMARY.md: 4 probers, 9 unit tests, nodemailer installed
- STATE.md: advanced to phase 2 plan 2, added 5 key decisions
- ROADMAP.md: updated phase 2 progress (2/4 summaries)
- REQUIREMENTS.md: marked HLTH-02 and HLTH-04 complete
2026-02-24 14:25:45 -05:00
admin
cf30811b97 test(02-01): add analyticsService unit tests
- 6 tests: recordProcessingEvent (4 tests) + deleteProcessingEventsOlderThan (2 tests)
- Verifies fire-and-forget: void return (undefined), no throw on Supabase failure
- Verifies error logging on Supabase failure without rethrowing
- Verifies null coalescing for optional fields (duration_ms, error_message, stage)
- Verifies cutoff date math (~30 days ago) and row count return
- Uses makeSupabaseChain() pattern from Phase 1 model tests
2026-02-24 14:23:42 -05:00
admin
a8ba884043 test(02-02): add healthProbeService unit tests
- 9 tests covering all 4 probers and orchestrator
- Verifies all probes return 4 ProbeResults with correct service names
- Verifies results persisted via HealthCheckModel.create 4 times
- Verifies one probe failure does not abort other probes
- Verifies LLM probe 429 returns degraded not down
- Verifies Supabase probe uses getPostgresPool (not PostgREST)
- Verifies Firebase Auth distinguishes expected vs unexpected errors
- Verifies latency_ms is a non-negative number
- Verifies HealthCheckModel.create failure is isolated
2026-02-24 14:23:35 -05:00
admin
41298262d6 feat(02-02): install nodemailer and create healthProbeService
- Install nodemailer + @types/nodemailer (needed by Plan 03)
- Create healthProbeService.ts with 4 probers: document_ai, llm_api, supabase, firebase_auth
- Each probe makes a real authenticated API call
- Each probe returns structured ProbeResult with status, latency_ms, error_message
- LLM probe uses cheapest model (claude-haiku-4-5) with max_tokens 5
- Supabase probe uses getPostgresPool().query('SELECT 1') not PostgREST
- Firebase Auth probe distinguishes expected vs unexpected errors
- runAllProbes orchestrator uses Promise.allSettled for fault isolation
- Results persisted via HealthCheckModel.create() after each probe
2026-02-24 14:22:38 -05:00
admin
ef88541511 feat(02-01): create analytics migration and analyticsService
- Migration 013: document_processing_events table with CHECK constraint on event_type
- Indexes on created_at (retention queries) and document_id (per-doc history)
- RLS enabled following migration 012 pattern
- analyticsService.recordProcessingEvent(): void return (fire-and-forget, never throws)
- analyticsService.deleteProcessingEventsOlderThan(): Promise<number> (retention delete)
- getSupabaseServiceClient() called per-method, never cached at module level
2026-02-24 14:22:05 -05:00
admin
73f8d8271e docs(02-backend-services): create phase plan 2026-02-24 14:14:54 -05:00
admin
fcb3987c56 docs(02): research phase backend services domain 2026-02-24 14:08:02 -05:00
admin
13454fe860 docs(phase-01): complete phase execution and verification 2026-02-24 14:00:10 -05:00
admin
20e3bec887 docs(01-02): complete model unit tests plan
- SUMMARY: 33 tests for HealthCheckModel (14) and AlertEventModel (19)
- STATE: Advanced to plan 02 complete, added 3 mock pattern decisions
- ROADMAP: Phase 1 Data Foundation marked complete (2/2 plans)
2026-02-24 12:22:12 -05:00
admin
e630ff744a test(01-02): add AlertEventModel unit tests
- Tests cover create (valid, default status active, explicit status, with details JSONB)
- Input validation (empty name, invalid alert_type, invalid status)
- Supabase error handling (throws descriptive message)
- findActive (all active, filtered by service, empty array)
- acknowledge (sets status+timestamp, throws on not found via PGRST116)
- resolve (sets status+timestamp, throws on not found)
- findRecentByService (found within window, null when absent — deduplication use case)
- deleteOlderThan (cutoff date, returns count)
- All 41 tests pass (14 HealthCheck + 19 AlertEvent + 8 existing)
2026-02-24 12:22:12 -05:00
admin
61c2b9fc73 test(01-02): add HealthCheckModel unit tests
- Tests cover create (valid, minimal, probe_details), input validation (empty name, invalid status)
- Supabase error handling (throws, logs error)
- findLatestByService (found, not found — PGRST116 null return)
- findAll (default limit 100, filtered by service, custom limit)
- deleteOlderThan (cutoff date calculation, returns count)
- Establishes Supabase chainable mock pattern for future model tests
- Mocks getSupabaseServiceClient confirming INFR-04 compliance
2026-02-24 12:22:12 -05:00
admin
1e4bc99fd1 feat(01-01): add HealthCheckModel and AlertEventModel with barrel exports
- HealthCheckModel: typed interfaces (ServiceHealthCheck, CreateHealthCheckData), static methods create/findLatestByService/findAll/deleteOlderThan, input validation, getSupabaseServiceClient() per-method, Winston logging
- AlertEventModel: typed interfaces (AlertEvent, CreateAlertEventData), static methods create/findActive/acknowledge/resolve/findRecentByService/deleteOlderThan, input validation, PGRST116 handled as null, Winston logging
- Update models/index.ts to re-export both models and their types
- Strict TypeScript: Record<string, unknown> for JSONB fields, no any types
2026-02-24 12:22:12 -05:00
admin
94d1c0adae feat(01-01): create monitoring tables migration
- Add service_health_checks table with status CHECK constraint, JSONB probe_details, checked_at column
- Add alert_events table with alert_type and status CHECK constraints, lifecycle timestamps
- Add created_at indexes on both tables (INFR-01 requirement)
- Add composite indexes for common query patterns
- Enable RLS on both tables (service role bypasses RLS per Supabase pattern)
2026-02-24 12:22:12 -05:00
admin
fec5d0319e Merge branch 'upgrade/firebase-functions-v7-nodejs22' into production
Upgrade Firebase Functions runtime to Node.js 22 and firebase-functions v7.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 11:44:02 -05:00
admin
e606027ddc docs(01-01): complete monitoring data foundation plan
- Add 01-01-SUMMARY.md documenting migration + model layer delivery
- Update STATE.md: plan 1 complete, decisions recorded, session updated
- Update ROADMAP.md: phase 1 progress 1/2 plans complete
- Mark INFR-01 and INFR-04 requirements complete in REQUIREMENTS.md
2026-02-24 11:42:53 -05:00
admin
9a5ff52d12 chore: upgrade Firebase Functions to Node.js 22 and firebase-functions v7
Node.js 20 is being decommissioned 2026-10-30. This upgrades the runtime
to Node.js 22 (LTS), bumps firebase-functions from v6 to v7, removes the
deprecated functions.config() fallback, and aligns the TS target to ES2022.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 11:41:00 -05:00
admin
6429e98f58 docs(01): create phase plan 2026-02-24 11:24:17 -05:00
admin
c480d4b990 docs(01): research phase data foundation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 11:20:24 -05:00
admin
54157fe74d docs(01): capture phase context 2026-02-24 11:15:10 -05:00
admin
fcaf4579e1 docs: create roadmap (4 phases) 2026-02-24 11:08:13 -05:00
admin
503f39bd9c docs: define v1 requirements 2026-02-24 11:05:34 -05:00
admin
f9cc71b959 chore: add project config 2026-02-24 10:52:04 -05:00
admin
972760b957 docs: initialize project 2026-02-24 10:49:52 -05:00
admin
e6e1b1fa6f docs: map existing codebase 2026-02-24 10:28:22 -05:00
admin
9a906763c7 Remove 15 stale planning and analysis docs
These are completed implementation plans, one-time analysis artifacts,
and generic guides that no longer reflect the current codebase.
All useful content is either implemented in code or captured in
TODO_AND_OPTIMIZATIONS.md.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 10:12:23 -05:00
admin
3d01085b10 Fix hardcoded processing strategy in document controller
The confirmUpload and inline processing paths were hardcoded to
'document_ai_agentic_rag', ignoring the config setting. Now reads
from config.processingStrategy so the single-pass processor is
actually used when configured.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 22:37:38 -05:00
admin
5cfb136484 Add single-pass CIM processor: 2 LLM calls, ~2.5 min processing
New processing strategy `single_pass_quality_check` replaces the multi-pass
agentic RAG pipeline (15-25 min) with a streamlined 2-call approach:

1. Full-document LLM extraction (Sonnet) — single call with complete CIM text
2. Delta quality-check (Haiku) — reviews extraction, returns only corrections

Key changes:
- New singlePassProcessor.ts with extraction + quality check flow
- llmService: qualityCheckCIMDocument() with delta-only corrections array
- llmService: improved prompt requiring professional inferences for qualitative
  fields instead of defaulting to "Not specified in CIM"
- Removed deterministic financial parser from single-pass flow (LLM outperforms
  it — parser matched footnotes and narrative text as financials)
- Default strategy changed to single_pass_quality_check
- Completeness scoring with diagnostic logging of empty fields

Tested on 2 real CIMs: 100% completeness, correct financials, ~150s each.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 22:28:45 -05:00
admin
f4bd60ca38 Fix CIM processing pipeline: embeddings, model refs, and timeouts
- Fix invalid model name claude-3-7-sonnet-latest → use config.llm.model
- Increase LLM timeout from 3 min to 6 min for complex CIM analysis
- Improve RAG fallback to use evenly-spaced chunks when keyword matching
  finds too few results (prevents sending tiny fragments to LLM)
- Add model name normalization for Claude 4.x family
- Add googleServiceAccount utility for unified credential resolution
- Add Cloud Run log fetching script
- Update default models to Claude 4.6/4.5 family

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 18:33:31 -05:00
admin
b00700edd7 Note runtime/dependency upgrades in to-do list 2026-02-23 14:50:56 -05:00
admin
9480a3c994 Add acceptance tests and align defaults to Sonnet 4 2026-02-23 14:45:57 -05:00
admin
14d5c360e5 Set up clean Firebase deploy workflow from git source
- Add @google-cloud/functions-framework and ts-node deps to match deployed
- Add .env.bak ignore patterns to firebase.json
- Fix adminService.ts: inline axios client (was importing non-existent module)
- Clean .env to exclude GCP Secret Manager secrets (prevents deploy overlap error)
- Verified: both frontend and backend build and deploy successfully

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-23 13:41:00 -05:00
admin
ecd4b13115 Fix EBITDA margin auto-correction and TypeScript compilation error
- Added auto-correction logic for EBITDA margins when difference >15pp
- Fixed missing closing brace in revenue validation block
- Enhanced margin validation to catch cases like 95% -> 22.3%
2025-11-10 15:53:17 -05:00
admin
59e0938b72 Implement Claude Haiku 3.5 for financial extraction
- Use Haiku 3.5 (claude-3-5-haiku-latest) for financial extraction by default
- Automatically adjust maxTokens to 8192 for Haiku (vs 16000 for Sonnet)
- Add intelligent fallback to Sonnet 4.5 if Haiku validation fails
- Add comprehensive test script for Haiku financial extraction
- Fix TypeScript errors in financial validation logic

Benefits:
- ~50% faster processing (13s vs 26s estimated)
- ~92% cost reduction (--.014 vs --.15 per extraction)
- Maintains accuracy with validation fallback

Tested successfully with Stax Holding Company CIM:
- Correctly extracted FY3=4M, FY2=1M, FY1=6M, LTM=1M
- Processing time: 13.15s
- Cost: --.0138
2025-11-10 14:44:37 -05:00
admin
e1411ec39c Fix financial summary generation issues
- Fix period ordering: Display periods in chronological order (FY3 → FY2 → FY1 → LTM)
- Add missing metrics: Include Gross Profit and Gross Margin rows in summary table
- Enhance financial parser: Improve column alignment validation and logging
- Strengthen LLM prompts: Add better examples, validation checks, and column alignment guidance
- Improve validation: Add cross-period validation, trend checking, and margin consistency checks
- Add test suite: Create comprehensive tests for financial summary workflow

All tests passing. Summary table now correctly displays periods chronologically and includes all required metrics.
2025-11-10 14:00:42 -05:00