diff --git a/.planning/phases/03-api-layer/03-RESEARCH.md b/.planning/phases/03-api-layer/03-RESEARCH.md new file mode 100644 index 0000000..6cd1b29 --- /dev/null +++ b/.planning/phases/03-api-layer/03-RESEARCH.md @@ -0,0 +1,550 @@ +# Phase 3: API Layer - Research + +**Researched:** 2026-02-24 +**Domain:** Express.js admin route construction, Firebase Auth middleware, Supabase analytics queries +**Confidence:** HIGH + +--- + + +## User Constraints (from CONTEXT.md) + +### Locked Decisions + +**Response shape & contracts** +- Analytics endpoint accepts a configurable time range via query param (e.g., `?range=24h`, `?range=7d`) with a sensible default +- Field naming convention: match whatever the existing codebase already uses (camelCase or snake_case) — stay consistent + +**Auth & error behavior** +- Non-admin users receive 404 on admin endpoints — do not reveal that admin routes exist +- Unauthenticated requests: Claude decides whether to return 401 or same 404 based on existing auth middleware patterns + +**Analytics instrumentation** +- Best-effort with logging: emit events asynchronously, log failures, but never let instrumentation errors propagate to processing +- Key milestones only — upload started, processing complete, processing failed (not every pipeline stage) +- Include duration/timing data per event — enables avg processing time metric in the analytics endpoint + +**Endpoint conventions** +- Route prefix: match existing Express app patterns +- Acknowledge semantics: Claude decides (one-way, toggle, or with note — whatever fits best) + +### Claude's Discretion +- Envelope pattern vs direct data for API responses +- Health endpoint detail level (flat status vs nested with last-check times) +- Admin role mechanism (Firebase custom claims vs Supabase role check vs other) +- Unauthenticated request handling (401 vs 404) +- Alert pagination strategy +- Alert filtering support +- Rate limiting on admin endpoints + +### Deferred Ideas (OUT OF SCOPE) + +None — discussion stayed within phase scope + + +--- + + +## Phase Requirements + +| ID | Description | Research Support | +|----|-------------|-----------------| +| INFR-02 | Admin API routes protected by Firebase Auth with admin email check | Firebase Auth `verifyFirebaseToken` middleware exists; need `requireAdmin` layer that checks `req.user.email` against `process.env.EMAIL_WEEKLY_RECIPIENT` (already configured for alerts) or a dedicated `ADMIN_EMAIL` env var | +| HLTH-01 | Admin can view live health status (healthy/degraded/down) for Document AI, Claude/OpenAI, Supabase, and Firebase Auth | `HealthCheckModel.findLatestByService()` already exists; need a query across all four service names or a loop; service names must match what `healthProbeService` writes | +| ANLY-02 | Admin can view processing summary: upload counts, success/failure rates, avg processing time | `document_processing_events` table exists with `event_type`, `duration_ms`, `created_at`; need a Supabase aggregation query grouped by `event_type` over a time window; `recordProcessingEvent()` must be called from `jobProcessorService.processJob()` (not yet called there) | + + +--- + +## Summary + +Phase 3 is entirely additive — it exposes data from Phase 1 and Phase 2 via admin-protected HTTP endpoints, and instruments the existing `jobProcessorService.processJob()` method with fire-and-forget analytics calls. No database schema changes are needed; all tables and models exist. + +The three technical sub-problems are: (1) a two-layer auth middleware — Firebase token verification (existing `verifyFirebaseToken`) plus an admin email check (new, 5-10 lines); (2) three new route handlers reading from `HealthCheckModel`, `AlertEventModel`, and a new `getAnalyticsSummary()` function in `analyticsService`; and (3) inserting `recordProcessingEvent()` calls at three points inside `processJob()` without altering success/failure semantics. + +The codebase is well-factored and consistent: route files live in `backend/src/routes/`, middleware in `backend/src/middleware/`, service functions in `backend/src/services/`. The existing `verifyFirebaseToken` middleware plus a new `requireAdminEmail` middleware compose cleanly onto the new `/admin` router. The existing `{ success: true, data: ..., correlationId: ... }` envelope is the established pattern and should be followed. + +**Primary recommendation:** Add `adminRoutes.ts` to the existing routes directory, mount it at `/admin` in `index.ts`, compose `verifyFirebaseToken` + `requireAdminEmail` as router-level middleware, and wire three handlers to existing model/service methods. Instrument `processJob()` at job-start, completion, and failure using the existing `recordProcessingEvent()` signature. + +--- + +## Standard Stack + +### Core + +| Library | Version | Purpose | Why Standard | +|---------|---------|---------|--------------| +| express | already in use | Router, Request/Response types | Project standard | +| firebase-admin | already in use | Token verification (`verifyIdToken`) | Existing auth layer | +| @supabase/supabase-js | already in use | Database reads via `getSupabaseServiceClient()` | Project data layer | + +### Supporting + +| Library | Version | Purpose | When to Use | +|---------|---------|---------|-------------| +| (none new) | — | All needed libraries already present | No new npm installs required | + +### Alternatives Considered + +| Instead of | Could Use | Tradeoff | +|------------|-----------|----------| +| Email-based admin check | Firebase custom claims | Custom claims require Firebase Admin SDK `setCustomUserClaims()` call — more setup; email check works with zero additional config since `EMAIL_WEEKLY_RECIPIENT` is already defined | +| Email-based admin check | Supabase role column | Cross-system lookup adds latency and a new dependency; email check is synchronous against the already-decoded token | + +**Installation:** No new packages needed. + +--- + +## Architecture Patterns + +### Recommended Project Structure + +``` +backend/src/ +├── routes/ +│ ├── admin.ts # NEW — /admin router with health, analytics, alerts endpoints +│ ├── documents.ts # existing +│ ├── monitoring.ts # existing +│ └── ... +├── middleware/ +│ ├── firebaseAuth.ts # existing — verifyFirebaseToken +│ ├── requireAdmin.ts # NEW — requireAdminEmail middleware (10-15 lines) +│ └── ... +├── services/ +│ ├── analyticsService.ts # extend — add getAnalyticsSummary() query function +│ ├── jobProcessorService.ts # modify — add recordProcessingEvent() calls +│ └── ... +└── index.ts # modify — mount /admin routes +``` + +### Pattern 1: Two-Layer Admin Auth Middleware + +**What:** `verifyFirebaseToken` handles token signature + expiry; `requireAdminEmail` checks that `req.user.email` equals the configured admin email. Admin routes apply both in sequence. + +**When to use:** All `/admin/*` routes. + +**Example:** +```typescript +// backend/src/middleware/requireAdmin.ts +import { Response, NextFunction } from 'express'; +import { FirebaseAuthenticatedRequest } from './firebaseAuth'; +import { logger } from '../utils/logger'; + +const ADMIN_EMAIL = process.env['ADMIN_EMAIL'] ?? process.env['EMAIL_WEEKLY_RECIPIENT']; + +export function requireAdminEmail( + req: FirebaseAuthenticatedRequest, + res: Response, + next: NextFunction +): void { + const userEmail = req.user?.email; + + if (!userEmail || userEmail !== ADMIN_EMAIL) { + // 404 — do not reveal admin routes exist (per locked decision) + logger.warn('requireAdminEmail: access denied', { + uid: req.user?.uid ?? 'unauthenticated', + email: userEmail ?? 'none', + path: req.path, + }); + res.status(404).json({ error: 'Not found' }); + return; + } + + next(); +} +``` + +**Unauthenticated handling:** `verifyFirebaseToken` already returns 401 for missing/invalid tokens. Since it runs first, unauthenticated requests never reach `requireAdminEmail`. The 404 behavior (hiding admin routes) only applies to authenticated non-admin users — this is consistent with the existing middleware chain. No change needed to `verifyFirebaseToken`. + +### Pattern 2: Admin Router Construction + +**What:** A dedicated Express Router with both middleware applied at router level, then individual route handlers. + +**When to use:** All admin endpoints. + +**Example:** +```typescript +// backend/src/routes/admin.ts +import { Router, Request, Response } from 'express'; +import { verifyFirebaseToken } from '../middleware/firebaseAuth'; +import { requireAdminEmail } from '../middleware/requireAdmin'; +import { addCorrelationId } from '../middleware/validation'; +import { HealthCheckModel } from '../models/HealthCheckModel'; +import { AlertEventModel } from '../models/AlertEventModel'; +import { getAnalyticsSummary } from '../services/analyticsService'; +import { logger } from '../utils/logger'; + +const router = Router(); + +// Auth chain: verify Firebase token, then assert admin email +router.use(verifyFirebaseToken); +router.use(requireAdminEmail); +router.use(addCorrelationId); + +const SERVICE_NAMES = ['document_ai', 'llm', 'supabase', 'firebase_auth'] as const; + +router.get('/health', async (req: Request, res: Response): Promise => { + try { + const results = await Promise.all( + SERVICE_NAMES.map(name => HealthCheckModel.findLatestByService(name)) + ); + const health = SERVICE_NAMES.map((name, i) => ({ + service: name, + status: results[i]?.status ?? 'unknown', + checkedAt: results[i]?.checked_at ?? null, + latencyMs: results[i]?.latency_ms ?? null, + errorMessage: results[i]?.error_message ?? null, + })); + res.json({ success: true, data: health, correlationId: req.correlationId }); + } catch (error) { + logger.error('GET /admin/health failed', { error, correlationId: req.correlationId }); + res.status(500).json({ success: false, error: 'Health query failed', correlationId: req.correlationId }); + } +}); +``` + +### Pattern 3: Analytics Summary Query + +**What:** A new `getAnalyticsSummary(range: string)` function in `analyticsService.ts` that queries `document_processing_events` aggregated over a time window. Supabase JS client does not support `COUNT`/`AVG` aggregations directly — use the Postgres pool (`getPostgresPool().query()`) for aggregate SQL, consistent with how `runRetentionCleanup` and the scheduled function's health check already use the pool. + +**When to use:** `GET /admin/analytics?range=24h` + +**Range parsing:** `24h` → `24 hours`, `7d` → `7 days`. Default: `24h`. + +**Example:** +```typescript +// backend/src/services/analyticsService.ts (addition) +import { getPostgresPool } from '../config/supabase'; + +export interface AnalyticsSummary { + range: string; + totalUploads: number; + succeeded: number; + failed: number; + successRate: number; + avgProcessingMs: number | null; + generatedAt: string; +} + +export async function getAnalyticsSummary(range: string): Promise { + const interval = parseRange(range); // '24h' -> '24 hours', '7d' -> '7 days' + const pool = getPostgresPool(); + + const { rows } = await pool.query<{ + total_uploads: string; + succeeded: string; + failed: string; + avg_processing_ms: string | null; + }>(` + SELECT + COUNT(*) FILTER (WHERE event_type = 'upload_started') AS total_uploads, + COUNT(*) FILTER (WHERE event_type = 'completed') AS succeeded, + COUNT(*) FILTER (WHERE event_type = 'failed') AS failed, + AVG(duration_ms) FILTER (WHERE event_type = 'completed') AS avg_processing_ms + FROM document_processing_events + WHERE created_at >= NOW() - INTERVAL $1 + `, [interval]); + + const row = rows[0]!; + const total = parseInt(row.total_uploads, 10); + const succeeded = parseInt(row.succeeded, 10); + const failed = parseInt(row.failed, 10); + + return { + range, + totalUploads: total, + succeeded, + failed, + successRate: total > 0 ? succeeded / total : 0, + avgProcessingMs: row.avg_processing_ms ? parseFloat(row.avg_processing_ms) : null, + generatedAt: new Date().toISOString(), + }; +} + +function parseRange(range: string): string { + if (/^\d+h$/.test(range)) return range.replace('h', ' hours'); + if (/^\d+d$/.test(range)) return range.replace('d', ' days'); + return '24 hours'; // fallback +} +``` + +### Pattern 4: Analytics Instrumentation in jobProcessorService + +**What:** Three `recordProcessingEvent()` calls in `processJob()` at existing lifecycle points. The function signature already matches — `document_id`, `user_id`, `event_type`, optional `duration_ms` and `error_message`. The return type is `void` (not `Promise`) so no `await` is possible. + +**Key instrumentation points:** +1. After `ProcessingJobModel.markAsProcessing(jobId)` — emit `upload_started` (no duration) +2. After `ProcessingJobModel.markAsCompleted(...)` — emit `completed` with `duration_ms = Date.now() - startTime` +3. In the catch block before `ProcessingJobModel.markAsFailed(...)` — emit `failed` with `duration_ms` and `error_message` + +**Example:** +```typescript +// In processJob(), after markAsProcessing: +recordProcessingEvent({ + document_id: job.document_id, + user_id: job.user_id, + event_type: 'upload_started', +}); + +// After markAsCompleted: +recordProcessingEvent({ + document_id: job.document_id, + user_id: job.user_id, + event_type: 'completed', + duration_ms: Date.now() - startTime, +}); + +// In catch, before markAsFailed: +recordProcessingEvent({ + document_id: job.document_id, + user_id: job.user_id ?? '', + event_type: 'failed', + duration_ms: Date.now() - startTime, + error_message: errorMessage, +}); +``` + +**Constraint:** `job` may be null in the catch block if `findById` failed. Guard with `job?.document_id` or skip instrumentation when `job` is null (it's already handled by the early return in that case). + +### Pattern 5: Alert Acknowledge Semantics + +**Decision:** One-way acknowledge (active → acknowledged). `AlertEventModel.acknowledge(id)` already implements exactly this. No toggle, no note field. The endpoint returns the updated alert object. + +```typescript +router.post('/alerts/:id/acknowledge', async (req: Request, res: Response): Promise => { + const { id } = req.params; + try { + const updated = await AlertEventModel.acknowledge(id); + res.json({ success: true, data: updated, correlationId: req.correlationId }); + } catch (error) { + // AlertEventModel.acknowledge throws a specific error when id not found + const msg = error instanceof Error ? error.message : String(error); + if (msg.includes('not found')) { + res.status(404).json({ success: false, error: 'Alert not found', correlationId: req.correlationId }); + return; + } + logger.error('POST /admin/alerts/:id/acknowledge failed', { id, error: msg }); + res.status(500).json({ success: false, error: 'Acknowledge failed', correlationId: req.correlationId }); + } +}); +``` + +### Anti-Patterns to Avoid + +- **Awaiting `recordProcessingEvent()`:** Its return type is `void`, not `Promise`. Calling `await recordProcessingEvent(...)` is a TypeScript error and would break the fire-and-forget guarantee. +- **Supabase JS `.select()` for aggregates:** Supabase JS client does not support SQL aggregate functions (`COUNT`, `AVG`). Use `getPostgresPool().query()` for analytics queries. +- **Caching admin email at module level:** Firebase Secrets are not available at module load time. Read `process.env['ADMIN_EMAIL']` inside the middleware function, not at the top of the file — or use lazy evaluation. The alertService precedent (creating transporter inside function scope) demonstrates this pattern. +- **Revealing admin routes to non-admin users:** Never return 403 on admin routes — always return 404 to unauthenticated/non-admin callers (per locked decision). Since `verifyFirebaseToken` runs first and returns 401 for unauthenticated requests, unauthenticated callers get 401 (expected, token verification precedes admin check). Authenticated non-admin callers get 404. +- **Mutating existing `processJob()` logic:** Analytics calls go around existing `markAsProcessing`, `markAsCompleted`, `markAsFailed` calls — never replacing or wrapping them. + +--- + +## Don't Hand-Roll + +| Problem | Don't Build | Use Instead | Why | +|---------|-------------|-------------|-----| +| Token verification | Custom JWT validation | `verifyFirebaseToken` (already exists) | Handles expiry, revocation, recovery from session | +| Health data retrieval | Raw SQL or in-memory aggregation | `HealthCheckModel.findLatestByService()` (already exists) | Validated input, proper error handling, same pattern as Phase 2 | +| Alert CRUD | New Supabase queries | `AlertEventModel.findActive()`, `AlertEventModel.acknowledge()` (already exist) | Consistent error handling, deduplication-aware | +| Correlation IDs | Custom header logic | `addCorrelationId` middleware (already exists) | Applied at router level like other route files | + +**Key insight:** Phase 3 is primarily composition, not construction. Nearly all data access is through existing models. The only new code is the admin router, the admin email middleware, the `getAnalyticsSummary()` function, and three `recordProcessingEvent()` call sites. + +--- + +## Common Pitfalls + +### Pitfall 1: Admin Email Source +**What goes wrong:** `ADMIN_EMAIL` env var is not defined; admin check silently passes (if the check is `email === undefined`) or silently blocks all admin access. +**Why it happens:** The codebase uses `EMAIL_WEEKLY_RECIPIENT` for the alert recipient — there is no `ADMIN_EMAIL` variable yet. If `ADMIN_EMAIL` is not set and the check falls back to `undefined`, `email !== undefined` would always be true (blocking all) or the inverse. +**How to avoid:** Read `ADMIN_EMAIL ?? EMAIL_WEEKLY_RECIPIENT` as fallback. Log a `logger.warn` at startup/first call if neither is defined. If neither is set, fail closed (deny all admin access) with a logged warning. +**Warning signs:** Admin endpoints return 404 even when authenticated with the correct email. + +### Pitfall 2: Service Name Mismatch on Health Endpoint +**What goes wrong:** `GET /admin/health` returns `status: null` / `checkedAt: null` for all services because the service names in the query don't match what `healthProbeService` writes. +**Why it happens:** `HealthCheckModel.findLatestByService(serviceName)` does an exact string match. If the route handler uses `'document-ai'` but the probe writes `'document_ai'`, the join finds nothing. +**How to avoid:** Read `healthProbeService.ts` to confirm the exact service name strings used in `HealthCheckResult` / passed to `HealthCheckModel.create()`. Use those exact strings in the admin route. +**Warning signs:** Response data has `status: 'unknown'` for all services. + +### Pitfall 3: `job.user_id` Type in Analytics Instrumentation +**What goes wrong:** TypeScript error or runtime `undefined` when emitting `recordProcessingEvent` in the catch block. +**Why it happens:** `job` can be `null` if `ProcessingJobModel.findById()` threw before `job` was assigned. The catch block handles all errors, including the pre-assignment path. +**How to avoid:** Guard instrumentation with `if (job)` in the catch block. `ProcessingEventData.user_id` is typed as `string`, so pass `job.user_id` only when `job` is non-null. +**Warning signs:** TypeScript compile error on `job.user_id` in catch block. + +### Pitfall 4: `getPostgresPool()` vs `getSupabaseServiceClient()` for Aggregates +**What goes wrong:** Using `getSupabaseServiceClient().from('document_processing_events').select(...)` for the analytics summary and getting back raw rows instead of aggregated counts. +**Why it happens:** Supabase JS PostgREST client does not support SQL aggregate functions in the query builder. +**How to avoid:** Use `getPostgresPool().query(sql, params)` for the analytics aggregate query, consistent with how `processDocumentJobs` scheduled function performs its DB health check and how `cleanupOldData` runs bulk deletes. +**Warning signs:** `getAnalyticsSummary` returns row-level data instead of aggregated counts. + +### Pitfall 5: Route Registration Order in index.ts +**What goes wrong:** Admin routes conflict with or shadow existing routes. +**Why it happens:** Express matches routes in registration order. Registering `/admin` before `/documents` is fine as long as there are no overlapping paths. +**How to avoid:** Add `app.use('/admin', adminRoutes)` alongside the existing route registrations. The `/admin` prefix is unique — no conflicts expected. +**Warning signs:** Existing document/monitoring routes stop working after adding admin routes. + +--- + +## Code Examples + +Verified patterns from the existing codebase: + +### Existing Route File Pattern (from routes/monitoring.ts) +```typescript +// Source: backend/src/routes/monitoring.ts +import { Router, Request, Response } from 'express'; +import { addCorrelationId } from '../middleware/validation'; +import { logger } from '../utils/logger'; + +const router = Router(); +router.use(addCorrelationId); + +router.get('/some-endpoint', async (req: Request, res: Response): Promise => { + try { + // ... data access + res.json({ + success: true, + data: someData, + correlationId: req.correlationId || undefined, + }); + } catch (error) { + logger.error('Failed', { + category: 'monitoring', + operation: 'some_op', + error: error instanceof Error ? error.message : 'Unknown error', + correlationId: req.correlationId || undefined, + }); + res.status(500).json({ + success: false, + error: 'Failed to retrieve data', + correlationId: req.correlationId || undefined, + }); + } +}); + +export default router; +``` + +### Existing Middleware Pattern (from middleware/firebaseAuth.ts) +```typescript +// Source: backend/src/middleware/firebaseAuth.ts +export interface FirebaseAuthenticatedRequest extends Request { + user?: admin.auth.DecodedIdToken; +} + +export const verifyFirebaseToken = async ( + req: FirebaseAuthenticatedRequest, + res: Response, + next: NextFunction +): Promise => { + // ... verifies token, sets req.user, calls next() or returns 401 +}; +``` + +### Existing Model Pattern (from models/HealthCheckModel.ts) +```typescript +// Source: backend/src/models/HealthCheckModel.ts +static async findLatestByService(serviceName: string): Promise { + const supabase = getSupabaseServiceClient(); + const { data, error } = await supabase + .from('service_health_checks') + .select('*') + .eq('service_name', serviceName) + .order('checked_at', { ascending: false }) + .limit(1) + .single(); + if (error?.code === 'PGRST116') return null; + // ... +} +``` + +### Existing Analytics Record Pattern (from services/analyticsService.ts) +```typescript +// Source: backend/src/services/analyticsService.ts +// Return type is void (NOT Promise) — prevents accidental await on critical path +export function recordProcessingEvent(data: ProcessingEventData): void { + const supabase = getSupabaseServiceClient(); + void supabase + .from('document_processing_events') + .insert({ ... }) + .then(({ error }) => { + if (error) logger.error('analyticsService: failed to insert processing event', { ... }); + }); +} +``` + +### Route Registration Pattern (from index.ts) +```typescript +// Source: backend/src/index.ts +app.use('/documents', documentRoutes); +app.use('/vector', vectorRoutes); +app.use('/monitoring', monitoringRoutes); +app.use('/api/audit', auditRoutes); +// New: +app.use('/admin', adminRoutes); +``` + +--- + +## State of the Art + +| Old Approach | Current Approach | When Changed | Impact | +|--------------|------------------|--------------|--------| +| Legacy auth middleware (auth.ts) | Firebase Auth (firebaseAuth.ts) | Pre-Phase 3 | `auth.ts` is fully deprecated and returns 501 — do not use it | +| In-memory monitoring (uploadMonitoringService) | Supabase-persisted health checks and analytics | Phase 1-2 | Admin endpoints must read from Supabase, not in-memory state | +| Direct `console.log` | Winston logger (`logger` from `utils/logger.ts`) | Pre-Phase 3 | Always use `logger.info/warn/error/debug` | + +**Deprecated/outdated:** +- `backend/src/middleware/auth.ts`: All exports (`authenticateToken`, `requireAdmin`, `requireRole`) return 501. Do not import. Use `firebaseAuth.ts`. +- `uploadMonitoringService`: In-memory service. Not suitable for admin health dashboard — data does not survive cold starts. + +--- + +## Open Questions + +1. **Exact service name strings written by healthProbeService** + - What we know: The service names come from whatever `healthProbeService.ts` passes to `HealthCheckModel.create({ service_name: ... })` + - What's unclear: The exact strings — likely `'document_ai'`, `'llm'`, `'supabase'`, `'firebase_auth'` but must be verified before writing the health handler + - Recommendation: Read `healthProbeService.ts` during plan/implementation to confirm exact strings before writing `SERVICE_NAMES` constant in the admin route + +2. **`job.user_id` field type confirmation** + - What we know: `ProcessingEventData.user_id` is typed as `string`; `ProcessingJob` model has `user_id` field + - What's unclear: Whether `ProcessingJob.user_id` can ever be `undefined`/nullable in practice + - Recommendation: Check `ProcessingJobModel` type definition during implementation; add defensive `?? ''` if nullable + +3. **Alert pagination for GET /admin/alerts** + - What we know: `AlertEventModel.findActive()` returns all active alerts without limit; for a single-admin system this is unlikely to be an issue + - What's unclear: Whether a limit/offset param is needed + - Recommendation: Claude's discretion — default to returning all active alerts (no pagination) given single-admin use case; add `?limit=N` support as optional param using `.limit()` on the Supabase query + +--- + +## Sources + +### Primary (HIGH confidence) +- Codebase: `backend/src/middleware/firebaseAuth.ts` — verifyFirebaseToken implementation, FirebaseAuthenticatedRequest interface, 401 error responses +- Codebase: `backend/src/models/HealthCheckModel.ts` — findLatestByService, findAll, deleteOlderThan patterns +- Codebase: `backend/src/models/AlertEventModel.ts` — findActive, acknowledge, resolve, findRecentByService patterns +- Codebase: `backend/src/services/analyticsService.ts` — recordProcessingEvent (void return), deleteProcessingEventsOlderThan (pool.query pattern) +- Codebase: `backend/src/services/jobProcessorService.ts` — processJob lifecycle: startTime capture, markAsProcessing, markAsCompleted, markAsFailed, catch block structure +- Codebase: `backend/src/routes/monitoring.ts` — route file pattern, envelope shape `{ success, data, correlationId }` +- Codebase: `backend/src/index.ts` — route registration, Express app structure, existing `/health` endpoint shape +- Codebase: `backend/src/models/migrations/012_create_monitoring_tables.sql` — exact column names for service_health_checks, alert_events +- Codebase: `backend/src/models/migrations/013_create_processing_events_table.sql` — exact column names for document_processing_events + +### Secondary (MEDIUM confidence) +- Codebase: `backend/src/services/alertService.ts` — pattern for reading `process.env['EMAIL_WEEKLY_RECIPIENT']` inside function (not at module level) to avoid Firebase Secrets timing issue + +--- + +## Metadata + +**Confidence breakdown:** +- Standard stack: HIGH — all libraries already in use; no new dependencies +- Architecture: HIGH — patterns derived from existing codebase, not assumptions +- Pitfalls: HIGH — three of five pitfalls are directly observable from reading the existing code +- Open questions: LOW confidence only on exact service name strings (requires reading one more file) + +**Research date:** 2026-02-24 +**Valid until:** 2026-03-24 (stable codebase; valid until significant refactoring)