teatea-pension/.trellis/tasks/07-05-07-06-harden-ai-analysis-path/design.md

Harden AI Analysis Path Design

Overview

This change is an in-place hardening pass over the existing AI MVP. It keeps the current module boundaries and synchronous workflow, then adds tests and narrows citation behavior.

Boundaries

Backend AI module

• modules/ai/types.ts
  • Keep runtime validators as the source of truth for output shape.
  • Add/adjust helpers only if needed for citation ID validation.
• modules/ai/server/analysis.ts
  • Validate model output against the allowed citation set before persisting a completed history row.
  • Derive final persisted/display citations from citation IDs actually referenced by findings/recommendations.
  • Keep all provider and validation failures sanitized.
• modules/ai/server/knowledge.ts
  • Preserve current save-time embedding and scope-filtered retrieval behavior.
  • Add tests around existing behavior rather than introducing background embedding state.

API routes

• app/api/ai/elders/[id]/analyses/route.ts
• app/api/ai/knowledge/route.ts
• app/api/ai/knowledge/[id]/route.ts

Route behavior stays unchanged except tests protect permission ordering, structured failures, and audit behavior.

Frontend

• modules/ai/components/KnowledgeManagementClient.tsx
  • Make AI configuration / vector generation failures more explicit.
• modules/ai/components/ElderAiAnalysisDialog.tsx
  • Make failed latest history state clearer without exposing raw errors.

Permissions

• modules/core/server/permissions.ts
  • Clarify ai:manage as reserved/currently governance-oriented wording only.

Citation Contract

Allowed citations are built by buildElderAiContext plus knowledge retrieval. The model may reference only these IDs.

Final persisted result.citations must be derived from actual references:

1. Collect citation IDs from all keyFindings[].citationIds and recommendations[].citationIds.
2. If any referenced ID is absent from the allowed citation map, reject the output as schema_validation_failed.
3. Persist/display only the allowed citations whose IDs were referenced.
4. Preserve stable ordering from the original allowed citation list.

This avoids showing unused evidence and prevents hallucinated source IDs.

Test Strategy

Tests should mock database/provider boundaries and assert observable contracts, not implementation details.

• Validator tests use pure functions.
• Knowledge tests mock getDatabase and OpenAIEmbeddings or use light fake query builders where practical.
• Analysis tests mock config, elder context, knowledge retrieval, model invocation, database insert, and audit logging.
• Route tests mock requirePermission, AI services, validation as needed, and audit logging.

Compatibility

No database migration is required for this hardening pass. No new dependency is required. Existing persisted resultJson.citations remain readable; new analyses will have stricter citation lists.

Rollback

• Revert citation strictness if a provider cannot satisfy citation references, but keep tests documenting expected loosened behavior.
• Revert UI message changes independently if copy causes product concern.
• Tests are additive and can stay even if implementation is adjusted.