chore(task): archive 07-05-07-06-harden-ai-analysis-path
This commit is contained in:
@@ -0,0 +1,74 @@
|
||||
# Harden AI Analysis Path Design
|
||||
|
||||
## Overview
|
||||
|
||||
This change is an in-place hardening pass over the existing AI MVP. It keeps the current module boundaries and synchronous workflow, then adds tests and narrows citation behavior.
|
||||
|
||||
## Boundaries
|
||||
|
||||
### Backend AI module
|
||||
|
||||
- `modules/ai/types.ts`
|
||||
- Keep runtime validators as the source of truth for output shape.
|
||||
- Add/adjust helpers only if needed for citation ID validation.
|
||||
- `modules/ai/server/analysis.ts`
|
||||
- Validate model output against the allowed citation set before persisting a completed history row.
|
||||
- Derive final persisted/display citations from citation IDs actually referenced by findings/recommendations.
|
||||
- Keep all provider and validation failures sanitized.
|
||||
- `modules/ai/server/knowledge.ts`
|
||||
- Preserve current save-time embedding and scope-filtered retrieval behavior.
|
||||
- Add tests around existing behavior rather than introducing background embedding state.
|
||||
|
||||
### API routes
|
||||
|
||||
- `app/api/ai/elders/[id]/analyses/route.ts`
|
||||
- `app/api/ai/knowledge/route.ts`
|
||||
- `app/api/ai/knowledge/[id]/route.ts`
|
||||
|
||||
Route behavior stays unchanged except tests protect permission ordering, structured failures, and audit behavior.
|
||||
|
||||
### Frontend
|
||||
|
||||
- `modules/ai/components/KnowledgeManagementClient.tsx`
|
||||
- Make AI configuration / vector generation failures more explicit.
|
||||
- `modules/ai/components/ElderAiAnalysisDialog.tsx`
|
||||
- Make failed latest history state clearer without exposing raw errors.
|
||||
|
||||
### Permissions
|
||||
|
||||
- `modules/core/server/permissions.ts`
|
||||
- Clarify `ai:manage` as reserved/currently governance-oriented wording only.
|
||||
|
||||
## Citation Contract
|
||||
|
||||
Allowed citations are built by `buildElderAiContext` plus knowledge retrieval. The model may reference only these IDs.
|
||||
|
||||
Final persisted `result.citations` must be derived from actual references:
|
||||
|
||||
1. Collect citation IDs from all `keyFindings[].citationIds` and `recommendations[].citationIds`.
|
||||
2. If any referenced ID is absent from the allowed citation map, reject the output as `schema_validation_failed`.
|
||||
3. Persist/display only the allowed citations whose IDs were referenced.
|
||||
4. Preserve stable ordering from the original allowed citation list.
|
||||
|
||||
This avoids showing unused evidence and prevents hallucinated source IDs.
|
||||
|
||||
## Test Strategy
|
||||
|
||||
Tests should mock database/provider boundaries and assert observable contracts, not implementation details.
|
||||
|
||||
- Validator tests use pure functions.
|
||||
- Knowledge tests mock `getDatabase` and `OpenAIEmbeddings` or use light fake query builders where practical.
|
||||
- Analysis tests mock config, elder context, knowledge retrieval, model invocation, database insert, and audit logging.
|
||||
- Route tests mock `requirePermission`, AI services, validation as needed, and audit logging.
|
||||
|
||||
## Compatibility
|
||||
|
||||
No new dependency is required.
|
||||
Deployment hardening additionally changes the still-unapplied `0007` AI migration from pgvector to JSONB embeddings, so production PostgreSQL 18 Alpine can run the migration without a `vector` extension.
|
||||
Existing persisted `resultJson.citations` remain readable; new analyses will have stricter citation lists.
|
||||
|
||||
## Rollback
|
||||
|
||||
- Revert citation strictness if a provider cannot satisfy citation references, but keep tests documenting expected loosened behavior.
|
||||
- Revert UI message changes independently if copy causes product concern.
|
||||
- Tests are additive and can stay even if implementation is adjusted.
|
||||
Reference in New Issue
Block a user