test: harden AI analysis path

.trellis/tasks/07-05-07-06-harden-ai-analysis-path/check.jsonl

@@ -0,0 +1 @@
+{"_example": "Fill with {\"file\": \"<path>\", \"reason\": \"<why>\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."}

.trellis/tasks/07-05-07-06-harden-ai-analysis-path/design.md

@@ -0,0 +1,74 @@
+# Harden AI Analysis Path Design
+
+## Overview
+
+This change is an in-place hardening pass over the existing AI MVP. It keeps the current module boundaries and synchronous workflow, then adds tests and narrows citation behavior.
+
+## Boundaries
+
+### Backend AI module
+
+- `modules/ai/types.ts`
+  - Keep runtime validators as the source of truth for output shape.
+  - Add/adjust helpers only if needed for citation ID validation.
+- `modules/ai/server/analysis.ts`
+  - Validate model output against the allowed citation set before persisting a completed history row.
+  - Derive final persisted/display citations from citation IDs actually referenced by findings/recommendations.
+  - Keep all provider and validation failures sanitized.
+- `modules/ai/server/knowledge.ts`
+  - Preserve current save-time embedding and scope-filtered retrieval behavior.
+  - Add tests around existing behavior rather than introducing background embedding state.
+
+### API routes
+
+- `app/api/ai/elders/[id]/analyses/route.ts`
+- `app/api/ai/knowledge/route.ts`
+- `app/api/ai/knowledge/[id]/route.ts`
+
+Route behavior stays unchanged except tests protect permission ordering, structured failures, and audit behavior.
+
+### Frontend
+
+- `modules/ai/components/KnowledgeManagementClient.tsx`
+  - Make AI configuration / vector generation failures more explicit.
+- `modules/ai/components/ElderAiAnalysisDialog.tsx`
+  - Make failed latest history state clearer without exposing raw errors.
+
+### Permissions
+
+- `modules/core/server/permissions.ts`
+  - Clarify `ai:manage` as reserved/currently governance-oriented wording only.
+
+## Citation Contract
+
+Allowed citations are built by `buildElderAiContext` plus knowledge retrieval. The model may reference only these IDs.
+
+Final persisted `result.citations` must be derived from actual references:
+
+1. Collect citation IDs from all `keyFindings[].citationIds` and `recommendations[].citationIds`.
+2. If any referenced ID is absent from the allowed citation map, reject the output as `schema_validation_failed`.
+3. Persist/display only the allowed citations whose IDs were referenced.
+4. Preserve stable ordering from the original allowed citation list.
+
+This avoids showing unused evidence and prevents hallucinated source IDs.
+
+## Test Strategy
+
+Tests should mock database/provider boundaries and assert observable contracts, not implementation details.
+
+- Validator tests use pure functions.
+- Knowledge tests mock `getDatabase` and `OpenAIEmbeddings` or use light fake query builders where practical.
+- Analysis tests mock config, elder context, knowledge retrieval, model invocation, database insert, and audit logging.
+- Route tests mock `requirePermission`, AI services, validation as needed, and audit logging.
+
+## Compatibility
+
+No database migration is required for this hardening pass.
+No new dependency is required.
+Existing persisted `resultJson.citations` remain readable; new analyses will have stricter citation lists.
+
+## Rollback
+
+- Revert citation strictness if a provider cannot satisfy citation references, but keep tests documenting expected loosened behavior.
+- Revert UI message changes independently if copy causes product concern.
+- Tests are additive and can stay even if implementation is adjusted.

.trellis/tasks/07-05-07-06-harden-ai-analysis-path/implement.jsonl

@@ -0,0 +1 @@
+{"_example": "Fill with {\"file\": \"<path>\", \"reason\": \"<why>\"}. Put spec/research files only — no code paths. Run `python3 .trellis/scripts/get_context.py --mode packages` to list available specs. Delete this line once real entries are added."}

.trellis/tasks/07-05-07-06-harden-ai-analysis-path/implement.md

@@ -0,0 +1,43 @@
+# Harden AI Analysis Path Implementation Plan
+
+## Steps
+
+1. Add pure validator tests for `modules/ai/types.ts`.
+2. Add/adjust analysis citation helper behavior in `modules/ai/server/analysis.ts`.
+3. Add analysis service tests for missing config, invalid output, provider errors, retrieval degradation, and redaction.
+4. Add knowledge service tests for scope, disabled entries, organization isolation, and embedding failure.
+5. Add AI route tests for permission failures, invalid input, service failure propagation, success responses, and audit calls.
+6. Improve knowledge UI and analysis dialog failure copy.
+7. Clarify `ai:manage` permission description.
+8. Run targeted AI tests, then project verification.
+
+## Validation Commands
+
+- `pnpm test -- modules/ai/types.test.ts`
+- `pnpm test -- modules/ai/server/analysis.test.ts modules/ai/server/knowledge.test.ts app/api/ai/ai-routes.test.ts`
+- `pnpm lint`
+- `pnpm type-check`
+- `pnpm test`
+- `pnpm build`
+
+## Risky Files
+
+- `modules/ai/server/analysis.ts`
+- `modules/ai/server/knowledge.ts`
+- `modules/ai/types.ts`
+- `app/api/ai/*/route.ts`
+- `modules/core/server/permissions.ts`
+
+## Review Gates
+
+- Citation helper must reject unknown citation IDs and avoid appending unused citations.
+- Tests must not call real AI providers.
+- Tests must not require a live PostgreSQL instance unless explicitly scoped to migration smoke testing.
+- No new `any`, non-null assertions, `@ts-ignore`, or `@ts-expect-error`.
+- Failed history records must remain sanitized.
+
+## Rollback Points
+
+- After citation strictness: revert helper and related tests if provider output compatibility proves too brittle.
+- After route tests: routes should remain behavior-compatible except status/copy assertions.
+- After UI copy: copy changes can be reverted without affecting backend contracts.

.trellis/tasks/07-05-07-06-harden-ai-analysis-path/prd.md

@@ -0,0 +1,49 @@
+# Harden AI Analysis Path PRD
+
+## Goal
+
+Harden the existing elder AI analysis and knowledge retrieval MVP so the critical safety boundaries are covered by tests and the displayed evidence chain matches the citations actually used by the model output.
+
+## Background
+
+The current MVP already implements LangChain-backed elder AI analysis, pgvector-backed knowledge retrieval, permission-scoped resident context, persisted analysis history, and knowledge management UI.
+
+Current evidence from repository inspection:
+
+- AI service files live under `modules/ai/`.
+- API routes live under `app/api/ai/`.
+- AI tables and vector extension are in `modules/core/server/schema.ts` and `drizzle/0007_purple_puff_adder.sql`.
+- Existing project verification passes: `pnpm lint`, `pnpm type-check`, and `pnpm test`.
+- No AI-specific test file currently covers schema validation, permission scoping, history redaction, knowledge organization isolation, disabled knowledge exclusion, or provider failure behavior.
+- `normalizeCitations` currently appends allowed citations that the model did not necessarily cite, weakening the evidence chain.
+
+## Requirements
+
+1. Add focused automated tests for AI validators, knowledge service behavior, analysis service behavior, and AI route authorization/response behavior.
+2. Tighten citation handling so the persisted/rendered `citations` list is derived from citation IDs actually referenced by findings and recommendations.
+3. Reject AI outputs whose finding or recommendation `citationIds` include IDs not present in the allowed context/knowledge citation set.
+4. Keep failed analysis history sanitized: no full prompt, resident context, API key, or raw provider error.
+5. Preserve the existing synchronous generation MVP behavior; do not introduce queues, streaming, background jobs, new dependencies, or a new vector store.
+6. Improve user-facing failure text where configuration/vector generation failure would otherwise look like a generic save/generation failure.
+7. Clarify `ai:manage` as a reserved/future governance permission instead of implying a currently implemented management surface.
+
+## Acceptance Criteria
+
+- `validateKnowledgeEntryInput`, `validateElderAiAnalysisOutput`, and `parseDataScopes` have direct unit coverage for valid and invalid edge cases.
+- Knowledge tests cover writable scope failures, disabled entry exclusion, platform/current-organization visibility, other-organization exclusion, and embedding failure behavior without hitting a real provider.
+- Analysis tests cover missing config, provider error, schema validation failure, retrieval degradation, and history redaction without hitting a real provider.
+- AI route tests cover missing `ai:read`, missing `elder:read`, missing `knowledge:read`, missing `knowledge:manage`, invalid request body, service failure status propagation, and success audit behavior.
+- Citation normalization no longer appends unused allowed citations.
+- Any unknown citation ID in findings/recommendations causes a structured schema validation failure and failed history record.
+- Knowledge UI and AI analysis dialog keep sensitive details out of failure messages while making configuration/vector-generation failures understandable.
+- `ai:manage` permission description no longer overpromises current UI/API capability.
+- `pnpm lint`, `pnpm type-check`, `pnpm test`, and `pnpm build` pass.
+
+## Out of Scope
+
+- Async generation jobs.
+- Streaming AI responses.
+- AI usage billing/rate limiting.
+- AI configuration UI.
+- File upload/OCR/web crawling knowledge ingestion.
+- AI-generated business record drafts or one-click mutations.

.trellis/tasks/07-05-07-06-harden-ai-analysis-path/task.json

@@ -0,0 +1,26 @@
+{
+  "id": "07-06-harden-ai-analysis-path",
+  "name": "07-06-harden-ai-analysis-path",
+  "title": "Harden AI Analysis Path",
+  "description": "Add tests and tighten citation evidence chain for the AI knowledge analysis MVP.",
+  "status": "in_progress",
+  "dev_type": null,
+  "scope": null,
+  "package": null,
+  "priority": "P1",
+  "creator": "TalexDreamSoul",
+  "assignee": "TalexDreamSoul",
+  "createdAt": "2026-07-05",
+  "completedAt": null,
+  "branch": null,
+  "base_branch": "main",
+  "worktree_path": null,
+  "commit": null,
+  "pr_url": null,
+  "subtasks": [],
+  "children": [],
+  "parent": null,
+  "relatedFiles": [],
+  "notes": "",
+  "meta": {}
+}