02
Commissioning Knowledge Retrieval & Troubleshooting System (RAG)
设备调试知识检索与故障辅助排查系统(RAG)
- Core retrieval and evidence-chain development, testing and regression verification.
- RAG · Hybrid Retrieval · Rerank · FastAPI
- 0.993 Recall@5 · 2200 tests passed · 89% line coverage
Problem
A RAG system for looking up commissioning documentation and supporting new-hire training: four-way hybrid retrieval plus rerank, answers that must cite source evidence, and a refusal or follow-up question instead of a made-up answer when evidence falls short.
The system only provides knowledge and suggestions. It has no ability to control any equipment.
Internal equipment-commissioning documentation was scattered and slow to search. New-hire training ran on a tight schedule, and common questions depended on senior staff to answer.
The system supports looking up commissioning documentation and helps with new-hire training.
- Commissioning documentation is scattered and technical documents are slow to search.
- New-hire training runs on a tight schedule.
- Common questions depend on senior staff to answer.
Developed the core retrieval and evidence chain, and owned testing and regression verification.
Build
Ingestion with provenance: PDF / DOCX manuals are parsed — including tables, embedded table files and OCR of screenshots — the chapter tree and stable page numbers are rebuilt, and the content is split into 600+ traceable knowledge units, each recording document, chapter, page and risk level. Rule-based redaction runs before ingestion; external models only receive redacted text, and images are never sent out.
Hybrid retrieval: four recall paths — trigram exact match, BM25, character n-grams and vectors — are fused with weighted RRF and then reranked, with a retrieval trace kept for every stage.
Evidence Gate: low relevance is treated as no basis for an answer; when key conditions such as product line or version are missing, the system asks first; L0–L3 risk levels restrict how high-risk content is shown; any request to bypass approval is refused.
Citation Validation: every citation must belong to the current evidence set, and every number, path and command in the answer must appear verbatim in the cited source. If any check fails, the system fails closed and falls back to listing the source evidence only.
Knowledge sources
- PDF manualsPDF 手册
- DOCX manualsDOCX 手册
- Tables & embedded table files表格与内嵌表格文件
- Screenshot OCR截图 OCR
- Knowledge Ingestion
- Trigram ExactBM25 (jieba)Char n-gramVector
- Weighted RRF
- Rerank
- Evidence Gate
- LLM Generation
- Citation Validation
- Answer / Evidence-only Fallback
Results from the four recall paths are fused with weighted RRF and then reranked; the retrieval trace of every stage is kept.
If either the Evidence Gate or citation validation fails, the system fails closed: no generated answer, only the source evidence. When the cloud service fails, a circuit breaker trips and retrieval falls back to local-only.
- Query
- 4-way Recall
- RRF + Rerank
- Evidence Gate
- Generate
- Citation Check
- 03› Weighted RRF fusion, then rerank
- 04› Relevance sufficient → generate
- 05› Generate an answer from the evidence set
- 06› Citations verified → answer with source citations
Query
Q: alarm XX after start-up — what should I do?
4-way Recall
Four-way recall: trigram / BM25 / char n-gram / vector
RRF + Rerank
Weighted RRF fusion, then rerank
Evidence Gate
Relevance sufficient → generate
Generate
Generate an answer from the evidence set
Citation Check
Citations verified → answer with source citations
Q ›The equipment shows alarm XX after start-up — what should I do?
Retrieved evidence
- PDF manual
- Screenshot OCR
- DOCX manual
answerable→ LLM, answer with citations
- 01
Four recall paths, not vectors alone
The retrieval ablation shows it: vectors alone reach Recall@5 0.755, local multi-path recall 0.976, and multi-path + vectors + rerank 0.993.
- 02
Brute-force cosine, not FAISS
With only 600+ knowledge units the data set is small, so plain cosine similarity is enough and a FAISS index is not needed.
- 03
Multi-path agreement bypass
Rerank tends to push down literal matches, so a multi-path agreement bypass keeps results that several recall paths agree on from being pushed out by the reranker.
- 04
Refuse or ask rather than make things up
Low relevance means no basis for an answer, missing key conditions mean a follow-up question first, and any failed check falls back to the source evidence only.
- 05
External models only see redacted text
Rule-based redaction runs before ingestion. External models only receive redacted text, and images are never sent out.
Engineering
- Python + FastAPI; SQLite FTS5, jieba BM25, weighted RRF, rerank, vector retrieval (cloud embedding); Pydantic.
- Web app with authentication, audit logging, Q&A history and screenshots attached to answers.
- Circuit breaker on cloud-service failure, falling back to local retrieval.
- Two-stage Docker image and a compose deployment setup.
- Evaluation: auto-generated reverse questions, a frozen gold set, an independent held-out set and a red-team set; rule-based verdicts and human review are recorded separately, and repeated replays separate model variance from real regressions.
- 2200 tests passing, 89% line coverage, about 98% on the core retrieval and Q&A modules.
- Quality gates: ruff, pyright, pytest and a secret-leak scan.
- Recall@5 · 457 auto eval questions · multi-path + vector + rerank
- 0.993
- tests passed
- 2200
- line coverage · core modules ~98%
- 89%
Retrieval ablation (457 auto-generated eval questions, Recall@5): vectors only 0.755 → local multi-path 0.976 → multi-path + vectors + rerank 0.993.
Gold set: Recall@5 0.983, Recall@10 1.000.
Passed internal pilot acceptance.
Notes
Rerank is not a free win: it pushes down literal matches, so it needs a multi-path agreement bypass as a backstop.
Evaluation has to tell model variance apart from real regressions: replay several times, and record rule-based verdicts and human review separately.
- Python
- FastAPI
- SQLite FTS5
- jieba BM25
- Weighted RRF
- Rerank
- Vector Retrieval
- Cloud Embedding
- Pydantic
- Pytest
- Docker
- RAG