Skip to content
All projects

02Enterprise Knowledge Retrieval · RAG

Commissioning Knowledge Retrieval & Troubleshooting System (RAG)

设备调试​知识检索​与​故障辅助排查​系统(RAG)

Role
Core retrieval and evidence-chain development, testing and regression verification.
Stack
RAG · Hybrid Retrieval · Rerank · FastAPI
Proof
0.993 Recall@5 · 2200 tests passed · 89% line coverage

01

Problem

01.1Overview

A RAG system for looking up commissioning documentation and supporting new-hire training: four-way hybrid retrieval plus rerank, answers that must cite source evidence, and a refusal or follow-up question instead of a made-up answer when evidence falls short.

The system only provides knowledge and suggestions. It has no ability to control any equipment.

01.2Context

Internal equipment-commissioning documentation was scattered and slow to search. New-hire training ran on a tight schedule, and common questions depended on senior staff to answer.

The system supports looking up commissioning documentation and helps with new-hire training.

01.3Problem

  • Commissioning documentation is scattered and technical documents are slow to search.
  • New-hire training runs on a tight schedule.
  • Common questions depend on senior staff to answer.

01.4My Role

Developed the core retrieval and evidence chain, and owned testing and regression verification.

02

Build

02.1Solution

Ingestion with provenance: PDF / DOCX manuals are parsed — including tables, embedded table files and OCR of screenshots — the chapter tree and stable page numbers are rebuilt, and the content is split into 600+ traceable knowledge units, each recording document, chapter, page and risk level. Rule-based redaction runs before ingestion; external models only receive redacted text, and images are never sent out.

Hybrid retrieval: four recall paths — trigram exact match, BM25, character n-grams and vectors — are fused with weighted RRF and then reranked, with a retrieval trace kept for every stage.

Evidence Gate: low relevance is treated as no basis for an answer; when key conditions such as product line or version are missing, the system asks first; L0–L3 risk levels restrict how high-risk content is shown; any request to bypass approval is refused.

Citation Validation: every citation must belong to the current evidence set, and every number, path and command in the answer must appear verbatim in the cited source. If any check fails, the system fails closed and falls back to listing the source evidence only.

Knowledge sources

  • PDF manualsPDF 手册
  • DOCX manualsDOCX 手册
  • Tables & embedded table files表格与内嵌表格文件
  • Screenshot OCR截图 OCR

02.2Architecture

  1. Knowledge Ingestion
  2. Trigram Exact
    BM25 (jieba)
    Char n-gram
    Vector
  3. Weighted RRF
  4. Rerank
  5. Evidence Gate
  6. LLM Generation
  7. Citation Validation
  8. Answer / Evidence-only Fallback
fig. 02a — four-way recall, rerank, evidence gate & citation check

Results from the four recall paths are fused with weighted RRF and then reranked; the retrieval trace of every stage is kept.

If either the Evidence Gate or citation validation fails, the system fails closed: no generated answer, only the source evidence. When the cloud service fails, a circuit breaker trips and retrieval falls back to local-only.

02Alarm XX after start-up — what should I do?
  1. Query

    Q: alarm XX after start-up — what should I do?

  2. 4-way Recall

    Four-way recall: trigram / BM25 / char n-gram / vector

  3. RRF + Rerank

    Weighted RRF fusion, then rerank

  4. Evidence Gate

    Relevance sufficient → generate

  5. Generate

    Generate an answer from the evidence set

  6. Citation Check

    Citations verified → answer with source citations

One run of the pipeline, step by step.
EVIDENCE GATEillustration

Q ›The equipment shows alarm XX after start-up — what should I do?

Retrieved evidence

  • PDF manual
  • Screenshot OCR
  • DOCX manual
Evidence levelThreshold

answerable→ LLM, answer with citations

02.3Key Decisions

  1. 01

    Four recall paths, not vectors alone

    The retrieval ablation shows it: vectors alone reach Recall@5 0.755, local multi-path recall 0.976, and multi-path + vectors + rerank 0.993.

  2. 02

    Brute-force cosine, not FAISS

    With only 600+ knowledge units the data set is small, so plain cosine similarity is enough and a FAISS index is not needed.

  3. 03

    Multi-path agreement bypass

    Rerank tends to push down literal matches, so a multi-path agreement bypass keeps results that several recall paths agree on from being pushed out by the reranker.

  4. 04

    Refuse or ask rather than make things up

    Low relevance means no basis for an answer, missing key conditions mean a follow-up question first, and any failed check falls back to the source evidence only.

  5. 05

    External models only see redacted text

    Rule-based redaction runs before ingestion. External models only receive redacted text, and images are never sent out.

03

Engineering

03.1Engineering

  • Python + FastAPI; SQLite FTS5, jieba BM25, weighted RRF, rerank, vector retrieval (cloud embedding); Pydantic.
  • Web app with authentication, audit logging, Q&A history and screenshots attached to answers.
  • Circuit breaker on cloud-service failure, falling back to local retrieval.
  • Two-stage Docker image and a compose deployment setup.
  • Evaluation: auto-generated reverse questions, a frozen gold set, an independent held-out set and a red-team set; rule-based verdicts and human review are recorded separately, and repeated replays separate model variance from real regressions.
  • 2200 tests passing, 89% line coverage, about 98% on the core retrieval and Q&A modules.
  • Quality gates: ruff, pyright, pytest and a secret-leak scan.

03.2Results

Recall@5 · 457 auto eval questions · multi-path + vector + rerank
0.993
tests passed
2200
line coverage · core modules ~98%
89%

Retrieval ablation (457 auto-generated eval questions, Recall@5): vectors only 0.755 → local multi-path 0.976 → multi-path + vectors + rerank 0.993.

Gold set: Recall@5 0.983, Recall@10 1.000.

Passed internal pilot acceptance.

04

Notes

04.1What I Learned

Rerank is not a free win: it pushes down literal matches, so it needs a multi-path agreement bypass as a backstop.

Evaluation has to tell model variance apart from real regressions: replay several times, and record rule-based verdicts and human review separately.

04.2Tech Stack

  • Python
  • FastAPI
  • SQLite FTS5
  • jieba BM25
  • Weighted RRF
  • Rerank
  • Vector Retrieval
  • Cloud Embedding
  • Pydantic
  • Pytest
  • Docker
  • RAG