结构化与非结构化资料:处理、索引和检索全流程

Structured + unstructured ingestion · indexing · search · source evidence

结构化与非结构化资料:处理、索引和检索全流程Structured records and unstructured documents follow distinct ingestion paths, converge into a versioned catalog and search projections, then a query path applies authorization filters, retrieves lexical and optional vector candidates, fuses and reranks them, and fetches source evidence. 结构化与非结构化资料:处理、索引和检索全流程 SEPARATE INGESTION PATHS · VERSIONED CATALOG · SEARCH PROJECTIONS · AUTHORIZED EVIDENCE 01 · STRUCTURED DATA 02 · UNSTRUCTURED CONTENT 03 · SEARCH PROJECTIONSBM25 · METADATA FILTER · DENSE ANN — PARALLEL INDEXES 04 · QUERY AND EVIDENCE CDC / ingest validate persist archive extract normalize parallel index build question identity scope candidates evidence feedback query projections STRUCTURED 结构化来源 数据库 · 行情 · Vendor事件 · CSV · 数据表 INGEST CDC / API / Batch 增量水位 · 去重Schema Registry QUALITY Schema 与质量 类型校验 · 单位归一实体映射 · 时效验证 CATALOG 结构化事实 Parquet / Iceberg / SQL业务主键 · 时间维度 UNSTRUCTURED 非结构化来源 PDF · Office · Email · Web扫描件 · 音频 · 会议 RAW S3 原始归档 原件 · checksum · version来源许可 · 采集时间 EXTRACT 内容抽取 Parser · OCR · ASR表格 · 标题 · 页码 · 时间戳 NORMALIZE 规范化与 Chunk 去重 · Entity · ACL段落 / 表格 / 时间片 DOCUMENT CATALOG 统一目录与版本 Document / Version / ChunkSource · Entity · ACL · Lineage LEXICAL BM25 / Full-text 所有合格文本倒排索引 · 字段权重 FILTER Structured Filters 时间 · Tenant · SourceCountry · Market · ACL DENSE Dense ANN(可选) 按查询评测选择 ChunkModel / Dimension Version REQUEST 用户 / Agent 自然语言问题结构化条件(可选) API Search API 鉴权 · 频控 · TraceQuery normalize POLICY 权限与查询范围 强制 ACL / Tenant时间 · 来源 · 市场 RETRIEVE 并行召回 BM25 + StructuredDense ANN(如启用) FUSION RRF / Rerank 去重 · 融合 · 重排Top-K + provenance EVIDENCE 证据回取 Chunk / Page / TimestampCatalog + S3 original FEEDBACK 评测与反馈 点击 / 相关性标签索引修订 · 回放 Embedding is an optional projection, not a prerequisite for document ingestion. Keep the original and provenance independently recoverable.