feat: complete M1-M3 infrastructure — OSS client, native execution chain, rule lifecycle API, system docs
- M1: unified OSS client (upload/download/presign) + path utils + config - M2: rule service with validate/create/publish/rollback + binding CRUD endpoints - M3: native AuditCtx runner, file/rule resolvers, storage adapter with full persistence - docs: SYSTEM_OVERVIEW.md as comprehensive architecture reference - fix: double finalize — terminal state now written once by finalize_run
This commit is contained in:
@@ -0,0 +1,193 @@
|
||||
# LeAudit Platform — 系统现状总览
|
||||
|
||||
> 最后更新:2026-04-28
|
||||
|
||||
## 一、目标架构
|
||||
|
||||
```
|
||||
┌─ API ───────────────────────────────────────────────────────────┐
|
||||
│ AuditController (/audit) RuleController (/rule-sets) │
|
||||
│ POST /run 触发评查 GET / 规则集列表 │
|
||||
│ GET /run/:id 查询状态 GET /{type}/versions 版本列表 │
|
||||
│ GET /result/:id 查询结果 GET /versions/:id/content 正文│
|
||||
│ POST /{type}/validate 校验 │
|
||||
│ POST /{type}/versions 创建 │
|
||||
│ POST /{type}/publish 发布 │
|
||||
│ POST /{type}/rollback 回滚 │
|
||||
│ GET /bindings 绑定列表 │
|
||||
│ POST /{type}/bindings 创建绑定│
|
||||
│ PUT /bindings/{id} 更新绑定│
|
||||
│ DELETE /bindings/{id} 删除绑定│
|
||||
├─ Service ───────────────────────────────────────────────────────┤
|
||||
│ AuditServiceImpl RuleServiceImpl + OssServiceImpl│
|
||||
├─ Bridge ────────────────────────────────────────────────────────┤
|
||||
│ FileSourceResolver RuleVersionResolver RulesLoader │
|
||||
│ AuditCtxBuilder AuditServiceFactory NativeRunner │
|
||||
│ StorageAdapter ResultAdapter RuleValidator │
|
||||
│ tasks.py (dispatch_leaudit_task) │
|
||||
├─ leaudit (不改) ────────────────────────────────────────────────│
|
||||
│ AuditCtx → AuditService.audit() → OCR/Normalize/Extract/ │
|
||||
│ Evaluate/Rescue → 填充 ctx.normalized_doc/extraction/ │
|
||||
│ evaluation/fallback_tasks/timing/extraction_errors │
|
||||
├─ 存储 ──────────────────────────────────────────────────────────┤
|
||||
│ OSS (MinIO) PostgreSQL │
|
||||
│ bdocs/{region}/{type}/{doc_id}/ leaudit_documents │
|
||||
│ rules/{rule_type}/{version_no}/ leaudit_document_files │
|
||||
│ artifacts/{region}/{run_id}/ leaudit_audit_runs │
|
||||
│ leaudit_rule_sets │
|
||||
│ leaudit_rule_versions │
|
||||
│ leaudit_rule_type_bindings │
|
||||
│ leaudit_rule_results │
|
||||
│ leaudit_field_results │
|
||||
│ leaudit_run_metrics │
|
||||
│ leaudit_run_errors │
|
||||
│ leaudit_rescue_outcomes │
|
||||
│ leaudit_artifacts │
|
||||
└─────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## 二、两条核心数据流
|
||||
|
||||
### 流 A:规则生命周期
|
||||
|
||||
```
|
||||
编辑 YAML
|
||||
→ POST /rule-sets/{type}/validate → RuleValidator (YAML语法 + DSL语义)
|
||||
→ POST /rule-sets/{type}/versions → 上传 OSS + INSERT leaudit_rule_versions
|
||||
→ POST /rule-sets/{type}/publish → UPDATE leaudit_rule_sets.current_version_id
|
||||
→ POST /rule-sets/{type}/bindings → INSERT leaudit_rule_type_bindings
|
||||
→ 新 run 自动绑定新版本到对应文档类型
|
||||
```
|
||||
|
||||
### 流 B:评查执行
|
||||
|
||||
```
|
||||
POST /audit/run { documentId }
|
||||
→ AuditServiceImpl.Run()
|
||||
1. 查 leaudit_documents + leaudit_document_files
|
||||
2. 查 leaudit_rule_type_bindings → rule_set_id + rule_version_id
|
||||
3. INSERT leaudit_audit_runs (rule_version_id, rule_source_oss_url, sha256)
|
||||
4. FileSourceResolver → 下载文档 bytes
|
||||
5. dispatch_leaudit_task()
|
||||
┌─ RuleVersionResolver → OSS 下载规则 YAML → SHA256 校验
|
||||
├─ RulesLoader → RulesFile
|
||||
├─ NativeRunner.run() → AuditCtx → AuditService.audit()
|
||||
└─ NativeRunner.persist_result()
|
||||
├─ save_ocr_result() → leaudit_artifacts
|
||||
├─ save_extraction_result() → leaudit_field_results
|
||||
├─ save_evaluation_results() → leaudit_rule_results + 分数
|
||||
├─ save_run_errors() → leaudit_run_errors
|
||||
├─ save_rescue_outcomes() → leaudit_rescue_outcomes
|
||||
├─ save_run_metrics() → leaudit_run_metrics
|
||||
└─ finalize_run() → 终态 (result_status/finished_at)
|
||||
```
|
||||
|
||||
## 三、模块完成度
|
||||
|
||||
### M1:OSS 基础设施 — 100%
|
||||
|
||||
| 文件 | 状态 | 说明 |
|
||||
|---|---|---|
|
||||
| `fastapi_common/fastapi_common_storage/oss_client.py` | 已完成 | 上传/下载/Presign/ObjectExists |
|
||||
| `fastapi_common/fastapi_common_storage/oss_path_utils.py` | 已完成 | BuildBusinessDocKey/ArtifactKey/RuleYamlKey |
|
||||
| `fastapi_admin/config/_settings.py` | 已完成 | OSS_ENDPOINT/ACCESS_KEY/SECRET_KEY/BUCKET/REGION/USE_SSL/PRESIGN_EXPIRE |
|
||||
| `fastapi_admin/config/__init__.pyi` | 已完成 | 类型声明 |
|
||||
|
||||
### M2:规则管理后端 — 100%
|
||||
|
||||
| 文件 | 状态 | 说明 |
|
||||
|---|---|---|
|
||||
| `controllers/ruleController.py` | 已完成 | 11 个端点(含绑定管理) |
|
||||
| `services/ruleService.py` | 已完成 | IRuleService 接口 |
|
||||
| `services/impl/ruleServiceImpl.py` | 已完成 | 规则 CRUD + 绑定 CRUD |
|
||||
| `leaudit_bridge/ruleValidator.py` | 已完成 | YAML 语法 + DSL 语义校验 |
|
||||
| `domian/Dto/ruleVersionCreateDto.py` | 已完成 | 创建版本 DTO |
|
||||
| `domian/Dto/ruleValidateDto.py` | 已完成 | 校验 DTO |
|
||||
| `domian/Dto/rulePublishDto.py` | 已完成 | 发布/回滚 DTO |
|
||||
| `domian/Dto/ruleBindingDto.py` | 已完成 | 绑定 DTO |
|
||||
| `domian/vo/ruleVo.py` | 已完成 | RuleSetVO/VersionVO/ContentVO/ValidationVO/BindingVO |
|
||||
|
||||
### M3:执行链与持久化 — ~95%
|
||||
|
||||
| 文件 | 状态 | 说明 |
|
||||
|---|---|---|
|
||||
| `leaudit_bridge/auditCtxBuilder.py` | 已完成 | 构建原生 AuditCtx |
|
||||
| `leaudit_bridge/auditServiceFactory.py` | 已完成 | 创建 AuditService |
|
||||
| `leaudit_bridge/nativeRunner.py` | 已完成 | 原生执行入口 + persist_result |
|
||||
| `leaudit_bridge/fileSourceResolver.py` | 已完成 | 文档来源解析(接 OSS) |
|
||||
| `leaudit_bridge/ruleVersionResolver.py` | 已完成 | 规则版本解析(含 SHA256) |
|
||||
| `leaudit_bridge/storage_adapter.py` | 已完成 | 全部持久化方法(已修双重 finalize) |
|
||||
| `leaudit_bridge/tasks.py` | 已完成 | 任务入口 + 失败处理 |
|
||||
| `leaudit_bridge/pipeline.py` | 待退役 | 旧管线(已被 nativeRunner 替代) |
|
||||
| `leaudit_bridge/client_factory.py` | 90% | OCR/LLM/VLM 客户端工厂 |
|
||||
|
||||
### M4:全流程联调 — 0%
|
||||
|
||||
| 项目 | 状态 |
|
||||
|---|---|
|
||||
| 上传入口梳理 | 待做 |
|
||||
| 上传后自动触发评查 | 待做 |
|
||||
| 结果查询展示 | 待做 |
|
||||
| 联调样例准备 | 待做 |
|
||||
| E2E 验证 | 待做 |
|
||||
|
||||
## 四、持久化执行顺序与终态
|
||||
|
||||
`persist_result()` 严格按以下顺序执行,`finalize_run` 在最后是唯一的终态写入者:
|
||||
|
||||
```
|
||||
save_ocr_result()
|
||||
save_extraction_result()
|
||||
save_evaluation_results() ← 只写分数/计数,不写 finished_at
|
||||
save_run_errors()
|
||||
save_rescue_outcomes()
|
||||
save_run_metrics()
|
||||
finalize_run() ← 唯一写 result_status / finished_at / rescue_applied / phase
|
||||
```
|
||||
|
||||
**终态来源(直接从 AuditCtx 读取,不另存平行状态):**
|
||||
|
||||
| 平台表 | 来源 |
|
||||
|---|---|
|
||||
| `leaudit_run_metrics` | `ctx.timing` |
|
||||
| `leaudit_run_errors` | `ctx.extraction_errors` + `ctx.extraction.all_errors` |
|
||||
| `leaudit_rescue_outcomes` | `ctx.fallback_tasks` |
|
||||
| `leaudit_audit_runs.result_status` | 综合 `ctx.fallback_tasks` + `ctx.evaluation` 推导 |
|
||||
| `leaudit_audit_runs.finished_at` | `finalize_run` 写入 now() |
|
||||
| `leaudit_audit_runs.rescue_applied` | `bool(ctx.fallback_tasks)` |
|
||||
|
||||
## 五、数据库表关系
|
||||
|
||||
```
|
||||
leaudit_document_types
|
||||
└── leaudit_rule_type_bindings (doc_type_id → rule_set_id)
|
||||
└── leaudit_rule_sets (current_version_id)
|
||||
└── leaudit_rule_versions (oss_url, sha256)
|
||||
└── leaudit_audit_runs (rule_version_id, rule_source_oss_url)
|
||||
├── leaudit_rule_results
|
||||
├── leaudit_field_results
|
||||
├── leaudit_run_metrics
|
||||
├── leaudit_run_errors
|
||||
├── leaudit_rescue_outcomes
|
||||
└── leaudit_artifacts
|
||||
```
|
||||
|
||||
## 六、关键设计原则
|
||||
|
||||
1. **leaudit 核心不改** — 所有定制在 bridge 层,leaudit 保持纯 Python 包
|
||||
2. **只读 AuditCtx** — 执行完只从 ctx 上读取,不自己模拟 stage
|
||||
3. **规则版本溯源** — 每个 run 绑定具体 `rule_version_id`,老 run 不受新发布影响
|
||||
4. **OSS 真源 + DB 索引** — YAML 正文存 OSS,元数据索引存 PostgreSQL
|
||||
5. **终态单点写入** — `finalize_run` 是 run 主表终态的唯一写入者
|
||||
6. **独立 session 提交** — 每个 `save_*` 独立会话,按"先子表后主表"顺序保证最终一致性
|
||||
|
||||
## 七、已知待补项
|
||||
|
||||
| 缺口 | 优先级 |
|
||||
|---|---|
|
||||
| `run_metrics.llm_call_count` / `vlm_call_count` 仍为空 | P1(可从 RescueTask 汇总) |
|
||||
| `ctx.timing` 缺 normalize/rescue 独立 key | P2(等 leaudit 原生补) |
|
||||
| 文档状态更新逻辑分散在两处 | P1(应收敛到 StorageAdapter) |
|
||||
| Celery 异步化 | P2(当前同步可跑通,生产需异步) |
|
||||
| `_TYPE_ID_RULES_MAP` 硬编码兜底 | P2(等 bindings 全覆盖后移除) |
|
||||
| M4 E2E 联调 | P0(下个里程碑) |
|
||||
Reference in New Issue
Block a user