yuanshuai/storage-labos
1
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

chore(repo): initialize team collaboration repository
CI / Python 3.12 (push) Waiting to run
Details
CI / Python 3.9 (push) Waiting to run
Details

Browse Source
This commit is contained in:
yuanshuai
2026-07-27 20:40:12 +08:00
commit c91a64fddb
109 changed files with 21121 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
flashops/.gitignore
+26
View File
@@ -0,0 +1,26 @@
# Python
.venv/
__pycache__/
*.py[cod]
.pytest_cache/
.mypy_cache/
*.egg-info/
# Node
node_modules/
.next/
out/
.turbo/
# 本地运行产物（开发态的 SQLite 库、对象存储、日志）
var/
*.db
*.db-journal
*.log
# 环境
.env
.env.local
# macOS
.DS_Store
flashops/Makefile
+74
View File
@@ -0,0 +1,74 @@
.PHONY: help setup setup-py setup-web db-reset seed demo dev dev-api dev-web dev-agent dev-agent-loop test test-py e2e clean
SHELL := /bin/bash
ROOT := $(CURDIR)
PY := "$(ROOT)/.venv/bin/python"
PIP := "$(ROOT)/.venv/bin/pip"
CP := $(ROOT)/services/control-plane
AGENT := $(ROOT)/services/host-agent
CONSOLE := $(ROOT)/../前端UI八页面完成
export PYTHONPATH := $(CP):$(AGENT)
help:
@echo "FlashOps —— 常用命令"
@echo ""
@echo " make setup 建 venv、装依赖、建库、灌种子数据"
@echo " make demo 无硬件跑通一整条固件 A/B 回归（含故障注入与恢复）"
@echo " make dev 起控制平面与同源控制台 :8000"
@echo " make dev-agent 向已启动的控制平面注册并发送一次真实 Agent 心跳"
@echo " make test 后端单测 + 端到端 smoke"
@echo " make db-reset 删库重建并重灌种子"
@echo " make clean 清理本地运行产物"
setup: setup-py db-reset
@echo ""
@echo "✅ 就绪。下一步：make demo（看它跑）或 make dev（起服务）"
setup-py:
@test -d .venv || python3 -m venv .venv
@$(PIP) install --quiet --upgrade pip
@$(PIP) install --quiet -r services/control-plane/requirements.txt
@echo "✅ 后端依赖就绪"
db-reset:
@rm -f "$(ROOT)/var/flashops.db"
@rm -rf "$(ROOT)/var/objects"
@mkdir -p "$(ROOT)/var/objects"
@$(PY) -m flashops_control.cli init-db
@$(PY) -m flashops_control.cli seed
@echo "✅ 数据库已重建并灌入种子数据"
seed:
@$(PY) -m flashops_control.cli seed
demo:
@$(PY) -m flashops_control.cli demo
dev:
@echo "控制台 http://localhost:8000/console/Dashboard.dc.html · API http://localhost:8000/docs"
@$(PY) -m uvicorn flashops_control.main:app --host 0.0.0.0 --port 8000 --reload
dev-api:
@$(PY) -m uvicorn flashops_control.main:app --host 0.0.0.0 --port 8000 --reload
dev-web:
@cd "$(CONSOLE)" && "$(PY)" -m http.server 3000
dev-agent:
@$(PY) -m flashops_agent --server http://127.0.0.1:8000 --once
dev-agent-loop:
@$(PY) -m flashops_agent --server http://127.0.0.1:8000
test: test-py
test-py:
@"$(ROOT)/.venv/bin/pytest" services/control-plane/tests services/host-agent/tests -q
e2e:
@"$(ROOT)/.venv/bin/pytest" services/control-plane/tests/test_e2e_vertical_slice.py -q -s
clean:
@rm -rf "$(ROOT)/var" "$(ROOT)/.pytest_cache"
@find "$(ROOT)" -name __pycache__ -type d -prune -exec rm -rf {} +
@echo "✅ 已清理"
flashops/README.md
+103
View File
@@ -0,0 +1,103 @@
# FlashOps — 存储实验室无人值守执行层
> 固件回归测试的**控制平面**：把一条真实 SOP 变成可执行、可恢复、可取证的状态机，
> 让工程师不必通宵值守。
**命名是临时的。** `FlashOps` 只是后端代号，控制台目前仍显示 `STORAGE LABOS`；
商标与最终品牌确认后，需要统一后端常量、控制台标题和文档。
---
## 当前可运行基线
现在已有一条**无需硬件即可端到端跑通**的真实纵向链路：
```
建任务 → 六项安全预检 → 资源加锁 → N 次模拟循环
→ 注入 Agent 心跳丢失 → L1/L2 失败 → L3 带外 Reset 恢复
→ 从循环检查点续跑 → Evidence Bundle 打包 → 前端实时可见
```
全程使用内置模拟 DUT、测试主机与带外控制器，一条命令即可完成。当前实现重点是
事件溯源写路径、安全门禁、状态转移、恢复记录和可复现演示；独立 Host Agent
已经完成注册、token 与心跳最小链路，但步骤租约、真实硬件适配器、失败聚类执行逻辑
与完整权限体系仍是后续工作，不在此基线中伪装完成。
---
## 快速开始
前置：Python 3.9+。**不需要 Node 或 Docker。**
```bash
make setup # 建 venv、装依赖、建库、灌种子数据
make demo # 跑一条 8 轮模拟回归（含故障注入、恢复与证据包）
make dev # 起控制平面与控制台 :8000
make dev-agent # 注册独立 Host Agent 并发送一次心跳
make test # 控制平面 + Agent + API 端到端测试
```
`make demo` 结束会打印 Run ID 与 Evidence Bundle 路径；`make dev` 后打开
<http://localhost:8000/console/Tasks.dc.html>，运行预检并创建任务，再进入“实时运行”
即可看到同一条 Run 的事件时间线。API 文档在 <http://localhost:8000/docs>。
---
## 仓库结构
```
flashops/
├─ docs/ 架构与规范（先读这里）
│ ├─ architecture.md 五层架构 → 代码目录的逐层映射
│ ├─ domain-model.md 7 个核心领域对象 → 数据库表
│ ├─ state-machine.md Run/Step 状态机、恢复阶梯、检查点语义
│ ├─ adapter-sdk.md 工具适配器契约（产品扩张的关键边界）
│ ├─ workflow-spec.md SOP → YAML 工作流的 schema
│ └─ decisions/ ADR：为什么这么选，以及什么时候该推翻
├─ services/
│ ├─ control-plane/ FastAPI 控制平面（事件、安全、模拟执行、证据）
│ └─ host-agent/ 独立拉模式 Agent（注册、凭据、心跳）
├─ ../前端UI八页面完成/ 控制台原型；由 FastAPI 挂载到 /console
├─ docker-compose.yml 目标生产拓扑草案（尚未作为发布物验收）
└─ Makefile
```
---
## 三条必须守住的架构红线
写后续代码时，这三条一旦破了，返工成本最高：
1. **控制面与数据面分离**——任务状态的唯一真相在控制平面数据库，测试主机本地只有缓存。
Agent 挂了、主机重装了，Run 状态不丢。
2. **事件溯源**——所有状态变化以 append-only 事件写入 `run_events`，
当前状态是事件的投影而非独立真相。时间线、审计、复现全部由此派生。
*禁止*绕过 `events/recorder.py` 直接 UPDATE Run 状态。
3. **安全默认拒绝**——当前序列号、破坏性授权、系统盘、固件型号、Host 与 OOB
任一不满足即在 `safety/` 层拒绝。真实 Agent 接入前还必须补签名命令模板，
保证 Agent 只执行控制平面批准的动作。
---
## 当前状态与下一步
| 能力 | 状态 | 说明 |
|---|---|---|
| 14 张持久化表 | ✅ 已落地 | SQLite 建表与种子数据可重复执行 |
| Run 事件溯源 | ✅ 已落地 | 状态只能经 `events/recorder.py` 投影 |
| Run 状态转移 | ✅ 已落地 | 纯函数转移表，终态不可恢复 |
| 六项安全门禁 | ✅ 已落地 | 序列号、破坏性授权、系统盘、固件、Host、OOB |
| L1–L5 恢复决策 | ✅ 已落地 | 纯函数；演示链路实际执行 L1–L3 |
| 循环检查点续跑 | ✅ 模拟落地 | 循环边界保存，恢复后续跑 |
| Evidence Bundle | ✅ 最小实现 | `manifest.json` + `events.ndjson`，完整率 100% |
| 控制台 | 🟨 部分接入 | 任务中心与实时运行接 API；其余页面仍为静态样例 |
| 独立 Host Agent | 🟨 部分落地 | 注册、token 摘要、心跳、主机探针与 15s/30s 状态老化已实现；Step 租约待实现 |
| 真实 nvme-cli 与 OOB | ⬜ 待实现 | 需要可牺牲 DUT 与 JetKVM/PDU |
| 失败签名执行与聚类 | ⬜ 待实现 | 数据表已定义，服务逻辑未接入 |
| 多工位调度、认证与权限 | ⬜ 待实现 | 进入真实实验室前必须补齐 |
下一步建议实现 Step 租约、ACK、续租、完成上报与幂等事件，再把内置模拟执行器迁到
独立 Agent；完成这层后再接真实 `nvme-cli` 与 JetKVM，才能保证硬件接入不绕过
控制面安全规则。
flashops/deploy/README.md
+35
View File
@@ -0,0 +1,35 @@
# FlashOps production deployment
- Domain: `flashops.imagebrewing.com`
- App root: `/data/wangzhan/app/storage-labos`
- Backend bind: `127.0.0.1:18080`
- Process manager: `systemd` unit `flashops.service`
- Reverse proxy: Baota Nginx vhost `flashops.imagebrewing.com.conf`
- Persistent data: `/data/wangzhan/app/storage-labos/flashops/var`
- Private environment file: `/etc/flashops/flashops.env`
The service deliberately uses one worker because the current production store is
SQLite. The current preview console is anonymously readable and writable through
Nginx; do not connect real hardware or production data until application RBAC and
approval gates are implemented. The ACME challenge path remains available so
certificate renewal can complete.
Before enabling Agent enrollment, create the environment file without putting
the secret in the repository or unit:
```bash
sudo install -d -o root -g root -m 0755 /etc/flashops
sudo install -o root -g root -m 0600 /dev/null /etc/flashops/flashops.env
sudoedit /etc/flashops/flashops.env
```
Add `FLASHOPS_AGENT_ENROLLMENT_TOKEN=<random secret>`, then restart the service.
Useful checks:
```bash
sudo systemctl status flashops
curl http://127.0.0.1:18080/api/v1/health
sudo nginx -t
sudo certbot renew --dry-run
```
flashops/deploy/certbot-reload-nginx.sh
+4
View File
@@ -0,0 +1,4 @@
#!/bin/sh
set -eu
/usr/bin/systemctl reload nginx
flashops/deploy/flashops.service
+31
View File
@@ -0,0 +1,31 @@
[Unit]
Description=FlashOps control plane
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=ubuntu
Group=ubuntu
WorkingDirectory=/data/wangzhan/app/storage-labos/flashops
EnvironmentFile=-/etc/flashops/flashops.env
Environment=PYTHONPATH=/data/wangzhan/app/storage-labos/flashops/services/control-plane:/data/wangzhan/app/storage-labos/flashops/services/host-agent
Environment=FLASHOPS_ENV=production
Environment=FLASHOPS_API_HOST=127.0.0.1
Environment=FLASHOPS_API_PORT=18080
Environment=FLASHOPS_DATABASE_URL=sqlite+aiosqlite:////data/wangzhan/app/storage-labos/flashops/var/flashops.db
Environment=FLASHOPS_OBJECT_STORE_URL=/data/wangzhan/app/storage-labos/flashops/var/objects
Environment=FLASHOPS_CORS_ORIGINS=https://flashops.imagebrewing.com
ExecStart=/data/wangzhan/app/storage-labos/flashops/.venv/bin/uvicorn flashops_control.main:app --host 127.0.0.1 --port 18080 --workers 1 --proxy-headers --forwarded-allow-ips=127.0.0.1
Restart=on-failure
RestartSec=3
TimeoutStopSec=15
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectHome=true
ReadWritePaths=/data/wangzhan/app/storage-labos/flashops/var
UMask=0027
[Install]
WantedBy=multi-user.target
flashops/deploy/gitea/README.md
+41
View File
@@ -0,0 +1,41 @@
# Gitea deployment
The team Git service runs as Gitea 1.27.0 on the existing CVM:
- public URL: `https://git.imagebrewing.com`;
- container HTTP bind: `127.0.0.1:13000`;
- persistent data: `/data/gitea/data`;
- database: Gitea-managed SQLite;
- Git transport: HTTPS only; no extra public SSH port;
- anonymous users may browse public repositories;
- account registration requires administrator approval before sign-in.
- daily backup target: `/chucun/wangzhan-production/backups/gitea/`.
The service is intentionally separate from the FlashOps runtime and database.
Back up `/data/gitea/data` before an image upgrade. Pin the image version and
review the Gitea release notes before changing it.
The current server does not have the Docker Compose plugin. The checked-in
`docker-compose.yml` is the declarative service record for future rebuilds;
the running container was created with the equivalent pinned `docker run`
settings. Routine checks and restarts therefore use Docker directly:
```bash
sudo docker ps --filter name=^gitea$
sudo docker restart gitea
sudo docker logs --tail 100 gitea
curl -fsS http://127.0.0.1:13000/api/healthz
```
Install Compose or recreate the container from the checked-in definition only
during a planned maintenance window, after a verified backup.
Nginx terminates HTTPS and proxies to the loopback-only container port. Run
`nginx -t` before every reload. Credentials and API tokens are never stored in
this directory or committed to Git.
`gitea-backup.timer` runs daily at 03:20 Asia/Shanghai with a randomized delay.
It uses Gitea's own dump command, writes the ZIP and SHA-256 sidecar to the
mounted COS bucket, and removes the temporary local dump after a successful copy.
The live repository and SQLite database remain on the local filesystem because
COSFS is a backup destination, not a POSIX database filesystem.
flashops/deploy/gitea/backup.sh
+25
View File
@@ -0,0 +1,25 @@
#!/usr/bin/env bash
set -euo pipefail
backup_dir=/chucun/wangzhan-production/backups/gitea
container_dump=/data/gitea-backup.zip
host_dump=/data/gitea/data/gitea-backup.zip
stamp=$(date -u +%Y%m%dT%H%M%SZ)
target="$backup_dir/gitea-$stamp.zip"
install -d -o ubuntu -g ubuntu -m 0770 "$backup_dir"
if [ -e "$host_dump" ]; then
unlink "$host_dump"
fi
docker exec -u git gitea sh -c \
'mkdir -p /tmp/gitea-dump && find /tmp/gitea-dump -mindepth 1 -delete'
docker exec -u git gitea gitea dump \
--config /data/gitea/conf/app.ini \
--file "$container_dump" \
--tempdir /tmp/gitea-dump
cp "$host_dump" "$target"
sha256sum "$target" > "$target.sha256"
unlink "$host_dump"
printf 'Gitea backup created: %s\n' "$target"
flashops/deploy/gitea/docker-compose.yml
+28
View File
@@ -0,0 +1,28 @@
services:
gitea:
image: gitea/gitea:1.27.0
container_name: gitea
restart: unless-stopped
environment:
USER_UID: "1000"
USER_GID: "1000"
GITEA__database__DB_TYPE: sqlite3
GITEA__database__PATH: /data/gitea/gitea.db
GITEA__server__DOMAIN: git.imagebrewing.com
GITEA__server__ROOT_URL: https://git.imagebrewing.com/
GITEA__server__HTTP_PORT: "3000"
GITEA__server__DISABLE_SSH: "true"
GITEA__server__OFFLINE_MODE: "true"
GITEA__service__DISABLE_REGISTRATION: "false"
GITEA__service__REGISTER_MANUAL_CONFIRM: "true"
GITEA__service__REQUIRE_SIGNIN_VIEW: "false"
GITEA__service__DEFAULT_ALLOW_CREATE_ORGANIZATION: "false"
GITEA__repository__DEFAULT_PRIVATE: public
GITEA__security__INSTALL_LOCK: "true"
GITEA__cron__ENABLED: "true"
GITEA__log__MODE: console
GITEA__log__LEVEL: Info
ports:
- "127.0.0.1:13000:3000"
volumes:
- /data/gitea/data:/data
flashops/deploy/gitea/gitea-backup.service
+11
View File
@@ -0,0 +1,11 @@
[Unit]
Description=Back up Gitea to mounted COS bucket
Requires=docker.service
After=docker.service network-online.target
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/flashops-gitea-backup
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7
flashops/deploy/gitea/gitea-backup.timer
+11
View File
@@ -0,0 +1,11 @@
[Unit]
Description=Daily Gitea backup timer
[Timer]
OnCalendar=*-*-* 03:20:00 Asia/Shanghai
RandomizedDelaySec=10m
Persistent=true
Unit=flashops-gitea-backup.service
[Install]
WantedBy=timers.target
flashops/deploy/gitea/nginx-http.conf
+25
View File
@@ -0,0 +1,25 @@
server {
listen 80;
server_name git.imagebrewing.com;
access_log /www/wwwlogs/git.imagebrewing.com.log;
error_log /www/wwwlogs/git.imagebrewing.com.error.log;
location ^~ /.well-known/acme-challenge/ {
root /var/www/html;
default_type text/plain;
}
location / {
proxy_pass http://127.0.0.1:13000;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 300s;
proxy_send_timeout 300s;
}
}
flashops/deploy/gitea/nginx.conf
+47
View File
@@ -0,0 +1,47 @@
server {
listen 80;
server_name git.imagebrewing.com;
location ^~ /.well-known/acme-challenge/ {
root /var/www/html;
default_type text/plain;
}
location / {
return 301 https://$host$request_uri;
}
}
server {
listen 443 ssl;
http2 on;
server_name git.imagebrewing.com;
ssl_certificate /etc/letsencrypt/live/git.imagebrewing.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/git.imagebrewing.com/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_session_timeout 1d;
ssl_session_cache shared:GiteaSSL:10m;
ssl_session_tickets off;
access_log /www/wwwlogs/git.imagebrewing.com.log;
error_log /www/wwwlogs/git.imagebrewing.com.error.log;
client_max_body_size 100m;
add_header X-Content-Type-Options nosniff always;
add_header Referrer-Policy strict-origin-when-cross-origin always;
add_header X-Frame-Options SAMEORIGIN always;
location / {
proxy_pass http://127.0.0.1:13000;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 300s;
proxy_send_timeout 300s;
}
}
flashops/deploy/nginx-http.conf
+26
View File
@@ -0,0 +1,26 @@
server {
listen 80;
server_name flashops.imagebrewing.com;
access_log /www/wwwlogs/flashops.imagebrewing.com.log;
error_log /www/wwwlogs/flashops.imagebrewing.com.error.log;
location ^~ /.well-known/acme-challenge/ {
auth_basic off;
root /var/www/html;
default_type text/plain;
}
location / {
proxy_pass http://127.0.0.1:18080;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 300s;
proxy_send_timeout 300s;
}
}
flashops/deploy/nginx-https.conf
+47
View File
@@ -0,0 +1,47 @@
server {
listen 80;
server_name flashops.imagebrewing.com;
location ^~ /.well-known/acme-challenge/ {
auth_basic off;
root /var/www/html;
default_type text/plain;
}
location / {
return 301 https://$host$request_uri;
}
}
server {
listen 443 ssl;
http2 on;
server_name flashops.imagebrewing.com;
ssl_certificate /etc/letsencrypt/live/flashops.imagebrewing.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/flashops.imagebrewing.com/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_session_timeout 1d;
ssl_session_cache shared:FlashOpsSSL:10m;
ssl_session_tickets off;
access_log /www/wwwlogs/flashops.imagebrewing.com.log;
error_log /www/wwwlogs/flashops.imagebrewing.com.error.log;
add_header X-Content-Type-Options nosniff always;
add_header Referrer-Policy strict-origin-when-cross-origin always;
add_header X-Frame-Options SAMEORIGIN always;
location / {
proxy_pass http://127.0.0.1:18080;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_read_timeout 300s;
proxy_send_timeout 300s;
}
}
flashops/docker-compose.yml
+78
View File
@@ -0,0 +1,78 @@
# 生产形态（客户内网私有化单机部署）。
# 开发不需要它：make dev 用 SQLite + 进程内总线 + 本地目录，零外部依赖。
#
# 注意：Host Agent 不在这里。它跑在客户的测试主机上（Windows 服务 / systemd），
# 不进容器——它要碰真实设备、真实驱动、真实串口。
services:
postgres:
image: postgres:16-alpine
environment:
POSTGRES_DB: flashops
POSTGRES_USER: flashops
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?必须设置}
volumes:
- pgdata:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U flashops"]
interval: 5s
timeout: 3s
retries: 10
restart: unless-stopped
redis:
image: redis:7-alpine
command: redis-server --appendonly yes
volumes:
- redisdata:/data
restart: unless-stopped
minio:
image: minio/minio:latest
command: server /data --console-address ":9001"
environment:
MINIO_ROOT_USER: ${MINIO_USER:-flashops}
MINIO_ROOT_PASSWORD: ${MINIO_PASSWORD:?必须设置}
volumes:
- miniodata:/data
ports:
- "9001:9001" # 控制台，仅内网暴露
restart: unless-stopped
control-plane:
build:
context: .
dockerfile: services/control-plane/Dockerfile
environment:
FLASHOPS_DATABASE_URL: postgresql+asyncpg://flashops:${POSTGRES_PASSWORD}@postgres:5432/flashops
FLASHOPS_BUS_URL: redis://redis:6379/0
FLASHOPS_OBJECT_STORE_URL: s3://minio:9000/flashops
FLASHOPS_OBJECT_STORE_KEY: ${MINIO_USER:-flashops}
FLASHOPS_OBJECT_STORE_SECRET: ${MINIO_PASSWORD}
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_started
minio:
condition: service_started
ports:
- "8000:8000" # Agent 要能连上，所以对实验室网段开放
restart: unless-stopped
console:
build:
context: .
dockerfile: apps/console/Dockerfile
environment:
FLASHOPS_API_BASE: http://control-plane:8000
depends_on:
- control-plane
ports:
- "3000:3000"
restart: unless-stopped
volumes:
pgdata:
redisdata:
miniodata:
flashops/docs/adapter-sdk.md
+90
View File
@@ -0,0 +1,90 @@
# Adapter SDK：产品扩张的关键边界
方案 §6.4 的判断是对的——**适配器是产品扩张的关键**。每接一个客户，
新增的工作量应该收敛到"写一个适配器"，而不是改引擎。这一波把这条边界钉死。
实现：`services/host-agent/flashops_agent/adapters/`
## 契约（7 个方法，一个都不能少）
```python
class ToolAdapter(Protocol):
name: str
version: str
def discover(self, ctx: AdapterContext) -> Capabilities: ...
def precheck(self, ctx: AdapterContext) -> CheckResult: ...
def execute(self, step: StepSpec, ctx: AdapterContext) -> ExecutionHandle: ...
def collect(self, handle: ExecutionHandle) -> EvidenceArtifact: ...
def cancel(self, handle: ExecutionHandle) -> None: ...
def health(self) -> HealthStatus: ...
def normalize_result(self, raw: RawResult) -> UnifiedResult: ...
```
| 方法 | 职责 | 什么时候被调 |
|---|---|---|
| `discover` | 报告本工具在这台主机上能干什么（版本、支持的设备、能力位） | Agent 启动 / 资产刷新 |
| `precheck` | 执行前自检：工具在不在、权限够不够、设备在不在、参数合不合法 | 每个步骤执行前，**失败即拒绝执行** |
| `execute` | 启动执行，**立即返回句柄，不阻塞** | 步骤开始 |
| `collect` | 从句柄收集产物：stdout/stderr、结果文件、设备日志 | 步骤结束 / 中途取证 |
| `cancel` | 终止执行（含进程树），保证不留孤儿进程 | 紧急停止 / 超时 / 恢复前 |
| `health` | 适配器自身健康（工具还在吗、许可证过期没） | 心跳周期 |
| `normalize_result` | 把工具原生输出翻译成统一结果模型 | `collect` 之后 |
## `normalize_result` 是最重要的一个
它决定了失败签名能不能跨工具聚类。原生输出千奇百怪，统一模型只有一种：
```python
@dataclass
class UnifiedResult:
outcome: Literal["PASS", "FAIL", "ERROR", "TIMEOUT"]
error_codes: List[str] # 归一化后的错误码，如 ["NVME_STATUS_0x2002"]
log_templates: List[str] # 日志模板化后的指纹，数字/路径已替换为占位符
metrics: Dict[str, float] # iops / latency_p99 / temperature_c ...
device_state: DeviceState # 枚举状态、固件版本、SMART 关键项
integrity: IntegrityState # OK / MISMATCH / NOT_CHECKED
artifacts: List[str] # 产物 URI
```
**适配器的实现者只需要保证一件事**：同样的故障，`error_codes` 与
`log_templates` 要稳定。不稳定 → 签名散开 → 聚类失效 → 同一个问题重复提单，
客户第一时间就会发现这个系统在制造噪音。
日志模板化规则在 `flashops_agent/adapters/templating.py`：
数字 → `<N>`，十六进制 → `<HEX>`，路径 → `<PATH>`，UUID → `<UUID>`，
时间戳 → `<TS>`。所以
`"nvme0n1: I/O error, sector 12345678 at 2026-07-27T03:14:15"`
归一为 `"nvme<N>n<N>: I/O error, sector <N> at <TS>"`。
## 这一波带的适配器
| 适配器 | 状态 | 说明 |
|---|---|---|
| `ShellAdapter` | ✅ 真实 | 执行签名命令模板，进程树管理、超时、输出捕获 |
| `NvmeCliAdapter` | ⚠️ 真实骨架 + 模拟回退 | 命令与解析是真的；无真实设备时走模拟器 |
| `FioAdapter` | ⚠️ 真实骨架 + 模拟回退 | 解析 fio `--output-format=json` |
| `SimulatedDutAdapter` | ✅ 模拟 | 可注入故障的假 DUT，让全链路无硬件可跑 |
| `VendorFlashAdapter` | ⬜ 占位 | 客户私有刷写工具——**这个必须等拿到真实 SOP 再写** |
`VendorFlashAdapter` 故意留空并在导入时抛 `NotImplementedError`。
猜客户的刷写工具长什么样是纯浪费——报告 Phase 0 的结论是先拿 SOP 再写代码。
## 写一个新适配器的清单
1. 继承 `BaseAdapter`，实现 7 个方法
2. 在 `adapters/__init__.py` 的 `REGISTRY` 里注册（key 就是工作流 YAML 里的 `adapter:`）
3. 危险命令**必须**在控制平面 `safety/templates.py` 注册签名模板；
适配器里不允许拼接任意命令行——参数只能来自模板 schema 校验过的字典
4. 写一个 `tests/adapters/test_<name>.py`：至少覆盖
`precheck` 失败路径、超时路径、`normalize_result` 的两个不同故障输出
5. 跑 `make test` 里的适配器契约测试（`test_adapter_contract.py` 会对
`REGISTRY` 里每个适配器自动断言 7 个方法齐全且签名正确）
## 边界纪律
- 适配器**不知道** Run、Workflow、状态机的存在。它只认 `StepSpec` 和 `AdapterContext`。
- 适配器**不做**重试和恢复决策。挂了就如实报告，重试与恢复是控制平面的事。
- 适配器**不写**数据库，产物写本地临时目录，由 Agent 上传。
- 适配器里**不允许**出现 `if customer == "X"` 这种分支。客户差异靠不同适配器 +
工作流参数表达，不靠代码里的 if。
flashops/docs/agent-protocol.md
+90
View File
@@ -0,0 +1,90 @@
# 控制平面 ↔ Host Agent 协议
> **当前状态（2026-07-27）**：独立 Host Agent 已实现注册、注册口令、bearer token、
> 本地 `0600` 凭据文件、主机指纹和心跳；控制平面只保存 token SHA-256 摘要。
> 心跳迟到 15 秒会把 Host 降级为 `DEGRADED`，丢失 30 秒会标记为 `OFFLINE`；
> 安全预检会先刷新该状态，避免失联 Host 被误判为可用。
> Step 租约、事件、完成与产物端点仍是下一阶段契约。`make demo` 暂时继续使用
> 控制平面内置模拟执行器验证状态、恢复与证据链。
Agent 是**拉模式**。控制平面永远不主动连接测试主机。
理由：客户测试主机在内网/NAT 后面，很多实验室根本不给外部反连；
拉模式还顺带解决了 Agent 重启后的重新接入问题。
## 端点
| 状态 | 方法 | 路径 | 用途 |
|---|---|---|---|
| 已实现 | POST | `/api/v1/agent/register` | 首次注册 / token 轮换；上报主机指纹与工具能力 |
| 已实现 | POST | `/api/v1/agent/{agent_id}/heartbeat` | bearer 心跳，上报健康和设备探针；返回待办指令 |
| 已实现 | GET | `/api/v1/agent/hosts` | 查看 Host 与 Agent 在线投影，不返回 token |
| 待实现 | POST | `/api/v1/agent/{agent_id}/lease` | 领取下一个步骤（长轮询，最多挂 20s） |
| 待实现 | POST | `/api/v1/agent/{agent_id}/steps/{step_id}/events` | 批量上报执行事件与日志片段 |
| 待实现 | POST | `/api/v1/agent/{agent_id}/steps/{step_id}/complete` | 上报步骤终态 + UnifiedResult + 产物清单 |
| 待实现 | POST | `/api/v1/agent/{agent_id}/artifacts` | 上传产物（日志 / 结果文件 / 画面） |
带外控制器走独立端点（它必须能在主机死后独立上报）：
| 方法 | 路径 | 用途 |
|---|---|---|
| POST | `/api/v1/oob/{controller_id}/heartbeat` | 带外心跳 + 电源状态 + 温度 + 画面指纹 |
| POST | `/api/v1/oob/{controller_id}/observation` | 主动上报观测（蓝屏画面、掉电、按钮触发） |
## 心跳返回的指令
心跳响应已经带 `commands` 字段；当前固定为空数组。命令队列接入后，Agent 收到后立即执行：
```json
{
"ok": true,
"server_time": "2026-07-27T03:14:15Z",
"commands": [
{"kind": "CANCEL_STEP", "step_id": "..."},
{"kind": "SOFT_RESTART", "reason": "recovery_L1"},
{"kind": "EMERGENCY_STOP", "run_id": "..."},
{"kind": "REFRESH_ASSETS"}
]
}
```
紧急停止走心跳而不是新连接——心跳是唯一能保证到达的通道。
最坏情况延迟 = 心跳间隔（5s）。真正需要毫秒级的停止靠带外物理急停，不靠软件。
## 租约（lease）语义
```
Agent ──lease──▶ 控制平面
├─ 有活：分配步骤，step → DISPATCHED，写租约（TTL 60s）
└─ 没活：长轮询挂起最多 20s，返回 204
Agent ──ack───▶ step → RUNNING
（租约到期未 ack 或未 complete）→ step 回 PENDING，重新派发
```
租约 TTL 必须**大于**步骤超时？不——恰恰相反：租约由 Agent 在执行期间通过心跳续期。
Agent 死了 → 心跳停 → 租约到期 → 步骤回收。这是 Agent 崩溃能被发现的机制之一
（另一条是心跳超时判定，见 state-machine.md §5）。
## 认证
首次注册带 `X-FlashOps-Enrollment-Token`。开发态未配置注册口令时允许本机演示；
生产态未配置时注册接口返回 503。注册成功下发 bearer token，之后每个请求带
`Authorization: Bearer <token>`。`agent_id` 与 `test_host_id` 绑定，重新注册会轮换 token。
原始 token 仅存于 Agent 本机 `0600` state 文件；控制平面数据库只保存 SHA-256 摘要。
Agent 客户端保留可选 gateway Basic Auth 能力，供未来重新启用边界认证或部署在其他网关后使用；
当前 `flashops.imagebrewing.com` 预览环境未启用这层认证。
**下一波必须升级**：方案 §6.2 要求"每个 Agent 使用唯一证书或密钥"。
bearer token 在客户内网够用，但要进客户安全评审得上 mTLS。
升级点在 `api/deps.py::require_agent()` 一处。
## 幂等（待实现）
Step 租约接入时，所有 Agent → 控制平面的 Step 写请求必须带
`Idempotency-Key`（Agent 生成的 UUID）。控制平面需要记录并拒绝重复副作用；
`run_events` 已有唯一约束 `(run_id, idempotency_key)`，但当前注册 / 心跳链路尚未实现
通用的请求幂等存储。
这一条在真实实验室里不是可选项：主机重启、网线松动、Wi-Fi 掉线是常态，
没有幂等就会出现"一次故障记了三条"的时间线，取证时说不清。
flashops/docs/architecture.md
+99
View File
@@ -0,0 +1,99 @@
# 架构：五层 → 代码目录的逐层映射
> **实现状态（2026-07-27）**：本文描述目标架构。当前已落地控制平面的模型、
> 安全门禁、事件记录、Run 转移表、恢复决策、最小证据包、内置模拟执行器，以及
> 独立 Host Agent 的注册、token、心跳和 Host 状态老化链路；Step 租约、Adapter
> 执行、消息总线替换和生产对象存储仍未落地。
方案 §5.1 定义了五层架构。这里给出每一层**落到哪个目录、边界在哪、谁不能调用谁**。
```
┌─────────────────────────────────────────────────────────────┐
│ 交互与集成层 apps/console (Next.js) · CLI · Open API │
├─────────────────────────────────────────────────────────────┤
│ 控制平面 services/control-plane/flashops_control/ │
│ services/ 编排 safety/ 门禁 api/ 接口 │
├─────────────────────────────────────────────────────────────┤
│ 执行平面 services/host-agent/flashops_agent/ │
│ adapters/ 工具适配器 │
├─────────────────────────────────────────────────────────────┤
│ 带外平面 services/host-agent/flashops_agent/oob/ │
│ （独立进程/独立网络，模拟实现在 simulator/） │
├─────────────────────────────────────────────────────────────┤
│ 证据与智能层 control-plane/.../evidence/ · signatures/ │
│ events/ 事件总线 │
└─────────────────────────────────────────────────────────────┘
```
## 依赖方向（单向，不可逆）
```
console ──HTTP/SSE──▶ control-plane ◀──HTTP──── host-agent
│ │
│ ├─▶ adapters ─▶ 客户工具
▼ │
数据库 └─▶ oob ─▶ 带外控制器
```
**硬规则：**
- `host-agent` **不得**导入 `control-plane` 的任何模块。两者只通过 `docs/agent-protocol.md`
定义的 HTTP 契约通信。理由：Agent 要能单文件分发到客户 Windows 机器，
控制平面代码不进客户测试主机。
- `control-plane` **不得**主动连接 Agent（Agent 是拉模式）。理由：客户测试主机在
内网/NAT 后面，反向连接在真实实验室里活不下来。
- `engine/` **不得**直接读写数据库。它只操作 `Snapshot` 值对象，返回 `Effect` 列表。
持久化由 `runtime.py` 施加。理由：状态机要能不起数据库单测。
- `api/` **不得**包含业务判断。路由只做校验 + 调用 `engine`/`services`。
- 任何危险动作（刷写、Format、Sanitize、断电）**必须**先过 `safety/`，
且 `safety/` 的拒绝发生在下发给 Agent 之前。
## 控制平面内部分层
```
flashops_control/
├─ config.py 配置（产品名、DSN、路径、超时策略）
├─ db.py 引擎/会话/建表
├─ models/ SQLAlchemy 表定义（贫血模型，只管持久化）
├─ schemas.py Pydantic 出入参（当前最小 API 契约）
├─ events/
│ ├─ bus.py 进程内事件总线（生产换 NATS/Redis，接口不变）
│ └─ recorder.py 事件溯源写入：唯一允许改 Run 状态的入口
├─ engine/
│ ├─ states.py Run/Step 状态枚举 + 转移表（纯数据）
│ ├─ recovery.py 五级恢复阶梯策略（纯函数）
│ └─ （目标）planner/runtime：独立 Agent 接入后补齐
├─ safety/
│ ├─ gates.py DUT 绑定 / 系统盘保护 / 破坏性白名单
│ └─ templates.py 签名命令模板注册表 + 参数 schema 校验
├─ signatures/ （目标）失败签名（8 要素哈希）与聚类
├─ evidence/ 最小 Evidence Bundle（manifest + 事件流）
├─ services/runs.py 当前内置模拟编排（后续拆成 Agent + runtime）
├─ seed.py 可重复执行的模拟资产与工作流
└─ api/ FastAPI 路由
```
**`engine/` 的纯度是刻意的。** 当前 `states.py` / `recovery.py`
不导入 SQLAlchemy、不导入 FastAPI、不做 I/O、不看时钟
（时间从 `Snapshot.now` 传入）。这让状态机的全部分支都能用普通 pytest 覆盖，
不需要起库、不需要 sleep。后面几波会不断往状态机里加分支（新故障类型、
新恢复策略、多工位调度），这条边界是唯一能防止它烂掉的东西。
## 开发形态 vs 生产形态
| 组件 | 开发（`make dev`） | 生产（`docker-compose`） | 换的时候动哪 |
|---|---|---|---|
| 数据库 | SQLite (aiosqlite) | PostgreSQL 16 | 只改 `DATABASE_URL` |
| 队列/总线 | 进程内 asyncio | Redis / NATS | `events/bus.py` 换实现 |
| 对象存储 | `var/objects/` 本地目录 | MinIO | `storage.py` 换实现 |
| Agent | 内置模拟执行器 + 独立 Agent 心跳 | 客户主机上的 systemd/Windows 服务 | 补齐 Step lease/runtime |
| 带外 | 模拟控制器 | 树莓派 / JetKVM / 智能 PDU | `oob/` 换实现 |
三处"换实现"都在接口后面，且开发态实现本身就是测试替身——不存在"生产才发现跑不通"。
## 为什么不用 Temporal / Airflow
见 [ADR-0003](decisions/0003-own-state-machine-over-temporal.md)。一句话：
恢复语义（带外断电、冻结现场、误盘防护）是本产品的差异化本身，
不能交给通用工作流引擎的重试模型。等到多工位跨节点调度成为瓶颈时再评估 Temporal，
`runtime.py` 是唯一需要替换的文件。
flashops/docs/decisions/0001-monorepo-and-stack.md
+39
View File
@@ -0,0 +1,39 @@
# ADR-0001：Monorepo 与技术栈选型
- 日期：2026-07-27
- 状态：已接受
## 背景
第一波要同时出前端控制台、控制平面、Host Agent 三个可交付物，团队规模是 1-2 人。
## 决定
单仓库（monorepo），三个可独立部署的单元：`apps/console`、`services/control-plane`、
`services/host-agent`。技术栈按方案 §6.1：Next.js + FastAPI + PostgreSQL。
**开发态零外部依赖**：SQLite 替 Postgres、进程内 asyncio 替 Redis、
本地目录替 MinIO。三者都在接口后面，`docker-compose.yml` 给出生产形态。
## 理由
- 1-2 人团队用多仓库，跨仓改一个接口要开三个 PR，纯损耗。
- 控制平面与 Agent 的 HTTP 契约会在接第一个客户时频繁改。同仓库能一次改完、
一次跑通端到端测试。
- 开发态零依赖是为了**降低启动摩擦**：这台机器上没有 Docker，客户现场的
Windows 测试主机上也大概率装不了 Docker。能 `python -m` 直接跑起来的东西，
在实验室里活得久。
- Python 3.9 兼容：客户测试主机的 Python 版本不可控，Agent 必须往低了兼容。
所以全仓库用 `Optional[X]` 而不是 `X | None`，用 `List[X]` 而不是 `list[X]`。
## 代价
- SQLite 与 Postgres 有行为差异（并发写、JSON 查询、事务隔离）。
缓解：ORM 层不用任何方言特有类型；CI 后续要在 Postgres 上再跑一遍测试。
- monorepo 在团队超过 5 人后会需要更强的 CI 分区。届时再拆。
## 什么时候推翻
- 团队 >5 人且前后端分离开发节奏明显不同步 → 拆仓。
- Agent 需要单文件分发到无 Python 环境的 Windows → 把 Agent 用 Go 重写
（方案 §6.1 已经把这条写成演进建议）。届时 HTTP 契约不变，只换实现语言。
flashops/docs/decisions/0002-event-sourcing.md
+44
View File
@@ -0,0 +1,44 @@
# ADR-0002：以事件溯源作为 Run 的唯一真相
- 日期：2026-07-27
- 状态：已接受
## 背景
产品要交付的三件事——**统一时间线**、**审计**、**自动复现**——都需要回答同一个问题：
"当时到底按什么顺序发生了什么"。
同时，观测是**双通道**的：Agent 报一套，带外控制器报一套，两者时钟不同步，
主机断电时 Agent 那一路会直接消失。
## 决定
`run_events` 是 append-only 的唯一真相。`runs` / `run_steps` / `recovery_actions`
是可以从事件重放重建的**读模型**。
序号 `seq` 由控制平面单点分配（不是时间戳排序），Agent 事件与带外事件进同一条序列。
仓储层**不提供** `update_run_state()` 类 API。改状态的唯一入口是
`events/recorder.py::record()`，它写事件并同步更新投影。
## 理由
- 主机断电时，Agent 的最后几条日志可能永远到不了。带外控制器的观测
（掉电时刻、画面指纹）是那段时间唯一的证据。两路事件必须在同一条时间线上
才能交叉校验——"Agent 最后一条日志 seq=812，带外报掉电 seq=813"这种判断
只有单点序号能给。
- 审计要求记录"操作者、审批者、命令模板版本、输入参数、结果、撤销动作"
（方案 §6.6）。这些天然是事件，硬塞进当前状态表会做成一堆冗余字段。
- 自动复现（§8.4）需要"失败前后的最小步骤区间"。有完整事件流才能裁剪。
## 代价
- 写路径变长，每次状态变化多一次插入。
- `run_events` 会很大：100 循环 × 每轮 ~30 事件 = 3000 条/Run。
缓解：事件按 run_id 分区友好，日志正文不进事件表（只存对象存储 URI）。
- 开发者容易忍不住直接 UPDATE。缓解：仓储层不给这个 API，code review 卡这一条。
## 什么时候推翻
不推翻。真要优化，是加快照（每 N 个事件存一次状态快照加速重放），
不是放弃事件溯源。
flashops/docs/decisions/0003-own-state-machine-over-temporal.md
+51
View File
@@ -0,0 +1,51 @@
# ADR-0003：自研状态机，不用 Temporal / Airflow
- 日期：2026-07-27
- 状态：已接受（方案 §6.1 把 Temporal 列为演进建议，这里给出不在首版用的理由）
## 背景
工作流编排有成熟方案：Temporal、Airflow、Prefect、OpenTAP。
自研状态机通常是坏味道。
## 决定
首版自研：`engine/states.py` 转移表 + `engine/state_machine.py` 纯函数
+ `engine/runtime.py` 异步循环。
## 理由
产品的差异化恰好落在通用引擎**不管**的那一层：
1. **恢复语义不是重试。** Temporal 的重试模型是"再调一次这个 activity"。
我们需要的是"主机蓝屏了 → 带外触发主板 Reset → 等主机起来 → 验证 DUT
重新枚举 → 从检查点续跑"。这是跨越进程、跨越主机生死的物理恢复，
不是函数重试。硬套通用引擎会变成在 activity 里塞一堆状态判断，比自研更难维护。
2. **非幂等步骤默认不重跑。** 通用引擎的默认值是"重试是安全的"。
我们这里最贵的一条规则是"刷写步骤挂了**不要**自动重跑"。
跟框架默认值对着干，不如自己控制。
3. **冻结现场是一等状态。** 数据完整性失败要立刻停止一切覆盖性动作并保留现场。
通用引擎里这是"失败"，我们这里它是一个需要保持很久、等人来看的活状态。
4. **可测试性。** 纯函数状态机能把"30 秒心跳超时后带外在线则走 L3"这种规则
在毫秒内测完。挂在 Temporal 上要起测试环境。后面几波会疯狂往状态机加分支，
测试成本是决定性因素。
5. 依赖成本：Temporal 需要自己的数据库和服务集群。客户是**内网私有化部署**，
多一个必须运维的组件就多一个交付摩擦。
## 代价
- 跨节点调度、持久化定时器、版本化工作流这些 Temporal 白送的能力要自己做。
这一波不需要（单工位），Phase 3 多工位时会痛。
- 自研状态机容易随时间腐化成 if 堆。缓解：转移表是纯数据、`decide()` 是纯函数、
每加一个分支必须配单测，三条规矩写进了 `docs/state-machine.md §6`。
## 什么时候推翻
多工位跨节点调度成为瓶颈时（Phase 3，预计 20+ 工位）。
届时替换范围仅限 `runtime.py`——`states.py` / `state_machine.py` / `recovery.py`
是纯逻辑，可以原样搬进 Temporal 的 workflow 定义里。
这也是把它们做成纯函数的第二个理由。
flashops/docs/decisions/0004-safety-in-control-plane.md
+51
View File
@@ -0,0 +1,51 @@
# ADR-0004：安全门禁放在控制平面，Agent 保持"笨"
- 日期：2026-07-27
- 状态：已接受
## 背景
危险动作有三类：刷固件、Format/Sanitize、断电。任何一次误盘 = PoC 判定失败
（方案 §9.3：0 次错盘、0 次系统盘破坏）。
门禁可以放在 Agent（离设备近，判断准）或控制平面（离决策近，可审计）。
## 决定
**全部放控制平面**，且拒绝发生在**下发之前**。Agent 只执行已批准的命令模板 +
schema 校验过的参数，自己不做任何"这条命令安不安全"的判断。
Agent 侧保留的唯一防线是 `precheck`（设备在不在、路径对不对），
它是**执行前自检**，不是安全决策。
## 理由
- **可审计**：审批记录、模板版本、操作者、参数必须与决策在同一个地方，
否则审计链断在 Agent 上。客户安全评审第一个问的就是这个。
- **Agent 可信度低**：Agent 跑在客户的测试主机上，那台机器会蓝屏、会被人手动改配置、
会装乱七八糟的工具。安全判断不能依赖一个随时可能处于异常状态的环境。
- **升级路径**：安全规则改了，改控制平面就全网生效。要靠 Agent 判断，
就得推送 Agent 升级到每台客户主机——在实验室里这是以周计的事。
- **可测试**：门禁是纯函数（`safety/gates.py`），误盘场景可以在单测里穷举，
不需要真的接一块盘来试。
## 具体门禁（`safety/gates.py`）
1. **DUT 绑定**：序列号 + 设备路径 + `allow_destructive` 标签三信号一致才放行。
任一不匹配 → 拒绝，记 `SAFETY_REJECTED` 事件。
2. **系统盘保护**：解析目标设备的挂载点/启动标志/分区表，命中系统盘立即拒绝。
这一条**没有** override 开关，代码里不给绕过的口子。
3. **命令模板白名单**：危险命令只能来自 `safety/templates.py` 注册表，
参数受 JSON Schema 约束。不接受自由文本命令行。
4. **环境指纹一致性**：A/B 对比的两次运行环境指纹不一致 → 结论标记为不可比。
## 代价
- 控制平面必须掌握足够的设备信息才能判断（依赖 Agent 上报的设备探针数据）。
上报延迟 → 判断基于稍旧的数据。缓解：危险步骤执行前强制刷新一次探针，
Agent 的 `precheck` 再做一次现场核对，两者不一致直接拒绝。
- 多一次往返。危险动作本来就不该快。
## 什么时候推翻
不推翻。可以增强（加人工审批门、加双人复核），但不下放到 Agent。
flashops/docs/decisions/0005-simulator-as-first-class.md
+43
View File
@@ -0,0 +1,43 @@
# ADR-0005：模拟器是一等公民，不是测试脚手架
- 日期：2026-07-27
- 状态：已接受
## 背景
第一波没有真实硬件（没有可牺牲的 DUT、没有带外控制器、没有温箱）。
常规做法是先写代码、等硬件到位再联调。
## 决定
把**模拟 DUT + 模拟带外控制器 + 故障注入器**做成产品代码的一部分
（`services/host-agent/flashops_agent/simulator/`），与真实适配器共用同一套接口，
而不是塞进 `tests/` 当替身。
模拟器支持注入的故障至少覆盖报告 Phase 1a 要求验证的三类：
杀 Agent、蓝屏/panic、拔网线；外加掉盘与数据完整性失败。
## 理由
1. **它是长期资产，不是临时替身。** 有了真硬件之后，模拟器仍然是唯一能
在 CI 里跑"100 循环 + 注入 20 次故障"的东西。真设备跑一轮要几小时，
CI 里不可能天天跑。
2. **恢复逻辑的测试覆盖只能靠它。** 五级恢复阶梯里的 L4/L5（ATX 长按、AC 断电）
在真机上每测一次都有风险且很慢。这条路径恰恰是产品价值所在，必须能高频回归。
3. **它是谈判道具的底座。** 报告把 Phase 1a 的产出定义为"一台会自救的演示台——
同时是合伙人与客户谈判的最强道具"。在拿到硬件之前，模拟器版本已经能把
完整故事演一遍：注入蓝屏 → 看着它自己救回来 → 断点续跑 → 出证据包。
4. **它划清了接口。** 能被模拟器替换掉的地方，就是硬件接入的边界。
写模拟器的过程本身就在逼迫接口设计正确。
## 代价
- 模拟器与真实实现可能漂移（模拟器里能过、真机上过不了）。
缓解：`tests/test_adapter_contract.py` 对模拟与真实适配器跑**同一套契约测试**；
真实适配器接入后，契约测试是第一道闸。
- 有"在模拟器上跑通了就以为完事了"的风险。缓解：README 的能力表里，
模拟实现一律标 ⚠️ 或"模拟"，不标 ✅ 真实。
## 什么时候推翻
不推翻。真实硬件接入后模拟器保留，作为 CI 的默认执行后端。
flashops/docs/domain-model.md
+96
View File
@@ -0,0 +1,96 @@
# 领域模型：7 个核心对象 → 14 张表
方案 §5.3 列了 7 个核心领域对象。当前 SQLAlchemy 元数据包含 14 张表。
额外表用于事件溯源、恢复动作、资源锁、失败实例与审计；它们不是新的顶层产品概念，
而是核心对象的行为记录和读模型。
## 对象 → 表
| 方案对象 | 表 | 补充说明 |
|---|---|---|
| DUT | `duts` | 序列号是防错盘的第一信号 |
| Test Host | `test_hosts` + `oob_controllers` | 带外控制器独立成表：它必须能在主机死后独活 |
| Firmware Artifact | `firmware_artifacts` | 哈希 + 签名 + 适用型号，缺一不可审计 |
| Workflow | `workflows` | 存 spec 原文 + spec_hash，改一个字就是新版本 |
| Run | `runs` + `run_steps` + `run_events` + `recovery_actions` | 见下 |
| Failure Signature | `failure_signatures` + `run_failures` | 签名是聚类锚点，失败实例挂在它下面 |
| Evidence Bundle | `evidence_bundles` | manifest 记录完整率，缺字段要能查出来 |
| —（新增） | `resource_locks` | DUT/主机独占，防并发踩踏 |
| —（新增） | `audit_log` | 谁在什么时候用哪个模板做了什么 |
## Run 为什么拆成四张表
`runs` 只存**当前投影**（状态、进度、结论）。真相在 `run_events`。
```
run_events (append-only, 唯一真相)
│ projection
▼
runs / run_steps / recovery_actions (可重建的读模型)
```
任何时刻都能靠重放 `run_events` 重建 `runs` 的状态——这是"事件溯源"落地的含义，
也是统一时间线、审计、自动复现三个功能的共同地基。
**代价**：写路径必须走 `events/recorder.py`。仓储层故意**没有**
`update_run_state()` 这种 API。想改状态？记一条事件，投影自己会跟上。
## 关键字段的设计理由
### `duts.allow_destructive` + `duts.serial`
破坏性动作（Format / Sanitize / 刷写）的三重校验：序列号匹配、
`allow_destructive=True`、设备路径不是系统盘。三个信号缺一个就拒绝执行。
误盘一次 = PoC 判定失败（方案 §9.3），所以这个字段不给 API 直接改，
只能走带审批记录的资产管理接口。
### `runs.env_fingerprint`（JSON，不可变快照）
任务开始时冻结：主板 / BIOS / OS / 内核 / 驱动 / Agent 版本 / 工具版本 / DUT 固件。
A/B 对比的前提是环境相同——固件之外任何一项变了，对比结论就不成立。
签名哈希也吃这个指纹，所以"换了台机器复现不出来"能被自动识别为不同签名。
### `runs.unattended_completion` + `runs.human_touches`
这两个字段是**销售武器**，不是技术指标。UCR（无人值守完成率）和人工触碰次数
是客户签字确认 ROI 的凭据（方案 §9.3、报告 Phase 2 现场指标）。
从第一行代码就记，不要等到要卖了才补。
### `run_events.seq`（每个 run 内单调递增）
时间戳会因为主机断电、时钟漂移、带外控制器与主机时钟不同步而乱序。
`seq` 由控制平面单点分配，保证时间线可重放。带外事件与 Agent 事件
进同一条序列——双通道观测的交叉校验靠它。
### `failure_signatures.hash`（8 要素）
```
hash(workflow_step, normalized_error_codes, log_templates, host_state,
dut_enumeration_state, data_integrity_state, recovery_outcome,
environment_fingerprint)
```
只用错误码会把"掉盘"和"脚本超时"归成一类，聚类就废了。
`log_templates` 是日志模板化后的结果（数字/路径/时间被替换为占位符），
不是原始日志——否则每条日志都是新签名。
### `run_failures.failure_class`
`DUT_DEFECT` / `INFRA_FAILURE` / `SCRIPT_FAILURE` / `DATA_INTEGRITY` / `UNKNOWN`。
**`INFRA_FAILURE` 必须与 `DUT_DEFECT` 分开统计**（方案 §4.3）。
把网络断了、磁盘满了算成 SSD 缺陷，客户第一周就不信任这个系统了。
基础设施误报率 <5% 是 MVP 硬指标。
## 状态字段一览
| 表 | 字段 | 取值 |
|---|---|---|
| `runs` | `state` | QUEUED / PREFLIGHT / RUNNING / RECOVERING / PAUSED / FROZEN / COMPLETED / ABORTED / REJECTED |
| `runs` | `verdict` | PASS / FAIL / INCONCLUSIVE / null |
| `run_steps` | `state` | PENDING / DISPATCHED / RUNNING / SUCCEEDED / FAILED / TIMED_OUT / SKIPPED / CANCELLED |
| `test_hosts` | `status` | ONLINE / DEGRADED / OFFLINE / UNKNOWN |
| `duts` | `status` | IDLE / IN_USE / QUARANTINED / MISSING |
| `recovery_actions` | `outcome` | RECOVERED / FAILED / ESCALATED / FROZEN |
详见 [state-machine.md](state-machine.md)。
flashops/docs/state-machine.md
+134
View File
@@ -0,0 +1,134 @@
# 状态机、恢复阶梯与检查点语义
这是整个产品的核心。差异化不在"能跑命令"，在**跑挂了之后会发生什么**。
当前实现：`engine/states.py`（Run 转移表）、`engine/recovery.py`（恢复纯函数）、
`events/recorder.py`（事件与投影写入口）以及 `services/runs.py`（内置模拟编排）。
Step 租约状态机与独立 runtime 属于下一阶段。
---
## 1. Run 状态机
```
┌──────────┐
│ QUEUED │ 等资源（DUT/主机独占锁）
└────┬─────┘
│ 锁到手
┌────▼─────┐
│ PREFLIGHT│ 安全门禁：序列号绑定、系统盘保护、
└──┬────┬──┘ 破坏性白名单、环境指纹冻结
门禁拒绝 │ │ 全过
┌────▼┐ │
│REJEC│ │
│ TED │ │
└─────┘ │
┌───────▼──────┐
┌─────────▶│ RUNNING │◀────────┐
│ └──┬───┬───┬───┘ │ 恢复成功，
│ 人工继续 │ │ │ │ 从检查点续跑
┌────┴───┐ │ │ │ ┌────┴──────┐
│ PAUSED │◀───────┘ │ └───────▶│ RECOVERING│
└────────┘ 人工暂停 │ 心跳超时/ └────┬──────┘
│ 蓝屏/掉盘 │ 阶梯耗尽
全部循环完成 │ 或数据完整性失败
┌─────▼─────┐ ┌────▼────┐
│ COMPLETED │ │ FROZEN │ 冻结现场，
│ PASS/FAIL │ └─────────┘ 停止一切覆盖性动作
└───────────┘
```
任何非终态 → `ABORTED`（紧急停止：Web / 物理按钮 / 带外均可触发）。
**终态**：`COMPLETED` / `ABORTED` / `FROZEN` / `REJECTED`。终态不可再转移，
`assert_transition()` 会抛 `IllegalTransition`——这个异常在生产里意味着有代码
绕过了事件溯源，属于必须修的 bug，不是可以吞掉的告警。
转移表是 `states.py` 里的一份纯数据 `RUN_TRANSITIONS: Dict[RunState, FrozenSet[RunState]]`。
加新状态时只改这张表 + 补一条单测，不改 `runtime.py`。
## 2. Step 状态机
```
PENDING ──▶ DISPATCHED ──▶ RUNNING ──┬──▶ SUCCEEDED
▲ ├──▶ FAILED ──┐
│ ├──▶ TIMED_OUT┤
└──── 重试（attempt+1）◀───────────┴─────────────┘
└──▶ CANCELLED / SKIPPED
```
- `DISPATCHED`：控制平面已把步骤租给某个 Agent，等待 Agent 确认接手。
租约有超时——Agent 领了活就死了，租约到期步骤回到 `PENDING` 重新派发。
- 重试次数由工作流步骤的 `retry` 声明，**默认 0**。
危险步骤（刷写、Format）默认不重试：重试一次刷写可能把盘刷成砖。
## 3. 五级恢复阶梯
对应方案 §7.2。触发条件是**双通道观测**的判定结果，不是单一信号：
| 级别 | 动作 | 前提 | 典型触发 |
|---|---|---|---|
| L1 | `AGENT_SOFT` Agent 优雅停止/软重启进程 | Agent 心跳还在 | 测试脚本卡死、进程僵死 |
| L2 | `OS_REBOOT` 通过 OS 远程通道重启 | 主机网络还通 | Agent 进程崩溃、驱动异常 |
| L3 | `OOB_RESET` 带外触发主板 Reset | 带外控制器在线 | 心跳超时 + 带外仍在线（蓝屏典型） |
| L4 | `OOB_ATX_POWER` 带外模拟 ATX 长按关机再开机 | 带外控制器在线 | Reset 无效（挂在 BIOS/固件态） |
| L5 | `OOB_AC_CYCLE` 整机 AC 断电 → 安全间隔 → 上电 | 带外控制器在线 | ATX 无效；也是 DUT 掉盘的最后手段 |
| — | `FREEZE` 冻结现场，停止自动恢复 | — | 超过限定次数；或**数据完整性失败立即触发** |
**关键规则（`recovery.py` 里是硬编码的，不给配置覆盖）：**
1. **数据完整性失败直接跳到 FREEZE**，不走阶梯。
哈希/读回比较不一致意味着现场有价值，任何重启都可能毁掉证据。
2. **带外控制器离线时，L3-L5 不可用**，直接降级到 FREEZE 并标记
`INFRA_FAILURE`——不能因为带外没接就把 DUT 判成坏盘。
3. **每一级恢复都要留证**：恢复前抓画面/串口/温度，恢复后验证 DUT 重新枚举。
`recovery_actions` 表记录每一级的 trigger / outcome / evidence_uri。
4. **恢复成功 ≠ 步骤成功**。恢复只是把系统救回可执行状态，
原步骤按检查点语义决定是续跑还是重跑。
## 4. 检查点与断点续跑
检查点在**步骤边界**创建，不在步骤中间——中间态无法保证幂等。
```
loop_index=37, step=fio-workload, state=SUCCEEDED
└─▶ checkpoint 写入：{loop_index: 37, next_step: verify-enumeration, dut_fw: "A"}
```
恢复后的续跑规则：
| 挂在哪 | 恢复后 |
|---|---|
| 步骤已 `SUCCEEDED`，检查点已落 | 从下一步继续 |
| 步骤 `RUNNING` 时挂了，步骤幂等（`idempotent: true`） | 重跑该步骤 |
| 步骤 `RUNNING` 时挂了，步骤非幂等（默认，如刷写） | **不重跑**，标记 `INCONCLUSIVE`，进入人工确认 |
| 步骤是循环体中的一环 | 回到该 `loop_index` 的起点重跑整轮 |
非幂等步骤不自动重跑，是这一层最保守也最重要的默认值。
"自动重试把盘刷坏"是这个产品最容易砸招牌的失败模式。
## 5. 心跳与失联判定
```
Agent 心跳间隔 5s
├─ 15s 无心跳 → 主机 DEGRADED，记事件，不动作
├─ 30s 无心跳 → 判定失联，查带外：
│ ├─ 带外在线 + 主机有电 → 走恢复阶梯 L3
│ ├─ 带外在线 + 主机无电 → INFRA_FAILURE（供电问题，不是 DUT）
│ └─ 带外也离线 → INFRA_FAILURE + FREEZE（不能瞎判）
└─ 心跳恢复 → 校验 Run 状态一致性，续跑
```
阈值在 `config.py`，但**判定逻辑在 `state_machine.py` 里是纯函数**：
输入 `(last_heartbeat_age, oob_online, oob_power_state, now)`，输出判定。
所以"30 秒超时"这种行为可以不等 30 秒就测出来。
## 6. 怎么给状态机加东西（后续几波会反复做）
1. 在 `states.py` 加状态/事件枚举 + 改转移表
2. 在 `state_machine.py::decide()` 加分支，返回新的 `Effect`
3. 在 `runtime.py` 加该 `Effect` 的施加逻辑（唯一碰 I/O 的地方）
4. 在 `tests/test_state_machine.py` 加纯函数单测——不起库、不 sleep
如果第 2 步发现需要在 `decide()` 里做 I/O，说明抽象漏了，
应该把需要的数据加进 `Snapshot`，而不是在纯函数里开个口子。
flashops/docs/workflow-spec.md
+123
View File
@@ -0,0 +1,123 @@
# 工作流规范：SOP → 可执行状态机
> **当前状态**：这是待实现的发布与物化规范。数据库中已经保存一份符合该结构的
> 演示工作流，但 `WorkflowSpec` 完整 schema 校验、YAML 发布器与 `planner.py`
> 尚未落地；当前 API 不接受任意工作流上传。
工作流必须是**确定性的结构化配置**，不是让大模型自由生成命令（方案 §6.3、§8.1）。
AI 可以把自然语言 SOP 草拟成下面这份 YAML，但**必须过 schema 校验 + 人工发布**才能执行。
目标实现位置：`engine/planner.py`（物化）、`schemas.py::WorkflowSpec`（校验）
与 `workflows/fw-ab-regression.yaml`（版本化示例）。
## Schema
```yaml
key: fw-ab-regression # 全局唯一，改 key = 新工作流
name: 固件 A/B 回归
version: 3 # 每次发布 +1；spec_hash 变了但 version 没变 → 拒绝发布
danger_level: high # none | low | high —— high 需要审批记录
source_sop: "客户X_固件回归SOP_v2.1（脱敏）"
params: # 运行时参数，带类型与默认值
loops: {type: int, default: 100, min: 1, max: 10000}
dut_serial: {type: string, required: true}
host: {type: string, required: true}
fw_a: {type: string, required: true} # firmware_artifact id
fw_b: {type: string, required: true}
resources: # 独占锁，PREFLIGHT 阶段获取，终态释放
- {type: dut, ref: params.dut_serial, exclusive: true}
- {type: test_host, ref: params.host, exclusive: true}
policy:
heartbeat_timeout_s: 30
max_recovery_per_loop: 3 # 单轮循环内恢复超过 3 次 → FREEZE
max_recovery_total: 20
on_data_integrity_failure: freeze # 硬编码值，写在这里只是为了显式
steps:
- key: preflight
type: precheck
adapter: safety
- key: flash-fw-a
type: command
adapter: nvme_cli
template: nvme_fw_download_commit # 只能引用已注册的签名模板
params: {firmware: "{{ params.fw_a }}", slot: 1, action: 3}
danger: high
timeout_s: 300
idempotent: false # 默认值，写出来是为了提醒：挂了不自动重跑
checkpoint: true
- key: regression-loop
type: loop
count: "{{ params.loops }}"
body:
- key: workload
type: command
adapter: fio
template: fio_seq_rw
timeout_s: 600
retry: 1
idempotent: true
- key: enumeration-check
type: device_check
adapter: nvme_cli
expect: {enumerated: true, link_speed_min: "8GT/s"}
- key: integrity-check
type: command
adapter: shell
template: sha256_readback
on_fail: freeze # 覆盖默认处置
checkpoint: true # 每轮结束落检查点
- key: report
type: report
adapter: builtin
```
## 步骤类型（MVP 六种）
| type | 语义 | 备注 |
|---|---|---|
| `command` | 执行签名命令模板 | 唯一能碰危险动作的类型 |
| `device_check` | 设备探针断言（枚举/固件版本/链路速率/SMART） | 不改变设备状态 |
| `wait` | 等待固定时长或条件 | 用于温度稳定、上电间隔 |
| `loop` | 固定次数循环，body 是步骤列表 | 支持嵌套一层 |
| `precheck` | 安全门禁 | 每个工作流的第一步都应该是它 |
| `report` | 生成报告与 Evidence Bundle | 内置适配器 |
企业版才加的：`http`、`parallel`、`subflow`、`approval`、`instrument`。
这一波故意不做——`planner.py` 里遇到未知 type 直接拒绝发布，不静默跳过。
## 模板变量
只支持 `{{ params.X }}` 和 `{{ loop.index }}` 两种插值，**不支持表达式求值**。
理由：能求值就能注入。需要计算的场景写进适配器，不写进 YAML。
## 版本与 Git
工作流 YAML 进 `workflows/` 目录、进 Git。发布时控制平面记录：
```
workflows.spec 原文
workflows.spec_hash sha256(规范化后的 spec)
workflows.version 发布号
```
`spec_hash` 变了而 `version` 没变 → 拒绝发布。理由：A/B 对比的结论必须能追溯到
**具体哪一版流程**，否则"上周跑的和这周跑的是不是同一个流程"就说不清了。
## `on_fail` 的可选处置
| 值 | 行为 |
|---|---|
| `retry`（默认，受 `retry:` 次数约束） | 重试该步骤 |
| `fail_run` | 整个 Run 判 FAIL 并结束 |
| `continue` | 记失败，继续下一步（用于非关键采集步骤） |
| `freeze` | 立刻冻结现场，停止一切覆盖性动作 |
| `recover` | 进入恢复阶梯 |
数据完整性相关的步骤**只允许** `freeze`，planner 校验时强制。
flashops/services/control-plane/flashops_control/__init__.py
+5
View File
@@ -0,0 +1,5 @@
"""FlashOps 控制平面。"""
from .config import APP_NAME
__all__ = ["APP_NAME"]
flashops/services/control-plane/flashops_control/api/__init__.py
+1
View File
@@ -0,0 +1 @@
"""FastAPI 路由。"""
flashops/services/control-plane/flashops_control/api/agents.py
+276
View File
@@ -0,0 +1,276 @@
"""拉模式 Host Agent 的注册、认证与心跳 API。"""
from __future__ import annotations
import asyncio
import hmac
import secrets
import time
import uuid
from typing import Dict, List, Optional
from fastapi import APIRouter, Header, HTTPException, Response, status
from sqlalchemy import select
from ..config import get_settings
from ..db import session_scope
from ..enums import HostStatus
from ..models import TestHost, iso_z, utcnow
from ..schemas import (
AgentHeartbeatBody,
AgentLeaseBody,
AgentRegisterBody,
AgentStepAttemptBody,
AgentStepCompleteBody,
AgentStepEventsBody,
)
from ..services.agent_status import refresh_agent_host
from ..services.step_leases import (
StepLeaseConflict,
acknowledge_step,
complete_step,
record_step_events,
renew_step,
try_lease_next_step,
)
from .deps import agent_token_digest, require_agent
router = APIRouter(prefix="/api/v1/agent", tags=["host-agent"])
def _idempotency_key(value: str) -> str:
if len(value) > 60:
raise HTTPException(status_code=422, detail="Idempotency-Key 过长")
try:
uuid.UUID(value)
except (ValueError, AttributeError) as exc:
raise HTTPException(
status_code=422, detail="Idempotency-Key 必须是 UUID"
) from exc
return value
def _lease_conflict(exc: StepLeaseConflict) -> HTTPException:
return HTTPException(status_code=status.HTTP_409_CONFLICT, detail=str(exc))
def _check_enrollment_token(provided: Optional[str]) -> None:
settings = get_settings()
required = settings.agent_enrollment_token
if settings.env == "production" and not required:
raise HTTPException(
status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
detail="生产环境尚未配置 Agent 注册口令",
)
if required and not hmac.compare_digest(provided or "", required):
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Agent 注册口令无效",
)
async def _resolve_host(body: AgentRegisterBody) -> Dict[str, object]:
async with session_scope() as session:
host: Optional[TestHost] = None
if body.host_id:
host = await session.get(TestHost, body.host_id)
if host is None:
raise HTTPException(status_code=404, detail="Test Host 不存在")
elif body.host_name:
host = await session.scalar(
select(TestHost).where(TestHost.name == body.host_name)
)
else:
host = await session.scalar(select(TestHost).order_by(TestHost.created_at))
if host is None:
host = TestHost(name=body.host_name or "agent-host-" + uuid.uuid4().hex[:8])
session.add(host)
await session.flush()
raw_token = secrets.token_urlsafe(32)
host.agent_id = host.agent_id or "agent_" + uuid.uuid4().hex[:20]
host.agent_token = agent_token_digest(raw_token)
host.agent_version = body.agent_version
host.os_family = body.os_family
if body.os_version is not None:
host.os_version = body.os_version
if body.kernel is not None:
host.kernel = body.kernel
if body.cpu is not None:
host.cpu = body.cpu
if body.motherboard is not None:
host.motherboard = body.motherboard
if body.bios_version is not None:
host.bios_version = body.bios_version
if body.ip_address is not None:
host.ip_address = body.ip_address
if body.toolchain:
host.toolchain = body.toolchain
if body.labels:
host.labels = {**(host.labels or {}), **body.labels}
host.status = HostStatus.ONLINE.value
host.last_heartbeat_at = utcnow()
payload = {
"agent_id": host.agent_id,
"host_id": host.id,
"token": raw_token,
"heartbeat_interval_s": get_settings().timing.heartbeat_interval_s,
"server_time": iso_z(host.last_heartbeat_at),
}
return payload
@router.post("/register", status_code=status.HTTP_201_CREATED)
async def register_agent(
body: AgentRegisterBody,
enrollment_token: Optional[str] = Header(
default=None, alias="X-FlashOps-Enrollment-Token"
),
) -> Dict[str, object]:
_check_enrollment_token(enrollment_token)
return await _resolve_host(body)
@router.post("/{agent_id}/heartbeat")
async def heartbeat(
agent_id: str,
body: AgentHeartbeatBody,
authorization: Optional[str] = Header(default=None),
) -> Dict[str, object]:
async with session_scope() as session:
host = await require_agent(session, agent_id, authorization)
host.status = (
HostStatus.ONLINE.value if body.healthy else HostStatus.DEGRADED.value
)
host.last_heartbeat_at = utcnow()
if body.agent_version:
host.agent_version = body.agent_version
if body.ip_address:
host.ip_address = body.ip_address
if body.toolchain:
host.toolchain = body.toolchain
if body.device_probe:
host.labels = {
**(host.labels or {}),
"agent_probe": body.device_probe,
}
return {
"ok": True,
"server_time": iso_z(host.last_heartbeat_at),
"heartbeat_interval_s": get_settings().timing.heartbeat_interval_s,
"commands": [],
}
@router.post("/{agent_id}/lease")
async def lease_step(
agent_id: str,
body: AgentLeaseBody,
authorization: Optional[str] = Header(default=None),
) -> object:
deadline = time.monotonic() + min(
body.wait_timeout_s, get_settings().timing.lease_poll_timeout_s
)
while True:
async with session_scope() as session:
host = await require_agent(session, agent_id, authorization)
lease = await try_lease_next_step(session, host)
if lease is not None:
return lease
remaining = deadline - time.monotonic()
if remaining <= 0:
return Response(status_code=status.HTTP_204_NO_CONTENT)
await asyncio.sleep(min(get_settings().engine_tick_s, remaining))
@router.post("/{agent_id}/steps/{step_id}/ack")
async def ack_step(
agent_id: str,
step_id: str,
body: AgentStepAttemptBody,
authorization: Optional[str] = Header(default=None),
idempotency_key: str = Header(alias="Idempotency-Key"),
) -> Dict[str, object]:
key = _idempotency_key(idempotency_key)
try:
async with session_scope() as session:
host = await require_agent(session, agent_id, authorization)
return await acknowledge_step(session, host, step_id, body.attempt, key)
except StepLeaseConflict as exc:
raise _lease_conflict(exc) from exc
@router.post("/{agent_id}/steps/{step_id}/renew")
async def renew_step_lease(
agent_id: str,
step_id: str,
body: AgentStepAttemptBody,
authorization: Optional[str] = Header(default=None),
idempotency_key: str = Header(alias="Idempotency-Key"),
) -> Dict[str, object]:
key = _idempotency_key(idempotency_key)
try:
async with session_scope() as session:
host = await require_agent(session, agent_id, authorization)
return await renew_step(session, host, step_id, body.attempt, key)
except StepLeaseConflict as exc:
raise _lease_conflict(exc) from exc
@router.post("/{agent_id}/steps/{step_id}/events")
async def step_events(
agent_id: str,
step_id: str,
body: AgentStepEventsBody,
authorization: Optional[str] = Header(default=None),
idempotency_key: str = Header(alias="Idempotency-Key"),
) -> Dict[str, object]:
key = _idempotency_key(idempotency_key)
try:
async with session_scope() as session:
host = await require_agent(session, agent_id, authorization)
return await record_step_events(
session, host, step_id, body.attempt, body.events, key
)
except StepLeaseConflict as exc:
raise _lease_conflict(exc) from exc
@router.post("/{agent_id}/steps/{step_id}/complete")
async def finish_step(
agent_id: str,
step_id: str,
body: AgentStepCompleteBody,
authorization: Optional[str] = Header(default=None),
idempotency_key: str = Header(alias="Idempotency-Key"),
) -> Dict[str, object]:
key = _idempotency_key(idempotency_key)
try:
async with session_scope() as session:
host = await require_agent(session, agent_id, authorization)
return await complete_step(session, host, step_id, body, key)
except StepLeaseConflict as exc:
raise _lease_conflict(exc) from exc
@router.get("/hosts")
async def list_agent_hosts() -> List[Dict[str, object]]:
async with session_scope() as session:
hosts = (
await session.scalars(select(TestHost).order_by(TestHost.name))
).all()
for host in hosts:
refresh_agent_host(host)
return [
{
"host_id": host.id,
"host_name": host.name,
"agent_id": host.agent_id,
"agent_version": host.agent_version,
"status": host.status,
"last_heartbeat_at": iso_z(host.last_heartbeat_at),
"ip_address": host.ip_address,
"toolchain": host.toolchain,
}
for host in hosts
]
flashops/services/control-plane/flashops_control/api/deps.py
+47
View File
@@ -0,0 +1,47 @@
"""HTTP dependencies shared by Agent endpoints."""
from __future__ import annotations
import hashlib
import hmac
from typing import Optional
from fastapi import HTTPException, status
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from ..models import TestHost
def agent_token_digest(token: str) -> str:
return hashlib.sha256(token.encode("utf-8")).hexdigest()
def bearer_token(authorization: Optional[str]) -> str:
scheme, separator, value = (authorization or "").partition(" ")
if not separator or scheme.lower() != "bearer" or not value.strip():
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Agent bearer token 缺失",
headers={"WWW-Authenticate": "Bearer"},
)
return value.strip()
async def require_agent(
session: AsyncSession,
agent_id: str,
authorization: Optional[str],
) -> TestHost:
host = await session.scalar(select(TestHost).where(TestHost.agent_id == agent_id))
token = bearer_token(authorization)
if (
host is None
or not host.agent_token
or not hmac.compare_digest(host.agent_token, agent_token_digest(token))
):
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Agent 身份无效",
headers={"WWW-Authenticate": "Bearer"},
)
return host
flashops/services/control-plane/flashops_control/api/routes.py
+258
View File
@@ -0,0 +1,258 @@
"""控制台所需的最小纵向 API。"""
from __future__ import annotations
from typing import Dict, List
from fastapi import APIRouter, HTTPException, status
from sqlalchemy import func, select
from ..db import session_scope
from ..enums import EventKind, EventSource, RunState, Severity
from ..events import append_event
from ..models import EvidenceBundle, Run, RunEvent, ResourceLock, utcnow
from ..schemas import CreateRunBody, HumanActionBody, PreflightBody
from ..services.runs import (
cancel_simulation,
create_run,
event_to_dict,
prepare_agent_run,
preflight_for_body,
run_to_dict,
start_simulation,
)
router = APIRouter(prefix="/api/v1")
@router.get("/health")
async def health() -> Dict[str, str]:
return {"status": "ok", "service": "flashops-control-plane"}
@router.post("/preflight")
async def preflight(body: PreflightBody) -> Dict[str, object]:
try:
async with session_scope() as session:
return await preflight_for_body(session, body)
except LookupError as exc:
raise HTTPException(status_code=404, detail=str(exc)) from exc
@router.post("/runs", status_code=status.HTTP_201_CREATED)
async def create_run_route(body: CreateRunBody) -> Dict[str, object]:
try:
async with session_scope() as session:
run = await create_run(session, body)
if body.execution_mode == "agent_dry_run":
await prepare_agent_run(session, run)
payload = run_to_dict(run)
if body.execution_mode == "simulated":
start_simulation(run.id)
return payload
except LookupError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
@router.post("/runs/demo", status_code=status.HTTP_201_CREATED)
async def create_demo_run() -> Dict[str, object]:
return await create_run_route(CreateRunBody())
@router.get("/runs")
async def list_runs(limit: int = 30) -> List[Dict[str, object]]:
async with session_scope() as session:
runs = (
await session.scalars(
select(Run).order_by(Run.created_at.desc()).limit(min(max(limit, 1), 100))
)
).all()
return [run_to_dict(run) for run in runs]
@router.get("/runs/{run_id}")
async def get_run(run_id: str) -> Dict[str, object]:
async with session_scope() as session:
run = await session.get(Run, run_id)
if run is None:
raise HTTPException(status_code=404, detail="Run 不存在")
payload = run_to_dict(run)
bundle = await session.scalar(
select(EvidenceBundle)
.where(EvidenceBundle.run_id == run_id)
.order_by(EvidenceBundle.created_at.desc())
)
payload["evidence"] = (
{
"id": bundle.id,
"uri": bundle.uri,
"completeness": bundle.completeness,
"size_bytes": bundle.size_bytes,
}
if bundle
else None
)
return payload
@router.get("/runs/{run_id}/events")
async def get_events(run_id: str, after_seq: int = 0) -> List[Dict[str, object]]:
async with session_scope() as session:
exists = await session.get(Run, run_id)
if exists is None:
raise HTTPException(status_code=404, detail="Run 不存在")
events = (
await session.scalars(
select(RunEvent)
.where(RunEvent.run_id == run_id, RunEvent.seq > after_seq)
.order_by(RunEvent.seq)
)
).all()
return [event_to_dict(event) for event in events]
@router.post("/runs/{run_id}/pause")
async def pause_run(run_id: str, body: HumanActionBody) -> Dict[str, object]:
async with session_scope() as session:
run = await session.get(Run, run_id)
if run is None:
raise HTTPException(status_code=404, detail="Run 不存在")
if run.state != RunState.RUNNING.value:
raise HTTPException(status_code=409, detail="只有 RUNNING 任务可以暂停")
await append_event(
session,
run,
EventKind.HUMAN_TOUCH.value,
"{0}: {1}".format(body.actor, body.reason),
source=EventSource.HUMAN.value,
projection={
"human_touches": run.human_touches + 1,
"unattended_completion": False,
},
)
await append_event(
session,
run,
EventKind.RUN_PAUSED.value,
"任务已在安全检查点暂停",
source=EventSource.HUMAN.value,
target_state=RunState.PAUSED.value,
)
return run_to_dict(run)
@router.post("/runs/{run_id}/resume")
async def resume_run(run_id: str, body: HumanActionBody) -> Dict[str, object]:
async with session_scope() as session:
run = await session.get(Run, run_id)
if run is None:
raise HTTPException(status_code=404, detail="Run 不存在")
if run.state != RunState.PAUSED.value:
raise HTTPException(status_code=409, detail="只有 PAUSED 任务可以继续")
await append_event(
session,
run,
EventKind.HUMAN_TOUCH.value,
"{0}: {1}".format(body.actor, body.reason),
source=EventSource.HUMAN.value,
projection={
"human_touches": run.human_touches + 1,
"unattended_completion": False,
},
)
await append_event(
session,
run,
EventKind.RUN_RESUMED.value,
"人工确认后从检查点继续",
source=EventSource.HUMAN.value,
target_state=RunState.RUNNING.value,
)
return run_to_dict(run)
@router.post("/runs/{run_id}/emergency-stop")
async def emergency_stop(run_id: str, body: HumanActionBody) -> Dict[str, object]:
async with session_scope() as session:
run = await session.get(Run, run_id)
if run is None:
raise HTTPException(status_code=404, detail="Run 不存在")
if run.state in {
RunState.COMPLETED.value,
RunState.ABORTED.value,
RunState.FROZEN.value,
RunState.REJECTED.value,
}:
raise HTTPException(status_code=409, detail="终态任务不能再次停止")
await append_event(
session,
run,
EventKind.HUMAN_TOUCH.value,
"{0}: {1}".format(body.actor, body.reason),
source=EventSource.HUMAN.value,
severity=Severity.CRITICAL.value,
projection={
"human_touches": run.human_touches + 1,
"unattended_completion": False,
},
)
await append_event(
session,
run,
EventKind.RUN_ABORTED.value,
"紧急停止已执行，现场与检查点已保留",
source=EventSource.HUMAN.value,
severity=Severity.CRITICAL.value,
target_state=RunState.ABORTED.value,
projection={"ended_at": utcnow()},
)
await session.execute(
ResourceLock.__table__.delete().where(ResourceLock.run_id == run.id)
)
payload = run_to_dict(run)
cancel_simulation(run_id)
return payload
@router.get("/dashboard/summary")
async def dashboard_summary() -> Dict[str, object]:
async with session_scope() as session:
total = await session.scalar(select(func.count()).select_from(Run)) or 0
running = (
await session.scalar(
select(func.count()).select_from(Run).where(
Run.state.in_(
[
RunState.QUEUED.value,
RunState.PREFLIGHT.value,
RunState.RUNNING.value,
RunState.RECOVERING.value,
]
)
)
)
or 0
)
completed = (
await session.scalar(
select(func.count()).select_from(Run).where(
Run.state == RunState.COMPLETED.value
)
)
or 0
)
unattended = (
await session.scalar(
select(func.count()).select_from(Run).where(
Run.state == RunState.COMPLETED.value,
Run.unattended_completion.is_(True),
)
)
or 0
)
ucr = round(unattended / completed * 100, 1) if completed else 0.0
return {
"total_runs": total,
"active_runs": running,
"completed_runs": completed,
"unattended_completion_rate": ucr,
}
flashops/services/control-plane/flashops_control/cli.py
+74
View File
@@ -0,0 +1,74 @@
"""开发与演示命令行。"""
from __future__ import annotations
import argparse
import asyncio
from sqlalchemy import select
from .db import drop_db, init_db, session_scope
from .models import EvidenceBundle, Run
from .schemas import CreateRunBody
from .seed import ensure_seed_data
from .services.runs import create_run, run_to_dict, simulate_run
async def _init_db() -> None:
await init_db()
async def _seed() -> None:
await init_db()
async with session_scope() as session:
await ensure_seed_data(session)
async def _reset() -> None:
await drop_db()
await init_db()
await _seed()
async def _demo() -> None:
await _seed()
async with session_scope() as session:
run = await create_run(
session,
CreateRunBody(loops=8, inject_failure=True, created_by="cli-demo"),
)
run_id = run.id
print("Run {0} 已启动：将注入一次心跳丢失并自动升级到 L3 恢复。".format(run_id))
await simulate_run(run_id)
async with session_scope() as session:
run = await session.get(Run, run_id)
bundle = await session.scalar(
select(EvidenceBundle)
.where(EvidenceBundle.run_id == run_id)
.order_by(EvidenceBundle.created_at.desc())
)
assert run is not None
payload = run_to_dict(run)
print(
"完成：{state} / {verdict}，循环 {loop_done}/{loop_target}，恢复 {recovery_count} 次。".format(
**payload
)
)
if bundle:
print("Evidence Bundle: {0}".format(bundle.uri))
def main() -> None:
parser = argparse.ArgumentParser(prog="flashops")
parser.add_argument("command", choices=["init-db", "seed", "reset-db", "demo"])
args = parser.parse_args()
commands = {
"init-db": _init_db,
"seed": _seed,
"reset-db": _reset,
"demo": _demo,
}
asyncio.run(commands[args.command]())
if __name__ == "__main__":
main()
flashops/services/control-plane/flashops_control/config.py
+107
View File
@@ -0,0 +1,107 @@
"""配置。
产品名只出现在这里和 apps/console/lib/brand.ts —— 改名成本 = 改两个常量。
商标检索未完成前，任何地方都不要再硬编码产品名。
"""
from __future__ import annotations
import os
from dataclasses import dataclass, field
from pathlib import Path
from typing import Optional
APP_NAME = "FlashOps"
APP_TAGLINE = "存储实验室无人值守执行层"
# 仓库根目录：.../flashops
REPO_ROOT = Path(__file__).resolve().parents[3]
def _env(key: str, default: str) -> str:
return os.environ.get(f"FLASHOPS_{key}", default)
def _env_int(key: str, default: int) -> int:
return int(os.environ.get(f"FLASHOPS_{key}", str(default)))
def _env_float(key: str, default: float) -> float:
return float(os.environ.get(f"FLASHOPS_{key}", str(default)))
@dataclass(frozen=True)
class TimingPolicy:
"""时间策略。
全部集中在这里，且引擎不直接读它 —— 引擎从 Snapshot 拿数值。
这样"30 秒心跳超时"这类规则可以在测试里用 0.03 秒验证。
"""
heartbeat_interval_s: float = 5.0
heartbeat_degraded_after_s: float = 15.0
heartbeat_lost_after_s: float = 30.0
lease_ttl_s: float = 60.0
lease_poll_timeout_s: float = 20.0
step_default_timeout_s: float = 600.0
recovery_settle_s: float = 20.0 # 恢复动作后等待主机回来的时间
ac_cycle_off_s: float = 10.0 # AC 断电后的安全间隔（方案 §7.2）
max_recovery_per_loop: int = 3
max_recovery_total: int = 20
@dataclass(frozen=True)
class Settings:
app_name: str = APP_NAME
env: str = field(default_factory=lambda: _env("ENV", "development"))
# 开发态：SQLite。生产：postgresql+asyncpg://...
database_url: str = field(
default_factory=lambda: _env(
"DATABASE_URL", "sqlite+aiosqlite:///" + str(REPO_ROOT / "var" / "flashops.db")
)
)
# 开发态：本地目录。生产：s3://minio:9000/flashops
object_store_url: str = field(
default_factory=lambda: _env("OBJECT_STORE_URL", str(REPO_ROOT / "var" / "objects"))
)
# 开发态：进程内 asyncio。生产：redis://... / nats://...
bus_url: Optional[str] = field(default_factory=lambda: os.environ.get("FLASHOPS_BUS_URL"))
# Agent 首次注册口令。开发态未配置时允许本机演示；生产态缺失时拒绝注册。
agent_enrollment_token: Optional[str] = field(
default_factory=lambda: os.environ.get("FLASHOPS_AGENT_ENROLLMENT_TOKEN")
)
api_host: str = field(default_factory=lambda: _env("API_HOST", "0.0.0.0"))
api_port: int = field(default_factory=lambda: _env_int("API_PORT", 8000))
cors_origins: str = field(default_factory=lambda: _env("CORS_ORIGINS", "http://localhost:3000"))
# 引擎主循环的 tick 间隔。demo/测试里调小以加速。
engine_tick_s: float = field(default_factory=lambda: _env_float("ENGINE_TICK_S", 0.5))
# 演示/仿真加速倍率：1.0 = 真实速度
time_scale: float = field(default_factory=lambda: _env_float("TIME_SCALE", 1.0))
timing: TimingPolicy = field(default_factory=TimingPolicy)
@property
def is_sqlite(self) -> bool:
return self.database_url.startswith("sqlite")
@property
def object_store_path(self) -> Path:
return Path(self.object_store_url)
_settings: Optional[Settings] = None
def get_settings() -> Settings:
global _settings
if _settings is None:
_settings = Settings()
return _settings
def reset_settings() -> None:
"""测试用：让下一次 get_settings() 重新读环境变量。"""
global _settings
_settings = None
flashops/services/control-plane/flashops_control/db.py
+106
View File
@@ -0,0 +1,106 @@
"""异步数据库连接与会话。
连接按当前配置惰性创建，测试可以安全地切换到临时 SQLite 数据库。
"""
from __future__ import annotations
import asyncio
from contextlib import asynccontextmanager
from typing import AsyncIterator, Optional
from weakref import WeakKeyDictionary
from sqlalchemy.ext.asyncio import (
AsyncEngine,
AsyncSession,
async_sessionmaker,
create_async_engine,
)
from .config import get_settings
from .models import Base
_engine: Optional[AsyncEngine] = None
_engine_url: Optional[str] = None
_session_factory: Optional[async_sessionmaker[AsyncSession]] = None
_sqlite_session_locks: WeakKeyDictionary[asyncio.AbstractEventLoop, asyncio.Lock] = (
WeakKeyDictionary()
)
def _sqlite_session_lock() -> asyncio.Lock:
"""Return one transaction lock per event loop for the SQLite prototype.
SQLite permits only one writer at a time. Serializing the complete
session_scope transaction prevents the simulator and human control routes
from reading the same event sequence and then racing on commit. PostgreSQL
deployments skip this application-level lock.
"""
loop = asyncio.get_running_loop()
lock = _sqlite_session_locks.get(loop)
if lock is None:
lock = asyncio.Lock()
_sqlite_session_locks[loop] = lock
return lock
def get_engine() -> AsyncEngine:
global _engine, _engine_url, _session_factory
url = get_settings().database_url
if _engine is None or _engine_url != url:
_engine = create_async_engine(url, future=True)
_engine_url = url
_session_factory = async_sessionmaker(_engine, expire_on_commit=False)
return _engine
def get_session_factory() -> async_sessionmaker[AsyncSession]:
get_engine()
assert _session_factory is not None
return _session_factory
@asynccontextmanager
async def _transaction_scope() -> AsyncIterator[AsyncSession]:
session = get_session_factory()()
try:
yield session
await session.commit()
except Exception:
await session.rollback()
raise
finally:
await session.close()
@asynccontextmanager
async def session_scope() -> AsyncIterator[AsyncSession]:
if get_settings().is_sqlite:
async with _sqlite_session_lock():
async with _transaction_scope() as session:
yield session
return
async with _transaction_scope() as session:
yield session
async def init_db() -> None:
engine = get_engine()
async with engine.begin() as connection:
await connection.run_sync(Base.metadata.create_all)
async def drop_db() -> None:
engine = get_engine()
async with engine.begin() as connection:
await connection.run_sync(Base.metadata.drop_all)
async def dispose_db() -> None:
global _engine, _engine_url, _session_factory
if _engine is not None:
await _engine.dispose()
_engine = None
_engine_url = None
_session_factory = None
flashops/services/control-plane/flashops_control/engine/__init__.py
+5
View File
@@ -0,0 +1,5 @@
"""无 I/O 的编排规则。"""
from .states import IllegalTransition, assert_run_transition
__all__ = ["IllegalTransition", "assert_run_transition"]
flashops/services/control-plane/flashops_control/engine/planner.py
+159
View File
@@ -0,0 +1,159 @@
"""Materialize a published Workflow snapshot into sequential RunStep rows.
The planner deliberately supports a small, deterministic subset of the workflow
schema. It never evaluates expressions and it never turns free-form text into a
command. Host Agents only receive already-published adapter/template references.
"""
from __future__ import annotations
import re
from typing import Any, Dict, List, Optional
from sqlalchemy.ext.asyncio import AsyncSession
from ..config import get_settings
from ..enums import OnFail, StepType
from ..models import Run, RunStep, Workflow
_PARAM = re.compile(r"^\{\{\s*params\.([A-Za-z_][A-Za-z0-9_]*)\s*\}\}$")
_LOOP_INDEX = re.compile(r"^\{\{\s*loop\.index\s*\}\}$")
_HOST_STEP_TYPES = {
StepType.COMMAND.value,
StepType.DEVICE_CHECK.value,
StepType.WAIT.value,
}
class WorkflowPlanningError(ValueError):
"""The published workflow cannot be safely materialized."""
def _resolve(value: Any, run_params: Dict[str, Any], loop_index: Optional[int]) -> Any:
"""Resolve only exact, allow-listed template tokens; no expression evaluation."""
if isinstance(value, str):
parameter = _PARAM.fullmatch(value)
if parameter:
key = parameter.group(1)
if key not in run_params:
raise WorkflowPlanningError("工作流参数不存在: {0}".format(key))
return run_params[key]
if _LOOP_INDEX.fullmatch(value):
if loop_index is None:
raise WorkflowPlanningError("loop.index 只能在循环体内使用")
return loop_index
if "{{" in value or "}}" in value:
raise WorkflowPlanningError("模板只允许完整的 params.X 或 loop.index 占位符")
return value
if isinstance(value, dict):
return {key: _resolve(item, run_params, loop_index) for key, item in value.items()}
if isinstance(value, list):
return [_resolve(item, run_params, loop_index) for item in value]
return value
def _step_row(
*,
run: Run,
spec: Dict[str, Any],
seq: int,
loop_index: Optional[int],
checkpoint_after: bool,
) -> RunStep:
step_type = str(spec.get("type", ""))
if step_type not in _HOST_STEP_TYPES:
raise WorkflowPlanningError("Host Agent 不支持步骤类型: {0}".format(step_type or "<empty>"))
key = str(spec.get("key", "")).strip()
adapter = str(spec.get("adapter", "")).strip()
if not key or not adapter:
raise WorkflowPlanningError("步骤 key 与 adapter 不能为空")
retry = int(spec.get("retry", 0))
timeout_s = float(spec.get("timeout_s", get_settings().timing.step_default_timeout_s))
if retry < 0 or retry > 20:
raise WorkflowPlanningError("步骤 retry 必须在 0..20 之间")
if timeout_s <= 0:
raise WorkflowPlanningError("步骤 timeout_s 必须大于 0")
return RunStep(
run_id=run.id,
seq=seq,
loop_index=loop_index,
step_key=key,
step_type=step_type,
adapter=adapter,
template=spec.get("template"),
params=_resolve(dict(spec.get("params") or {}), run.params, loop_index),
max_retry=retry,
timeout_s=timeout_s,
idempotent=bool(spec.get("idempotent", False)),
danger=str(spec.get("danger", "none")),
on_fail=str(spec.get("on_fail", OnFail.RETRY.value)),
checkpoint_after=checkpoint_after,
)
def plan_run_steps(run: Run, workflow: Workflow) -> List[RunStep]:
"""Expand one workflow snapshot into the exact ordered steps for a Run."""
if not workflow.published:
raise WorkflowPlanningError("工作流尚未发布")
raw_steps = workflow.spec.get("steps") if isinstance(workflow.spec, dict) else None
if not isinstance(raw_steps, list) or not raw_steps:
raise WorkflowPlanningError("工作流没有可执行步骤")
planned: List[RunStep] = []
seq = 1
for raw in raw_steps:
if not isinstance(raw, dict):
raise WorkflowPlanningError("工作流步骤必须是 object")
step_type = str(raw.get("type", ""))
# Safety preflight and evidence/report generation live in the control plane.
if step_type in {StepType.PRECHECK.value, StepType.REPORT.value}:
continue
if step_type == StepType.LOOP.value:
body = raw.get("body")
if not isinstance(body, list) or not body:
raise WorkflowPlanningError("循环步骤必须包含非空 body")
for loop_index in range(1, run.loop_target + 1):
for index, child in enumerate(body):
if not isinstance(child, dict):
raise WorkflowPlanningError("循环体步骤必须是 object")
planned.append(
_step_row(
run=run,
spec=child,
seq=seq,
loop_index=loop_index,
checkpoint_after=bool(child.get("checkpoint", False))
or (bool(raw.get("checkpoint", False)) and index == len(body) - 1),
)
)
seq += 1
continue
planned.append(
_step_row(
run=run,
spec=raw,
seq=seq,
loop_index=None,
checkpoint_after=bool(raw.get("checkpoint", False)),
)
)
seq += 1
if not planned:
raise WorkflowPlanningError("工作流没有 Host Agent 可执行步骤")
return planned
async def materialize_run_steps(
session: AsyncSession, run: Run, workflow: Workflow
) -> List[RunStep]:
planned = plan_run_steps(run, workflow)
session.add_all(planned)
await session.flush()
return planned
flashops/services/control-plane/flashops_control/engine/recovery.py
+37
View File
@@ -0,0 +1,37 @@
"""五级恢复阶梯的纯函数决策。"""
from __future__ import annotations
from dataclasses import dataclass
from ..enums import RecoveryLevel, RecoveryTrigger
@dataclass(frozen=True)
class RecoveryContext:
trigger: RecoveryTrigger
attempt: int
agent_online: bool
host_network_online: bool
oob_online: bool
def next_recovery_level(context: RecoveryContext) -> RecoveryLevel:
if context.trigger == RecoveryTrigger.DATA_INTEGRITY:
return RecoveryLevel.FREEZE
ladder = []
if context.agent_online:
ladder.append(RecoveryLevel.AGENT_SOFT)
if context.host_network_online:
ladder.append(RecoveryLevel.OS_REBOOT)
if context.oob_online:
ladder.extend(
[
RecoveryLevel.OOB_RESET,
RecoveryLevel.OOB_ATX_POWER,
RecoveryLevel.OOB_AC_CYCLE,
]
)
if context.attempt >= len(ladder):
return RecoveryLevel.FREEZE
return ladder[context.attempt]
flashops/services/control-plane/flashops_control/engine/states.py
+44
View File
@@ -0,0 +1,44 @@
"""Run 状态转移表。"""
from __future__ import annotations
from typing import Dict, FrozenSet
from ..enums import RunState
class IllegalTransition(ValueError):
pass
RUN_TRANSITIONS: Dict[RunState, FrozenSet[RunState]] = {
RunState.QUEUED: frozenset({RunState.PREFLIGHT, RunState.ABORTED}),
RunState.PREFLIGHT: frozenset(
{RunState.RUNNING, RunState.REJECTED, RunState.ABORTED}
),
RunState.RUNNING: frozenset(
{
RunState.RECOVERING,
RunState.PAUSED,
RunState.FROZEN,
RunState.COMPLETED,
RunState.ABORTED,
}
),
RunState.RECOVERING: frozenset(
{RunState.RUNNING, RunState.FROZEN, RunState.ABORTED}
),
RunState.PAUSED: frozenset({RunState.RUNNING, RunState.ABORTED}),
RunState.FROZEN: frozenset(),
RunState.COMPLETED: frozenset(),
RunState.ABORTED: frozenset(),
RunState.REJECTED: frozenset(),
}
def assert_run_transition(current: str, target: str) -> None:
current_state = RunState(current)
target_state = RunState(target)
if target_state not in RUN_TRANSITIONS[current_state]:
raise IllegalTransition(
"Run 状态不允许从 {0} 转到 {1}".format(current_state.value, target_state.value)
)
flashops/services/control-plane/flashops_control/enums.py
+234
View File
@@ -0,0 +1,234 @@
"""全部领域枚举。
放在包根、不依赖任何东西 —— models / engine / schemas / api 都从这里取，
保证"状态"这个概念在全仓库只有一份定义。
"""
from __future__ import annotations
from enum import Enum
class StrEnum(str, Enum):
"""Python 3.9 没有 enum.StrEnum，自己来一个。"""
def __str__(self) -> str: # pragma: no cover - 只影响日志可读性
return self.value
# ---------------------------------------------------------------- Run / Step
class RunState(StrEnum):
QUEUED = "QUEUED" # 等资源
PREFLIGHT = "PREFLIGHT" # 安全门禁 + 环境指纹冻结
RUNNING = "RUNNING"
RECOVERING = "RECOVERING" # 恢复阶梯进行中
PAUSED = "PAUSED" # 人工暂停
FROZEN = "FROZEN" # 冻结现场，停止一切覆盖性动作（终态）
COMPLETED = "COMPLETED" # 终态
ABORTED = "ABORTED" # 紧急停止（终态）
REJECTED = "REJECTED" # 安全门禁拒绝（终态）
class StepState(StrEnum):
PENDING = "PENDING"
DISPATCHED = "DISPATCHED" # 已租给 Agent，等确认接手
RUNNING = "RUNNING"
SUCCEEDED = "SUCCEEDED"
FAILED = "FAILED"
TIMED_OUT = "TIMED_OUT"
SKIPPED = "SKIPPED"
CANCELLED = "CANCELLED"
class Verdict(StrEnum):
PASS = "PASS"
FAIL = "FAIL"
INCONCLUSIVE = "INCONCLUSIVE" # 非幂等步骤中断后不自动重跑 → 需人工确认
# ------------------------------------------------------------------ 恢复阶梯
class RecoveryLevel(StrEnum):
"""方案 §7.2 的五级阶梯。顺序即升级顺序。"""
AGENT_SOFT = "L1_AGENT_SOFT" # Agent 优雅停止 / 软重启进程
OS_REBOOT = "L2_OS_REBOOT" # OS 远程通道重启
OOB_RESET = "L3_OOB_RESET" # 带外触发主板 Reset
OOB_ATX_POWER = "L4_OOB_ATX_POWER" # 带外模拟 ATX 长按关机再开机
OOB_AC_CYCLE = "L5_OOB_AC_CYCLE" # 整机 AC 断电 → 安全间隔 → 上电
FREEZE = "FREEZE" # 冻结现场，不再自动恢复
@property
def requires_oob(self) -> bool:
return self in (
RecoveryLevel.OOB_RESET,
RecoveryLevel.OOB_ATX_POWER,
RecoveryLevel.OOB_AC_CYCLE,
)
class RecoveryOutcome(StrEnum):
RECOVERED = "RECOVERED"
FAILED = "FAILED"
ESCALATED = "ESCALATED"
FROZEN = "FROZEN"
class RecoveryTrigger(StrEnum):
HEARTBEAT_LOST = "HEARTBEAT_LOST"
STEP_TIMEOUT = "STEP_TIMEOUT"
HOST_CRASH = "HOST_CRASH" # 蓝屏 / kernel panic
DUT_NOT_ENUMERATED = "DUT_NOT_ENUMERATED"
DATA_INTEGRITY = "DATA_INTEGRITY" # → 直接 FREEZE，不走阶梯
MANUAL = "MANUAL"
# -------------------------------------------------------------------- 资产
class HostStatus(StrEnum):
ONLINE = "ONLINE"
DEGRADED = "DEGRADED" # 心跳迟到但未判失联
OFFLINE = "OFFLINE"
UNKNOWN = "UNKNOWN"
class DutStatus(StrEnum):
IDLE = "IDLE"
IN_USE = "IN_USE"
QUARANTINED = "QUARANTINED" # 疑似坏盘，隔离待人工确认
MISSING = "MISSING" # 探针找不到
class PowerState(StrEnum):
ON = "ON"
OFF = "OFF"
UNKNOWN = "UNKNOWN"
class DangerLevel(StrEnum):
NONE = "none"
LOW = "low"
HIGH = "high" # 需要审批记录
# -------------------------------------------------------------------- 失败
class FailureClass(StrEnum):
"""INFRA_FAILURE 必须与 DUT_DEFECT 分开统计 —— 见 domain-model.md。
把网络断了算成 SSD 缺陷，客户第一周就不信任这套系统。
"""
DUT_DEFECT = "DUT_DEFECT"
INFRA_FAILURE = "INFRA_FAILURE"
SCRIPT_FAILURE = "SCRIPT_FAILURE"
DATA_INTEGRITY = "DATA_INTEGRITY"
UNKNOWN = "UNKNOWN"
class IntegrityState(StrEnum):
OK = "OK"
MISMATCH = "MISMATCH"
NOT_CHECKED = "NOT_CHECKED"
# ------------------------------------------------------------- 事件（时间线）
class EventSource(StrEnum):
CONTROL_PLANE = "control_plane"
AGENT = "agent"
OOB = "oob" # 带外通道 —— 主机死后唯一还在说话的
HUMAN = "human"
class Severity(StrEnum):
DEBUG = "DEBUG"
INFO = "INFO"
WARNING = "WARNING"
ERROR = "ERROR"
CRITICAL = "CRITICAL"
class EventKind(StrEnum):
"""统一时间线的事件种类。
新增事件种类是常事；删除/改名不是 —— 历史事件已经落库，改名会让旧 Run 的
时间线读不出来。只增不改。
"""
# Run 生命周期
RUN_CREATED = "RUN_CREATED"
RUN_QUEUED = "RUN_QUEUED"
RUN_PREFLIGHT_STARTED = "RUN_PREFLIGHT_STARTED"
RUN_PREFLIGHT_PASSED = "RUN_PREFLIGHT_PASSED"
RUN_REJECTED = "RUN_REJECTED"
RUN_STARTED = "RUN_STARTED"
RUN_PAUSED = "RUN_PAUSED"
RUN_RESUMED = "RUN_RESUMED"
RUN_COMPLETED = "RUN_COMPLETED"
RUN_ABORTED = "RUN_ABORTED"
RUN_FROZEN = "RUN_FROZEN"
# 循环与步骤
LOOP_STARTED = "LOOP_STARTED"
LOOP_COMPLETED = "LOOP_COMPLETED"
STEP_DISPATCHED = "STEP_DISPATCHED"
STEP_LEASE_RENEWED = "STEP_LEASE_RENEWED"
STEP_LEASE_EXPIRED = "STEP_LEASE_EXPIRED"
STEP_STARTED = "STEP_STARTED"
STEP_OUTPUT = "STEP_OUTPUT"
STEP_SUCCEEDED = "STEP_SUCCEEDED"
STEP_FAILED = "STEP_FAILED"
STEP_TIMED_OUT = "STEP_TIMED_OUT"
STEP_RETRYING = "STEP_RETRYING"
STEP_CANCELLED = "STEP_CANCELLED"
# 检查点
CHECKPOINT_SAVED = "CHECKPOINT_SAVED"
CHECKPOINT_RESUMED = "CHECKPOINT_RESUMED"
# 心跳与双通道观测
HEARTBEAT_DEGRADED = "HEARTBEAT_DEGRADED"
HEARTBEAT_LOST = "HEARTBEAT_LOST"
HEARTBEAT_RECOVERED = "HEARTBEAT_RECOVERED"
OOB_OBSERVATION = "OOB_OBSERVATION"
OOB_POWER_STATE = "OOB_POWER_STATE"
# 恢复
RECOVERY_STARTED = "RECOVERY_STARTED"
RECOVERY_ESCALATED = "RECOVERY_ESCALATED"
RECOVERY_SUCCEEDED = "RECOVERY_SUCCEEDED"
RECOVERY_FAILED = "RECOVERY_FAILED"
# 设备与数据
DUT_ENUMERATION_LOST = "DUT_ENUMERATION_LOST"
DUT_ENUMERATION_RESTORED = "DUT_ENUMERATION_RESTORED"
INTEGRITY_MISMATCH = "INTEGRITY_MISMATCH"
# 安全与资源
SAFETY_APPROVED = "SAFETY_APPROVED"
SAFETY_REJECTED = "SAFETY_REJECTED"
LOCK_ACQUIRED = "LOCK_ACQUIRED"
LOCK_RELEASED = "LOCK_RELEASED"
# 人与产物
HUMAN_TOUCH = "HUMAN_TOUCH" # 每一次人工介入都记 —— UCR 的分母来源
FAILURE_SIGNED = "FAILURE_SIGNED"
EVIDENCE_BUNDLED = "EVIDENCE_BUNDLED"
class StepType(StrEnum):
COMMAND = "command"
DEVICE_CHECK = "device_check"
WAIT = "wait"
LOOP = "loop"
PRECHECK = "precheck"
REPORT = "report"
class OnFail(StrEnum):
RETRY = "retry"
FAIL_RUN = "fail_run"
CONTINUE = "continue"
FREEZE = "freeze"
RECOVER = "recover"
flashops/services/control-plane/flashops_control/events/__init__.py
+5
View File
@@ -0,0 +1,5 @@
"""事件溯源写入。"""
from .recorder import append_event
__all__ = ["append_event"]
flashops/services/control-plane/flashops_control/events/recorder.py
+54
View File
@@ -0,0 +1,54 @@
"""Run 事件唯一写入口，同时维护当前状态投影。"""
from __future__ import annotations
from typing import Any, Dict, Optional
from sqlalchemy.ext.asyncio import AsyncSession
from ..engine.states import assert_run_transition
from ..enums import EventSource, Severity
from ..models import Run, RunEvent, utcnow
async def append_event(
session: AsyncSession,
run: Run,
kind: str,
message: str,
*,
source: str = EventSource.CONTROL_PLANE.value,
severity: str = Severity.INFO.value,
payload: Optional[Dict[str, Any]] = None,
target_state: Optional[str] = None,
projection: Optional[Dict[str, Any]] = None,
idempotency_key: Optional[str] = None,
step_id: Optional[str] = None,
loop_index: Optional[int] = None,
) -> RunEvent:
if target_state is not None and target_state != run.state:
assert_run_transition(run.state, target_state)
run.event_seq += 1
event = RunEvent(
run_id=run.id,
seq=run.event_seq,
ts=utcnow(),
source=source,
kind=kind,
severity=severity,
message=message,
payload=payload or {},
idempotency_key=idempotency_key,
step_id=step_id,
loop_index=loop_index,
)
session.add(event)
if target_state is not None:
run.state = target_state
for key, value in (projection or {}).items():
if not hasattr(run, key):
raise AttributeError("Run 不存在投影字段: {0}".format(key))
setattr(run, key, value)
await session.flush()
return event
flashops/services/control-plane/flashops_control/evidence/__init__.py
+5
View File
@@ -0,0 +1,5 @@
"""Evidence Bundle 生成。"""
from .bundle import build_evidence_bundle
__all__ = ["build_evidence_bundle"]
flashops/services/control-plane/flashops_control/evidence/bundle.py
+69
View File
@@ -0,0 +1,69 @@
"""把一次 Run 的可重放事件打成最小证据包。"""
from __future__ import annotations
import json
from pathlib import Path
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from ..config import get_settings
from ..models import EvidenceBundle, Run, RunEvent, iso_z
async def build_evidence_bundle(session: AsyncSession, run: Run) -> EvidenceBundle:
events = (
await session.scalars(
select(RunEvent).where(RunEvent.run_id == run.id).order_by(RunEvent.seq)
)
).all()
root = get_settings().object_store_path / "runs" / run.id
root.mkdir(parents=True, exist_ok=True)
timeline_path = root / "events.ndjson"
timeline = "\n".join(
json.dumps(
{
"seq": event.seq,
"ts": iso_z(event.ts),
"source": event.source,
"kind": event.kind,
"severity": event.severity,
"message": event.message,
"payload": event.payload,
},
ensure_ascii=False,
sort_keys=True,
)
for event in events
)
timeline_path.write_text(timeline + ("\n" if timeline else ""), encoding="utf-8")
manifest = {
"schema_version": 1,
"run_id": run.id,
"workflow": {
"key": run.workflow_key,
"version": run.workflow_version,
"spec_hash": run.spec_hash,
},
"environment_fingerprint": run.env_fingerprint,
"checkpoint": run.checkpoint,
"event_count": len(events),
"files": [{"path": "events.ndjson", "bytes": timeline_path.stat().st_size}],
}
manifest_path = root / "manifest.json"
manifest_path.write_text(
json.dumps(manifest, ensure_ascii=False, indent=2, sort_keys=True),
encoding="utf-8",
)
size = manifest_path.stat().st_size + timeline_path.stat().st_size
bundle = EvidenceBundle(
run_id=run.id,
uri=str(root),
manifest=manifest,
size_bytes=size,
completeness=1.0,
missing_fields=[],
)
session.add(bundle)
await session.flush()
return bundle
flashops/services/control-plane/flashops_control/main.py
+53
View File
@@ -0,0 +1,53 @@
"""FlashOps FastAPI 应用。"""
from __future__ import annotations
import asyncio
from contextlib import asynccontextmanager
from pathlib import Path
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import RedirectResponse
from fastapi.staticfiles import StaticFiles
from .api.agents import router as agent_router
from .api.routes import router
from .config import APP_NAME, APP_TAGLINE, REPO_ROOT, get_settings
from .db import init_db, session_scope
from .seed import ensure_seed_data
from .services.agent_status import run_agent_watchdog, stop_agent_watchdog
@asynccontextmanager
async def lifespan(_: FastAPI):
await init_db()
async with session_scope() as session:
await ensure_seed_data(session)
watchdog = asyncio.create_task(run_agent_watchdog(), name="agent-heartbeat-watchdog")
try:
yield
finally:
await stop_agent_watchdog(watchdog)
app = FastAPI(title=APP_NAME, description=APP_TAGLINE, version="0.1.0", lifespan=lifespan)
app.add_middleware(
CORSMiddleware,
allow_origins=[origin.strip() for origin in get_settings().cors_origins.split(",")],
allow_credentials=True,
allow_methods=["*"],
allow_headers=["*"],
)
app.include_router(router)
app.include_router(agent_router)
console_dir = REPO_ROOT.parent / "前端UI八页面完成"
if console_dir.exists():
app.mount("/console", StaticFiles(directory=str(console_dir), html=True), name="console")
@app.get("/", include_in_schema=False)
async def root() -> RedirectResponse:
if console_dir.exists():
return RedirectResponse("/console/Dashboard.dc.html")
return RedirectResponse("/docs")
flashops/services/control-plane/flashops_control/models/__init__.py
+31
View File
@@ -0,0 +1,31 @@
"""11 张表 → 方案 §5.3 的 7 个核心领域对象。映射见 docs/domain-model.md。"""
from .analysis import AuditLog, EvidenceBundle, FailureSignature, RunFailure
from .assets import Dut, FirmwareArtifact, OobController, TestHost
from .base import Base, iso_z, new_id, utcnow
from .run import RecoveryAction, ResourceLock, Run, RunEvent, RunStep
from .workflow import Workflow
__all__ = [
"Base",
"utcnow",
"new_id",
"iso_z",
# 资产
"TestHost",
"OobController",
"Dut",
"FirmwareArtifact",
# 工作流
"Workflow",
# Run
"Run",
"RunStep",
"RunEvent",
"RecoveryAction",
"ResourceLock",
# 分析
"FailureSignature",
"RunFailure",
"EvidenceBundle",
"AuditLog",
]
flashops/services/control-plane/flashops_control/models/analysis.py
+112
View File
@@ -0,0 +1,112 @@
"""失败签名、失败实例、Evidence Bundle、审计。"""
from __future__ import annotations
import datetime as _dt
from typing import Any, Dict, List, Optional
from sqlalchemy import (
DateTime,
Float,
ForeignKey,
Index,
Integer,
JSON,
String,
Text,
UniqueConstraint,
)
from sqlalchemy.orm import Mapped, mapped_column
from ..enums import FailureClass, IntegrityState
from .base import Base, TimestampMixin, pk
class FailureSignature(Base, TimestampMixin):
"""失败签名 —— 聚类锚点。
8 要素哈希（方案 §8.3）。只用错误码会把"掉盘"和"脚本超时"归成一类，
聚类就废了；用原始日志则每条日志都是新签名。所以用模板化后的日志指纹。
"""
__tablename__ = "failure_signatures"
__table_args__ = (UniqueConstraint("hash", name="uq_signature_hash"),)
id: Mapped[str] = pk("sig")
hash: Mapped[str] = mapped_column(String(64), nullable=False)
title: Mapped[str] = mapped_column(String(255), nullable=False)
# 8 要素
workflow_step: Mapped[str] = mapped_column(String(64), nullable=False)
error_codes: Mapped[List[Any]] = mapped_column(JSON, default=list)
log_templates: Mapped[List[Any]] = mapped_column(JSON, default=list)
host_state: Mapped[str] = mapped_column(String(32), default="UNKNOWN")
dut_enumeration_state: Mapped[str] = mapped_column(String(32), default="UNKNOWN")
data_integrity_state: Mapped[str] = mapped_column(String(16), default=IntegrityState.NOT_CHECKED.value)
recovery_outcome: Mapped[Optional[str]] = mapped_column(String(16))
environment_fingerprint_hash: Mapped[Optional[str]] = mapped_column(String(64))
failure_class: Mapped[str] = mapped_column(String(24), default=FailureClass.UNKNOWN.value)
occurrences: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
first_seen_at: Mapped[Optional[_dt.datetime]] = mapped_column(DateTime)
last_seen_at: Mapped[Optional[_dt.datetime]] = mapped_column(DateTime)
# 人工确认后的处置：如已提单号、已知问题、误报
triage_note: Mapped[Optional[str]] = mapped_column(Text)
issue_ref: Mapped[Optional[str]] = mapped_column(String(64))
class RunFailure(Base):
"""一次具体的失败实例，挂在签名下面。"""
__tablename__ = "run_failures"
__table_args__ = (Index("ix_failure_run", "run_id"),)
id: Mapped[str] = pk("fail")
run_id: Mapped[str] = mapped_column(ForeignKey("runs.id"), nullable=False)
signature_id: Mapped[Optional[str]] = mapped_column(ForeignKey("failure_signatures.id"))
step_id: Mapped[Optional[str]] = mapped_column(String(32))
loop_index: Mapped[Optional[int]] = mapped_column(Integer)
ts: Mapped[_dt.datetime] = mapped_column(DateTime, nullable=False)
failure_class: Mapped[str] = mapped_column(String(24), default=FailureClass.UNKNOWN.value)
summary: Mapped[str] = mapped_column(Text, nullable=False)
detail: Mapped[Dict[str, Any]] = mapped_column(JSON, default=dict)
# 记录固件版本，A/B 对比直接从这里聚合
firmware_label: Mapped[Optional[str]] = mapped_column(String(64))
class EvidenceBundle(Base, TimestampMixin):
"""证据包。completeness < 1 说明有关键字段缺失 —— MVP 硬指标是 ≥95%。"""
__tablename__ = "evidence_bundles"
id: Mapped[str] = pk("bundle")
run_id: Mapped[str] = mapped_column(ForeignKey("runs.id"), nullable=False)
uri: Mapped[str] = mapped_column(String(512), nullable=False)
manifest: Mapped[Dict[str, Any]] = mapped_column(JSON, default=dict)
size_bytes: Mapped[int] = mapped_column(Integer, default=0)
completeness: Mapped[float] = mapped_column(Float, default=0.0)
missing_fields: Mapped[List[Any]] = mapped_column(JSON, default=list)
class AuditLog(Base):
"""审计：谁、什么时候、用哪个模板版本、对什么、做了什么、结果如何。
方案 §6.6 的硬要求。与决策放在同一处 —— 见 ADR-0004。
"""
__tablename__ = "audit_log"
__table_args__ = (Index("ix_audit_ts", "ts"),)
id: Mapped[str] = pk("audit")
ts: Mapped[_dt.datetime] = mapped_column(DateTime, nullable=False)
actor: Mapped[str] = mapped_column(String(64), nullable=False)
action: Mapped[str] = mapped_column(String(64), nullable=False)
target_type: Mapped[Optional[str]] = mapped_column(String(32))
target_id: Mapped[Optional[str]] = mapped_column(String(32))
command_template: Mapped[Optional[str]] = mapped_column(String(64))
template_version: Mapped[Optional[str]] = mapped_column(String(16))
params: Mapped[Dict[str, Any]] = mapped_column(JSON, default=dict)
result: Mapped[Optional[str]] = mapped_column(String(32))
approved_by: Mapped[Optional[str]] = mapped_column(String(64))
reason: Mapped[Optional[str]] = mapped_column(Text)
flashops/services/control-plane/flashops_control/models/assets.py
+111
View File
@@ -0,0 +1,111 @@
"""资产：测试主机、带外控制器、DUT、固件包。"""
from __future__ import annotations
import datetime as _dt
from typing import Any, Dict, List, Optional
from sqlalchemy import Boolean, DateTime, ForeignKey, Integer, JSON, String, UniqueConstraint
from sqlalchemy.orm import Mapped, mapped_column
from ..enums import DutStatus, HostStatus, PowerState
from .base import Base, TimestampMixin, pk
class OobController(Base, TimestampMixin):
"""带外控制器。
独立成表而不是挂在 TestHost 上，因为它的核心价值就是**主机死后独活**：
它有自己的网络、自己的心跳、自己的生命周期。主机 OFFLINE 时它必须还 ONLINE，
这个"矛盾"状态是恢复阶梯 L3-L5 可用性的判断依据。
"""
__tablename__ = "oob_controllers"
id: Mapped[str] = pk("oob")
name: Mapped[str] = mapped_column(String(128), nullable=False)
kind: Mapped[str] = mapped_column(String(32), default="simulated") # raspberry_pi / jetkvm / pdu / simulated
endpoint: Mapped[Optional[str]] = mapped_column(String(255))
# 能力位：{"power": true, "reset": true, "atx": true, "ac": true, "hdmi": true, "temp": true}
capabilities: Mapped[Dict[str, Any]] = mapped_column(JSON, default=dict)
status: Mapped[str] = mapped_column(String(16), default=HostStatus.UNKNOWN.value)
power_state: Mapped[str] = mapped_column(String(16), default=PowerState.UNKNOWN.value)
last_seen_at: Mapped[Optional[_dt.datetime]] = mapped_column(DateTime)
temperature_c: Mapped[Optional[float]] = mapped_column()
class TestHost(Base, TimestampMixin):
"""测试主机。环境指纹的主要来源 —— A/B 对比的可比性靠它固定。"""
__tablename__ = "test_hosts"
id: Mapped[str] = pk("host")
name: Mapped[str] = mapped_column(String(128), nullable=False, unique=True)
os_family: Mapped[str] = mapped_column(String(32), default="linux") # linux / windows
os_version: Mapped[Optional[str]] = mapped_column(String(128))
kernel: Mapped[Optional[str]] = mapped_column(String(128))
cpu: Mapped[Optional[str]] = mapped_column(String(128))
motherboard: Mapped[Optional[str]] = mapped_column(String(128))
bios_version: Mapped[Optional[str]] = mapped_column(String(64))
agent_version: Mapped[Optional[str]] = mapped_column(String(32))
ip_address: Mapped[Optional[str]] = mapped_column(String(64))
status: Mapped[str] = mapped_column(String(16), default=HostStatus.UNKNOWN.value)
last_heartbeat_at: Mapped[Optional[_dt.datetime]] = mapped_column(DateTime)
agent_id: Mapped[Optional[str]] = mapped_column(String(32))
# 只保存 bearer token 的 SHA-256 摘要；原始 token 仅在注册响应中返回一次。
agent_token: Mapped[Optional[str]] = mapped_column(String(64))
oob_controller_id: Mapped[Optional[str]] = mapped_column(ForeignKey("oob_controllers.id"))
# 已安装工具与版本，来自 Agent 的 discover()：{"nvme-cli": "2.8", "fio": "3.36"}
toolchain: Mapped[Dict[str, Any]] = mapped_column(JSON, default=dict)
labels: Mapped[Dict[str, Any]] = mapped_column(JSON, default=dict)
class Dut(Base, TimestampMixin):
"""被测设备。
allow_destructive 是防误盘的第二道信号（第一道是序列号，第三道是系统盘检测）。
它不给普通 API 改 —— 只能走带审批记录的资产接口，见 ADR-0004。
"""
__tablename__ = "duts"
__table_args__ = (UniqueConstraint("serial", name="uq_dut_serial"),)
id: Mapped[str] = pk("dut")
serial: Mapped[str] = mapped_column(String(64), nullable=False)
model: Mapped[str] = mapped_column(String(128), nullable=False)
vendor: Mapped[Optional[str]] = mapped_column(String(64))
capacity_gb: Mapped[Optional[int]] = mapped_column(Integer)
form_factor: Mapped[str] = mapped_column(String(32), default="M.2") # M.2 / U.2 / E1.S
interface: Mapped[str] = mapped_column(String(32), default="NVMe")
current_firmware: Mapped[Optional[str]] = mapped_column(String(64))
bdf: Mapped[Optional[str]] = mapped_column(String(32)) # PCIe 地址，如 0000:03:00.0
device_path: Mapped[Optional[str]] = mapped_column(String(64)) # /dev/nvme0n1
test_host_id: Mapped[Optional[str]] = mapped_column(ForeignKey("test_hosts.id"))
status: Mapped[str] = mapped_column(String(16), default=DutStatus.IDLE.value)
# 破坏性测试白名单标记。默认 False —— 安全默认拒绝。
allow_destructive: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False)
health: Mapped[Dict[str, Any]] = mapped_column(JSON, default=dict) # SMART 关键项、温度、寿命
labels: Mapped[Dict[str, Any]] = mapped_column(JSON, default=dict)
class FirmwareArtifact(Base, TimestampMixin):
"""固件包。哈希 + 签名 + 适用型号，缺一样就不可审计。"""
__tablename__ = "firmware_artifacts"
id: Mapped[str] = pk("fw")
version: Mapped[str] = mapped_column(String(64), nullable=False)
label: Mapped[Optional[str]] = mapped_column(String(64)) # "A" / "B" / "候选版"
sha256: Mapped[str] = mapped_column(String(64), nullable=False)
signature: Mapped[Optional[str]] = mapped_column(String(512))
applicable_models: Mapped[List[Any]] = mapped_column(JSON, default=list)
flash_tool: Mapped[str] = mapped_column(String(64), default="nvme_cli")
file_uri: Mapped[Optional[str]] = mapped_column(String(512))
uploaded_by: Mapped[Optional[str]] = mapped_column(String(64))
flashops/services/control-plane/flashops_control/models/base.py
+64
View File
@@ -0,0 +1,64 @@
"""ORM 基类与通用字段。
刻意保持"贫血模型"：这些类只管持久化，不放业务逻辑。
业务逻辑在 engine/（纯函数）与 services/（编排），这样才能不起数据库单测。
"""
from __future__ import annotations
import datetime as _dt
import uuid
from typing import Any, Dict
from sqlalchemy import DateTime, JSON, String
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
class Base(DeclarativeBase):
"""所有表的基类。
type_annotation_map 里只用方言中立的类型：SQLite（开发）与 PostgreSQL（生产）
行为一致。不要引入 JSONB / ARRAY / UUID 这些 PG 特有类型 —— 见 ADR-0001。
"""
type_annotation_map = {
Dict[str, Any]: JSON,
dict: JSON,
list: JSON,
}
def utcnow() -> _dt.datetime:
"""统一的当前时间：naive UTC。
SQLite 不保存时区，所以全仓库统一存 naive UTC，序列化时再补 Z。
禁止在任何地方用 datetime.now()（本地时区）—— 实验室里跨时区协作会出事。
"""
return _dt.datetime.utcnow()
def new_id(prefix: str) -> str:
"""带前缀的 ID，如 run_7f3a9c2e...。
前缀让日志和时间线一眼可读 —— 排障时不用回表查这个 ID 是什么东西。
"""
return "{0}_{1}".format(prefix, uuid.uuid4().hex[:20])
def iso_z(value: Any) -> Any:
"""naive UTC datetime → ISO8601 带 Z。非 datetime 原样返回。"""
if isinstance(value, _dt.datetime):
return value.replace(microsecond=value.microsecond).isoformat() + "Z"
return value
class TimestampMixin:
created_at: Mapped[_dt.datetime] = mapped_column(DateTime, default=utcnow, nullable=False)
updated_at: Mapped[_dt.datetime] = mapped_column(
DateTime, default=utcnow, onupdate=utcnow, nullable=False
)
def pk(prefix: str) -> Mapped[str]:
return mapped_column(
String(32), primary_key=True, default=lambda: new_id(prefix)
)
flashops/services/control-plane/flashops_control/models/run.py
+188
View File
@@ -0,0 +1,188 @@
"""Run 及其行为记录。
runs / run_steps / recovery_actions 是**读模型**（投影）。
唯一真相是 run_events（append-only）—— 见 ADR-0002。
仓储层刻意不提供 update_run_state()：改状态的唯一入口是 events/recorder.py。
"""
from __future__ import annotations
import datetime as _dt
from typing import Any, Dict, List, Optional
from sqlalchemy import (
Boolean,
DateTime,
Float,
ForeignKey,
Index,
Integer,
JSON,
String,
Text,
UniqueConstraint,
)
from sqlalchemy.orm import Mapped, mapped_column
from ..enums import EventSource, RunState, Severity, StepState
from .base import Base, TimestampMixin, pk
class Run(Base, TimestampMixin):
__tablename__ = "runs"
id: Mapped[str] = pk("run")
name: Mapped[Optional[str]] = mapped_column(String(255))
workflow_id: Mapped[str] = mapped_column(ForeignKey("workflows.id"), nullable=False)
workflow_key: Mapped[str] = mapped_column(String(64), nullable=False)
workflow_version: Mapped[int] = mapped_column(Integer, nullable=False)
spec_hash: Mapped[str] = mapped_column(String(64), nullable=False)
state: Mapped[str] = mapped_column(String(16), default=RunState.QUEUED.value, nullable=False)
verdict: Mapped[Optional[str]] = mapped_column(String(16))
dut_id: Mapped[Optional[str]] = mapped_column(ForeignKey("duts.id"))
test_host_id: Mapped[Optional[str]] = mapped_column(ForeignKey("test_hosts.id"))
firmware_a_id: Mapped[Optional[str]] = mapped_column(ForeignKey("firmware_artifacts.id"))
firmware_b_id: Mapped[Optional[str]] = mapped_column(ForeignKey("firmware_artifacts.id"))
params: Mapped[Dict[str, Any]] = mapped_column(JSON, default=dict)
# 任务开始时冻结的不可变快照。固件之外任何一项变了，A/B 对比结论就不成立。
env_fingerprint: Mapped[Dict[str, Any]] = mapped_column(JSON, default=dict)
env_fingerprint_hash: Mapped[Optional[str]] = mapped_column(String(64))
loop_target: Mapped[int] = mapped_column(Integer, default=1)
loop_done: Mapped[int] = mapped_column(Integer, default=0)
# 最近一次检查点：{"loop_index": 37, "next_step_key": "...", "dut_fw": "A"}
checkpoint: Mapped[Dict[str, Any]] = mapped_column(JSON, default=dict)
started_at: Mapped[Optional[_dt.datetime]] = mapped_column(DateTime)
ended_at: Mapped[Optional[_dt.datetime]] = mapped_column(DateTime)
# ROI 凭据（方案 §9.3）：这两个字段是销售武器，从第一行代码就记。
unattended_completion: Mapped[bool] = mapped_column(Boolean, default=True, nullable=False)
human_touches: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
recovery_count: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
# 事件序号分配器（单点递增，保证时间线可重放，见 ADR-0002）
event_seq: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
created_by: Mapped[Optional[str]] = mapped_column(String(64))
freeze_reason: Mapped[Optional[str]] = mapped_column(Text)
class RunStep(Base, TimestampMixin):
"""物化后的步骤实例。循环体在物化时按 loop_index 展开。"""
__tablename__ = "run_steps"
__table_args__ = (
UniqueConstraint("run_id", "seq", name="uq_step_run_seq"),
Index("ix_step_run_state", "run_id", "state"),
)
id: Mapped[str] = pk("step")
run_id: Mapped[str] = mapped_column(ForeignKey("runs.id"), nullable=False)
seq: Mapped[int] = mapped_column(Integer, nullable=False)
loop_index: Mapped[Optional[int]] = mapped_column(Integer)
step_key: Mapped[str] = mapped_column(String(64), nullable=False)
step_type: Mapped[str] = mapped_column(String(32), nullable=False)
adapter: Mapped[str] = mapped_column(String(32), nullable=False)
template: Mapped[Optional[str]] = mapped_column(String(64))
params: Mapped[Dict[str, Any]] = mapped_column(JSON, default=dict)
state: Mapped[str] = mapped_column(String(16), default=StepState.PENDING.value, nullable=False)
attempt: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
max_retry: Mapped[int] = mapped_column(Integer, default=0, nullable=False)
timeout_s: Mapped[float] = mapped_column(Float, default=600.0, nullable=False)
# 非幂等步骤（默认）中断后不自动重跑 —— "自动重试把盘刷坏"是最容易砸招牌的失败模式
idempotent: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False)
danger: Mapped[str] = mapped_column(String(16), default="none")
on_fail: Mapped[str] = mapped_column(String(16), default="retry")
checkpoint_after: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False)
# 租约：Agent 领了活就死了 → 租约到期 → 步骤回 PENDING 重新派发
leased_by: Mapped[Optional[str]] = mapped_column(String(32))
lease_expires_at: Mapped[Optional[_dt.datetime]] = mapped_column(DateTime)
started_at: Mapped[Optional[_dt.datetime]] = mapped_column(DateTime)
ended_at: Mapped[Optional[_dt.datetime]] = mapped_column(DateTime)
exit_code: Mapped[Optional[int]] = mapped_column(Integer)
error_class: Mapped[Optional[str]] = mapped_column(String(32))
result: Mapped[Dict[str, Any]] = mapped_column(JSON, default=dict) # UnifiedResult
log_uri: Mapped[Optional[str]] = mapped_column(String(512))
class RunEvent(Base):
"""Append-only 事件流 —— 唯一真相。
seq 由控制平面单点分配（不按时间戳排序）：主机断电、时钟漂移、
带外与主机时钟不同步都会让时间戳乱序。Agent 事件与带外事件进同一条序列，
双通道观测的交叉校验靠它。
"""
__tablename__ = "run_events"
__table_args__ = (
UniqueConstraint("run_id", "seq", name="uq_event_run_seq"),
# 网络抖动重发不产生重复事件（见 agent-protocol.md）
UniqueConstraint("run_id", "idempotency_key", name="uq_event_idem"),
Index("ix_event_run_seq", "run_id", "seq"),
)
id: Mapped[str] = pk("ev")
run_id: Mapped[str] = mapped_column(ForeignKey("runs.id"), nullable=False)
seq: Mapped[int] = mapped_column(Integer, nullable=False)
ts: Mapped[_dt.datetime] = mapped_column(DateTime, nullable=False)
source: Mapped[str] = mapped_column(String(16), default=EventSource.CONTROL_PLANE.value)
kind: Mapped[str] = mapped_column(String(48), nullable=False)
severity: Mapped[str] = mapped_column(String(16), default=Severity.INFO.value)
step_id: Mapped[Optional[str]] = mapped_column(String(32))
loop_index: Mapped[Optional[int]] = mapped_column(Integer)
message: Mapped[Optional[str]] = mapped_column(Text)
payload: Mapped[Dict[str, Any]] = mapped_column(JSON, default=dict)
idempotency_key: Mapped[Optional[str]] = mapped_column(String(64))
class RecoveryAction(Base):
"""每一级恢复的留证记录。恢复前抓画面/温度，恢复后验证 DUT 重新枚举。"""
__tablename__ = "recovery_actions"
id: Mapped[str] = pk("rec")
run_id: Mapped[str] = mapped_column(ForeignKey("runs.id"), nullable=False)
step_id: Mapped[Optional[str]] = mapped_column(String(32))
loop_index: Mapped[Optional[int]] = mapped_column(Integer)
level: Mapped[str] = mapped_column(String(24), nullable=False)
trigger: Mapped[str] = mapped_column(String(32), nullable=False)
outcome: Mapped[Optional[str]] = mapped_column(String(16))
started_at: Mapped[_dt.datetime] = mapped_column(DateTime, nullable=False)
ended_at: Mapped[Optional[_dt.datetime]] = mapped_column(DateTime)
detail: Mapped[Dict[str, Any]] = mapped_column(JSON, default=dict)
evidence_uris: Mapped[List[Any]] = mapped_column(JSON, default=list)
class ResourceLock(Base):
"""DUT / 主机独占锁。PREFLIGHT 获取，终态释放。
唯一约束在 (resource_type, resource_id) —— 数据库来保证互斥，不靠应用层自觉。
"""
__tablename__ = "resource_locks"
__table_args__ = (
UniqueConstraint("resource_type", "resource_id", name="uq_lock_resource"),
)
id: Mapped[str] = pk("lock")
resource_type: Mapped[str] = mapped_column(String(16), nullable=False) # dut / test_host / oob
resource_id: Mapped[str] = mapped_column(String(32), nullable=False)
run_id: Mapped[str] = mapped_column(ForeignKey("runs.id"), nullable=False)
acquired_at: Mapped[_dt.datetime] = mapped_column(DateTime, nullable=False)
expires_at: Mapped[Optional[_dt.datetime]] = mapped_column(DateTime)
flashops/services/control-plane/flashops_control/models/workflow.py
+35
View File
@@ -0,0 +1,35 @@
"""工作流：SOP 的可执行形态。"""
from __future__ import annotations
from typing import Any, Dict, Optional
from sqlalchemy import Boolean, Integer, JSON, String, UniqueConstraint
from sqlalchemy.orm import Mapped, mapped_column
from ..enums import DangerLevel
from .base import Base, TimestampMixin, pk
class Workflow(Base, TimestampMixin):
"""工作流定义。
spec 存原文，spec_hash 存规范化后的哈希。spec_hash 变了而 version 没变 →
拒绝发布（见 workflow-spec.md）。理由：A/B 对比的结论必须能追溯到
具体哪一版流程，否则"上周跑的和这周跑的是不是同一个流程"说不清。
"""
__tablename__ = "workflows"
__table_args__ = (UniqueConstraint("key", "version", name="uq_workflow_key_version"),)
id: Mapped[str] = pk("wf")
key: Mapped[str] = mapped_column(String(64), nullable=False)
name: Mapped[str] = mapped_column(String(128), nullable=False)
version: Mapped[int] = mapped_column(Integer, default=1, nullable=False)
spec: Mapped[Dict[str, Any]] = mapped_column(JSON, nullable=False)
spec_hash: Mapped[str] = mapped_column(String(64), nullable=False)
danger_level: Mapped[str] = mapped_column(String(16), default=DangerLevel.NONE.value)
source_sop: Mapped[Optional[str]] = mapped_column(String(255))
published: Mapped[bool] = mapped_column(Boolean, default=False, nullable=False)
approved_by: Mapped[Optional[str]] = mapped_column(String(64))
flashops/services/control-plane/flashops_control/safety/__init__.py
+5
View File
@@ -0,0 +1,5 @@
"""危险动作的安全门禁。"""
from .gates import PreflightRequest, PreflightResult, run_preflight
__all__ = ["PreflightRequest", "PreflightResult", "run_preflight"]
flashops/services/control-plane/flashops_control/safety/gates.py
+90
View File
@@ -0,0 +1,90 @@
"""安全默认拒绝的预检规则。"""
from __future__ import annotations
from dataclasses import asdict, dataclass
from pathlib import PurePath
from typing import Dict, List, Sequence
@dataclass(frozen=True)
class Check:
key: str
name: str
passed: bool
note: str
@dataclass(frozen=True)
class PreflightRequest:
expected_serial: str
discovered_serial: str
allow_destructive: bool
device_path: str
system_device_paths: Sequence[str]
firmware_model_allowed: bool
host_online: bool
oob_online: bool
@dataclass(frozen=True)
class PreflightResult:
passed: bool
checks: List[Check]
def as_dict(self) -> Dict[str, object]:
return {
"passed": self.passed,
"checks": [asdict(check) for check in self.checks],
}
def _same_device(left: str, right: str) -> bool:
left_name = PurePath(left).name
right_name = PurePath(right).name
if left_name == right_name:
return True
# /dev/nvme0n1p2 属于 /dev/nvme0n1；拒绝把其任一分区当作测试盘。
return left_name.startswith(right_name + "p") or right_name.startswith(left_name + "p")
def run_preflight(request: PreflightRequest) -> PreflightResult:
serial_ok = bool(request.expected_serial) and (
request.expected_serial == request.discovered_serial
)
system_disk_safe = bool(request.device_path) and not any(
_same_device(request.device_path, path) for path in request.system_device_paths
)
checks = [
Check("serial", "DUT 序列号白名单绑定", serial_ok, request.discovered_serial or "未发现"),
Check(
"destructive",
"破坏性写入授权",
request.allow_destructive,
"已审批" if request.allow_destructive else "DUT 未授权破坏性写入",
),
Check(
"system_disk",
"系统盘 / 分区保护",
system_disk_safe,
"启动盘已排除" if system_disk_safe else "目标命中系统盘或其分区",
),
Check(
"firmware",
"固件适配型号",
request.firmware_model_allowed,
"型号匹配" if request.firmware_model_allowed else "固件不适用于该型号",
),
Check(
"host",
"测试主机与 Agent",
request.host_online,
"在线" if request.host_online else "主机离线",
),
Check(
"oob",
"带外控制器",
request.oob_online,
"Power / Reset 可用" if request.oob_online else "带外通道离线",
),
]
return PreflightResult(all(check.passed for check in checks), checks)
flashops/services/control-plane/flashops_control/schemas.py
+80
View File
@@ -0,0 +1,80 @@
"""HTTP API 的 Pydantic 契约。"""
from __future__ import annotations
from typing import Any, Dict, List, Literal, Optional
from pydantic import BaseModel, Field
class PreflightBody(BaseModel):
dut_id: Optional[str] = None
host_id: Optional[str] = None
firmware_id: Optional[str] = None
discovered_serial: Optional[str] = None
class CreateRunBody(BaseModel):
name: str = "固件 A/B 无人值守回归"
workflow_id: Optional[str] = None
dut_id: Optional[str] = None
host_id: Optional[str] = None
firmware_a_id: Optional[str] = None
firmware_b_id: Optional[str] = None
loops: int = Field(default=8, ge=1, le=500)
inject_failure: bool = True
execution_mode: Literal["simulated", "agent_dry_run"] = "simulated"
created_by: str = "console-demo"
class HumanActionBody(BaseModel):
actor: str = "console-user"
reason: str = "人工操作"
class AgentRegisterBody(BaseModel):
host_id: Optional[str] = None
host_name: Optional[str] = None
agent_version: str = Field(min_length=1, max_length=32)
os_family: str = Field(default="linux", min_length=1, max_length=32)
os_version: Optional[str] = Field(default=None, max_length=128)
kernel: Optional[str] = Field(default=None, max_length=128)
cpu: Optional[str] = Field(default=None, max_length=128)
motherboard: Optional[str] = Field(default=None, max_length=128)
bios_version: Optional[str] = Field(default=None, max_length=64)
ip_address: Optional[str] = Field(default=None, max_length=64)
toolchain: Dict[str, Any] = Field(default_factory=dict)
labels: Dict[str, Any] = Field(default_factory=dict)
class AgentHeartbeatBody(BaseModel):
healthy: bool = True
agent_version: Optional[str] = Field(default=None, max_length=32)
ip_address: Optional[str] = Field(default=None, max_length=64)
toolchain: Dict[str, Any] = Field(default_factory=dict)
device_probe: Dict[str, Any] = Field(default_factory=dict)
class AgentLeaseBody(BaseModel):
wait_timeout_s: float = Field(default=0.0, ge=0.0, le=20.0)
class AgentStepAttemptBody(BaseModel):
attempt: int = Field(ge=1)
class AgentStepEvent(BaseModel):
message: str = Field(min_length=1, max_length=2000)
severity: Literal["DEBUG", "INFO", "WARNING", "ERROR"] = "INFO"
payload: Dict[str, Any] = Field(default_factory=dict)
class AgentStepEventsBody(AgentStepAttemptBody):
events: List[AgentStepEvent] = Field(min_length=1, max_length=100)
class AgentStepCompleteBody(AgentStepAttemptBody):
status: Literal["SUCCEEDED", "FAILED"]
exit_code: Optional[int] = None
error_class: Optional[str] = Field(default=None, max_length=32)
result: Dict[str, Any] = Field(default_factory=dict)
log_uri: Optional[str] = Field(default=None, max_length=512)
flashops/services/control-plane/flashops_control/seed.py
+141
View File
@@ -0,0 +1,141 @@
"""可重复执行的演示数据。"""
from __future__ import annotations
import hashlib
import json
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from .enums import DutStatus, HostStatus, PowerState
from .models import Dut, FirmwareArtifact, OobController, TestHost, Workflow, utcnow
def _spec_hash(spec: dict) -> str:
raw = json.dumps(spec, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
return hashlib.sha256(raw.encode("utf-8")).hexdigest()
async def ensure_seed_data(session: AsyncSession) -> None:
existing = await session.scalar(select(Workflow.id).limit(1))
if existing:
return
oob = OobController(
name="OOB-05",
kind="simulated",
endpoint="sim://oob-05",
capabilities={"power": True, "reset": True, "atx": True, "ac": True, "hdmi": True},
status=HostStatus.ONLINE.value,
power_state=PowerState.ON.value,
last_seen_at=utcnow(),
temperature_c=42.6,
)
session.add(oob)
await session.flush()
host = TestHost(
name="WS-05",
os_family="linux",
os_version="Ubuntu 22.04.4 LTS",
kernel="6.8.0-40-generic",
cpu="Intel Core i7-13700",
motherboard="FlashOps SIM-X1",
bios_version="F26b",
agent_version="0.1.0-sim",
ip_address="192.0.2.15",
status=HostStatus.ONLINE.value,
last_heartbeat_at=utcnow(),
oob_controller_id=oob.id,
toolchain={"fio": "3.36", "nvme-cli": "2.9.1"},
labels={"system_device_paths": ["/dev/nvme0n1"]},
)
session.add(host)
await session.flush()
dut = Dut(
serial="S6XNNA0T609",
model="NV-E1T92",
vendor="Lab Sample",
capacity_gb=1920,
form_factor="U.2",
current_firmware="3.2.1",
bdf="0000:03:00.0",
device_path="/dev/nvme1n1",
test_host_id=host.id,
status=DutStatus.IDLE.value,
allow_destructive=True,
health={"temperature_c": 43, "percentage_used": 12, "critical_warning": 0},
labels={"purpose": "test-only"},
)
session.add(dut)
firmware_a = FirmwareArtifact(
version="3.2.1",
label="A / 基线",
sha256="4bd1" + "0" * 56 + "08ce",
signature="simulated-valid-signature",
applicable_models=[dut.model],
file_uri="sim://firmware/FW_3.2.1.bin",
uploaded_by="seed",
)
firmware_b = FirmwareArtifact(
version="3.3.0-rc3",
label="B / 候选",
sha256="9f2c" + "0" * 56 + "41aa",
signature="simulated-valid-signature",
applicable_models=[dut.model],
file_uri="sim://firmware/FW_3.3.0-rc3.bin",
uploaded_by="seed",
)
session.add_all([firmware_a, firmware_b])
spec = {
"key": "fw-ab-regression",
"name": "固件 A/B 电源状态回归",
"version": 3,
"danger_level": "high",
"params": {"loops": {"type": "int", "default": 8, "min": 1, "max": 10000}},
"steps": [
{"key": "preflight", "type": "precheck", "adapter": "safety"},
{
"key": "flash-fw-b",
"type": "command",
"adapter": "nvme_cli",
"template": "nvme_fw_download_commit",
"danger": "high",
"idempotent": False,
"checkpoint": True,
},
{
"key": "regression-loop",
"type": "loop",
"body": [
{
"key": "workload",
"type": "command",
"adapter": "fio",
"idempotent": True,
"retry": 1,
},
{"key": "enumeration-check", "type": "device_check", "adapter": "nvme_cli"},
{"key": "integrity-check", "type": "command", "adapter": "shell"},
],
"checkpoint": True,
},
{"key": "report", "type": "report", "adapter": "builtin"},
],
}
workflow = Workflow(
key=spec["key"],
name=spec["name"],
version=spec["version"],
spec=spec,
spec_hash=_spec_hash(spec),
danger_level="high",
source_sop="内置模拟 SOP v1",
published=True,
approved_by="system-seed",
)
session.add(workflow)
await session.flush()
flashops/services/control-plane/flashops_control/services/__init__.py
+1
View File
@@ -0,0 +1 @@
"""应用服务。"""
flashops/services/control-plane/flashops_control/services/agent_status.py
+95
View File
@@ -0,0 +1,95 @@
"""Agent heartbeat aging and Host status projection."""
from __future__ import annotations
import asyncio
import datetime as dt
import logging
from contextlib import suppress
from typing import Optional
from sqlalchemy import select
from ..config import TimingPolicy, get_settings
from ..db import session_scope
from ..enums import HostStatus
from ..models import TestHost, utcnow
logger = logging.getLogger(__name__)
def projected_agent_status(
last_heartbeat_at: Optional[dt.datetime],
current_status: str,
*,
now: Optional[dt.datetime] = None,
timing: Optional[TimingPolicy] = None,
) -> str:
"""Return the status implied by heartbeat age.
A fresh explicit DEGRADED report remains degraded until the Agent sends a
healthy heartbeat. Only Agents with an ``agent_id`` use this projection;
seeded simulator Hosts retain their configured status.
"""
if last_heartbeat_at is None:
return HostStatus.OFFLINE.value
policy = timing or get_settings().timing
age_s = max(0.0, ((now or utcnow()) - last_heartbeat_at).total_seconds())
if age_s >= policy.heartbeat_lost_after_s:
return HostStatus.OFFLINE.value
if age_s >= policy.heartbeat_degraded_after_s:
return HostStatus.DEGRADED.value
if current_status == HostStatus.DEGRADED.value:
return HostStatus.DEGRADED.value
return HostStatus.ONLINE.value
def refresh_agent_host(host: TestHost, *, now: Optional[dt.datetime] = None) -> str:
"""Refresh one registered Host in the caller's transaction."""
if not host.agent_id:
return host.status
host.status = projected_agent_status(
host.last_heartbeat_at,
host.status,
now=now,
)
return host.status
async def refresh_all_agent_hosts(*, now: Optional[dt.datetime] = None) -> int:
"""Persist heartbeat-derived states for all registered Agents."""
changed = 0
async with session_scope() as session:
hosts = (
await session.scalars(
select(TestHost).where(TestHost.agent_id.is_not(None))
)
).all()
for host in hosts:
previous = host.status
refresh_agent_host(host, now=now)
if host.status != previous:
changed += 1
return changed
async def run_agent_watchdog() -> None:
"""Continuously age registered Agent heartbeats."""
interval_s = max(1.0, get_settings().timing.heartbeat_interval_s)
while True:
try:
await refresh_all_agent_hosts()
except asyncio.CancelledError:
raise
except Exception:
logger.exception("Agent heartbeat watchdog failed")
await asyncio.sleep(interval_s)
async def stop_agent_watchdog(task: "asyncio.Task[None]") -> None:
task.cancel()
with suppress(asyncio.CancelledError):
await task
flashops/services/control-plane/flashops_control/services/runs.py
+560
View File
@@ -0,0 +1,560 @@
"""Run 创建、查询与内置模拟执行器。"""
from __future__ import annotations
import asyncio
import hashlib
import json
from typing import Any, Dict, Optional, Set
from sqlalchemy import delete, select
from sqlalchemy.ext.asyncio import AsyncSession
from ..config import get_settings
from ..db import session_scope
from ..enums import (
EventKind,
EventSource,
HostStatus,
RecoveryLevel,
RecoveryOutcome,
RecoveryTrigger,
RunState,
Severity,
Verdict,
)
from ..events import append_event
from ..evidence import build_evidence_bundle
from ..engine.planner import materialize_run_steps
from ..models import (
Dut,
FirmwareArtifact,
OobController,
RecoveryAction,
ResourceLock,
Run,
RunEvent,
TestHost,
Workflow,
iso_z,
utcnow,
)
from ..safety import PreflightRequest, run_preflight
from ..schemas import CreateRunBody, PreflightBody
from .agent_status import refresh_agent_host
_tasks: Dict[str, asyncio.Task[None]] = {}
_task_refs: Set[asyncio.Task[None]] = set()
async def _first(session: AsyncSession, model: Any) -> Any:
value = await session.scalar(select(model).limit(1))
if value is None:
raise LookupError("演示数据尚未初始化")
return value
async def resolve_assets(
session: AsyncSession,
*,
dut_id: Optional[str] = None,
host_id: Optional[str] = None,
firmware_id: Optional[str] = None,
) -> tuple[Dut, TestHost, Optional[FirmwareArtifact], Optional[OobController]]:
dut = await session.get(Dut, dut_id) if dut_id else await _first(session, Dut)
host = await session.get(TestHost, host_id) if host_id else await _first(session, TestHost)
firmware = (
await session.get(FirmwareArtifact, firmware_id)
if firmware_id
else await _first(session, FirmwareArtifact)
)
if dut is None or host is None:
raise LookupError("DUT 或测试主机不存在")
oob = await session.get(OobController, host.oob_controller_id) if host.oob_controller_id else None
return dut, host, firmware, oob
async def preflight_for_body(session: AsyncSession, body: PreflightBody) -> Dict[str, object]:
dut, host, firmware, oob = await resolve_assets(
session,
dut_id=body.dut_id,
host_id=body.host_id,
firmware_id=body.firmware_id,
)
refresh_agent_host(host)
system_paths = host.labels.get("system_device_paths", []) if host.labels else []
allowed = firmware is not None and (
not firmware.applicable_models or dut.model in firmware.applicable_models
)
result = run_preflight(
PreflightRequest(
expected_serial=dut.serial,
discovered_serial=body.discovered_serial or dut.serial,
allow_destructive=dut.allow_destructive,
device_path=dut.device_path or "",
system_device_paths=system_paths,
firmware_model_allowed=allowed,
host_online=host.status == HostStatus.ONLINE.value,
oob_online=oob is not None and oob.status == HostStatus.ONLINE.value,
)
)
payload = result.as_dict()
payload.update({"dut_id": dut.id, "host_id": host.id, "firmware_id": firmware.id if firmware else None})
return payload
async def create_run(session: AsyncSession, body: CreateRunBody) -> Run:
workflow = (
await session.get(Workflow, body.workflow_id)
if body.workflow_id
else await _first(session, Workflow)
)
dut, host, _, _ = await resolve_assets(
session, dut_id=body.dut_id, host_id=body.host_id
)
firmwares = (
await session.scalars(select(FirmwareArtifact).order_by(FirmwareArtifact.created_at))
).all()
if not firmwares:
raise LookupError("固件数据尚未初始化")
firmware_a = (
await session.get(FirmwareArtifact, body.firmware_a_id)
if body.firmware_a_id
else firmwares[0]
)
firmware_b = (
await session.get(FirmwareArtifact, body.firmware_b_id)
if body.firmware_b_id
else firmwares[-1]
)
assert workflow is not None and firmware_a is not None and firmware_b is not None
env = {
"host": host.name,
"os": host.os_version,
"kernel": host.kernel,
"bios": host.bios_version,
"agent": host.agent_version,
"toolchain": host.toolchain,
"dut_serial": dut.serial,
"dut_model": dut.model,
}
env_hash = hashlib.sha256(
json.dumps(env, ensure_ascii=False, sort_keys=True).encode("utf-8")
).hexdigest()
run = Run(
name=body.name,
workflow_id=workflow.id,
workflow_key=workflow.key,
workflow_version=workflow.version,
spec_hash=workflow.spec_hash,
dut_id=dut.id,
test_host_id=host.id,
firmware_a_id=firmware_a.id,
firmware_b_id=firmware_b.id,
params={
"inject_failure": body.inject_failure,
"execution_mode": body.execution_mode,
"loops": body.loops,
"fw_a": firmware_a.id,
"fw_b": firmware_b.id,
"firmware_a_version": firmware_a.version,
"firmware_b_version": firmware_b.version,
},
env_fingerprint=env,
env_fingerprint_hash=env_hash,
loop_target=body.loops,
created_by=body.created_by,
)
session.add(run)
await session.flush()
await append_event(
session,
run,
EventKind.RUN_CREATED.value,
"任务已创建，等待独占资源",
payload={"loops": body.loops, "inject_failure": body.inject_failure},
)
await append_event(
session, run, EventKind.RUN_QUEUED.value, "任务已进入调度队列"
)
return run
def run_to_dict(run: Run) -> Dict[str, Any]:
progress = round((run.loop_done / run.loop_target) * 100, 1) if run.loop_target else 0
return {
"id": run.id,
"name": run.name,
"state": run.state,
"verdict": run.verdict,
"workflow_key": run.workflow_key,
"workflow_version": run.workflow_version,
"execution_mode": run.params.get("execution_mode", "simulated"),
"dut_id": run.dut_id,
"test_host_id": run.test_host_id,
"loop_target": run.loop_target,
"loop_done": run.loop_done,
"progress": progress,
"checkpoint": run.checkpoint,
"environment": run.env_fingerprint,
"firmware_a_version": run.params.get("firmware_a_version"),
"firmware_b_version": run.params.get("firmware_b_version"),
"recovery_count": run.recovery_count,
"unattended_completion": run.unattended_completion,
"human_touches": run.human_touches,
"created_at": iso_z(run.created_at),
"started_at": iso_z(run.started_at),
"ended_at": iso_z(run.ended_at),
"freeze_reason": run.freeze_reason,
}
async def prepare_agent_run(session: AsyncSession, run: Run) -> None:
"""Run safety gates, lock resources and materialize an Agent dry-run."""
await append_event(
session,
run,
EventKind.RUN_PREFLIGHT_STARTED.value,
"已锁定执行计划，开始 Agent 任务安全预检",
target_state=RunState.PREFLIGHT.value,
)
preflight = await preflight_for_body(
session,
PreflightBody(
dut_id=run.dut_id,
host_id=run.test_host_id,
firmware_id=run.firmware_b_id,
),
)
if not preflight["passed"]:
await append_event(
session,
run,
EventKind.RUN_REJECTED.value,
"安全预检未通过",
severity=Severity.CRITICAL.value,
payload=preflight,
target_state=RunState.REJECTED.value,
projection={"ended_at": utcnow()},
)
return
existing_lock = await session.scalar(
select(ResourceLock.id).where(
ResourceLock.resource_type.in_(["dut", "test_host"]),
ResourceLock.resource_id.in_([run.dut_id or "", run.test_host_id or ""]),
)
)
if existing_lock:
raise LookupError("DUT 或测试主机已被其他任务锁定")
workflow = await session.get(Workflow, run.workflow_id)
if workflow is None:
raise LookupError("Run 关联工作流不存在")
steps = await materialize_run_steps(session, run, workflow)
session.add_all(
[
ResourceLock(
resource_type="dut",
resource_id=run.dut_id or "",
run_id=run.id,
acquired_at=utcnow(),
),
ResourceLock(
resource_type="test_host",
resource_id=run.test_host_id or "",
run_id=run.id,
acquired_at=utcnow(),
),
]
)
await append_event(
session,
run,
EventKind.SAFETY_APPROVED.value,
"6/6 安全门禁通过",
payload=preflight,
)
await append_event(
session,
run,
EventKind.LOCK_ACQUIRED.value,
"DUT 与测试主机独占锁已获取",
)
await append_event(
session,
run,
EventKind.RUN_STARTED.value,
"独立 Host Agent dry-run 已就绪，共 {0} 个步骤".format(len(steps)),
payload={"step_count": len(steps), "execution_mode": "agent_dry_run"},
target_state=RunState.RUNNING.value,
projection={"started_at": utcnow()},
)
def event_to_dict(event: RunEvent) -> Dict[str, Any]:
return {
"id": event.id,
"seq": event.seq,
"ts": iso_z(event.ts),
"source": event.source,
"kind": event.kind,
"severity": event.severity,
"message": event.message,
"payload": event.payload,
}
async def _sleep(seconds: float) -> None:
await asyncio.sleep(max(0.01, seconds * get_settings().time_scale))
async def _load_run(session: AsyncSession, run_id: str) -> Run:
run = await session.get(Run, run_id)
if run is None:
raise LookupError("Run 不存在: {0}".format(run_id))
return run
async def _run_recovery(run_id: str, loop_index: int) -> None:
levels = [
(RecoveryLevel.AGENT_SOFT, RecoveryOutcome.FAILED, "Agent 软恢复无响应"),
(RecoveryLevel.OS_REBOOT, RecoveryOutcome.FAILED, "OS 重启通道超时"),
(RecoveryLevel.OOB_RESET, RecoveryOutcome.RECOVERED, "带外 Reset 后主机与 DUT 已恢复"),
]
for level, outcome, message in levels:
async with session_scope() as session:
run = await _load_run(session, run_id)
action = RecoveryAction(
run_id=run.id,
loop_index=loop_index,
level=level.value,
trigger=RecoveryTrigger.HEARTBEAT_LOST.value,
outcome=outcome.value,
started_at=utcnow(),
ended_at=utcnow(),
detail={"simulated": True, "message": message},
evidence_uris=["sim://oob/frame-{0}".format(loop_index)],
)
session.add(action)
await append_event(
session,
run,
EventKind.RECOVERY_STARTED.value,
"{0} 已执行".format(level.value),
severity=Severity.WARNING.value,
payload={"level": level.value, "outcome": outcome.value},
projection={"recovery_count": run.recovery_count + 1},
)
if outcome == RecoveryOutcome.RECOVERED:
await append_event(
session,
run,
EventKind.RECOVERY_SUCCEEDED.value,
message,
source=EventSource.OOB.value,
payload={"level": level.value},
target_state=RunState.RUNNING.value,
)
else:
await append_event(
session,
run,
EventKind.RECOVERY_ESCALATED.value,
message + "，升级恢复阶梯",
severity=Severity.ERROR.value,
payload={"level": level.value},
)
await _sleep(0.28)
async def simulate_run(run_id: str) -> None:
try:
async with session_scope() as session:
run = await _load_run(session, run_id)
await append_event(
session,
run,
EventKind.RUN_PREFLIGHT_STARTED.value,
"已锁定工位，开始安全预检",
target_state=RunState.PREFLIGHT.value,
)
await _sleep(0.25)
async with session_scope() as session:
run = await _load_run(session, run_id)
preflight = await preflight_for_body(
session,
PreflightBody(
dut_id=run.dut_id,
host_id=run.test_host_id,
firmware_id=run.firmware_b_id,
),
)
if not preflight["passed"]:
await append_event(
session,
run,
EventKind.RUN_REJECTED.value,
"安全预检未通过",
severity=Severity.CRITICAL.value,
payload=preflight,
target_state=RunState.REJECTED.value,
projection={"ended_at": utcnow()},
)
return
session.add_all(
[
ResourceLock(
resource_type="dut",
resource_id=run.dut_id or "",
run_id=run.id,
acquired_at=utcnow(),
),
ResourceLock(
resource_type="test_host",
resource_id=run.test_host_id or "",
run_id=run.id,
acquired_at=utcnow(),
),
]
)
await append_event(
session,
run,
EventKind.SAFETY_APPROVED.value,
"6/6 安全门禁通过",
payload=preflight,
)
await append_event(
session,
run,
EventKind.LOCK_ACQUIRED.value,
"DUT 与测试主机独占锁已获取",
)
await append_event(
session,
run,
EventKind.RUN_STARTED.value,
"模拟 Agent 已接管执行",
target_state=RunState.RUNNING.value,
projection={"started_at": utcnow()},
)
async with session_scope() as session:
run = await _load_run(session, run_id)
target = run.loop_target
inject_failure = bool(run.params.get("inject_failure", False))
failure_loop = min(3, target) if inject_failure and target >= 2 else -1
for loop_index in range(1, target + 1):
while True:
async with session_scope() as session:
run = await _load_run(session, run_id)
if run.state != RunState.PAUSED.value:
break
await _sleep(0.2)
async with session_scope() as session:
run = await _load_run(session, run_id)
if run.state in {
RunState.ABORTED.value,
RunState.FROZEN.value,
RunState.REJECTED.value,
}:
return
await append_event(
session,
run,
EventKind.LOOP_STARTED.value,
"循环 {0}/{1} 开始".format(loop_index, target),
payload={"loop_index": loop_index},
)
await _sleep(0.22)
if loop_index == failure_loop:
async with session_scope() as session:
run = await _load_run(session, run_id)
await append_event(
session,
run,
EventKind.HEARTBEAT_LOST.value,
"Agent 心跳丢失；OOB 仍在线，启动自动恢复",
source=EventSource.OOB.value,
severity=Severity.ERROR.value,
payload={"loop_index": loop_index, "oob_online": True},
target_state=RunState.RECOVERING.value,
)
await _sleep(0.25)
await _run_recovery(run_id, loop_index)
async with session_scope() as session:
run = await _load_run(session, run_id)
await append_event(
session,
run,
EventKind.CHECKPOINT_RESUMED.value,
"主机指纹一致，从循环检查点续跑",
payload={"loop_index": loop_index},
)
async with session_scope() as session:
run = await _load_run(session, run_id)
checkpoint = {
"loop_index": loop_index,
"next_step_key": "regression-loop",
"dut_fw": "B",
}
await append_event(
session,
run,
EventKind.LOOP_COMPLETED.value,
"循环 {0}/{1} 通过，检查点已写入".format(loop_index, target),
payload={"loop_index": loop_index},
projection={"loop_done": loop_index, "checkpoint": checkpoint},
)
await _sleep(0.14)
async with session_scope() as session:
run = await _load_run(session, run_id)
bundle = await build_evidence_bundle(session, run)
await append_event(
session,
run,
EventKind.EVIDENCE_BUNDLED.value,
"Evidence Bundle 已生成，完整率 100%",
payload={"bundle_id": bundle.id, "uri": bundle.uri, "completeness": 1.0},
)
await append_event(
session,
run,
EventKind.LOCK_RELEASED.value,
"DUT 与测试主机独占锁已释放",
)
await session.execute(delete(ResourceLock).where(ResourceLock.run_id == run.id))
await append_event(
session,
run,
EventKind.RUN_COMPLETED.value,
"全部循环完成，结论 PASS",
target_state=RunState.COMPLETED.value,
projection={"verdict": Verdict.PASS.value, "ended_at": utcnow()},
)
except asyncio.CancelledError:
raise
finally:
_tasks.pop(run_id, None)
def start_simulation(run_id: str) -> None:
existing = _tasks.get(run_id)
if existing and not existing.done():
return
task = asyncio.create_task(simulate_run(run_id))
_tasks[run_id] = task
_task_refs.add(task)
task.add_done_callback(_task_refs.discard)
def cancel_simulation(run_id: str) -> None:
task = _tasks.pop(run_id, None)
if task and not task.done():
task.cancel()
flashops/services/control-plane/flashops_control/services/step_leases.py
+547
View File
@@ -0,0 +1,547 @@
"""Reliable pull-mode Step lease lifecycle for Host Agents."""
from __future__ import annotations
import datetime as _dt
from typing import Any, Dict, List, Optional
from sqlalchemy import delete, func, select
from sqlalchemy.ext.asyncio import AsyncSession
from ..config import get_settings
from ..enums import (
EventKind,
EventSource,
OnFail,
RunState,
Severity,
StepState,
Verdict,
)
from ..events import append_event
from ..evidence import build_evidence_bundle
from ..models import ResourceLock, Run, RunEvent, RunStep, TestHost, iso_z, utcnow
from ..schemas import AgentStepCompleteBody, AgentStepEvent
_ACTIVE_STEP_STATES = {
StepState.PENDING.value,
StepState.DISPATCHED.value,
StepState.RUNNING.value,
}
class StepLeaseConflict(ValueError):
"""The Agent request does not match the current lease generation/state."""
def step_status(step: RunStep, *, replayed: bool = False) -> Dict[str, Any]:
return {
"step_id": step.id,
"run_id": step.run_id,
"state": step.state,
"attempt": step.attempt,
"lease_expires_at": iso_z(step.lease_expires_at),
"replayed": replayed,
}
def step_lease(step: RunStep, run: Run) -> Dict[str, Any]:
return {
"step_id": step.id,
"run_id": step.run_id,
"seq": step.seq,
"loop_index": step.loop_index,
"step_key": step.step_key,
"step_type": step.step_type,
"adapter": step.adapter,
"template": step.template,
"params": step.params,
"danger": step.danger,
"idempotent": step.idempotent,
"timeout_s": step.timeout_s,
"attempt": step.attempt,
"lease_expires_at": iso_z(step.lease_expires_at),
# This slice exercises the real process/network protocol but never executes
# a real hardware command. A future signed adapter registry will remove it.
"dry_run": run.params.get("execution_mode") == "agent_dry_run",
}
async def _locked_run(session: AsyncSession, run_id: str) -> Run:
run = await session.scalar(
select(Run).where(Run.id == run_id).with_for_update()
)
if run is None:
raise StepLeaseConflict("Run 不存在")
return run
async def _owned_step(
session: AsyncSession, host: TestHost, step_id: str
) -> tuple[RunStep, Run]:
step = await session.scalar(
select(RunStep)
.join(Run, Run.id == RunStep.run_id)
.where(RunStep.id == step_id, Run.test_host_id == host.id)
.with_for_update()
)
if step is None:
raise StepLeaseConflict("步骤不存在或不属于当前 Agent")
run = await _locked_run(session, step.run_id)
return step, run
async def _replayed(
session: AsyncSession,
*,
run_id: str,
step_id: str,
idempotency_key: str,
operation: str,
) -> bool:
event = await session.scalar(
select(RunEvent).where(
RunEvent.run_id == run_id,
RunEvent.idempotency_key == idempotency_key,
)
)
if event is None:
return False
if event.step_id != step_id or event.payload.get("_operation") != operation:
raise StepLeaseConflict("Idempotency-Key 已用于其他步骤或操作")
return True
def _assert_current(step: RunStep, agent_id: str, attempt: int) -> None:
now = utcnow()
if step.leased_by != agent_id or step.attempt != attempt:
raise StepLeaseConflict("租约代次不匹配，步骤可能已被重新派发")
if step.lease_expires_at is None or step.lease_expires_at <= now:
raise StepLeaseConflict("步骤租约已过期")
if (
step.state == StepState.RUNNING.value
and step.started_at is not None
and step.started_at + _dt.timedelta(seconds=step.timeout_s) <= now
):
raise StepLeaseConflict("步骤执行已超过 timeout_s")
async def _freeze_run(
session: AsyncSession, run: Run, step: RunStep, reason: str
) -> None:
if run.state not in {RunState.RUNNING.value, RunState.RECOVERING.value}:
return
await append_event(
session,
run,
EventKind.RUN_FROZEN.value,
reason,
severity=Severity.CRITICAL.value,
target_state=RunState.FROZEN.value,
projection={
"verdict": Verdict.INCONCLUSIVE.value,
"freeze_reason": reason,
"ended_at": utcnow(),
},
step_id=step.id,
loop_index=step.loop_index,
)
await session.execute(delete(ResourceLock).where(ResourceLock.run_id == run.id))
async def reclaim_expired_leases(session: AsyncSession, host: TestHost) -> int:
"""Requeue safe work and freeze ambiguous non-idempotent work."""
now = utcnow()
expired = (
await session.scalars(
select(RunStep)
.join(Run, Run.id == RunStep.run_id)
.where(
Run.test_host_id == host.id,
Run.state.in_([RunState.RUNNING.value, RunState.RECOVERING.value]),
RunStep.state.in_([StepState.DISPATCHED.value, StepState.RUNNING.value]),
RunStep.lease_expires_at.is_not(None),
RunStep.lease_expires_at <= now,
)
.order_by(RunStep.run_id, RunStep.seq)
.with_for_update()
)
).all()
for step in expired:
run = await _locked_run(session, step.run_id)
previous = step.state
step.leased_by = None
step.lease_expires_at = None
if previous == StepState.DISPATCHED.value:
step.state = StepState.PENDING.value
await append_event(
session,
run,
EventKind.STEP_LEASE_EXPIRED.value,
"Agent 未 ACK，步骤租约已回收",
severity=Severity.WARNING.value,
payload={"attempt": step.attempt, "previous_state": previous},
step_id=step.id,
loop_index=step.loop_index,
)
continue
can_retry = step.idempotent and step.attempt <= step.max_retry
if can_retry:
step.state = StepState.PENDING.value
await append_event(
session,
run,
EventKind.STEP_RETRYING.value,
"Agent 执行中失联，幂等步骤已安全回收等待重试",
severity=Severity.WARNING.value,
payload={"attempt": step.attempt, "max_retry": step.max_retry},
step_id=step.id,
loop_index=step.loop_index,
)
continue
step.state = StepState.TIMED_OUT.value
step.ended_at = now
await append_event(
session,
run,
EventKind.STEP_TIMED_OUT.value,
"执行租约到期，步骤结果不确定",
severity=Severity.ERROR.value,
payload={
"attempt": step.attempt,
"idempotent": step.idempotent,
"retry_exhausted": step.idempotent,
},
step_id=step.id,
loop_index=step.loop_index,
)
await _freeze_run(
session,
run,
step,
"步骤 {0} 执行结果不确定，已冻结现场禁止盲目重跑".format(step.step_key),
)
return len(expired)
async def try_lease_next_step(
session: AsyncSession, host: TestHost
) -> Optional[Dict[str, Any]]:
await reclaim_expired_leases(session, host)
runs = (
await session.scalars(
select(Run)
.where(
Run.test_host_id == host.id,
Run.state == RunState.RUNNING.value,
)
.order_by(Run.created_at, Run.id)
.with_for_update(skip_locked=True)
)
).all()
for run in runs:
steps = (
await session.scalars(
select(RunStep)
.where(RunStep.run_id == run.id, RunStep.state.in_(_ACTIVE_STEP_STATES))
.order_by(RunStep.seq)
.with_for_update(skip_locked=True)
)
).all()
if not steps:
continue
step = steps[0]
if step.state != StepState.PENDING.value:
continue
step.state = StepState.DISPATCHED.value
step.attempt += 1
step.leased_by = host.agent_id
step.lease_expires_at = utcnow() + _dt.timedelta(
seconds=get_settings().timing.lease_ttl_s
)
await append_event(
session,
run,
EventKind.STEP_DISPATCHED.value,
"步骤已租给 Host Agent，等待 ACK",
payload={"agent_id": host.agent_id, "attempt": step.attempt},
step_id=step.id,
loop_index=step.loop_index,
)
return step_lease(step, run)
return None
async def acknowledge_step(
session: AsyncSession,
host: TestHost,
step_id: str,
attempt: int,
idempotency_key: str,
) -> Dict[str, Any]:
step, run = await _owned_step(session, host, step_id)
if await _replayed(
session,
run_id=run.id,
step_id=step.id,
idempotency_key=idempotency_key,
operation="ack",
):
return step_status(step, replayed=True)
if step.state != StepState.DISPATCHED.value:
raise StepLeaseConflict("只有 DISPATCHED 步骤可以 ACK")
_assert_current(step, host.agent_id or "", attempt)
now = utcnow()
step.state = StepState.RUNNING.value
step.started_at = now
step.lease_expires_at = min(
now + _dt.timedelta(seconds=get_settings().timing.lease_ttl_s),
now + _dt.timedelta(seconds=step.timeout_s),
)
await append_event(
session,
run,
EventKind.STEP_STARTED.value,
"Host Agent 已确认接手步骤",
source=EventSource.AGENT.value,
payload={"_operation": "ack", "attempt": attempt},
idempotency_key=idempotency_key,
step_id=step.id,
loop_index=step.loop_index,
)
return step_status(step)
async def renew_step(
session: AsyncSession,
host: TestHost,
step_id: str,
attempt: int,
idempotency_key: str,
) -> Dict[str, Any]:
step, run = await _owned_step(session, host, step_id)
if await _replayed(
session,
run_id=run.id,
step_id=step.id,
idempotency_key=idempotency_key,
operation="renew",
):
return step_status(step, replayed=True)
if step.state != StepState.RUNNING.value:
raise StepLeaseConflict("只有 RUNNING 步骤可以续租")
_assert_current(step, host.agent_id or "", attempt)
assert step.started_at is not None
now = utcnow()
step.lease_expires_at = min(
now + _dt.timedelta(seconds=get_settings().timing.lease_ttl_s),
step.started_at + _dt.timedelta(seconds=step.timeout_s),
)
await append_event(
session,
run,
EventKind.STEP_LEASE_RENEWED.value,
"Host Agent 已续租执行步骤",
source=EventSource.AGENT.value,
severity=Severity.DEBUG.value,
payload={"_operation": "renew", "attempt": attempt},
idempotency_key=idempotency_key,
step_id=step.id,
loop_index=step.loop_index,
)
return step_status(step)
async def record_step_events(
session: AsyncSession,
host: TestHost,
step_id: str,
attempt: int,
events: List[AgentStepEvent],
idempotency_key: str,
) -> Dict[str, Any]:
step, run = await _owned_step(session, host, step_id)
first_key = "{0}:0".format(idempotency_key)
if await _replayed(
session,
run_id=run.id,
step_id=step.id,
idempotency_key=first_key,
operation="events",
):
return {**step_status(step, replayed=True), "accepted": len(events)}
if step.state != StepState.RUNNING.value:
raise StepLeaseConflict("只有 RUNNING 步骤可以上报事件")
_assert_current(step, host.agent_id or "", attempt)
for index, item in enumerate(events):
await append_event(
session,
run,
EventKind.STEP_OUTPUT.value,
item.message,
source=EventSource.AGENT.value,
severity=item.severity,
payload={"_operation": "events", "attempt": attempt, **item.payload},
idempotency_key="{0}:{1}".format(idempotency_key, index),
step_id=step.id,
loop_index=step.loop_index,
)
return {**step_status(step), "accepted": len(events)}
async def _finish_run(session: AsyncSession, run: Run, verdict: str) -> None:
await append_event(
session,
run,
EventKind.LOCK_RELEASED.value,
"DUT 与测试主机独占锁已释放",
)
await session.execute(delete(ResourceLock).where(ResourceLock.run_id == run.id))
await append_event(
session,
run,
EventKind.RUN_COMPLETED.value,
"独立 Host Agent 已完成全部步骤，结论 {0}".format(verdict),
target_state=RunState.COMPLETED.value,
projection={"verdict": verdict, "ended_at": utcnow()},
)
bundle = await build_evidence_bundle(session, run)
await append_event(
session,
run,
EventKind.EVIDENCE_BUNDLED.value,
"Evidence Bundle 已生成，完整率 100%",
payload={"bundle_id": bundle.id, "uri": bundle.uri, "completeness": 1.0},
)
async def complete_step(
session: AsyncSession,
host: TestHost,
step_id: str,
body: AgentStepCompleteBody,
idempotency_key: str,
) -> Dict[str, Any]:
step, run = await _owned_step(session, host, step_id)
if await _replayed(
session,
run_id=run.id,
step_id=step.id,
idempotency_key=idempotency_key,
operation="complete",
):
return step_status(step, replayed=True)
if step.state != StepState.RUNNING.value:
raise StepLeaseConflict("只有 RUNNING 步骤可以完成")
_assert_current(step, host.agent_id or "", body.attempt)
step.exit_code = body.exit_code
step.error_class = body.error_class
step.result = body.result
step.log_uri = body.log_uri
step.ended_at = utcnow()
step.leased_by = None
step.lease_expires_at = None
succeeded = body.status == StepState.SUCCEEDED.value
step.state = StepState.SUCCEEDED.value if succeeded else StepState.FAILED.value
await append_event(
session,
run,
EventKind.STEP_SUCCEEDED.value if succeeded else EventKind.STEP_FAILED.value,
"步骤执行成功" if succeeded else "步骤执行失败",
source=EventSource.AGENT.value,
severity=Severity.INFO.value if succeeded else Severity.ERROR.value,
payload={
"_operation": "complete",
"attempt": body.attempt,
"exit_code": body.exit_code,
"error_class": body.error_class,
"result": body.result,
"log_uri": body.log_uri,
},
idempotency_key=idempotency_key,
step_id=step.id,
loop_index=step.loop_index,
)
if not succeeded:
if (
step.on_fail == OnFail.RETRY.value
and step.idempotent
and step.attempt <= step.max_retry
):
step.state = StepState.PENDING.value
await append_event(
session,
run,
EventKind.STEP_RETRYING.value,
"幂等步骤失败，已进入受限重试队列",
severity=Severity.WARNING.value,
payload={"attempt": step.attempt, "max_retry": step.max_retry},
step_id=step.id,
loop_index=step.loop_index,
)
return step_status(step)
if step.on_fail in {OnFail.FREEZE.value, OnFail.RECOVER.value} or not step.idempotent:
await _freeze_run(
session,
run,
step,
"步骤 {0} 失败且不可安全自动重跑，已冻结现场".format(step.step_key),
)
return step_status(step)
if step.on_fail == OnFail.FAIL_RUN.value:
await _finish_run(session, run, Verdict.FAIL.value)
return step_status(step)
remaining = await session.scalar(
select(func.count())
.select_from(RunStep)
.where(RunStep.run_id == run.id, RunStep.state.in_(_ACTIVE_STEP_STATES))
)
if succeeded and step.checkpoint_after:
next_step = await session.scalar(
select(RunStep)
.where(RunStep.run_id == run.id, RunStep.seq > step.seq)
.order_by(RunStep.seq)
.limit(1)
)
checkpoint = {
"loop_index": step.loop_index,
"completed_step_key": step.step_key,
"next_step_key": next_step.step_key if next_step else None,
}
loop_done = max(run.loop_done, step.loop_index or run.loop_done)
await append_event(
session,
run,
EventKind.CHECKPOINT_SAVED.value,
"安全检查点已保存",
payload=checkpoint,
projection={"checkpoint": checkpoint, "loop_done": loop_done},
step_id=step.id,
loop_index=step.loop_index,
)
if not remaining:
failed = await session.scalar(
select(func.count())
.select_from(RunStep)
.where(RunStep.run_id == run.id, RunStep.state == StepState.FAILED.value)
)
await _finish_run(
session,
run,
Verdict.FAIL.value if failed else Verdict.PASS.value,
)
return step_status(step)
flashops/services/control-plane/requirements.txt
+19
View File
@@ -0,0 +1,19 @@
# 控制平面依赖。刻意保持短——每加一个依赖，客户内网私有化部署就多一分摩擦。
fastapi>=0.115,<1.0
uvicorn[standard]>=0.30
sqlalchemy>=2.0.30
greenlet>=3.0 # SQLAlchemy 异步会话的协程桥接
aiosqlite>=0.20 # 开发态数据库
pydantic>=2.9,<3.0
pyyaml>=6.0 # 工作流定义
httpx>=0.27 # 测试客户端 + Agent 侧复用
python-dotenv>=1.0
# 生产（docker-compose）才需要，本地开发可不装：
# asyncpg>=0.29
# redis>=5.0
# boto3>=1.34
# 开发/测试
pytest>=8.0
pytest-asyncio>=0.24
flashops/services/control-plane/tests/test_agent_enrollment.py
+24
View File
@@ -0,0 +1,24 @@
from fastapi import HTTPException
import pytest
from flashops_control.api.agents import _check_enrollment_token
from flashops_control.config import reset_settings
def test_production_agent_enrollment_is_fail_closed(monkeypatch):
monkeypatch.setenv("FLASHOPS_ENV", "production")
monkeypatch.delenv("FLASHOPS_AGENT_ENROLLMENT_TOKEN", raising=False)
reset_settings()
with pytest.raises(HTTPException) as missing:
_check_enrollment_token(None)
assert missing.value.status_code == 503
monkeypatch.setenv("FLASHOPS_AGENT_ENROLLMENT_TOKEN", "enroll-secret")
reset_settings()
with pytest.raises(HTTPException) as invalid:
_check_enrollment_token("wrong")
assert invalid.value.status_code == 401
_check_enrollment_token("enroll-secret")
# Leave the global cache empty so later tests read the restored environment.
reset_settings()
flashops/services/control-plane/tests/test_agent_status.py
+53
View File
@@ -0,0 +1,53 @@
import datetime as dt
from flashops_control.config import TimingPolicy
from flashops_control.enums import HostStatus
from flashops_control.services.agent_status import projected_agent_status
NOW = dt.datetime(2026, 7, 27, 12, 0, 0)
TIMING = TimingPolicy(
heartbeat_interval_s=5,
heartbeat_degraded_after_s=15,
heartbeat_lost_after_s=30,
)
def test_fresh_agent_heartbeat_is_online():
status = projected_agent_status(
NOW - dt.timedelta(seconds=14),
HostStatus.ONLINE.value,
now=NOW,
timing=TIMING,
)
assert status == HostStatus.ONLINE.value
def test_late_agent_heartbeat_is_degraded():
status = projected_agent_status(
NOW - dt.timedelta(seconds=15),
HostStatus.ONLINE.value,
now=NOW,
timing=TIMING,
)
assert status == HostStatus.DEGRADED.value
def test_lost_agent_heartbeat_is_offline():
status = projected_agent_status(
NOW - dt.timedelta(seconds=30),
HostStatus.DEGRADED.value,
now=NOW,
timing=TIMING,
)
assert status == HostStatus.OFFLINE.value
def test_fresh_unhealthy_agent_stays_degraded_until_healthy_report():
status = projected_agent_status(
NOW,
HostStatus.DEGRADED.value,
now=NOW,
timing=TIMING,
)
assert status == HostStatus.DEGRADED.value
flashops/services/control-plane/tests/test_e2e_vertical_slice.py
+413
View File
@@ -0,0 +1,413 @@
import os
import time
import uuid
from pathlib import Path
TEST_DB = Path("/tmp/flashops-e2e-{0}.db".format(os.getpid()))
TEST_OBJECTS = Path("/tmp/flashops-e2e-objects-{0}".format(os.getpid()))
os.environ["FLASHOPS_DATABASE_URL"] = "sqlite+aiosqlite:///" + str(TEST_DB)
os.environ["FLASHOPS_OBJECT_STORE_URL"] = str(TEST_OBJECTS)
os.environ["FLASHOPS_TIME_SCALE"] = "0.01"
from fastapi.testclient import TestClient
from flashops_control.config import reset_settings
from flashops_control.config import get_settings
from flashops_control.main import app
def setup_module():
reset_settings()
TEST_DB.unlink(missing_ok=True)
def teardown_module():
TEST_DB.unlink(missing_ok=True)
def test_demo_run_recovers_and_builds_evidence():
with TestClient(app) as client:
health = client.get("/api/v1/health")
assert health.status_code == 200
preflight = client.post("/api/v1/preflight", json={})
assert preflight.status_code == 200
assert preflight.json()["passed"] is True
assert len(preflight.json()["checks"]) == 6
created = client.post(
"/api/v1/runs",
json={"loops": 4, "inject_failure": True, "created_by": "pytest"},
)
assert created.status_code == 201
run_id = created.json()["id"]
run = None
for _ in range(100):
response = client.get("/api/v1/runs/" + run_id)
assert response.status_code == 200
run = response.json()
if run["state"] == "COMPLETED":
break
time.sleep(0.02)
assert run is not None
assert run["state"] == "COMPLETED"
assert run["verdict"] == "PASS"
assert run["loop_done"] == 4
assert run["recovery_count"] == 3
assert run["evidence"]["completeness"] == 1.0
assert Path(run["evidence"]["uri"], "manifest.json").exists()
events = client.get("/api/v1/runs/{0}/events".format(run_id)).json()
kinds = [event["kind"] for event in events]
assert [event["seq"] for event in events] == list(range(1, len(events) + 1))
assert "HEARTBEAT_LOST" in kinds
assert kinds.count("RECOVERY_STARTED") == 3
assert "RECOVERY_SUCCEEDED" in kinds
assert kinds[-1] == "RUN_COMPLETED"
def test_mismatched_serial_is_rejected_before_run_creation():
with TestClient(app) as client:
response = client.post(
"/api/v1/preflight", json={"discovered_serial": "SYSTEM-DISK"}
)
assert response.status_code == 200
payload = response.json()
assert payload["passed"] is False
serial = next(check for check in payload["checks"] if check["key"] == "serial")
assert serial["passed"] is False
def test_pause_resume_and_emergency_stop_are_audited():
with TestClient(app) as client:
created = client.post(
"/api/v1/runs",
json={"loops": 50, "inject_failure": False, "created_by": "pytest-actions"},
)
assert created.status_code == 201
run_id = created.json()["id"]
run = created.json()
for _ in range(50):
run = client.get("/api/v1/runs/" + run_id).json()
if run["state"] == "RUNNING":
break
time.sleep(0.01)
assert run["state"] == "RUNNING"
paused = client.post(
"/api/v1/runs/{0}/pause".format(run_id),
json={"actor": "pytest", "reason": "检查暂停"},
)
assert paused.status_code == 200
assert paused.json()["state"] == "PAUSED"
resumed = client.post(
"/api/v1/runs/{0}/resume".format(run_id),
json={"actor": "pytest", "reason": "检查继续"},
)
assert resumed.status_code == 200
assert resumed.json()["state"] == "RUNNING"
stopped = client.post(
"/api/v1/runs/{0}/emergency-stop".format(run_id),
json={"actor": "pytest", "reason": "检查急停"},
)
assert stopped.status_code == 200
assert stopped.json()["state"] == "ABORTED"
assert stopped.json()["human_touches"] == 3
events = client.get("/api/v1/runs/{0}/events".format(run_id)).json()
kinds = [event["kind"] for event in events]
assert "RUN_PAUSED" in kinds
assert "RUN_RESUMED" in kinds
assert kinds[-1] == "RUN_ABORTED"
def test_agent_registration_heartbeat_and_token_rotation():
with TestClient(app) as client:
registered = client.post(
"/api/v1/agent/register",
json={
"host_name": "WS-05",
"agent_version": "0.1.0-test",
"os_family": "linux",
"os_version": "pytest",
"kernel": "test-kernel",
"ip_address": "192.0.2.50",
"toolchain": {"fio": "fio-3.36"},
"labels": {"system_device_paths": ["/dev/nvme0n1"]},
},
)
assert registered.status_code == 201
credentials = registered.json()
assert credentials["agent_id"].startswith("agent_")
assert credentials["host_id"]
assert len(credentials["token"]) >= 32
missing_token = client.post(
"/api/v1/agent/{0}/heartbeat".format(credentials["agent_id"]),
json={"healthy": True},
)
assert missing_token.status_code == 401
heartbeat = client.post(
"/api/v1/agent/{0}/heartbeat".format(credentials["agent_id"]),
headers={"Authorization": "Bearer " + credentials["token"]},
json={
"healthy": True,
"agent_version": "0.1.1-test",
"device_probe": {"nvme": ["nvme1n1"]},
},
)
assert heartbeat.status_code == 200
assert heartbeat.json()["ok"] is True
assert heartbeat.json()["commands"] == []
rotated = client.post(
"/api/v1/agent/register",
json={
"host_id": credentials["host_id"],
"agent_version": "0.1.1-test",
},
)
assert rotated.status_code == 201
rotated_credentials = rotated.json()
assert rotated_credentials["agent_id"] == credentials["agent_id"]
assert rotated_credentials["token"] != credentials["token"]
old_token = client.post(
"/api/v1/agent/{0}/heartbeat".format(credentials["agent_id"]),
headers={"Authorization": "Bearer " + credentials["token"]},
json={"healthy": True},
)
assert old_token.status_code == 401
new_token = client.post(
"/api/v1/agent/{0}/heartbeat".format(credentials["agent_id"]),
headers={"Authorization": "Bearer " + rotated_credentials["token"]},
json={"healthy": False},
)
assert new_token.status_code == 200
hosts = client.get("/api/v1/agent/hosts")
assert hosts.status_code == 200
current = next(
item for item in hosts.json() if item["host_id"] == credentials["host_id"]
)
assert current["agent_id"] == credentials["agent_id"]
assert current["status"] == "DEGRADED"
assert "token" not in current
def _register_seed_agent(client):
response = client.post(
"/api/v1/agent/register",
json={"host_name": "WS-05", "agent_version": "0.2.0-pytest"},
)
assert response.status_code == 201
return response.json()
def _idem_headers(credentials, key=None):
return {
"Authorization": "Bearer " + credentials["token"],
"Idempotency-Key": key or str(uuid.uuid4()),
}
def _lease(client, credentials):
return client.post(
"/api/v1/agent/{0}/lease".format(credentials["agent_id"]),
headers={"Authorization": "Bearer " + credentials["token"]},
json={"wait_timeout_s": 0},
)
def _ack(client, credentials, lease, key=None):
return client.post(
"/api/v1/agent/{0}/steps/{1}/ack".format(
credentials["agent_id"], lease["step_id"]
),
headers=_idem_headers(credentials, key),
json={"attempt": lease["attempt"]},
)
def _complete(client, credentials, lease, key=None):
return client.post(
"/api/v1/agent/{0}/steps/{1}/complete".format(
credentials["agent_id"], lease["step_id"]
),
headers=_idem_headers(credentials, key),
json={
"attempt": lease["attempt"],
"status": "SUCCEEDED",
"exit_code": 0,
"result": {"simulated": True},
},
)
def test_agent_step_lease_idempotency_and_full_dry_run():
with TestClient(app) as client:
credentials = _register_seed_agent(client)
created = client.post(
"/api/v1/runs",
json={
"loops": 2,
"inject_failure": False,
"execution_mode": "agent_dry_run",
"created_by": "pytest-agent-lease",
},
)
assert created.status_code == 201
run_id = created.json()["id"]
assert created.json()["state"] == "RUNNING"
assert created.json()["execution_mode"] == "agent_dry_run"
unauthorized = client.post(
"/api/v1/agent/{0}/lease".format(credentials["agent_id"]),
json={"wait_timeout_s": 0},
)
assert unauthorized.status_code == 401
first = _lease(client, credentials)
assert first.status_code == 200
lease = first.json()
assert lease["step_key"] == "flash-fw-b"
assert lease["attempt"] == 1
assert lease["dry_run"] is True
ack_key = str(uuid.uuid4())
acked = _ack(client, credentials, lease, ack_key)
assert acked.status_code == 200
assert acked.json()["state"] == "RUNNING"
replayed_ack = _ack(client, credentials, lease, ack_key)
assert replayed_ack.status_code == 200
assert replayed_ack.json()["replayed"] is True
renew = client.post(
"/api/v1/agent/{0}/steps/{1}/renew".format(
credentials["agent_id"], lease["step_id"]
),
headers=_idem_headers(credentials),
json={"attempt": lease["attempt"]},
)
assert renew.status_code == 200
event_key = str(uuid.uuid4())
output = client.post(
"/api/v1/agent/{0}/steps/{1}/events".format(
credentials["agent_id"], lease["step_id"]
),
headers=_idem_headers(credentials, event_key),
json={
"attempt": lease["attempt"],
"events": [
{"message": "dry-run output 1", "payload": {"line": 1}},
{"message": "dry-run output 2", "severity": "DEBUG"},
],
},
)
assert output.status_code == 200
assert output.json()["accepted"] == 2
replayed_output = client.post(
"/api/v1/agent/{0}/steps/{1}/events".format(
credentials["agent_id"], lease["step_id"]
),
headers=_idem_headers(credentials, event_key),
json={
"attempt": lease["attempt"],
"events": [
{"message": "dry-run output 1", "payload": {"line": 1}},
{"message": "dry-run output 2", "severity": "DEBUG"},
],
},
)
assert replayed_output.status_code == 200
assert replayed_output.json()["replayed"] is True
complete_key = str(uuid.uuid4())
completed = _complete(client, credentials, lease, complete_key)
assert completed.status_code == 200
replayed_complete = _complete(client, credentials, lease, complete_key)
assert replayed_complete.status_code == 200
assert replayed_complete.json()["replayed"] is True
step_count = 1
while True:
leased = _lease(client, credentials)
if leased.status_code == 204:
break
assert leased.status_code == 200
item = leased.json()
assert _ack(client, credentials, item).status_code == 200
assert _complete(client, credentials, item).status_code == 200
step_count += 1
assert step_count == 7 # flash + (3 loop-body steps * 2 loops)
run = client.get("/api/v1/runs/" + run_id).json()
assert run["state"] == "COMPLETED"
assert run["verdict"] == "PASS"
assert run["loop_done"] == 2
assert run["evidence"]["completeness"] == 1.0
events = client.get("/api/v1/runs/{0}/events".format(run_id)).json()
assert [event["seq"] for event in events] == list(range(1, len(events) + 1))
assert sum(event["kind"] == "STEP_DISPATCHED" for event in events) == 7
assert sum(event["kind"] == "STEP_SUCCEEDED" for event in events) == 7
def test_expired_running_lease_retries_only_idempotent_steps():
timing = get_settings().timing
original_ttl = timing.lease_ttl_s
object.__setattr__(timing, "lease_ttl_s", 0.01)
try:
with TestClient(app) as client:
credentials = _register_seed_agent(client)
# The first workflow step is a non-idempotent firmware action. Once
# ACKed, an expired lease must freeze the run instead of rerunning it.
created = client.post(
"/api/v1/runs",
json={"loops": 1, "execution_mode": "agent_dry_run"},
)
assert created.status_code == 201
run_id = created.json()["id"]
lease = _lease(client, credentials).json()
assert lease["idempotent"] is False
assert _ack(client, credentials, lease).status_code == 200
time.sleep(0.02)
assert _lease(client, credentials).status_code == 204
frozen = client.get("/api/v1/runs/" + run_id).json()
assert frozen["state"] == "FROZEN"
assert frozen["verdict"] == "INCONCLUSIVE"
# On another run, finish the firmware step and let the idempotent fio
# workload expire. It is safely reissued once with attempt=2.
created_retry = client.post(
"/api/v1/runs",
json={"loops": 1, "execution_mode": "agent_dry_run"},
)
assert created_retry.status_code == 201
retry_run_id = created_retry.json()["id"]
firmware = _lease(client, credentials).json()
assert _ack(client, credentials, firmware).status_code == 200
assert _complete(client, credentials, firmware).status_code == 200
workload = _lease(client, credentials).json()
assert workload["step_key"] == "workload"
assert workload["idempotent"] is True
assert _ack(client, credentials, workload).status_code == 200
time.sleep(0.02)
retried = _lease(client, credentials)
assert retried.status_code == 200
assert retried.json()["step_id"] == workload["step_id"]
assert retried.json()["attempt"] == 2
stopped = client.post(
"/api/v1/runs/{0}/emergency-stop".format(retry_run_id),
json={"actor": "pytest", "reason": "清理重试场景"},
)
assert stopped.status_code == 200
finally:
object.__setattr__(timing, "lease_ttl_s", original_ttl)
flashops/services/control-plane/tests/test_recovery.py
+40
View File
@@ -0,0 +1,40 @@
from flashops_control.engine.recovery import RecoveryContext, next_recovery_level
from flashops_control.enums import RecoveryLevel, RecoveryTrigger
def test_integrity_failure_always_freezes():
level = next_recovery_level(
RecoveryContext(
trigger=RecoveryTrigger.DATA_INTEGRITY,
attempt=0,
agent_online=True,
host_network_online=True,
oob_online=True,
)
)
assert level == RecoveryLevel.FREEZE
def test_offline_oob_removes_destructive_recovery_levels():
level = next_recovery_level(
RecoveryContext(
trigger=RecoveryTrigger.HEARTBEAT_LOST,
attempt=2,
agent_online=True,
host_network_online=True,
oob_online=False,
)
)
assert level == RecoveryLevel.FREEZE
def test_recovery_escalates_in_order():
context = dict(
trigger=RecoveryTrigger.HEARTBEAT_LOST,
agent_online=True,
host_network_online=True,
oob_online=True,
)
assert next_recovery_level(RecoveryContext(attempt=0, **context)) == RecoveryLevel.AGENT_SOFT
assert next_recovery_level(RecoveryContext(attempt=1, **context)) == RecoveryLevel.OS_REBOOT
assert next_recovery_level(RecoveryContext(attempt=2, **context)) == RecoveryLevel.OOB_RESET
flashops/services/control-plane/tests/test_safety.py
+39
View File
@@ -0,0 +1,39 @@
from flashops_control.safety import PreflightRequest, run_preflight
def valid_request(**overrides):
values = {
"expected_serial": "DUT-001",
"discovered_serial": "DUT-001",
"allow_destructive": True,
"device_path": "/dev/nvme1n1",
"system_device_paths": ["/dev/nvme0n1", "/dev/nvme0n1p2"],
"firmware_model_allowed": True,
"host_online": True,
"oob_online": True,
}
values.update(overrides)
return PreflightRequest(**values)
def test_all_safety_gates_pass_for_bound_test_disk():
result = run_preflight(valid_request())
assert result.passed is True
assert len(result.checks) == 6
def test_serial_mismatch_is_rejected():
result = run_preflight(valid_request(discovered_serial="WRONG-DISK"))
assert result.passed is False
assert next(check for check in result.checks if check.key == "serial").passed is False
def test_system_partition_is_rejected():
result = run_preflight(valid_request(device_path="/dev/nvme0n1p2"))
assert result.passed is False
assert next(check for check in result.checks if check.key == "system_disk").passed is False
def test_destructive_opt_in_is_required():
result = run_preflight(valid_request(allow_destructive=False))
assert result.passed is False
flashops/services/control-plane/tests/test_state_machine.py
+22
View File
@@ -0,0 +1,22 @@
import pytest
from flashops_control.engine.states import IllegalTransition, assert_run_transition
from flashops_control.enums import RunState
def test_happy_path_transitions_are_allowed():
assert_run_transition(RunState.QUEUED.value, RunState.PREFLIGHT.value)
assert_run_transition(RunState.PREFLIGHT.value, RunState.RUNNING.value)
assert_run_transition(RunState.RUNNING.value, RunState.RECOVERING.value)
assert_run_transition(RunState.RECOVERING.value, RunState.RUNNING.value)
assert_run_transition(RunState.RUNNING.value, RunState.COMPLETED.value)
def test_terminal_state_cannot_transition():
with pytest.raises(IllegalTransition):
assert_run_transition(RunState.COMPLETED.value, RunState.RUNNING.value)
def test_rejected_cannot_be_resumed():
with pytest.raises(IllegalTransition):
assert_run_transition(RunState.REJECTED.value, RunState.QUEUED.value)
flashops/services/host-agent/README.md
+56
View File
@@ -0,0 +1,56 @@
# FlashOps Host Agent
The Host Agent is a pull-mode process that runs on a test host. The first
implemented slice supports:
- read-only host fingerprint discovery;
- enrollment into a Test Host record;
- one-time bearer token issuance;
- local credential persistence with mode `0600`;
- authenticated heartbeats and Host status updates;
- server-side heartbeat aging to `DEGRADED` after 15s and `OFFLINE` after 30s;
- optional gateway Basic Auth for deployments that enable an authenticated proxy.
Step leases, command templates, event uploads, artifacts, and completion
reporting remain the next slice.
## Local run
Start the control plane in one terminal:
```bash
make dev
```
Register and send one heartbeat in another:
```bash
make dev-agent
```
Run continuously:
```bash
make dev-agent-loop
```
By default the Agent state is stored at `~/.flashops/agent-state.json`. It
contains the raw bearer token and is created with mode `0600`. The control
plane stores only its SHA-256 digest.
## Configuration
| Variable | Purpose |
|---|---|
| `FLASHOPS_AGENT_SERVER` | Control plane base URL |
| `FLASHOPS_AGENT_HOST_ID` | Bind to an existing Test Host ID |
| `FLASHOPS_AGENT_HOST_NAME` | Bind to or create a Test Host by name |
| `FLASHOPS_AGENT_STATE_PATH` | Local credential state path |
| `FLASHOPS_AGENT_ENROLLMENT_TOKEN` | First-registration secret |
| `FLASHOPS_AGENT_BASIC_USER` | Optional authenticated-proxy user |
| `FLASHOPS_AGENT_BASIC_PASSWORD` | Optional authenticated-proxy password |
Production registration is denied when the control plane does not have
`FLASHOPS_AGENT_ENROLLMENT_TOKEN` configured. Store that value in
`/etc/flashops/flashops.env` with root-only permissions; never put it in the
systemd unit or repository.
flashops/services/host-agent/flashops_agent/__init__.py
+3
View File
@@ -0,0 +1,3 @@
"""FlashOps pull-mode Host Agent."""
__version__ = "0.1.0"
flashops/services/host-agent/flashops_agent/__main__.py
+180
View File
@@ -0,0 +1,180 @@
"""Command-line entry point for the pull-mode Host Agent."""
from __future__ import annotations
import argparse
import os
import sys
import time
import uuid
from pathlib import Path
from typing import Optional, Tuple
from .client import AgentUnauthorized, ControlPlaneClient, ControlPlaneError
from .fingerprint import heartbeat_payload, registration_payload
from .state import AgentState, StateStore
def _execute_dry_run(lease: dict) -> dict:
"""Exercise the distributed worker path without invoking any host command."""
if lease.get("dry_run") is not True:
raise RuntimeError("Agent 拒绝执行未注册的真实适配器")
return {
"simulated": True,
"adapter": lease.get("adapter"),
"template": lease.get("template"),
"step_key": lease.get("step_key"),
}
def _process_one_lease(
client: ControlPlaneClient, state: AgentState
) -> Optional[str]:
lease = client.lease(state.agent_id, state.token, wait_timeout_s=0)
if lease is None:
return None
step_id = str(lease["step_id"])
attempt = int(lease["attempt"])
client.ack(
state.agent_id,
state.token,
step_id,
attempt,
str(uuid.uuid4()),
)
client.report_events(
state.agent_id,
state.token,
step_id,
attempt,
[{"message": "Agent dry-run 正在验证步骤协议", "severity": "INFO"}],
str(uuid.uuid4()),
)
result = _execute_dry_run(lease)
client.complete(
state.agent_id,
state.token,
step_id,
attempt,
status="SUCCEEDED",
exit_code=0,
result=result,
idempotency_key=str(uuid.uuid4()),
)
return str(lease.get("step_key") or step_id)
def _parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(description="FlashOps Host Agent")
parser.add_argument(
"--server",
default=os.environ.get("FLASHOPS_AGENT_SERVER", "http://127.0.0.1:8000"),
)
parser.add_argument("--host-id", default=os.environ.get("FLASHOPS_AGENT_HOST_ID"))
parser.add_argument("--host-name", default=os.environ.get("FLASHOPS_AGENT_HOST_NAME"))
parser.add_argument(
"--state-path",
type=Path,
default=Path(
os.environ.get(
"FLASHOPS_AGENT_STATE_PATH",
str(Path.home() / ".flashops" / "agent-state.json"),
)
),
)
parser.add_argument(
"--enrollment-token",
default=os.environ.get("FLASHOPS_AGENT_ENROLLMENT_TOKEN"),
)
parser.add_argument(
"--basic-user",
default=os.environ.get("FLASHOPS_AGENT_BASIC_USER"),
)
parser.add_argument(
"--basic-password",
default=os.environ.get("FLASHOPS_AGENT_BASIC_PASSWORD"),
)
parser.add_argument("--interval", type=float, default=None)
parser.add_argument("--once", action="store_true")
return parser
def _basic_auth(user: Optional[str], password: Optional[str]) -> Optional[Tuple[str, str]]:
if bool(user) != bool(password):
raise ValueError("Basic Auth 用户名和密码必须同时提供")
return (user, password) if user and password else None
def _register(
client: ControlPlaneClient,
store: StateStore,
server: str,
host_id: Optional[str],
host_name: Optional[str],
) -> AgentState:
payload = client.register(
registration_payload(host_id=host_id, host_name=host_name)
)
state = AgentState.from_payload(server, payload)
store.save(state)
print("Agent 已注册: {0} → Host {1}".format(state.agent_id, state.host_id))
return state
def main() -> int:
args = _parser().parse_args()
server = args.server.rstrip("/")
store = StateStore(args.state_path)
try:
basic_auth = _basic_auth(args.basic_user, args.basic_password)
with ControlPlaneClient(
server,
enrollment_token=args.enrollment_token,
basic_auth=basic_auth,
) as client:
state = store.load()
if state is None or state.server != server:
state = _register(
client, store, server, args.host_id, args.host_name
)
while True:
try:
response = client.heartbeat(
state.agent_id,
state.token,
heartbeat_payload(),
)
except AgentUnauthorized:
state = _register(
client, store, server, args.host_id, args.host_name
)
response = client.heartbeat(
state.agent_id,
state.token,
heartbeat_payload(),
)
print(
"心跳已确认: {0} · commands={1}".format(
response.get("server_time"),
len(response.get("commands", [])),
)
)
completed_step = _process_one_lease(client, state)
if completed_step:
print("dry-run 步骤已完成: {0}".format(completed_step))
if args.once:
return 0
interval = args.interval or state.heartbeat_interval_s
time.sleep(max(0.5, interval))
except KeyboardInterrupt:
print("Agent 已停止")
return 0
except (ControlPlaneError, OSError, RuntimeError, ValueError) as exc:
print("Agent 错误: {0}".format(exc), file=sys.stderr)
return 1
if __name__ == "__main__":
raise SystemExit(main())
flashops/services/host-agent/flashops_agent/client.py
+247
View File
@@ -0,0 +1,247 @@
"""Small HTTP client for the control-plane Agent contract."""
from __future__ import annotations
from typing import Any, Dict, List, Optional, Tuple
import httpx
class ControlPlaneError(RuntimeError):
pass
class AgentUnauthorized(ControlPlaneError):
pass
class StepLeaseConflict(ControlPlaneError):
pass
class ControlPlaneClient:
def __init__(
self,
server: str,
*,
enrollment_token: Optional[str] = None,
basic_auth: Optional[Tuple[str, str]] = None,
timeout_s: float = 15.0,
transport: Optional[httpx.BaseTransport] = None,
) -> None:
self.server = server.rstrip("/")
self.enrollment_token = enrollment_token
self._client = httpx.Client(
base_url=self.server,
auth=httpx.BasicAuth(*basic_auth) if basic_auth else None,
timeout=timeout_s,
transport=transport,
)
def close(self) -> None:
self._client.close()
def __enter__(self) -> "ControlPlaneClient":
return self
def __exit__(self, *_: object) -> None:
self.close()
@staticmethod
def _payload(response: httpx.Response) -> Dict[str, Any]:
try:
value = response.json()
except ValueError as exc:
raise ControlPlaneError(
"控制平面返回了非 JSON 响应: HTTP {0}".format(response.status_code)
) from exc
if not isinstance(value, dict):
raise ControlPlaneError("控制平面响应不是 JSON object")
return value
def register(self, fingerprint: Dict[str, Any]) -> Dict[str, Any]:
headers = {}
if self.enrollment_token:
headers["X-FlashOps-Enrollment-Token"] = self.enrollment_token
response = self._client.post(
"/api/v1/agent/register",
json=fingerprint,
headers=headers,
)
if response.status_code == 401:
raise AgentUnauthorized("Agent 注册口令无效")
if response.status_code >= 400:
raise ControlPlaneError(
"Agent 注册失败: HTTP {0} {1}".format(
response.status_code, response.text[:200]
)
)
payload = self._payload(response)
for field in ("agent_id", "host_id", "token"):
if not payload.get(field):
raise ControlPlaneError("注册响应缺少字段: {0}".format(field))
return payload
def heartbeat(
self,
agent_id: str,
token: str,
heartbeat: Dict[str, Any],
) -> Dict[str, Any]:
response = self._client.post(
"/api/v1/agent/{0}/heartbeat".format(agent_id),
json=heartbeat,
headers={"Authorization": "Bearer " + token},
)
if response.status_code == 401:
raise AgentUnauthorized("Agent token 已失效")
if response.status_code >= 400:
raise ControlPlaneError(
"Agent 心跳失败: HTTP {0} {1}".format(
response.status_code, response.text[:200]
)
)
payload = self._payload(response)
if payload.get("ok") is not True:
raise ControlPlaneError("控制平面未确认心跳")
return payload
@staticmethod
def _agent_headers(token: str, idempotency_key: Optional[str] = None) -> Dict[str, str]:
headers = {"Authorization": "Bearer " + token}
if idempotency_key:
headers["Idempotency-Key"] = idempotency_key
return headers
def lease(
self,
agent_id: str,
token: str,
*,
wait_timeout_s: float = 0.0,
) -> Optional[Dict[str, Any]]:
response = self._client.post(
"/api/v1/agent/{0}/lease".format(agent_id),
json={"wait_timeout_s": wait_timeout_s},
headers=self._agent_headers(token),
)
if response.status_code == 204:
return None
return self._step_response(response, "领取步骤")
def _step_response(self, response: httpx.Response, action: str) -> Dict[str, Any]:
if response.status_code == 401:
raise AgentUnauthorized("Agent token 已失效")
if response.status_code == 409:
detail = response.text[:300]
try:
detail = str(response.json().get("detail", detail))
except (ValueError, AttributeError):
pass
raise StepLeaseConflict("{0}冲突: {1}".format(action, detail))
if response.status_code >= 400:
raise ControlPlaneError(
"{0}失败: HTTP {1} {2}".format(
action, response.status_code, response.text[:200]
)
)
return self._payload(response)
def _step_post(
self,
agent_id: str,
token: str,
step_id: str,
action: str,
payload: Dict[str, Any],
idempotency_key: str,
) -> Dict[str, Any]:
response = self._client.post(
"/api/v1/agent/{0}/steps/{1}/{2}".format(
agent_id, step_id, action
),
json=payload,
headers=self._agent_headers(token, idempotency_key),
)
return self._step_response(response, action)
def ack(
self,
agent_id: str,
token: str,
step_id: str,
attempt: int,
idempotency_key: str,
) -> Dict[str, Any]:
return self._step_post(
agent_id,
token,
step_id,
"ack",
{"attempt": attempt},
idempotency_key,
)
def renew(
self,
agent_id: str,
token: str,
step_id: str,
attempt: int,
idempotency_key: str,
) -> Dict[str, Any]:
return self._step_post(
agent_id,
token,
step_id,
"renew",
{"attempt": attempt},
idempotency_key,
)
def report_events(
self,
agent_id: str,
token: str,
step_id: str,
attempt: int,
events: List[Dict[str, Any]],
idempotency_key: str,
) -> Dict[str, Any]:
return self._step_post(
agent_id,
token,
step_id,
"events",
{"attempt": attempt, "events": events},
idempotency_key,
)
def complete(
self,
agent_id: str,
token: str,
step_id: str,
attempt: int,
*,
status: str,
idempotency_key: str,
exit_code: Optional[int] = None,
error_class: Optional[str] = None,
result: Optional[Dict[str, Any]] = None,
log_uri: Optional[str] = None,
) -> Dict[str, Any]:
return self._step_post(
agent_id,
token,
step_id,
"complete",
{
"attempt": attempt,
"status": status,
"exit_code": exit_code,
"error_class": error_class,
"result": result or {},
"log_uri": log_uri,
},
idempotency_key,
)
flashops/services/host-agent/flashops_agent/fingerprint.py
+97
View File
@@ -0,0 +1,97 @@
"""Read-only host discovery used during registration and heartbeat."""
from __future__ import annotations
import os
import platform
import shutil
import socket
import subprocess
from pathlib import Path
from typing import Any, Dict, List, Optional
from . import __version__
def _ip_address() -> Optional[str]:
try:
with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as sock:
sock.connect(("192.0.2.1", 9))
return str(sock.getsockname()[0])
except OSError:
return None
def _tool_version(command: str, args: List[str]) -> Optional[str]:
executable = shutil.which(command)
if not executable:
return None
try:
result = subprocess.run(
[executable] + args,
check=False,
capture_output=True,
text=True,
timeout=3,
)
except (OSError, subprocess.TimeoutExpired):
return None
output = (result.stdout or result.stderr).strip().splitlines()
return output[0][:128] if output else None
def discover_toolchain() -> Dict[str, str]:
probes = {
"fio": ("fio", ["--version"]),
"nvme-cli": ("nvme", ["version"]),
"smartctl": ("smartctl", ["--version"]),
}
versions: Dict[str, str] = {}
for name, (command, args) in probes.items():
version = _tool_version(command, args)
if version:
versions[name] = version
return versions
def system_device_paths() -> List[str]:
if platform.system().lower() != "linux":
return []
try:
for line in Path("/proc/mounts").read_text(encoding="utf-8").splitlines():
fields = line.split()
if len(fields) >= 2 and fields[1] == "/" and fields[0].startswith("/dev/"):
return [os.path.realpath(fields[0])]
except OSError:
return []
return []
def registration_payload(
*,
host_id: Optional[str] = None,
host_name: Optional[str] = None,
) -> Dict[str, Any]:
return {
"host_id": host_id,
"host_name": host_name or platform.node() or socket.gethostname(),
"agent_version": __version__,
"os_family": platform.system().lower() or "unknown",
"os_version": platform.platform(),
"kernel": platform.release(),
"cpu": platform.processor() or None,
"motherboard": None,
"bios_version": None,
"ip_address": _ip_address(),
"toolchain": discover_toolchain(),
"labels": {"system_device_paths": system_device_paths()},
}
def heartbeat_payload() -> Dict[str, Any]:
return {
"healthy": True,
"agent_version": __version__,
"ip_address": _ip_address(),
"toolchain": discover_toolchain(),
"device_probe": {"system_device_paths": system_device_paths()},
}
flashops/services/host-agent/flashops_agent/state.py
+65
View File
@@ -0,0 +1,65 @@
"""Agent credential persistence with restrictive local permissions."""
from __future__ import annotations
import json
import os
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Any, Dict, Optional
@dataclass(frozen=True)
class AgentState:
server: str
agent_id: str
host_id: str
token: str
heartbeat_interval_s: float = 5.0
@classmethod
def from_payload(
cls,
server: str,
payload: Dict[str, Any],
) -> "AgentState":
return cls(
server=server.rstrip("/"),
agent_id=str(payload["agent_id"]),
host_id=str(payload["host_id"]),
token=str(payload["token"]),
heartbeat_interval_s=float(payload.get("heartbeat_interval_s", 5.0)),
)
class StateStore:
def __init__(self, path: Path) -> None:
self.path = path
def load(self) -> Optional[AgentState]:
try:
raw = json.loads(self.path.read_text(encoding="utf-8"))
return AgentState(**raw)
except FileNotFoundError:
return None
except (OSError, TypeError, ValueError, json.JSONDecodeError) as exc:
raise RuntimeError("Agent state 文件无效: {0}".format(self.path)) from exc
def save(self, state: AgentState) -> None:
self.path.parent.mkdir(parents=True, exist_ok=True)
temporary = self.path.with_suffix(self.path.suffix + ".tmp")
descriptor = os.open(
str(temporary),
os.O_WRONLY | os.O_CREAT | os.O_TRUNC,
0o600,
)
try:
with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
json.dump(asdict(state), handle, ensure_ascii=False, indent=2)
handle.write("\n")
handle.flush()
os.fsync(handle.fileno())
os.chmod(temporary, 0o600)
os.replace(temporary, self.path)
finally:
if temporary.exists():
temporary.unlink()
flashops/services/host-agent/tests/test_client.py
+141
View File
@@ -0,0 +1,141 @@
from pathlib import Path
import stat
import uuid
import httpx
import pytest
from flashops_agent.client import AgentUnauthorized, ControlPlaneClient
from flashops_agent.state import AgentState, StateStore
def test_control_plane_client_registers_and_sends_heartbeat():
requests = []
def handler(request: httpx.Request) -> httpx.Response:
requests.append(request)
if request.url.path == "/api/v1/agent/register":
return httpx.Response(
201,
json={
"agent_id": "agent_test",
"host_id": "host_test",
"token": "raw-token",
"heartbeat_interval_s": 5,
},
)
if request.url.path == "/api/v1/agent/agent_test/heartbeat":
return httpx.Response(
200,
json={
"ok": True,
"server_time": "2026-07-27T00:00:00Z",
"commands": [],
},
)
return httpx.Response(404)
with ControlPlaneClient(
"http://control.test",
enrollment_token="enroll-test",
transport=httpx.MockTransport(handler),
) as client:
registered = client.register(
{
"host_name": "pytest-host",
"agent_version": "0.1.0",
"os_family": "linux",
}
)
heartbeat = client.heartbeat(
registered["agent_id"],
registered["token"],
{"healthy": True},
)
assert heartbeat["ok"] is True
assert requests[0].headers["x-flashops-enrollment-token"] == "enroll-test"
assert requests[1].headers["authorization"] == "Bearer raw-token"
def test_control_plane_client_rejects_unauthorized_heartbeat():
def handler(_: httpx.Request) -> httpx.Response:
return httpx.Response(401, json={"detail": "invalid"})
with ControlPlaneClient(
"http://control.test",
transport=httpx.MockTransport(handler),
) as client:
with pytest.raises(AgentUnauthorized):
client.heartbeat("agent_test", "bad-token", {"healthy": True})
def test_state_store_round_trip_uses_private_permissions(tmp_path: Path):
path = tmp_path / "agent-state.json"
store = StateStore(path)
expected = AgentState(
server="http://control.test",
agent_id="agent_test",
host_id="host_test",
token="raw-token",
heartbeat_interval_s=5.0,
)
store.save(expected)
assert store.load() == expected
assert stat.S_IMODE(path.stat().st_mode) == 0o600
def test_control_plane_client_runs_step_lease_protocol():
requests = []
def handler(request: httpx.Request) -> httpx.Response:
requests.append(request)
path = request.url.path
if path.endswith("/lease"):
return httpx.Response(
200,
json={"step_id": "step_test", "attempt": 1, "dry_run": True},
)
if path.endswith("/ack"):
return httpx.Response(200, json={"step_id": "step_test", "state": "RUNNING"})
if path.endswith("/renew"):
return httpx.Response(200, json={"step_id": "step_test", "state": "RUNNING"})
if path.endswith("/events"):
return httpx.Response(200, json={"step_id": "step_test", "accepted": 1})
if path.endswith("/complete"):
return httpx.Response(200, json={"step_id": "step_test", "state": "SUCCEEDED"})
return httpx.Response(404)
token = "raw-token"
keys = [str(uuid.uuid4()) for _ in range(4)]
with ControlPlaneClient(
"http://control.test", transport=httpx.MockTransport(handler)
) as client:
lease = client.lease("agent_test", token)
assert lease is not None
client.ack("agent_test", token, "step_test", 1, keys[0])
client.renew("agent_test", token, "step_test", 1, keys[1])
client.report_events(
"agent_test",
token,
"step_test",
1,
[{"message": "hello"}],
keys[2],
)
client.complete(
"agent_test",
token,
"step_test",
1,
status="SUCCEEDED",
exit_code=0,
result={"simulated": True},
idempotency_key=keys[3],
)
assert len(requests) == 5
assert requests[0].headers["authorization"] == "Bearer raw-token"
assert requests[1].headers["idempotency-key"] == keys[0]