Skip to content

docs(research): add harness strength protocol - #306

Open
seekskyworld wants to merge 1 commit into
openpi-dev:mainfrom
seekskyworld:docs/issue-45-harness-strength-protocol
Open

docs(research): add harness strength protocol#306
seekskyworld wants to merge 1 commit into
openpi-dev:mainfrom
seekskyworld:docs/issue-45-harness-strength-protocol

Conversation

@seekskyworld

Copy link
Copy Markdown

Problem

Issue #45 has a detailed cross-harness investigation, but its reusable protocol and evidence boundaries are not represented in the repository research index. This makes it easy to confuse existing observations with a completed benchmark or a runtime decision.

Value

A versioned research record gives future benchmark work a stable, reviewable contract for model/harness identity, paired execution, usage accounting, failure classification, and conservative interpretation. It keeps OpenPI's Pi-native default intact until evidence supports a change.

Approach

  • Add docs/research/OPENPI_HARNESS_STRENGTH_PROTOCOL_2026-08-30.md.
  • Separate verified facts, interpretation, unknown provider behavior, and experiment gates.
  • Define four comparison arms, isolation requirements, frozen identities, metrics, and result thresholds.
  • Link the record from the research index without publishing new benchmark scores or changing runtime behavior.

Validation

  • git diff --check — passed.
  • npx --yes bun@1.3.14 run lint — passed.
  • npx --yes bun@1.3.14 run typecheck — passed.
  • Markdown was reviewed against docs/README.md and docs/research/README.md; Biome intentionally ignores Markdown in this repository.

Impact

  • User-visible behavior: None.
  • Model-visible context/tools: None.
  • Runtime/lifecycle: None.
  • Persisted config/data: None.
  • Compatibility/risk: Documentation-only; no benchmark result or product Decision is claimed.

Related to #45

Signed-off-by: seekskyworld <djh1813553759@gmail.com>
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Aug 30, 2026

@tt-a1i tt-a1i left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review:Changes Requested

固定版本 e5c54204c9ce1a7f8afe7b343fec5a1b7b404eb9

问题 / 价值 / 方法

将模型能力与 Harness 强度的比较整理为研究协议,明确身份、隔离、指标和解释门槛,减少后续实验口径漂移。仅改两份 Markdown;没有新增 runtime、工具或配置,保守解释方向合理。

Standards

1 项 P2:2026-08-30 新建文档误放入早于 Decision 0001 的 Legacy records,见索引行内意见。

Spec

1 项独立 P2:新增 profile 对照替换了来源中的实验安排,未注明为新建议,见正文行内意见。Spec 也发现了上述 Legacy 归类问题。

非阻塞:明确四臂反转/配对的含义,并补 Issue 回链和适用来源信息。

不要求在这份研究协议中补做正式 Benchmark,也不要求上传原始日志或改变默认 runtime。

验证

该 exact head 的隔离工作树中:frozen-lockfile 安装后 bun run check 通过;git diff --check 和4个本地 Markdown 链接检查通过。两路规范/Spec 审查已完成。没有重跑全量 tests 或模型实验,仓库检查不能证明研究结论。现有 CI 绿色,分支落后 main。

本次仅提交 review,未改文件、未合并。

Comment thread docs/research/README.md

- [`CLAUDE_CODE_WORKFLOW_FANOUT_POLICY_2026-08-23.md`](CLAUDE_CODE_WORKFLOW_FANOUT_POLICY_2026-08-23.md) — official-source research on dynamic fan-out and bounded execution.
- [`CLAUDE_CODE_WORKFLOW_RUNTIME_CONTRACT_2026-08-23.md`](CLAUDE_CODE_WORKFLOW_RUNTIME_CONTRACT_2026-08-23.md) — version-scoped Workflow contract interview and evidence boundary.
- [`OPENPI_HARNESS_STRENGTH_PROTOCOL_2026-08-30.md`](OPENPI_HARNESS_STRENGTH_PROTOCOL_2026-08-30.md) — Issue #45 的模型能力 × Harness 强度配对研究协议;不包含新的 Benchmark 结果。

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [P2] 新建记录不应列为 Legacy

本列表声明文档早于 Decision 0001、尚未迁移到新 metadata 契约。但新增记录创建日期为 2026-08-30,Decision 0001 已于 2026-08-29 生效。这个位置会错误表达其历史豁免身份。

请移到当前研究小节,按现有 docs/README.md 约定补适用来源边界、相关 PR 和替代关系;无需新增格式框架或批量迁移旧文档。


1. Bare Pi:Pi 原生最小编码面。
2. OpenPI Explicit:普通 turn 保持 Pi-native,明确请求时才加载能力。
3. OpenPI capability profile:只加载实验指定的一个能力组。

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ [P2] 保留来源协议的 Maka 对照臂,或明确记录方案变更

Issue #45 §7 明确列出 Bare Pi、OpenPI Explicit、OMP、Maka 四个 Harness。这里将 Maka 换成 OpenPI capability profile,意味着完全按这份协议执行也无法完成原定的四 Harness 比较,而记录没有说明这一调整。

请恢复原四臂;单能力 profile 可另标为可选消融。若确实要改变主实验,明确给出新建议和理由,不要把它表现为已核验的原协议。四臂顺序也请沿用来源的 Latin-square/反转安排,或说明 AB/BA 实际配对对象,无需新增调度机制。

@tt-a1i

tt-a1i commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Code Review Summary

Changes Requested:2 项 P2。 完整 review,审查版本 e5c5420

  • 新建记录误归 Legacy,需移至当前研究区。
  • profile 对照组合替换了来源中的实验安排,需保留原方案或明确标为新建议。
  • 研究方向合理,只需文档修正;不新增 runtime、框架或模型实验。

隔离 checkout 的 bun run check、diff 与本地链接检查通过;未重跑全量测试或 Benchmark。只提交审查,未改文件、未合并。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants