Skip to content

fix(qoder): preserve offsets in scaffold scanning - #5366

Closed
luvs01 wants to merge 1 commit into
lidge-jun:devfrom
luvs01:fix/qoder-scaffold-marker-offsets
Closed

luvs01 wants to merge 1 commit into
lidge-jun:devfrom
luvs01:fix/qoder-scaffold-marker-offsets

Conversation

@luvs01

@luvs01 luvs01 commented Sep 20, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • The streaming Qoder scaffold guard computed marker offsets from a lowercased copy and then sliced the original buffer. Unicode case-folding can expand one code unit into several (for example 캅 becomes i plus a combining dot), so an offset taken from the lowered string can point into the middle of a scaffold block in the original and leak suppressed scaffold text to the client before the refusal lands.
  • Add indexOfMarker(text, marker, from), an ASCII case-insensitive search that runs directly on the original UTF-16 string so returned offsets stay valid for slice().
  • Replace every toLowerCase().indexOf(...) use ??opener/closer detection, nested-closer scanning, and the split-marker tail in heldSuffixLength ??and keep reminderOpensHere working on the original string so token-boundary checks stay offset-correct.

Verification

  • bun test tests/providers/qoder-scaffold-guard.test.ts tests/providers/qoder-adapter.test.ts ??29 pass, 0 fail, including new regressions that repeat 캅 before and inside a reminder block and assert no scaffold text reaches the client.
  • bun x tsc --noEmit ??clean.

Checklist

  • Scope stays focused and avoids unrelated cleanup.
  • Docs or release notes were updated when needed.
  • Security-sensitive changes were reviewed for secrets, auth, and unsafe defaults.

Review readiness checklist

This PR stays in draft until every box below is ticked. Tick all four boxes once the requirements are met:

  • All CI tests are green on my local testing.

  • I pushed my PR to the latest dev commit.

  • I resolved all correct Codex and CodeRabbit findings.

  • My PR is ready for review.

@coderabbitai

coderabbitai Bot commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Deterministic PR hygiene checks passed.

@github-actions github-actions Bot added the bug Something isn't working label Sep 20, 2026
@github-actions

github-actions Bot commented Sep 20, 2026 •

Copy link
Copy Markdown
Contributor

✅ READY

  • all PR quality gates passed; the review readiness checklist is complete.

Review readiness checklist

  • ✅ All CI tests are green on my local testing.
  • ✅ I pushed my PR to the latest dev commit.
  • ✅ I resolved all correct Codex and CodeRabbit findings.
  • ✅ My PR is ready for review.

✅ 4/4 boxes ticked.

Automatic ready-for-review conversion failed; please mark the pull request ready manually if it is still a draft.
The review-ready label marks this PR as ready; review automation runs independently.
Maintainers notified: @lidge-jun @Ingwannu

@lidge-jun

Copy link
Copy Markdown
Owner

리뷰 · 우선순위 63 / 80

이 PR은 qoder 스트림에서 벤더가 흘리는 <system-reminder> 같은 뼈대 글을 가릴 때, 글자 위치를 잘못 잡던 구멍을 막습니다. 예전에는 버퍼를 toLowerCase() 한 복사본에서 표식을 찾은 뒤, 그 번호를 원래 글에 slice로 썼습니다. İ처럼 소문자로 바꾸면 한 글자가 두 칸으로 늘어나는 글이 앞에 있으면, 찾은 번호가 원래 글의 표식 한가운데를 가리킬 수 있습니다. 그러면 막아야 할 reminder 안의 MCP 서버 목록 같은 내용이 잠깐 클라이언트로 나갈 수 있습니다. 이번 코드는 indexOfMarker로 원래 UTF-16 글에서만 ASCII 대소문자를 맞춰 찾고, opener/closer·중첩 closer·조각난 꼬리(heldSuffixLength)·reminderOpensHere까지 같은 방식으로 바꿉니다. 표식은 원래부터 영어 ASCII라서, 로케일 toLowerCase 대신 A–Z만 접는 선택이 맞습니다. 테스트는 İ를 표식 앞·안에 넣고 본문이 새지 않는지, 대문자 closer도 닫히는지 확인합니다. base는 dev이고, 같은 주제의 다른 열린 PR은 목록에 없습니다. draft이며 준비 체크리스트는 아직 비어 있습니다.

라인 src/adapters/qoder/scaffold-guard.ts indexOfMarker - 표식 쪽 글자는 소문자라고 가정하고, 버퍼 쪽 A–Z만 +32로 맞춥니다. 지금 ALL_MARKERS는 모두 소문자라 동작이 맞습니다. 나중에 표식 상수에 대문자가 들어가면 소문자 입력과 영원히 안 맞습니다. 한 줄 주석이나 표식도 같이 접는 쪽이 실수 방지에 낫습니다.

라인 CI / 본문 Verification - 이 head에서는 enforce-target·hygiene·label·resolve-pr만 보이며, 본문에 적은 bun test·tsc 결과는 깃허브 체크로는 안 보입니다. draft라 테스트 job이 안 돌았거나, 로컬 증명만 있는 상태입니다. MCP 목록 유출을 막는 수정이라, 머지 전에 그 테스트가 CI에서도 한 번 통과한 기록이 있으면 좋습니다.

라인 tests/providers/qoder-scaffold-guard.test.ts - İ 접두·블록 안·가드 경로 회귀는 잘 잡았습니다. 표식이 델타 중간에 잘리고 그 앞에 확장 유니코드가 붙는 heldSuffixLength 조합은 직접 넣은 케이스가 없습니다. 지금 가드 테스트로 큰 구멍은 막혀 있고, 조각 꼬리만 따로 더 볼지는 선택입니다.

메인테이너의 판단이 필요한 지점
draft 체크리스트를 채우고 ready로 올린 뒤에 머지할지, 로컬 테스트 증명만으로 dev에 넣을지 정해 주세요. 보안·유출 성격이라 CI에 해당 테스트가 한 번 보이는 편이 안전합니다. types.ts/config.ts 분할·프리뷰 배포는 해당 없습니다. 닫을 중복·무효 PR은 보이지 않습니다.

너의 추천
방향과 범위는 dev에 넣기 좋습니다. indexOfMarker에 “marker는 소문자 ASCII” 가정을 짧게 적거나 표식도 접게 한 뒤, draft 해제·준비 체크·해당 bun 테스트가 CI에 보이면 머지하면 됩니다. 조각+İ 꼬리 테스트는 있어도 좋고, 없어도 이번 핵심 회귀는 이미 커버됩니다.

이 댓글은 grok-bot이 작성했습니다

@lidge-jun

Copy link
Copy Markdown
Owner

Thanks @luvs01. This is carried in #5600 with your commit (f6504df) kept as authored. Review found that ASCII-only folding dropped a match the old lowercased scan made: U+212A KELVIN SIGN lowercases to an ASCII k, so <invo\u212Ae> markup would pass through. A follow-up commit (b185f24) folds each code unit the way toLowerCase() does when the result is one code unit, with whole and split-delta regressions. Closing in favor of #5600.

@lidge-jun lidge-jun closed this Sep 22, 2026
lidge-jun added a commit that referenced this pull request Sep 23, 2026
…top, reauth unknown_flow, Raycast probe, pool golden, no-renames) (#5600)

* docs: harden branch content classification against renames

* docs: date the no-renames correction and align sibling commands

* test(oauth): exercise configured generic pool validators

* test(oauth): prove the generic null-strategy clear and harden test teardown

* test(oauth): require the strategy property in the cleared response

* fix(integrations): harden Raycast defaults probe

* test(integrations): cover killed defaults probe in Raycast detection

* fix(reauth): stop polling terminal unknown flows

* fix(qoder): preserve offsets in scaffold scanning

* fix(responses): keep a cyber-policy stop when a 5xx body has malformed UTF-8

consumeComboFailure read 5xx bodies with fatalUtf8, so a single malformed
byte rejected the whole read and replaced an otherwise recognizable
cyber-policy refusal with "Provider error <status>". The combo then hopped
instead of stopping.

readBoundedResponseBody gains reportUtf8Validity: it decodes with
replacement characters and reports utf8Valid at EOF (true by construction
when fatalUtf8 is also set). consumeComboFailure keeps every existing trust
rule for malformed 5xx bodies -- no quota evidence, usage, or ordinary
classification -- and only lets the lenient decode through when it
identifies a cyber-policy refusal. The quota agreement with
shouldRetryCodexPoolAccountQuota is unchanged.

Reimplements #5307 with a narrower classification gate.

Co-authored-by: luvs01 <27862058+luvs01@users.noreply.github.com>

* docs: separate the campaign command from the corrected rerun command

The branch and PR classification summaries showed the --no-renames form as though the campaign had used it. State the command that produced the recorded verdicts and the form any rerun must use, matching the correction in 010_method.md.

Follow-up to #5461.

Co-authored-by: luvs01 <27862058+luvs01@users.noreply.github.com>

* test(oauth): restore the pool-validator home even when shutdown throws

A throwing server.stop skipped the OPENCODEX_HOME restore and temp-dir removal, leaking both into later cases. Run cleanup in an inner finally.

Follow-up to #5442.

Co-authored-by: luvs01 <27862058+luvs01@users.noreply.github.com>

* test(integrations): model a killed defaults probe with a type-safe result

The timeout case cast a result with exitCode null directly to typeof Bun.spawnSync, which strict TypeScript can reject, and its empty stdout could not tell an exit-code check from an empty read. Cast through unknown, cover null and non-zero exits, and return "1" on stdout so ignoring the exit code would visibly report Pro.

Follow-up to #5244.

Co-authored-by: luvs01 <27862058+luvs01@users.noreply.github.com>

* docs(structure): record the terminal unknown_flow GET in the reauth contract

The dashboard contract said a non-2xx GET keeps cancellation ownership and polling, and that no replacement login POST can appear before DELETE settles. A GET 404 unknown_flow now ends the flow the same way the DELETE path does, so qualify both statements as applying to retryable GET errors and state the exception in the overview.

Follow-up to #5428.

Co-authored-by: luvs01 <27862058+luvs01@users.noreply.github.com>

* fix(qoder): keep single-unit Unicode case folding in scaffold scanning

Matching markers with ASCII-only folding kept offsets correct but dropped matches the lowercased scan used to make: U+212A KELVIN SIGN lowercases to an ASCII k, so <invo\u212Ae> tool markup passed through unsuppressed, whole or split across deltas. Fold each code unit as toLowerCase() does when the result is a single code unit; characters that expand, such as U+0130, still cannot shift offsets.

Follow-up to #5366.

Co-authored-by: luvs01 <27862058+luvs01@users.noreply.github.com>

* docs(pr-assets): add the reauth unknown_flow GET before/after capture

Main-account card rendered with the dev hook and the branch hook against a mocked management API (Cancel DELETE 503, then GET 404 unknown_flow). Synthetic identity only.

Follow-up to #5428.

Co-authored-by: luvs01 <27862058+luvs01@users.noreply.github.com>

* test(responses): keep a non-replayable malformed cyber stop free of retry metadata

Carries the #5307-related part of cc466ed, which the author added after consolidating #5307 into #5553: a malformed 502 cyber-policy body that was marked non-replayable must keep the marker, carry no Retry-After or quota reset, and still stop the combo. Document the malformed-body contract in the responses structure doc, matching the narrower classification gate this branch implements.

Follow-up to #5307 (via #5553).

Co-authored-by: luvs01 <27862058+luvs01@users.noreply.github.com>

---------

Co-authored-by: luvs01 <27862058+luvs01@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working review-ready

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants