-
Notifications
You must be signed in to change notification settings - Fork 1.2k
fix(openai): drop tool-schema patterns Python re cannot compile #4072
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,8 +1,7 @@ | ||
| // Codex multi-agent v2 stamps a Responses-only `encrypted: true` marker on | ||
| // collaboration tool schemas (openai/codex 5f4d06ef; issue #85). It is an | ||
| // annotation for the ChatGPT backend only, so translated provider schemas must | ||
| // drop it without removing properties or definitions literally named `encrypted`. | ||
| const ENCRYPTED_MARKER_NAME_BAG_KEYS = new Set([ | ||
| // Keys whose *children's names* are caller-chosen rather than schema keywords, and keys whose | ||
| // values are literal payloads rather than schemas. Shared by both strippers below: each one has | ||
| // to tell "the keyword `x`" apart from "a property someone named `x`". | ||
| const SCHEMA_NAME_BAG_KEYS = new Set([ | ||
| "properties", | ||
| "patternProperties", | ||
| "$defs", | ||
|
|
@@ -11,9 +10,14 @@ const ENCRYPTED_MARKER_NAME_BAG_KEYS = new Set([ | |
| "dependentSchemas", | ||
| "dependentRequired", | ||
| ]); | ||
| const ENCRYPTED_MARKER_LITERAL_VALUE_KEYS = new Set(["const", "default", "enum", "examples"]); | ||
| const SCHEMA_LITERAL_VALUE_KEYS = new Set(["const", "default", "enum", "examples"]); | ||
|
|
||
| /** | ||
| * Codex multi-agent v2 stamps a Responses-only `encrypted: true` marker on collaboration tool | ||
| * schemas (openai/codex 5f4d06ef; issue #85). It is an annotation for the ChatGPT backend only, | ||
| * so translated provider schemas must drop it without removing properties or definitions | ||
| * literally named `encrypted`. | ||
| * | ||
| * The schema is caller-supplied, so its nesting depth is attacker-influenced. Native recursion | ||
| * would turn a deep schema into a stack overflow that takes down the request path, so this walks | ||
| * an explicit stack instead: depth costs heap, which is bounded and recoverable. | ||
|
|
@@ -52,11 +56,11 @@ export function stripResponsesOnlyEncryptedMarker(node: unknown, inNameBag = fal | |
| // Inside a name bag every key is a caller-chosen name, so `encrypted` here is data. | ||
| stack.push({ node: value, inNameBag: false, assign: v => { out[key] = v; } }); | ||
| } else if (key !== "encrypted") { | ||
| if (ENCRYPTED_MARKER_LITERAL_VALUE_KEYS.has(key)) { | ||
| if (SCHEMA_LITERAL_VALUE_KEYS.has(key)) { | ||
| // Literal payloads are values, not schemas: an `encrypted` key inside them is data. | ||
| out[key] = value; | ||
| } else { | ||
| const childInNameBag = ENCRYPTED_MARKER_NAME_BAG_KEYS.has(key); | ||
| const childInNameBag = SCHEMA_NAME_BAG_KEYS.has(key); | ||
| stack.push({ node: value, inNameBag: childInNameBag, assign: v => { out[key] = v; } }); | ||
| } | ||
| } | ||
|
|
@@ -65,3 +69,96 @@ export function stripResponsesOnlyEncryptedMarker(node: unknown, inNameBag = fal | |
|
|
||
| return result; | ||
| } | ||
|
|
||
| /** | ||
| * `\p{…}` is an escape only when the backslash introducing it is itself unescaped: in `\\p{2}` | ||
| * the pair is a literal backslash and the `p{2}` that follows is an ordinary quantified `p`, | ||
| * which Python compiles fine. Scanning for the raw substring would misread that as a property | ||
| * escape and discard a working pattern. | ||
| */ | ||
| function usesUnicodePropertyEscape(pattern: string): boolean { | ||
| for (let i = 0; i < pattern.length; i++) { | ||
| if (pattern[i] !== "\\") continue; | ||
| const next = pattern[i + 1]; | ||
| if (next === "\\") { | ||
| i++; | ||
| continue; | ||
| } | ||
| if ((next === "p" || next === "P") && pattern[i + 2] === "{") return true; | ||
| } | ||
| return false; | ||
| } | ||
|
|
||
| /** | ||
| * ECMA-262 regexes may use Unicode property escapes (`\p{Cc}`, `\P{L}`); Python's `re` cannot | ||
| * compile them. OpenAI-family upstreams validate a function tool's JSON Schema `pattern` by | ||
| * compiling it with `re`, so a schema authored in JavaScript is refused whole, before routing: | ||
| * | ||
| * Invalid schema for function 'Artifact': | ||
| * '^(?!__.*__$)[^\p{Cc}\p{Cf}\p{Zl}\p{Zp}"\\./[\]]{1,200}$' is not a 'regex'. | ||
| * | ||
| * A client that ships such a pattern on a built-in tool therefore loses every request, not just | ||
| * the calls to that tool — Claude Code 2.1.265 does exactly this on its `Artifact` tool. | ||
| * Dropping only the patterns the destination cannot compile keeps the tool's shape while letting | ||
| * the request through, the same trade the Kiro adapter makes for the validation keywords Bedrock | ||
| * rejects. What is given up is bounded: an upstream that enforces `pattern` does so under strict | ||
| * Structured Outputs, and a pattern this function drops is one that upstream could not have | ||
| * compiled in the first place — it refuses the whole schema before any argument is generated. So | ||
| * the choice is a dropped constraint versus no request at all, not a silently weakened one that | ||
| * would otherwise have been enforced. | ||
| * | ||
| * Returns `node` itself when nothing was dropped, so callers can use identity to tell whether | ||
| * the schema changed. Walks an explicit stack for the same reason as the stripper above. | ||
| */ | ||
| export function stripUnicodePropertyPatterns(node: unknown, inNameBag = false): unknown { | ||
| type Assign = (value: unknown) => void; | ||
| interface Frame { node: unknown; inNameBag: boolean; assign: Assign } | ||
|
|
||
| let result: unknown; | ||
| let dropped = 0; | ||
| const stack: Frame[] = [{ node, inNameBag, assign: value => { result = value; } }]; | ||
|
|
||
| while (stack.length > 0) { | ||
| const frame = stack.pop()!; | ||
| const current = frame.node; | ||
|
|
||
| if (Array.isArray(current)) { | ||
| const out: unknown[] = new Array(current.length); | ||
| frame.assign(out); | ||
| // Array items are schemas in their own right, never a name bag. | ||
| for (let i = current.length - 1; i >= 0; i--) { | ||
| stack.push({ node: current[i], inNameBag: false, assign: value => { out[i] = value; } }); | ||
| } | ||
| continue; | ||
| } | ||
| if (!current || typeof current !== "object") { | ||
| frame.assign(current); | ||
| continue; | ||
| } | ||
|
|
||
| // A schema name may be `__proto__`; a null-prototype record keeps it as data. | ||
| const out: Record<string, unknown> = Object.create(null) as Record<string, unknown>; | ||
| frame.assign(out); | ||
|
|
||
| for (const [key, value] of Object.entries(current as Record<string, unknown>)) { | ||
| if (frame.inNameBag) { | ||
| // Inside a name bag every key is a caller-chosen name, so `pattern` here is a property | ||
| // name; its value is still a schema and is walked as one. | ||
| stack.push({ node: value, inNameBag: false, assign: v => { out[key] = v; } }); | ||
| continue; | ||
| } | ||
| if (key === "pattern" && typeof value === "string" && usesUnicodePropertyEscape(value)) { | ||
| dropped++; | ||
| continue; | ||
| } | ||
| if (SCHEMA_LITERAL_VALUE_KEYS.has(key)) { | ||
| // Literal payloads are values, not schemas: a `pattern` key inside them is data. | ||
| out[key] = value; | ||
| continue; | ||
| } | ||
| stack.push({ node: value, inNameBag: SCHEMA_NAME_BAG_KEYS.has(key), assign: v => { out[key] = v; } }); | ||
| } | ||
| } | ||
|
|
||
| return dropped === 0 ? node : result; | ||
| } | ||
|
Comment on lines
+113
to
+164
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win Extract the shared schema walk instead of copying Lines 110-161 duplicate lines 25-71 almost verbatim. The frame types, the array branch, the null-prototype rebuild, and the name-bag rule are identical. Only three things differ: the Two consequences:
Extract one walker that takes a per-key decision, then express both strippers through it. That also allows a single pass at the chat call site. ♻️ Sketch of the shared walkertype KeyVerdict = { action: "drop" } | { action: "literal" } | { action: "walk" };
function rewriteSchema(
node: unknown,
inNameBag: boolean,
decide: (key: string, value: unknown) => KeyVerdict,
): { value: unknown; dropped: number } {
// single explicit-stack walk, shared by both strippers
}
export function stripUnicodePropertyPatterns(node: unknown, inNameBag = false): unknown {
const { value, dropped } = rewriteSchema(node, inNameBag, (key, v) =>
key === "pattern" && typeof v === "string" && usesUnicodePropertyEscape(v)
? { action: "drop" }
: SCHEMA_LITERAL_VALUE_KEYS.has(key)
? { action: "literal" }
: { action: "walk" });
return dropped === 0 ? node : value;
}The guideline "Do not combine unrelated responsibilities to avoid creating another large shared module" still holds: the walker stays inside this module and keeps one responsibility. 🤖 Prompt for AI AgentsSource: Coding guidelines |
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
Add a chat-format wire test for the new strip.
Line 1334 changes what
toolsToChatFormatsends to every chat destination, including the xAI and Moonshot branches at lines 1329-1333. The current coverage proves the helper in isolation and proves the Responses passthrough body. No test asserts the serialized chat body.Add a focused test in
tests/adapters/openai/openai-chat-hardening.test.ts. Build a request with one tool whoseparameters.properties.field.patterncontains\p{Cc}, then assert the serializedtools[0].function.parametersdrops thatpatternand keeps a lookahead pattern. Assert the xAI branch too, becausenormalizeXaiToolParametersruns before the strip and can returnundefined.Based on path instructions: "A behavior change in src/ should come with a focused regression test near the existing tests for that subsystem."
🤖 Prompt for AI Agents
Source: Path instructions