fix(filter): count the nesting depth in the grammar - #157
Merged
Merged
Conversation
MaxNestingDepth counted every '(' and ')' in the raw filter text, also inside quoted values. A quoted ')' lowered the count, so a filter such as Title == ")))..." || (((...))) went past the limit and overflowed the call stack. The process stopped, even with both parse limits on.
The grammar now counts each parenthesized group (logical group, arithmetic group, and property list) when the parser enters it. Past the limit, the parser throws QueryKitNestingDepthExceededException. A '(' or ')' inside a quoted value does not change the count. The defaults stay off, so a filter without limits gives the same result as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MaxNestingDepthcounted every(and)in the raw filter text, also inside quoted values. A quoted)lowered the count. A filter such asTitle == ")))…" || (((…Title == "Pancakes"…)))went past the limit and overflowed the call stack. .NET cannot catch a stack overflow, so the process stopped, even with both limits on (H1 in the review of main against v1.14.2).The grammar now does the count. Each parenthesized group parser increments a per-thread counter when it enters the group and decrements it when it leaves.
(or)inside a quoted value is part of the value, so it does not change the count. A fix that only clamps the old count at zero does not stop((((Title == "))))" && ((((…. The grammar count stops it.int.MaxValue). With the limits off, every input gives the same result as before.(no longer counts. For example,Title == "((((((((("withMaxNestingDepth = 3parsed before this change, but it threwQueryKitNestingDepthExceededException. Now it parses. Real groups count the same as before.IQueryKitParseLimitscomment, and the wrongFilterParsercomment ("Counting everything can only reject too much") are corrected. The README now also says thatMaxNestingDepthdoes not limit a long flat&&or||chain (L2). The grammar for that case is not changed.verify-querykitharness has a newdepth-10preset (MaxNestingDepth = 10, no length limit) to drive the overflow.Evidence
Harness (
qk run, publicApplyQueryKitFilter, memory and Postgres targets), presetdepth-10:Title == "<20 x )>" || <20 x (>Title == "Pancakes"<20 x )>[Pancakes],WHERE r."Title" IN ('))))…', 'Pancakes')QueryKitNestingDepthExceededException: … depth of 11 … maximum allowed depth of 10.(both targets)((((((((Title == "))))))))" && ((((((((Title == "Pancakes"))))))))))))))))(clamp-proof shape)QueryKitNestingDepthExceededException(both targets)Stack overflow., exit 134QueryKitNestingDepthExceededException, exit 2Stack overflow., exit 134QueryKitNestingDepthExceededException, exit 2((Title == ")))" || Title == "Pancakes" || Title == """((("""))[Pancakes][Pancakes][Pancakes]New tests in
ParseLimitsTests(run against the oldFilterParser.cs, then the new one):Full suites on net10.0: unit 410 passed (402 before, plus 8 new), integration (Postgres via Testcontainers) 283 passed.
dotnet build QueryKit -c Releasebuilds net6.0 to net10.0 with no warnings.Merge Danger
Door: two-way
The change has no public API change and no data change. A revert brings back the old pre-scan.
Blast Radius: small
Only apps that set
MaxNestingDepthsee a difference. Two kinds of input now parse that threw before: a quoted(that pushed the old count over the limit, and an input whose quoted)hid real depth. The second kind now throws. That is the fix. TheQueryKitNestingDepthExceededExceptionnow throws during parsing, not before it. The message format is the same. The out-of-scope #135 turns the limits on by default and will rebase on this change.