Sync with Microsoft ONNX Runtime - 21082026 - #1265
Open
ai-fw-intg wants to merge 20 commits into
Open
Conversation
…osoft#32053) ### Description Add a new EP allowlist that includes WebGPU (`cpu_acl_cuda_dml_webgpu_eps`), and use it for the Level-2 `GeluFusion` and `BiasGeluFusion` transformers. This allows the erf-based GELU pattern (`Div` -> `Erf` -> `Add` -> `Mul` -> `Mul`, optionally preceded by a bias Add) to fuse into the WebGPU EP's kMSDomain Gelu and BiasGelu kernels instead of running as separate elementwise dispatches. This covers models using older opsets that the opset >= 20 Level-1 `GeluFusion` transformer does not handle. ### Performance Impact In native WebGPU build, the vision submodel of zero-shot image classification achieved a 1.11x wall-clock speedup on Panther Lake and a 1.15x speedup on Wildcat Lake. | Platform | Latency reduction | Speedup | |--------------------------|-------------------|---------| | Intel Wildcat Lake (WCL) | −13.0% | 1.15× | | Intel Panther Lake (PTL) | −9.9% | 1.11× | This PR addresses the `Gelu` and `BiasGelu` items listed in [microsoft#29841](microsoft#29841). --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Upgrade cutlass from 4.4.2 to 4.7 Upgrade cudnn-frontend from 1.24 to 1.27
### Description <!-- Describe your changes. --> - Use commit timestamps for plugin EP dev versions - Update set_plugin_ep_build_variables.py to use argparse ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Python dev package versions were using only the commit date. This prevented packages from multiple commits on the same day from being published. Using a finer granularity (from day to seconds) should mitigate it. It's probably good enough if we only expect to publish from `main` and release branches. --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
To avoid OOM in cuda 13 plugin ep build: ``` ##[error]The agent worker exited with code 137, which means it ran out of memory. Make sure the agent (container) host has sufficient memory configured. ```
…t#32057) Widen each 8-lane int8 product group into the int32 accumulators before multiplying the second half of the packed K block. This avoids the signed int16 overflow that occurs when two `-128 * -128` products share a halfword lane. ### Description * Update the plain-NEON ARM64/AArch64 SymmQgemm S8 kernels to reduce each 8-lane product group into int32 before the next multiply. * Keep the ARM64 and AArch64 assembly implementations in sync. * Remove the temporary non-dotprod test guard introduced in microsoft#31606 so the existing signed-input regression coverage runs on the plain-NEON path again. ### Motivation and Context Fixes microsoft#31573. The previous `smull` + `smlal` sequence accumulated two int8 products in a signed int16 lane before widening. For the extreme case, `(-128 * -128) + (-128 * -128) = 32768`, which overflows int16. The revised sequence reduces each product group into the int32 accumulators before processing the second half of the packed K block. ### Validation The existing signed-input regression test is re-enabled for non-dotprod ARM64. The repository also contains `onnxruntime_mlas_benchmark` with `SYMMQGEMM/SignedActivation`; no representative non-dotprod Arm64 hardware was available for a trustworthy throughput comparison, so no performance numbers are claimed here.
This pull request addresses edge cases in the `TensorScatter` operator, particularly around handling large or potentially overflowing `write_indices` values. It also improves test coverage for these scenarios, ensuring robust and correct behavior in both linear and circular modes. ### Bug fixes and safety improvements * Fixed a potential overflow bug in linear mode by updating the bounds check to prevent `write_indices` values that could cause overflow when added to `sequence_length`. (`onnxruntime/core/providers/cpu/llm/tensorscatter.cc`) * Refined the circular mode logic to correctly handle very large `write_indices` values, ensuring correct wraparound behavior without overflow. (`onnxruntime/core/providers/cpu/llm/tensorscatter.cc`) ### Test coverage enhancements * Added a test for linear mode that verifies the operator fails gracefully when `write_indices` addition would overflow, ensuring the new bounds check is enforced. (`onnxruntime/test/providers/cpu/llm/tensorscatter_op_test.cc`) * Added a test for circular mode to confirm that very large `write_indices` values wrap correctly without overflow, verifying correct wraparound logic. (`onnxruntime/test/providers/cpu/llm/tensorscatter_op_test.cc`) * Included `<limits>` header to support the use of `std::numeric_limits<int64_t>::max()` in tests. (`onnxruntime/test/providers/cpu/llm/tensorscatter_op_test.cc`)
This pull request improves the robustness and correctness of handling external data references in ONNX Runtime, particularly focusing on rejecting invalid or ambiguous "location" entries and ensuring in-memory reference tags are not treated as file paths. The changes also enhance path validation logic and add new unit tests to cover these scenarios. **Validation and error handling improvements:** - Added a check in `ExternalDataInfo::Create` to reject duplicate "location" entries in the external data info, ensuring only one "location" is allowed per tensor. [[1]](diffhunk://#diff-df6326d7af04c52e54c61249ff8b1980da9bac81192ee2e63c5b2280d7b91e05R32) [[2]](diffhunk://#diff-df6326d7af04c52e54c61249ff8b1980da9bac81192ee2e63c5b2280d7b91e05L42-R47) - Updated path validation logic in `ValidateExternalDataPathFromDir` and `ValidateExternalDataPath` to explicitly reject in-memory reference tags (used for internal memory mapping) as invalid file paths, alongside empty and absolute paths. [[1]](diffhunk://#diff-d31e9fbe0f5334fcd949833e035f2b25d5ae810dcd505c545f6b372b546b1406L410-R417) [[2]](diffhunk://#diff-d31e9fbe0f5334fcd949833e035f2b25d5ae810dcd505c545f6b372b546b1406L423-R437) [[3]](diffhunk://#diff-d31e9fbe0f5334fcd949833e035f2b25d5ae810dcd505c545f6b372b546b1406L438-R446) [[4]](diffhunk://#diff-d31e9fbe0f5334fcd949833e035f2b25d5ae810dcd505c545f6b372b546b1406L448-R457) [[5]](diffhunk://#diff-d31e9fbe0f5334fcd949833e035f2b25d5ae810dcd505c545f6b372b546b1406L469-R477) [[6]](diffhunk://#diff-d31e9fbe0f5334fcd949833e035f2b25d5ae810dcd505c545f6b372b546b1406L497-R509) **Functionality and code clarity:** - Refined `HasExternalDataInMemory` to correctly detect in-memory references even if there are multiple "location" entries, improving reliability. **Testing enhancements:** - Added new unit tests to verify that duplicate "location" entries are rejected and that in-memory reference tags are correctly handled and rejected by validation logic. [[1]](diffhunk://#diff-d75ec5db9cc4642f78b6ff568aff6d10398fc211b0fb7c862d3ec88738e3eda6R146-R188) [[2]](diffhunk://#diff-d75ec5db9cc4642f78b6ff568aff6d10398fc211b0fb7c862d3ec88738e3eda6R771-R782) These changes collectively strengthen the validation of external data references, prevent ambiguous or invalid configurations, and ensure that in-memory tags are never misinterpreted as file paths.
This pull request improves the robustness and correctness of the `scale_by_axis` method in the `Initializer` class and adds new tests to ensure proper handling of edge cases, especially with empty tensors and invalid scaler inputs. **Improvements to `scale_by_axis` implementation:** * Updated the calculation of `block_size` and `num_blocks` in the `scale_by_axis` method of the `Initializer` class to use more precise and consistent dimension handling, addressing potential bugs with axis indexing. (`onnxruntime/core/optimizer/initializer.cc`) **Testing enhancements:** * Added a new test case (`ScaleByAxisEmptyTensor`) to cover scenarios where the target or scaler tensors are empty, ensuring that the method does not throw unexpectedly and returns the correct size. The test also checks for proper exception handling with invalid scaler sizes and data types. (`onnxruntime/test/optimizer/initializer_test.cc`) * Included the `span_utils.h` header in the test file to support the new test cases using spans. (`onnxruntime/test/optimizer/initializer_test.cc`)
This pull request updates the logic for determining whether a node's input is a valid initializer for fusion and adds a new unit test to ensure that initializers which can be overridden (i.e., initializers that are also graph inputs) are properly skipped during fusion. The main changes are: ### Logic Update * In `IsNodeValidForFusion`, the check for a valid initializer now uses `graph_utils::GetConstantInitializer`, ensuring that only constant initializers (not those that can be overridden by graph inputs) are considered valid for fusion. ### Testing Improvements * Added a new test, `FuseInitializersSkipsOverridableInitializer`, to verify that the transformer skips fusion for initializers that are also graph inputs and can be overridden. This test ensures the fusion transformation does not incorrectly modify such initializers. * Included the `graph_transform_test_builder.h` header to support the new test.
This pull request improves input validation for the `Normalizer` operator by adding a check to reject scalar (rank-0) inputs and introducing a corresponding unit test to ensure this behavior. The main changes are as follows: **Input validation improvements:** * Added a check in `normalizer.cc` to return an error if the input tensor has rank 0, ensuring only rank 1 or 2 inputs are accepted. **Testing enhancements:** * Added a new test case in `normalizer_test.cc` (`ScalarInputRejected`) to verify that scalar inputs are correctly rejected with an appropriate error message.
This pull request strengthens input validation for the Conv operator by adding explicit checks for the bias tensor shape and size, and adds a new unit test to verify this behavior. The main changes are as follows: **Input validation improvements:** * Added a check in `Conv<T>::Compute` (and the float specialization) to ensure that the bias tensor `B`, if provided, is a 1D tensor whose size matches the number of output channels (`M`). If not, an informative error message is returned. [[1]](diffhunk://#diff-17bd0d956bdc88650a3b0fc11efe67ee921f0e2f0d5e5ab3e73925cd4bb510d7R75-R77) [[2]](diffhunk://#diff-17bd0d956bdc88650a3b0fc11efe67ee921f0e2f0d5e5ab3e73925cd4bb510d7R261-R263) **Testing enhancements:** * Introduced a new test case `Conv2D_InvalidBiasSize` in `conv_op_test.cc` to verify that the Conv operator correctly fails when the bias tensor does not have the expected size, ensuring the new validation logic works as intended.
This pull request improves the handling and validation of empty set reductions in the ONNX Runtime CPU reduction operators. It enforces stricter checks for the axes input, prevents unnecessary memory operations, and adds comprehensive tests to verify correct behavior for various edge cases. **Validation and Error Handling Improvements:** - Enforces that the `axes` tensor input must be a 1D vector and only processes it if it is present, improving robustness and error messages for invalid input shapes. - Adds a check to ensure that memory copying only occurs when the input tensor is non-empty, avoiding redundant operations. **Logic and Flow Adjustments:** - Refactors the order of reduction logic to first handle empty axes cases before checking for empty set input, ensuring correct execution flow and output. [[1]](diffhunk://#diff-61c3aeab1bffa9fc6cac3ace83df3a778d14c8f8d02a53ab46e5b776e567cdd4L971-R986) [[2]](diffhunk://#diff-61c3aeab1bffa9fc6cac3ace83df3a778d14c8f8d02a53ab46e5b776e567cdd4L1016-R1031) **Testing Enhancements:** - Adds new tests to verify: - Reduction when the optional `axes` input is missing (should reduce all dimensions). - Validation that the `axes` tensor must be a vector, with expected error messages for invalid cases. - Correct behavior for the `noop_with_empty_axes` attribute, both when axes are omitted and when an empty axes tensor is provided.
This pull request enhances the validation logic for segment embedding inputs in the quantized `QEmbedLayerNorm` operator and significantly improves test coverage to ensure robustness against partial or inconsistent segment embedding input scenarios. **Validation Logic Improvements:** - Enforces that all segment embedding-related inputs (`segment_ids`, `segment_embedding`, `segment_embedding_scale`, `segment_embedding_zero_point`) must be provided together or all omitted, preventing invalid partial configurations. [[1]](diffhunk://#diff-ba9db626fd260b1ca5d0794268c616200df2929f2cf66cf67c383634a3e66727R190-R191) [[2]](diffhunk://#diff-ba9db626fd260b1ca5d0794268c616200df2929f2cf66cf67c383634a3e66727R203-L202) **Test Suite Enhancements:** - Refactors the test harness (`RunTest`) to allow fine-grained control over which segment embedding inputs are present, using a bitmask for flexible test scenarios. - Updates test input logic to conditionally add each segment embedding input based on the bitmask, enabling the simulation of all possible partial input combinations. [[1]](diffhunk://#diff-0396e3396c5f66dd6e1d6daac33d667ee873c93bc750b09650a9409af8df9a0aL67-R72) [[2]](diffhunk://#diff-0396e3396c5f66dd6e1d6daac33d667ee873c93bc750b09650a9409af8df9a0aL82-R87) [[3]](diffhunk://#diff-0396e3396c5f66dd6e1d6daac33d667ee873c93bc750b09650a9409af8df9a0aL114-R119) [[4]](diffhunk://#diff-0396e3396c5f66dd6e1d6daac33d667ee873c93bc750b09650a9409af8df9a0aL140-R145) - Adds a new test, `PartialSegmentInputsRejected`, which systematically verifies that any partial provision of segment embedding inputs is correctly rejected by the operator, aligning with the new validation logic. - Modifies the test execution to expect failures with a clear error message when partial segment embedding inputs are provided.
This pull request improves input validation for the `ScatterND` operator and adds a corresponding unit test to ensure that invalid input is properly rejected. Input validation: * Added a check in `scatter_nd.h` to ensure that the last dimension of the `indices` tensor is at least 1, returning an error if it is not. Testing: * Added a unit test in `scatter_nd_op_test.cc` (`ScatterND_rejects_zero_index_depth`) to verify that the operator rejects `indices` tensors with a zero-sized last dimension.
This pull request updates the ONNX Runtime Python bindings to improve memory management and API flexibility for asynchronous inference sessions. The main changes ensure that Python objects used as inputs to `run_async` are properly referenced and not prematurely garbage collected, and that the API is more Pythonic by accepting `py::object` for session and run options. It also adds a test to verify that input arrays are kept alive during asynchronous execution. **Improvements to memory management and API flexibility:** * The `AsyncResource` struct now stores references to Python feed objects, the session, and run options to ensure they are kept alive for the duration of async execution (`onnxruntime/python/onnxruntime_pybind_state.cc`). * The `run_async` binding now accepts `py::object` for the session and run options, and internally casts them as needed. This makes the API more Pythonic and flexible (`onnxruntime/python/onnxruntime_pybind_state.cc`). * When preparing feeds for `run_async`, the code now stores each input object in `feed_objects` before creating the corresponding `OrtValue`, ensuring the Python objects are not garbage collected too early (`onnxruntime/python/onnxruntime_pybind_state.cc`). **Testing and validation:** * A test using `weakref` was added to verify that input arrays passed to `run_async` remain alive until the callback is invoked, preventing premature garbage collection (`onnxruntime/test/python/onnxruntime_test_python.py`) [[1]](diffhunk://#diff-bc2d3954a8ed883e3036dc7a675c7ba13b6299e1079728e7b6069cab1f49479cR690) [[2]](diffhunk://#diff-bc2d3954a8ed883e3036dc7a675c7ba13b6299e1079728e7b6069cab1f49479cR702) [[3]](diffhunk://#diff-bc2d3954a8ed883e3036dc7a675c7ba13b6299e1079728e7b6069cab1f49479cL712-R721). * The test also exercises explicit deletion and garbage collection of inputs, session, and run options to ensure robustness (`onnxruntime/test/python/onnxruntime_test_python.py`).
…OOB read (microsoft#29461) ### Description The Split operator's split-as-attribute path validates that each split size is non-negative (in the constructor), but the split-as-input-tensor path skips that check and only validates the aggregate (sum == axis dim, count == num_outputs). A crafted negative split size like [6, -2] on an axis of size 4 passes the aggregate check ( 6 + (-2) = 4 ) and causes the kernel to copy 6 rows from a 4-row input — an out-of-bounds read. Changes: • split.h ( PrepareForCompute ): Per-element >= 0 validation. Covers CPU, WebGPU, shared-provider paths. • cuda/tensor/split.cc ( PrepareForComputeLocal ): Same fix in the CUDA copy. • split_op_test.cc : NegativeSplitSizeInputTensor test with split = [6, -2] expecting failure. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1d814472-441f-441d-bd46-931956efc1cd
ai-fw-intg
requested review from
Jaswanth51,
ankitm3k,
jatinwadhwa921 and
vthaniel
August 20, 2026 20:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
10 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated daily backmerge from ORT main to ovep-develop. No conflicts detected. Do NOT squash or rebase - use merge commit only.