Add benchmark tests for TMVA Sofie - #239
Closed
lmoneta wants to merge 35 commits into
Closed
Conversation
sitongan
approved these changes
Dec 2, 2021
lmoneta
force-pushed
the
tmva_sofie_benchmark
branch
from
March 8, 2022 09:37
11d7d82 to
5018380
Compare
lmoneta
force-pushed
the
tmva_sofie_benchmark
branch
from
April 3, 2022 10:10
03cf673 to
8f79b34
Compare
lmoneta
force-pushed
the
tmva_sofie_benchmark
branch
2 times, most recently
from
July 4, 2022 17:14
0e87df1 to
bebe103
Compare
lmoneta
force-pushed
the
tmva_sofie_benchmark
branch
from
March 6, 2024 09:47
babcd71 to
9c562ba
Compare
Disable automatic generation of test code, using template benchmark is trivial to add a new test
Add corresponding files (including new .dat wiles for weights in compiled_models)
Use also a given number of events (e.g. 64)
Add also all needed input files
for SOFIE tests Remove then header file from repository Remove also obsolate onnx file that cannot be parsed by SOFIE Add new FindONNXRuntime and a new FindLWTNN to find the required dependency package. Use minimial FIndModule file, which requires only to define an env variable to find the package
Add optimization flags for auto-vectorization : -mavx2 -fno-signaling-nans -fno-trapping-math. With autovec, COnv1D test is 3 times faster. These changes are used to produce the benchmark results presented ad ACAT.
…sage of state.thread_index
Add json file for Generator model
Modifiy noth ONNXRuntimeinference and SOFIEINference to support models with multiple inputs
- set in ONNXRuntime also the interopnumthreads to 1
lmoneta
force-pushed
the
tmva_sofie_benchmark
branch
from
June 4, 2024 14:26
9c562ba to
0f948b0
Compare
Do not use empty path when creating Session classes Improve tests by adding possibility to pass location of input files
guitargeek
pushed a commit
that referenced
this pull request
Sep 5, 2026
Benchmark the inference performance of models compiled with the TMVA
SOFIE code generator, with three complementary benchmark binaries:
* SOFIEInference: inference with the C++ code emitted by SOFIE for a
set of ONNX models (dense, convolutional, recurrent, and models from
experiments), compiled at build time with the emitFromONNX tool
* SOFIEInference_Reader: inference going through RSofieReader, which
parses the ONNX file and JITs the generated code at runtime
* RDF_SOFIE_Inference: SOFIE inference inside an RDataFrame event loop
via a SofieFunctor helper
If ONNXRuntime is found, an equivalent ONNXRuntimeInference benchmark is
generated over the same models for comparison.
This is a squash of the development history of PR #239. Compared to the
original branch, the following content is left out:
* the LWTNN comparison benchmarks, since the LWTNN project is no
longer actively developed
* the ONNX model files, which will be generated by a script instead of
being committed to the repository (see follow-up commits)
* benchmark result log files and other stray files
guitargeek
added a commit
that referenced
this pull request
Sep 5, 2026
Running the benchmarks against ROOT master showed that six of the models benchmarked in PR #239 no longer work with the current version of SOFIE: * RNN_d10_L20_h8_B1, GRU_d10_L20_h8_B1, LSTM_d10_L20_h8_B1, and DDB_B1 parse fine, but the generated code does not compile (references to non-existing Session members and assignments to const-qualified tensor pointers) * Conv2DTranspose_Relu_Sigmoid fails to parse with a dynamic-tensor error * resnet18v1 fails to parse with an "intermediate tensor already exists" error These failures should be reported as SOFIE bugs. Until they are fixed, the corresponding benchmarks are disabled so that the remaining suite builds and runs.
guitargeek
added a commit
that referenced
this pull request
Sep 5, 2026
The models are built directly with the onnx helper API, using seeded random weights: only the network architecture matters for benchmarking the inference speed, so nothing needs to be trained and no binary model files need to be committed to the repository or downloaded from elsewhere. This follows the same approach as the SOFIE unit tests in ROOT itself. The script reproduces the architectures of the models benchmarked in PR #239, including the recurrent models that are currently disabled (so that re-enabling them later requires no binary files either). The models that are not reproduced are DDB_B1, Conv2DTranspose_Relu_Sigmoid, and resnet18v1, which are all unsupported by current SOFIE anyway. The script requires Python with the onnx and numpy packages.
guitargeek
added a commit
that referenced
this pull request
Sep 5, 2026
Main changes compared to the configuration in PR #239: * SOFIE is detected by probing for its libraries (as CMake targets in ROOT-builtin builds, via find_library in standalone builds). The previous ROOT_tmva-sofie_FOUND check never passed with recent ROOT versions, where SOFIE is built unconditionally with TMVA, so the whole benchmark suite was silently disabled. * The ONNX input models are generated at build time with make_input_models.py instead of being committed to the repository. Missing Python, onnx, BLAS, or ONNXRuntime now disable the respective benchmarks with a clear status message. * The inference code is generated with one custom command per model instead of a POST_BUILD loop on a custom target, so the generation runs in parallel and is not repeated on every build. * The benchmarks are compiled with -O3 to get auto-vectorization like in an optimized user build, but no longer with -march=native and -ffast-math: those made the results machine-dependent and changed the numerical behavior of generated code that relies on infinities (the compiler warned about undefined behavior). * The ONNXRuntime benchmark registrations are generated from the model list instead of a directory glob, which would have been empty at configure time now that the models only appear at build time. * The unused Use_SOFIE_TEMPLATE section, commented-out code, and personal-machine paths are removed.
Contributor
|
Thank you @lmoneta for all the work in this PR! This is now superseded by #311 rather #311 keeps your benchmarks (the first commit is a squash of this branch, under your authorship) and adapts them to today's ROOT: the SOFIE detection is fixed for versions where the Reviving the branch also turned out to be a useful stress test of SOFIE itself: six of the models no longer work with current SOFIE (RNN, GRU, LSTM, and DDB_B1 generate code that does not compile; Conv2DTranspose_Relu_Sigmoid and resnet18v1 fail to parse). The details are in the commit messages of #311. These are probably worth fixing in SOFIE independently of the benchmarks, and the affected benchmarks can be re-enabled once they are. Closing in favor of #311. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ann new tests for TMVA SOFIE including tests using ONNXRuntime and LWTNN
Add corresponding Find.. cmake modules to find the needed package
Make also use of FindBLAS to find teh correct BLAS version. Need to use -DBLA_VENDOR to configure to the right BLAS type installed in the system.
For example -DBLA_VENDOR= -DBLA_VENDOR=Intel10_64lp_seq to use the sequential MKL library that is used in the benchmark presented at ACAT
This PR is built on top of #236 tnaks to Federico Sossai