Add TMVA SOFIE benchmarks - #311
Merged
Merged
Conversation
Benchmark the inference performance of models compiled with the TMVA
SOFIE code generator, with three complementary benchmark binaries:
* SOFIEInference: inference with the C++ code emitted by SOFIE for a
set of ONNX models (dense, convolutional, recurrent, and models from
experiments), compiled at build time with the emitFromONNX tool
* SOFIEInference_Reader: inference going through RSofieReader, which
parses the ONNX file and JITs the generated code at runtime
* RDF_SOFIE_Inference: SOFIE inference inside an RDataFrame event loop
via a SofieFunctor helper
If ONNXRuntime is found, an equivalent ONNXRuntimeInference benchmark is
generated over the same models for comparison.
This is a squash of the development history of PR root-project#239. Compared to the
original branch, the following content is left out:
* the LWTNN comparison benchmarks, since the LWTNN project is no
longer actively developed
* the ONNX model files, which will be generated by a script instead of
being committed to the repository (see follow-up commits)
* benchmark result log files and other stray files
Running the benchmarks against ROOT master showed that six of the models benchmarked in PR root-project#239 no longer work with the current version of SOFIE: * RNN_d10_L20_h8_B1, GRU_d10_L20_h8_B1, LSTM_d10_L20_h8_B1, and DDB_B1 parse fine, but the generated code does not compile (references to non-existing Session members and assignments to const-qualified tensor pointers) * Conv2DTranspose_Relu_Sigmoid fails to parse with a dynamic-tensor error * resnet18v1 fails to parse with an "intermediate tensor already exists" error These failures should be reported as SOFIE bugs. Until they are fixed, the corresponding benchmarks are disabled so that the remaining suite builds and runs.
The models are built directly with the onnx helper API, using seeded random weights: only the network architecture matters for benchmarking the inference speed, so nothing needs to be trained and no binary model files need to be committed to the repository or downloaded from elsewhere. This follows the same approach as the SOFIE unit tests in ROOT itself. The script reproduces the architectures of the models benchmarked in PR root-project#239, including the recurrent models that are currently disabled (so that re-enabling them later requires no binary files either). The models that are not reproduced are DDB_B1, Conv2DTranspose_Relu_Sigmoid, and resnet18v1, which are all unsupported by current SOFIE anyway. The script requires Python with the onnx and numpy packages.
Main changes compared to the configuration in PR root-project#239: * SOFIE is detected by probing for its libraries (as CMake targets in ROOT-builtin builds, via find_library in standalone builds). The previous ROOT_tmva-sofie_FOUND check never passed with recent ROOT versions, where SOFIE is built unconditionally with TMVA, so the whole benchmark suite was silently disabled. * The ONNX input models are generated at build time with make_input_models.py instead of being committed to the repository. Missing Python, onnx, BLAS, or ONNXRuntime now disable the respective benchmarks with a clear status message. * The inference code is generated with one custom command per model instead of a POST_BUILD loop on a custom target, so the generation runs in parallel and is not repeated on every build. * The benchmarks are compiled with -O3 to get auto-vectorization like in an optimized user build, but no longer with -march=native and -ffast-math: those made the results machine-dependent and changed the numerical behavior of generated code that relies on infinities (the compiler warned about undefined behavior). * The ONNXRuntime benchmark registrations are generated from the model list instead of a directory glob, which would have been empty at configure time now that the models only appear at build time. * The unused Use_SOFIE_TEMPLATE section, commented-out code, and personal-machine paths are removed.
The SOFIE benchmarks wrote the first inference output of every model to a file in the current directory on each run. This debugging aid for validating results across ROOT versions is still available with the new -o command line option of SOFIEInference, but is now disabled by default so that benchmark runs don't litter the working directory.
With the onnx Python package available, the CI also builds the SOFIE benchmarks, exercising the full pipeline of generating the ONNX input models and compiling them with emitFromONNX against the ROOT version from conda-forge. Without the package, the SOFIE benchmarks would be silently skipped in CI.
ROOT versions up to 6.40 have a dedicated tmva-sofie build option that is advertised as a ROOT feature, so accept ROOT_tmva-sofie_FOUND as evidence that SOFIE is available, in addition to probing for the library targets (ROOT-builtin builds) and the installed libraries (standalone builds against later ROOT versions, where SOFIE is built unconditionally with TMVA).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds inference benchmarks for TMVA SOFIE, superseding #239. The core of the work is @lmoneta's: the first commit is a squash of his branch, and the follow-up commits bring it into shape for current ROOT.
Three benchmark binaries cover complementary SOFIE code paths:
emitFromONNXtoolRSofieReader, which parses the ONNX file and JITs the generated code at runtimeIf ONNXRuntime is found, an equivalent ONNXRuntimeInference benchmark is generated over the same models for comparison.
Main changes compared to #239:
make_input_models.py, which builds the same architectures with the onnx helper API and seeded random weights — following the approach of the SOFIE unit tests in ROOT. The models are compiled byemitFromONNXat build time, so the suite always exercises the SOFIE version of the ROOT build it runs against.tmva-sofiefeature), ROOT-builtin builds (via the library targets), and later standalone builds where SOFIE is built unconditionally with TMVA. The originalROOT_tmva-sofie_FOUND-only check silently disabled everything with recent ROOT.-O3but no longer-march=native -ffast-math(machine-dependent results, and UB warnings from generated code using infinities), model code generation is parallel and incremental, and the per-run output dumps are behind a new-ooption instead of littering the working directory.Verified on Linux (standalone build against ROOT master): all three binaries build and run cleanly, and the benchmark timings with the generated models match the original model files within noise (e.g. higgs_model_dense 0.57 ms both ways, Generator_B1 27 ms both ways). Untested here and left to CI/review: the ROOT-builtin build, the ONNXRuntime benchmarks, and macOS. The CI environment now installs the Python
onnxpackage so the full generation pipeline is exercised there.