Develop an Nsight System plugin to profile network bandwidth - #999
Develop an Nsight System plugin to profile network bandwidth#999kingcrimsontianyu wants to merge 23 commits into
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
kingcrimsontianyu
added
improvement
Sample UIBelow is an example of the NIC bandwidth profile in nsys UI. 
|
Sample documentation
|
wence-
left a comment
wence-
left a comment
There was a problem hiding this comment.
flushing some small comments
| // loop breaks and the process exits cleanly with code 0 instead of an abnormal termination. Also | ||
| // catch SIGINT to have the same clean exit for Ctrl-C. | ||
| extern "C" { | ||
| static void kvikio_nic_handle_signal(int /*signum*/) { g_stop = 1; } |
There was a problem hiding this comment.
And this:
g_stop.test_and_set();
There was a problem hiding this comment.
Done. Passed the relaxed memory ordering.
| auto prev_time = clock::now(); | ||
| auto next_deadline = prev_time; | ||
|
|
||
| while (stop == 0) { |
There was a problem hiding this comment.
Done, and passed the relaxed memory ordering.
|
@wence- Thanks! These are great suggestions. I didn't know the modern C++ way of doing signal handling, so this is a good learning experience. |
madsbk
left a comment
madsbk
left a comment
There was a problem hiding this comment.
LGTM, thanks @kingcrimsontianyu
| option(KvikIO_BUILD_EXAMPLES "Configure CMake to build examples" ON) | ||
| option(KvikIO_BUILD_TESTS "Configure CMake to build tests" ON) | ||
| option(KvikIO_REMOTE_SUPPORT "Configure CMake to build with remote IO support" ON) | ||
| option(KvikIO_BUILD_NSYS_PLUGIN "Configure CMake to build the Nsight Systems NIC plugin" ON) |
There was a problem hiding this comment.
Do we want to build this by default? I guess, yes?
There was a problem hiding this comment.
Yes. I hope to have it built by default
|
small suggestion for the CMake code (via codex): [P2] Link the plugin explicitly against Threads::Threads. The executable uses std::thread, but its link declaration only includes NVTX and ${CMAKE_DL_LIBS}. The Rocky Linux CI linker command confirms no -pthread; it currently succeeds through an indirect runtime dependency, which is toolchain-dependent. Since the parent already finds Threads, use:target_link_libraries( |
|
Thanks. Done. @msarahan |
Matt711
left a comment
Matt711
left a comment
There was a problem hiding this comment.
Just noting that I've also tested with it and it's helped tremendously, so thank you!
6 participants
When developing and optimizing remote I/O, a commonly asked question is: how do I know the achieved bandwidth of my application? While nsys does offer network-related features, limitations exist:
--nic-metrics: Requires OFED/InfiniBand dependency that cannot be fulfilled on many systems, includingg7e.8xlarge.--enable=network_interface: Shows raw counter instead of bandwidth value. Does not register signals and therefore always returns non-zero exit code.This PR develops an nsys plugin to allow flexible, generic NIC bandwidth profiling without the above limitations.