Skip to content

[BUG] Downloading and decompressing cuvs_conda_cpp_libcuvs_x86_64_cu12 occasionally takes a very long time #2500

Description

@imotov

Describe the bug
I keep noticing cuvs-java and cuvs-lucene builds that occasionally take a very long time to complete. It looks like the majority of the time is spent on these two lines (depending on the type of the build):

CPP_CHANNEL=$(rapids-download-from-github "$(rapids-artifact-name conda_cpp libcuvs cuvs --cuda "$RAPIDS_CUDA_VERSION")")

CPP_CHANNEL=$(rapids-download-from-github "$(rapids-artifact-name conda_cpp libcuvs cuvs --cuda "$RAPIDS_CUDA_VERSION")")

Most of the time these lines take just a couple of minutes. For example, in https://github.com/NVIDIA/cuvs/actions/runs/32545903651/job/96968831522?pr=2499#step:13:72 this line is executed pretty fast:

RAPIDS logger » [08/22/26 02:58:18]
┌────────────────────────────────────────────────┐
|    Downloading artifacts from previous jobs    |
└────────────────────────────────────────────────┘

[rapids-download-from-github] Detected CI=true. Skipping running 'gh auth status'.
[rapids-github-run-id] Detected CI=true. Skipping running 'gh auth status'.
[rapids-github-run-id] Assuming artifacts should be fetched from this same run of 'pr.yaml'
Downloading and decompressing cuvs_conda_cpp_libcuvs_x86_64_cu13 from Run ID 32545903651 into /tmp/tmp.m7yJdKDcm7

RAPIDS logger » [08/22/26 03:00:40]

However, in other cases such as https://github.com/NVIDIA/cuvs/actions/runs/32545903651/job/96968831551?pr=2499#step:13:72 it can take almost 30 minutes

RAPIDS logger » [08/22/26 02:59:48]
┌────────────────────────────────────────────────┐
|    Downloading artifacts from previous jobs    |
└────────────────────────────────────────────────┘

[rapids-download-from-github] Detected CI=true. Skipping running 'gh auth status'.
[rapids-github-run-id] Detected CI=true. Skipping running 'gh auth status'.
[rapids-github-run-id] Assuming artifacts should be fetched from this same run of 'pr.yaml'
Downloading and decompressing cuvs_conda_cpp_libcuvs_x86_64_cu12 from Run ID 32545903651 into /tmp/tmp.nC3Cq1kuVk

RAPIDS logger » [08/22/26 03:28:29]

And occasionally it can take almost hour and a half https://github.com/NVIDIA/cuvs/actions/runs/32545903651/job/96973240844?pr=2499

RAPIDS logger » [08/22/26 03:36:51]
┌────────────────────────────────────────────────┐
|    Downloading artifacts from previous jobs    |
└────────────────────────────────────────────────┘

[rapids-download-from-github] Detected CI=true. Skipping running 'gh auth status'.
[rapids-github-run-id] Detected CI=true. Skipping running 'gh auth status'.
[rapids-github-run-id] Assuming artifacts should be fetched from this same run of 'pr.yaml'
Downloading and decompressing cuvs_conda_cpp_libcuvs_x86_64_cu13 from Run ID 32545903651 into /tmp/tmp.0HAanNDBql
[rapids-download-from-github] Detected CI=true. Skipping running 'gh auth status'.
[rapids-github-run-id] Detected CI=true. Skipping running 'gh auth status'.
[rapids-github-run-id] Assuming artifacts should be fetched from this same run of 'pr.yaml'
Downloading and decompressing cuvs-java-cuda13.3.0 from Run ID 32545903651 into /tmp/tmp.bgbx0u4QWB

RAPIDS logger » [08/22/26 05:02:21]

I actually watched this one and it was stuck on cuvs_conda_cpp_libcuvs_x86_64_cu13 for most of that time.

Steps/Code to reproduce bug
Start a build in CI that builds cuvs-java and cuvs-lucene.

Expected behavior
Download is performed in reasonable and predictable time.

Environment details (please complete the following information):
CI

Additional context
N/A

Metadata

Metadata

Assignees

No one assigned

    Labels

    BuildbugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions