Skip to content

When running at scale, sleep in code can have dramatic affect. #2

Description

@mike-gangl

https://github.com/SWOT-Confluence/ssc_input/blob/e9930c51e78673d2b09bd569674d6ea30e8c5d78/ssc_input.py#L133C9-L137C39

These sleep commands, which will almost certainly happen as we are hitting CMR with many queries, cause constant delays (up to two minutes). For reference, we have ~200k reaches to process, so at 1% of those hitting a CMR Error, we might be looking at (assuming average) a 2000 minutes of delay in SSC_input processing.

We can respect the 'retry-after' header in the CMR Queries:
https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html#request-moderation

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions