-
Notifications
You must be signed in to change notification settings - Fork 11
Show how to access SPHEREx data from S3 efficiently #367
Copy link
Copy link
Open
Labels
content: cloudContent related issues/PRs for notebooks with cloud hosted data relevanceContent related issues/PRs for notebooks with cloud hosted data relevancecontent: spherexContent related issues/PRs for notebooks with SPHEREx relevanceContent related issues/PRs for notebooks with SPHEREx relevanceenhancementNew notebook or featureNew notebook or feature
Description
Activity
Metadata
Metadata
Assignees
Labels
content: cloudContent related issues/PRs for notebooks with cloud hosted data relevanceContent related issues/PRs for notebooks with cloud hosted data relevancecontent: spherexContent related issues/PRs for notebooks with SPHEREx relevanceContent related issues/PRs for notebooks with SPHEREx relevanceenhancementNew notebook or featureNew notebook or feature
We've encountered at least two users who say they tried to access the S3 copy of SPHEREx data but it was much slower than accessing the on-prem copy. Both seem to be cases where they tried to apply code written for local data access directly to S3 access without accounting for how to access S3 data efficiently. We have cloud data tutorials but SPHEREx data is a bit complicated so we should provides examples for it explicitly. We may also want another general cloud tutorial about common pitfalls and what to do instead.
Here's an example I found in user's code that's fine when the FITS file (
hdul) is local but inefficient for S3 data.This has to touch the data in the bucket twice (two GET requests):
Change it to this so the data is only touched once:
I found several instances of this kind of thing in user's code. Removing the extra touches sped up the S3 code by 5-10x.