Allow declaring override remotes on a RubyGems source

[eileencodes]

A source may declare one or more overrides: — secondary RubyGems-compatible repositories that supply alternate builds (for example, prebuilt binaries) of gems that already exist in the primary source.

source "https://rubygems.org",
       overrides: ["https://build-farm.example.com"]

Override remotes are consulted only for gems that are also present in the primary source; they cannot introduce new gems. When an override publishes a spec whose name and version match one in the primary source, that spec is preferred for installation if it is compatible with the local platform. If no matching build is available, or fetching the override fails (authentication, SSL, or network errors), Bundler falls back to the primary source.

Override remotes are recorded in Gemfile.lock under override: lines:

GEM
  remote: https://rubygems.org/
  override: https://build-farm.example.com/
  specs:

Questions

I don't know if overrides is the correct name. We considered mirror but it's not a true mirror. and binary _sources but that feels too specific. Open to ideas and changes. I decided not to spend more time on the name since we will end up discussing and debating on the PR anyway.

What was the end-user or developer problem that led to this PR?

If we were going to build a build farm, we'd need a way to define those sources. We could use the source block format but that's not a super user-friendly API since users would need to know which gems are precompiled. We considered reintroducing top-level mulit-source but we've need to handle the security implications. We came up with this API that has a source that can't introduce new gems because they must be present in the primary source.

What is your fix for the problem, implemented in this PR?

See commit message

Make sure the following tasks are checked

• Describe the problem / feature
• Write tests for features and bug fixes
• Write code to solve the problem
• Make sure you follow the current code style and write meaningful commit messages without tags

cc/ @tenderlove @jenshenny

[tenderlove]

If I'm reading this correctly, it looks like we just emit a warning (and skip) if there's an override that doesn't exist in the main source. Shouldn't we raise an exception instead? It seems like a bug if there's an override but no canonical version

[jenshenny]

Something I've noticed, it's unclear from the lockfile if a gem installed comes from the override or the rubygems.org source and it becomes harder to audit your lockfile. For example:

GEM
  remote: https://rubygems.org/
  override: https://override-source.com/
  specs:
    ruby_dsp (0.0.6-x86_64-linux)
      rice (~> 4.11.2)

PLATFORMS
  ruby
  x86_64-linux

DEPENDENCIES
  ruby_dsp (= 0.0.6)

CHECKSUMS
  ruby_dsp (0.0.6) sha256=eb6ff50f612e1d501a1e0126782bca4e6ddb867e63ee898492e19484064f05cf
  ruby_dsp (0.0.6-x86_64-linux) sha256=c5a447d914616a21215cb1ab2ffae0211eb971214974ff49b80ef30c72d49693

BUNDLED WITH
  4.1.0.dev

It's hard to verify if ruby_dsp installed is actually from the primary or override source without checking the checksum entry. I think it'll be better if the blocks were separated somehow.

[tenderlove]

It's hard to verify if ruby_dsp installed is actually from the primary or override source without checking the checksum entry. I think it'll be better if the blocks were separated somehow.

I agree with this. It makes sense to separate them, I'm just not sure if we should make that part of this PR since I'm not sure what the impact of changing the lockfile format would be. I think we should land this and then figure out a way to fix the lockfile format. wdyt?