Implement source/registry overlays

[mkatychev]

#212 (comment)

@vados-cosmonic I agree, one of my hesitations in doing config/data driven coverage testing is that I plan to add overlay support cargo style sometime after #215. Note this commit from the multi package publishing PR:

wasm-pkg-tools/crates/wkg/src/main.rs

Lines 273 to 304 in 9de6af4

	let mut config = self.common.load_config().await?;
	let cache = self.common.load_cache().await?;
	let (overlay, tmp_dir) = wasm_pkg_client::local::LocalBackend::temp_dir()?;
	let reg_config = RegistryConfig::default().with_default_backend(
	"local",
	LocalConfig {
	root: tmp_dir.path().to_path_buf(),
	},
	)?;
	// Route every package in the plan to the local overlay registry
	// backed by `reg_config`, so the client used in `build_wit_dir`
	// resolves these packages against the local overlay instead of
	// an upstream remote.
	let local_registry: Registry = "tmp_local_publish".parse()?;
	config
	.get_or_insert_registry_config_mut(&local_registry)
	.merge(reg_config);
	let plan = PublishPlan::from_paths(paths)?;
	// TODO(mkatychev): Add support for `PackageLoader::get_release` to handle
	// querying on a per package, namespace, and registry level
	// to handle cargo style overlays.
	// see this reference of `cargo::core::Dependency` usage for local overlays in Cargo:
	// https://github.com/rust-lang/cargo/blob/d6900d00af2644ea1c0068c5694d9dbe11a3ab39/src/cargo/sources/overlay.rs#L47
	for spec in plan.iter() {
	config.set_package_registry_override(
	spec.package.clone(),
	RegistryMapping::Registry(local_registry.clone()),
	);
	}

In Cargo there is one type to handle overrides at all scopes (package, registry, workspace source replacement):
src/cargo/sources/overlay.rs#L14-L20
And the pattern for applying overlays happens once per pacakge/registry overrides are already done:
src/cargo/sources/config.rs#L240-L248

The current wasm_pkg_client::Client::resolve_source approach should wrap any self.sources.read() in an overlay lookup for better control over how and when sources/registries are overridden:

• wasm-pkg-tools/crates/wasm-pkg-client/src/lib.rs

  Line 298 in d40f518

  Ok(self.sources.read().await.get(&registry).unwrap().clone())

Note the overlays field in cargo::sources::config::SourceConfigMap:
https://github.com/rust-lang/cargo/blob/a595d0da21f228b7fdae64d3d5c0e527ea66bb59/src/cargo/sources/config.rs#L23-L31

There is only a SourceId to SourceId mapping and the resolution is done in scoped order: global -> workspace -> package.

Right now we do not resolve/normalize overrides in a layered approach so reasoning about the config aht various scopes becomes difficult and results in disjointed logic that does not use prior resolutions:

• wasm-pkg-tools/crates/wasm-pkg-client/src/lib.rs

  Lines 231 to 246 in d40f518

  	.and_then(|meta| {
  	// If the overridden registry matches the registry we are trying to resolve, we
  	// should use the metadata, otherwise we'll need to fetch the metadata from the
  	// registry
  	match (meta, is_override) {
  	(RegistryMapping::Custom(custom), true)
  	if custom.registry == registry =>
  	{
  	Some(custom.metadata.clone())
  	}
  	(RegistryMapping::Custom(custom), false) => {
  	Some(custom.metadata.clone())
  	}
  	_ => None,
  	}
  	})

In essence we should implement: an overlay flow to handle overrides with these rules:

• layering overlays: last write wins for the same key
• local always wins for conflicts inside an overlay
• creating an initial workspace overlay for handling per package overrides (this would involve restricting what one can specify in the global config)

[vados-cosmonic]

We might want to use config-rs if we can here

[mkatychev]

That's a great idea, and Eric is a maintainer, that's a seal of quality if any.