Value-weighted, lane-driven market scan budgeting (replace uniform 1:1 probe coverage)

[jdkajewski]

Problem

Market prices are read per-waypoint (GET /systems/{sys}/waypoints/{wp}/market) and only return live prices/volumes when a ship is present at that waypoint. There is no batch market endpoint. So refreshing N markets costs N API calls against the 2 req/s per-account ceiling shared by the entire fleet.

Today the bot parks ~1 probe per market and refreshes them roughly uniformly. But markets vary ~100x in value, and a market only matters if it's an endpoint of a profitable lane. Uniform coverage spends scarce request budget on dead markets (low tradeVolume, all-EXCHANGE, no arbitrage), starving the trades that actually earn.

Goal: make scan budget a managed resource allocated by profit contribution — concentrate reads on the markets that feed the best lanes, prune dead markets to near-zero cost, and reduce probe count. This directly lifts credits-per-request, the only lever that matters under the rate ceiling.

Core shift: market-centric 1:1 → value-weighted, lane-driven coverage

1. Score every market by lane contribution — V(market)

• Realized profit attribution: track net credits per lane, attribute to its src + sink markets, decay as an EWMA. Markets feeding top lanes score high.
• Structural potential: from the market's own data — Σ over goods (tradeVolume × estimated_margin), margin from cross-market price gaps. EXPORT = buy-low source, IMPORT = sell-high sink, all-EXCHANGE + low volume ≈ 0.
• Volume: high tradeVolume enables bigger lanes (more cargo/trip = more credits/request).

2. Value-weighted adaptive scan interval

T(market) = base / (V × volatility), clamped [Tmin, Tmax]. Extends the existing volatility-only adaptive TTL by adding value, so hot+volatile markets refresh often and calm/low-value ones rarely (or drop out).

3. Coverage tiering + probe pruning

• Hot → dedicated fresh coverage
• Warm → periodic
• Cold/dead → scan once to classify, then drop the probe (don't buy / redeploy)

Run with fewer probes concentrated on high-value markets. Stop paying recurring scan budget for markets that produce nothing.

4. Lane registry as source of truth

Maintain top-K lanes (srcMkt, good, sinkMkt, margin, volume, lastSeen) ranked by net credits/min. A market must be fresh only if it's an endpoint of an active/candidate lane; markets in no profitable lane are demoted automatically.

5. Traders as free scanners

A trader docking to trade already reads that market for free. Route traders through markets needing refresh so the scan folds into the earning trip — shrinking dedicated probe scans to only the markets traders don't naturally pass.

6. Global scan-budget scheduler

A priority queue keyed by value × staleness, draining at the available req/s. Makes the 2 req/s allocation explicit and profit-optimal instead of emergent; naturally starves dead markets.

7. Periodic cheap re-evaluation

Markets shift over time. Cold markets get a counts-only re-check on a long interval to catch newly-opened opportunities without paying full-scan cost.

Expected outcome

Fewer probes, scan budget concentrated on the markets feeding the best lanes, dead markets pruned to near-zero cost → higher credits-per-request and more headroom for actual trading under the 2 req/s ceiling.

Relation to the galaxy crawler (dk-galaxy-crawler-tie-in)

The crawler's SystemRichness is a useful prior (which systems are worth entering). This issue is a distinct runtime concern: operational scan scheduling driven by observed lane profitability. They share the value concept and should exchange signals (crawler seeds priors → runtime refines from realized lanes), but remain separate subsystems. The crawler's data model should leave room for per-market value scoring + lane attribution.

Phasing (suggested)

1. Lane-profit attribution + per-market realized-value tracking (data + EWMA).
2. Structural-potential scoring from market structure (tradeVolume, EXPORT/IMPORT spreads).
3. Value-weighted adaptive scan interval (extend existing TTL).
4. Coverage tiering + probe pruning + redeploy.
5. Global scan-budget scheduler (priority queue over value × staleness).
6. Traders-as-scanners routing integration.
7. Periodic cold-market cheap re-check.

Notes

• Per-market reads are the scarce resource — treat every GET /market as budget spend.
• Keep all thresholds/intervals tunable (levers), consistent with the existing config-lever pattern.