Repository navigation
Conversation
The worldwide stations query paged with OFFSET over every station, so each page re-ran the whole query and risked the service's time limit. One query per country stays small. A failed batch of sights is retried five towns at a time, and the run fails only when more than 1% of towns end up without sights. The log shows elapsed minutes.
The first full build timed out (HTTP 504) on every batch of sights and on the United States' towns. Sights now run in the written order, from the cities down, in batches of 20. Towns are found first and their details read once each, instead of multiplying populations, coordinates and labels into hundreds of rows per town.
Walking every subclass of human settlement from the top timed out even for Luxembourg; checking each town's class upwards is what worked.
A plain subquery was joined after the outer patterns, so every item with a population was read and even Monaco timed out. The live CI job tries each query once, so a query that times out fails the check quickly.
…x hours Great Britain's 8,000 stations timed out once with labels read inside the search. A full fetch of towns and sights takes a few hours.
Great Britain's towns and stations timed out now and then. Stations are now found from Wikidata's railway stations, then the country's; towns fall back to reading every item with a population when the country-first query fails. The live CI job checks Great Britain and the United States.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before: the first full build ran for two hours without finishing. Every batch of sights and the United States' towns failed with HTTP 504, Wikidata's time limit, and each failure was retried twice before moving on. The worldwide stations query paged with OFFSET over every station, so each page re-ran the whole query. A single failed batch of sights failed the whole run.
After: the queries fit the time limit, and a run survives the odd failure. Sights are found from the cities down, in batches of 20, with the query run in the written order. Towns are found first in a named subquery and their details read once each, instead of multiplying populations, coordinates and labels into hundreds of rows per town; when that quick query times out for a large country, a slower population-first query takes over. Stations are fetched with one query per country, also in a named subquery. A failed batch of sights is retried four towns at a time, and the run fails only when more than 1% of towns end up without sights, or more than two countries are missing towns or stations. Every log line shows the minutes elapsed.
How:
citiesQuery,sightsQueryandstationsQuery(countryIso2)insrc/wikidata.ts, and the loops inscripts/fetch.ts. The station filter "UIC code or Wikipedia article" becomes "UIC code or any Wikimedia page", which is much cheaper to check. The live CI job now fetches Monaco, Great Britain and the United States with a single try per query, so the fallbacks are exercised against the real Wikidata: the last run loaded 8,740 towns, 11,702 stations and 2,375 sights in 13 minutes. A full worldwide build should take a few hours, inside the publish job's six-hour limit.