Skip to content

fix: empty RT entity types - #1856

Merged
davidgamez merged 2 commits into
mainfrom
fix_rt_entity_types_empty
Oct 6, 2026
Merged

davidgamez merged 2 commits into
mainfrom
fix_rt_entity_types_empty

Conversation

@davidgamez

@davidgamez davidgamez commented Oct 5, 2026 •

Copy link
Copy Markdown
Member

Summary:

Fixes #1847. GTFS-RT feeds with a blank entity type returned HTTP 500 on every access path, and broke any /v1/search page spanning them.

process_entity_types in api/src/scripts/populate_db_gtfs.py split the entity_type cell with "".split("-"), which returns [""] rather than []. The loop therefore ran once with an empty token and created Entitytype(name=""); the else branch was dead code. The generated models validate entity_types against the vp/tu/sa enum from DatabaseCatalogAPI.yaml, so a blank name failed serialization for the feed and for every unrelated feed on the same page.

Three layers:

  • Populate script — new normalize_entity_types trims, lowercases, drops blanks, dedupes, and keeps only EntityType members (warn + skip otherwise). Entity types are replaced only when the normalized list is non-empty, preserving the existing "empty cell does not wipe" convention.
  • Migration (liquibase/changes/fix_empty_entity_types.sql) — deletes blank entitytypefeed then entitytype rows, adds CHECK (name IN ('vp','tu','sa')), refreshes feedsearch.
  • Tests — fixture rows for an empty cell, a padded/mixed-case cell (VP | sa), and a partly invalid cell (tu-bogus), plus a DB-wide assertion that no blank entity type names exist.

Expected behavior:

  • GET /v1/search?data_type=gtfs_rt&limit=100&offset=0 no longer 500s.
  • A blank, padded, or unknown entity_type cell in the catalog CSV produces no entitytype row; VP | sa yields ["vp","sa"], tu-bogus yields ["tu"] with a logged warning.
  • INSERT INTO entitytype(name) VALUES ('') is rejected by the DB.

No public API surface change. Note: get_gtfs_rt_feeds_query inner-joins Entitytype, so the two fixed feeds stay out of the /v1/gtfs_rt_feeds list endpoint until they have real entity types (pre-existing behaviour, not changed here).

Testing tips:

# Apply the migration locally
docker compose --env-file ./config/.env.local up liquibase liquibase-test

# Constraint blocks bad names
docker exec database psql -U postgres -d MobilityDatabase \
  -c "INSERT INTO entitytype(name) VALUES ('');"   # expect: violates check constraint

# Only canonical names survive population
./scripts/docker-localdb-rebuild-data.sh --populate-db --populate-test-data
docker exec database psql -U postgres -d MobilityDatabase \
  -c "SELECT name, length(name) FROM entitytype;"  # expect only vp/tu/sa

./scripts/api-tests.sh

After deploy, confirm GET /v1/gtfs_rt_feeds/mdb-3438, mdb-3439, and a full-catalog paginated crawl of /v1/search?data_type=gtfs_rt.

  • Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

Comment thread docs/OperationsAPI.yaml
description: Unique identifier of the GTFS feed.
example: mdb-123
latest_state:
$ref: "#/components/schemas/GtfsFeedContinuousCoverage"

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes in the operations API are missing updates from the previous PRs.

@davidgamez
davidgamez marked this pull request as ready for review October 6, 2026 14:56
@davidgamez davidgamez changed the title fix : empty RT entity types fix: empty RT entity types Oct 6, 2026

@cka-y cka-y left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

# An empty cell leaves the stored entity types alone.
self.logger.warning(f"Entity types array is empty for feed {stable_id}")
feed.entitytypes.clear()
return

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@davidgamez
davidgamez merged commit 3ee7de5 into main Oct 6, 2026
32 of 33 checks passed
@davidgamez
davidgamez deleted the fix_rt_entity_types_empty branch October 6, 2026 15:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Two GTFS-RT feed records return HTTP 500 on every access path (mdb-3438, mdb-3439)

2 participants