Update workflow to support import list overrides - #711
Conversation
Not up to standards ⛔🔴 Issues
|
| Category | Results |
|---|---|
| CodeStyle | 2 minor |
🟢 Metrics 6 complexity
Metric Results Complexity 6
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
There was a problem hiding this comment.
Code Review
This pull request updates the import workflow pipeline to support structured dictionary objects containing optional overrides for latestVersion and graphPath instead of only plain strings. Feedback on these changes highlights critical issues in get_import_info, including a graph path mismatch when only the version is overridden, a lack of backward compatibility for string inputs, and potential NoneType exceptions. Additionally, a bug was identified in parse_import_list where dictionaries missing the importName key are incorrectly stringified and appended as invalid import names.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request updates the ingestion pipeline and helper workflows to support passing import configurations as a list of dictionaries containing 'importName' and optional 'latestVersion' (representing the full GCS path), rather than just a list of strings. This change simplifies the data structure by consolidating 'graphPath' and 'latestVersion' into a single field. The feedback highlights two main improvement opportunities: first, validating input types in the aggregation helper and raising errors for malformed data instead of silently converting them to strings; second, reducing code duplication in the Spanner client by unifying the database query execution and removing unnecessary defensive string-stripping logic.
d71a506 to
0ac17a1
Compare
7bcbc07 to
1beabb4
Compare
Currently, ingestion workflow takes a list of import names as an argument. It then fetches the latest version information from the spanner ImportStatus table. This PR allows passing an optional latest version path as an input argument for easy backfilling. For this, we make the import list argument support a JSON body (importName and latestVersion). This input path eliminates the need for a spanner look up via an override. As part of this, code is cleaned up to merge graphPath and latestVersion params into a single latestVersion parameter.