Fixes #29996: parse Delta struct field names containing colons

[ulixius9]

Describe your changes:

Fixes #29996

I fixed ColumnTypeParser._parse_struct_fields_string so it splits each struct field on the last top-level : instead of requiring exactly name:type, because Unity Catalog Delta column-mapped struct fields carry : inside the field name itself (e.g. struct<baselineproportions:1:behavioral_segment_search_for_dm:bigint>) and previously crashed ingestion with a ValueError. A field type never contains a top-level colon (colons only appear inside bracketed types, which the split already protects), so the last segment is unambiguously the type and everything before it is the name.

Type of change:

• Bug fix

High-level design:

N/A — small change.

Tests:

Use cases covered

• Unity Catalog metadata ingestion of a Delta table whose struct column has a colon-bearing (column-mapped) field name now succeeds instead of raising expected format is: 'field_name:field_type'.

Unit tests

• I added unit tests for the changed logic.
• Files added/updated: ingestion/tests/unit/test_column_type_parser.py — the exact issue string, backtick-quoted colon names, mixed/nested fields, and a negative no-colon case.

Backend integration tests

• Not applicable (no backend API changes).

Ingestion integration tests

• Not applicable (parser-level fix; covered by unit tests).

Playwright (UI) tests

• Not applicable (no UI changes).

Manual testing performed

1. Created a real Databricks Unity Catalog Delta table (column mapping on) with events STRUCT<+""+baselineproportions:1:behavioral_segment_search_for_dm+""+: BIGINT>.
2. Ran the connector's parser against the live table: before the fix it raised the exact issue ValueError (exit 1); after the fix both columns parse cleanly (exit 0).
3. make-equivalent unit run: pytest ingestion/tests/unit/test_column_type_parser.py → 8 passed; ruff check/ruff format clean.

UI screen recording / screenshots:

Not applicable.

Checklist:

• I have read the CONTRIBUTING document.
• My PR title is Fixes <issue-number>: <short explanation>
• My PR is linked to a GitHub issue via Fixes #<issue-number> above.
• I have commented on my code, particularly in hard-to-understand areas.
• I have added tests (unit) and listed them above.
• I have added a test that covers the exact scenario we are fixing (issue Unity Catalog connector fails to parse Delta mapped struct field names containing colon paths #29996 referenced in the test docstring).

Greptile Summary

This PR allows Delta struct field names to contain colons. The main changes are:

• Uses the last top-level colon as the field name/type separator.
• Preserves colons within unquoted and backtick-quoted field names.
• Adds tests for nested, mixed, quoted, and invalid struct fields.

Confidence Score: 5/5

This looks safe to merge.

• No blocking issues found in the changed code.

Important Files Changed

Filename	Overview
ingestion/src/metadata/ingestion/source/database/column_type_parser.py	Parses the final top-level colon as the struct field name/type separator while preserving earlier colons in the field name.
ingestion/tests/unit/test_column_type_parser.py	Adds focused parser tests for colon-bearing field names and malformed fields without a separator.

_{Reviews (1): Last reviewed commit: "fix(ingestion): parse Delta struct field..." | Re-trigger Greptile}

Context used (3)

• Context used - CLAUDE.md (source)
• Context used - openmetadata-ui-core-components/CLAUDE.md (source)
• Context used - AGENTS.md (source)

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

• Issue Unity Catalog connector fails to parse Delta mapped struct field names containing colon paths #29996 is missing Shipping → Status.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

[gitar-bot]

Code Review ✅ Approved

Updates ColumnTypeParser to split struct field names on the final colon, allowing Unity Catalog Delta tables with column-mapped field names to parse correctly. Comprehensive unit tests covering nested and edge cases have been included.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}

[github-actions]

🟡 Playwright Results — all passed (28 flaky)

✅ 4525 passed · ❌ 0 failed · 🟡 28 flaky · ⏭️ 95 skipped

Shard	Passed	Failed	Flaky	Skipped
🟡 Shard 1	421	0	3	16
✅ Shard 2	11	0	0	0
🟡 Shard 3	823	0	9	8
🟡 Shard 4	820	0	3	18
🟡 Shard 5	836	0	3	5
🟡 Shard 6	785	0	3	46
🟡 Shard 7	829	0	7	2

🟡 28 flaky test(s) (passed on retry)

• Flow/TestConnectionModal.spec.ts › raw log toggle shows and hides connection log (shard 1, 1 retry)
• Pages/Lineage/LineageRightPanel.spec.ts › Verify custom properties tab IS visible for supported type: metric (shard 1, 1 retry)
• Flow/SearchRBAC.spec.ts › the browse tree only shows the asset-type categories a user can access (shard 1, 1 retry)
• Features/BulkImportWithDotInName.spec.ts › Full import cycle with dot in service name (shard 3, 1 retry)
• Features/BulkImportWithDotInName.spec.ts › Service name with multiple dots (shard 3, 1 retry)
• Features/ContextCenterArchive.spec.ts › archive page lazy-loads more rows on scroll within its own scroll container (shard 3, 1 retry)
• Features/ContextCenterArticles.spec.ts › Article listing search filters, clears, and shows empty state (shard 3, 1 retry)
• Features/ContextCenterArticles.spec.ts › Article list cards, recently viewed widget, and pagination work (shard 3, 1 retry)
• Features/ContextCenterMemories.spec.ts › clearing search restores the unfiltered list (shard 3, 1 retry)
• Features/ContextCenterPermission.spec.ts › created-by-me filter on the archive page shows only the current user's archived documents (shard 3, 1 retry)
• Features/Glossary/GlossaryHierarchy.spec.ts › should drag nested term to root level of same glossary (shard 3, 1 retry)
• Features/Glossary/GlossaryHierarchy.spec.ts › should move term with children to different glossary (shard 3, 1 retry)
• Features/Glossary/GlossaryRelationsGraph.spec.ts › search in the Relations Graph filters to the matching node and its neighbours (shard 4, 1 retry)
• Features/Glossary/GlossaryTermRelationsGraphNested.spec.ts › viewing a child term: parent appears as a 1-hop neighbour via parentOf edge (shard 4, 1 retry)
• Features/SearchExport.spec.ts › Export queues a background job and downloads from the jobs tray (shard 4, 1 retry)
• Pages/CustomProperties.spec.ts › Should verify property name is visible for apiCollection in right panel (shard 5, 1 retry)
• Pages/DataContracts.spec.ts › Contract Status badge should be visible on condition if Contract Tab is present/hidden by Persona (shard 5, 1 retry)
• Pages/DataContractsSemanticRules.spec.ts › Validate Description Rule Is_Set (shard 5, 1 retry)
• Pages/Entity.spec.ts › Delete Api Endpoint (shard 6, 1 retry)
• Pages/ExploreBrowse.spec.ts › service type drill-down disables unrelated roots and query-panel Clear resets it (shard 6, 1 retry)
• Pages/ExplorePageRightPanel_KnowledgeCenter.spec.ts › Should remove user owner for knowledgeCenter (shard 6, 1 retry)
• Pages/ExplorePageRightPanel.spec.ts › Should NOT show restricted edit buttons for Data Steward for dashboardDataModel (shard 7, 1 retry)
• Pages/GlossaryImportExport.spec.ts › Glossary Bulk Import Export (shard 7, 1 retry)
• Pages/GlossaryImportExport.spec.ts › Import partial success - some terms pass, some fail (shard 7, 1 retry)
• Pages/Lineage/LineageFilters.spec.ts › Verify Impact Analysis service filter selection (shard 7, 1 retry)
• Pages/Lineage/LineageRightPanel.spec.ts › Verify custom properties tab is NOT visible for pipelineService in platform lineage (shard 7, 1 retry)
• Pages/Lineage/LineageRightPanel.spec.ts › Verify custom properties tab is NOT visible for apiService in platform lineage (shard 7, 1 retry)
• Pages/TestSuite.spec.ts › Logical TestSuite (shard 7, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud