Fixes #29492: persist column extensions in bulk entity create path

[ivnvMkhl]

Description of Changes

Fixes #29492

createManyEntitiesFlushBody (the bulk PUT /v1/tables/bulk create path) called storeExtensions for table-level custom properties but never called storeColumnExtensions, so column-level custom properties were silently dropped on bulk create. The single-entity createNewEntityFlush path was not affected.

The fix mirrors what createNewEntityFlush already does — iterating entities and calling storeColumnExtensions(entity.getId(), getColumnsForExtensionPersistence(entity)) inside the storeEntities phase.

Type of Change

• Bug fix

High-level Design

N/A — 3-line change, no architectural impact. Adds the missing storeColumnExtensions loop to createManyEntitiesFlushBody to match the single-entity create path.

Testing Coverage

• Manual testing: created tables with column custom properties via PUT /v1/tables/bulk, confirmed properties are persisted in entity_extension and visible in the Custom Properties tab.
• Unit/integration tests: existing tests for storeColumnExtensions cover the logic; no new test infrastructure required for this call-site addition.

UI Evidence

Not applicable — backend-only fix.

Checklist

• Read the CONTRIBUTING documentation
• Title follows Fixes #<issue>: <description> format
• PR linked to issue PUT /v1/tables/bulk silently drops column-level custom properties on CREATE #29492

---

Summary by Gitar

• Bug fix:
  • Added missing storeColumnExtensions call to createManyEntitiesFlushBody to ensure column-level custom properties persist during bulk operations.
  • Extended storeColumnExtensions persistence to createManyEntitiesForImport and updateManyEntitiesForImport to prevent column extension data loss during CSV imports.
• Testing:
  • Added comprehensive integration tests in ColumnCustomPropertiesIT covering bulk creation and bulk updates of tables with column extensions.

_{This will update automatically on new commits.}

Greptile Summary

This PR fixes a silent data loss bug where column-level custom properties were dropped on all bulk entity paths. The single-entity createNewEntityFlush path already called storeColumnExtensions; the three bulk paths (createManyEntitiesFlushBody, createManyEntitiesForImport, and updateManyEntitiesForImport) were missing this call.

• Adds the missing storeColumnExtensions loop to createManyEntitiesFlushBody (the PUT /v1/tables/bulk path), createManyEntitiesForImport (CSV import create), and updateManyEntitiesForImport (CSV import update), each mirroring the already-correct single-entity path.
• Adds two integration tests in ColumnCustomPropertiesIT covering bulk create and bulk update of tables with column-level custom properties via the bulkCreateOrUpdate client method.

Confidence Score: 5/5

Safe to merge — the three-line additions are additive, each mirroring a call that already existed and was already tested in the single-entity path.

Each added loop calls storeColumnExtensions, which is an upsert and a no-op for entity types that do not override getColumnsForExtensionPersistence. The change is isolated to the bulk write path, does not alter read logic, and the two new integration tests validate both the create and update scenarios end-to-end.

No files require special attention beyond the one minor test assertion noted in ColumnCustomPropertiesIT.

Important Files Changed

Filename	Overview
openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java	Adds storeColumnExtensions loop in three bulk paths (createManyEntitiesFlushBody, createManyEntitiesForImport, updateManyEntitiesForImport) to match the single-entity create/update behavior; all three previous P1 findings are resolved.
openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/ColumnCustomPropertiesIT.java	Two new integration tests for bulk create/update column extension persistence; minor missing size assertion before getColumns().get(0) in the update test.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Client
    participant BulkAPI as PUT /v1/tables/bulk
    participant Repo as EntityRepository
    participant DB as entity_extension

    Client->>BulkAPI: PUT /v1/tables/bulk [tables with column extensions]
    BulkAPI->>Repo: createManyEntitiesFlushBody(entities)
    Repo->>DB: storeEntities(entities)
    Repo->>DB: storeExtensions(entities) — table-level custom props
    loop for each entity [NEW]
        Repo->>DB: storeColumnExtensions(entity.getId(), columns) — column-level custom props
    end
    Repo->>DB: storeRelationshipsInternal(entities)
    BulkAPI-->>Client: 200 OK (column extensions now persisted)

    note over Repo,DB: Same fix applied to createManyEntitiesForImport (CSV create) and updateManyEntitiesForImport (CSV update)

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Client
    participant BulkAPI as PUT /v1/tables/bulk
    participant Repo as EntityRepository
    participant DB as entity_extension

    Client->>BulkAPI: PUT /v1/tables/bulk [tables with column extensions]
    BulkAPI->>Repo: createManyEntitiesFlushBody(entities)
    Repo->>DB: storeEntities(entities)
    Repo->>DB: storeExtensions(entities) — table-level custom props
    loop for each entity [NEW]
        Repo->>DB: storeColumnExtensions(entity.getId(), columns) — column-level custom props
    end
    Repo->>DB: storeRelationshipsInternal(entities)
    BulkAPI-->>Client: 200 OK (column extensions now persisted)

    note over Repo,DB: Same fix applied to createManyEntitiesForImport (CSV create) and updateManyEntitiesForImport (CSV update)

Loading

Comments Outside Diff (3)

1. openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java, line 3763-3768 (link)

   createManyEntitiesForImport has the same missing storeColumnExtensions call

   createManyEntitiesForImport (the CSV import path) calls storeEntities → storeExtensions → storeRelationshipsInternal inside its transaction but never calls storeColumnExtensions. This is structurally identical to the bug being fixed in createManyEntitiesFlushBody — column-level custom properties will be silently dropped when tables (or dashboard data models) are bulk-created via CSV import, just as they were via PUT /v1/tables/bulk before this patch.

2. openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java, line 3847-3852 (link)

   createManyEntitiesForImport has the same missing storeColumnExtensions call

   createManyEntitiesForImport (the CSV import path) calls storeExtensions but never storeColumnExtensions inside its transaction. This is structurally identical to the bug fixed in createManyEntitiesFlushBody — column-level custom properties on tables (and dashboard data models) will be silently dropped when entities are bulk-created via CSV import, just as they were via PUT /v1/tables/bulk before this patch.

3. openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java, line 3896-3909 (link)

   updateManyEntitiesForImport deletes column extensions and never restores them

   removeExtensions(originals) calls deleteAllBatch on each entity's ID, which removes every row in entity_extension for that entity — including column-level extensions stored under column FQN hashes. storeExtensions(updatedEntities) then re-writes only entity-level custom properties. Column-level extensions are permanently deleted on every CSV import update, which is structurally worse than the create-side omission fixed in this PR.

_{Reviews (6): Last reviewed commit: "Mirrors storeColumnExtensions into CSV-i..." | Re-trigger Greptile}

[github-actions]

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

• Linked issue PUT /v1/tables/bulk silently drops column-level custom properties on CREATE #29492 is not added to the Shipping project.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

[github-actions]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[github-actions]

Jest test Coverage

UI tests summary

Lines	Statements	Branches	Functions
	63.42% (70688/111446)	45.9% (40625/88504)	47.81% (12369/25866)

[github-actions]

🔴 Playwright Results — 4 pipeline/setup failure(s)

✅ 0 passed · ❌ 0 failed · 🟡 0 flaky · ⏭️ 0 skipped

Pipeline and setup failures

• The build job finished with status failure.
• Shard detection finished with status failure.
• The Playwright shard matrix was unexpectedly skipped.
• No expected Playwright shards were declared.

Shard	Passed	Failed	Flaky	Skipped

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ui'

Issues
1241 New issues
0 Accepted issues

Measures
13 Security Hotspots
0.0% Coverage on New Code
3.4% Duplication on New Code

See analysis details on SonarQube Cloud

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[harshach]

@ivnvMkhl cna you please add integration-test for this

[ivnvMkhl]

@harshach Added an integration test — test_tableColumn_bulkCreatePersistsExtension in ColumnCustomPropertiesIT.java, exercising the bulk-create path this PR fixes. Also added test_tableColumn_bulkUpdatePersistsExtension as a smoke check on the neighboring bulk-update path.

[ivnvMkhl]

@gitar-bot Looked into the CSV import path more closely before mirroring the fix there.

createManyEntitiesForImport/updateManyEntitiesForImport are indeed missing the storeColumnExtensions call that createManyEntitiesFlushBody just got — but it's not currently a live/reachable bug: the CSV row schema for entityType=column (EntityCsv.updateColumnsFromCsvRecursive, and TableRepository.TableCsv) has no column.extension header at all — only dataType/description/tags/etc. So every Column built from a CSV row always has extension == null, and storeColumnExtensions would be a no-op there regardless of this change.

Fixing just the repository method without also teaching the CSV format/parser to carry column.extension wouldn't have any observable effect for users importing via CSV. I've parked a defense-in-depth fix (mirrors the two methods to match createManyEntitiesFlushBody) on a draft branch rather than opening a separate issue/PR for something not end-to-end reachable yet: https://github.com/ivnvMkhl/OpenMetadata/tree/draft/csv-import-column-extensions

Happy to turn that into a proper issue/PR (plus the CSV format work) if the team wants it prioritized — for now keeping focus on this PR and #29496/#29497.

[ivnvMkhl]

@gitar-bot Both findings from this review addressed:

1. CSV import path still drops column custom properties — mirrored the storeColumnExtensions call into createManyEntitiesForImport and updateManyEntitiesForImport (pushed in 484b8a8), matching the pattern in createManyEntitiesFlushBody. One caveat: today the CSV row schema for entityType=column has no column.extension header, so every Column parsed from CSV currently has extension == null and this call is a no-op end-to-end for now — but it brings both methods in line with their sibling and stops the silent drop the moment CSV format gains that field (or anything else populates it through this path).

2. writeEpochById/writeEpochByName naming — checked git blame on those lines: they're not introduced by this PR. They're pre-existing on main (commit 7eee9f956b, unrelated to this change) and just happen to fall inside this diff's context window. Renaming them here would be touching unrelated code outside the scope of this fix, so leaving as-is in this PR — happy to file a separate small PR for the rename if the team wants it.

[gitar-bot]

Code Review 👍 Approved with suggestions 1 resolved / 2 findings

Persists column extensions in bulk creation by mirroring the single-entity persistence logic in createManyEntitiesFlushBody. Integration tests confirm property retention, though similar data-loss issues persist in the CSV import paths for createManyEntitiesForImport and updateManyEntitiesForImport.

💡 Quality: writeEpochById/writeEpochByName read rather than write

📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:382-390

The new @VisibleForTesting helpers writeEpochById/writeEpochByName simply delegate to readEpochById/readEpochByName, so their write prefix is misleading — they observe the current epoch, they do not bump it. Rename to something like currentEpochById/currentEpochByName (or readEpoch*) to avoid confusing future readers/tests into thinking they mutate the write epoch.

✅ 1 resolved

✅ Bug: CSV import path still drops column custom properties

📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:3847-3852 📄 openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:5297-5308
createManyEntitiesForImport (the CSV bulk-import path) runs the same storeEntities → storeExtensions → storeRelationshipsInternal sequence as the just-fixed createManyEntitiesFlushBody but never calls storeColumnExtensions, so column-level custom properties are silently dropped when tables/data models are created via CSV import — the exact bug this PR fixes for PUT /v1/tables/bulk. Add the same loop after storeExtensions(entities); getColumnsForExtensionPersistence returns an empty list for non-column entities and storeColumnExtensions short-circuits on empty, so it is a no-op for other types.

🤖 Prompt for agents

Code Review: Persists column extensions in bulk creation by mirroring the single-entity persistence logic in `createManyEntitiesFlushBody`. Integration tests confirm property retention, though similar data-loss issues persist in the CSV import paths for `createManyEntitiesForImport` and `updateManyEntitiesForImport`.

1. 💡 Quality: writeEpochById/writeEpochByName read rather than write
   Files: openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/EntityRepository.java:382-390

   The new @VisibleForTesting helpers `writeEpochById`/`writeEpochByName` simply delegate to `readEpochById`/`readEpochByName`, so their `write` prefix is misleading — they observe the current epoch, they do not bump it. Rename to something like `currentEpochById`/`currentEpochByName` (or `readEpoch*`) to avoid confusing future readers/tests into thinking they mutate the write epoch.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}

[ivnvMkhl]

Completed