fix(search): accept index schemas whose analyzers and CORS options are objects

[Rana Singh (ranadeepsingh)]

Why

Fixes #2143. An Azure AI Search index that uses a custom analyzer (or char filter, tokenizer, token filter, suggester, or CORS configuration) could not be written to at all:

spray.json.DeserializationException: Expected String as JsString, but got
{"@odata.type":"#Microsoft.Azure.Search.CustomAnalyzer","charFilters":[],
 "name":"keyword_analyzer","tokenFilters":["lowercase"],"tokenizer":"keyword_v2"}

Reported against 0.11.0 and still reproducible on master.

Root cause

IndexInfo declared six members as Option[Seq[String]], but the Search REST API returns objects for every one:

Member	Declared	Actually returned
suggesters	Option[Seq[String]]	array of objects
analyzers	Option[Seq[String]]	array of objects
charFilters	Option[Seq[String]]	array of objects
tokenizers	Option[Seq[String]]	array of objects
tokenFilters	Option[Seq[String]]	array of objects
corsOptions	Option[Seq[String]]	a single object, not an array

The right-hand column was confirmed against the live service, not inferred from the issue.

IndexInfo is deserialized twice from the caller's indexJson, both before any document is sent — parseIndexJson(...).name.get at the top of createIfNoneExists, and validateIndexInfo. So the write threw straight out of AzureSearchWriter.write regardless of apiVersion or whether the index already existed.

The fix

Type the six members as opaque JsValue.

The writer never inspects them — it only round-trips them to the service — so JsValue is both sufficient and strictly more robust: it won't break again when the service adds a new analyzer or tokenizer kind. Same class of defect as the standing TODO about synonymMaps.

In-repo compatibility: nothing reads these fields (verified by grep), and the only construction site passes None for all six. Existing callers compile unchanged, and serialized output is byte-identical since None fields are omitted.

⚠️ External breaking change (for release notes): IndexInfo is a public case class returned by the public IndexParser.parseIndexJson, so retyping changes its apply/copy/accessor signatures. Real exposure is near zero — under the old typing any index carrying these features threw on read, so no working downstream code could have depended on the Seq[String] shape. There is no MiMa gate here, so CI won't flag it; noting it so it can be captured in the next release notes.

Tests

IndexSchemaParsingSuite — 7 offline tests driven by the exact index definition from #2143: custom analyzer parses; custom analyzer survives parse → re-serialize byte-identically (the service rejects an analyzer it can't identify, so round-trip fidelity is the property that matters); object-valued charFilters/tokenizers/tokenFilters parse; object-valued suggesters parse; corsOptions parses as an object not an array; all six at once parse; omitting them yields empty rather than failure. Every one fails on master with Expected String as JsString.

IndexSchemaLiveRoundTripSuite — 1 end-to-end test. The offline suite asserts against hand-written JSON, so alone it proves the parser accepts these shapes, not that they're what the service emits. This creates a real index using all six features, reads it back through the same production getIndexJsonFromExistingIndex path the writer uses, parses it, and deletes the index in a finally. ~21s.

pipeline.yaml registers both in the misc leg — PipelineTestCoverageSuite fails the build otherwise, since nothing globs the services.search parent package.

Proving it's a real fix, not a cosmetic retype

Reverting AzureSearchSchemas.scala fails at compile time (the tests use the new types), which shows type-safety but not the runtime bug. So a payload captured live from mmlspark-azure-search (API 2026-04-01) was parsed through parseIndexJson alone under both schema versions:

Schema	Result
master's Option[Seq[String]]	DeserializationException: Expected String as JsString, but got {"name":"sml_suggester",...} — verbatim #2143
this PR's JsValue	parses successfully

Incidental finding worth recording: a suggester's sourceFields cannot reference a field using a custom analyzer — the service returns 400. The test points its suggester at a separate default-analyzer field.

Validation

IndexSchemaParsingSuite 7/7 · IndexSchemaLiveRoundTripSuite passes live · PipelineTestCoverageSuite passes · cognitive/scalastyle + Test/scalastyle 0 errors · full ADO pipeline 77/77 green, 0 failures, including the UnitTests misc leg that runs the live suite.

Checklist

• Tests added, including a live end-to-end test
• No dependency changes
• Bug fix, not a new feature — no website samples needed

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[github-actions]

Hey Rana Singh (@ranadeepsingh) 👋!
Thank you so much for contributing to our repository 🙌.
Someone from SynapseML Team will be reviewing this pull request soon.

We use semantic commit messages to streamline the release process.
Before your pull request can be merged, you should make sure your first commit and PR title start with a semantic prefix.
This helps us to create release messages and credit you for your hard work!

Examples of commit messages with semantic prefixes:

• fix: Fix LightGBM crashes with empty partitions
• feat: Make HTTP on Spark back-offs configurable
• docs: Update Spark Serving usage
• build: Add codecov support
• perf: improve LightGBM memory usage
• refactor: make python code generation rely on classes
• style: Remove nulls from CNTKModel
• test: Add test coverage for CNTKModel

To test your commit locally, please follow our guild on building from source.
Check out the developer guide for additional guidance on testing your change.

[Copilot]

Pull request overview

Fixes Azure AI Search index JSON deserialization so indexes that define object-valued schema features (custom analyzers/filters/tokenizers/suggesters and corsOptions) can be validated and written to without spray.json.DeserializationException. This targets SynapseML’s Search service integration (cognitive/services/search) where index JSON is parsed/validated before any document upload.

Changes:

• Update IndexInfo to treat suggesters, analyzer/tokenizer/filter collections, and corsOptions as opaque JsValue pass-through fields to match the Search REST API’s object shapes.
• Add a focused regression suite (IndexSchemaParsingSuite) covering parsing and round-trip preservation for the affected schema members (including the #2143 repro).
• Register the new suite in pipeline.yaml under the misc test leg to satisfy pipeline test coverage expectations.

Show a summary per file

File	Description
cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/search/AzureSearchSchemas.scala	Changes IndexInfo field types to JsValue/Seq[JsValue] to avoid failing on object-valued schema members returned by the Search API.
cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/search/IndexSchemaParsingSuite.scala	Adds regression tests that validate parsing of object-valued analyzers/tokenizers/filters/suggesters and object corsOptions, including the issue repro payload.
pipeline.yaml	Includes the new parsing suite in the CI test matrix (misc leg).

Review details

Tip

Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 3/3 changed files
• Comments generated: 0
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.05%. Comparing base (2c21cf2) to head (9884920).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2624   +/-   ##
=======================================
  Coverage   87.05%   87.05%           
=======================================
  Files         338      338           
  Lines       18843    18843           
  Branches     1805     1818   +13     
=======================================
  Hits        16403    16403           
  Misses       2440     2440           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Rana Singh (ranadeepsingh)]

Fixed a real failure this PR introduced, caught by the Release Branch Compatibility Check spark4.1 leg.

That check replays the PR as a patch onto the spark4.1 branch and compiles it. master imports spray-json by wildcard (import spray.json._), but spark4.1 imports it explicitly:

\\scala
import spray.json.{DefaultJsonProtocol, JsonFormat, RootJsonFormat}
\\

My IndexInfo hunk doesn't touch the import block, so JsValue resolved on master but not after replay — six not found: type JsValue errors.

Writing spray.json.JsValue in full makes the hunk self-contained and immune to import drift on any release branch. Adding an import line instead would have been fragile: the import blocks differ between the branches, so that hunk could fail to apply on its own.

Re-validated: IndexSchemaParsingSuite 7/7 green, cognitive/scalastyle 0 errors.

/azp run

[Copilot]

Review details

Suppressed comments (2)

cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/search/IndexSchemaParsingSuite.scala:46

• This test asserts JSON AST equality for the analyzer payload, not a byte-identical string round-trip. Updating the test name/comment avoids implying stronger guarantees than are actually being checked (which could confuse future debugging).

  test("a custom analyzer survives a parse and re-serialize round trip") {
    val original = indexJson(customAnalyzer)
    val roundTripped = parseIndexJson(original).toJson.asJsObject
    // The service rejects an analyzer it cannot identify, so every member has to come back intact.
    assert(roundTripped.fields("analyzers") == original.parseJson.asJsObject.fields("analyzers"))

cognitive/src/main/scala/com/microsoft/azure/synapse/ml/services/search/AzureSearchSchemas.scala:22

• The comment reads ungrammatically ("replayed onto import ..."). Rewording will make the rationale clearer for future maintainers.

// spray.json.JsValue is written out in full because the release branches this change is replayed
// onto import spray.json explicitly rather than by wildcard.

• Files reviewed: 3/3 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Rana Singh (ranadeepsingh)]

End-to-end validation against the live Azure Search service

The offline IndexSchemaParsingSuite asserts against hand-written JSON, so it proved the parser accepts the shapes reported in #2143 but not that those are the shapes the service actually emits. I closed that gap with a real round-trip.

What the live service actually returns

I created an index on mmlspark-azure-search (API version 2026-04-01) exercising all six object-valued schema features, then read the definition back. The service returns:

Field	Shape returned
suggesters	array of objects
analyzers	array of objects
charFilters	array of objects
tokenizers	array of objects
tokenFilters	array of objects
corsOptions	single object, not an array

That matches this PR exactly: five fields typed Option[Seq[JsValue]] and corsOptions typed Option[JsValue].

One incidental finding worth recording: a suggester's sourceFields cannot reference a field that uses a custom analyzer — the service rejects it with a 400. The test therefore points its suggester at a separate default-analyzer field.

A/B proof that this is a real fix, not a cosmetic retype

Reverting AzureSearchSchemas.scala to master doesn't fail at runtime — it fails at compile time, because the tests use the new types. That demonstrates type-safety but not the actual bug. So I captured the real live payload and parsed it through parseIndexJson alone, under both versions of the schema file:

• master's Option[Seq[String]] types → DeserializationException: Expected String as JsString, but got {"name":"sml_suggester",...} — verbatim issue [BUG] writeToAzureSearch fails when the index has custom analyzers or tokenizers since 0.11.0 #2143.
• this PR's JsValue types → parses successfully.

So the failure reproduces on genuine service output, and this change is what fixes it.

New regression test

Added IndexSchemaLiveRoundTripSuite, which creates a real index, reads it back through the production getIndexJsonFromExistingIndex path that AzureSearchWriter uses, parses it with parseIndexJson, and asserts the six fields deserialize. The index is deleted in a finally block. Runs in ~21s.

It's registered in the misc leg of pipeline.yaml — required, since PipelineTestCoverageSuite fails the build on any unclaimed suite and the services.search parent package isn't covered by a glob.

Verification run

scalastyle Found 0 errors
PipelineTestCoverageSuite: Tests: succeeded 1, failed 0
IndexSchemaLiveRoundTripSuite + IndexSchemaParsingSuite: Tests: succeeded 8, failed 0

All test indexes created during this work have been deleted; the service is back to its original 12 indexes.

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

Suppressed comments (1)

cognitive/src/test/scala/com/microsoft/azure/synapse/ml/services/search/IndexSchemaLiveRoundTripSuite.scala:93

• The cleanup in the finally block can mask the real test failure: deleteIndex calls safeSend, which throws on non-2xx responses. If the test body fails (e.g., parse assertion) and deletion also fails (transient network/429/403), the thrown exception from deleteIndex will override the original failure and make debugging harder. Wrap deleteIndex in a best-effort try/catch (or allow expected codes like 404) so the primary assertion failure is preserved.

    } finally {
      deleteIndex(indexName)
    }

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Rana Singh (ranadeepsingh)]

/azp run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[Copilot]

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Brendan Walsh (BrendanWalsh)]

Approved — automated multi-model review cycle.

Reviewed by Claude Opus 5 and GPT-5.6 Sol (both at maximum reasoning effort) against a blocking-issues-only rubric, followed by an independent adjudication pass. Final result: 0 blocking findings on commit 9884920c. All 30 checks are green.

One finding was raised and was overturned on verification:

"Preserve the published IndexInfo field types" (AzureSearchSchemas.scala:26) — not blocking.
The concern was that retyping suggesters/analyzers/charFilters/tokenizers/tokenFilters to Option[Seq[JsValue]] and corsOptions to Option[JsValue] is a source-compatibility break for downstream Scala consumers. The supporting evidence was mis-cited: VectorSchemaMigrationSuite.scala:188 locks down VectorSearch and IndexField only and asserts nothing whatsoever about IndexInfo. Beyond that:

• There is no MiMa or equivalent binary/source compatibility enforcement in this repo (project/plugins.sbt has none; zero hits in build.sbt).
• IndexInfo appears in zero docs and zero notebooks; a repo-wide grep finds zero accesses to any of the six retyped members.
• No working downstream consumer can exist in the first place, because the old Seq[String] typing threw on every real Azure AI Search payload — which is precisely the bug this PR fixes.

Backward compatibility on the read path was confirmed decisively: spray-json 1.3.5's AdditionalFormats.JsValueFormat.read(value) = value is a total identity read, so schemas that previously parsed as plain strings still parse (JsString is a JsValue). jsonFormat11 still derives, Option fields still omit on None, and prepareEntity sends aligned.compactPrint of the raw JSON rather than a re-serialized IndexInfo, so the wire format is untouched. Validation is unweakened — validateIndexInfo only checks name and the per-field rules, which this PR does not modify.

Non-blocking observations, recorded but not gating:

• IndexSchemaLiveRoundTripSuite.scala:78 reads the index back immediately after creation without retryWithBackoff, unlike all six existing call sites, and the misc leg carries no FLAKY: "true". Azure AI Search index creation is eventually consistent, so this can flake. Mitigated in practice by retryCountOnTaskFailure: 1 on the Unit Test task, which is why it is not gating.
• IndexSchemaLiveRoundTripSuite.scala:92 calls deleteIndex in a finally via safeSend, which defaults to expectedCodes = Set() and therefore throws on a non-2xx cleanup — that would mask the real assertion failure in a compound-failure scenario.