Skip to content

Keep online endpoint/deployment operations pinned to workspace scope for registry-backed clients - #48325

Open
Chakradhar886 with Copilot wants to merge 16 commits into
mainfrom
copilot/fix-ml-registry-deployment-issue
Open

Chakradhar886 with Copilot wants to merge 16 commits into
mainfrom
copilot/fix-ml-registry-deployment-issue

Conversation

Copilot AI commented Jul 29, 2026 •

Copy link
Copy Markdown
Contributor

Online deployment requests could be sent to the registry resource group when a client was initialized to consume registry assets and target a workspace in a different resource group. In that configuration, endpoint/deployment operations looked up the workspace endpoint under the wrong RG and failed with ResourceNotFound.

  • Scope isolation for online operations

    • Introduce a dedicated workspace-scoped OperationScope for online endpoint and online deployment operations.
    • Keep registry-backed asset operations on registry scope, but force online control-plane calls to use the workspace subscription/resource group/workspace.
  • Dedicated service clients for online calls

    • Create workspace-scoped 2022-02 and 2023-04-preview ARM clients specifically for online endpoint/deployment operations.
    • Route OnlineEndpointOperations and OnlineDeploymentOperations through those workspace-scoped clients instead of the registry-shifted shared scope.
  • Regression coverage

    • Add a unit test that initializes MLClient with a registry plus workspace_reference and asserts:
      • online endpoint/deployment operations retain workspace scope
      • model/data operations continue to use registry scope
  • Behavioral effect

    • Registry assets can still be referenced from a different resource group, while online endpoint/deployment resolution stays anchored to the target workspace.
ml_client = MLClient(
    credential=credential,
    subscription_id=workspace_sub,
    resource_group_name=workspace_rg,
    registry_name="my-registry",
    workspace_name="my-workspace",
)

# Registry-backed assets may resolve via registry scope,
# but online endpoint/deployment operations stay on workspace scope.
ml_client.online_deployments.begin_create_or_update(deployment)

Testing

Samples validations: Azure/azureml-examples#4110

…d clients

Co-authored-by: Chakradhar886 <259224138+Chakradhar886@users.noreply.github.com>
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

…0 chars)

Co-authored-by: Chakradhar886 <259224138+Chakradhar886@users.noreply.github.com>
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
9 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Pins online endpoint and deployment operations to workspace scope while preserving registry scope for assets.

Changes:

  • Adds workspace-scoped online operation clients and scopes.
  • Adds regression coverage for mixed workspace/registry configuration.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
azure/ai/ml/_ml_client.py Routes online operations through workspace-scoped clients.
tests/internal_utils/unittests/test_ml_client.py Tests workspace and registry scope isolation.

Comment thread sdk/ml/azure-ai-ml/tests/internal_utils/unittests/test_ml_client.py Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (3)

sdk/ml/azure-ai-ml/tests/conftest.py:496

  • The end-to-end regression likewise exercises registry_reference + workspace_name, rather than the registry_name + workspace_reference configuration described by this fix. These forms take different scope-selection branches, so the test can pass while the reported scenario is broken.
        workspace_name=e2e_ws_scope.workspace_name,
        registry_reference="sdkv2-testFeed",

sdk/ml/azure-ai-ml/tests/internal_utils/unittests/test_ml_client.py:546

  • This test does not exercise the reported registry_name + workspace_reference construction path. It instead uses registry_reference + workspace_name, which follows different conditionals for asset operation scopes (for example, DataOperations remains workspace-scoped in that path). Use the customer-facing configuration so a regression in the branch fixed by this PR is detected.
                workspace_name=workspace_name,
                registry_reference="test-registry",

sdk/ml/azure-ai-ml/tests/internal_utils/unittests/test_ml_client.py:565

  • The regression criteria say both model and data operations must retain registry scope, but this test only asserts the model scope. Capture and verify DataOperations too; this also protects the asset-scope half of the isolation contract.
        model_scope = captured["ModelOperations"]["scope"]

Copilot AI review requested due to automatic review settings August 14, 2026 04:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (3)

sdk/ml/azure-ai-ml/tests/internal_utils/unittests/test_ml_client.py:546

  • This test reverses the configuration from the reported regression. Passing registry_reference made the pre-change online scope select _ws_operation_scope, so it does not reproduce the wrong-resource-group path caused by registry_name with workspace_reference. Construct the client with those arguments so the test fails on the old implementation and validates the intended fix.
                workspace_name=workspace_name,
                registry_reference="test-registry",

sdk/ml/azure-ai-ml/tests/conftest.py:496

  • The end-to-end fixture also uses the inverse configuration from the failing scenario. With registry_reference, the old code already selected the workspace operation scope; use registry_name plus workspace_reference to exercise the cross-resource-group regression described by this PR.
        workspace_name=e2e_ws_scope.workspace_name,
        registry_reference="sdkv2-testFeed",

sdk/ml/azure-ai-ml/tests/internal_utils/unittests/test_ml_client.py:564

  • The regression coverage described for this PR also requires verifying that data operations remain registry-scoped, but the test only checks models. Add corresponding data-scope assertions; this will also guard the scope isolation for both asset operation types.
        assert model_scope.subscription_id == registry_sub
        assert model_scope.resource_group_name == registry_rg

Copilot AI review requested due to automatic review settings August 14, 2026 06:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (1)

sdk/ml/azure-ai-ml/tests/internal_utils/unittests/test_ml_client.py:565

  • The advertised regression configuration is registry_name plus workspace_reference, but this test uses the inverse pairing; those paths initialize asset scopes differently, and the promised DataOperations registry-scope assertion is also absent. Exercise the reported configuration directly and assert that data operations remain registry-scoped.
            MLClient(
                credential=mock_credential,
                subscription_id=workspace_sub,
                resource_group_name=workspace_rg,
                workspace_name=workspace_name,

Copilot AI review requested due to automatic review settings August 14, 2026 06:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (3)

sdk/ml/azure-ai-ml/tests/internal_utils/unittests/test_ml_client.py:546

  • This setup exercises the registry_reference path, whose online operation scope was already selected from _ws_operation_scope before this change. The reported configuration is registry_name with workspace_reference, so this test would still pass if the new registry_name branch in _online_operation_scope regressed. Construct the client with that argument pair and omit workspace_name, which is mutually exclusive with registry_name.
                workspace_name=workspace_name,
                registry_reference="test-registry",

sdk/ml/azure-ai-ml/tests/internal_utils/unittests/test_ml_client.py:564

  • The advertised regression coverage also requires data operations to remain registry-scoped, but DataOperations is captured and never asserted. Add its subscription/resource-group checks; with the intended registry_name + workspace_reference setup, these assertions also guard against accidentally shifting asset operations back to workspace scope.
        assert model_scope.subscription_id == registry_sub
        assert model_scope.resource_group_name == registry_rg

sdk/ml/azure-ai-ml/tests/conftest.py:496

  • This E2E fixture uses registry_reference, for which the endpoint/deployment operation scope was already workspace-scoped before this PR. It therefore does not reproduce the reported wrong-resource-group path, which is reached by combining registry_name with workspace_reference. Configure that pair here so the E2E test actually validates the new registry_name scope isolation.
        workspace_name=e2e_ws_scope.workspace_name,
        registry_reference="sdkv2-testFeed",

@github-actions

This comment has been minimized.

Copilot AI review requested due to automatic review settings August 14, 2026 07:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

Suppressed comments (2)

sdk/ml/azure-ai-ml/tests/internal_utils/unittests/test_ml_client.py:564

  • The PR states that both model and data operations remain registry-scoped, but only the model scope is asserted. DataOperations is wired through a separate conditional in MLClient, so add explicit assertions to prevent its scope from silently changing. These assertions should be used with the registry_name + workspace_reference setup above.
        assert model_scope.subscription_id == registry_sub
        assert model_scope.resource_group_name == registry_rg

sdk/ml/azure-ai-ml/tests/internal_utils/unittests/test_ml_client.py:546

  • This regression test exercises workspace_name + internal registry_reference, not the reported registry_name + workspace_reference configuration. Those paths behaved differently before this change: registry_reference already selected _ws_operation_scope, so this test does not reproduce the wrong-resource-group bug. Instantiate the client with the exact public configuration so the test fails if that branch regresses.
                workspace_name=workspace_name,
                registry_reference="test-registry",

@github-actions

Copy link
Copy Markdown
Contributor
[Pilot] PR Pipeline Failure Analysis

A CI pipeline failed on this pull request. Here is an automated analysis of what went wrong and how to get the build green.

What failed

The python - pullrequest pipeline (build 6702812) failed across all six test matrix legs (macOS 3.11, Ubuntu 3.10, Ubuntu 3.10 coverage, Ubuntu 3.13, Ubuntu 3.14, Windows 3.12). Every leg failed on the same single test:

sdk.ml.azure-ai-ml.tests.online_services.e2etests.test_online_deployment.TestOnlineDeployment.test_online_deployment_create_when_registry_assets

This is a test failure — the end-to-end test test_online_deployment_create_when_registry_assets is failing consistently across all platforms and Python versions, which strongly suggests the PR's code changes to online deployment/registry scope logic are not yet fully correct (or a test recording needs to be updated).

Recommended next steps

  • Review the failing test test_online_deployment_create_when_registry_assets in sdk/ml/azure-ai-ml/tests/online_services/e2etests/test_online_deployment.py to understand what it asserts.
  • Check whether the test uses recorded cassettes/playback — if so, the recordings may need to be regenerated to reflect the new workspace-scoped client behavior introduced in this PR.
  • If running live, verify that the registry-backed MLClient with workspace_reference correctly routes online endpoint/deployment calls to workspace scope as described in the PR description.
  • Inspect the detailed per-job logs for the specific assertion error or exception:
  • See the CI troubleshooting guide: https://aka.ms/ci-fix
  • Push new commits to address the failures; this comment updates automatically on the next failing run.
Raw pipeline analysis (azsdk ci analyze)
Analyzing pipeline https://github.com/Azure/azure-sdk-for-python/pull/48325...
Getting failed workflow runs for commit f2e645be331dc326dfdb75c4c8c4bed25e1e812c in Azure/azure-sdk-for-python
Build: 6702812 Project: public PipelineUrl: https://dev.azure.com/azure-sdk/public/_build/results?buildId=6702812
--------------------------------------------------------------------------------
Failed Tests
--------------------------------------------------------------------------------
{
  "macos311": [
    "sdk.ml.azure-ai-ml.tests.online_services.e2etests.test_online_deployment.TestOnlineDeployment.test_online_deployment_create_when_registry_assets",
    "sdk.ml.azure-ai-ml.tests.online_services.e2etests.test_online_deployment.TestOnlineDeployment.test_online_deployment_create_when_registry_assets"
  ],
  "Ubuntu2404_313": [
    "sdk.ml.azure-ai-ml.tests.online_services.e2etests.test_online_deployment.TestOnlineDeployment.test_online_deployment_create_when_registry_assets",
    "sdk.ml.azure-ai-ml.tests.online_services.e2etests.test_online_deployment.TestOnlineDeployment.test_online_deployment_create_when_registry_assets"
  ],
  "ubuntu2404_310": [...],
  "ubuntu2404_310_coverage": [...],
  "Ubuntu2404_314": [...],
  "windows2022_312": [...]
}
--------------------------------------------------------------------------------
Failing checks on the pull request
--------------------------------------------------------------------------------
  python - pullrequest [FAILURE] https://dev.azure.com/azure-sdk/public/_build/results?buildId=6702812
  (Build Test Ubuntu2404_313) [FAILURE]
  (Build Test Ubuntu2404_314) [FAILURE]
  (Build Test macos311) [FAILURE]
  (Build Test ubuntu2404_310) [FAILURE]
  (Build Test ubuntu2404_310_coverage) [FAILURE]
  (Build Test windows2022_312) [FAILURE]

Copilot detected the failing pipeline and generated the analysis above. To have it attempt a fix automatically, reply with @copilot please fix the failing pipeline on this PR.

Generated by Pipeline Analysis - Next Steps · 21.9 AIC · ⌖ 9.1 AIC · ⊞ 6.6K · ◷

Copilot AI review requested due to automatic review settings August 14, 2026 09:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Suppressed comments (2)

sdk/ml/azure-ai-ml/tests/internal_utils/unittests/test_ml_client.py:565

  • This regression test constructs the alternate workspace_name + registry_reference path, not the reported registry_name + workspace_reference configuration. Those paths differ here: only the latter must copy workspace_reference into the online scope, and it also keeps DataOperations registry-scoped. As written, the test would still pass if that exact path regressed, and it omits the promised data-scope assertion. Construct the reported configuration and assert the captured data scope as well.
            MLClient(
                credential=mock_credential,
                subscription_id=workspace_sub,
                resource_group_name=workspace_rg,
                workspace_name=workspace_name,

sdk/ml/azure-ai-ml/tests/conftest.py:506

  • This E2E fixture also uses workspace_name + registry_reference, for which online operations were already given the workspace resource-group scope before this PR. It therefore does not exercise the reported wrong-resource-group path (registry_name + workspace_reference) through the real online calls. Build the client with the affected argument combination so this test regresses if workspace scope isolation is removed.
            subscription_id=e2e_ws_scope.subscription_id,
            resource_group_name=e2e_ws_scope.resource_group_name,
            workspace_name=e2e_ws_scope.workspace_name,
            registry_reference="sdkv2-testFeed",

Copilot AI review requested due to automatic review settings September 7, 2026 04:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The documented client construction is rejected by the API, and the claimed data-operation registry scope is neither implemented nor tested.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 9/9 changed files
  • Comments generated: 2
  • Review effort level: Balanced

online_deployment_scope = captured["OnlineDeploymentOperations"]["scope"]
online_endpoint_client = captured["OnlineEndpointOperations"]["args"][2]
online_deployment_client = captured["OnlineDeploymentOperations"]["args"][2]
model_scope = captured["ModelOperations"]["scope"]
Comment on lines +284 to +285
if registry_name or registry_reference
else self._operation_scope

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants