Azure ML workspace stuck in soft-deleted state: create blocked, permanently-delete cannot find workspace

[vairakkumaarsvs]

Describe the bug

Azure CLI Version:
2.88.0

ML Extension:
2.44.1

Subscription:

Resource Group:
stmlwus2ws01-rg

Workspace:
stmlwus2ws01

Problem:

I deleted an Azure ML workspace.

When I attempt to recreate the workspace, deployment fails with:

"Soft-deleted workspace exists. Please purge or recover it."

However, attempting to permanently delete the workspace fails because the workspace cannot be found:

az ml workspace delete
--name myws
--resource-group my-rg
--subscription
--permanently-delete
--yes

Error:

(UserError) Unable to find workspace:
/subscriptions/.../resourcegroups/my-rg/providers/Microsoft.MachineLearningServices/workspaces/myws

This creates a deadlock:

• Workspace creation fails because a soft-deleted workspace exists.
• Permanent delete fails because the workspace cannot be found.
• No CLI command appears available to enumerate or purge the soft-deleted workspace.

Expected behavior:

Either:

1. az ml workspace delete --permanently-delete should purge the soft-deleted workspace, or
2. The CLI should provide a supported command to list and purge soft-deleted workspaces.

Actual behavior:

Workspace name remains blocked but cannot be purged.

Related command

az ml workspace delete
--name myws
--resource-group my-rg
--subscription
--permanently-delete
--yes

Errors

(UserError) Unable to find workspace:
Code: UserError
Message: Unable to find workspace:

Issue script & Debug output

cli: Received HttpResponseError: Traceback (most recent call last):
File "/home/myubuntuuser/.azure/cliextensions/ml/azext_mlv2/manual/custom/workspace.py", line 278, in ml_workspace_delete
del_result = ml_client.workspaces.begin_delete(
name=name, delete_dependent_resources=all_resources, permanently_delete=permanently_delete
)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_telemetry/activity.py", line 288, in wrapper
return f(*args, **kwargs)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations.py", line 355, in begin_delete
return super().begin_delete(
~~~~~~~~~~~~~~~~~~~~^
name, delete_dependent_resources=delete_dependent_resources, permanently_delete=permanently_delete, **kwargs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations_base.py", line 494, in begin_delete
workspace: Any = self.get(name, **kwargs)
~~~~~~~~^^^^^^^^^^^^^^^^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_telemetry/activity.py", line 288, in wrapper
return f(*args, **kwargs)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations.py", line 143, in get
return super().get(workspace_name=name, **kwargs)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations_base.py", line 91, in get
obj = self._operation.get(resource_group, workspace_name)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_restclient/v2024_10_01_preview_tsp/operations/_operations.py", line 10422, in get
map_error(status_code=response.status_code, response=response, error_map=error_map)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.14/site-packages/azure/core/exceptions.py", line 163, in map_error
raise error
azure.core.exceptions.ResourceNotFoundError: (ResourceNotFound) The Resource 'Microsoft.MachineLearningServices/workspaces/mymlws' under resource group 'mymlws-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
Code: ResourceNotFound
Message: The Resource 'Microsoft.MachineLearningServices/workspaces/mymlws' under resource group 'mymlws-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix

cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "/opt/az/lib/python3.14/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/init.py", line 677, in execute
raise ex
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/init.py", line 820, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/init.py", line 789, in _run_job
result = cmd_copy(params)
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/init.py", line 335, in call
return self.handler(*args, **kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/command_operation.py", line 120, in handler
return op(**command_args)
File "/home/myubuntuuser/.azure/cliextensions/ml/azext_mlv2/manual/custom/workspace.py", line 285, in ml_workspace_delete
log_and_raise_error(err, debug)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File "/home/myubuntuuser/.azure/cliextensions/ml/azext_mlv2/manual/custom/raise_error.py", line 193, in log_and_raise_error
raise cli_error
File "/home/myubuntuuser/.azure/cliextensions/ml/azext_mlv2/manual/custom/workspace.py", line 278, in ml_workspace_delete
del_result = ml_client.workspaces.begin_delete(
name=name, delete_dependent_resources=all_resources, permanently_delete=permanently_delete
)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_telemetry/activity.py", line 288, in wrapper
return f(*args, **kwargs)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations.py", line 355, in begin_delete
return super().begin_delete(
~~~~~~~~~~~~~~~~~~~~^
name, delete_dependent_resources=delete_dependent_resources, permanently_delete=permanently_delete, **kwargs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations_base.py", line 494, in begin_delete
workspace: Any = self.get(name, **kwargs)
~~~~~~~~^^^^^^^^^^^^^^^^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_telemetry/activity.py", line 288, in wrapper
return f(*args, **kwargs)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations.py", line 143, in get
return super().get(workspace_name=name, **kwargs)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations_base.py", line 91, in get
obj = self._operation.get(resource_group, workspace_name)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_restclient/v2024_10_01_preview_tsp/operations/_operations.py", line 10422, in get
map_error(status_code=response.status_code, response=response, error_map=error_map)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.14/site-packages/azure/core/exceptions.py", line 163, in map_error
raise error
azure.core.exceptions.ResourceNotFoundError: (ResourceNotFound) The Resource 'Microsoft.MachineLearningServices/workspaces/mymlws' under resource group 'mymlws-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
Code: ResourceNotFound
Message: The Resource 'Microsoft.MachineLearningServices/workspaces/mymlws' under resource group 'mymlws-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix

cli.azure.cli.core.azclierror: (ResourceNotFound) The Resource 'Microsoft.MachineLearningServices/workspaces/mymlws' under resource group 'mymlws-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
Code: ResourceNotFound
Message: The Resource 'Microsoft.MachineLearningServices/workspaces/mymlws' under resource group 'mymlws-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
az_command_data_logger: (ResourceNotFound) The Resource 'Microsoft.MachineLearningServices/workspaces/mymlws' under resource group 'mymlws-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
Code: ResourceNotFound
Message: The Resource 'Microsoft.MachineLearningServices/workspaces/mymlws' under resource group 'mymlws-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7e58e1b0c3b0>]
az_command_data_logger: exit code: 3
cli.main: Command ran in 1.545 seconds (init: 0.049, invoke: 1.497)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 4607 in cache file under /home/myubuntuuser/.azure/telemetry/20260720191821574
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/opt/az/bin/python3 /opt/az/lib/python3.14/site-packages/azure/cli/telemetry/init.py /home/myubuntuuser/.azure /home/myubuntuuser/.azure/telemetry/20260720191821574"
telemetry.process: Return from creating process 1362084
telemetry.main: Finish creating telemetry upload process.

Expected behavior

purged the soft-deleted workspacee

Environment Summary

{
"azure-cli": "2.88.0",
"azure-cli-core": "2.88.0",
"azure-cli-telemetry": "1.1.0",
"extensions": {
"ml": "2.44.1"
}
}
which python3
/usr/bin/python3
Python 3.14.4

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[azure-client-tools-agent]

Routed to Azure/azure-cli-extensions as #10126 (Azure/azure-cli-extensions#10126) because this targets the machinelearningservices extension, whose source lives in Azure/azure-cli-extensions.

---

Posted by agent-assist (autonomous bug-fix pipeline).

[moozechen]

This issue appears to stem from Azure CLI’s az ml workspace delete not fully recognizing soft-deleted workspaces, which blocks creation while preventing permanent deletion. A workaround is to use the Azure Resource Graph REST API to query and manually purge the workspace by name with 2023-04-01-preview API version, which supports soft-delete management. To unblock creation, you can also try creating the workspace with a different name or using the Azure Portal’s “Recover” option if available, though permanent purge via portal or CLI --permanently-delete should eventually resolve the stale state after 7–30 days depending on regional soft-delete retention.

[saisharan0103]

The repro details point to a split between create and delete lookup behavior: create sees the soft-deleted stmlwus2ws01, while az ml workspace delete --permanently-delete resolves only the active workspace path and returns Unable to find workspace. workspace.py:ml_workspace_delete is the part I would inspect first, specifically whether the permanently-delete path can address the soft-deleted resource or needs a separate list/purge operation.

[vairakkumaarsvs]

This issue appears to stem from Azure CLI’s az ml workspace delete not fully recognizing soft-deleted workspaces, which blocks creation while preventing permanent deletion. A workaround is to use the Azure Resource Graph REST API to query and manually purge the workspace by name with 2023-04-01-preview API version, which supports soft-delete management. To unblock creation, you can also try creating the workspace with a different name or using the Azure Portal’s “Recover” option if available, though permanent purge via portal or CLI --permanently-delete should eventually resolve the stale state after 7–30 days depending on regional soft-delete retention.

Do you mean to say the 2023-04-01-preview` API version will allow to purge-delete before the 7-30 day window to purge-delete?