Azure Databricks Secret Scope from Azure Key Vault— REST API throws 401 Unauthorized Error

I had a requirement to add secret scope from Azure Key Vault in Azure Databricks and there are 2 options to implement the requirement

1. Terraform — use the terraform module available here -https://registry.terraform.io/providers/databricks/databricks/latest/docs/resources/secret_scope
2. Databricks REST API — use the REST API available here — https://docs.databricks.com/api/azure/workspace/secrets/createscope

I developed an Azure DevOps pipeline and used Databricks REST APIs to authenticate and create secret scope.

For authentication, I used the Databricks Service Principal (SPN) credentials to get a bearer and management token which I can use in my REST API request. The SPN has the Admin permission on the Databricks Workspace.

However, when I tried to run the pipeline, it failed with below error-

requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://adb-xxx.0.azuredatabricks.net/api/2.0/secrets/scopes/create

Surprised!!! SPN has admin access on Databricks but still it is failing with 401 error

I drilled down further and checked SPN permission on Azure Key Vault. I tried with different roles — Key Vault Secrets User, Key Vault Secrets Officer, Key Vault Administrator. No luck!!!

Then I provided “Contributor” role on Azure Key Vault and the REST API call worked. It created a secret scope from Azure Key Vault in my Databricks Workspace.

Hurray!!