By: Akhil Mahajan | Comments | Related: > Azure
Problem
In a previous tip, Securely Manage Secrets in Azure Databricks Using Databricks-Backed, we looked at how to secure credentials that can be used by many users connecting to many different data sources. In this tip we look at how we can secure secrets with Azure Databricks using Azure Key Vault-backed scoped secrets.Solution
To manage credentials Azure Databricks offers Secret Management. Secret Management allows users to share credentials in a secure mechanism. Currently Azure Databricks offers two types of Secret Scopes:
- Azure Key Vault-backed: To reference secrets stored in an Azure Key Vault, you can create a secret scope backed by Azure Key Vault. Azure Key Vault-backed secrets are only supported for Azure Databricks Premium Plan.
- Databricks-backed: A Databricks-backed scope is stored in (backed by) an Azure Databricks database. You create a Databricks-backed secret scope using the Databricks CLI (version 0.7.1 and above).
In this tip we will learn about creating Azure Key Vault-backed secret scopes. To learn about Databricks-backed secret scopes, check my previous article for more information.
Requirements
Following are the requirements for setting up Databricks-Backed secret scope and secret(s):
- An Azure Subscription
- An Azure Key Vault
- An Azure Databricks workspace
- An Azure Databricks Cluster (Runtime 4.0 or above)
Creating Azure Key Vault
Open a Web Browser. I am using Chrome.
Enter the URL https://portal.azure.com and hit enter.
![Web Browser](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.001.png)
Sign in into your Azure Account.
![Azure Portal - Login Information](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.002.png)
After successfully logging to Azure Portal, you should see the following screen.
![Azure Portal - Home Page](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.003.png)
Click on "All Services" on the top left corner.
![Azure Portal - All Services](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.004.png)
Search for "Azure Key Vault" in the "All Services" search text box.
![Azure Portal - Search for Azure Key Vault service](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.005.png)
Click on "Key vaults". It will open the blade for "Key vaults".
![Azure Portal - Azure Key Vault Service View](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.006.png)
Click on "Add". It will open a new blade for creating a key vault "Create key vault".
![Azure Portal - Create Azure Key Vault](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.007.png)
Enter all the information and click the "Create" button. Once the resource is created, refresh the screen and it will show the new "key vault" which we created.
![Azure Portal - Azure Key Vault Service View with the newly created Azure Key Vault service](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.008.png)
Click on the "key vault" name.
![Azure Portal - Azure Key Vault Overview Page](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.009.png)
Scroll down and click on the "Properties".
![Azure Portal - Azure Key Vault Menu](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.010.png)
Save the following information for the "key vault" created. We would be using these properties when we connect to the "key Vault" from "databricks"
- DNS Name
- Resource ID
![Azure Portal - Azure Key Vault Properties (DNS Name and Resource ID)](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.011.png)
Creating Secret in Azure Key Vault
Click on "Secrets" on the left-hand side.
![Azure Portal - Azure Key Vault Menu](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.012.png)
Click on "Generate/Import". We will be creating a secret for the "access key" for the "Azure Blob Storage".
![Azure Portal - Azure Key Vault Generate/Import View](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.013.png)
Enter the required information for creating the "secret".
![Azure Portal - Azure Key Vault Secret creation view](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.014.png)
After entering all the information click on the "Create" button.
![Azure Portal - Azure Key Vault Generate/Import View](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.015.png)
Note down the "Name" of the secret.
Creating Azure Key Vault Secret Scope in Databricks
Open a Web Browser. I am using Chrome.
Enter the URL https://portal.azure.com and hit enter.
![Web Browser](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.016.png)
Sign in into your Azure Account.
![Azure Portal - Login Information](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.002.png)
Open the Azure Databricks workspace created as part of the Azure Databricks Workspace mentioned in the Requirements section at the top of the article.
![Azure Databricks - Workspace
](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.017.png)
Click on Launch Workspace to open Azure Databricks.
![Azure Databricks - Home Page](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.018.png)
Copy the "URL" from the browser window.
![Azure Databricks - Home Page](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.019.png)
Build the "URL" for creating the secret scope. https://<Databricks_url>#secrets/createScope.
![Azure Databricks - Creating the Azure Key Vault backed secret scope.](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.020.png)
Enter all the required information:
- Scope Name.
- DNS Name (this is the "DNS name" which we saved when we created the "Azure Key Vault").
- Resource ID (this is the "Resource ID" which we saved when we created the "Azure Key Vault").
![Azure Databricks - Creating the Azure Key Vault backed secret scope.](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.021.png)
Click the "Create" button.
"Databricks" is now connected with "Azure Key Vault".
Using Azure Key Vault Secret Scope and Secret in Azure Databricks Notebook
Open a Web Browser. I am using Chrome.
Enter the URL https://portal.azure.com and hit enter.
![Web Browser](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.001.png)
Sign in into your Azure Account.
![Azure Portal - Login Information](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.002.png)
Open the Azure Databricks workspace created as part of the "Azure Databricks Workspace" mentioned in the Requirements section at the top of the article.
![Azure Databricks - Workspace](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.022.png)
Click on "Launch Workspace" to open the "Azure Databricks".
![Azure Databricks - Home page](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.023.png)
In the left pane, click Workspace. From the Workspace drop-down, click Create, and then click Notebook.
In the Create Notebook dialog box, enter a name, select Python as the language.
![Azure Databricks - Create a Python Notebook](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.024.png)
Enter the following code in the Notebook
dbutils.secrets.get(scope = "azurekeyvault_secret_scope", key = "BlobStorageAccessKey") #azurekeyvault_secret_scope --> Azure Key Vault based scope which we created in Databricks #BlobStorageAccessKey --> Secret name which we created in Azure Key Vault
![command line](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.025.png)
When you run the above command, it should show [REDACTED] which confirms that the secret was used from the Azure Key Vault secrets.
![command line](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.026.png)
In the same notebook we are going to add another command section and use Scala as the language.
%scala val blob_storage_account_access_key = dbutils.secrets.get(scope = "azurekeyvault_secret_scope", key = "BlobStorageAccessKey") //azurekeyvault_secret_scope --> Azure Key Vault based scope which we created in Databricks //BlobStorageAccessKey --> Secret name which we created in Azure Key Vault
![command line](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.027.png)
When you run the above command, it should show [REDACTED] which confirms that the secret was used from the Azure Key Vault secrets.
![command line](/tipimages2/6094_managing-secrets-in-azure-databricks-using-azure-key-vault.028.png)
References
- https://docs.microsoft.com/en-us/azure/azure-databricks/what-is-azure-databricks
- https://azure.microsoft.com/en-us/pricing/details/key-vault/
- https://docs.microsoft.com/en-us/azure/azure-databricks/what-is-azure-databricks
- https://docs.azuredatabricks.net/user-guide/secrets/index.html#secrets-user-guide
- https://docs.azuredatabricks.net/user-guide/secrets/secret-scopes.html
- https://docs.azuredatabricks.net/user-guide/secrets/secret-scopes.html#create-an-azure-key-vault-backed-secret-scope
Next Steps
- In this article, we learned how to create Azure Key Vault secret scope.
- Using Azure Key Vault secrets to connect to Azure SQL DB from Databricks.
- Check out the previous tip Securely Manage Secrets in Azure Databricks Using Databricks-Backed
About the author
![MSSQLTips author Akhil Mahajan](/images/Akhil-Mahajan.png)
This author pledges the content of this article is based on professional experience and not AI generated.
View all my tips