Hello fellow Data Enthusiasts and welcome to my Blog,
if you’ve missed my Blogpost about the Azure Basics please check the Hub and Spoke Part of it before you go on in this post. Make sure you have a managed identity, which has acess to the encryption key vault in the Azure Landing Zone.
Networking
If you want to setup a new Databricks environment I highly recommend to do so in a new Spoke VNet which has at least the size /24 and a peering to the Hub Net. Split the VNet into at least 2*/26 Subnets. This will allow you to create 59 cluster nodes. In case you need more nodes check this page for the networking requirements. Here an example of and subnet configuration:
Deligate the 2 /26 subnets to Databricks:
Create a NSG and associate it with the two /26 Subnets
Add these Rules to the NSG
Storage Account
Create Storage, with private access only. Don’t create private Enpoints here, since Microsoft will then add a random values at the end of the private Endpoint name.
Instead, create the private Endpoints afterwards and I recommend the following naming pattern pe0<storage account name>0<endpoint> and to put them into a separate ressource group.
Set the Storage to use the encryption key from the key vault of the Azure Landing Zone
Databricks
For all security features, you’ll need a Premium workspace. For the managed ressource group I suggest to use mrg0<databricksworkspace name>.
In the Networking tab, you get to define the address spaces and names for Spoke VNet 1 & 2. I suggest refraining from creating private endpoints within the initial setup again, because here, you’ll get again a random string at the end of the name.
The encryption of databricks works a bit different than the one of the storage account. Here the setup requests to enter a key identifier. Someone with access to your encryption key vault could provide this String, which can be found within the key
So this encryption tab should look like this (You could enable to automatically rotate the key).
Note: Make sure the ‘AzureDatabricks’ Service Principal has acess to the Key Vault Keys. You’ll find this Service Pricipal only in Key Vault ressources.
Now create the private endpoints for the databricks workspace, with your naming pattern.
Note: You only need one browser_authentication endpoint for the whole network. So make sure, this private Endpoint is either in the Hub or in your production environment.
Now setup the private DNS Zones in your Hub and then everything is done.
I hope this little guide helps you in your further projects.
