Bicep Template – Azure Data Factory with KeyVault, ADLS and SQLDB Linked Services

A common pattern I have needed to use recently is to move data from Azure Blob Storage (ADLS Gen2 in this case) to SQL so it can be modelled by people with existing SQL skills.

The full bicep template can be found here.

The pattern pretty much has the same setup

• A – The data is sat in an existing ADLS Gen 2 account
• B – The serving layer will be an Azure SQL Database
• A Data Factory pipeline will be used to move data from A to B
• Items such as connection strings and DFS uris are held in a Keyvault for portability across environments.

I’ve recently been experimenting with Bicep templates to ensure a repeatable deployment process, not only across applications, but also across development environments.

One challenge I faced was deploying a set of basic linked services along with the ADF in order to tie the components together via a pipeline. The end result needed to be

This Bicep template described below can be used as a module for repeatability and has the following pertinent sections

Data Factory Deployment

Fairly straightforward, this declares a datafactory resource

resource dataFactory 'Microsoft.DataFactory/factories@2018-06-01' = {
  name: name
  location: location
  tags: resourceTags
  identity: {
    type: 'SystemAssigned'
  }
}

Linked Service for Keyvault

Declares a linked service for a keyvault.

resource linkedServiceKeyVault 'Microsoft.DataFactory/factories/linkedservices@2018-06-01' = {
  name: 'LS_KeyVault'
  parent: dataFactory
  properties: {
    description: 'Linked service relating to the key vault for this application'
    parameters: {}
    type: 'AzureKeyVault'
    typeProperties: {
      baseUrl: linkedServiceKeyVaultURL
    }
  }
}

Linked Service for ADLS

In this case the URI is passed as a parameter (linkedServiceADLSUri).

resource linkedServiceADLS 'Microsoft.DataFactory/factories/linkedservices@2018-06-01' = {
  name: 'LS_PrimaryADLS'
  parent: dataFactory
  properties: {
    description: 'Linked service for the primary ADLS account'
    parameters: {}
    type: 'AzureBlobFS'
    typeProperties: {
      azureCloudType: 'AzurePublic'
      url: linkedServiceADLSUri
    }
  }
}

Linked Service for Azure SQL DB

In this case the connection string is a reference to a secret in a keyvault.

resource linkedServicePrimarySQL 'Microsoft.DataFactory/factories/linkedservices@2018-06-01' = {
  name: 'LS_PrimarySQL'
  parent: dataFactory
  properties: {
    description: 'Linked service for the primary AzureSQL Database'
    parameters: {}
    type: 'AzureSqlDatabase'
    typeProperties: {
      connectionString: {
        type: 'AzureKeyVaultSecret'
        store: {
          referenceName: linkedServiceKeyVault.name
          type: 'LinkedServiceReference'
        }
        secretName: 'primary-sql-connectionstring'
      }
    }
  }
}

Points to note

• In my scenario, the ADLS storage account and SQL server are already deployed.
• There are no cross-region resources or explicit security requirements to handle.
• This is about as bog standard as its gets.

Useful Resources

This has been a worthwhile learning experience and in future will no doubt evolve to become an even bigger time saver, here are the resources I used to go from zero knowledge of bicep to a working template.

Getting Started with Bicep Ebook by Freek Berson	https://www.amazon.com/Getting-started-Bicep-Infrastructure-Azure/dp/B098WK3MR7
Microsofts Bicep documentation	Bicep documentation | Microsoft Learn