Setup Azure Data Factory Environment


If you want to follow along with the examples in this tutorial, you'll need to install ADF in your Azure tenant. Either make sure you have the appropriate permissions to do so, or you can opt for a free trial. We cover how to setup Azure Data Factory, a Storage Account and Azure SQL Database.

Setup Azure Data Factory

When logged in the Azure Portal, click on "Create a resource" at the top of your screen.

create a resource in portal

Search for "data factory" in the marketplace and choose the result from Microsoft.

choose data factory from results

On the next page, you'll get an overview of the product. Click on create to get started with configuring your ADF environment. You need to select a subscription. You can either create a new resource group (which is a logical container for your resources) or select an existing one. You need to select a region (take one close by your location to minimize latency) and choose a name. Finally, you need to select a version. It's highly recommended you choose V2. Version 1 of ADF is almost never used and practically all documentation you'll find online is about V2.

config ADF

Click on Review + create at the bottom. It's possible you might get a validation error about the Git configuration. Integration with Git and Azure Devops is out of scope for this tutorial.

git validation error

If you get the error, go to the Git configuration tab and select Configure Git later.

configure git later

When the validation passes, click on Create to have Azure create the ADF environment for you. This might take a couple of minutes. When the resource is deployed, you can check it out in the portal.

ADF in the portal

Typically, you don't spend a lot of time here. You can configure access control to give people permission to develop in ADF, or you can set up monitoring and alerting. The actual development itself is done in Azure Data Factory Studio, which is a separate environment. Click on the Studio icon to go to the development environment, which should open in a new browser tab.

ADF Studio welcome page

Setup Storage Account

Before we can start creating pipelines in ADF, we need to set up our source and destination (called sink in ADF). We begin by creating a storage account in the Azure Portal. Search for the "storage account" resource in the marketplace and click on Create.

create storage account marketplace

In the Basics tab, choose your subscription and the same resource group as the ADF environment. Specify a name for the storage account and choose the same region as your ADF.

configure storage basics

For the redundancy, choose "Locally-redundant storage (LRS)", which is the cheapest option. Go to the Advanced tab and switch the access tier to Cool. This is a cheaper option than the default Hot access tier.

storage account advanced config

Click on Review + Create and then Create to provision your storage account. When it has been deployed, go to the resource and then to Containers in the Data Storage section.

create new container in blob

Specify "data-input" as the new container name and then click on Create.

specify container name

Setup Azure SQL Database

Next, we need our destination, which is going to be an Azure SQL database. Search for "SQL database" in the marketplace and click on Create.

add azure sql db

In the Basics tab, choose your subscription and the same resource group as before. Give the database the name "tutorial".

azure sql db basics tab

Before we can create the database however, we need to assign it to a "SQL Server". This is not an actual SQL Server, but rather a logical container for our databases. Some configurations are applied to the server level. Since we do not have a server yet, we need to create it first. Click on Create new to create one.

Specify a name for the server. This will be the server name you'll enter in a database tool like SQL Server Management Studio or Azure Data Studio to connect to your database. I choose "adf-tutorial-sql", but if you want to add other databases later on that have nothing to do with this tutorial, you might want to choose another name.

create sql server

If you choose SQL authentication, you need to specify a login name for the server admin and a strong password. If you choose Azure Active Directory authentication, you'll need to specify an Azure AD admin. You can choose either one of the authentication methods, or both. If you use Azure AD, click on Set admin. Search for an AD user you want to grant admin rights.

add ad user as admin

Once you have selected your admin, click on OK at the bottom to finish the configuration of the server. Back in the configuration of the database, set the redundancy to "locally-redundant backup storage".

database config continued

The default configuration of the database is a bit too pricy for our tutorial, so let's set this to a cheaper option. Click on Configure database to see the various options.

default database config

Switch to DTU-based purchasing and choose the Basic workload. You can see the price has considerably dropped!

dtu set to basic

The downside is we can have only 2GB for our database, but that should be plenty for this tutorial. Just one more setting before we can create our database. In the Additional settings tab, choose Sample as the data source. This will install the AdventureWorksLT sample database.

config sample db

Click on Review + create and then on create to create the SQL Server and the Azure SQL database. This might take a couple of minutes. Once the deployment is done, go the SQL Server and then to Firewalls and virtual networks, which can be found in the Security section.

firewall config

To make sure we can access our database from our machine, we need to add our current IP address to the firewall. At the top, click on Add client IP. This will add a new rule to the firewall. Don't forget to click Save at the top!

add firewall rule

While we're in the firewall config, let's set the property "Allow Azure services and resources to access this server" to Yes. This will make our lives a lot easier when we try to connect to the server from ADF.

allow azure services in firewall

In the Overview pane, you can find the name of the server. Hover over it with your mouse and click the copy icon to copy the name to your clipboard. Start SQL Server Management Studio (SSMS) or Azure Data Studio to connect to the server. For the remainder of the tutorial, SSMS is used. In SSMS, create a new connection. Paste the server name and choose the authentication method you configured earlier. If you're using Azure AD, don't choose Windows Authentication but rather one of the Azure AD authentication methods listed: Universal with MFA, Password or Integrated. The correct one depends on your environment.

connect to azure sql

Don't click on Connect just yet! First, go to options and enter the database name in the upper text box.

specify db name in conn

If you don't do this, SSMS will automatically try to connect to the master database, which might or might not work, depending on your permissions. You can now click on Connect. Once you're connected, you can view the tables that were automatically created for us because we chose the sample database:

ssms connected to azure sql db, showing tables
Additional Information

Comments For This Article

Saturday, May 14, 2022 - 1:19:25 PM - Hussain Back To Top (90086)
Thanks for providing azure knowledge, please share all concepts related to adf.thanks in advance.

get free sql tips
agree to terms