Problem
We have data from a couple of IoT devices streaming into Azure Event Hubs. We would like to analyze this data in Microsoft Fabric, preferably using a Lakehouse table. Is there an easy way to get the data from the Event Hub into a Fabric Lakehouse?
Solution
Microsoft Fabric is the new centralized cloud data platform. You can find an introduction in this tip: What is Microsoft Fabric? There are several types of compute services available, each with different purposes. There are options available to handle real-time data, such as the KQL databases (Fabric version of Azure Data Explorer databases). You can learn more about them in the tip: What are Kusto Query Language (KQL) databases in Microsoft Fabric? Another option is Eventstreams, which offers a centralized platform in Fabric to ingest, transform, and route real-time data to certain destinations. Like most services in Fabric, this is done in a low-to-no-code SaaS fashion.
In Part 1, we’ll introduce the concepts of Eventstream with the use case of vending machines sending data to Azure Event Hubs. We’ll focus on the configuration of Azure Event Hubs and the source in the Eventstream. In Part 2, we’ll concentrate on the PowerShell scripts to generate the sample data and the destination configuration in Eventstream.
To follow along with this tip, you will need both Azure and Fabric, which offer free trials.
Reading Data from Event Hubs into Eventstream
Luckily, we don’t need to start from scratch. Fellow MSSQLTips.com author John Miner wrote a tip, Azure Event Hub Service Telemetry Example with PowerShell, where he colorfully explains the history of vending machines, how he created his sample data, and shares a PowerShell script to generate some data to send to Azure Event Hubs. We’ll reuse his scripts for our use case here.
Before we start with PowerShell, we need to configure Azure Event Hubs and Microsoft Fabric first.
Setting Up Azure Event Hubs
For an introduction to Azure Event Hubs, check out the tip: The tale of two Azure Hubs – IoT Hub and Event Hub.
In the Azure portal, click on “Create a resource.”
In the Marketplace, search for “event hubs.” Click on Create to create a new Event Hubs namespace.
In the configuration pane, choose a subscription, a resource group, a region, and a name. The name needs to be globally unique.
For the pricing tier, choose Standard. The basic tier cannot be selected currently as it doesn’t support the creation of extra consumer groups (relevancy to be explained later). All the other settings can be left to the defaults.
Click on Review + create to go to the validation page. When the validation succeeds, hit Create to provision the new Azure Event Hubs namespace. This might take a couple of minutes.
When the deployment is finished, go to the new resource. Go to the Event Hubs pane to create a new Event Hub instance.
In the basics pane, give the new Event Hub a name. This time, it doesn’t need to be globally unique.
I also changed the retention time from 1 hour to 4 hours. (This might be overkill, but at least the data can survive a lunch break.) Leave everything else to the defaults. Hit Review + create to go to validation and click Create once validation finishes.
The deployment of the new Event Hub should be quite fast.
Go to the new Event Hub and navigate to Consumer groups.
Create a new consumer group called Eventstream.
Next, we need to ensure that the PowerShell script can send events to the Hub and that the Fabric Eventstream can read them. To do this, we’ll create a Shared Access policy, which allows us to generate keys. You can find the Shared access policies in the Settings section.
Click on Addto create a new policy. Give the policy a name and check the Manage option.
When the policy is created, you can click on it to see the generated keys.
Setting Up an Eventstream in Microsoft Fabric
Once our Azure Event Hub configuration is complete, we can focus our efforts on Microsoft Fabric. Make sure you have a Fabric-enabled workspace (either by having Power BI Premium or creating an Azure Fabric capacity).
In the workspace, create a lakehouse. This will serve as the destination of the Eventstream. You can learn more about Fabric lakehouses and how to create one in the tip, What are Lakehouses in Microsoft Fabric?
In Microsoft Fabric, switch to the real-time analytics persona.
In the New section, click on Eventstream.
Give the Eventstream a name.
A canvas will open with a new Eventstream object without sources or destinations.
Click on New source and select Azure Event Hubs from the list.
You can also choose to use some sample data from Fabric itself, an Azure IoT Hub, or a Custom App. With a custom app, you can create your own logic that sends event data to the Eventstream. When you choose a Custom App, you will be provided with some sample code to get you started (for some reason, only Java is available currently) for either Event Hub, AMQP, or Kafka:
When you choose Azure Event Hub, a pane will open on the right side of your screen where you will need to enter the connection information:
Give the source a name and create a new connection. You need to specify the Event Hub namespace (globally unique), the Event Hub name, the shared access policy name, and the key of the specified policy.
Don’t forget to change the auto-generated name to something more user-friendly (“vendingmachines_eventhub” is used in the screenshot above). If you don’t and you have multiple connections to Azure Event Hub, you might end up with something like this:
It’s impossible to tell which connection is which from the dropdown. This is a known bug, and Microsoft is working on it. For now, you can edit existing connections in Microsoft Fabric through the Manage connections and gateways option in the Settings menu.
In the list of connections, find your EventHub connection and click on the ellipsis to go to its settings menu.
There you can change its name.
Once you configured the connection, you need to select a consumer group. Pick the one we created, as the default $Default consumer group is currently not supported (Microsoft is working on this):
Because of this, we need to create another consumer group, which is not possible in the basic tier of Azure Event Hubs. In the end, your configuration should look like this:
Click on Add to add the Event Hubs source to the canvas. It may take some time to start streaming.
The data preview will be empty since no data is being streamed yet. Before we configure our destination, it would be helpful to see data coming into the Eventstream, which will be the focus of Part 2 of this tip.
Next Steps
- Try it out yourself! You can get free Azure and Microsoft Fabric trials to get you started.
- More information about Azure Event Hubs:
- The official documentation for Eventstream can be found here.
- Check out this overview for all Microsoft Fabric-related tips.
