What is Eventstream in Microsoft Fabric? - Part 1

By:   |   Updated: 2024-02-27   |   Comments (2)   |   Related: 1 | 2 | > Microsoft Fabric


We have data from a couple of IoT devices streaming into Azure Event Hubs. We would like to analyze this data in Microsoft Fabric, preferably using a Lakehouse table. Is there an easy way to get the data from the Event Hub into a Fabric Lakehouse?


Microsoft Fabric is the new centralized cloud data platform. You can find an introduction in this tip: What is Microsoft Fabric? There are several types of compute services available, each with different purposes. There are options available to handle real-time data, such as the KQL databases (Fabric version of Azure Data Explorer databases). You can learn more about them in the tip: What are Kusto Query Language (KQL) databases in Microsoft Fabric? Another option is Eventstreams, which offers a centralized platform in Fabric to ingest, transform, and route real-time data to certain destinations. Like most services in Fabric, this is done in a low-to-no-code SaaS fashion.

evenstream example

In Part 1, we'll introduce the concepts of Eventstream with the use case of vending machines sending data to Azure Event Hubs. We'll focus on the configuration of Azure Event Hubs and the source in the Eventstream. In Part 2, we'll concentrate on the PowerShell scripts to generate the sample data and the destination configuration in Eventstream.

To follow along with this tip, you will need both Azure and Fabric, which offer free trials.

Reading Data from Event Hubs into Eventstream

Luckily, we don't need to start from scratch. Fellow MSSQLTips.com author John Miner wrote a tip, Azure Event Hub Service Telemetry Example with PowerShell, where he colorfully explains the history of vending machines, how he created his sample data, and shares a PowerShell script to generate some data to send to Azure Event Hubs. We'll reuse his scripts for our use case here.

Before we start with PowerShell, we need to configure Azure Event Hubs and Microsoft Fabric first.

Setting Up Azure Event Hubs

For an introduction to Azure Event Hubs, check out the tip: The tale of two Azure Hubs - IoT Hub and Event Hub.

In the Azure portal, click on "Create a resource."

create new azure resource

In the Marketplace, search for "event hubs." Click on Create to create a new Event Hubs namespace.

event hubs in the market place

In the configuration pane, choose a subscription, a resource group, a region, and a name. The name needs to be globally unique.

event hubs namespace configuration

For the pricing tier, choose Standard. The basic tier cannot be selected currently as it doesn't support the creation of extra consumer groups (relevancy to be explained later). All the other settings can be left to the defaults.

Click on Review + create to go to the validation page. When the validation succeeds, hit Create to provision the new Azure Event Hubs namespace. This might take a couple of minutes.

When the deployment is finished, go to the new resource. Go to the Event Hubs pane to create a new Event Hub instance.

create new event hub

In the basics pane, give the new Event Hub a name. This time, it doesn't need to be globally unique.

event hub configuration

I also changed the retention time from 1 hour to 4 hours. (This might be overkill, but at least the data can survive a lunch break.) Leave everything else to the defaults. Hit Review + create to go to validation and click Create once validation finishes.

event hub validation

The deployment of the new Event Hub should be quite fast.

new event hub created

Go to the new Event Hub and navigate to Consumer groups.

event hub landing page

Create a new consumer group called Eventstream.

create new consumer group

Next, we need to ensure that the PowerShell script can send events to the Hub and that the Fabric Eventstream can read them. To do this, we'll create a Shared Access policy, which allows us to generate keys. You can find the Shared access policies in the Settings section.

create new shared access policy

Click on Addto create a new policy. Give the policy a name and check the Manage option.

policy config

When the policy is created, you can click on it to see the generated keys.

generated keys for the policy

Setting Up an Eventstream in Microsoft Fabric

Once our Azure Event Hub configuration is complete, we can focus our efforts on Microsoft Fabric. Make sure you have a Fabric-enabled workspace (either by having Power BI Premium or creating an Azure Fabric capacity).

In the workspace, create a lakehouse. This will serve as the destination of the Eventstream. You can learn more about Fabric lakehouses and how to create one in the tip, What are Lakehouses in Microsoft Fabric?

In Microsoft Fabric, switch to the real-time analytics persona.

go to real-time analytics persona

In the New section, click on Eventstream.

create new eventstream in Fabric

Give the Eventstream a name.

name the new evenstream

A canvas will open with a new Eventstream object without sources or destinations.

new evenstream on design canvas

Click on New source and select Azure Event Hubs from the list.

choose event hubs as the source

You can also choose to use some sample data from Fabric itself, an Azure IoT Hub, or a Custom App. With a custom app, you can create your own logic that sends event data to the Eventstream. When you choose a Custom App, you will be provided with some sample code to get you started (for some reason, only Java is available currently) for either Event Hub, AMQP, or Kafka:

custom app source with java sample code

When you choose Azure Event Hub, a pane will open on the right side of your screen where you will need to enter the connection information:

azure event hub source configuration pane

Give the source a name and create a new connection. You need to specify the Event Hub namespace (globally unique), the Event Hub name, the shared access policy name, and the key of the specified policy.

create new connection to azure event hub

Don't forget to change the auto-generated name to something more user-friendly ("vendingmachines_eventhub" is used in the screenshot above). If you don't and you have multiple connections to Azure Event Hub, you might end up with something like this:

A screenshot of a computer

Description automatically generated

It's impossible to tell which connection is which from the dropdown. This is a known bug, and Microsoft is working on it. For now, you can edit existing connections in Microsoft Fabric through the Manage connections and gateways option in the Settings menu.

manage connections and gateways in settings

In the list of connections, find your EventHub connection and click on the ellipsis to go to its settings menu.

edit an existing connection

There you can change its name.

change the name of an existing connection

Once you configured the connection, you need to select a consumer group. Pick the one we created, as the default $Default consumer group is currently not supported (Microsoft is working on this):

cannot use default consumer group

Because of this, we need to create another consumer group, which is not possible in the basic tier of Azure Event Hubs. In the end, your configuration should look like this:

evenstream event hub source final config

Click on Add to add the Event Hubs source to the canvas. It may take some time to start streaming.

configured azure event hubs source in eventstream

The data preview will be empty since no data is being streamed yet. Before we configure our destination, it would be helpful to see data coming into the Eventstream, which will be the focus of Part 2 of this tip.

Next Steps

sql server categories

sql server webinars

subscribe to mssqltips

sql server tutorials

sql server white papers

next tip

About the author
MSSQLTips author Koen Verbeeck Koen Verbeeck is a seasoned business intelligence consultant at AE. He has over a decade of experience with the Microsoft Data Platform in numerous industries. He holds several certifications and is a prolific writer contributing content about SSIS, ADF, SSAS, SSRS, MDS, Power BI, Snowflake and Azure services. He has spoken at PASS, SQLBits, dataMinds Connect and delivers webinars on MSSQLTips.com. Koen has been awarded the Microsoft MVP data platform award for many years.

This author pledges the content of this article is based on professional experience and not AI generated.

View all my tips

Article Last Updated: 2024-02-27

Comments For This Article

Thursday, March 7, 2024 - 2:50:59 AM - Koen Verbeeck Back To Top (92047)
Hi Scott,
as explained in the article, I use a shared access policy to access the event hub consumer group.
In the connection dialog in Eventstream, I use the shared access key to make the connection.


Wednesday, March 6, 2024 - 10:51:34 AM - Scott Back To Top (92046)
Koen, thanks for the post. We are running into an issue using the same setup with a Power BI Gateway connection. The connection shows successful and appears to test connectivity; however when we try to create the event stream connection in Fabric we get the error below and are unable to see any options for Consumer Group. The SAS policy name and primary key are correct, but we can't get past the "authentication is invalid" error. We have also tried to create a connection from KQL to Event Hubs and don't receive any error, but the NEXT button is disabled when trying to select a Consumer Group. I was wondering if there are additional permissions we have to grant to the connection like a Fabric service principle or something similar?

Invalid cloud connection configuration: type is EventHub, host is valid, authentication is invalid

get free sql tips
agree to terms