Problem
For various reasons, your data infrastructure may require two or more Azure Data Factory (ADF) or Synapse instances that need to communicate using different trigger schedules. For example, you are using ADF and Synapse Analytics together because Synapse does not support all available features of ADF, or you are using two instances of Synapse Analytics for high availability. This tip will help you design an event-based trigger initiated in one instance of ADF/Synapse and consumed by another instance of ADF/Synapse.
Solution
The solution is to use the Event Grid Topic resource in Azure to manage events created using POST Web request in ADF, which in turn are consumed by a Custom Trigger of Azure Synapse Analytics via an event subscription, as shown in the illustration below:
Figure 1: Architecture Drawing
Azure Resources
For this demo, you will need the following Azure resources:
- Azure Data Factory
- Azure Synapse Analytics
- Event Grid Topic
Please follow the hyperlinks below to create ADF and Synapse. In the next tip, we will cover how to create a new Event Grid Topic for managing events.
Create New Event Grid Topic
In the Azure marketplace, locate Event Grid Topic.
Figure 2: Event Grid Topic on Azure Marketplace
Click Create. The following window will appear, starting on the Basics tab.
Figure 3: Event Grid Basic Settings
Next, click on the Networking tab. For this demonstration, I will set up public network access. If you prefer, you can set up a private network using subnets.
Figure 4: Event Grid Topic Public Access
We will set up Firewall rules later to allow network traffic from ADF and Synapse.
Click on the Security tab. Both Managed Identity and Shared Access Signature will be used for this demonstration. Please enable both.
Figure 5: Azure Event Grid Topic Security
Now, on the Advanced tab, use “Event Grid Schema” for Event Schema. For disaster recovery, choose “Regional” for demo purposes, but for production, please consider “Cross-Geo.”
Figure 6: Advanced Setting of Event Grid Topic
Add any tags you wish to add and create the resource. Once the resource is created, proceed to ADF to create a pipeline with POST request.
Create New Pipeline in ADF to Send Events to Event Grid Topic
Navigate to your ADF and select Create new pipeline. Add Web activity located under Activities/General/Web:
Figure 7: ADF Web Activity named “POST to Event Grid.”
This web activity will post a request to Event Grid. Before it can do that, it needs to be configured correctly. Click Web Activity to configure the General settings.
Figure 8: Web Activity General Settings
Here you can define the name of your activity and specify properties like Timeout, Retry, Retry interval, etc.
Furthermore, for correct functionality, the Settings tab must be fully configured. Below is an illustration of a configured POST request.
Figure 9: Settings of Web Activity
URL. This is the URL of the Event Grid Topic to which you would like to post your data. It can be found by going to your Event Grid Topic resource overview page, property named “Topic Endpoint,” as shown below:
Figure 10: URL == Topic Endpoint
Method. This is the REST API method. Select “POST” from the drop-down menu.
Body. The json format body sent to Event Grid shown in the code example below:
[{
"id": "1",
"eventType": "ADF.Completed",
"subject": "StartCustomTrigger",
"eventTime": "@{utcNow()}",
"data": "",
"dataVersion": 1,
"metadataVersion": 1
}]
This code example can be extended with additional properties. Additional help can be found in this Microsoft article: Publish events to Azure Event Grid custom topics using access keys.
Authentication. Set to “None,” as we will use SAS-KEY to authenticate. When writing this article, Managed Identity authentication is still not supported by Event Grid Topic accepting new events.
Headers. Add two new properties named Content-Type and aeg-sas-key.. Content-Type value should be “application/json”. For aeg-sas-key, copy/paste one of your Event Grid Topic access keys that can be found by going to your Event Grid Topic and selecting Settings/Access keys as shown below:
Figure 11: Event Grid Topic Access Keys
Advanced. Leave the advanced settings as-is.
For more information on Web Activity, check out the following article: Web activity in Azure Data Factory and Azure Synapse Analytics.
Let’s save and execute the pipeline to test our new POST request. I ran my pipeline in debug mode several times to generate some events in Event Grid Topic. An example of successful pipeline execution is below:
Figure 12: Successful Web Activity Execution
Create New Custom Trigger in Synapse Analytics
For Event Grid Topic events to be correctly matched, we need to create a trigger in Synapse Analytics with a valid subscription in Event Grid to consume events created by ADF.
Open Synapse Analytics and navigate to Manage/Triggers, and select Create New Trigger with the properties shown below:
Figure 13: Azure Synapse Analytics: Create new Custom Events trigger.
For trigger type, select “Custom events” and then choose your Azure Subscription where the desired Event Grid Topic is located, followed by the event grid topic.
You have an option in “Subject filters” to filter events by prefix or suffix. For this demo, I will filter all events but the one we will be sending from our ADF with subject starting with “StartCustomTrigger”. Same as the subject value in the body of the ADF POST request.
For “Event types,” add a new property value called “ADF.Completed”, which is the same as the eventType in the body of our POST request.
Create Trigger. You can only activate a trigger if you have a pipeline associated with it. I created a dummy pipeline for this demo to activate a custom trigger. Use the following tip to help create with ADF or Synapse Analytics pipeline: Create Azure Data Factory Pipeline.
For the trigger to consume events from Event Grid Topic, you need to start the trigger and publish it in Synapse Analytics. Only then will a valid event subscription be created in Event Grid Topic.
Figure 14: Event Subscription created for Custom Trigger
Monitoring Events in Event Grid Topic
Event Grid Topic offers a good interface to monitor all incoming and consumed events. It will also show if events have failed to submit, for example, due to security or configuration issues.
I have already generated some events in ADF before activating the trigger, and these events will be marked as unmatched.
I have sent a POST request from ADF with the wrongly configured body. These will be marked as failed events.
Run your ADF POST pipeline a few times to generate matched events consumed by the Synapse trigger subscription.
Open your Event Grid Topic, and on the overview page, you will find metrics and subscription information as shown below:
Figure 15: Metrics and Subscription information of Event Grid Topic
First and foremost, under “Event Subscriptions,” you will find a subscription created for our custom trigger in Azure Synapse Analytics displaying the prefix filter and event types.
In the diagram above, you will find an overview of all events for the selected period. This is a very useful overview as it shows:
- Published Events: number of events sent by ADF POST request, events that have successfully reached Event Grid.
- Publish Failed: number of events that failed to reach Event Grid from ADF due to misconfiguration or other issues.
- Matched Events: received events that are matched to event subscription.
- Delivered Events: received events successfully delivered to the subscriber.
- Unmatched Events: received events that are unmatched to event subscriptions.
Ideally, published events should equal delivered events with no publish errors. Under normal operation, any deviation from that means that not all POST events from ADF are matched to the Synapse trigger.
Summary
When you need synchronized execution between Azure Data Factory and Synapse Analytics or two ADF, Synapse instances, Event Grid Topic fills the gap for messaging service between the pipeline, sending the POST request and receiving instance custom trigger. It also provides monitoring capabilities to review published and matched events and can further be extended with Azure Monitor and email notifications in case not all events reach their destinations.
Next Steps
