By: Koen Verbeeck | Updated: 2024-06-17 | Comments | Related: > Microsoft Fabric
Problem
Microsoft Build (MS Build) is a global developer conference where you can learn about various announcements and product updates concerning Azure, AI, data, .NET, and many other technologies. The conference was hosted from May 21-23, 2024 in Seattle, Washington. The event is a huge source of news concentrated on all the different Microsoft products. In this tip, we'll focus on announcements regarding the Microsoft Fabric data platform.
Solution
Microsoft Fabric is a centralized SaaS analytics platform, which was announced at the previous MS Build conference in 2023, that became generally available in November 2023 (see the tip Microsoft Fabric Announcements at Microsoft Ignite 2023 for more info) and has a monthly cadence of releasing new features and updates. Many sessions from Build 2024 were recorded and can be viewed on-demand after registration. Make sure to check out the session Microsoft Fabric: What's new and what's next by Microsoft's Amir Netz, Arun Ulagaratchagan, and many others (such as a demo by Patrick from Guy in a Cube).
Build Announcements for Microsoft Fabric
Below are various announcements for Microsoft Fabric.
Real-Time Intelligence
During MS Build, it became clear Microsoft has been investing heavily in providing a robust and intelligent end-to-end analytics platform for streaming data. The goal is to make data analytics on real-time data easier and more accessible for business intelligence. The focus on real-time intelligence begins with the introduction of Real-Time Intelligence, a new workload (previously called a persona) in Fabric. This workload replaces the Real-Time Analytics workload and includes the Data Activator workload (still available as a separate choice).
Switching to this new experience will shortcut you to the available task for dealing with streaming event data:
As part of the increased efforts for real-time data, the following features were announced as well:
- Real-Time hub. This hub is a central location for your data-in-motion. What OneLake is for data in rest, the Real-Time hub is for streaming data. You can use it as a single place to ingest, process, and route events.
Source: Microsoft Fabric Blog
- Event Streams got some updates as well, with Enhanced Event Stream for more cross-cloud sources, the Edit and Live modes, and Default and Derived Streams.
- The real-time dashboards from Azure Data Explorer are now available in Fabric, giving you an alternative for Power BI reports. This dashboard visualizes data from your KQL databases. Learn more about them: Visualize data with Real-Time Dashboards, Create a Real-Time Dashboard (preview), and Visualize and Explore Data with Real-Time Dashboards.
Source: Microsoft Fabric Blog
And there's more! Check out the official blog post announcement, the documentation, or the May 2024 Fabric update page.
To understand these updates, check out the MS Build session Reimagine Real-Time Intelligence with Microsoft Fabric.
Fabric Workload Development Kit
With this new feature, partners (Independent Software Vendors or ISVs) can develop a custom workload right in the Fabric service. Workloads are the different types of compute supported by Fabric. The native workloads are, for example, the warehouse (T-SQL), the lakehouse (Spark), KQL database, event streams, data activator, and Power BI. With the workload development kit, you can develop your own workload that will run inside the Fabric user interface. At Build, some partnerships were announced and demonstrated during several sessions, such as creating a graph database in Neo4J or a decision tree in SAS.
Source: Microsoft Fabric Blog
With all those new workloads, potential use cases in Fabric will grow exponentially. For more information: Official announcement and Fabric workload development kit. You can watch this feature in action in this MS Build recording.
GraphQL API
Not to be mistaken with Microsoft's GraphAPI, GraphQL is a query language for your API. It's a specification, not a technology or a specific implementation. At Build, GraphQL support for Fabric data sources was announced, and it is now in preview. With a few clicks, you can create a GraphQL layer on top of your data to automatically create endpoints to read or update each table. It can also work with parameterized stored procedures.
Source: Microsoft Fabric Blog
Using the GraphQL API will make integration with other apps easier. GraphQL can be more efficient than a REST API since you can specify your own query (and select only columns that you actually need, for example). If you're interested in learning more about this feature, check out the recording or the official documentation. You can also learn how to set up REST and GraphQL API endpoints for your Azure Databases with the Data API Builder tool in the tip Build REST API with Azure Data API Builder (DAB).
AI and Copilot
Nowadays, you can't watch a vendor keynote without hearing the words AI and Copilot at least a dozen times. At MS Build, there was no exception. Microsoft continues to invest in Copilot and AI integration. Copilot in Fabric is now generally available (from F64 and up), and it is also enabled by default for eligible tenants for most workload experiences, such as data factory, data engineering, data warehouse, data science, and real-time intelligence. Copilot for Power BI will roll out soon, and Copilot for KQL is on the roadmap.
AI Skills is also introduced, which is an evolution of the Q&A feature that already exists in Power BI. You can learn more about Copilot here or in the official announcement blog post.
Snowflake Apache Iceberg Compatibility and Direct Integration
Microsoft expands its partnership with Snowflake to make the data integration between the two platforms more seamless. Two major announcements are the result of this partnership:
- OneLake will support the Apache Iceberg format, an open table format for large analytic datasets. Iceberg can be regarded as a "competitor" for the delta/parquet format. This makes OneLake a more open lake and takes the OneCopy experience a step further. You will be able to create shortcuts to Iceberg data (previously, shortcuts only supported the delta format).
- There will also be a direct integration between Fabric and Snowflake. You will be able to create an Iceberg database in Snowflake and choose OneLake as the external storage layer. This will be visible in Fabric, and changes made in Snowflake will be available in Fabric and vice versa. This will be available in preview later this year (2024).
More information can be found in the official blog post. You can also view a demo of the upcoming features in this recording.
Data Factory Announcements
There are several interesting announcements made for the Data Factory workload in Fabric which is the next generation of Azure Data Factory:
- The introduction of Data workflows, which are managed Airflow environments. Airflow is an open-source, Python-based data orchestration tool. Data workflows will allow you to develop, schedule, and monitor directed acyclic graphs (DAGs). You can learn more about this feature: Introducing Data workflows in Microsoft Fabric.
- Blob Storage Event triggers are added for data pipelines. You can create a trigger on a pipeline that is activated each time a file arrives or is deleted from a blob container. Behind the scenes, Reflex items (from Data Activator) and Event Streams are used to make this happen. More info can be found here.
- There are two new activities: the HDInsight activity and the Spark Job Definition activity.
- You can now use Trusted Workspace Access in your pipelines to access firewall-enabled Azure Data Lake Storage accounts. This feature uses the workspace identity, which currently can only be created for workspaces with a Fabric capacity of F64 or higher.
- There are even more announcements for Data Factory; check out this overview to learn more.
Task Flows
Tasks Flows are a new feature that allow you to create an architectural diagram of your data solution in Fabric. There are pre-defined templates, such as a medaillon architecture, a lambda architecture, or an event analytics platform.
Choosing a template will create a task flow for you. For each block, you can add a new Fabric item.
For example, if you click on batch data storage, the task flow will recommend that you create a lakehouse in your workspace:
You can also make your own task flow. A task flow is a visual guide of your data solution and shows the dependencies between the various items in your workspace. It's an interesting documentation tool for your Fabric data platform.
More information can be found in the official blog post.
Data Functions
The final announcement for this tip is the new User Data Functions. These Functions are reusable pieces of code that can be created in Fabric (using either C# or Python) and used in other Fabric workloads. This sounds a lot like Azure Functions – that's because it is (it even has the same icon in Fabric Pipelines). In a Function, you can connect to data stored in a warehouse, lakehouse, or mirrored database. The code for the Function can be edited in Visual Studio code using a new Fabric extension.
Source: Microsoft Fabric Blog
At the time of writing, this feature is in private preview.
Next Steps
- You can find most announcements in the blog post, Unlock real-time insights with AI-powered analytics in Microsoft Fabric. Most new features are demonstrated during the various sessions at MS Build, for which recordings are available.
- Keep an eye on this website, as tips will be published that explain new features in more detail to help you get started!
- All Microsoft Fabric tips and tutorials are available in this overview.
About the author
Koen Verbeeck is a seasoned business intelligence consultant at AE. He has over a decade of experience with the Microsoft Data Platform in numerous industries. He holds several certifications and is a prolific writer contributing content about SSIS, ADF, SSAS, SSRS, MDS, Power BI, Snowflake and Azure services. He has spoken at PASS, SQLBits, dataMinds Connect and delivers webinars on MSSQLTips.com. Koen has been awarded the Microsoft MVP data platform award for many years.This author pledges the content of this article is based on professional experience and not AI generated.
View all my tips
Article Last Updated: 2024-06-17
