Problem
Without proper Microsoft Fabric capacity planning, you may experience several ramifications within your Fabric platform. This includes overpaying for unused capacity or exceeding capacity limits quickly, resulting in emergency unbudgeted increases in capacity cost.
If you are on a Power BI Premium P-SKU, you will need to either migrate or re-point to an F-SKU managed in the Azure Portal before Microsoft retires the P SKU. You must understand the capabilities of the F-SKU vs P-SKU. For example, the F-SKU will no longer have access to auto-scale, meaning the capacity planning initiative becomes even more critical. Also, you need at least an F-64 SKU to enjoy AI skills and Copilot within the workspace.
Without a thorough understanding of these capabilities and planning approaches, you could negatively impact your business and the consumers of Fabric items.
Solution
Microsoft Fabric capacity is the allocated computing resources for the workspace’s performance and scalability. Here are some tips for effectively planning and managing your Fabric Capacities. These will impact your budgeted cost, uptime, and performance.
Regularly Monitor Fabric Capacity Metrics App
If your organization has an existing Fabric environment, consider delegating an administrator to regularly monitor and set alerts. They can use the Fabric Capacity Metrics PBI app. This will give you valuable insights into the current environments and capacity usage. Find more details about this app and alerts, from: Microsoft Fabric Monitoring with Metrics App and Data Activator.
This app can help to visually understand trends around your capacity usage over a period. For example, the blue shade in the following diagram indicates scheduled dataflow and pipeline refreshes. The red reflects interactive compute. This could be Excel downloads of reports, ad-hoc analysis and reports on existing datasets, or manual report refreshes and consumption. Start by drilling into red spikes that approach 100%. You can identify trends around reports that require performance optimizations or complete rewrites based on inefficiencies. So, let’s say your organization is on a P3 premium capacity SKU. If most of the red spikes are related to an inefficient report and dataset, once the report is rewritten and optimized, resulting in an elimination of red spikes, you may be able to drop your capacity SKU from a P3 (256CU) to a P2 (128CU).
This capacity metrics app is also extremely useful when your organization is new to Fabric and is estimating capacity size. Start with a trial capacity and monitor utilization in the capacity metrics app. You can then scale up capacity based on utilization, ensuring workloads are within capacity limits going forward.
Purchase Reserved Instances to Benefit from Optimal Cost Savings
There is approximately a 41% cost savings per month if you purchase a reservation for your Fabric SKU. This is typically a 1-year commitment. The image below shows some of the costs associated with SKUs and differences between pay-as-you-go versus reservations. Note: With Fabric capacities, auto scale is no longer available. It was originally available with the premium P SKUs until Microsoft decided to retire it.
Benefits associated with your Fabric capacity include:
- Bursting allows workloads to temporarily use more resources to handle sudden demand spikes.
- Smoothing spreads compute usage evaluation over time to ensure jobs run smoothly without needing more capacity.
- Throttling limits operations when capacity is over-utilized to maintain stability.
- Capacity buy-back lets you exchange or refund unused capacity reservations to optimize costs.
You can learn more about smoothing and throttling functionality here.
Assign Capacity Based on Usage Expectations
Creating a trial capacity and testing usage patterns via the capacity metrics app may take some time. However, you can accumulate insightful usage data. If you need to plan a more extensive enterprise-level migration from a PaaS, IaaS cloud, or on-premises SQL Server or virtual machine (VM) to Fabric, consider the following table as a reference to determine how much Fabric Capacity Units your organization might need.
Column heading definitions:
- 30-Second CU Use: The number of capacity units (CUs) consumed over a 30-second period for each SKU. This metric helps quantify the compute resources used in a short, consistent time frame. This allows for easier comparison and understanding of resource consumption for different SKUs.
- On-Prem Compute Equivalent: A rough comparison between the CUs available in Microsoft Fabric and traditional on-premises VM configurations. This helps understand the compute power needed to run similar workloads in an on-premises environment. It should be easier to select the right Microsoft Fabric SKU based on the estimated workload requirements and available on-premises resources.
- Data Volume: This indicates the total size of datasets. This is the amount of data being processed, stored, and managed within Microsoft Fabric. It helps gauge the scale of your workloads. Ensuring the selection of the appropriate SKU to handle the expected data loads. This could include the total size of datasets, the volume of data being ingested, and the amount of data processed. This could be from various workloads like ETL pipelines, Power BI report refreshes, and Spark compute jobs. Understanding data volume helps to select the appropriate capacity SKU to ensure optimal performance and resource allocation.
- Example Compute Workloads: This provides specific, detailed examples of how different SKUs can be used for various data processing tasks. This includes ETL pipelines, Power BI report refreshes, and Spark compute jobs. These examples help understand the types of workloads each SKU can handle and choose the right capacity based on the organization’s needs.
Understand Premium Capabilities Based on SKU
Microsoft recently announced that it will be retiring the Power BI Premium P SKU. As such, customers need to migrate to the F SKU. Note: The F SKU will no longer have auto scale capabilities and rely solely on bursting and smoothing. Furthermore, certain Fabric features, shown below, will only be available in the F64 SKU or higher. This is important to plan around. Developers may need access to these AI Skills, Copilot, or other features in their workspaces. However, they may not need the compute power of an F64. Also note that when moving from a P SKU to an F SKU in the same region, it will be designated a simple re-point. However, when moving from a P SKU to an F SKU in different regions, this will be considered a migration. This may involve additional complexities.
Python Code to List Workspaces and Premium Capacities
The following Python code can be run in a Fabric notebook using Fabric semantic link. This code lists all workspaces and premium capacities in tabular form.
import pandas as pd
import sempy.fabric as fabric
# List workspaces and capacities
workspaces = fabric.list_workspaces()
capacities = fabric.list_capacities()
# Convert to pandas DataFrames
workspaces_df = pd.DataFrame(workspaces)
capacities_df = pd.DataFrame(capacities)
# Join workspaces with capacities on the capacity ID
workspaces_with_capacity = workspaces_df.merge(capacities_df, left_on='Capacity Id', right_on='Id', suffixes=('_workspace', '_capacity'))
# Filter the workspaces that are on dedicated capacity
workspaces_on_dedicated_capacity = workspaces_with_capacity.query('`Is On Dedicated Capacity` == True')
# Select relevant columns
result = workspaces_on_dedicated_capacity[['Name', 'Display Name', 'Capacity Id', 'Sku', 'Region', 'State']].rename(columns={'Name': 'Workspace Name', 'Display Name': 'Capacity Name'})
# Display the result in a tabular format
display(result)The first seven columns in the chart below are the result of this Python code. Additionally, you can include the Environment, Department, and F-SKU columns to plan which F-SKU to assign to the workspace. For example, all Prod workspaces originally on a P3 (F256) but were only utilizing 50% of the capacity could be assigned an F-128. And the remaining Dev workspaces could be assigned F-8 or F-16 SKUs.
Once you have identified which capacities should be assigned to each workspace, consider purchasing the capacities in the Azure portal with reserved instances. They will immediately show up in the Fabric Admin portal. Then, click on the capacity and assign multiple workspaces to it. The new capacities will also become available in the Fabric capacity metrics app.
There are several other Fabric Capacity features (shown below) to familiarize yourself with to administer and manage this capacity.
Next Steps
