Overview
SQL Server 2017 introduced the concept of scale out clusters, where a master node can orchestrate the execution of packages on several worker nodes.
Explanation
What is a Scale Out Cluster?
The SSIS Scale Out cluster resembles the distributed computing concepts that are nowadays typically found in Big Data processing frameworks such as Spark or Synapse Dedicated SQL Pools. There’s a master node that accepts work (executing SSIS packages) and it orchestrates the execution of those packages amongst the worker nodes.
The architecture looks like this:
The diagram depicts 3 workers, but it is possible to add or drop workers.
With a Scale Out cluster, you can scale SSIS horizontally instead of only vertically. Scaling vertically means that if you want to get more work done with SSIS, you install a better and faster (and more expensive) server. By scaling horizontally, you can use multiple cheaper servers to get to the same goal. Keep in mind though that it is possible to install all nodes on the same machine if you desire so.
Configuring a Scale Out Cluster
The detailed set up of a SSIS Scale Out Cluster is out of scope for this tutorial. However, the following tips are excellent guidelines to get you started:
