Generative AI with Azure Machine Learning

By:   |   Updated: 2024-01-29   |   Comments   |   Related: More > Artificial Intelligence


Azure Machine Learning (AML) is a platform used by several organizations and their Data Science teams worldwide for many years. It has provided Data Scientists with an interactive UI and code-driven experience for creating ML Models and orchestrating MLOps processes with Continuous Integration / Continuous Deployments (CI/CD) and Azure DevOps. Recently, several cloud platform leaders, such as Databricks, Synapse, Fabric, and Snowflake, are beginning to integrate the capability to easily develop Generative AI solutions through their platform through agnostic open-source libraries and out-of-the-box ML runtime compute, LLMOps, and more. AML has also entered this space with its new features of the Gen AI Model Catalog and Prompt Flow. What are some of the features and capabilities of AML's Gen AI Model Catalog and Prompt Flow, and how can development teams and Data Scientists quickly start building Gen AI solutions in AML?


With the Model Catalog in AML, developers can access a wide range of LLM models from Azure OpenAI, Meta, Hugging Face, and more. With Prompt Flow, developers can leverage the power of these LLMs by quickly getting started with pre-built workflows and evaluation metrics that can be integrated with LLMOps capabilities for deployment, tracking, model monitoring, and more. This tip will explore the Model Catalog and Prompt Flow within AML.

Model Catalog

Model Catalog is a new tab within AML. It provides a user interface (UI) to help you select the right model for your use case. You can search for the model directly or filter by inference and fine-tune tasks to have AML suggest a suitable model.

ModelCatalog Illustration of the model catalog options

When you select a model, you can further test sample inference queries and results. Also, you have access to batch and real-time inference samples in Python notebooks and CLIs with YAML to enable your continued experimentation. You also have similar samples for model evaluation. Notice that you can get an idea of model performance based on the VM SKU selected for your compute resource.

ModelDetails Details of available samples and more in the model UI

When ready to deploy the model, the scoring script and environment will be auto-generated. You will need to specify your VM type, instance count, and endpoint name. Notice that a new endpoint URL will be created after creating the endpoint. This is an excellent way of quickly deploying out-of-the-box LLMs through an easy-to-use UI.

ModelDeployment Steps to deploy model

Prompt Flow

Prompt Flow is a development tool that simplifies the creation, debugging, and deployment of AI applications powered by LLMs. It allows users to create executable flows linking prompts and Python tools and provides a visualized graph for easy navigation. It also supports large-scale testing for prompt performance evaluation. A new flow can be created as a standard, chat, or evaluation flow. Flows can also be created from a gallery and include web classifications, along with "bring your own data Chat Q and A," among other flows.

NewFlow New Prompt Flow gallery

You can also build evaluation flows by cloning a flow template from the gallery.

Evaluation Evaluation Gallery of prompt flows

To get started, select the flow template from the gallery and clone it.

CloneFlow Step to clone flow

You will have instant access to the customizable flow template, displayed in a visual format to the right of the flow canvas.

FlowDetails1 visual depiction of flow details.

To the left of the flow canvas, you will see the various detailed stages of the entire flow. Since this template is customizable, you can easily add more LLMs, Prompts, Python code, embeddings, Vector DB lookups, and more. You can also move around the flow components and customize them to your needs. Even if you are new to Gen AI development, it is easy to comprehend the flow details here and change blob path parameters to customize the flow to your environment and data.

FlowStudio Options for customizing the flow

Here is some Python code to generate the prompt context that comes with the template. Again, you can easily customize this to your needs.

FlowStudioPython Options for customizing the flow with python

Here are more details that come with the flow template, which can be customized and validated in real-time.

FlowStudioPrompt Options for customizing the flow with prompt templates

You must create a compute instance to deploy and run this prompt flow. A compute instance is a fully managed cloud-based workstation optimized for your machine learning development environment. This can be created in the UI by defining a compute name and choosing between CPU or GPU VM types. CPU and GPU VM types differ in their processing capabilities. CPU VMs are general-purpose and handle sequential tasks efficiently, while GPU VMs, with their parallel processing capabilities, are ideal for compute-intensive tasks like deep fine tuning. The choice between the two depends on the specific needs of your LLM Gen AI application.

ComputeInstance Steps to create new compute

The Advanced Settings allow you to enable idle shutdown after a specific time of inactivity. You could also schedule start and shutdown, enable SSH and virtual network access, assign to another user, provision with setup script, and download a template for automation. These capabilities are quite impressive and easy to configure with the code-free UI.

ComputeInstance2 Steps to create new compute and configure settings

After creating the compute instance, you need to create a runtime and link it to the compute instance from the previous step. You could create a custom environment or use the default environment. Prompt Flow's predefined runtime environment in AML is a Docker image equipped with built-in tools for flow execution. It provides a convenient starting point and is regularly updated to align with the latest Docker image version. You can customize this environment by adding preferred packages or configurations.

Runtime Steps to create new runtime

With the compute instance and runtime created, you can now deploy the flow to an endpoint by specifying a new or existing endpoint name, authentication types, and identity types. Key-based authentication uses a non-expiring key for access, while token-based authentication uses a token that expires and needs to be refreshed or regenerated. System-assigned identities are created and managed by Azure and tied to the lifecycle of the service instance. User-assigned identities are created by the user and can be assigned to multiple instances.

Endpoint Steps to create new endpoint and authentication

To take it a step further, you could create a new vector index in AML Prompt Flow using a similar UI. A vector index stores numerical representations of concepts for understanding relationships. The available vector store options are the Azure Cognitive Search Index and the Faiss Index. Faiss is an open-source library that provides a local file-based store ideal for development and testing. Azure Cognitive Search is an Azure resource. It supports information retrieval over vector and textual data stored in search indexes and meets enterprise-level requirements.

VectorIndex Steps to create new vector index

Note that Prompt flow development is not limited to the AML Studio. With the prompt flow for VS Code extension available on the Visual Studio Marketplace, you can easily develop prompt flows in the same format but within your familiar VS Code environment.

VSCodeExt Step to install VS code extension for prompt flow.


This tip demonstrated how to begin developing Generative AI LLM apps with Prompt Flow in Azure Machine Learning. There are other Data Science and ML Platforms for Gen AI development. However, this is an excellent platform if you are looking for a code-free, UI-driven process to customize and deploy pre-built templates or learn how to get started with Gen AI development quickly and easily. Furthermore, as you refine your skills, you can easily leverage VS Code with the prompt flow extensions to create more customized and code-driven Gen AI applications. As cloud AI/ML platforms mature, you could also explore working with prompt flow in other platforms, like Databricks, Synapse, Fabric, and more.

Next Steps

sql server categories

sql server webinars

subscribe to mssqltips

sql server tutorials

sql server white papers

next tip

About the author
MSSQLTips author Ron L'Esteve Ron L'Esteve is a trusted information technology thought leader and professional Author residing in Illinois. He brings over 20 years of IT experience and is well-known for his impactful books and article publications on Data & AI Architecture, Engineering, and Cloud Leadership. Ron completed his Masterís in Business Administration and Finance from Loyola University in Chicago. Ron brings deep tec

This author pledges the content of this article is based on professional experience and not AI generated.

View all my tips

Article Last Updated: 2024-01-29

Comments For This Article

get free sql tips
agree to terms