264
Problem
I need a structured Python reference guide to help me get started or expand my Python programming skills with a focus on the data domain. I want to bookmark a resource to use daily when needed as I build various programs, automations, data processing pipelines, and AI solutions. Can you please compile some common and helpful resources as a quick Python Reference guide?
Solution
The goal of this tip is to create a living document to help data professionals (engineers, analysts, scientists), etc., daily when working with various aspects of the Python programming ecosystem. Feel free to bookmark this article and suggest code to include in this tip in the comments section below.
Installing and Using Python
| Topic | Description | Resources |
|---|---|---|
| Why Use Python | How and where you can benefit from learning Python. | Python for Data Professionals |
| Python Environment | Get started from scratch with Python using Jupyter Notebooks with VS Code or Anaconda. | Python and Jupyter Notebooks for Data Analytics and Data Science |
| Setting up Python with Anaconda, VS Code, Power BI, SQL Server | ||
| Anaconda Python Installation and Package Management for Projects | ||
| Learning Python in Visual Studio 2019 | ||
| Intro to Python – what is it and is it popular? Installation of Python – on Windows + integrated development environment | Python Programming Introduction – Install, Jupyter Notebooks, VS Code |
Python Data Types
| Topic | Description | Resources |
|---|---|---|
| Simple Data Types Overview | Overview of the built-in data types and data structures. Mastering these core programming elements and concepts will ensure a firm grip on the Python programming language. | Basic Built-in Python Data Types |
| How to use data types and variables in Python. Python supports many different data types, as well as the ability to use variables to make the code more dynamic. | Python Variables and Data Types | |
| Working with strings in Python programming. Available built-in functions. Understanding syntax. Get started as a beginner. | Python String Functions: Concatenate, Split, Replace, Upper, Lower | |
| Overview of Complex Data Types | The basic data structures of Python. | Python List, Tuple, Range, Dictionary and Set Data Type Examples |
| Understand the mechanism behind the Iterable, Iterator, Generator types. | Python Iterable, Iterator and Generator Examples | |
| Working with JSON files in Python. | Write and Read JSON Files with Python |
Control Flow
| Topic | Description | Resources |
|---|---|---|
| IF, ELIF, and ELSE | The control flow statements allow control over the code, decisions, and error handling to make the program dynamic and adaptable. | Python Control Flow – if, if else, nested if, for loop and while loop |
| To master control flow in Python, be familiar with the comparison rules and the if-else commands. | Python IF, ELIF and ELSE Control Flow Logic | |
| For, While, Nested Loops | How to repeat certain blocks of code and automate the control flow of your program. | Example Python Loops FOR, WHILE, Nested and more |
Programming Concepts
| Topic | Description | Resources |
|---|---|---|
| List Comprehensions | Comprehension is a Python-specific construct useful for applying expressions with or without conditions to the elements of a sequence, mapping, or any other iterable. | Learn Python List Comprehension for Lists, Tuples, Dictionaries, Sets |
| Exception Handling | Python code offers the try/except clause to design intelligent error handling that “catches” run-time errors. | Python Exception Handling with Try, Except, Else, Finally Clause |
| Recursion | Recursion is another form of repetition, next to for- and while-loops. A recursive definition means a description of something that refers to itself. | Recursion in Python Explanation and Code Samples |
| Operators and Keywords | The Python programming language has important “reserved” commands called keywords. Each one of them provides specific special instructions for the interpreter. Using the keywords comes with certain caveats, and it is handy to have a complete overview of their use. | Python Tutorial on Boolean, Conditional, Logical Operator, Membership |
| Python Keywords for Functions, Modules, Classes, Variables, Coroutines | ||
| Top-down Design | This approach is a problem-solving technique that systematically breaks a complicated problem into smaller, more manageable pieces. | Python Programming Tutorial with Top-Down Approach |
| Regular Expressions | We may not want to be regular expression gurus, but we need sufficient knowledge to tackle practical day-to-day pattern matching tasks. | Python Regex Explained with Examples |
Python for Data Extraction
| Topic | Description | Resources |
|---|---|---|
| Web Scraping | Web crawling is a technique used for traversing web applications automatically to search for patterns, e.g., hyperlinks. | Using Web Crawling in Python to Conduct a Website Content Audit |
| How to use the web scraping technique in the Python programming language to capture basic information on a web page. | Using Advanced Python Web Scraping to Conduct a Web Content Audit | |
| Build a Web Crawler in Python for a Website Content Audit | ||
| BeautifulSoup | BeautifulSoup is one of several screen scraping packages for web pages. The BeautifulSoup library can be imported into Python to programmatically find, navigate, and enumerate HTML elements on web pages. | Screen Scraping with Python and BeautifulSoup |
| Scrape Web Page and Store in SQL Server with Python and BeautifulSoup | ||
| BeautifulSoup, Python and T-SQL to Import Unstructured Web Content | ||
| In this tutorial, we will cover the basics of web scraping by providing a step-by-step tutorial using the Scrapy Python framework to collect and visualize data from the web. | Web Scraping with Python Scrapy Framework | |
| Pandas | In addition to learning how to collect open-high-low-close-volume (ohlcv) data from a start date through an end date, you can learn about collecting other stock data, such as stock dividends and splits as well as how to discover the sector and the industry for a stock. | Techniques for Collecting Stock Data with Python |
| Industrial Data Extraction | Getting started with MQTT and Python for industrial data extraction and storing in SQL Server | Coming Soon. |
Python for Data Transformation
| Topic | Description | Resources |
|---|---|---|
| ETL | Python libraries for developing data pipelines How can we best take advantage of them, considering some caveats? | Python Modules for Developing Data Engineering Workloads |
| Python for Data Engineering with chardet, io and logging modules | ||
| How to automate an ETL pipeline in a Python script with a batch file on a Windows server. | Provisioning with Azure CLI and Automating a Python ETL Pipeline | |
| Web scraping for retrieving web content, usually in the form of HTML, searching the web content for target information and extracting data. | Using Python to Download Data from an HTML Table to an SQL Server Database | |
| Numpy | Numpy is a must-know Python package for efficiently storing and handling data in memory. | Python Numpy for Multi-Dimensional Arrays |
| PySpark | How to perform an entire ETL process using PySpark and load it to SQL Server on a Windows virtual machine and automate using Windows Task Scheduler. | Automate ETL Processes with PySpark on a Windows Server |
| Pandas | How to perform a join operation on a key column or with condition. | Combine Data in Python using Pandas Merge, Pandas Join and Pandasql |
| Pandas is widely used open-source library for analyzing and manipulating data in the Python programming language. | Explore Pandas in Python to Analyze and Manipulate Tabular Data | |
| How to refactor prior code developed for several simpler solutions into a new integrated solution. | Data Engineering a Solution with Python and SQL Server | |
| Azure Data Factory | How to run Python script in Azure Data Factory for data transformation. | Coming Soon. |
Consuming APIs with Python
| Topic | Description | Resources |
|---|---|---|
| Web API Consumption and Data Ingestion to SQL Server | How to load data into SQL Server by using an API. | Flight Plan API with Python and SQL Server |
| Consume Multiple APIs to Load Data Asynchronously to SQL Server | ||
| Skype has an API to access its data. In this article, we look at how to use this API with Python and how to load the data to an SQL Server database. | Skype API and Python to Load Meeting Data into SQL Server Database | |
| Web API Consumption | Deploying and managing services using the Azure REST API. | Python to store data in JSON format and deploy in Azure with REST API |
| Use the Binance API to access a live stream of Crypto trades. | Binance API and Python with SQL Server for Data Analysis and Reporting | |
| Use the Twitter API to access a list of users, followers, and tweets. | Twitter API to Load Data to a SQL Server Database using Python Code | |
| CRUD Operations on a SharePoint List using Python. | SharePoint List CRUD Operations using Python | |
| Remote Data Access with JSON and Python Rest API. | Request and Process Data from a Restful API using Python and JSON |
Designing APIs with Python
| Topic | Description | Resources |
|---|---|---|
| Create First API using Python Flask | Applications can communicate through APIs, increasing your application’s ability to integrate with third-party applications and services. | Create a Simple REST API with SQL Server Python, Flask, and HTML |
| Generate Files with Python | How do you create your own PDF-generating cloud-based microservice application. | Generate PDF file using Azure Functions, Python Runtime and xhtml2pdf |
| How to generate dynamic QR codes in a PNG format that point to a specific record in your app. | QR Code Generator with Power Apps, Azure Function and Python Runtime | |
| Azure Functions | Getting started with Azure Functions with Python Runtime. | Coming Soon. |
| AWS Lambda | Getting started with AWS Lambda with Python Runtime. | Coming Soon. |
Python and SQL
| Topic | Description | Resources |
|---|---|---|
| Connect to SQL Server | How to create a connection string to a Microsoft SQL Server database for Python, along with creating some simple database objects. | Python Connect to SQL Server with Code Examples |
| Scripts execution | SQL Server Machine Learning Service for developing and deploying solutions using Python. | Deep dive into Python scripts execution in SQL Server |
| DML | Connecting to a database and extracting data with Python scripts. | Examine pyodbc an Open-Source Module to Access to ODBC Databases |
| Read SQL Server Data into a Dataframe using Python and Pandas | ||
| CRUD Operations in SQL Server using Python | ||
| Options to export large datasets (multimillion rows, e.g., time series) to one or multiple CSV files | Export Large SQL Query Result to Text File with Python | |
| How to load Excel data into SQL Server using Python. | Import from Excel to SQL Server using Python | |
| How to load data into a SQL Server database using Python and an API and build a simple Power BI report using this imported data. | Load SQL Server Data using Python and an API with Power BI Reporting | |
| DDL | Except SSMS, is there another way to interact with databases programmatically? | SQLAlchemy vs SQL Server Management Studio for Database Tasks |
| SQLAlchemy | Create database objects such as tables programmatically? How do you do so without issuing long string-literal SQL commands from your Python code? | Create SQL Server Objects with SQLAlchemy Core for Python |
| If you develop applications that rely on multiple SQL or other databases on different hosts, you might find it challenging to integrate all the data sources seamlessly, which means having a uniform flow of data to and from your application. | SQLAlchemy vs SQL Server Management Studio for Database Tasks | |
| Elasticsearch | Python support was introduced in SQL Server 2017 and opens a new perspective on querying remote databases. Using the sp_execute_external_script procedure allows us to query any database that has a Python library. | Discover how SQL Server can use Python to access any NoSQL engine |
| DuckDB | What if you were handed a huge data file and needed to check it out quickly and efficiently? | DuckDB for Fast Analytical Query Workloads on Large Datasets |
Statistics and Data Analysis with Python
| Topic | Description | Resources |
|---|---|---|
| Statistics | Several ways to describe your data by using pure Python programming language with no additional libraries. | Python Statistics – mean, median, mode, min, max, range, variance |
| Several ways to describe your data by using the Python statistics package. | Python Functions – Median, Quantiles, Variance, STDev and Correlation | |
| Analyze normally and lognormally distributed data. | Python Normal Distribution Example | |
| Assess if selected datasets contain lognormally distributed values. | Assessing if Dataset Values are Lognormal Distributed with Python | |
| How to determine the probability density fit of a data set. | Coming Soon. | |
| Bayesian Models | Introduction to a simple probabilistic, yet powerful classifier, the Naïve Bayes Model, and implement it in Python. | Naive Bayes Model for Machine Learning and AI |
| How to construct a Bayesian network and model for uncovering data dependency and multivariate prediction. | Getting Started with Predictive Modeling Using Bayesian Network | |
| Linear Regression | How to do linear regression with SQL Server data. | Linear Regression with Python in SQL Server 2017 |
| How to check if two variables are linearly correlated and how to measure and express their degree of correlation. | Linear Correlation Analysis using Python with Code Examples | |
| Linear regression is a parametric model widely used in business analytics, econometrics, research, and development. | Implement Linear Regression in Python for Machine Learning | |
| Time Series | How to predict the target variable in a time series dataset | Time Series Forecasting using Python Prophet |
| How to plot financial time series data from SQL Server. Use the plots to reveal the value of exponential moving averages with different period lengths to make decisions about time series values. | Plotting in Python Financial Time Series from SQL Server | |
| Highlight critical aspects of data quality management within the context of time-series data. Use Python to demonstrate working with time-series data. | Data Quality Management for Time Series Analysis Resolved with Python | |
| How to organize data within SQL Server for export to a CSV file, import the CSV file into Python, and prepare performance charts for the time series data. | Performance Charts for Time Series Data with SQL Server and Python | |
| Non-linear Regression | There are two types of correlation analysis depending on how the two variables relate: linear and non-linear. How to check if two variables are non-linearly correlated | Non-Linear Relationship Analysis with Python |
| Data Profiling, Cleansing, EDA | Create a dataset profile for analyzing a dataset. | Reading and Profiling Data with Python Pandas |
| Data cleansing is the process of identifying and correcting inaccurate records from a record set, table, or database. Here are some data cleansing techniques. | Data Science: Cleansing Your Data Using Python | |
| Jupyter Notebook combines live code execution with textual comments, equations, and graphical visualizations. Here is how to use this tool for data exploration. | Data Exploration with Python and SQL Server using Jupyter Notebooks | |
| Explore data in a sample dataset using matplotlib. | Exploratory Data Analysis with Python in SQL Server 2017 |
Python for Data Visualization
| Topic | Description | Resources |
|---|---|---|
| Interactive Data Visualization | How to create interactive plots with plotly and matplotlib in VS Code. | Plotly to Visualize Time Series Data in Python |
| Python Matplotlib to Present Data Interactively in VS Code | ||
| Using the Python Bokeh library to visualize real-time streaming data. It will allow us to ingest streaming data and visualize them live in the browser. | Python Bokeh for Visualizing Real-Time Data | |
| How to portray security price time series data in line and candlestick charts: sample Python code to illustrate core concepts. | Create Subplots for Line and Candlestick Charts with Python and Plotly | |
| How to visualize REST API data representing the position of the International Space Station’s (ISS) in real time. | Real-Time Data Plotting with Python for Scientific Applications | |
| Static Data Visualization | How to create static plots with matplotlib and Python in VS Code. | Python matplotlib for Data Visualizations |
| How to create line and bar charts. | Python Bar Charts and Line Charts Examples | |
| Flask Python Reporting for SQL Server | ||
| How to create a variety of different pie charts for displaying counts based on output from SQL Server queries. | Pie Charts in Python from SQL Server Data | |
| How to visualize app logs using matplotlib. | Visualize Application Log Data with Python Matplotlib Charts | |
| Python can work with any of several charting libraries. This tip focuses on the Matplotlib library and the subplots method within the pyplot application programming interface. | Subplots Visualization for SQL Server Data with Python and Matplotlib | |
| How to create animated line plots with Python. | Creating Animated Line Plots with Python | |
| How to use SQL Server and Python for creating tree map charts from SQL data sources. | Treemap Charts for SQL Server | |
| How to use Python to generate plots from the data to analyze patterns, correlations, and trends | Data Visualization with Python Matplotlib | |
| How to analyze the data using Python and perform customer churn analysis to determine why customers stop using a service. | Customer Churn Analysis with Python | |
| Python in Power BI | How to source the output directly from Python scripts in Power BI Desktop, for reporting, visualization, and analytics. | Import data dynamically using R and Python in Power BI Desktop |
| Swarmplot is a specialized visualization in Python Seaborn library, which can be used with the Python script control in Power BI to render categorical scatterplots. | Visualize categorical scatterplots in Power BI with Python | |
| How to load data into a SQL Server database using Python and an API and build a simple Power BI report using this imported data. | Load SQL Server Data using Python and an API with Power BI Reporting | |
| Python scripts executed from the Python control and powered by Python installation of SQL Server enables creation of visualizations in Power BI. | Generate visualizations in Power BI using Python Scripts |
AI Engineering with Python
| Topic | Description | Resources |
|---|---|---|
| Object detection | How to run a lightweight object detection model on your home or business PC, regardless of the hardware you have. | Build an Object Detection Machine Learning Algorithm using Python |
| RAG | You have a vast amount of data on an Azure SQL Server and would like to take advantage of the retrieval augment generation (RAG). RAG enables large language models (LLMs) such as GPT to be grounded in your company-specific data and provide answers to complex questions and queries that would otherwise require time-consuming data mining. | Large Language Models with Azure AI Search and Python for OpenAI RAG |
| With the introduction of modern technologies like ChatGPT4o and all the industry-leading companies having AI incorporated into their businesses, it is safe to admit that we have reached a point of no return, and AI will only increase in popularity and influence in all businesses. | Build an LLM Application using LangChain | |
| Summarization | How to design a document summarization pipeline customized for large documents | Automated PDF Document Summarization with Azure OpenAI |
| Chatbot | How to enable an LLM to answer questions based on data stored in Azure SQL Database. How to package this in a custom application, such as a chatbot, for user-friendly interaction. | Build Chatbot with a LLM and Azure SQL Database to Answer Questions |
| How to add message history and a user interface for an LLM application. | AI Chatbot with Message History using LangChain and SQL | |
| Prompting | How to combine LangChain with prompt templates to be utilized with OpenAI’s LLMs effectively to enhance the quality of generated text and improve the efficiency of natural language processing tasks. | Prompt Templates for gpt-4 LLM with OpenAI and LangChain in Databricks |
| AI Agents | What are AI Agents? How to use them? What are ways to benefit from integrating AI Agents in existing code? | Coming Soon. |
| Getting started with Model Context Protocol (MCP) | Coming Soon. | |
| MLOps | As the tech industry voraciously develops and fine-tunes Large Language Models (LLMs), there comes the need to operationalize the management, maintenance, monitoring, optimization, comparison, and deployment of these LLMs as they work their way into production environments. With LLMOps, Gen AI enthusiasts will have the necessary infrastructure and tools to build and deploy LLMs easily, and the risks, challenges, and time to market for Gen AI LLMs can be reduced. | Get Started with LLMOps with MLFlow for Model Tracking with Open AI |
| Transcription | How to convert audio or video files stored in SQL Server and automatically create a transcript from these files. | Automatically Transcribe MP3 or MP4 Files and Store in SQL Server |
Python for Machine Learning and Deep Learning
| Topic | Description | Resources |
|---|---|---|
| Machine Learning | Introduction and implementation of the K-Nearest Neighbors model in Python. Although it is quite old, it remains immensely popular due to its simplicity and intuitiveness. | Machine Learning Introduction: KNN Model |
| Introduction to Logistic Regression model and K-Fold cross-validation, which will help us locate the better hyperparameters for our model. | Logistic Regression and K-Fold Cross Validation in Machine Learning | |
| Introduction to Neural Networks in a simplified manner to help beginners understand how they work and how to leverage their power to solve problems. | Introduction to Machine Learning with Neural Networks | |
| Introduction and implementation of basic text preprocessing and cleaning techniques with Python. | Text Cleaning and Preprocessing with Python for NLP Tasks | |
| Understanding of different model evaluation metrics and discuss suitability with the help of Python. | Model Evaluation Metrics in Machine Learning with Python | |
| Tensorflow | How to use machine learning and Python to make predictions | Machine Learning to make Predictions using Python and TensorFlow |
| Scikit-learn | Coming Soon. | Coming Soon. |
| Kerasno info | Coming Soon. | Coming Soon. |
Essential Python Packages for Data Professionals
| Topic | Description | Resources |
|---|---|---|
| Built-in Packages | Overview of the built-in Python packages such as os, sys, logging, etc. | Coming Soon. |
| Custom Packages | Create a Python Wheel File to Package and Distribute Custom Code | Create a Python Wheel File to Package and Distribute Custom Code |
| Numpy | Overview of the numpy package. | Coming Soon. |
| Pandas | Overview of the pandas package. | Coming Soon. |
| Scipy | Overview of the scipy package. | Coming Soon. |
| Requests | Overview of the requests package. | Coming Soon. |
| langchain | langchain for SQL data integration with LLMs. | Coming Soon. |
| ollama | SQL Query generation with the ollama package. | Coming Soon. |
Python and Azure Databricks
| Topic | Description | Resources |
|---|---|---|
| Overview | Getting started with Python and Azure Databricks (how and where to use) | Coming Soon. |
| Engineering a Lakehouse with Azure Databricks with Spark Dataframes | ||
| Delta Live Tables (DLT) | Getting started with DLT under Python | Coming Soon. |
| Comparison | Python vs Pyspark vs Pandas | Coming Soon. |
Python Functions
| Topic | Description | Resources |
|---|---|---|
| Defining functions | Using Python functions allows you to solve the problem of having to write repeated code, make code easier to understand, and reduce code complexity. | Python Functions for Reducing Code Complexity |
| Explaining the benefits of user-defined functions in Python programming is. | Python User Defined Functions for SQL Server Professionals | |
| How to avoid repetition using functions. | Learn Python Functions with Parameters, Nested Functions, Scope Rules | |
| Built-in functions | As a scripting language, Python also offers built-in functions that are helpful to the programmer. While it is difficult to enumerate all of them, there are some that are considered important and popular. | Built-in Functions in Python with Code Samples |
Object Oriented Programming (OOP) with Python
| Topic | Description | Resources |
|---|---|---|
| How to define classes | What are classes in Python? How do classes help to implement object-oriented programming (OOP) concepts? | Python Classes for Code Reusability |
| Static vs Class vs Instance methods | Overview and how to use the three diverse types of class methods. | Coming Soon. |
| Inheritance | Inheritance is a core OOP principle and powerful technique enabling better code structure and code reuse. | Coming Soon. |
| Polymorphism | Polymorphism is a core OOP principle and powerful technique. An object can take multiple forms in diverse ways in polymorphism. | Polymorphism in Python Code Examples – Functions, Classes, Inheritance |
Python Desktop and Web Applications
| Topic | Description | Resources |
|---|---|---|
| GUI with Tkinter | Using a standard GUI library for Python, i.e., tkinter, to create GUI applications. | Build Python GUI with Tkinter |
| Python executable | Creating a distributable executable in Python using PyInstaller | Build Standalone Python Executable Application |
| Windows | How to package Python app in various forms, e.g., MSI installer, executable, CLI app, or Windows service. | Create MSI Installer for a Python Application for Easy Distribution |
| Build Standalone Python Executable Application | ||
| Python as a Windows Service Example | ||
| Using Windows Task Scheduler to Run a Python Script at Prescribed Times | ||
| Use Task Scheduler to run Python scripts and then employ Windows Event Viewer to check the Python script execution history. So, how can we use all these techniques to automate routine tasks using Python on Windows? | How to Run Python Script in Windows | |
| How to Run a Python Script as a Windows Service using NSSM | ||
| Web Apps | Create Content Management System using Python Flask with SQL Server providing the database, and use the good old HTML to handle your views. | Build Content Management System using SQL Server, Python and Flask |
| Introduction to Django as an ideal framework to develop websites and web applications. | Python Django Tutorial for Website with SQL Server Database | |
| PyQt | Introduction to PyQT – what is it, how to get started, and how to build a simple app. | Coming Soon. |
| Flask | Create an AI-powered knowledge base. | Create a Knowledge Base with React, Strapi, and AI with Python Backend |
Article selection cut-off date: 2025-05-09.
Next Steps
- Be sure to bookmark this Python reference guide and come back to it often to improve your Python skills.
