Databricks Notebook

You are a Databricks notebook developer. The user wants to create PySpark and SQL notebooks with interactive widgets and visualizations.

What to check first

• Verify you're in a Databricks workspace (check the URL contains databricks.com)
• Confirm PySpark and SQL are available by running spark.version in a cell
• Check if displayHTML() and dbutils are accessible (built-in to Databricks notebooks)

Steps

1. Create a new notebook by clicking "Create" → "Notebook" and select Python or SQL as the default language
2. Import required libraries at the top: from pyspark.sql.functions import * and import pandas as pd
3. Add widgets using dbutils.widgets.text(), .dropdown(), or .multiselect() to capture user input
4. Create a DataFrame from a data source using spark.read.format("parquet").load() or spark.sql("SELECT * FROM table")
5. Filter or transform data based on widget values retrieved with dbutils.widgets.get("widget_name")
6. Use display() function to render DataFrames with automatic visualizations (tables, charts, maps)
7. Create custom visualizations by converting to Pandas and using displayHTML() with matplotlib or plotly
8. Use %sql, %python, %r, or %scala magic commands to mix languages in a single notebook

Code

# Databricks Notebook Source

# COMMAND ----------
# Import libraries
from pyspark.sql.functions import col, count, sum, avg, year, month
from pyspark.sql.types import StructType, StructField, StringType, IntegerType
import pandas as pd
import matplotlib.pyplot as plt

# COMMAND ----------
# Create interactive widgets
dbutils.widgets.text("start_date", "2024-01-01", "Start Date")
dbutils.widgets.text("end_date", "2024-12-31", "End Date")
dbutils.widgets.dropdown("region", "US", ["US", "EU", "APAC", "ALL"])
dbutils.widgets.multiselect("product_types", ["Electronics"], ["Electronics", "Clothing", "Books", "Food"])

# COMMAND ----------
# Retrieve widget values
start_date = dbutils.widgets.get("start_date")
end_date = dbutils.widgets.get("end_date")
region = dbutils.widgets.get("region")
product_types = dbutils.widgets.get("product_types").split(",")

# COMMAND ----------
# Create or read a sample DataFrame
data = [
    ("2024-01-15", "US", "Electronics", 1500),
    ("2024-02-20", "EU", "Clothing", 800),
    ("2024-03-10", "APAC", "Electronics", 2200),
    ("2024-04-05", "US", "Books", 350),

Note: this example was truncated in the source. See the GitHub repo for the latest full version.

Common Pitfalls

• Treating this skill as a one-shot solution — most workflows need iteration and verification
• Skipping the verification steps — you don't know it worked until you measure
• Applying this skill without understanding the underlying problem — read the related docs first

When NOT to Use This Skill

• When a simpler manual approach would take less than 10 minutes
• On critical production systems without testing in staging first
• When you don't have permission or authorization to make these changes

How to Verify It Worked

• Run the verification steps documented above
• Compare the output against your expected baseline
• Check logs for any warnings or errors — silent failures are the worst kind

Production Considerations

• Test in staging before deploying to production
• Have a rollback plan — every change should be reversible
• Monitor the affected systems for at least 24 hours after the change