Databricks MLflow

You are a Machine Learning Engineer integrating MLflow with Databricks to track experiments, register models, and manage production deployments.

What to check first

• Verify MLflow is installed: pip list | grep mlflow (should show mlflow >= 2.0)
• Check Databricks workspace connection: databricks workspace list or verify ~/.databrickscfg exists
• Confirm Spark is available: pyspark --version or test import pyspark in Python

Steps

1. Set the MLflow tracking URI to Databricks: Use mlflow.set_tracking_uri("databricks") to automatically connect to your workspace's managed MLflow instance
2. Create or get an experiment: Call mlflow.set_experiment("/Users/user@example.com/my_experiment") with a workspace path (not just a name)
3. Start a run and log parameters: Use mlflow.start_run() context manager, then mlflow.log_param("learning_rate", 0.01) for hyperparameters
4. Log metrics during training: Call mlflow.log_metric("accuracy", accuracy_value, step=epoch_number) to track validation metrics over steps
5. Log artifacts (models, plots, data): Use mlflow.log_artifact("model.pkl") or mlflow.log_figure(fig, "confusion_matrix.png") to store files
6. Register the model in MLflow Registry: After run completes, use mlflow.register_model("runs:/<run_id>/model", "model_name") to register in the workspace catalog
7. Transition model to Production stage: Query the registry with client.transition_model_version_stage() to move from Staging → Production
8. Load and deploy: Use mlflow.pyfunc.load_model("models:/<model_name>/Production") to fetch the latest production model for serving

Code

import mlflow
import mlflow.sklearn
from mlflow.tracking import MlflowClient
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix
import numpy as np

# Set tracking URI to Databricks workspace MLflow
mlflow.set_tracking_uri("databricks")

# Set experiment (use workspace path format)
mlflow.set_experiment("/Users/your-email@company.com/iris_classification")

# Initialize client for model registry operations
client = MlflowClient()

# Load data and split
iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(
    iris.data, iris.target, test_size=0.2, random_state=42
)

# Start MLflow run
with mlflow.start_run(run_name="rf_baseline") as run:
    run_id = run.info.run_id

Note: this example was truncated in the source. See the GitHub repo for the latest full version.

Common Pitfalls

• Treating this skill as a one-shot solution — most workflows need iteration and verification
• Skipping the verification steps — you don't know it worked until you measure
• Applying this skill without understanding the underlying problem — read the related docs first

When NOT to Use This Skill

• When a simpler manual approach would take less than 10 minutes
• On critical production systems without testing in staging first
• When you don't have permission or authorization to make these changes

How to Verify It Worked

• Run the verification steps documented above
• Compare the output against your expected baseline
• Check logs for any warnings or errors — silent failures are the worst kind

Production Considerations

• Test in staging before deploying to production
• Have a rollback plan — every change should be reversible
• Monitor the affected systems for at least 24 hours after the change