Prometheus Setup

You are a DevOps engineer or systems administrator. The user wants to set up Prometheus with scraping targets, recording rules, and AlertManager integration for production monitoring and alerting.

What to check first

• Verify Prometheus is installed: prometheus --version
• Check if AlertManager is installed: alertmanager --version
• Confirm port 9090 (Prometheus) and 9093 (AlertManager) are available: netstat -tulpn | grep -E '909[03]'
• Validate existing prometheus.yml: promtool check config /etc/prometheus/prometheus.yml

Steps

1. Create the main Prometheus config at /etc/prometheus/prometheus.yml with global scrape interval, evaluation interval, and alertmanager address
2. Define scrape_configs with job_name, static_configs with targets, and scrape_interval overrides for specific jobs
3. Create recording rules file at /etc/prometheus/rules/recording_rules.yml with expr, for, and labels fields
4. Create alerting rules file at /etc/prometheus/rules/alert_rules.yml with alert name, expr, for duration, and annotations
5. Add rule_files paths to prometheus.yml pointing to both recording and alerting rule files
6. Configure AlertManager config at /etc/prometheus/alertmanager.yml with global settings, route tree, and receiver definitions
7. Update prometheus.yml alerting section with AlertManager target: alerting: alertmanagers: - static_configs: - targets: ['localhost:9093']
8. Reload Prometheus with promtool check config validation, then kill -HUP $(pgrep prometheus) or restart the service

Code

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: 'production'
    environment: 'prod'

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - localhost:9093

rule_files:
  - '/etc/prometheus/rules/recording_rules.yml'
  - '/etc/prometheus/rules/alert_rules.yml'

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node-exporter'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:9100', 'node2.example.com:9100']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

  - job_name: 'application'
    metrics_path: '/metrics'
    scrape_interval: 30s
    static_configs:
      - targets: ['app1:8080', 'app2:8080']
---
# /etc/prometheus/rules/recording_rules

Note: this example was truncated in the source. See the GitHub repo for the latest full version.

Common Pitfalls

• Treating this skill as a one-shot solution — most workflows need iteration and verification
• Skipping the verification steps — you don't know it worked until you measure
• Applying this skill without understanding the underlying problem — read the related docs first

When NOT to Use This Skill

• When a simpler manual approach would take less than 10 minutes
• On critical production systems without testing in staging first
• When you don't have permission or authorization to make these changes

How to Verify It Worked

• Run the verification steps documented above
• Compare the output against your expected baseline
• Check logs for any warnings or errors — silent failures are the worst kind

Production Considerations

• Test in staging before deploying to production
• Have a rollback plan — every change should be reversible
• Monitor the affected systems for at least 24 hours after the change