Splunk SPL

You are a Splunk Search Processing Language (SPL) expert. The user wants to write SPL queries for search, stats, and timechart operations to analyze and visualize data in Splunk.

What to check first

• Verify your Splunk instance is running and you have access to the search bar (usually at http://localhost:8000 or your Splunk URL)
• Run a basic search like index=main | head 10 to confirm data is indexed and searchable
• Check which fields are available in your dataset using index=main | fields or the field picker in the UI

Steps

1. Start with a base search using index=<indexname> to narrow down your data source, optionally add source=<sourcename> or sourcetype=<sourcetypename> filters
2. Use the pipe operator | to chain commands; SPL reads left-to-right and each pipe passes results to the next command
3. Apply the search command with field-value pairs like search status=200 user=admin to filter records after the initial index search
4. Use stats command with aggregation functions: stats count, stats sum(bytes), stats avg(response_time), stats dc(user) (distinct count)
5. Group stats by field using by clause: stats count by status creates rows for each unique status value with counts
6. Chain multiple grouping fields: stats sum(bytes) by host, status groups by both host and status
7. Use timechart command to create time-series data: timechart count by status groups counts over time buckets (default 10-minute spans)
8. Control time bucket size in timechart with timechart span=1h count for hourly buckets, or span=5m for 5-minute intervals

Code

# Search: Find all failed authentication attempts in the last 24 hours
index=main sourcetype=auth status=failure earliest=-24h | search user!="system"

# Stats: Count login attempts per user with average response time
index=main sourcetype=auth earliest=-7d
| stats count as login_attempts, avg(response_time) as avg_response_ms by user
| where login_attempts > 5
| sort - login_attempts

# Timechart: HTTP traffic volume by status code over the last week, hourly buckets
index=main sourcetype=http earliest=-7d
| timechart span=1h count by status
| fillnull value=0

# Complex stats: Calculate percentiles and multiple aggregations
index=main sourcetype=app_logs earliest=-1d
| stats count, avg(latency_ms) as avg_latency, max(latency_ms) as max_latency, 
         perc95(latency_ms) as p95_latency, dc(session_id) as unique_sessions by app_name
| where avg_latency > 500

# Timechart with stats

Note: this example was truncated in the source. See the GitHub repo for the latest full version.

Common Pitfalls

• Treating this skill as a one-shot solution — most workflows need iteration and verification
• Skipping the verification steps — you don't know it worked until you measure
• Applying this skill without understanding the underlying problem — read the related docs first

When NOT to Use This Skill

• When a simpler manual approach would take less than 10 minutes
• On critical production systems without testing in staging first
• When you don't have permission or authorization to make these changes

How to Verify It Worked

• Run the verification steps documented above
• Compare the output against your expected baseline
• Check logs for any warnings or errors — silent failures are the worst kind

Production Considerations

• Test in staging before deploying to production
• Have a rollback plan — every change should be reversible
• Monitor the affected systems for at least 24 hours after the change