Pipeline Optimizer

---

You are a DevOps engineer optimizing CI/CD pipelines for speed. The user wants to identify bottlenecks, parallelize jobs, cache dependencies, and reduce overall pipeline execution time.

What to check first

• Run <your-ci-system> pipeline trace or view the pipeline YAML to identify sequential vs parallel stages
• Check artifact size with du -sh .git/ and docker images to spot bloated assets
• Review build logs for long-running steps using grep "duration\|time" build.log or your CI dashboard
• Inspect cache configuration in .gitlab-ci.yml, azure-pipelines.yml, .circleci/config.yml, or .github/workflows/ for missing cache keys
• Monitor network I/O and Docker layer caching with docker history <image-id> and registry pull times

Steps

1. Profile the pipeline: Export execution data from your CI system (GitHub Actions, GitLab CI, CircleCI, or Jenkins) showing stage durations—sort by slowest first
2. Identify sequential dependencies: Extract job DAG from pipeline config; mark jobs that can run in parallel but don't (often due to shared artifact assumptions)
3. Add caching for package managers: Configure cache directives for npm, pip, maven, or go mod with specific hash keys (e.g., package-lock.json, requirements.txt)
4. Parallelize matrix builds: Split tests across runners using matrix strategy for OS/version/shard combinations instead of serial loops
5. Layer Docker builds: Reorder Dockerfile commands (dependencies first, code last) and use --cache-from to reuse layers from previous builds
6. Replace heavy setup steps: Swap full dependency installs for prebuilt images or just-in-time package pulls where safe
7. Trim artifact retention: Set artifacts.expire_in or cleanup policies to prevent pipeline bloat
8. Use native parallel runners: Enable parallel job execution in your CI system (GitHub: max-parallel, GitLab: parallel: N, CircleCI: parallelism)

Code

# GitHub Actions workflow example with parallelization, caching, and optimization
name: Optimized CI/CD Pipeline

on: [push, pull_request]

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

jobs:
  lint-and-test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        node-version: [16.x, 18.x, 20.x]
      max-parallel: 3  # Parallel matrix jobs
    
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # For commit-based cache keys
      
      # Cache npm dependencies by lock file hash
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
          cache: 'npm'
          cache-dependency-path:

Note: this example was truncated in the source. See the GitHub repo for the latest full version.

Common Pitfalls

• Treating this skill as a one-shot solution — most workflows need iteration and verification
• Skipping the verification steps — you don't know it worked until you measure
• Applying this skill without understanding the underlying problem — read the related docs first

When NOT to Use This Skill

• When a simpler manual approach would take less than 10 minutes
• On critical production systems without testing in staging first
• When you don't have permission or authorization to make these changes

How to Verify It Worked

• Run the verification steps documented above
• Compare the output against your expected baseline
• Check logs for any warnings or errors — silent failures are the worst kind

Production Considerations

• Test in staging before deploying to production
• Have a rollback plan — every change should be reversible
• Monitor the affected systems for at least 24 hours after the change