GitHub Actions Recipes for AI Testing (Copy-Paste Ready)

You know you should be testing your AI systems in CI/CD. You know you should fail builds when accuracy drops or safety scores degrade. The problem is nobody has written the workflows yet.

Here are five GitHub Actions workflows I've built and rebuilt across multiple companies. They're battle-tested, they're production-ready, and they're yours to copy-paste into your repo right now.

Workflow 1: Run Your LLM Evaluation Suite

This workflow runs your evaluation dataset every time you push code. It measures accuracy, latency, and semantic similarity against a baseline. If you drop below thresholds, the build fails.

name: LLM Evaluation Suite

on:
 push:
 branches: [main, develop]
 pull_request:
 branches: [main]

jobs:
 evaluate:
 runs-on: ubuntu-latest

 steps:
 - uses: actions/checkout@v3

 - name: Set up Python
 uses: actions/setup-python@v4
 with:
 python-version: '3.11'

 - name: Install dependencies
 run: |
 pip install -r requirements.txt
 pip install openai langchain scipy numpy

 - name: Run evaluation suite
 env:
 OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
 run: |
 python scripts/evaluate.py \
 --model models/my-model.gguf \
 --dataset data/test_cases.json \
 --output results.json

 - name: Check results against baseline
 run: |
 python scripts/check_baseline.py \
 --results results.json \
 --baseline baselines/accuracy_baseline.json \
 --min-accuracy 0.85 \
 --max-latency 2000 \
 --min-f1 0.82

 - name: Upload results
 if: always()
 uses: actions/upload-artifact@v3
 with:
 name: evaluation-results
 path: results.json

 - name: Comment on PR with results
 if: github.event_name == 'pull_request'
 uses: actions/github-script@v6
 with:
 script: |
 const fs = require('fs');
 const results = JSON.parse(fs.readFileSync('results.json', 'utf8'));
 const comment = `## Evaluation Results
 - Accuracy: ${(results.accuracy * 100).toFixed(2)}%
 - F1 Score: ${(results.f1 * 100).toFixed(2)}%
 - Latency (p95): ${results.latency_p95}ms
 - Tokens/sec: ${results.throughput.toFixed(1)}`;
 github.rest.issues.createComment({
 issue_number: context.issue.number,
 owner: context.repo.owner,
 repo: context.repo.repo,
 body: comment
 });

The key part: check_baseline.py compares your current results against known-good baselines. It's not enough to know your accuracy. You need to know if it's better or worse than last week.

Workflow 2: Detect Model Drift

This workflow runs every 6 hours (or on a schedule you define) and compares your model's performance to a baseline. If accuracy has dropped more than 5%, it alerts you.

This catches the silent killers: your model working fine locally but degrading gradually in production due to data drift or feature distribution changes.

name: Model Drift Detection

on:
 schedule:
 - cron: '0 */6 * * *' # Every 6 hours
 workflow_dispatch:

jobs:
 drift-check:
 runs-on: ubuntu-latest

 steps:
 - uses: actions/checkout@v3

 - name: Set up Python
 uses: actions/setup-python@v4
 with:
 python-version: '3.11'

 - name: Fetch production metrics
 env:
 DATADOG_API_KEY: ${{ secrets.DATADOG_API_KEY }}
 ARIZE_API_KEY: ${{ secrets.ARIZE_API_KEY }}
 run: |
 python scripts/fetch_production_metrics.py \
 --last-hours 6 \
 --output production_metrics.json

 - name: Run drift detection
 run: |
 python scripts/detect_drift.py \
 --production-metrics production_metrics.json \
 --baseline baselines/production_baseline.json \
 --output drift_report.json

 - name: Parse results and alert
 run: |
 python scripts/parse_drift_report.py \
 --report drift_report.json \
 --drift-threshold 0.05 \
 --output metrics.json

 - name: Send Slack alert if drift detected
 if: failure()
 uses: slackapi/[email protected]
 with:
 webhook-url: ${{ secrets.SLACK_WEBHOOK }}
 payload: |
 {
 "text": "Model drift detected!",
 "blocks": [
 {
 "type": "section",
 "text": {
 "type": "mrkdwn",
 "text": ":warning: *Model Drift Alert*\n\nAccuracy dropped more than 5% in the last 6 hours."
 }
 },
 {
 "type": "section",
 "text": {
 "type": "mrkdwn",
 "text": "${{ steps.parse.outputs.drift_summary }}"
 }
 }
 ]
 }

 - name: Create issue if severe drift
 if: failure()
 uses: actions/github-script@v6
 with:
 script: |
 github.rest.issues.create({
 owner: context.repo.owner,
 repo: context.repo.repo,
 title: 'Model drift detected - urgent investigation needed',
 body: 'Model accuracy has dropped significantly. See drift report in logs.',
 labels: ['model-quality', 'urgent']
 });

Pro tip: Feed your production metrics from Datadog, Arize, or WhyLabs. This workflow only works if you're actually monitoring real usage.

Workflow 3: Regression Testing for Conversation Flows

This workflow tests that your conversational AI still handles known conversation flows correctly. It's like unit testing but for conversations.

name: Conversation Regression Tests

on:
 push:
 branches: [main]
 paths:
 - 'models/**'
 - 'prompts/**'
 - '.github/workflows/regression-tests.yml'
 pull_request:
 branches: [main]

jobs:
 regression:
 runs-on: ubuntu-latest

 steps:
 - uses: actions/checkout@v3

 - name: Set up Node.js
 uses: actions/setup-node@v3
 with:
 node-version: '18'

 - name: Install dependencies
 run: npm ci

 - name: Run conversation regression tests
 env:
 OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
 run: |
 npm run test:conversations -- \
 --dataset test_conversations.json \
 --model-id ${{ github.sha }} \
 --output regression_results.json

 - name: Check for regressions
 run: |
 node scripts/check_regressions.js \
 --results regression_results.json \
 --baseline baselines/conversation_baseline.json \
 --fail-on-regression true

 - name: Generate report
 if: always()
 run: |
 node scripts/generate_regression_report.js \
 --results regression_results.json \
 --output regression_report.html

 - name: Upload report
 if: always()
 uses: actions/upload-artifact@v3
 with:
 name: regression-report
 path: regression_report.html
 retention-days: 30

Store your conversation flows as JSON. Each conversation is a series of user-model turns with expected intents and context retention checks. This catches regressions where your model suddenly starts losing context or contradicting itself.

Workflow 4: Cost Monitoring & Budget Alerts

Every inference costs money. This workflow tracks cost per inference, cost per model version, and alerts you when costs are trending up.

name: Cost Monitoring

on:
 schedule:
 - cron: '0 9 * * MON' # Every Monday at 9am
 workflow_dispatch:

jobs:
 cost-check:
 runs-on: ubuntu-latest

 steps:
 - uses: actions/checkout@v3

 - name: Set up Python
 uses: actions/setup-python@v4
 with:
 python-version: '3.11'

 - name: Fetch API usage logs
 env:
 OPENAI_ORG_ID: ${{ secrets.OPENAI_ORG_ID }}
 OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
 run: |
 python scripts/fetch_usage_logs.py \
 --last-days 7 \
 --output usage.json

 - name: Calculate costs
 run: |
 python scripts/calculate_costs.py \
 --usage usage.json \
 --output costs.json \
 --pricing-file pricing/openai_pricing.json

 - name: Compare to budget
 run: |
 python scripts/check_budget.py \
 --costs costs.json \
 --budget 1000 \
 --budget-period monthly \
 --output budget_status.json

 - name: Generate cost report
 run: |
 python scripts/generate_cost_report.py \
 --costs costs.json \
 --output cost_report.md

 - name: Create or update issue for cost tracking
 uses: actions/github-script@v6
 with:
 script: |
 const fs = require('fs');
 const costs = JSON.parse(fs.readFileSync('costs.json', 'utf8'));
 const body = `## Weekly Cost Report
 - This week: $${costs.weekly_total.toFixed(2)}
 - Monthly average: $${costs.monthly_average.toFixed(2)}
 - Cost per inference: $${costs.cost_per_inference.toFixed(4)}
 - Top models by cost: ${costs.top_models.join(', ')}`;

 // Find or create issue
 const issues = await github.rest.issues.listForRepo({
 owner: context.repo.owner,
 repo: context.repo.repo,
 state: 'open',
 labels: ['cost-tracking']
 });

 if (issues.data.length > 0) {
 github.rest.issues.update({
 owner: context.repo.owner,
 repo: context.repo.repo,
 issue_number: issues.data[0].number,
 body: body
 });
 }

This saves you from the surprise of a $10K bill. You can set budget thresholds and get alerted when you're trending toward overspending.

Workflow 5: Safety Scanning

This workflow runs safety tests: prompt injection attempts, jailbreaks, and other adversarial inputs. If the model fails a safety test, the build fails.

name: Safety Scanning

on:
 push:
 branches: [main]
 pull_request:
 branches: [main]

jobs:
 safety-scan:
 runs-on: ubuntu-latest

 steps:
 - uses: actions/checkout@v3

 - name: Set up Python
 uses: actions/setup-python@v4
 with:
 python-version: '3.11'

 - name: Install safety testing tools
 run: |
 pip install -r requirements-safety.txt

 - name: Run prompt injection tests
 env:
 OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
 run: |
 python scripts/test_prompt_injection.py \
 --model models/my-model.gguf \
 --output prompt_injection_results.json

 - name: Run jailbreak tests
 env:
 OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
 run: |
 python scripts/test_jailbreaks.py \
 --model models/my-model.gguf \
 --dataset data/known_jailbreaks.json \
 --output jailbreak_results.json

 - name: Check safety thresholds
 run: |
 python scripts/check_safety.py \
 --prompt-injection-results prompt_injection_results.json \
 --jailbreak-results jailbreak_results.json \
 --max-injection-rate 0.05 \
 --max-jailbreak-rate 0.10

 - name: Generate safety report
 if: always()
 run: |
 python scripts/generate_safety_report.py \
 --prompt-injection-results prompt_injection_results.json \
 --jailbreak-results jailbreak_results.json \
 --output safety_report.json

 - name: Comment on PR with safety results
 if: github.event_name == 'pull_request'
 uses: actions/github-script@v6
 with:
 script: |
 const fs = require('fs');
 const report = JSON.parse(fs.readFileSync('safety_report.json', 'utf8'));
 const comment = `## Safety Scan Results
 - Prompt injections caught: ${report.prompt_injections_caught}/${report.prompt_injections_attempted}
 - Jailbreaks caught: ${report.jailbreaks_caught}/${report.jailbreaks_attempted}
 - Success rate: ${(report.safety_success_rate * 100).toFixed(1)}%`;
 github.rest.issues.createComment({
 issue_number: context.issue.number,
 owner: context.repo.owner,
 repo: context.repo.repo,
 body: comment
 });

Putting It Together: A Complete CI/CD Pipeline

these workflows fit together in a complete pipeline:

1. Developer pushes code (new prompt, model update, config change)
2. Evaluation suite runs (5 min) - if accuracy drops, PR is blocked
3. Regression tests run (3 min) - conversation flows still work?
4. Safety scan runs (2 min) - any new jailbreaks?
5. If all pass: PR can be merged
6. Every 6 hours: drift detection checks production metrics
7. Every Monday: cost report generated

This setup gives you confidence that every deployed change is safe, accurate, and affordable.

Next Steps

Copy these workflows into your .github/workflows/ directory. Update the secrets (API keys), adjust thresholds to match your baselines, and customize the evaluation datasets. Then commit and watch your CI/CD system become a quality guardian.

Most teams I work with make one workflow their own first (usually evaluation), then gradually add the others as they understand their needs better. Start with evaluation. It's the foundation everything else builds on.