Run Metrics

Run Metrics

When to Use

• Generating monthly or custom-range metrics reports
• Reviewing adoption rates and comment quality across languages
• Producing charts for metrics visualization
• Reviewing companion quality trends across recent months
• Saving finalized metrics to the database (only when user explicitly requests)

Defaults

Unless the user says otherwise, always apply these defaults:

• Environment: production
• Charts: Always include --charts
• Languages: All languages (do not pass --exclude)
• Save: Do NOT pass --save unless the user explicitly asks to save (e.g. "save it", "persist", "write to DB")
• Format: JSON output (UTF-8 encoded)
• Quality trends follow-up: If the user asks for metrics but does not explicitly ask for quality trends, ask a short follow-up after sharing the metrics results: "Do you want the quality trends chart for the same period too?"
• Quality trends auto-run: If the user explicitly asks for quality trends, comment trends, or asks broadly for charts/trends/quality, also run the companion quality-trends report for the same period.

Date Resolution

The user will typically specify a calendar month by name (e.g. "March", "January 2025"). Resolve to the full month date range:

| User says | start_date | end_date |

|-----------|-----------|----------|

| "March" (current year) | YYYY-03-01 | YYYY-03-31 |

| "January 2025" | 2025-01-01 | 2025-01-31 |

| "March 1 to March 15" | YYYY-03-01 | YYYY-03-15 |

| "2025-06-01 to 2025-06-30" | 2025-06-01 | 2025-06-30 |

When only a month name is given without a year, use the current year. Be careful with month lengths (28/29/30/31 days).

Running Metrics

Step 1: Run the Command

Show the resolved command and run it immediately in a single foreground terminal invocation with a 180-second timeout (timeout: 180000). Before running, clean up stale output so you never accidentally present old results.

Important: Do NOT use 2>&1 — that merges stderr log messages (e.g. "Saved: output\charts\adoption.png") into the JSON output file, producing invalid JSON. Pipe stdout through Out-File -Encoding UTF8 so the output file is always valid UTF-8 JSON.

Full terminal command (cleanup + run):

New-Item -ItemType Directory -Path output -Force | Out-Null; if (Test-Path output/metrics_output.json) { Remove-Item output/metrics_output.json }; if (Test-Path output/charts) { Remove-Item output/charts/* -Force }; python cli.py report metrics -s <start_date> -e <end_date> --charts | Out-File -Encoding UTF8 output/metrics_output.json

After the command completes, read the output file with read_file to get the JSON results — do NOT use a separate terminal command. Then use view_image (not a terminal command) to display the chart PNGs. This means the entire workflow requires only one terminal invocation.

Examples

# March 2025, production, all languages, with charts (typical request)
python cli.py report metrics -s 2025-03-01 -e 2025-03-31 --charts

# Staging environment
python cli.py report metrics -s 2025-03-01 -e 2025-03-31 --charts --environment staging

# Exclude specific languages
python cli.py report metrics -s 2025-03-01 -e 2025-03-31 --charts --exclude Java Golang

# Custom date range
python cli.py report metrics -s 2025-03-10 -e 2025-03-20 --charts

# Save to database (ONLY when user explicitly requests)
python cli.py report metrics -s 2025-03-01 -e 2025-03-31 --charts --save

Companion Quality Trends

The quality-trends report is the companion chart view for metrics work.

• If the user explicitly asks for quality trends, comment trends, or a broader trends/chart view, run it automatically for the same time window.
• If the user only asked for metrics, ask whether they want the quality trends chart too after presenting the metrics result.

Translating the date window

The quality-trends command uses --months plus an optional --end-date, not a start date.

Convert the requested metrics window into an inclusive calendar-month count:

| Window | Use for --months |

|--------|--------------------|

| 2026-03-01 to 2026-03-31 | 1 |

| 2026-03-01 to 2026-03-15 | 1 |

| 2026-01-15 to 2026-03-02 | 3 |

Use the same resolved end date from the metrics request for --end-date.

Quality trends command

if (Test-Path output/charts/comment_bucket_trends.png) { Remove-Item output/charts/comment_bucket_trends.png -Force }; python cli.py report quality-trends --months <month_count> --end-date <end_date>

Optional additions:

• --environment staging
• --languages Python CSharp C# Java TypeScript JavaScript Go
• --exclude-human
• --neutral

After it completes, summarize the terminal output and use view_image to show the saved chart at output/charts/comment_bucket_trends.png.

Follow-up: "Save it"

If the user asks to save after a metrics run (e.g. "okay save it", "persist that", "write to DB"), re-run the same command with --save appended. Keep all other flags identical to the previous run.

Available Flags

| Flag | Type | Default | Description |

|------|------|---------|-------------|

| --start-date / -s | string | required | Start date (YYYY-MM-DD) |

| --end-date / -e | string | required | End date (YYYY-MM-DD) |

| --environment | string | production | production or staging |

| --exclude | list | none | Space-separated language names to exclude |

| --charts | flag | off | Generate PNG charts to output/charts/ |

| --save | flag | off | Persist metrics to Cosmos DB (never use unless user explicitly requests) |

Chart Outputs

When --charts is enabled, 4 PNGs are saved to output/charts/:

• adoption.png — Copilot vs non-Copilot reviews by language
• comment_quality.png — Comment quality breakdown (upvoted, implicit good/bad, downvoted, deleted)
• human_copilot_split.png — Human vs AI comment split
• human_comments_comparison.png — Human comments with vs without Copilot

Gotchas

• Redirect stdout to a file: Always pipe through | Out-File -Encoding UTF8 output/metrics_output.json and read the file afterward. Do NOT use > which produces UTF-16 in PowerShell 5.1.
• Clean up before running: Delete the previous output/metrics_output.json and output/charts/* before running. This prevents presenting stale results if the command fails silently.
• Do NOT use 2>&1: This merges stderr log lines (e.g. "Saved: ...") into the JSON output file, producing invalid JSON. Only redirect stdout.
• Use foreground with timeout: Run with isBackground=false and timeout: 180000 (3 min). Do NOT use isBackground=true and poll — that causes repeated user approval prompts.
• Use New-Item -ItemType Directory not mkdir: mkdir is aliased differently across shells. Use New-Item -ItemType Directory -Path output -Force | Out-Null for reliable directory creation.
• Read results with read_file and view_image: After the command finishes, use read_file for the JSON and view_image for charts. Do NOT launch additional terminal commands to read the file.
• Use python cli.py not .\avc: The avc.bat script may resolve to system Python. Use python cli.py report metrics ... to ensure the correct environment.
• Quality trends is separate: It uses report quality-trends with --months and optional --end-date, not --start-date.
• Month end dates: February has 28/29 days, April/June/Sept/Nov have 30 days. Get it right.
• Built-in exclusions: c, c++, typespec, swagger, xml are always excluded automatically — no need to add them.