Weekly Production Review

Weekly Production Review

Use this skill to produce a source-grounded production review across Datadog,

incident.io, and Linear. Keep the output factual, table-first, and easy to

audit from the linked source rows.

Scope

• Default "last week" to the previous Monday through Sunday in the user's

timezone. State both local and UTC query windows in one short scope line.

• Cover all production environments unless the user narrows scope:

prod-us, prod-eu, prod-hipaa, and prod-jp.

• Keep the first pass read-only. Do not create or update Linear issues,

comments, incident.io records, follow-ups, alerts, Datadog monitors, files,

Slack messages, or production systems unless the user explicitly asks after

reviewing the findings.

• For chat-only reviews, avoid creating report artifacts or local analysis

workspaces unless a required tool workflow explicitly does so or the user asks

for a file. If incident.io analysis tooling requires a local playbook

workspace, mention it only when relevant and keep production systems

unchanged.

• Write No measurements found when a requested signal cannot be queried or

measured.

Related Skills

• Use datadog-query-recipes for

production Datadog query shapes and environment/site routing.

• Use linear-bug-triage only after a human

explicitly approves a Linear write-back.

• Use incident-alert-tickets to check

each Datadog alert cluster against the per-monitor knowledge base and, only

after explicit human approval, record newly root-caused clusters there.

Workflow

• Confirm the review window and timezone. If the user says "last week", use the

previous calendar week, not a rolling seven-day window.

• Gather public/customer-facing incidents from incident.io. Prefer incident.io

for accepted incidents, public incident visibility, follow-ups, and incident

status. Always produce the incident.io table below, even when no rows are

found.

• Gather incident.io alert load for the same review window. Prefer

incident.io alert or escalation stats for the primary Langfuse escalation

path/team. If the user provides an incident.io pager-load dashboard URL, use

its escalation_path parameter as a scope hint; use the dashboard date range

only when the user explicitly scopes the review to that range instead of the

default weekly window. Group by paged engineer and incident.io time-of-day

bucket so the table shows working-hours, evening, and night load. Always

produce the incident.io alert load table below.

• Gather Linear bugs from the bug label first. Include all bug-labeled

tickets created, updated, completed, or still open with production evidence

during the window. Inspect likely production bugs with issue details and

comments when status, owner, or evidence is unclear. Always produce the

Linear bug table below.

• Gather Datadog alert/page signals for the window. Use incident.io alerts or

escalations when they represent pages; use Datadog monitor/event data when

available. Build the exhaustive alert universe by paginating until no more

results remain for the window. Cover every production environment in scope.

Group repeated firings by monitor/page title or ID, environment, service/team,

and trigger reason.

• For every Datadog alert/page cluster, perform the deep dive before writing the

final row. Do not stop at the monitor title or count. Check the monitor's

incident-alert ticket first (see

incident-alert-tickets); a

documented cause section may explain the cluster — cite the ticket in the

incident.io / Linear Link column. Inspect matching APM

spans, representative traces, related logs, error records, exception details,

failed job logs, dependency spans, queue backlog/delay context, and monitor

time windows. Put the relevant trace/span evidence and relevant logs/errors

directly in the Datadog alerts table. Do not create a separate Datadog Issue

Deep Dives table.

• Gather Datadog error log patterns for the window. Use logs with

status:error, scope to production environments, and group by the clustered

message pattern plus service/env where available. Always produce the

Datadog logs table below. Preserve exact Datadog patterns, including wildcard

tokens, instead of paraphrasing them.

• Classify each incident, bug, alert, alert-load row, and log pattern. Separate

production breakage from self-hosted, internal-only, duplicate, canceled,

expected/test, staging/dev, monitor-noise, or unknown signals.

• Cross-reference source rows on a best-effort basis. Link Datadog rows to

matching incident.io incidents or Linear bugs when evidence supports the

relationship. Link incident.io and Linear rows back to Datadog evidence when

available. If a relationship is inferential, say so in the row.

Output Contract

Return one short scope line followed by exactly these five source tables, in

this order:

• incident.io
• incident.io Alert Load
• Linear Bugs
• Datadog Alerts
• Datadog Logs

Do not add an executive summary, narrative summary, event-centric view, summary

table, source synthesis table, or separate Datadog Issue Deep Dives table. Put

counts and classifications inside the source tables.

If a table has no rows, keep the table heading and write one row or sentence

with No rows found or No measurements found plus the scoped source/query.

If a row is unclear, classify it as unclear or unknown/no measurements

instead of dropping it.

Cross-Source Linking

Keep incident.io incidents, incident.io alert load, Linear, Datadog alerts, and

Datadog logs as separate output tables. Use links inside each table to show

relationships instead of synthesizing a separate cross-source table.

Use Linear as the source of truth for deduplication across weeks and workflows.

Before reporting a bug, security finding, cost concern, or alert as new, search

Linear for matching issue keys, titles, source URLs, and comments — covering

both the bug label set and the incident-alert label set. If an

existing issue covers it, link to that issue and mark the row as already

tracked instead of reporting it again as fresh work.

Use short stable link labels:

• Datadog monitor: <monitor name>
• Datadog logs: <env/service/symptom>
• Datadog spans: <env/route/symptom>
• Datadog trace: <trace id or route>
• incident.io: <INC reference>
• incident.io alert load: <escalation path or team>
• Linear: <issue key>

Do not write links, create follow-ups, or update external systems unless the

user explicitly asks for changes after reviewing the report.

incident.io Table

Use this table for incident.io incidents with public or customer-facing impact

in the review window. Query incident.io with incident_list scoped to the

review window and relevant team when known. Include summary, roles,

custom_fields, timestamps, durations, and escalation_urgency when the

tool supports them. If the tool cannot filter by visibility server-side, list

the scoped incidents and include only rows where visibility is public or

the evidence supports customer-facing impact.

| Incident | Severity / Status | Start / End / Duration | Impact | Linked Sources | Follow-ups / Notes |

| --- | --- | --- | --- | --- | --- |

Column rules:

• Incident: incident.io reference linked to the incident.
• Severity / Status: severity and current lifecycle status.
• Start / End / Duration: reported, identified, resolved, and duration when

available.

• Impact: short impact statement grounded in the incident summary.
• Linked Sources: Datadog alerts/pages, Datadog logs/spans, Linear issues, or

none found.

• Follow-ups / Notes: follow-up count/status or none found.

incident.io Alert Load Table

Use this table for incident.io alert and pager load in the review window.

Prefer incident.io alert or escalation stats filtered to the Langfuse escalation

path or team. If the user provides a pager-load dashboard URL, parse and apply

the escalation_path[one_of] filter when available. Count alerts when the

source returns alert counts; otherwise count escalations/pages and label the

count source in Source / Notes.

incident.io time-of-day buckets are UTC:

• working_hours: 09:00-18:00 Monday-Friday.
• late_evening: 18:00-23:00 any day plus weekend daytime.
• overnight: 23:00-09:00.

Render one row per paged engineer, sorted by total descending, and include a

final All engineers row when measurements exist. If user identity is missing,

use Unassigned / no responder. Do not collapse this table into a narrative

summary.

| Engineer | Working Hours | Late Evening | Overnight | Total Alerts / Pages | Share | Source / Notes |

| --- | ---: | ---: | ---: | ---: | ---: | --- |

Column rules:

• Engineer: paged user or escalation target. Use the engineer's display name

when available.

• Working Hours: count in the working_hours bucket.
• Late Evening: count in the late_evening bucket.
• Overnight: count in the overnight bucket.
• Total Alerts / Pages: row total. Label pages versus alerts in

Source / Notes when the source does not expose alert counts directly.

• Share: row total divided by the measured alert/page total.
• Source / Notes: source query, escalation path/team filter, dashboard link,

or No measurements found.

Linear Bugs Table

Start from all Linear tickets with the bug label that were touched by the

window. Do not rely only on text searches for prod, incident, or Datadog;

those searches are useful for enrichment but are not the source universe. This

table is the Linear source inventory. A Linear bug can be classified as

non-production, duplicate, canceled, or no-action.

| Linear | Title | Summary | Owner | Status | Touched Last Week Because | Production Evidence | Classification | Counted? |

| --- | --- | --- | --- | --- | --- | --- | --- | --- |

Column rules:

• Linear: issue key linked to Linear, such as LFE-123.
• Title: Linear issue title in a separate column.
• Summary: one operational sentence based on the issue body, comments, and

evidence. Avoid fix guesses.

• Owner: assignee if present; otherwise owning team if clear; otherwise

Unassigned.

• Status: Linear status or state, plus completion timing when useful.
• Touched Last Week Because: created, updated, completed, or

open production bug.

• Production Evidence: prod env, customer impact, incident.io incident,

Datadog link, measured logs/spans/errors, or No measurements found.

• Classification: use production/customer-impacting, internal-only,

self-hosted, staging/dev, duplicate/canceled/no-action, or unclear.

• Counted?: yes only when the bug label and production/customer-impacting

evidence support including it in fixed/open production bug counts.

Datadog Alerts Table

Use this table for every production Datadog alert/page cluster found in the full

paginated alert/event pass, including clusters later classified as

expected/test, monitor noise, or unknown/no measurements.

| Monitor / Page Signal | Env / Service | Count / Window | Why It Alerted | Trace / Span Evidence | Relevant Logs / Errors | Verdict | incident.io / Linear Link |

| --- | --- | ---: | --- | --- | --- | --- | --- |

Column rules:

• Monitor / Page Signal: monitor/page title or stable ID.
• Env / Service: production env and service/team labels.
• Count / Window: grouped firing/page count and relevant time window.
• Why It Alerted: monitor threshold, trigger condition, route, queue, status

code, latency, or backlog signal.

• Trace / Span Evidence: representative trace/span links, error counts,

latency, status codes, dependency spans, or No measurements found.

• Relevant Logs / Errors: explicit exception class/message, exact or

normalized log message, failed job IDs when visible, DB/downstream errors,

retry exhaustion, validation failures, or No measurements found.

• Verdict: use customer incident, confirmed bug, infra/dependency,

expected/test, monitor noise, or unknown/no measurements.

• incident.io / Linear Link: matching incident.io reference, Linear issue key,

explicit disposition, or none found.

For API route and queue consumer errors, start from APM spans matching:

operation_name:(http.server OR bullmq.consumer) status:error env:<prod-env>

Then narrow by service, resource_name, route, queue, consumer, status code,

error type, or monitor time window. For failed trace samples, explicitly query

related Datadog logs and error records using the trace ID, span ID, service,

resource name, environment, and same time window. If no related logs or error

records are found, write No measurements found.

Before finalizing, compare the final Datadog alerts table against the full

paginated alert/event sweep. Confirm every production monitor title seen during

the window appears in the table or is explicitly excluded as non-prod.

Datadog Logs Table

Use this table for the most frequent production error-log patterns from the

review window. This is a broad log-health pass and is separate from the alert

cluster rows above, though rows should cross-link when possible.

Start from a Datadog Logs query shaped like:

status:error

Scope it to the review window and production environments in scope. Prefer the

Datadog pattern/clustering view using the log message field as the clustering

pattern field. Group or facet by service, env, and status when available,

and sort by count descending. Review at least the top 10 patterns overall, plus

any additional top pattern per production environment when the global top 10 is

dominated by one env or service.

| Exact Log Pattern | Env / Service | Count / Share | Representative Error | Related Signal / Link | Disposition |

| --- | --- | ---: | --- | --- | --- |

Column rules:

• Exact Log Pattern: exact Datadog clustered pattern or exact raw log message.

Preserve Datadog wildcard syntax such as [wildcard]...[/wildcard]. Do not

paraphrase this column.

• Env / Service: affected production envs and services.
• Count / Share: count in the review window and share if available.
• Representative Error: one exact short representative message, exception

class, or stack/log summary. Avoid long stack traces.

• Related Signal / Link: related Datadog alert row, trace, log query,

incident.io incident, Linear issue, or none found.

• Disposition: use known incident, tracked bug, needs investigation,

expected/test, monitor noise, or unknown.

If a high-volume pattern maps to a failed API route or queue consumer, ensure

the matching Datadog alert row includes the trace/log/error investigation. If a

high-volume pattern has no alert/page row, keep it in this table anyway and

mark it needs investigation or unknown based on evidence.

Output Format

Return valid Markdown only.

• Use valid Markdown syntax for headings, links, and tables.
• Include a space after list markers such as -, *, and 1..
• Close links and parentheses correctly.
• Do not emit malformed tables, dangling backticks, or partially opened code

fences.

• Escape table pipes inside log patterns or error messages when needed.
• If a section would be fragile to format, prefer a plain paragraph over broken

Markdown.