Chatbot Analytics Skill for Claude
Implement AI chatbot analytics and conversation monitoring. Use when adding conversation metrics, tracking AI usage, measuring user engagement with chat, or building conversation dashboards. Activates for AI analytics, token tracking, conversation categorization, and chat performance.
Install
one command, takes just this skill from the repositorynpx skills add https://github.com/curiositech/some_claude_skills --skill chatbot-analyticsThe instruction itself
18 sections, as written by the authorAI Chatbot Analytics
This skill helps you implement analytics for the AI coaching chat feature while maintaining HIPAA compliance.
Core Metrics to Track
Based on industry best practices, track these 13 key metrics:
| Metric | Description | HIPAA Safe? |
|--------|-------------|-------------|
| Total Sessions | Number of chat sessions | Yes |
| Avg Messages/Session | Messages per conversation | Yes |
| Avg Session Duration | Time spent in chat | Yes |
| Engagement Rate | % users who use chat | Yes |
| Completion Rate | Sessions ended naturally | Yes |
| Abandonment Rate | Sessions ended early | Yes |
| Response Time | AI response latency | Yes |
| Token Usage | Total/avg tokens consumed | Yes |
| Error Rate | Failed responses | Yes |
| Fallback Rate | "I don't understand" responses | Yes |
| Topic Categories | What users discuss | Metadata only |
| Sentiment Trend | Emotional direction | Derived only |
| Crisis Triggers | Emergency detection | Metadata only |
HIPAA-Compliant Analytics
What to Track
// Conversation metadata (SAFE)
interface ConversationAnalytics {
id: string;
conversationId: string;
userId: string; // For aggregation, not individual tracking
startedAt: Date;
endedAt: Date | null;
messageCount: number;
userMessageCount: number;
aiMessageCount: number;
totalTokens: number;
inputTokens: number;
outputTokens: number;
category: string; // Derived from metadata flags
outcome: 'completed' | 'abandoned' | 'error' | 'crisis_escalated';
avgResponseTime: number;
hadFallback: boolean;
}
What NOT to Track
// NEVER store these in analytics
interface PROHIBITED {
messageContent: string; // PHI
userQuery: string; // PHI
aiResponse: string; // PHI
specificTopics: string[]; // Could reveal health info
exactSentiment: 'sad'; // Could reveal mental state
}
Implementation Pattern
Tracking Conversation Start
// src/lib/ai/analytics.ts
export async function trackConversationStart(
conversationId: string,
userId: string
): Promise<void> {
await db.insert(conversationAnalytics).values({
id: generateId(),
conversationId,
userId,
startedAt: new Date(),
messageCount: 0,
totalTokens: 0,
category: 'unknown',
outcome: 'in_progress'
});
}
Tracking Message Exchange
export async function trackMessageExchange(
conversationId: string,
tokens: { input: number; output: number },
responseTimeMs: number,
flags: { hadFallback: boolean; hasCrisisIndicator: boolean }
): Promise<void> {
await db
.update(conversationAnalytics)
.set({
messageCount: sql`message_count + 1`,
totalTokens: sql`total_tokens + ${tokens.input + tokens.output}`,
inputTokens: sql`input_tokens + ${tokens.input}`,
outputTokens: sql`output_tokens + ${tokens.output}`,
avgResponseTime: sql`(avg_response_time * (message_count - 1) + ${responseTimeMs}) / message_count`,
hadFallback: flags.hadFallback,
...(flags.hasCrisisIndicator && { outcome: 'crisis_escalated' })
})
.where(eq(conversationAnalytics.conversationId, conversationId));
}
Tracking Conversation End
export async function trackConversationEnd(
conversationId: string,
outcome: 'completed' | 'abandoned' | 'error'
): Promise<void> {
await db
.update(conversationAnalytics)
.set({
endedAt: new Date(),
outcome
})
.where(eq(conversationAnalytics.conversationId, conversationId));
}
Category Detection (Metadata-Based)
Detect conversation categories WITHOUT reading content:
// Categories based on metadata flags from AI response
interface AIResponseMetadata {
usedCopingStrategies: boolean;
usedCrisisProtocol: boolean;
usedCheckInSupport: boolean;
usedGeneralChat: boolean;
requestedClarification: boolean;
}
function deriveCategory(metadata: AIResponseMetadata): string {
if (metadata.usedCrisisProtocol) return 'crisis_support';
if (metadata.usedCopingStrategies) return 'coping_strategies';
if (metadata.usedCheckInSupport) return 'checkin_support';
if (metadata.requestedClarification) return 'clarification';
return 'general_chat';
}
Dashboard Aggregations
Session Metrics
// Get aggregated session stats (HIPAA safe - no individual data)
async function getSessionStats(days: number = 30) {
const since = subDays(new Date(), days);
return db
.select({
totalSessions: count(),
avgMessages: avg(conversationAnalytics.messageCount),
avgDuration: avg(
sql`JULIANDAY(ended_at) - JULIANDAY(started_at)) * 24 * 60`
),
completionRate: sql`
CAST(SUM(CASE WHEN outcome = 'completed' THEN 1 ELSE 0 END) AS FLOAT) /
CAST(COUNT(*) AS FLOAT)
`,
crisisEscalations: sql`
SUM(CASE WHEN outcome = 'crisis_escalated' THEN 1 ELSE 0 END)
`
})
.from(conversationAnalytics)
.where(gte(conversationAnalytics.startedAt, since));
}
Token Usage for Cost Tracking
async function getTokenUsage(days: number = 30) {
const since = subDays(new Date(), days);
const result = await db
.select({
totalTokens: sum(conversationAnalytics.totalTokens),
inputTokens: sum(conversationAnalytics.inputTokens),
outputTokens: sum(conversationAnalytics.outputTokens),
avgTokensPerSession: avg(conversationAnalytics.totalTokens)
})
.from(conversationAnalytics)
.where(gte(conversationAnalytics.startedAt, since));
// Estimate cost (Claude pricing)
const inputCost = (result.inputTokens / 1_000_000) * 3.00; // $3/M input
const outputCost = (result.outputTokens / 1_000_000) * 15.00; // $15/M output
return {
...result,
estimatedCost: inputCost + outputCost
};
}
Category Breakdown
async function getCategoryBreakdown(days: number = 30) {
const since = subDays(new Date(), days);
return db
.select({
category: conversationAnalytics.category,
count: count(),
percentage: sql`
CAST(COUNT(*) AS FLOAT) * 100.0 /
(SELECT COUNT(*) FROM conversation_analytics WHERE started_at >= ${since})
`
})
.from(conversationAnalytics)
.where(gte(conversationAnalytics.startedAt, since))
.groupBy(conversationAnalytics.category)
.orderBy(desc(count()));
}
Alert Configuration
Set up alerts for concerning patterns:
interface AnalyticsAlert {
type: 'crisis_spike' | 'error_spike' | 'abandonment_spike';
threshold: number;
windowHours: number;
action: 'log' | 'email' | 'slack';
}
const alerts: AnalyticsAlert[] = [
{
type: 'crisis_spike',
threshold: 5, // 5+ crisis escalations
windowHours: 24,
action: 'email'
},
{
type: 'error_spike',
threshold: 10, // 10+ errors
windowHours: 1,
action: 'slack'
},
{
type: 'abandonment_spike',
threshold: 0.5, // 50%+ abandonment rate
windowHours: 24,
action: 'log'
}
];
Database Schema
CREATE TABLE conversation_analytics (
id TEXT PRIMARY KEY,
conversation_id TEXT NOT NULL,
user_id TEXT NOT NULL,
started_at TEXT NOT NULL,
ended_at TEXT,
message_count INTEGER DEFAULT 0,
user_message_count INTEGER DEFAULT 0,
ai_message_count INTEGER DEFAULT 0,
total_tokens INTEGER DEFAULT 0,
input_tokens INTEGER DEFAULT 0,
output_tokens INTEGER DEFAULT 0,
category TEXT DEFAULT 'unknown',
outcome TEXT DEFAULT 'in_progress',
avg_response_time REAL DEFAULT 0,
had_fallback INTEGER DEFAULT 0,
FOREIGN KEY (conversation_id) REFERENCES conversations(id),
FOREIGN KEY (user_id) REFERENCES users(id)
);
CREATE INDEX idx_conv_analytics_started ON conversation_analytics(started_at);
CREATE INDEX idx_conv_analytics_user ON conversation_analytics(user_id);
CREATE INDEX idx_conv_analytics_outcome ON conversation_analytics(outcome);
Testing Analytics
describe('Conversation Analytics', () => {
it('tracks session without PHI', async () => {
const analytics = await trackConversationStart('conv-123', 'user-456');
// Verify no PHI is stored
expect(analytics).not.toHaveProperty('messageContent');
expect(analytics).not.toHaveProperty('userQuery');
// Verify metadata is stored
expect(analytics.conversationId).toBe('conv-123');
expect(analytics.messageCount).toBe(0);
});
it('calculates aggregates correctly', async () => {
const stats = await getSessionStats(30);
expect(stats.totalSessions).toBeGreaterThanOrEqual(0);
expect(stats.completionRate).toBeBetween(0, 1);
});
});
Resources
- Chatbot Analytics Guide
- Botpress Analytics
- 13 Core Metrics
- Admin Suite Design:
docs/ADMIN-DEVELOPER-SUITE.md
Other skills for the same job
different authors, same section of the catalogue
Comprehensive spreadsheet creation, editing, and analysis with support for formulas, formatting, data analysis, and visualization. When Claude needs to work with spreadsheets (.xlsx, .xlsm, .csv, .tsv, etc) for: (1) Creating new spreadsheets with formulas and formatting, (2) Reading or analyzing data, (3) Modify existing spreadsheets while preserving formulas, (4) Data analysis and visualization in spreadsheets, or (5) Recalculating formulas
Use this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like \"the xlsx in my downloads\") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.
Picks random winners from lists, spreadsheets, or Google Sheets for giveaways, raffles, and contests. Ensures fair, unbiased selection with transparency.
Query openFDA API for drugs, devices, adverse events, recalls, regulatory submissions (510k, PMA), substance identification (UNII), for FDA regulatory data analysis and safety research.
MATLAB and GNU Octave numerical computing for matrix operations, data analysis, visualization, and scientific computing. Use when writing MATLAB/Octave scripts for linear algebra, signal processing, image processing, differential equations, optimization, statistics, or creating scientific visualizations. Also use when the user needs help with MATLAB syntax, functions, or wants to convert between MATLAB and Python code. Scripts can be executed with MATLAB or the open-source GNU Octave interpreter.
UMAP dimensionality reduction. Fast nonlinear manifold learning for 2D/3D visualization, clustering preprocessing (HDBSCAN), supervised/parametric UMAP, for high-dimensional data.
Creating interactive data visualisations using d3.js. This skill should be used when creating custom charts, graphs, network diagrams, geographic visualisations, or any complex SVG-based data visualisation that requires fine-grained control over visual elements, transitions, or interactions. Use this for bespoke visualisations beyond standard charting libraries, whether in React, Vue, Svelte, vanilla JavaScript, or any other environment.
Access AlphaFold 200M+ AI-predicted protein structures. Retrieve structures by UniProt ID, download PDB/mmCIF files, analyze confidence metrics (pLDDT, PAE), for drug discovery and structural biology.
How to use it
Take curiositech/chatbot-analytics from the repository into ~/.claude/skills for personal
use, or into .claude/skills inside a project.
The agent identifies a skill by the name field in its header. Two skills with the
same name cannot sit side by side — one of them will be ignored.