Testdriver:parse Agent Skill
Detect all UI elements on screen using OmniParser
Install
one command, takes just this skill from the repositorynpx skills add https://github.com/testdriverai/testdriverai --skill testdriver:parseThe instruction itself
17 sections, as written by the author<!-- Generated from parse.mdx. DO NOT EDIT. -->
Overview
Parse the current screen using OmniParser v2 to detect all visible UI elements. Returns structured data including element types, text content, interactivity levels, and bounding box coordinates.
This method analyzes the entire screen and returns every detected element. It's useful for:
- Understanding the full UI layout of a screen
- Finding all clickable or interactive elements
- Building custom element-based logic
- Debugging what elements TestDriver can detect
- Accessibility auditing
<Note>
Availability: parse() requires an enterprise or self-hosted plan. It uses OmniParser v2 server-side for element detection.
</Note>
Syntax
const result = await testdriver.parse()
Parameters
None.
Returns
Promise<ParseResult> - Object containing detected UI elements
ParseResult
| Property | Type | Description |
|----------|------|-------------|
| elements | ParsedElement[] | Array of detected UI elements |
| annotatedImageUrl | string | URL of the annotated screenshot with bounding boxes |
| imageWidth | number | Width of the analyzed screenshot |
| imageHeight | number | Height of the analyzed screenshot |
ParsedElement
| Property | Type | Description |
|----------|------|-------------|
| index | number | Element index |
| type | string | Element type (e.g. "text", "icon", "button") |
| content | string | Text content or description of the element |
| interactivity | string | Interactivity level (e.g. "clickable", "non-interactive") |
| bbox | object | Bounding box in pixel coordinates {x0, y0, x1, y1} |
| boundingBox | object | Bounding box as {left, top, width, height} |
Examples
Get All Elements on Screen
const result = await testdriver.parse();
console.log(`Found ${result.elements.length} elements`);
result.elements.forEach((el, i) => {
console.log(`${i + 1}. [${el.type}] "${el.content}" (${el.interactivity})`);
});
Find Clickable Elements
const result = await testdriver.parse();
const clickable = result.elements.filter(e => e.interactivity === 'clickable');
console.log(`Found ${clickable.length} clickable elements`);
clickable.forEach(el => {
console.log(`- "${el.content}" at (${el.bbox.x0}, ${el.bbox.y0})`);
});
Find and Click an Element by Content
const result = await testdriver.parse();
// Find a "Submit" button
const submitBtn = result.elements.find(e =>
e.content.toLowerCase().includes('submit') && e.interactivity === 'clickable'
);
if (submitBtn) {
// Calculate center of the bounding box
const x = Math.round((submitBtn.bbox.x0 + submitBtn.bbox.x1) / 2);
const y = Math.round((submitBtn.bbox.y0 + submitBtn.bbox.y1) / 2);
await testdriver.click({ x, y });
}
Filter by Element Type
const result = await testdriver.parse();
// Get all text elements
const textElements = result.elements.filter(e => e.type === 'text');
textElements.forEach(e => console.log(`Text: "${e.content}"`));
// Get all icons
const icons = result.elements.filter(e => e.type === 'icon');
console.log(`Found ${icons.length} icons`);
// Get all buttons
const buttons = result.elements.filter(e => e.type === 'button');
console.log(`Found ${buttons.length} buttons`);
Build Custom Assertions
import { describe, expect, it } from "vitest";
import { TestDriver } from "testdriverai/vitest/hooks";
describe("Login Page", () => {
it("should have expected form elements", async (context) => {
const testdriver = TestDriver(context);
await testdriver.provision.chrome({
url: 'https://myapp.com/login',
});
const result = await testdriver.parse();
// Assert expected elements exist
const textContent = result.elements.map(e => e.content.toLowerCase());
expect(textContent).toContain('email');
expect(textContent).toContain('password');
// Assert there are clickable elements
const clickable = result.elements.filter(e => e.interactivity === 'clickable');
expect(clickable.length).toBeGreaterThan(0);
});
});
Use Bounding Box Coordinates
const result = await testdriver.parse();
result.elements.forEach(el => {
// Pixel coordinates
console.log(`Element "${el.content}":`);
console.log(` bbox: (${el.bbox.x0}, ${el.bbox.y0}) to (${el.bbox.x1}, ${el.bbox.y1})`);
console.log(` size: ${el.boundingBox.width}x${el.boundingBox.height}`);
console.log(` position: left=${el.boundingBox.left}, top=${el.boundingBox.top}`);
});
View Annotated Screenshot
const result = await testdriver.parse();
// The annotated image shows all detected elements with bounding boxes
console.log('Annotated screenshot:', result.annotatedImageUrl);
console.log(`Image dimensions: ${result.imageWidth}x${result.imageHeight}`);
How It Works
- TestDriver captures a screenshot of the current screen
- The image is sent to the TestDriver API
- OmniParser v2 analyzes the image to detect all UI elements
- Each element is classified by type (text, icon, button, etc.) and interactivity
- Bounding box coordinates are returned in pixel coordinates matching the screen resolution
<Note>
OmniParser detects elements visually — it works with any UI framework, native apps, and even non-standard interfaces. It does not rely on DOM or accessibility trees.
</Note>
Best Practices
<AccordionGroup>
<Accordion title="Use find() for targeting specific elements">
For locating and interacting with a specific element, prefer find() which uses AI vision. Use parse() when you need a complete inventory of all elements on screen.
// Prefer this for clicking a specific element
await testdriver.find("Submit button").click();
// Use parse() for full UI analysis
const result = await testdriver.parse();
const allButtons = result.elements.filter(e => e.type === 'button');
</Accordion>
<Accordion title="Filter by interactivity">
Use the interactivity field to distinguish between clickable and non-interactive elements.
const result = await testdriver.parse();
const interactive = result.elements.filter(e => e.interactivity === 'clickable');
const static_ = result.elements.filter(e => e.interactivity === 'non-interactive');
</Accordion>
<Accordion title="Wait for content to load">
If elements aren't being detected, the page may not be fully loaded. Add a wait first.
// Wait for page to stabilize
await testdriver.wait(2000);
// Then parse
const result = await testdriver.parse();
</Accordion>
<Accordion title="Use the annotated image for debugging">
The annotatedImageUrl provides a visual overlay showing all detected elements with their bounding boxes — great for debugging.
const result = await testdriver.parse();
console.log('View annotated screenshot:', result.annotatedImageUrl);
</Accordion>
</AccordionGroup>
Related
- find() - AI-powered element location
- assert() - Make AI-powered assertions about screen state
- screenshot() - Capture screenshots
- Elements Reference - Complete Element API
Other skills for the same job
different authors, same section of the catalogue
Integration with protocols.io API for managing scientific protocols. This skill should be used when working with protocols.io to search, create, update, or publish protocols; manage protocol steps and materials; handle discussions and comments; organize workspaces; upload and manage files; or integrate protocols.io functionality into workflows. Applicable for protocol discovery, collaborative protocol development, experiment tracking, lab protocol management, and scientific documentation.
Analyzes job descriptions and generates tailored resumes that highlight relevant experience, skills, and achievements to maximize interview chances
Generate Excalidraw diagrams from natural language descriptions. Use when asked to "create a diagram", "make a flowchart", "visualize a process", "draw a system architecture", "create a mind map", or "generate an Excalidraw file". Supports flowcharts, relationship diagrams, mind maps, and system architecture diagrams. Outputs .excalidraw JSON files that can be opened directly in Excalidraw.
Build and distribute Expo development clients locally or via TestFlight
Use when you have a written implementation plan to execute in a separate session with review checkpoints
Data structure for annotated matrices in single-cell analysis. Use when working with .h5ad files or integrating with the scverse ecosystem. This is the data format skill—for analysis workflows use scanpy; for probabilistic models use scvi-tools; for population-scale queries use cellxgene-census.
Benchling R&D platform integration. Access registry (DNA, proteins), inventory, ELN entries, workflows via API, build Benchling Apps, query Data Warehouse, for lab data management automation.
Comprehensive molecular biology toolkit. Use for sequence manipulation, file parsing (FASTA/GenBank/PDB), phylogenetics, and programmatic NCBI/PubMed access (Bio.Entrez). Best for batch processing, custom bioinformatics pipelines, BLAST automation. For quick lookups use gget; for multi-service integration use bioservices.
How to use it
Take testdriverai/testdriver:parse from the repository into ~/.claude/skills for personal
use, or into .claude/skills inside a project.
The agent identifies a skill by the name field in its header. Two skills with the
same name cannot sit side by side — one of them will be ignored.