testdriverai/testdriverai-testdriver:parse
Detect all UI elements on screen using OmniParser
This is a copy. The original lives at testdriverai/testdriver:parse.
npx skills add https://github.com/testdriverai/testdriverai --skill testdriver:parse<!-- Generated from parse.mdx. DO NOT EDIT. -->
Parse the current screen using OmniParser v2 to detect all visible UI elements. Returns structured data including element types, text content, interactivity levels, and bounding box coordinates.
This method analyzes the entire screen and returns every detected element. It's useful for:
<Note>
Availability: parse() requires an enterprise or self-hosted plan. It uses OmniParser v2 server-side for element detection.
</Note>
const result = await testdriver.parse()
None.
Promise<ParseResult> - Object containing detected UI elements
| Property | Type | Description |
|----------|------|-------------|
| elements | ParsedElement[] | Array of detected UI elements |
| annotatedImageUrl | string | URL of the annotated screenshot with bounding boxes |
| imageWidth | number | Width of the analyzed screenshot |
| imageHeight | number | Height of the analyzed screenshot |
| Property | Type | Description |
|----------|------|-------------|
| index | number | Element index |
| type | string | Element type (e.g. "text", "icon", "button") |
| content | string | Text content or description of the element |
| interactivity | string | Interactivity level (e.g. "clickable", "non-interactive") |
| bbox | object | Bounding box in pixel coordinates {x0, y0, x1, y1} |
| boundingBox | object | Bounding box as {left, top, width, height} |
const result = await testdriver.parse();
console.log(`Found ${result.elements.length} elements`);
result.elements.forEach((el, i) => {
console.log(`${i + 1}. [${el.type}] "${el.content}" (${el.interactivity})`);
});
const result = await testdriver.parse();
const clickable = result.elements.filter(e => e.interactivity === 'clickable');
console.log(`Found ${clickable.length} clickable elements`);
clickable.forEach(el => {
console.log(`- "${el.content}" at (${el.bbox.x0}, ${el.bbox.y0})`);
});
const result = await testdriver.parse();
// Find a "Submit" button
const submitBtn = result.elements.find(e =>
e.content.toLowerCase().includes('submit') && e.interactivity === 'clickable'
);
if (submitBtn) {
// Calculate center of the bounding box
const x = Math.round((submitBtn.bbox.x0 + submitBtn.bbox.x1) / 2);
const y = Math.round((submitBtn.bbox.y0 + submitBtn.bbox.y1) / 2);
await testdriver.click({ x, y });
}
const result = await testdriver.parse();
// Get all text elements
const textElements = result.elements.filter(e => e.type === 'text');
textElements.forEach(e => console.log(`Text: "${e.content}"`));
// Get all icons
const icons = result.elements.filter(e => e.type === 'icon');
console.log(`Found ${icons.length} icons`);
// Get all buttons
const buttons = result.elements.filter(e => e.type === 'button');
console.log(`Found ${buttons.length} buttons`);
import { describe, expect, it } from "vitest";
import { TestDriver } from "testdriverai/vitest/hooks";
describe("Login Page", () => {
it("should have expected form elements", async (context) => {
const testdriver = TestDriver(context);
await testdriver.provision.chrome({
url: 'https://myapp.com/login',
});
const result = await testdriver.parse();
// Assert expected elements exist
const textContent = result.elements.map(e => e.content.toLowerCase());
expect(textContent).toContain('email');
expect(textContent).toContain('password');
// Assert there are clickable elements
const clickable = result.elements.filter(e => e.interactivity === 'clickable');
expect(clickable.length).toBeGreaterThan(0);
});
});
const result = await testdriver.parse();
result.elements.forEach(el => {
// Pixel coordinates
console.log(`Element "${el.content}":`);
console.log(` bbox: (${el.bbox.x0}, ${el.bbox.y0}) to (${el.bbox.x1}, ${el.bbox.y1})`);
console.log(` size: ${el.boundingBox.width}x${el.boundingBox.height}`);
console.log(` position: left=${el.boundingBox.left}, top=${el.boundingBox.top}`);
});
const result = await testdriver.parse();
// The annotated image shows all detected elements with bounding boxes
console.log('Annotated screenshot:', result.annotatedImageUrl);
console.log(`Image dimensions: ${result.imageWidth}x${result.imageHeight}`);
<Note>
OmniParser detects elements visually — it works with any UI framework, native apps, and even non-standard interfaces. It does not rely on DOM or accessibility trees.
</Note>
<AccordionGroup>
<Accordion title="Use find() for targeting specific elements">
For locating and interacting with a specific element, prefer find() which uses AI vision. Use parse() when you need a complete inventory of all elements on screen.
// Prefer this for clicking a specific element
await testdriver.find("Submit button").click();
// Use parse() for full UI analysis
const result = await testdriver.parse();
const allButtons = result.elements.filter(e => e.type === 'button');
</Accordion>
<Accordion title="Filter by interactivity">
Use the interactivity field to distinguish between clickable and non-interactive elements.
const result = await testdriver.parse();
const interactive = result.elements.filter(e => e.interactivity === 'clickable');
const static_ = result.elements.filter(e => e.interactivity === 'non-interactive');
</Accordion>
<Accordion title="Wait for content to load">
If elements aren't being detected, the page may not be fully loaded. Add a wait first.
// Wait for page to stabilize
await testdriver.wait(2000);
// Then parse
const result = await testdriver.parse();
</Accordion>
<Accordion title="Use the annotated image for debugging">
The annotatedImageUrl provides a visual overlay showing all detected elements with their bounding boxes — great for debugging.
const result = await testdriver.parse();
console.log('View annotated screenshot:', result.annotatedImageUrl);
</Accordion>
</AccordionGroup>
Take testdriverai/testdriverai-testdriver:parse from the repository into ~/.claude/skills for personal
use, or into .claude/skills inside a project.
The agent identifies a skill by the name field in its header. Two skills with the
same name cannot sit side by side — one of them will be ignored.