Act: Click action does not perform the action

[nimeshgurung]

Currently I am trying to use stagehand to see if it can be used to write automation test. The eventual goal is to have a descriptive step by step test plan converted to actual test's that are ran by some form of agentic setup, essentially replicating QA specialist.

But I have run into issue's using act, observe and extract seem to work fine. For example something as simple as:

await stagehand.page.act('Click on the menu item Account');

It's a pretty simple action which for one reason or another does not work. On the documentation it is suggested to use stagehand as back up when playwright fails which I get. But still does not explain why such a simple action would not work. Am I missing something here. Would be nice if there was a way to provide an expected result of an action and have stagehand retry the action couple of times.

More info:
I am running stagehand with AiSdk example along side vercel sdk and azue open ai and gpt4o. I am seeing an error that says

Cannot read properties of undefined (reading typename) which happens in zod which I can trace back to AISdkClient.createChatCompletion 

https://github.com/browserbase/stagehand/blob/main/examples/external_clients/aisdk.ts#L100

[adrianmaciuc]

I have the same error on act. I use local.

TypeError: Cannot read properties of undefined (reading 'typeName')

2025-02-05T19:47:38.747Z::[stagehand:act] running act  
2025-02-05T19:47:38.756Z::[stagehand:action] checking action cache  
2025-02-05T19:47:38.757Z::[stagehand:base_cache] Lock acquired  
2025-02-05T19:47:38.758Z::[stagehand:base_cache] Lock released  
2025-02-05T19:47:38.758Z::[stagehand:action] action cache miss  
2025-02-05T19:47:38.765Z::[stagehand:action] model does not support vision but useVision was not false. defaulting to false.  
2025-02-05T19:47:38.765Z::[stagehand:action] running / continuing action  
2025-02-05T19:47:38.765Z::[stagehand:action] processing DOM  
2025-02-05T19:47:40.403Z::[stagehand:action] looking at chunk  
2025-02-05T19:47:40.428Z::[stagehand:action] error performing action - b Cannot read properties of undefined (reading 'typeName') TypeError: Cannot read properties of undefined (reading 'typeName')    
at parseDef (file:///home/adrian/codeme/ai/stagehand-project/stagethree/node_modules/zod-to-json-schema/dist/esm/parseDef.js:49:46)

[andrewlu0]

Also getting the same error with Vercel AI SDK and Azure OpenAI provider set up. Here is the full log if useful

2025-02-05T21:49:36.838Z::[stagehand:action] error performing action - b
 Cannot read properties of undefined (reading 'typeName')
 TypeError: Cannot read properties of undefined (reading 'typeName')
    at parseDef (file:///Users/andrewluo/Desktop/dealpage/node_modules/.pnpm/[email protected][email protected]/node_modules/zod-to-json-schema/dist/esm/parseDef.js:49:46)
    at zodToJsonSchema (file:///Users/andrewluo/Desktop/dealpage/node_modules/.pnpm/[email protected][email protected]/node_modules/zod-to-json-schema/dist/esm/zodToJsonSchema.js:19:18)
    at zodSchema (file:///Users/andrewluo/Desktop/dealpage/node_modules/.pnpm/@[email protected][email protected]/node_modules/@ai-sdk/ui-utils/src/zod-schema.ts:10:5)
    at asSchema (file:///Users/andrewluo/Desktop/dealpage/node_modules/.pnpm/@[email protected][email protected]/node_modules/@ai-sdk/ui-utils/src/schema.ts:67:38)
    at file:///Users/andrewluo/Desktop/dealpage/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/ai/core/prompt/prepare-tools-and-tool-choice.ts:52:25
    at Array.map (<anonymous>)
    at prepareToolsAndToolChoice (file:///Users/andrewluo/Desktop/dealpage/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/ai/core/prompt/prepare-tools-and-tool-choice.ts:43:26)
    at fn (file:///Users/andrewluo/Desktop/dealpage/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/ai/core/generate-text/generate-text.ts:250:12)
    at file:///Users/andrewluo/Desktop/dealpage/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/ai/core/telemetry/record-span.ts:18:28
    at Object.startActiveSpan (file:///Users/andrewluo/Desktop/dealpage/node_modules/.pnpm/[email protected][email protected][email protected]/node_modules/ai/core/telemetry/noop-tracer.ts:21:14)

Just trying a basic act command like this:

  await page.goto(`https://${workspace}.slack.com`);
  await page.waitForLoadState("networkidle");
  await page.act("click the email input");

[kamath]

Hey! Sorry you guys are dealing with this -- can you provide a script that we can repro?

@andrewlu0 I was able to repro your error, and I believe it should be fixed in our v1.11 release! Try running this main function on a create-browser-app quickstart and lmk how it goes?

export async function main({
  page,
  context,
  stagehand,
}: {
  page: Page; // Playwright Page with act, extract, and observe methods
  context: BrowserContext; // Playwright BrowserContext
  stagehand: Stagehand; // Stagehand instance
}) {
  async function actWithCache(instruction: string) {
    // Observe the page and return the action to execute
    const results = await page.observe({
      instruction,
      onlyVisible: false, // Faster/better/cheaper, but uses Chrome a11y tree so may not always target directly visible elements
      returnAction: true, // return the action to execute
    });
    console.log(chalk.blue("Got results:"), results);

    // You can cache the playwright action to use it later with no additional LLM calls :)
    const actionToCache = results[0];
    console.log(chalk.blue("Taking cacheable action:"), actionToCache);

    // OPTIONAL: Draw an overlay over the relevant xpaths
    await drawObserveOverlay(page, results);
    await page.waitForTimeout(1000); // Can delete this line, just a pause to see the overlay
    await clearOverlays(page);

    // Execute the action
    await page.act(actionToCache);
  }

  const workspace = "stagehand-dev";
  await page.goto(`https://${workspace}.slack.com`);
  await page.waitForLoadState("networkidle");
  await actWithCache(`type [email protected] into the email input`);
  await page.waitForTimeout(5000);
}