Low-Level AI Actions

While ai.step() is the primary method for defining actions using natural language, the ai object also provides lower-level methods for more direct control over specific browser interactions, like clicking and typing into specific targets.

These actions bypass the planner and go straight to Moondream. This can be useful to use AI selectors directly as we refine the agent’s behavior on steps and checks.

​

Action Method Comparison

​

When to use low level actions

They can be useful when:

• You need precise control over a specific interaction.
• Moondream is having trouble clicking a specific target and you want to directly control the prompt

​

ai.click(target)

Directly performs a click action on an element identified by the target description.

await ai.click("the 'Save Settings' button");
await ai.click("the checkbox labeled 'I agree'");

​

ai.type(target, content)

Directly performs a typing action into a specified input element.

await ai.type("the search input field", "Magnitude Testing");
await ai.type("the password box", process.env.USER_PASSWORD); // Use secure methods for sensitive data

​

ai.exec(action)

Execute any action using its JSON representation.

import { ClickIntent } from 'magnitude-core'; // Adjust import path as needed

const specificClick: ClickIntent = {
  variant: 'click',
  target: '#submit-button-id' // Can be a selector or description
};

await ai.exec(specificClick);