Giving AI 'eyes'? The story of Saccade, which directly controls web browsers

Imagine this: You are so busy that every morning you ask your AI assistant, “Please find and summarize the three latest news articles to use as meeting materials today.” The AI assistant searches the internet brilliantly, but sometimes it gets frustrated by trying to process too much information at once, leading it to click the wrong button or slowing down. Just as humans glance quickly at only the necessary parts when looking at objects, couldn’t AI look at web pages like we do and pinpoint only the parts it needs to operate?

Saccade is a tool that has emerged to solve this dilemma.

Why is this important?

As AI agents advance, the era where they autonomously manipulate web browsers to find information and process tasks is fast approaching. However, while web pages are intuitive to humans, they are just massive chunks of data to an AI. Many current AI tools attempt to blindly pass all content of a web page to the AI. This is like trying to memorize every single piece of scenery before your eyes, causing an enormous waste of time and cost.

Saccade has transformed this process, similar to the human “saccade” (the physiological phenomenon of moving eyes quickly to focus only on necessary information when looking at objects). By allowing AI to filter out unnecessary information and focus only on essential parts, it has dramatically improved the processing speed and accuracy of AI agents.

Easy to understand: ‘Core route map’ instead of ‘entire map’

Let’s use an analogy. When traveling to a city you’ve never been to, which would be faster: carrying a massive map with every alleyway in the city drawn on it, or having a core subway map that only marks where you need to go?

If the existing method is handing the AI a “map with every alleyway drawn,” Saccade is a method of compressing only the buttons, input fields, and meaningful structures within the page and handing the AI a “core route map” Source: Saccade- gbjapdcoclbdjpcaogmjdbpmnmfgombn - Extpose.

Simply put, when an AI looks at a web page, it boldly omits unimportant advertisements or unnecessary background information and converts only core semantic objects (objects containing the meaning of the data), such as “where to click” or “what is written here,” into a format to deliver Source: Saccade- gbjapdcoclbdjpcaogmjdbpmnmfgombn - Extpose.

Where is it being used?

Using this tool, an AI agent can perform the following tasks:

1. Precise control: It finds and manipulates supported controls such as input fields or buttons within a web page [Source: Saccade — Live Web Truth for AI Agents

   NaN Logic](https://www.nanlogic.com/saccade).

2. Structural understanding: It identifies the logical structure and content of a web page, similar to how humans perceive it with their eyes Source: GitHub - nanlogic/saccade: Closed-loop browser control ….
3. Efficient analysis: It can analyze statistics on its own, such as how many tokens (units of words processed by AI) the AI agent consumed, the cost, and the processing time, by tracking the execution process Source: saccade · PyPI.

In fact, initial test results have confirmed that it processes information at a fast speed that is comparable to existing testing tools Source: ShowHN:Saccade–LivesemanticbrowsertruthforAIagents.

What will happen in the future?

Technology like Saccade will serve as a major bridge for AI agents to evolve from simple “writing tools” to “practical web assistants.” In the future, instead of an AI interpreting the complex code of a browser one by one, it is expected to process tasks much faster and more accurately by receiving only well-organized core information like Saccade does.

We will soon be able to accurately request, “Select and click only the buttons I need on the web page,” instead of telling the AI, “Read the whole web page.” As the precision of AI browsing increases, the repetitive clicking tasks we performed in front of computers might gradually disappear.

References

1. ShowHN:Saccade–LivesemanticbrowsertruthforAIagents
2. Saccade- gbjapdcoclbdjpcaogmjdbpmnmfgombn - Extpose

3. [Saccade — Live Web Truth for AI Agents

   NaN Logic](https://www.nanlogic.com/saccade)

4. GitHub - nanlogic/saccade: Closed-loop browser control …
5. saccade · PyPI