Experts point out that a compact representation of page state and continuous feedback are more critical to agent success than simply upgrading to more advanced language models.
Despite recent advances in language models, AI agents designed for web navigation continue to fail at basic workflows. The industry's current trend has been to address these limitations through improvements to the models themselves, such as sharper vision, longer contexts, and smarter planning. However, market analysis indicates that the primary performance bottleneck lies not in the model's cognitive capacity, but in the interface connecting it to the browser.
According to Kushan Raj, a machine learning engineer at ARK, developing browser agents requires a focus on three fundamental pillars: what the model sees, what it can execute, and what it learns from the process. Raj, who is also a founding engineer at Sarvam AI, where he built a real-time voice AI stack, argues that the solution requires building an adequate runtime for these agents.
Rather than feeding a raw data dump of the page to the model, the suggested approach involves a compact representation of the page state. Additionally, the actions executed by the agent should rely on fast, stable identifiers, avoiding the inefficiency of a single click per call. The third critical point is replacing a binary success-or-failure evaluation system at the end of a task with a step-by-step feedback mechanism during execution.
Initial tests demonstrated that simply changing this interaction interface was enough for the same model to go from a state of confusion to correctly executing multiple steps, even on web pages considered hostile. The evidence suggests that optimizing the browser state provided to the model acts as a much more effective performance lever than simply swapping in a more robust AI foundation.
Browser agents fail primarily due to poor interfaces connecting them to the browser, not because of limitations in the language model's cognitive capacity. The performance bottleneck lies in how the model interacts with the page.
Instead of feeding a raw data dump of the page to the model, developers should provide a compact representation of the page state. Actions should also rely on fast, stable identifiers rather than inefficient single clicks.
Performance should be evaluated using a step-by-step feedback mechanism during execution, rather than relying on a binary success-or-failure evaluation at the very end of the task.