The Dual Goldmines of E2E Automation: Driving Test Agents with Telemetry Data and Existing Page Objects

Slack Engineering’s recent exploration into agentic end-to-end (E2E) testing highlights a critical industry shift. By running over 200 workflows, their team demonstrated that AI agents can successfully change the testing paradigm from enforcing rigid paths to verifying high-level goals. However, the experiment also exposed major bottlenecks: agentic runs cost an astronomical $15 to $30 per execution, take upwards of 10 minutes, and suffer from high token accumulation.

To transition agentic testing from a costly novelty into a viable production framework, we must connect it to two massive, pre-existing organizational goldmines: Real User Monitoring (RUM) telemetry and existing Page Object Models (POMs). By grounding autonomous agents in actual production paths, wrapping existing engineering abstractions as toolsets, and evolving classic Model-Based Testing (MBT) principles, we can build an exploratory testing stack that is cost-effective, highly realistic, and deeply interpretable.

The Architectural Evolution: From GraphWalker to Dynamic Agents

To understand the true potential of AI agents, we must look at a testing methodology the industry has explored for years: Model-Based Testing. Popularized by engineering teams like Spotify using tools such as GraphWalker, MBT introduced the powerful concept of non-deterministic determinism. Testers explicitly define the application as a directed graph where nodes represent UI states and edges represent actions. The test runner then walks this graph dynamically, discovering edge-case bugs that rigid scripts miss.

The maintenance of these explicit state models is traditionally the primary bottleneck; engineers spend weeks mapping out every single UI permutation in structural files.

LLM agents completely evolve this paradigm. An agent treats the application's live Document Object Model (DOM) or accessibility tree as an implicit, self-generating graph. Instead of an engineer hand-crafting the nodes and edges, the agent builds a mental map of the interface dynamically to achieve a high-level instruction. It provides the same non-deterministic exploration pioneered by GraphWalker, but removes the manual model maintenance entirely.

The Critical Need for "Expert Interpreter" Reporting

Because the agent's pathing is fluid, standard pass or fail assertions are no longer sufficient. For agentic testing to be valuable, the runner must generate a highly structured Action Signature Map—akin to an execution graph trace—for human review. This report must explicitly log:

The exact sequence of normalized tool calls and UI actions taken across the dynamic graph.
State deviations and unexpected challenges encountered like API latency or dynamic layout shifts.
The alternative paths attempted to bypass those challenges.

This transforms the QA professional's role from writing repetitive scripts to acting as an expert interpreter, evaluating whether the agent's self-discovered workarounds represent viable user behavior or hidden user experience flaws.

Goldmine #1: Mining RUM and Session Replay for Agent Intuition

Left completely unconstrained, an LLM agent will explore an application's implicit graph inefficiently, inflating execution times and token counts. The solution lies directly within our existing observability stacks.

For years, the industry has treated telemetry as a graveyard for post-mortem analysis. Observability giants like Datadog, FullStory, and LogRocket have recently introduced AI features, but their scope remains restricted to passive summaries, sentiment analysis, and friction detection. This is a massive missed opportunity. Telemetry platforms sit on a goldmine of production event streams that should actively drive the testing lifecycle, yet existing platforms fail to cross the chasm to convert this data into executable, self-healing test suites.

We can achieve production-guided agentic testing by parsing raw user sessions into sequential markdown user journeys.

[Session Replay / RUM Data] 
       │
       ▼ (Extract Top 10% User Paths)
[LLM Context / System Prompt] 
       │
       ▼ (Informed Goal-Seeking)
[Playwright MCP Agent]

Feeding these high-frequency user paths into the agent's context as few-shot examples provides the model with an immediate intuition of realistic application usage. Instead of guessing how to navigate a complex workspace, the agent prioritizes paths that mirror actual customer behavior, significantly reducing random exploration and optimization overhead.

This bridges the gap between how users actually use an application and how it is tested. When a product update occurs, the agent does not just hunt for broken elements; it tests whether the top 10% of real-world user paths are still intact, adapting dynamically if a button moved or a workflow evolved.

Goldmine #2: Turning Page Object Models into Agentic Skills

Many missing engineering organizations hesitate to adopt AI testing because it implies abandoning years of investment in structured Page Object Models (POMs). Interestingly, Slack’s experiment noted that their code-generation framework degraded significantly (~48% failure rate on complex tasks) because the model struggled to programmatically reuse existing page object abstractions when writing static code lines.

We can solve this problem by flipping the hierarchy on its head: instead of asking the AI to write code using a POM, we wrap the existing POM inside the Model Context Protocol (MCP) server as explicit tools.

Instead of generic actions like clicking raw text or selectors, a method like BillingPage.selectPremiumPlan() is exposed directly to the live agent as a JSON tool schema.

This architectural shift completely alters the economics and execution of agentic QA:

Token Reduction: The agent no longer requires massive DOM retransmissions on every turn to discover what to click. It simply reviews a lightweight list of available text-based tools, dropping input token costs by orders of magnitude.
Balanced Execution: The test suite achieves component-level determinism through verified engineering abstractions, while retaining workflow-level non-determinism through the agent's goal-seeking reasoning.

The Pragmatic Blueprint: Put the Tool into the Tester's Hands

To bring this concept out of theory and into production, we must design how an expert tester actually triggers and utilizes this platform. Instead of running these expensive tests blindly on every basic code commit, the tester deploys the agent strategically across three distinct operational modes.

A. The New Feature Greenfield Run (Acceptance Criteria + POM)

When a completely new feature drops, an automated test suite does not exist yet, but the basic Page Objects for the new components have been drafted.

The Workflow: The tester feeds the feature's natural language Acceptance Criteria (Gherkin or user stories) directly into the agent alongside the new POM methods.
The Execution: The agent parses the criteria, maps the intent to the available page object tools, and attempts to drive the application to successful resolution.
The Deliverable: The agent returns a structured path log. If it succeeds, the tester can instantly export that successful path to generate a permanent, deterministic Playwright script. If it fails, the tester immediately uncovers missing edge cases or requirement gaps before code is merged.

B. The Feature Extension Run (RUM + POM)

When modifying or extending a mature feature, the primary risk is breaking complex, hidden user paths that engineers did not anticipate.

The Workflow: The tester initiates a regression loop by seeding the agent with the existing POM toolset and the top 10% of real-world user paths pulled from RUM data.
The Execution: The agent exercises the extended feature using the exact behavior signatures of real production users.
The Deliverable: The agent delivers a delta report to the expert tester detailing whether it completed the goal using historical user paths, and flagging any unexpected friction points or layout shifts caused by the new UI extension.

C. Continuous Production Smoke Testing (The Autonomous Monitor)

Because agentic runs carry an API cost, using them hourly in a CI pipeline is inefficient. However, running them as a scheduled, high-level production monitoring layer is an exceptional use of resources.

The Workflow: Every morning or following a production deployment, a scheduled worker triggers the agent with high-level business goals (e.g., "Complete a premium checkout workflow").
The Execution: The agent navigates production using a test account, leveraging the robust error-recovery of its LLM planning loop to bypass minor visual changes or transient network hiccups that would instantly flake out a rigid, traditional script.
The Deliverable: Instead of a simple pass/fail alert, the team receives an exploratory health report detailing exactly how smoothly critical paths are running, alerting engineers only if a goal becomes fundamentally unachievable.

Protocol Efficiency: MCP vs. CLI

Slack’s benchmarking data revealed a stark reliability and performance gap between execution models:

Agent + Playwright CLI: Failed 12% to 20% of the time, slowed down by shell-spawning overhead, session instability, and the need to repeatedly rebuild application state from scratch.
Agent + Playwright MCP: Maintained a 0% to 12% failure rate and consumed far fewer tokens.

The Model Context Protocol (MCP) architectural pattern accounts for this difference. While a CLI framework forces the agent to act through an isolated terminal abstraction, an MCP server exposes the browser state as a living, bi-directional API directly inside the model's tool-calling loop. MCP combines the action execution and state evaluation into a single round trip, minimizing the conversational turns that drive up API latency and failure rates.

A Playbook for Slashing Agentic Testing Costs

To transition agentic testing into a repeatable testing layer, the cost per run must decrease by orders of magnitude. Three concrete strategies can achieve this:

Context Compaction

The primary driver of token cost is the retransmission of massive DOM and accessibility trees on every single turn. Implementing custom middleware on the MCP server to filter out un-interactive elements, raw SVGs, and redundant structural divs ensures the LLM only processes critical interactive nodes like buttons, inputs, and anchors with explicit ARIA labels.

Aggressive Prompt Caching

Because system prompts, core tool definitions, and initial application landing states remain static across test cycles, structuring the API calls to maximize prompt caching allows teams to take advantage of substantial token discount structures offered by model providers.

The Hybrid Warm Start

Agents should not waste expensive tokens performing trivial, deterministic setup actions. Engineers can use standard, cost-free Playwright scripts to handle authentication, account setup, and baseline navigation. Once the application reaches the targeted complex state, the active browser context is handed over to the MCP agent for exploratory testing.

Conclusion

Agentic testing represents the next logical phase of end-to-end automation. By treating the UI as a dynamic state machine, guiding exploration with real-world RUM data, and utilizing native protocol integrations like MCP alongside existing Page Objects, engineering teams can build a self-healing, highly analytical QA layer that expands test coverage without introducing high maintenance costs. Turn passivity into activity: the future of testing belongs to those who turn production logs and existing code assets into automated intent.