Architecture Agent

Actual AI is a platform that helps engineering teams manage architecture and code quality in an era where both humans and AI agents are writing code. We partnered with their team through HCDE 517 to run a moderated usability study on one feature: the Architecture Agent, which surfaces and applies Architecture Decision Records (ADRs) across a codebase.

Over five sessions with practicing software engineers and delivery managers, we found that people could generally follow the interface's structure, but consistently lost track of what the system had actually done at the moments that mattered most: right after accepting a change, and while that change was syncing in the background.

This case study walks through three of our 29 findings in depth. A link to the full report is at the end.

Before I get into what we found, it's worth explaining what architecture does and what an ADR is.

The Architecture Agent is an AI agent that crawls a codebase, understands and spots technical architecture patterns, and proposes Architecture Decision Records, things like "adopt a consistent testing convention" or "standardize how environment configuration is handled." Architecture Decision Records (ADRs) are the architecture recommendations generated by the architecture agent.

In the product, a user looks at each ADR and accepts, rejects, or archives it. Once someone accepts an ADR, it still has to make its way into the codebase's Context Files. These are the reference docs that AI coding tools (Claude, Cursor, Kiro, you name it) actually read before they touch any code, so that whatever gets built next follows the same rules. Getting an ADR into a Context File means the system generates a pull request, and a human still has to go review and merge it on GitHub.

The team didn't know whether developers could actually understand what the AI was recommending or confidently apply those recommendations without guidance. Our usability study focused on answering two questions:

We conducted five 60-minute moderated usability sessions using the same test organization and sample repository.

Across sessions, our research team rotated between moderator, note-taker, and timekeeper. I independently moderated one complete session, while also contributing observations and synthesis across the study.

We collected both behavioral and attitudinal data:

Every participant successfully located and accepted an Architecture Decision Record. However, accepting the recommendation created a second required step: synchronizing Context Files. Because this status appeared at the bottom of the page, participants interpreted the confirmation message as the end of the workflow and missed the remaining action entirely. Rather than failing to perform an interaction, participants failed to understand the system's current state.

This finding directly answered the first research question. Participants could discover recommendations and act on them, but they could not reliably complete the workflow because the interface failed to communicate that another action was still required. The usability issue wasn't interaction — it was workflow visibility.

Before participants could evaluate Architecture Decision Records, they first had to understand how the Architecture Agent itself worked. Across sessions, participants spent considerable time exploring the interface, asking clarifying questions, and trying to infer relationships between Repositories, Categories, Decisions, Context Files, and pull requests. Rather than learning through the interface itself, participants relied on the moderator's explanations to understand the workflow.

This wasn't simply unfamiliarity with a new product. The Architecture Agent introduces concepts—such as ADRs and Context Files—that are unique to Actual AI and have no established mental model for most developers. Without guidance, participants entered the tasks with an incomplete understanding of how the system was organized. That increased cognitive load before meaningful work could begin.