Reducing the Bus Factor with AI-Assisted Engineering

|10 min read

Customer incidents often started the same way: someone noticed missing data in the middle of the night, and nobody knew exactly which integration owned that flow. The immediate reaction was always to ask around until we found one of the few engineers familiar with that customer.

Some integrations lived in Go services, others were n8n workflows, Python scripts, and other were Ruby services. Documentation was inconsistent because it was either half-baked or completely nonexistent. If often lacked details about how an integration was running, whether it was scheduled through cron or Kestra, or whether it was a long-running service serving HTTP endpoints for external calls.

The engineers who had originally built many of them had left the company.

The problem wasn’t writing documentation. It was discovering what should be documented.

Why not RAG?

Retrieval-Augmented Generation (RAG) was an option, but it solved a different problem. I didn’t need conversational search or semantic retrieval.

I wanted a structured document for each customer, listing their integrations and covering the following topics:

The workflow consisted of several repeatable stages with clear inputs and outputs, so agents and skills were a better approach for tackling this problem:

Traditional automation (think shell, Python or other types of scripts) works well when inputs and outputs are known ahead of time. In this case, discovery required interpreting inconsistent repository structures, naming conventions, deployment configurations, and documentation spread across multiple systems. The deterministic parts (calling CLIs, parsing YAML, querying APIs) were implemented as skills, while the LLM was used only where interpretation was required.

The architecture

Agents

Initially, I considered using a single agent responsible for discovery, documentation, and implementation.

I rejected that approach because:

Instead, I developed the agents so that each had a single responsibility and specialized in a specific task.

Splitting the workflow into independent stages also simplified operations. Since each agent consumed and produced well-defined artifacts, failures could be retried from the affected stage without repeating the entire discovery process. That made experimentation easier and reduced the cost of evolving individual components.

I found that the following agents were needed:

discovery

This agent wasn’t expected to produce truth. Instead, it produces evidence that integrations are associated with a customer and creates an evidence pack.

The evidence pack is a YAML file that is:

YAML was chosen because we needed a middle ground between what was implemented and what was documented. More importantly, it served as an explicit contract that decoupled discovery from documentation. We could improve either stage independently without changing the other. Every part of the documentation could be traced back to the YAML and, therefore, to the underlying implementation.

Every conclusion in the evidence pack is backed by the evidence that led to it. Rather than asking reviewers to trust the agent, the system exposes the artifacts it used so engineers can validate or challenge its reasoning. It also became a natural checkpoint. If a downstream stage failed or changed, we could rerun only that stage instead of repeating the entire discovery process.

More importantly, the evidence pack wasn’t just an intermediate data format. It was designed to make the discovery process traceable. Every finding included the repository, version control system, and artifact that led to the conclusion, allowing engineers to verify the evidence instead of treating the agent’s output as a black box.

The evidence pack became the contract between every stage of the pipeline.

Every downstream step assumed the evidence pack could be incomplete. The agent left open questions whenever it couldn’t determine an answer, because forcing it to provide confident answers could lead to incorrect documentation.

Sometimes the uncertainty required us to disagree with the agent because it identified something as needing migration when it was only relevant for documentation. In those cases, we needed to provide the correct answer and explicitly leave it as is.

A wrong answer with high confidence is significantly more dangerous than an explicit “I don’t know”.

Those open questions required an engineer to answer them, or they could later be assigned to someone else for investigation.

Discovery combined repository inspection, deployment metadata, naming conventions, and infrastructure configuration to build confidence that an integration belonged to a given customer.

documentation

Here, the agent reads the evidence pack, applies the documentation template, and produces a Notion page for each customer containing all of its integrations. The resulting page can then be easily shared across the company.

Notion was chosen because it was the company’s central knowledge base. There was no reason to reinvent the wheel, although it could easily be replaced with another documentation platform if needed.

The rest of the agents: work-tracking, implement-go-otel, implement-ruby-kestra, migration-decommission and verification

All the work performed by the agents is reviewed by a human. We can never be 100% certain about the state of the integrations, and the LLMs can hallucinate or make incorrect inferences, which happened occasionally. More importantly, engineers remain accountable for everything we deliver.

What surprised me

Repository organization mattered more than prompts

One of the biggest surprises was that repository organization had a greater impact on the quality of the generated documentation than prompt changes. Consistent naming conventions, clear project structure, and deployment metadata gave the agents reliable signals to reason about. In many cases, improving the engineering ecosystem produced larger gains than refining the prompts.

Infrastructure became another source of truth

I expected source code to be the primary source of truth. Instead, infrastructure turned out to be equally valuable. Deployment manifests, storage buckets, and scheduled jobs often revealed customer ownership and operational behavior that wasn’t obvious from the application code alone.

Uncertainty was more valuable than confidence

The most useful output wasn’t a confident answer, it was a well-defined question.

Good architecture made agents reusable

Separating discovery from documentation also made the agents reusable. Once discovery produced structured evidence, the same output could support migration planning, observability implementation, and future automation without repeating the discovery process.

AI as a tool, not a replacement

After a few iterations, I started noticing repetitive situations where the agents either asked how to proceed or repeatedly failed to perform a task the way I wanted. To address that, I introduced a “lessons learned” capability. The agents began recording timestamped lessons at the end of each run, whether from an agent or a skill. When the same lesson applied to two or more discovery, documentation, or migration tasks, it was promoted into the agent’s capabilities. This created a form of shared memory that improved consistency over time, allowing both the agents and the engineers using them to benefit from previous experience.

Because we needed to add observability, the agents could also help us choose alerts that would detect issues. Metrics alone aren’t useful unless they trigger some action or response. At the end of each observability implementation, we started creating runbooks (documents that help determine wheter an alert requires action) and linking them directly from the alert message. The on-call engineer no longer had to figure out where to start investigating, they could simply click the link and begin.

What I would improve

If this evolved into a long-lived internal platform, I’d invest in automated evaluation datasets, confidence scoring, and incremental discovery to better measure quality and reduce unnecessary processing.

Final thoughts

Separating the workflow into independent stages also simplified operations. Since each agent consumed and produced well-defined artifacts, failures could be retried from the affected stage without rerunning the entire pipeline.

Because we needed to collect knowledge about all customer integrations, the goal wasn’t to automate engineering decisions.

It was to automate the repetitive work involved in discovering and organizing engineering knowledge.

Along the way, the workflow also identified opportunities to improve existing integrations or create technical debt issues for future work. Finally, adding observability gave us better visibility into how integrations behaved in production, allowing the team to detect problems before customers reported them.

Before building the system, documenting a single customer typically required two engineers to spend roughly two and a half weeks tracing integrations across repositories, workflows, and deployment configurations.

After building it, most of that effort shifted from discovery to validation. Engineers spent their time reviewing evidence and making decisions instead of searching for information.

Looking back, the most valuable parts of the system weren’t the prompts or even the LLM. They were the architectural decisions around explicit contracts, traceable evidence, and keeping humans responsible for validating the results. Those choices made the system easier to trust, evolve, and maintain long after the initial implementation.

The bus factor isn’t just about people leaving. It’s about organizations losing the ability to understand and evolve their own systems. While AI helped reduce that risk in this project, the real value came from treating engineering knowledge as something that could be systematically discover, review, and maintain rather than something that existed only in a few people’s heads.

Share this post