How Sleuth Uses AI to Track Productivity Beyond DevOps

Software engineering measurement used to be easier to frame. Leaders used to care about things like how often teams deployed, how long it took for changes to code to reach production, how often releases failed, and how quickly teams could fix issues after they happened.

It was in that world that DORA metrics worked. They made it easier for engineering teams to see how well delivery was going by eliminating the need to rely on basic metrics such as the number of lines of code, man-hours worked, and tickets raised.

When it was first released, Sleuth was useful because it helped teams keep an eye on their deployments, make sense of their workflow, and link activities related to software delivery to real results.

AI changes what engineering teams need to measure

Things are different now, and AI-assisted development is changing what we need to keep an eye on. Now, a code change might involve both a human programmer and an AI assistant. There may also be some internal rules or prompts that control how the work is done. While an engineer is debugging, a code agent may be reading production logs, looking at recent deployments, looking at related code, and suggesting a fix before the engineer is even done gathering all the facts by hand.

And that’s what’s causing a whole new kind of visibility problem. Not knowing if software shipped faster isn’t enough these days. Now, engineering leaders need to know how that work was done, what the AI agent saw, what it was doing, and if the whole process was sound, safe, and repeatable.

This change is at the heart of Sleuth’s more recent work on Skills. It’s basically a control plane for engineering teams to manage, share, and enforce AI skills, rules, prompts, and MCP servers.

And that’s what moves Sleuth beyond traditional DevOps measurement and into a whole new problem: how do we manage AI-driven development work before it becomes just another layer of shadow tooling?

Why the old workflows just don’t cut it anymore

Traditional DevOps tooling assumes the main players are people and systems. A human makes a change. CI runs. The change is reviewed. Deployment happens. The observability tools report if the change caused any problems.

But AI agents blur those lines between people and systems big time.

An agent can help write code, check out a failed build, dig through production logs, and suggest a fix in the same session. If it has the necessary permissions, it can also navigate between repositories and assist with code writing. It can also look into a failed build at the same time, read production logs, and suggest a fix. If an agent has a certain skill, they can look at logs, traces, recent deploys, and code that is related to the problem to figure out what went wrong.

It works the same way with code generation. The agent will find it much easier to review the work if they are familiar with the repository’s rules, tests, security needs, and internal norms. It could still write code without that information, but it might need a lot of work to make it look better.

The risk is using it without supervision. It’s easy for local prompts, private plugins, copied files, and MCP configurations that are spread around to spread quickly. Organizations can develop “shadow AI” over time, which is agent activity that affects engineering work but isn’t seen by the leadership, security, or platform teams.

The three stages of AI adoption in engineering

Most organizations seem to move through three broad stages as AI enters engineering work.

Stage one: experimentation

The first stage is the spaghetti phase. Developers try different assistants, prompts, plugins, and MCP servers. Some teams use Cursor. Others use Claude Code, Copilot, Gemini, or Codex. Each group finds its own habits. 

It also creates a mess. Nobody has a complete view of what is running, what each agent can access, or which workflows are becoming part of daily development.

Stage two: standardization

The second stage begins when teams turn useful AI patterns into shared workflows.

Instead of every developer writing a different debugging prompt, the organization can define a standard skill for incident investigation. Instead of each repository carrying its own AI rules, teams can manage those rules from one place and apply them where needed.

This is where Sleuth Skills fits. It treats prompts, rules, MCP configs, and agent skills as structured, versioned assets. Define them once, then distribute them across different AI clients.

Stage three: governed usage

The third stage is governed, organization-wide AI use.

At this point, teams need approval workflows, audit trails, scoped access, RBAC, and runtime visibility. Agent skills and MCP servers are no longer personal productivity experiments. They become part of the software delivery system.

Sleuth’s Skills model is built around that idea. Skills can be managed, versioned, distributed, observed, and governed. Teams can scope them by organization, team, or repository. They can use version pinning, controlled rollout, canary releases, and rollback.

What Sleuth Skills adds to the AI development stack

Sleuth Skills sits in the layer between an organization and the AI agents its developers use.

That layer has three main jobs: manage, distribute, and govern.

• In the manage layer, things like prompts, rules, hooks, agent skills, and MCP configurations are turned into assets that can be saved, tracked, and viewed. 
• With the distribute layer, the right skills are sent to the right teams, tools, and repositories. 
• In the govern layer, approval workflows, audit trails, RBAC, and permissions that are only granted to certain people are added. 

Engineering measurement has a wider job now

The way Sleuth has changed over time shows how engineering management has changed in general. Teams still need to know if they can get software out the door quickly and fix problems as they come up.

Now they also need to know how AI agents help write, test, investigate, and change code.

That makes measuring productivity more than just using DevOps metrics. Leaders need to be able to see what skills agents used, what tools they used, what permissions they had, and which versions were being used by all teams.

For organizations dealing with shadow AI and tool sprawl, visibility could mean the difference between faster development and a delivery process that becomes more difficult to manage with each new agent.

The challenge is no longer whether engineering teams will use AI. It is how organizations will manage, govern, and standardize AI-driven development before agent workflows become another layer of shadow tooling. As AI agents gain access to more systems, repositories, and operational context, visibility becomes just as important as productivity.

Readers who want the fuller discussion, including Dylan Etkin’s explanation of Agent Skills, MCPs, AI governance, autonomous coding agents, and Sleuth’s approach to managing AI development workflows at scale, should check out the webinar featuring Dylan Etkin, CEO and Co-founder of Sleuth.