That Gap Is Still Widening, But The Bottleneck Was Never Engineering

In my previous post, The Gap Is Widening and It’s Not Slowing Down, I focused on the growing divide between individuals and teams that have embraced Generative AI and those that have not. That divide is real, measurable, and growing faster than many people seem willing to acknowledge. I’m not walking any of that back.

But after watching organizations over the last year attempt to integrate AI into their engineering practices, I’m increasingly convinced the widening gap isn’t actually about AI – at least not entirely. The technology is only exposing something that has existed for decades. The bottleneck was never engineering. It was never software development. It was never the people building things.

The bottleneck has almost always been the machinery *surrounding* the people building things: the approvals, the reporting structures, the committees, the prioritization processes, the disconnected leadership layers, the bureaucracy, the politics – the endless collection of organizational systems that somehow manage to consume vast amounts of energy while producing remarkably little forward progress. Generative AI didn’t create this problem. It simply made it impossible to hide.

Individual Engineers Are Experiencing a Massive Productivity Expansion

There is very little debate left about whether Generative AI increases the productivity of individual contributors. We’ve moved well past that question. The debate today isn’t whether productivity gains exist – it’s about how *much* gain exists and, more importantly, who is actually capable of capturing it.

Engineers today can create prototypes in hours that once required days or weeks. Architectural alternatives can be explored in an afternoon instead of consuming entire sprints of spike work. Documentation can be generated, revised, and maintained at speeds that would have seemed unrealistic even five years ago. Testing frameworks, infrastructure automation, deployment pipelines, and application scaffolding can all be stood up dramatically faster than before.

Even beyond code generation, AI has become an accelerator for thinking itself. It provides rapid feedback loops, architectural critiques, alternative approaches, and research capabilities that allow engineers to move through uncertainty faster than they ever have before. An engineer operating effectively with modern tooling can often accomplish what previously required several engineers. A small team can frequently deliver what once demanded a much larger one. This isn’t speculation anymore – we’re watching it happen every day.

Yet despite these gains, many organizations report only marginal improvements in overall delivery speed. Why? Because software development was never the slowest part of the process.

The Bottleneck Was Hiding Somewhere Else

For years, organizations convinced themselves that software engineers were the constraint. The logic seemed simple enough: if projects are late, development must be slow; if features take too long, engineering capacity must be insufficient; if delivery struggles, more developers must be needed. So organizations hired more engineers, added process around them, and watched delivery timelines stay roughly the same.

Then AI dramatically accelerated the development side of the equation – and something interesting happened. The overall system didn’t accelerate proportionally. Instead, the delays became easier to identify. Features completed quickly sat for weeks waiting for approval. Product decisions that took months were backed by implementation work that took days. Architecture reviews became calendar-management exercises instead of engineering exercises. Governance processes consumed more time than development itself, and procurement delays stalled technology adoption before it could even begin.

The moment development became faster, every other inefficiency suddenly became visible. The tide went out and revealed the rocks – and there were far more rocks than most organizations were prepared to acknowledge.

We’ve Been Trying To Solve This Problem For Over A Century

One of the most frustrating aspects of this situation is that the underlying problem isn’t new. Some of the most influential thinkers in management spent their entire careers attempting to solve exactly these issues, and we largely ignored them.

Chief among them was W. Edwards Deming. His work fundamentally reshaped manufacturing, quality management, and organizational thinking throughout the twentieth century. His influence helped transform post-war Japanese manufacturing and directly shaped the practices that would eventually become Lean methodology and the Toyota Production System. One of Deming’s most important observations was that organizations routinely blame individuals for failures that are actually caused by systems — the worker gets blamed, the engineer gets blamed, the frontline employee gets blamed, while the actual system producing poor outcomes remains untouched and unexamined.

Deming repeatedly argued that management’s primary responsibility was improving the system itself – not creating more reports, not creating more oversight, not creating more bureaucracy, but actually improving the system. This remains one of the most ignored lessons in modern business. When productivity stalls, organizations add process. When communication breaks down, they add meetings. When delivery slows, they add approvals. When uncertainty increases, they add governance layers. The response is almost always additional complexity, rarely simplification – yet simplification is often exactly what’s needed.

The Toyota Way Was Never About Manufacturing

One of the most persistently misunderstood management concepts in business is the Toyota Production System and the principles described in *The Toyota Way*. Organizations study Toyota and immediately focus on manufacturing techniques, kanban boards, and production flow. That completely misses the point.

The true innovation was not manufacturing. It was the relentless pursuit of removing waste from systems. Toyota became exceptional because it continuously questioned every activity that consumed effort without creating value – every unnecessary handoff, every unnecessary delay, every unnecessary approval, every unnecessary process step. Everything was examined through the lens of whether it actually moved something of value forward. If it didn’t, it was a candidate for elimination.

Modern software organizations often claim to embrace these ideas while operating with approval chains that require six or seven layers of sign-off, organizational structures where decisions travel further than the code itself, and workflows designed to optimize reporting while actively damaging delivery speed. The language of Lean has become extremely popular in tech. The discipline required to actually implement it remains genuinely rare. There’s a meaningful difference between saying “we practice continuous improvement” and operating a system that systematically identifies and eliminates its own waste – most organizations are doing the former and calling it the latter.

AI Is Exposing Management Debt

The software industry talks constantly about technical debt, and rightly so – I’ve spent decades fighting it. But I’m increasingly convinced that many organizations suffer more from *management debt* than technical debt, and management debt is considerably harder to see from the inside.

Management debt accumulates when organizations create layers of process that never get removed. It accumulates when reporting structures expand indefinitely, when approval chains grow with every re-org, when every novel problem gets solved by introducing another committee, another meeting, another workflow, or another governance layer. Over time these accumulate into a dense friction system that surrounds every team trying to build and ship something. Unlike technical debt, management debt is often invisible to leadership because leadership frequently created it – and organizations don’t typically build mechanisms for evaluating whether the management decisions made five years ago are still earning their overhead.

Generative AI is now exposing these accumulated liabilities with uncomfortable clarity. If engineers can produce ten times more output and delivery only improves ten percent, leadership should not be asking what’s wrong with the engineers. They should be asking what’s wrong with the system. The answer may be uncomfortable. It may involve examining years of accumulated organizational decisions, questioning structures that have become politically entrenched, and acknowledging that the bureaucracy itself is the liability. But that’s where the actual solution lives.

Systemic Thinking Matters More Than Ever

One of the most valuable disciplines organizations can adopt today is genuine systemic thinking – not as a framework to be installed and presented to the board, but as an actual way of seeing how the organization produces its outputs.

The reason it matters comes down to a simple but uncomfortable idea: organizations are systems, and systems produce exactly what they are designed to produce. Not what leadership intends. Not what the org chart implies. What the actual system, with its real incentives and real workflows, is built to produce. Many organizations want innovation while designing systems optimized for risk avoidance. They want speed while designing systems that optimize for approval coverage. They want accountability while designing systems optimized for blame diffusion. They want creativity while designing systems that reward conformity and punish variance.

The outputs shouldn’t be surprising – the system is behaving exactly as designed. Generative AI doesn’t alter this reality. If anything, it amplifies it. The faster individual contributors become, the more visible systemic dysfunction becomes. You can’t paper over a broken approval process with faster code generation. You just end up with more finished work sitting in queues.

The Competitive Advantage Isn’t AI

Here’s something I genuinely believe will be borne out over the next five years: the next generation of competitive advantage is unlikely to come from simply adopting AI. Everyone will eventually have access to similar models. Everyone will have copilots, agents, and increasingly capable automation. The models will keep getting better and access will continue to become more democratic. Access is not a moat.

The differentiator will be organizational capability – specifically, whether the organization can actually move. Can it make decisions quickly? Can it remove friction from its own processes? Can it empower teams to act without running everything through three layers of approval? Can it identify waste and eliminate it rather than building process around it? Organizations that can answer yes to these questions will compound the productivity gains AI provides and experience something genuinely transformative. Organizations that answer no will continue wondering why expensive AI investments fail to produce the results they see in press releases and conference talks – and they’ll blame the technology.

The Answers Already Exist

What’s remarkable about all of this is that very few of these ideas are new. Deming wrote extensively about them. Lean practitioners have written extensively about them. Systems thinkers like Peter Senge have written extensively about them. Toyota demonstrated them repeatedly over decades. The playbook already exists. It just requires organizational will to actually use it.

Generative AI simply raises the stakes. For decades, organizations could survive despite bureaucratic inefficiencies because software creation itself was difficult enough that organizational dysfunction stayed hidden behind the sheer complexity of engineering work. That cover is disappearing fast. The engineering side of the equation is accelerating rapidly, and the remaining constraints are becoming impossible to ignore.

If I had to estimate — and I’m willing to commit to this number — I’d say the overwhelming majority of organizations, probably 90% or more, still carry enough management debt, process debt, and bureaucratic drag to prevent them from realizing even half of the value Generative AI could deliver. The technology is arriving right on schedule. The organizations are not.

Organizations that fail to introspect, simplify, and dismantle their accumulated management structures will realize only a fraction of what’s possible. Organizations that embrace systems thinking, Lean principles, continuous improvement, and genuine organizational simplification will unlock extraordinary advantages — not because AI magically transformed their business, but because they finally removed the barriers that had been slowing them down all along.

The gap is widening. But the bottleneck was never engineering. It was management. And management now has nowhere left to hide.

Further Reading

These are the thinkers and resources worth going deep on if you want to move beyond reading about these ideas and actually do something about them.

  • W. Edwards Deming — Start with his Fourteen Points for Management, then read *Out of the Crisis*. His work is the foundation for almost everything else on this list.
  • The Deming Institute — The most accessible ongoing resource for understanding and applying Deming’s system of profound knowledge in modern organizations.
  • *The Toyota Way* by Jeffrey Liker — The definitive English-language treatment of Toyota’s actual management philosophy, not just the production tools. The tools without the philosophy are just theater.
  • Toyota Production System — Understanding the origins and evolution of TPS is worthwhile context before diving into the derivative frameworks that have followed it.
  • Lean Enterprise Institute — Practical, applied Lean thinking for people who want to actually implement rather than just read about it.
  • *The Fifth Discipline* by Peter Senge — The foundational text on systems thinking in organizations. If you read one book from this list, make it this one.
  • *The Goal* by Eliyahu Goldratt — Theory of Constraints explained through a novel. Surprisingly readable and genuinely transformative for how you see bottlenecks. The production setting feels dated; the ideas do not.
  • Kaizen / Continuous Improvement — Understanding what kaizen actually means in practice versus how the word gets casually deployed in tech organizations is worth the time. The gap between those two things is significant.

12 Days into 2026 – Status & update of my projects.

As mentioned in my end of 2025 post, there were several projects I intended to start, continue, or finish up in 2026. This is a quick status and an additional few things I’ve started working on.

https://interlinedlist.com – This is live. Albeit not very built out, but the start is live. You can even sign up for an account and play around with it. A caveat though, this is pre-alpha, and not very feature rich at all and I don’t have the programmer oriented tasks integrated in yet. It’s just the micro-blogging. But hey, some progress is better than no progress!

https://www.datadiluvium.com – This is still live and I’ve got some changes to push, but they’re breaking, so as soon as I dig up a bit of free coder time those will get resolved and I’ll get the latest tweaks pushed. I also need to get some basic documentation and also something posted here on the blog about what you can do, or would want to do with it.

dashingarrivals – First commit isn’t done. Working on it, it’ll be coming soon per the pervious post.

collectorstunetracker – No current update. Albeit I need to do something about my collection, cuz it has only grown and not shrunk! 🤘🏻

Writing – This is one of many posts. I wrote this one too, not some AI nonsense.

New News About New Projects

I discussed it a while back on one of the social medias (I’m on Threads, Mastadon, and Blue Sky – join me there) the idea of doing some videos on algorithms and data structures. The intent is to put together some videos similar to these “Coding with AI: A Comparative Analysis“. It could be good, and I’d tack onto the algorithms, data structures some additional lagniappe via concurrency patterns I previously wrote about. If interested, subscript to the blog here or subscribe to my YouTube at https://www.youtube.com/adronhall.

Until then, keep thrashing the code! 🤘🏻

Architects Evolving: How AI Is Reshaping the Role

In my last post, I broke down the many kinds of Software Architects, the ivory tower variety, the hands-on technical ones, and the practice architects who design how design happens. But here’s the thing: the role is shifting. The same forces that transformed how we build, deploy, and ship software are transforming what it even means to design (or “architect” as many say verb-izing the word) software.

We’re now entering an era where architects and principal engineers aren’t just bridging teams and systems. They’re orchestrating collaboration between humans and AI. The tools we use now think with us, and that changes everything about the craft.

The Changing Landscape

Ten years ago, a lot of architecture work was about managing complexity by building abstractions; frameworks, templates, deployment patterns. We spent weeks designing scaffolding so teams could build faster and more consistently. But that kind of work is getting automated. AI systems can generate scaffolding, boilerplate, and full service templates in minutes. The problem we used to solve, how to get started, isn’t really a problem anymore.

When the thing you used to design can now be generated instantly, the focus has to move. Architects aren’t defining structure anymore; they’re defining intelligence. You’re not asking “What should this system look like?” You’re asking “How do we make sure the AI knows why it should look that way?” It’s about shaping the inputs, context, and data so that what’s generated aligns with intent. The job shifts from building code foundations to engineering the thinking process that builds them.

The Rise of the AI-Literate Architect

We’re watching a new type of architect emerge: the AI-literate one. They still understand the core principles: separation of concerns, scalability, event-driven architecture, fault tolerance, all of it. But they also understand how generative systems influence those principles in real time.

An AI-literate architect knows how to embed architectural context into the ecosystem. They define how teams use AI-assisted tools safely and effectively. They make sure the AI understands the system’s constraints and style. They design for AI participation, not just human interaction.

This requires a mental shift. You stop thinking in one-way delivery lines and start thinking in loops. Human input feeds AI generation. AI output is validated and tuned by humans. That feedback updates the system and documentation automatically. It’s no longer a linear workflow, it’s a learning cycle. If you’re the architect, you design that cycle.

From Gatekeeper to Curator

In the old model, architects were the gatekeepers; reviewing, approving, enforcing standards through checklists and review boards. The problem is, AI doesn’t wait for review meetings. It just keeps generating.

That means the architect’s role evolves from enforcing architecture to curating it. You’re building adaptive systems that can evolve in real time while staying within safe boundaries. Architecture becomes an active process, not a static deliverable.

You start embedding your architectural intelligence directly into the system. Instead of writing 40 pages of guidelines that nobody reads, you teach the AI to enforce them. You build architectural context into templates, pipelines, and code generation rules. You define the patterns and anti-patterns that the tooling itself understands. It’s a shift from “humans enforcing rules” to “systems embedding rules.”

Practice Architects, in particular, are going to be the ones who make this leap first. They’ll design how AI participates in engineering: defining prompt libraries, training models on architectural standards, and creating governance systems that are continuous instead of ceremonial. Architecture review won’t be a meeting anymore; it’ll be a real-time feedback process that’s baked into every commit.

Bridging in the Age of Intelligent Systems

Architects and principal engineers have always been bridges between silos. That part doesn’t change it just gets more complex. Now you’re bridging not just Product, Engineering, and Operations, but also human and machine reasoning.

You’re designing how information moves between people, systems, and AI tools. You have to ensure that AI-driven design decisions stay aligned with business intent, because left unchecked, they’ll drift fast. AI is great at generating something that looks right but is completely wrong. So now part of the architect’s responsibility is to make sure the system learns properly and doesn’t hallucinate structure or logic that doesn’t exist.

This is where the role scales. Architects are no longer managing systems, they’re managing sociotechnical ecosystems. People, tools, code, feedback loops, all moving parts in one continuous adaptive system. That’s the new frontier.

The New Shape of Senior Engineering

Principal Engineers, Staff Engineers, and Architects are converging. The boundaries are blurring because the core responsibility is shifting toward orchestration of people, systems, and now AI agents. The job isn’t just designing systems; it’s designing the processes that generate and sustain them.

The most effective senior engineers in this new landscape are systems thinkers. They understand feedback loops, both technical and human. They can teach teams how to reason with AI and validate its output. They can embed governance and ethics into automation. And most importantly, they know how to keep humans in the loop without killing speed or creativity.

The point isn’t to resist AI. It’s to shape it, to make sure it becomes an extension of good engineering judgment, not a replacement for it.

The Path Forward

Software architecture isn’t fading away it’s evolving into something more dynamic. The diagrams and frameworks are still there, but now they’re part of a system that can reason about itself. The architect’s job is to make sure that system is learning the right lessons.

The best architects in this new era will stop treating architecture as documentation and start treating it as infrastructure. They’ll design processes that teach systems how to design themselves safely and intelligently. They’ll move from drawing boxes to defining the feedback loops between them. They’ll stop defending their ivory towers and start engineering adaptive ecosystems that learn and evolve continuously.

The next generation of architects won’t just design software. They’ll design intelligence into how software gets made. And that’s a far more interesting challenge.

What is the SITREP on Apache Kafka & Flink?

I’ve worked with (** references at end of article) a number of Apache projects over the years, often pretty closely; Apache Cassandra, Apache Flink, Apache Kafka, Apache Zookeeper and numerous others. But the last few years I’ve not been immediately hands on with the technology. A few questions popped up recently, that fortunately I was able to answer based on existing knowledge, but it made me real curious about what the SITREP (Situational Report) is for the Apache Kafka and Flink Projects for TODAY, i.e. rolling into 2025! The following is a quick dive into the history and then the latest details (and drama?) with Apache Kafka, Flink, and tangentially some other projects (Zookeeper?).

Apache Projects – Context & Quick Details

If you’re unfamiliar with the Apache Projects in a general sense, I highly suggest going and checking out the Apache Project Directory and Apache Projects List. There you will find all sorts of fascinating information about the organization itself, how the projects are organized, and the trend of committees and related details. For example, I always love checking out the initial charts on retired and active that show on the directory page, as I’ve snapshotted below.

Continue reading “What is the SITREP on Apache Kafka & Flink?”

Keeping it Lean: Building the Bare Essentials for Project Management

When you’re running a project that needs to stay lean — and I mean lean like taking a cargo bike to grab groceries instead of a 2+ ton automobile that’s slower, more cumbersome, and way overkill for the job — the tools you choose and the processes you define matter as much as the work itself. It’s easy to go overboard, drowning in Gantt charts, sprint boards, and daily standups that spiral into mini-retrospectives. But what if the goal is simplicity, agility, and clarity?

Let’s break it down.

1. Define Your Central Hub

The first thing you need is a single source of truth. This doesn’t mean a bloated Jira instance with workflows for every imaginable scenario. For a lean project, a simple Kanban-style board can do wonders. Tools like Trello or GitHub Projects (especially if you’re already using GitHub for version control) offer clean, intuitive interfaces that keep everything in one place.

Continue reading “Keeping it Lean: Building the Bare Essentials for Project Management”