Understanding is the new bottleneck

I think it funny how much average engineers are beginning to discover the challenges of engineering leadership and program management. This has always been the bottleneck. It's why managers and PMs want to be in standup. It's why slack exists and engineers are constantly being poked on it. It's why execs always talk about not getting too far away from the work. It's how seagull management happens. It's why program management is a job. All those behaviors engineers hated about their bosses that kept them away from being focused on the code...they're starting to feel what it's like on the other side and reinventing the solutions instead of just reading a book about engineering management. Maybe we'll rebrand program management to "understanding ops" or something. I wonder what AI would say about us if given the tokens to complain.

We have LLMs try to generate descriptions of PRs for us and they're pretty universally disliked. They're always overly-complex descriptions of the mechanical changes and have no sense of motivation. Also, a huge reason to understand the code yourself is to make sure the LLM isn't wrong, but this doesn't work if an LLM is itself generating the understanding.

I agree with the problem but not the solutions. The problem pre-dates LLM's: writing code that "works" but breaks the underlying model. Because it works, it always sounds reasonable and doesn't raise any flags. Only someone - human or LLM - who holds the model as the standard would see that this working solution breaks the model. (In theory, the model is to preserve scaling, flexibility or some other systemic feature not immediately invalidated by this working code, but as always the model itself could be bad.) LLM's are not bad at giving an account of the model; indeed, fighting with the LLM over what the model is can clarify things. But LLM's will happily hold on to a stream of inconsistent statements as their model, so they are not the authority.

"I read the code." -Mitchell Hashimoto Great code needs great understanding and agents need excellent guidance. Even in my current solo-dev work, I can't imagine making a production commit I haven't read until I understand it. I own the consequences of my code; that's a responsibility AI agents can't take.

It's more like this: We've always lacked understanding. However, it didn't feel like a bottleneck; in spite of lacking understanding, we developed huge, complex systems that became hard to maintain and that nobody understood completely. Now we want to scale that orders of magnitude, but when we do that, we feel the pesky lack of understanding. We previously worked around the lack of understanding by making the system gradually incomprehensible in small increments, upon each of which we observed it still working, more or less. If the whole thing materializes in one day, that doesn't work; the approach is gone. You can now bring into being something which statistically resembles the old kind of system that was iteratively evolved. But the thing has no such history. You can't go back to play archaeologist. It looks like something that would have had users, but it never did. It was never in production anywhere. Nobody ever submitted feedback, or a bug report, such that it was fixed or improved. There never existed a simpler version of it that several ex-maintainers understood perfectly; there are no such ex-maintainers and no such understanding. There is no documentation trail, or other historic trail if surrounding activity like discussions and negotiations which led to things being the way they are.