The Cost of Feedback

Introduction

The tools that now write code have changed how much it costs a software engineer to find out whether an idea works. The history of the field says that when that cost changes, the job changes with it. The careful programmer of 1960 and the engineer who runs a service today were both good at their jobs; the jobs were different because finding out cost something different. So the question asked here, what will be expected of us next, has a historical answer, and this essay works toward it.

A constant: how design happens

Start with what does not change. Most of our day goes to deciding what the code should be: how a function should behave, where a boundary belongs, what a change will do to the rest of the system. That deciding is design, and it has been watched happening. Glass reported on researchers who recorded software designers thinking aloud, and what they saw is how it goes.¹ The engineer imagines a candidate solution, runs it in the head against the problem, notices where it falls short, changes it, and runs it again. Every pass ends in a model that has just failed, and the failure is the point: it is how we learn what the next candidate has to handle. Glass’s phrase was that design happens “inside the head, inside the brain”; methods only organize it and notations only record it. The loop runs at the speed of thought, mostly before anything is written down, which is why typing was never the slow part of programming.

A variable: the cost of feedback

Trying a model in the head is nearly free, but it can only catch what we already know to look for. Everything else has to come back from reality as feedback, and feedback has a cost with three parts: how long it takes, how much of the idea it covers, and who it comes from, which is to say whose judgment the model is tested against: the machine’s, a colleague’s, or a user’s. That cost decides how the work gets done. When feedback is expensive, we reasonably stay in the loop longer and want more certainty before paying for it. When it is cheap, we ask early and often. Jack Reeves put it plainly in 1992, by which time building had become cheap enough that it was easier to build a design and test it than to prove it correct.²

How the cost has fallen

Batch computing

1950s–1960s

In the batch era that cost was at its highest on every count. Feedback came back overnight, covered one run, and came from the machine, which could say whether the program ran and nothing else. Brooks describes what that did to thinking: programmers lost the details, and sometimes the direction, of what they had been doing before the results arrived.³ A single typo could cost a day. The sensible response was to run the loop in the head as far as it would go and check the code by hand before submitting it. The job was to be careful enough not to need the machine.

Interactive computing

1960s–1980s

Sackman’s experiments, reported in 1968, caught the first fall as it happened.⁴ At a terminal feedback came back in seconds, and debugging took fewer programmer hours. The trade was machine time for human time, and it became worth making once machines were cheap enough to spend on finding mistakes. Time-sharing, then personal computers and integrated environments, made feedback at small scale nearly free. Finding mistakes moved from the desk to the machine, and a good programmer stopped being defined by carefulness alone.

Upfront design

1970s–1990s

But only the time had fallen. Feedback about a whole system still took months and reached users last, and Boehm’s finding that late mistakes cost most made it rational to get feedback on paper, before anything expensive existed: requirements, specifications, design reviews.⁵ This was not a failure of imagination. It was an adaptation to what feedback cost, and iteration was known and practiced wherever it could be afforded. But the way Royce’s 1970 paper was read, and then the standards written from that reading, held the sequential model in place for twenty years,⁶ long after the cost had begun to change. The job became building to a specification.

Iterative delivery

1990s–2010s

In 1999 Kent Beck saw that upfront design had been a response to the cost of change, not to feedback itself.⁷ If tests, refactoring, and small releases made change cheap, late feedback became affordable: what users made of the work, which had always come last, could be found out early, and the wait fell from months to days. The web made it available: software delivered through a browser could be changed for every user at once, so a release stopped being an event, and rented infrastructure made an architecture cheap to try. That is why the customer in the room, the frequent release, and the team that runs its own service followed. This era differed from the terminal era in one respect worth noticing. There, the cost fell and practice adapted. Here, practitioners lowered part of the cost themselves, by investing in tests and tooling before anyone had proved it would pay. The job grew to owning the path from idea to user.

AI assistance

2020s

The current tools lower the cost again, mainly on the second part, how much of the idea one round of feedback covers. A whole approach can be written, tried against alternatives, and discarded in days.⁸ The trade is Sackman’s again, machine time for human time, at a scale he could not have imagined. And as in the terminal era the cost fell on its own; the tools arrived, and practice is still catching up. That brings the work up to the line Brooks drew when he examined AI in 1986 and set it aside: the hard part of software is deciding what to say, not saying it.³ Saying it is what has become cheap. That was never a matter of typing. It is the distance from an idea to something that can be tried, and that distance is what the tools have closed.

What stayed expensive

Every earlier fall in the cost of feedback moved the job toward whatever had stayed expensive, and this one leaves two things expensive. Deciding what is worth building, because the tools will build whatever they are asked to. And putting the result in front of the people who can say it was not worth building, because where the feedback comes from is the one part of the cost that has not moved this time: the tools answer faster and about more of the idea at once, but they are still the machine, and the machine can say whether a thing works, not whether it was worth building. No tool will bring users any closer. Hearing from them gets cheaper only if we arrange it sooner.

That has always been the hard part of the job. Brooks named it in 1986, and each time feedback got cheaper the job moved further toward it. Building is now cheap. Deciding what to build, and finding out whether it was worth building, is not. If the pattern holds, that will show up on teams within a few years, in ways that can be checked.

Predictions

Design documents will come with working code or not at all. The proposal will be the prototype, and the writing will say what was tried and what happened.

Teams that keep the same feedback loop will ship more and change nothing. Most of the productivity gain will be wasted that way, and the organizations measuring output will be the ones let down.

What makes an engineer senior will change. It will stop being how well they write code, which is now the cheap part, and become how well they decide what to build and how soon they find out whether it was worth it.

Hiring will test for the wrong thing for years. Interviews will keep measuring the part that got cheap, because institutions change last, and the teams that notice first will have their pick of the people everyone else is filtering out.

The line between engineering and product management will narrow, from the engineering side. We will be expected to hear from users ourselves, because the part of the job that stayed expensive is the part that line was drawn around.