There is a fundamental discord between software creators and the businesses that want them to create software on their behalf that really just boils down to something like the following:
Programmers at all levels of experience want to believe they are artists, while businesses of all sizes want to believe that software can be manufactured.
There is an argument, often put forth by the business, that software is both art and manufacturing. That it is art in the sense that it is a creative process that requires a great deal of skill and experience to do well and that it is manufacturing in the sense that it is a process that can be broken down into discrete steps that can be repeated and improved upon. This point is not without merit, however it misses the mark when trying to make a concession to the artist because it fails to grant agency and autonomy to the artist. It is a bit like saying that a painter is an artist, but that the canvas, paint, and brushes are the manufacturing process. The painter is just a cog in the machine. This is not a good way to motivate a painter to do their best work.
A decade ago, this was an acceptable trade off for the business because even programmers who weren’t doing their best work created software that was good enough, and if it failed there were few consequences, both in damage to humans and liability to the business. However, this is not a recipe for software that we should trust with our lives. Autonomous vehicles, medical devices, aerospace systems, and other software that we rely on to keep us safe and healthy require what might be termed “NASA quality” software. That is, software that is as close to perfect as we can make it. Likewise, as software becomes integral to all parts of society, the era of businesses bearing no responsibility for what happens when their software is defective or used for harm is coming to an end. This too implies software that is as close to perfect as we can make it.
One does not create software that is nearly perfect using a manufacturing process. Programmers aren’t fungible, their creations are not interchangeable, and the process of iterating towards perfect is not linear. It is, as a process, less like what happens on the factory floor and more like what happens in the laboratory; a process of discovery, of experimentation, of failure, and of learning, and that takes unpredictable amounts of time.
For a business that needs to ship product, this is anathema. They don’t care about what happens in the laboratory, they care about how long it takes, and much it costs.
Instead of the manufacturing paradigm — which is rigid and fragile in even it’s most flexible embodiment — that has dominated the business mindset for decades, a new approach that is more fluid and adaptive is needed.
So too, do programmers need something better than artisanal code as an aspiration. Craftsmanship in creation is less like painting and more like cabinetmaking or blacksmithing. It is a process of learning and refinement that takes years to master but one that produces value with even basic skills. This is a problem if programmers like the status quo arrangement where their labor is highly sought and highly compensated. And it is a social problem. Learning to write code is a “now and continuing on forever” kind of skill; there is no finish line. Having form and structure to how you practice your craft is essential to making progress, but most jobs aren’t going to give you that structure. You have to create it for yourself, which you can’t do if you are a cog in the machine on the factory floor.
Both sides here are in trouble – business can’t keep shipping unsafe code without liability, and programmers can’t keep wasting their time in jobs that don’t support their artistic growth, no matter how much they pay.
My solution for programmers is to create a new kind of organization that is built around the idea that software is a creative process that requires a great deal of skill and experience to do well, and that the process of creating software is a process of discovery, of experimentation, of failure, and of learning, and that takes unpredictable amounts of time. And that becomes a place where programmers can learn and grow and create software that is as close to perfect as we can make it, like an atelier or a laboratory or a studio; the best analogy might be a dojo. There are members at all levels, and the goal is to help each other grow and learn and create better software. Software as a Practice. And these practices should be small and focused, fewer than a couple hundred people at the large end, but optimally around 25 – 50 people.
For business, the solution is to take a page from the elite military units and commit to mission-type orders that don’t dictate process and just ask for outcomes and a culture built around agency, empowerment, trust, and accountability. Then to see software like a weapons system — it amplifies the power of the people who use it, and it is a force multiplier for the business itself, not the business; it is not a product, it is a tool. Software as a Tool.
If you have an idea on how to make a tool better, you go to the tool makers and suggest (not command) an improvement. If you need a tool that does a particular thing, you do the same thing. The craftsmen are the experts, and they are the ones who know how to make the tools that make you more happy. You don’t tell them how to do their job, you tell them what you need and they figure out how to do it, then tell you what it costs to get it. Businesses, on the other hand, are the ones who need to use to the tools to make their business succeed.
This is a fundamental shift in how businesses and programmers interact, and, in my opinion, it is a shift that is long overdue. Software manufacturing isn’t going away, but it is irresponsible to use it as the methodology to produce software that we trust with our lives or allow to make life-altering decisions about people. Keep using it for grinding out enterprise CRM tools and the like if you want, but for the love of all that is holy, stop using it for medical devices, autonomous vehicles, and decision support systems embedded in the fabric of society like loan applications, risk assessments, and criminal sentencing. As these systems proliferate, they become a kind of social infrastructure accumulating so much impact that they have to aim for perfection, not profit.