The Canvas Cares Not
Computers don't care how programs are structured. Businesses don't care how systems are architected. The canvas does not care how paint is applied. Software engineering is an art.
Software engineering is also a culture of artists; despite that, we recoil at being called "artists." Art has no right answers: a work of art means something different to everyone who receives it. Software engineering denies that it is an art form, which leads to a deep-seated intellectual insecurity. It fetishizes science.
Until we come to terms with the reality of the profession, we delude ourselves. Our output, our programs, our systems suffer for our deliberate misunderstanding. What's more, in our pursuit of the trappings of science, we inflict suffering on one another needlessly.
A Front-Row Seat at the "Best Art" Awards
First, art. Why do I say software engineering is art? Art is communication: between people, between the artist's past self and present self, between the author and the audience. Music is art, prose is art, cooking is art, architecture is art.
Art is forever imperfect. It leans on shared context. Art is always "in reaction to" itself: an infinite regress.
Software engineering culturally focuses on the utilitarian byproducts of art: the food the cooking produced, the building the people live in, the album the musician pressed; we are obsessed with the utility produced while we ignore the aesthetic involved.
Aesthetics are, broadly, the values we bring to art: they're how we decide what is "good art" and what is "bad art." The concepts of "bad" and "good" are personal, not universal: it is entirely reasonable for two people to disgree about whether a book is bad, or whether an API is overengineered.
The art of software engineering is the voice of the engineers behind the code. The communication of the history of the system, the decisions made and options weighed. Every single engineer has different tastes, different aesthetic values, a different voice.
There is no such thing as a universal "best tool for the job" because our job is to communicate. There exists no meaningful award for "best art" because, as a form of communication, art is intrinsically subjective. So it is with programming.
A Zero-Sum Game
Software engineering is, as a culture, intellectually insecure.
We hold technologies in our periphery with some imagined faint praise or damnation.
We hold technologies in our direct line of sight with some self-assessed — but again, imagined — level of expertise, while we gloss over their essential complexity.
We imagine that our grasp of complex systems and topics grants us universal insight across all complex systems. We generalize prematurely. We pattern-match like a cheese grater, dragging complex ideas across the plane of our understanding until they come through the other side misshapen.
To some degree we do all of these things because we are human. But we make the mistake of thinking that we are special humans. That we are on one end of the "bell curve of intelligence", as if such an abstract concept could ever honestly be made concrete (don't @ me about IQ tests, they're bullshit.)
We're told we are the smartest and the best and that we should seek to work only with the smartest and the best, but:
We don't know everything and we're constantly reminded of just how average we are; just how small our understanding is. And if our understanding is small, then maybe we aren't the smartest and best. And if we're not the smartest and best, then no one wants to work with us. If no one wants to work with us, well, then, shelter and food are going to be a problem.
It is far easier to react defensively to that insecurity than to embrace it.
What makes us so rigid? Who told us to be this way?
Software engineering is art, yes, but it's also a job. The utility of our work attracts capital, and capitalists have have found it useful to tell us how great we are. Probably because it makes us turn a blind eye to the externalities of the systems we produce.
Because the utility of the systems we produce is the sole focus of capital — our sole value to business — we are systemically encouraged to avoid thinking of ourselves as artists, to recoil from the thought of having aesthetics.
Developers internalize this conflict: we have druthers, preferences, etc., but we're told that we shouldn't. Because we shouldn't have anything as silly as "a preference about semicolons", we try to rationalize it by picking some concrete truth and lashing our foibles to it. We're sandbagging our opinions both from each other and ourselves.
And so, as a rule, we're insecure.
As a result, we're constantly fighting both for and against our aesthetics and for and against the aesthetics of others. And our desire to continue drawing a salary prevents us from embracing the simple fact that we even have aesthetics in the first place. We seek to quantify our aesthetics, to ground them in universal truth: "For my expression to be be valid, all alternatives must be invalid." A zero-sum game.
We get in our own way. We get in each other's way.
A Successful Hire
It's no wonder that the state of technical interviews is so abysmal.
Technical interviews are a coarse values compatibility test between the interviewee and representatives from the hiring team. Ideally, this catches things like "does your preferred communication style match that of the team?", "how best do you learn and is the team suited to support that mode?", among similar questions.
Because of the deep-seated insecurity of engineering at large, interviews are often a form of earning cachet with the existing team. In the worst case, this looks like answering "gotcha questions", whiteboarding, and regurgitating trivia. Hazing, in other words.
Why do I say "worst case" when it's so widespread? Consider: it is common practice for most engineering managers to have some amount of programming background; thus those most frequently responsible for screening candidates ahead of technical interviews can already tell if the candidate knows how to program or not. Any discussion of "can the candidate program up to the quality level of the organization" can be linked directly or thematically back to "'A' players only want to work with 'A' players", which is generally misunderstood to mean "this candidate shares my aesthetics." A gentle thump on the offending interviewer's head may be dispensed by the hiring manager.
New hires bring in their own opinions, values, & aesthetics.
A successful hire always changes the values of the team that they join; and in turn the team always changes the values of the person that they hire.
The weight of the change — whether it tilts in favor of the team or the hire — depends on the experience of the hire and the role they are being hired into, but it NEVER goes to zero on either side. (If either side remains unchanged, it wasn't a successful hire: they were most likely shuffled onto another team or fired.)
Now, keep in mind: capital wants fungible engineers. They're focused solely on the utility we produce. From that frame, the conflict & compromise of a team's aesthetic values is a bug, not a feature. But they couldn't be more wrong: the primary value in hiring a new engineer is not simply "increasing parallelism of work", but "expanding the artistic scope of the team."
Invention happens because of unexpected connections; art is improved by resolving constructive conflict and integrating feedback. But if you leave that conflict unresolved — if our field is so stunted as to believe the conflict shouldn't exist — you merely end up with a team in discord.
Concede the Point
When aesthetic conflict can only be resolved by quantifiable arguments grounded in utility or universal fact, the result is tenuous rationalization of aesthetics. In other words, a team in this state lacks the means to resolve the conflict constructively.
This gives rise to bad behaviors:
- Argument to exhaustion: whoever argues the longest for their point wins.
- Oil and water: different parts of the system are built based on increasingly different aesthetic principles. The engineers involved "agree to disagree", so the conflict gets pushed into the interface between their two systems.
- Disingenuous arguments (or sandbagging): coming up with a solution and constructing arguments to support it instead of investigating a problem space, eroding trust.
- Appeal to the wisdom of the crowd: "everyone is using
$FRAMEWORK_OR_PROCESS, so it must be the right answer for us as well." This ignores the context of the team, who may not be in agreement with the practice.
Teams in discord are not stable. Unresolved conflict will show up in the system they produce. Ignoring the aesthetic of the team threatens the utility they produce.
The core of the problem is that, absent a means to talk about their personal desires & values for the solution, engineers "argue to win". The starting point of the argument is disingenuous, & they do not feel they can admit to their aesthetics because it would concede the point.
By way of comparison, a healthy team respects the tastes of its members, which lets them talk about them openly when faced with a problem. Conflicts can be resolved constructively by surfacing the values behind them.
Borrow These Values
Art is, as previously established, always in reaction to itself. It is subject to fads: a technique, thought process, or organizational principle sparks the imagination of a large number of artists at once. The fad might propose a novel solution to long-standing problems, reframe the art in a fresh way, or simply resurface an old technique for a new generation. Fads create a vacuum, a new space to explore in the art.
Invariably fads die out, either being subsumed into the practice completely or being replaced by new trends. They become contextualized, contained, understood. Penning them in extinguishes the initial enthusiasm that propelled them to popularity. You don't see a lot of art that centers pointillism these days. The fad has been explored and it no longer captures the imagination. There's just no point in focusing on it anymore.
Fads are a good thing in moderation. They expand the borders of the art. Long after their demise, the lessons learned still resonate and inform future artists. Fads, however, can produce zealotry: it's tempting to believe that "we've finally figured out the art! It is the end of history!" The fad can live forever, becoming synonymous with the art itself.
Software engineering is fertile ground for such zeal. Fads in this field live a charmed life: we call them "best practices." They're dumped on teams, with the expectation that the team's output will improve. Agile, scrum, devops, SRE, React, Node, matrix orgs, the Spotify model, Rails, Web 2.0! All fads.
When the team's productivity is down and your back is to the wall, there's a pressure to find a magic bullet. A process or tool that worked for some other team. Not your team -- a famous team, a successful team. After all, engineers ought to be fungible, right?
But all of these tools and processes are shaped by the aesthetic values of the teams that originated them. You can borrow these values, but they'll always fit a little like hand-me-downs: a little uncomfortably. Your team is not Google. You may not be able to afford Google's values: Google has a money printer, after all.
Your team needs to find its own voice. "Best practices" are the start of that work, not the end.
Finding One's Voice
First, release the idea that it is possible to be right about software engineering.
Second, I am going to quote John Gardner's "The Art of Fiction", which is as applicable to fabulists as it is to software engineers:
[...] in order to achieve mastery [they] must read widely and deeply and must write not just carefully but continually, thoughtfully assessing and reassessing what [they] write, because practice, for the writer as for the concert pianist, is the heart of the matter.
Practice, practice, practice. There are no silver bullets. Read lots of code, write lots of code, try out different approaches, see what speaks to you. Discard the rest.
Finally: I am going to give you the scariest bit of advice. In order to work well with others, you must be vulnerable. Admit that your tastes are your tastes. Concede the ground of universal truth; it was never yours to begin with.
Read old code together; react honestly when you find code that violates your values, but know that what you read isn't wrong, it merely offends you. There may be entirely valid reasons for it to be written the way it is! Build trust with your colleagues by demonstrating that you are willing to give authors the benefit of the doubt. Your colleagues can connect the dots: how you talk about code when the author's not around extends to how you talk about their code when they're not around.
Let your teammates see that you value their aesthetics. Invite them to inspect and adjust your values when you start a project. If you accommodate them, they are more likely to accommodate you. Expose your values and thinking whenever you go to solve a problem. Accept feedback graciously, and make sure you give credit when acting on it.
This profession doesn't need to be a zero-sum game. Treat it like an art, respect your colleagues as fellow artists, and practice it together.
Thanks @ceejbot and @jefflembeck for reviewing drafts of this post.