A long chill has taken hold of most of the US, and with it, you may find yourself laid up indoors, lollygagging on the couch. Here are a few reading recommendations to keep you off the doom scroll and focused on one of my favorite topics to ruminate on — keeping it simple (whatever it may be).
There are two methods in software design. One is to make the program so simple, there are obviously no errors. The other is to make it so complicated, there are no obvious errors.
– Tony Hoare
Preamble
When you start at Justworks, on your first day, you learn about COGIS. It’s our set of values:
Camaraderie — Going forward as a team within our own company and with the companies we support.
Openness — Transparency with our customers and ourselves until we have nothing to hide.
Grit — Toughing it out when things aren’t so easy, and staying on the paths that we know to be worth it.
Integrity — Doing the right thing to build trust and never lose it.
Simplicity — Ahhh simplicity…
At a company level, simplicity is about making hard problems for our customers easier, more streamlined, and entirely less daunting. But in engineering, it’s one of my favorite subjects. It’s the value that keeps me at Justworks, keeps me happy, and ultimately keeps my work connected to the wonderful folks we get to have as customers every day.
And so I’ve put together this post: it’s my recommended reading list for getting to know about this captivating subject, as well as a sort of New Year’s resolution to keeping my work and my team’s work in line with the messages within.
Two Kinds of Complexity
Our engineers work in a number of categories — Payroll, Benefits, Underwriting, etc — and these are hard, complex problems: maintaining tax remit schedules, dealing with school district taxes in Ohio and Pennsylvania, and entangled onboarding flows. “Hard” means the complexity is inherent to the problem’s domain, not its implementation. For instance, the difficulty in calculating paychecks correctly isn’t the math involved — it’s in understanding the regulations that drive the calculations in a nuanced and contextual way. The solution we put on top can either try to be minimal in complexity or incidentally add to the burden.
No Silver Bullet
For the first reading, you’ll guess that I’m hinting at No Silver Bullet(opens in a new tab) by Frederick P. Brooks, Jr., written in 1986. That’s a long time ago. But this is the paper that brings up this incredible concept of Accidental/Incidental and Essential Complexity. The idea is simple: the problem you’re solving can likely be solved with a spreadsheet or some analog equivalent. But that’s too slow, so we’re going to go digital and stick it in our magic thinking-rocks’ memories so it gets us to the moon. Once our solution makes it into a computer, every solution has a cost, and we have a duty to minimize that cost, both to ourselves and to our future teammates.
For example, once our app is on a server, there’s now the internet and more networks between the customer and us, providing ample opportunities for something to go wrong. Not to mention the programming language, framework, and opinions of the engineers layered on top that could be opportunities that steer us further from the essential priorities.
Separating Complexity and Effort
Rich Hickey adds to the subject of complexity in a more recent development, Simple Made Easy(opens in a new tab). He disambiguates the structure of the problem from the ability of the person designing or performing the solution. There’s also a more management-relevant book, Wiring the Winning Organization(opens in a new tab) by Gene Kim and Steven J. Spear, that also touches on the subject, but summarizes that managers need to align the problem’s required level of effort to solve with the ability of the team to solve it when those things overlap, great things Just Happen™.
Takeaways
Applying these learnings at Justworks means looking at a difficult problem and not solving every last bit of it in the first pass. We don’t need to design every piece of a new system from the start; rather, let’s design the simplest version that can grow into something more involved as we connect with customers who need it.
For you, dear reader, applying these materials to your life might involve a journey of self-discovery: what is actually important in this work? What can I do without and still answer the needs of the problem? More pointedly: am I designing this solution to answer the complexity or am I including my own interests in this effort?
Understanding Simple Design
There’s a classic, ancient tome called The Art of Unix Programming(opens in a new tab). Described within are themes of simple schemes for inter-process communication (plain text and a few delimiters) as well as a theme that you want “sharp tools” to compose together rather than programs that try to do everything. I’ve permalinked to the page relevant to this post.
Building up from Simplicity
As egotistical as it might sound, I have a gripe with this foundational text, Rule 17: “Design for the future, because it will be here sooner than you think.” My anecdotal experience with this is that there are engineers who take this to heart and begin to worry about every single possibility and even try to code defensively not just against inputs, but to their team changing the code later. It manifests as a bunch of interfaces or an abundance of tests trying to cement behavior rather than make it safer to change (you can go too far!).
On the other end of this is the approach described in Write Code that is Easy to Delete, Not to Extend(opens in a new tab), by tef. To summarize it: when writing code, start with a solution so simple that, when it needs to change, it can be rewritten in its entirety. It is, of course, aspirational, but it embraces the fallibility we all have: we don’t know what the right design is, so we must optimize to be iterative owners of the design.
There’s a spot in this post called the “corollary to the single responsibility principle.” A lot of folks read SRP(opens in a new tab) and then break their functions out into 10x the amount of code so that each function does one thing, but useless without context. It makes it harder to read, and your brain has to jump back and forth. It doesn’t seem like it really helps the developer iterate on it, which doesn’t help the product, organization, or team. The corollary reads: “every hard problem is solved in one place.” Now that is something useful to the developer. If all my payroll validation is in one spot, then when it breaks, I know where to go to fix it. I want to hold exactly the piece of software in my head to understand the hard, essentially complex problem to diagnose and treat.
Takeaways
The main lesson I want to drill home is that many times splitting out problems with abstraction and layers makes the mistake of making the whole of the solution harder to understand. It’s a symptom of trying to predict where changes will come from in the future and how they’ll affect our code, creating interfaces and tests that make us feel bullet-proof. Typically, this manifests in low locality, where the editor has many files open, or the same one open in different places because they have to jump around the code. Don’t fear the tall functions when it’s all relevant to telling the story of the problem it’s solving. You can always make a system more complex, but going in the opposite direction is exponentially more difficult.
Simplicity in Production
There’s a bunch of little talks and posts out in the real world from not-so-large companies that might apply when we talk about simplicity without totally falling victim to cargo-culting(opens in a new tab).
Simplicity in Go
At Justworks, we chose to adopt Go(opens in a new tab) a few years back. It’s a language that has historically been made fun of for missing features or shirking modern ones. Rob Pike discusses those decisions to keep the language guarded are in a wonderful talk, Simplicity Is Complicated(opens in a new tab). That video is a great intro to the idea that Go was defending against kitchen-sink features intentionally, and that if you appreciate that intentionality in your work, then it’s the tool for you. And as it turned out, was also the tool for us as we built our new platforms.
As a bonus, Rob Pike has his own blog with a very relevantly titled post, Simplicity(opens in a new tab). My favorite line there is:
[…] every time you add something, you add complexity. Add a new library, you add complexity. Add a new storage wrapper, you add complexity. Add an option to a subsystem, you complicate the configuration. And when you complicate something central, such as a networking library, you complicate everything.
Classifying Complex Systems
Bryan Cantrill recently gave this one: The Complexity of Simplicity(opens in a new tab). If you don’t know, Bryan co-founded this company that’s building server racks from scratch in this day and age, called Oxide Computing. They have a number of RFD’s(opens in a new tab) available online and serve as inspiration to me when I’m struggling to get my thinking together in a tech spec. His talk hits on the classification of types of systems. He argues some important systems are super-engineered — baroque in their complexity and design. While these systems may anchor critical functionality, they’re seldom pointed to as exemplars of how to actually solve hard problems. Then, there are the revolutionary systems. These learn from accreted systems, where the interfaces and abstractions became the thing you have to program around as essential pieces of the system’s inherent complexity, precisely because this functionality cannot be broken.
There’s about 5 mins in that talk where he says, “it’s all made up,” referring to abstraction/layers. And it’s true: the machine doesn’t see interfaces, it sees GOTO statements, and even those are eventually compiled to binary (okay, fine, the machine does see interfaces, kinda. At a technical level, they see v-tables and dynamic dispatch - but you get the point). The abstractions and layers and lines we draw are for us, for describing our problems to each other, and hopefully describing the way we’ve simplified them for the next person to understand and steward. As always, these additions can also be ways that we confuse the next steward.
Your tools will go further than you think
“But we have to engineer a little, right?” sayeth you, the reader. Well, sure, but in doses and where it actually matters. So now we’re gonna talk about innovation tokens. I think the phrase came from Dan McKinley, in his essay Choose Boring Technology(opens in a new tab). It’s a good read and caps everything you’ve read thus far: complexity and shiny new things cost something, and we need to control that cost. In the talk, he praises a super-simple implementation of Etsy’s activity feeds using cron jobs, surviving a 20x usage increase. The lesson here is that our DB’s and existing tooling will get us pretty dang far. ASO Payroll still doesn’t cache your YTD totals when viewing paychecks, we do that stuff on the fly, and it’s totally fine. No cache to bust, no recalculation to do. Sometimes we have the ability to burn a little CPU to simplify our lives and keep the architecture small.
But this lesson of doing more with less continues! Tailscale(opens in a new tab), up to a certain point, had their database as a flat JSON file. They eventually announced they were moving away from it in this blog post: An unlikely database migration(opens in a new tab), by Brad Fitzpatrick and David Crawshaw, the latter of whom is a master of using SQLite in production. Now, don’t get me wrong, the story ends with them moving to etcd and, eventually, hating that, moved to SQLite(opens in a new tab) a year later. The moral of the story is: you likely don’t need the scale you think you do.
Takeaways
The biggest takeaway is a broad, honest conversation about YAGNI(opens in a new tab). Turns out your database can handle billions of rows pretty well, so maybe designing a replicated system with shards is a long way off. You don’t need the shiniest thing because the boring, trusted solution has more than enough value for its cost. But to reiterate: you cannot avoid the cost in the decisions of your design, it will be ever-present. And that means focusing on what really matters to your business.
Wrapping Up
To take us home, there’s Dave Cheney’s Practical Go: Real world advice for writing maintainable Go programs(opens in a new tab). Dave famously helped in designing the great errors package of Go 1.13, alongside many contributions to evangelizing the language. I find myself coming back to this post often because it really hammers home that readability and understandability are the key components to success. Near the beginning, Dave makes this declaration:
Whatever programs we write, we should be able to agree that they are simple.
Wait. Who’s this we? Why, it’s all of us, of course. Which brings me to my last point: engineering is a team sport. We have to bring the team along wherever we go, either into the fire or across the ocean. Simplicityis the way that we enable things likeCamaraderie. And with our teams, we can embody the Openness and the Grit and the Integrity that our customers need.
Our customers won’t see the diagrams and the queues and the (*gesturing wildly*)architecture. Those are the things that we use to communicate ideas to each other, the things that grow our mutual understanding of the real issue rather than disguise it in complexity.
If you’re an engineer, the eureka moment is about learning to love solving the problem, not just programming around it.
Do you want to build products that help entrepreneurs and small businesses grow with confidence? We’re hiring across our Technology teams. Come build with us! Check out our Careers page.
