For those who work in product development, it seems we can never move fast enough. There is always another feature to build, problem to solve, or opportunity to exploit. If you feel like you can’t keep up, you’re not alone. By some estimates there is a shortage of over one million software developers in the United States 1. Demand exceeds capacity, and this often leads to tension between technology and other parts of an organization.
Everyone in the organization plays a role in solving the problem. At Realtracs, our Product Managers work hard to manage demand and ensure we work on the most valuable opportunities. That means we have to say “No” to many good ideas. To reciprocate, our engineers work hard to maximize our capacity for pumping out product using good architecture practices.
What makes a successful architecture? Maximum code re-use? Perhaps, but how slow and risky is it to make a change to that abstract class that everyone shares? Minimal code? We’ve seen some pretty asinine one-liners that nobody can understand. Great test coverage? Maybe all of your tests pass, but the code still didn’t do what the customer expected (the code was right, but it wasn’t the right code).
For us, a successful architecture means we have empowered our squads with the tools needed to autonomously produce valuable solutions for our customers quickly while protecting them from making structural mistakes that put the organization at risk as we scale. If your goal is anything else, there is a good chance your architecture is slowing you down.
Below, we provide our five architectural principles. These principles help guide decisions in how we architect our platform and help us navigate tough tradeoffs found in the inevitable gray areas of software design.
A Successful Architecture Empowers the Squads
In several organizations I’ve heard developers complain they have to do something “the architect made me do.” Those words don’t lead me to believe the developers felt empowered as much as enslaved. A good architecture empowers the squads (or teams) by providing tools they love to use because it makes them more successful.
When a squad starts working on a new feature, they shouldn’t have to worry about building logging, authentication, authorization, or batch scheduling infrastructure. At Realtracs, these cross-cutting concerns are handled for the squads. A squad simply pulls in the latest library to interface with the pre-built tools.
A squad also shouldn’t have to worry about long negotiations with an infrastructure team to provision storage, web server capacity, or cloud resources. At Realtracs, a squad simply commits a json based config file to git, which is then used to automatically script buckets, queues, topics, and other AWS resources—all neatly provisioned with an authentication boundary specific to their service.
Are your squads wrestling with devops and trying to figure out how to automate their builds and deploys? Or worse, are they left with no automation at all? At Realtracs, our architecture has solved this problem as well. So long as the squad interfaces with our runtime abstraction libraries, their code will automatically build on each commit and seamlessly deploy on the Realtracs runtime bus, automatically scaling across our multi-cloud footprint.
For developers who wants to create awesome products, it’s a blessing to not have to worry about the concerns above and be empowered to focus on the features.
A Successful Architecture Provides for Autonomy
Autonomy leads to happiness, so we don’t want a squad to be coupled to (or dependent on) other squads to complete any of their work. A squad should be able to design features on their own, build those features without outside help, and deploy their code independently.
At Realtracs, each squad works in their own code base, which is achieved through a micro-service architecture. This means a squad does not share code with other squads. Those difficult domain modeling negotiations don’t leave a squad. Discussions about when it’s safe to deploy don’t leave the squad either. As a by-product, accountability for each squad is high because they have complete control over their own destiny. If you have a monolith architecture, this isn’t possible.
The squads do share some components, but only when the benefit of reuse outweighs the benefits of squad autonomy. When these opportunities do arise, they offer a fun opportunity for cross-squad collaboration.
A Successful Architecture Focuses on Value
In business, it’s really easy to keep score. You either turn a profit or wind up gone. To win the game, we must deliver value. Your customers aren’t buying a cool architecture. They aren’t buying your cool code design. Your customers are buying features that make them efficient and bring them delight. As a result, we want our architecture to allow our squads to maximize time spent on features. Any work that is not directly related to a feature should be directly related to helping other engineers develop features faster.
We are able to maximize the time we spend on valuable features through heavy automation (builds and deploys), a healthy amount of test coverage, and libraries that handle cross-cutting concerns (logging, security, etc) in a standard way. We are often looking for logic we’ve repeated multiple times and abstracting the implementation away so we no longer have to repeat the code (or remember it must be done).
In the software profession, there is a lot of talk about “good architecture.” If you’re not careful, you can begin to think that a good architecture is the end-goal. To the contrary, a “good architecture” is just a means to an end: the ability to deliver more value at a cheaper cost. At Realtacs, our architecture has done a lot of the heavy lifting so that our engineers can spend time with customers, listen to our users and collaborate with stakeholders. These opportunities, in turn, lead to creative and innovation solutions making our product more valuable.
A Successful Architecture Makes Things Quick
We can focus on value, empower our teams, and help them work autonomously, but if they can’t respond to change quickly, we’ll always get out maneuvered by the competition. Therefore, a good architecture should maximize the ability to respond to change with agility.
We measure “quick” by how long it takes for us to get customer feedback. On average, we get feedback from a customer within five days of writing the first line of code for a feature. Much of this is achieved through adherence to lean principles. For example, we break work down into small chunks and use automation heavily (builds and deploys).
Our architecture also plays an important role. It supports an A/B stack (also known as a blue/green stack) so we can deploy multiple times during the day without incurring any downtime. Our architecture also supports “feature flags.” A feature flag allows us to deploy code to production using a toggle to determine if the features is visible (or usable) by a customer. This allows us to verify a production environment configuration early in the development of a feature and it allows us to release features to small groups of customers for feedback without disrupting our entire customer base.
A Successful Architecture Protects from Structural Mistakes
Working on a squad that is empowered to build valuable features is a delight; being able to move quickly is fun; and getting to work with autonomy is a joy. But autonomy doesn’t mean anarchy, so we don’t let teams use different frameworks or programming languages. A squad can’t roll their own logging solution or stand up an independent security framework. A football team can decide how they want to get to the end zone, but they can’t do it by running out-of-bounds. Likewise, there are rules our squads must follow and by following those rules, they stay within a framework that protects them. Our development managers live by one motto: empower and protect.
What kind of protection does our architecture provide? Like many teams, our builds fail when tests fail. Unlike many, our builds also fail when we detect semantic code rule violations. To keep the signal-to-noise ratio high on build messages, we fail builds with even a single warning message.
By using our logging architecture, a squad is automatically alerted (via Slack) when errors from their service exceed a pre-defined threshold within a window of time. By using the security architecture, a squad is protected from accidentally leaving a web service endpoint open to the public without any authentication. By using our micro-service architecture, a squad can release code multiple times per day (whenever they want) without impacting any other squad and can roll that change back quickly without impacting any other squad. By using our Web API framework, a squad is guaranteed that their payloads adhere to JSON standards that will easily play nice with our JavaScript based web apps.
Last, we make everything transparent. If a build fails in a primary branch, then the entire team is notified via Slack. Or if a deployment fails, the entire team is notified. Culturally, the organization swarms to help heal the injury quickly.
A Successful Architecture Scales
A good architecture scales. If your architecture does not scale, it is only good so long as your business doesn’t grow (in which case, your game is over). Most organizations don’t figure out their architecture doesn’t scale until it’s too late. Their customer base is growing fast and that growing customer base is demanding new features, but the organization can’t meet that demand because they are busy fixing their architecture. The time to ensure your architecture can scale is now, before it’s too late.
At Realtracs, our architecture focuses on horizontal scale over vertical scale. In general, it’s cheaper to throw more servers at a growing user base than more powerful servers. We achieve this through replicated, mirrored databases and a web-server farm that auto scales in AWS.
You must also anticipate scaling of your teams. An agile team with twenty members is too big. You can’t have hundreds of cooks in a kitchen, but dividing a team can be complicated if you can’t offer those teams autonomy over independent problem domains. At Realtracs, we are constantly anticipating those challenges, carefully creating micro-services by problem domain areas. As a team grows, we let those teams go through mitosis, each taking a subset of the micro-services the original team nurtured.
Architecting software is fun, but architectural goals are often misunderstood. We have to remember that software development is a team sport and requires hard-working, healthy, interpersonal relationships to be successful. Therefore, a good architecture has to design for people and how they work together. It must account for empowerment, autonomy, speed, feedback, and organizational growth – not just code organization.
If you’re architecture is not honoring the principles laid out above, you may be passively allowing the opposite: enslaved, tightly-coupled squads that build worthless software slowly and dangerously. If you’re in an architecture role, the developers in your organization don’t answer to you. You answer to them. What do they need to be successful? Software developers want to be helped and supported and the architecture should do that.
Realtracs started in 1996 and our technology has evolved dramatically over the last two decades. We haven’t always had a great architecture. In fact, we had one of the worst we’ve ever seen. But we fixed it and you can too.
