Feature Flags: Release Small and Often, Simplify Workflow
Last Updated: Oct 6, 2018
We’ve all heard the buzz about continuous delivery, but what does it mean exactly? Continuous delivery for an organization is the ability to deliver software safely, in short cycles and in a repeatable way. Ability to practice this process effectively will increase productivity, will keep your employees happy and most importantly will build a better rapport with your customers. But finding a development process that works for your organization does come with its challenges. Attempts to optimize engineering practices can sometimes increase complexity and tech debt if it’s not executed and maintained properly. We’ll discuss a few approaches to effective software delivery in more detail.
Two Extreme Approaches to Software Delivery
There are two extreme approaches that organizations take about software delivery. I’m going to take two real-world examples.
The first organization is a start-up and has 4 developers, responsible for implementing features, and the owner who assists with product management. In order to be competitive with other firms, the team wants to focus on speed and agility. Ultimately they want the client and customers who are funding the project to be happy. They are not too concerned with versioning and integration at this point since the project is in its early phases. The team is pretty small anyway so when builds break or a bug is introduced they can easily recover by undoing or fixing the problematic change. There are not too many customers using the product yet so instability is not desired but some of it is tolerated. The team is aware that they are cutting many corners but their velocity and productivity are pretty high and so is the satisfaction for them and the client.
The second organization is an enterprise with more than 10 different teams working on the same product and dealing with a large customer base.There is significant cross-team integration required for many of the features but teams have different work styles and release schedules. Different development workflows introduce many conflicts and bugs which often blocks other teams and causes frustration when teams aren’t able to meet their release deadlines. The outcome has management stepping in to introduce more steps to the development process. Teams get to try working with code freezes before releases, and engineers are asked to come up with a better stability solution. Engineers now introduce all possible tests they can think of and add that to continuous integration pipeline which slows the team further. Additional processes have significantly improved stability but productivity and number of features that are shipped has been reduced as well. Teams are happy about stability but frustrated about the team’s velocity and exhaustive testing and process. Customers are also not seeing much improvement in the product.
What is the Ideal Development Workflow?
“Survival rules are not stupid; they are simply over generalizations of rule we once needed for survival. We don’t want to simply throw them away. Survival rules can be transformed into less powerful forms, so that we can still use their wisdom without becoming incongruent”
— Jerry Weinberg
We could say that the start-up organization we mentioned is pretty agile, but also the codebase complexity is still manageable. The team is more or less aware of everything that is happening with the product and is able to recover quickly when the head of the codeline is broken. This is not the case for the enterprise organization. So, as the team grows, how can you achieve high velocity but still ensure quality and a stable bug free development environment? Well… you can’t! However, what you can do is alter the development workflow with one that works for your organization.
I want to focus on how we integrate our work because this is a crucial step in software development. Regardless of whether I am the sole developer or I am working with a team of developers, I can release software in small chunks or I can spend days’ worth of work and release them as larger chunks. Most often then not, the risk of deploying large chunks of work is higher. Sam Newman has a great talk on this; larger deltas means there is a higher chance that something can go wrong. This risk can come in various forms starting from bugs and defects but it can also be about subjective usability issues, eg. users don’t like the new feature or update.
When change is smaller the amount of change is smaller. This means bugs and defects can be traced, fixed or rolled back much faster. Also, you are able to get feedback much sooner from customers and iterate more quickly, eventually giving users what they really want.
To some teams, this might seem like a difficult task because, well, testing, peer reviews, deployments are all steps that take quite a bit of time. While the saying “if it hurts, do it more often” might not sound very helpful it could help you think creatively about optimizing the development workflow so that it is simpler to integrate code and deploy to production. Developers naturally want to do this and probably do it anyway.
Most of us have seen how Git Flow works right? GitFlow is a branching model for Git, created by Vincent Driessen. It has attracted a lot of attention because it is very well suited to collaboration and scaling the development team (1). A lot of companies today use this workflow. The main idea here is to keep trunk (or master in Git) stable so that it can be deployable at any time. This is a pretty good goal to have. However, to achieve this we introduce many branches. Consider what happens when your organization has hundreds of check-ins per day. With this branching model, it’s often convenient to keep feature branches for longer if they are stable enough and defer integration for later. Because each developer has their changes isolated from everyone else it becomes a race of who can check in changes sooner to avoid dealing with integration pains.
This might work for some companies and if it does that is great news. Branching, in principle, is intended for isolating your changes from the rest of developers. The more isolation of changes there is the higher the chance of collisions during integration with development codeline. In addition, the fact that all check-ins eventually end up in one development branch means that rolling back a feature branch becomes slightly tougher and riskier; your feature may include multiple check-ins and now many more check-ins from other developers. So instead of rolling back you might comment out some code, hide or disable a feature, and release anyway as is to avoid pains of reverting.
So we’ve talked about how small deltas and more frequent integration can reduce risk and also about the merge pains of using workflows like Git Flow. Trunk Based Development is a slightly different approach which companies like Google have been embracing for a few years now. The main idea is that everyone integrates small chunks of code, as frequently as possible and directly to trunk. But this sounds too good to be true. How can hundreds of developers integrate directly to the trunk and still ensure high quality? More on this below…
Few Words on Version Control and Policies
Before diving in trunk based development, it’s worth discussing the purpose of version control. First, it is a supporting discipline. When setting version control policies in place it is possible to have too many steps and have them to get in the way of “real work”. On the other hand, avoiding or misusing version control tools because of the perceived limitations causes problems too. For example, code freeze, in general, has problems because it forces developers to use alternate methods of versioning their code. That results in merge issues and stability issues right after code freeze has been lifted.
It’s important that the company’s branching model follows the business model. For example, if the company’s objective is to ship as often as possible then a complicated branching workflow is probably not ideal. Similarly, a mission-critical software with a very loose branching model might not be the most appropriate either.
Most companies fall somewhere in between where you want an active codeline that is usable by developers most of the time. In other words, it’s the balance between the complexity of the branching model and stability. If our goal is to integrate and ship often, smaller chunks at a time, then what do we need to make that happen?
When choosing the branching model we probably want to think about the tooling as well. What are the code scans and tests that we need to run and when? In the example of an enterprise organization, running tests has become slow because these are exhaustive tests. I’ve heard of companies where tests would run for hours and only then you would be able to merge your changes in. If we’re delivering code in smaller chunks do we necessarily run all tests? Having the option to run a selected test suite or smoke tests for smaller integrations and exhaustive tests before releases might make more sense as a workflow. There is no general pipeline that works for every project so it is up to teams to decide on how they want to optimize it.
Architecture with Organization in Mind
“Organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations.” (2)
— M. Conway
When a team is asked to come up with a solution about a feature or an entire system then systems architecture is a topic that comes to mind. Depending on who you ask you might get different answers. I’ve seen employees with different roles stepping in to make these decisions. If you ask an engineer how to build a system they will break it down into components, define their relationships and suggest an implementation strategy. Product managers will have a completely different perspective when asked the same question. They will be more concerned about the overall vision of the product and often act as feature gatekeepers preventing over-engineering irrelevant features.
There are many variables involved when making architecture decisions like how much will it cost? when can it be delivered? how productive will the teams be? How do the teams work together? All are important but I want to focus on the last one.
To come up with a product design we can’t just think of physical artifacts but must also consider how people interact with each other in the company. The product, after all, is the output of people in the organization. Are they in the same office or spread in different offices and on different time zones? Team dynamics can be important too. A team can be in the same office but have minimal interaction and the opposite can be true for teams in different offices.
Introducing new policies can influence how people communicate and do their work. While this can often help it can also frustrate people who are not used to the new working style. An alternative is to define architecture with organization structure in mind. This is often more accommodating to employees but not always easy, so re-structuring is usually needed. For example, people who don’t interact with teammates may be more comfortable with more process and structure, while people that interact with team frequently see too much process as an obstacle.
Trunk-Based Development (TBD)
A source-control branching model, where developers collaborate on code in a single branch called ‘trunk’ *, resist any pressure to create other long-lived development branches by employing documented techniques. They therefore avoid merge hell, do not break the build, and live happily ever after (3).
This is a branching model in the software development world and TrunkBasedDevelopment.com is a great resource for details. The main idea is that all developers collaborate in one single branch called the trunk, or mainline as it was called earlier. In Git this is master by default but you could also call it trunk.
With TBD developers are able to avoid long-lived branches which in this branching model are roots of all evil or so-called merge hell. Since everything is in trunk developers are able to integrate often with the mainline and are much more up to date then when using branching models like Git Flow.
Technically branches can exist but these are short-lived branches which typically live less than a day. Someone other than the original developer is responsible for doing peer reviews and these are followed by typical CI. Release branches are the only real branches which developers do not create. Release engineers are chosen to create releases, cherry-pick fixes or roll back changes when needed.
The fact that branches are short-lived it means that the changes with each commit are smaller. One of the challenges with TBD is when the tip of the mainline is broken and this can block other developers. Developers will have to be disciplined enough not to break trunk. If the trunk is broken then change will need to be reversed or fixed as soon as possible and any of the developers can do this.
CI and tooling in general is something that companies will invest in to ensure issues are caught before they go to trunk. But keeping code quality in check is good investment anyway! The fact that branches are short lived, however, means that changes are smaller too and they can be reversed easier.
Trunk Based Development With Feature Flags
A common question is how is it possible to have short-lived branches in cases where features are larger and span several development days? Using feature flags is one common approach to make sure we’re integrating often but are also controlling how these features are released. With this approach, a big feature can be broken down in smaller chunks each of which can be released in the dark. Only when the team is comfortable enabling the feature will the world see it.
This also gives developers the ability to test their features integrated with mainline incrementally while features are not yet visible. These can be automated tests that are run by CI but can also be tests that developers can run in isolation from what is visible in production. There are multiple methods to do this such as enabling the feature only for a group of users or IP addresses.
Ability to enable a feature for a segment of users can have additional benefits as well such as canary releasing and A/B testing. Features can be enabled to various user segments, feedback can be gathered, and multiple iterations may happen to validate a feature prior to rolling it out to everyone.
Another common feature flags use case incredibly handy is the ability to test major upgrades. For example, we might want to update a critical path algorithm but want to make sure that it we break the problem down into smaller pieces and test gradually until we are happy with the upgrade.
To illustrate this use case we can consider replacing wheels of a moving vehicle, except that this vehicle could have thousands of users and is always in construction (5). First, we de-couple wheels from the car such that new wheels can be easily mounted on. Then we have a decision point created which will use existing wheels by default. Then we slowly swap wheels from old wheel to new, one at a time. Finally, we are happy with new wheels and remove the unneeded decision point.
TBD with Feature Flags is a pretty powerful method for rapid development but can be misused as well leading to unwanted tech debt. Just like branches, feature flags are intended to be short-lived and ideally should be removed as soon as the feature is completed. Martin Fowler has a great article on some of the best practices when working with feature flags.
To Summarize…
- Integrate often and smaller chunks at a time
- Define workflow that works best for your organization business model
- Consider organization structure when defining dev workflow
- Consider benefits of Trunk Based Development and Feature Flags; it could work for you!
References
(1) Introducing Git Flow — GitHub.io
(2) Conway’s Law — Mel Conway
(3) Introduction — Trunk Based Development
(4) Feature Flag Use Cases — ProbeBeta.com
(5) Developing Gmail’s New Look — Google Blog
