Coding with Feature Flags: How-to Guide and Best Practices

The terms feature flag or feature toggle are pretty self-explanatory, probably because they have been around for a very long time. Most developers have used them in one way because they are so simple to use.

When I got my first couple of development projects, in small teams working on small applications, developers felt that the most effective way to control slightly larger features was through some global config file or environment file. This was not seen as a method but rather a convenience to prevent over-complicating things. You’ve probably seen this approach used in your views with, let's say, a new shopping cart that you want to try out:

{{#if config.showNewCart}}
   {{> newCart}}
{{else}}
   {{> cart}}
{{/if author}}

… or maybe you recently changed your payment gateway or shipping vendor. In your controller code, you create a decision point. In the development environment you use a new vendor, but on production environment you still use the existing vendor. Hopefully you have tests and if you do they will continue to test the old vendor.

if(config.useMoneris) {
   this.sendTransactionMoneris(...);
}
else {
   this.sendTransactionPayPal(...);
}

All of this makes sense right? When all is tested and working the config is updated and push a button: a new feature is released. Feature flags are still the same conceptually, except that a lot more has happened with them in the last several years. Complete open-source libraries have been written just to manage feature flags. There are also SaaS products that allow turning certain code paths on and off at run-time. Companies like Netflix [1], Google [2], Flickr [3] and Reddit have been using them not just to show and hide features but to also increase productivity, mitigate risk, target their audience, A/B test their traffic, and for various other use cases.

Working with Feature Flags

So let’s say you’re convinced that working with feature flags is going to bring you all the benefits that many claim that they bring. What are some of the best practices when using feature flags?

One of the main concerns that developers have with feature toggles is the idea of having toggles everywhere in code. Let’s face it, having if / else logic everywhere in your code does not look elegant at all. It’s also fairly easy to misuse them and make your implementation a lot more difficult than it should be. Below are some common best practices that you should consider.

Types of Feature Flags

There are a few types of feature flags and they can be categorized by a couple of main characteristics: longevity and dynamism.

To explain these categories further, I will list a few common ones that are often talked about. I’m sure you might have some of your own use cases too.

• Release toggles: More often than not these are on/off switches that are used to control whether a feature is enabled or not. They are short-lived and removed after the feature is released.
• Operational toggles: Typically used to control flow on the back end. An example could be algorithm changes, upgrading or retiring old APIs, etc. These are also short-lived and ought to be removed after the task is complete.
• Experimental Toggles: These are typically used for A/B or multivariate testing and are typically longer-lived flags that can be removed once we no longer need to collect user testing data.
• Permission Toggles: This is commonly talked about and is used to control the product experience users have. I think doing this is actually a bad use case for feature flags and personally don’t recommend it. This sort of information would be coming from your user properties.

Longevity, as the name suggests, has to do with how long we need the feature flag in our code.

It is a good practice to keep feature flags are short-lived as possible. Keeping their number low is equally important.

A feature flag splits your code paths into two or sometimes more. This means that now you have that many more paths to test. Releasing dead code in production can be useful for continuous delivery, but imagine what can happen if you toggle a 6-month-old feature flag. It could be a recipe for disaster. Even if all paths are active in production unless you are doing A/B testing, using a feature flag for a long time only increases complexity, adds to overall bloat in your code, and is something to be avoided.

Dynamism has to do with how much we need to modify a flag during its lifetime. To illustrate let’s consider a few ways that we store and retrieve feature flags:

• Hard-coded constants: Slightly hacky but we’ve all created constants which could serve as default input. Then we later change the input and re-deploy. These are pretty static since they require a re-deploy.
• Config files: These can come as JSON, template files, and could also be your environment config files. They’re convenient and also require a re-deploy.
• Database configs: We can store application settings in a database. It is a pretty standard and convenient place to store your settings. These settings can often be updated easily in the database and often use an administration interface.
• Open Source Libraries: Build specifically for feature flag management and can usually be updated on run-time. Installation and maintenance is often the biggest challenge with such libraries or even proprietary solutions.
• Third-Party Service: SaaS model type of solution, is highly dynamic, and usually comes with SDKs, documentation, analytics and so on.

Coding with Feature Flags

One of the pet peeves that many developers have is having too much branching in your code and feature flags are often avoided for this reason.

With the views layer, it’s usually easier to add and remove toggles. However, with flags being closer to core code, adding feature flags should be done more carefully. Not only can increase code complexity and look less elegant, but also will add to your overall tech debt that you later have to clean.

function search() {
   if(flagSdk.getFlag(NEW_ALGORITHM)) {
      // new implementation
   }
   else {
      // old implementation
   }
}

A slightly better approach could be to pass flags as parameters. While you still need to refactor this code later at least your function isn’t coupled with whatever feature flag solution you chose to use:

function search(useNewAlgorithm) {
   if(useNewAlgorithm) {
      // new implementation
   }
   else {
      // old implementation
   }
}

Can we do better? It is also possible to create a new module altogether and leave the existing code intact. This is a safer approach especially if you are updating core code. The other benefit to this is that you won’t need to update your test code when you add and remove the feature flag. The feature flag check will typically be on your controller level logic, eg. your controller if you are using MVC pattern or something similar.

... old search module
function search(...) {
   // old implementation
}
... new search module
function search(...) {
   // new implementation
}
... controller
let modSearch = useNewAlgorithm ? NewSearch : OldSearch;
modeSearch.search(…);

A common and useful pattern is to create an abstraction layer, a factory, that only deals with the provisioning of libraries or objects that the controller expects. With this approach, you will keep adding and removing libraries as needed, and the feature flag logic will be in the factory. After cleaning up the flags you can remove the decision point from factory and controller completely, or you can only remove the old path and leave the new path as a default option.

... controller
let factory = new MyFeatures(configsJson);
fff.getSearch().search(...);

Deciding When to Create a Feature Flags

There is often some confusion around when feature flags should be created. Feature flags, when they can be updated on run-time, decouple rollouts from deployments. This makes it easier for product managers to release features independently or with very little help from engineers. This is a great thing. There are times when product managers want to have more control over what is released which means more feature flags.

Giving control to release engineers or managers can work fine. That frees up developers to do work without having to worry about managing releases. However, it’s important to keep the number of feature flags low. Just like merging hell, there is also feature flag hell. Having too many feature flags means there are many more code paths, code complexity increases and testing becomes harder. Because there are many more paths and likely many dead paths the risk of those paths causing issues after they are enabled increases. Well… We’re always careful about writing our tests right? Even then I would ask why there are so many flags?

• Are flags being created for features that are too small? I’ve been asked once whether a flag should be used to fix a bug. While it’s possible to make bugs a lot worse, this is where I would draw the line.
• Are there many flags because they’re not being cleaned up? It’s easy to get caught up with other work and never clean up after ourselves. This is especially true in cases where stakeholders have seen the feature and are happy with it. It is the developer’s responsibility to ensure they are cleaned up.
• Are teams not coordinating about feature flags that they are using? If different teams are creating multiple feature flags for the same feature testing and releasing can be a pretty painful task.

So to avoid feature flag hell, create them only when it makes sense, clean them up after you are done and always check if it makes sense to re-use flags before creating your own.

Monitoring Feature Flag Usage and Clean Up

We touched a bit on why it’s important to remove an unneeded flag. When your codebase grows it can be a bit more challenging to see what code paths are being used. Removing feature flags is often easy but it usually comes with some extra effort, eg. deprecating old dead code or deprecated libraries.

Companies like Google and Facebook invest quite a bit in tooling to ensure their code quality is high. This is how they are able to have thousands of check-ins per day, use trunk-based development, and still have high-quality code. Automation and tools like Sonar or CodeClimate can be pretty handy to detect code issues which may be a result of a feature flag clean-up.

SaaS feature flag solutions, which I personally fond of, have pretty thorough stats and logs that help you determine the usage of each flag that’s still active in the system. Using these stats you can see what feature flags are no longer changing, which usually means a feature is released or abandoned and that the flag can now be safely removed.

Proprietary Solution vs Feature Flag Platform

When deciding on the feature flag solution it’s worth considering what your needs are before you start. If it’s a matter of enabling or disabling features where a deployment per feature release isn’t so prohibitive then maybe a simple config will do.

As you start scaling your product, you might find there are different benefits you can get from feature flags. For example, what if I wanted to roll out a change gradually to some users first and slowly ramp that up until it’s rolled out to all users. Companies like Netflix have a pretty large number of users and can’t afford to not test their new features with a smaller set of users first.

You might want to collect data about your users as well too to see how your users are responding to your releases. Using this feedback you can then iterate over features and improve the experience before rolling it out to everyone.

These use cases are examples where a simple feature flag solution will not meet your needs. Implementing your own robust feature flags solution is possible, but it can be quite costly and you will have to maintain it as well. A third-party solution that is within your budget could be a good option. There are plenty of companies like LaunchDarkly, Split, ProbeBeta, and Optimizely that could have the feature set you are looking for. Provided you have an abstraction layer, changing your feature flag solution later shouldn’t be a big deal since most of them function similarly.

Some History About … Branching

CI is about exposing our changes as frequently as possible, so that we can get great feedback on our ideas, branching, any form of branching, is about isolating change. A branch is, by-design, intended to hide change in one part of the code from other developers. It is antithetical to CI…

— Farley

In the old days, when developers were using versioning tools like CVS and Subversion, branching was a pretty big undertaking. You wouldn’t branch your code every time you wanted to add a button in a page. Most developers that I’ve talked to consider merging in CVS to be a hard and horrible experience. When Git came around you would have your own branch as soon as you cloned your repository. This gave developers the ability to branch as often as they like. They would have a few branches for experimentation, one for maintenance, one or more for active development and releases and so on.

With increasing number of branches, however, merging these branches becomes important too. How do you deal with merging when you have hundreds of developers contributing to a repository where each developer has several active branches around?

Git is amazing at how it handles branching. But because creating branches is cheap, it becomes important to find a method to manage all these branches. This is especially important, of course, when the intention is to bring them into trunk. There are many complex workflows introduced with GitFlow being one of them. I’ve read numerous articles on branching models used by several companies and the reason why it works for them. I honestly have not seen many that are simple. Sometimes there are too many development lines being maintained, all of which take time away from developers on daily basis.

I find that the branching model is a is very commonly talked about. Git workflow somehow always ends up being a discussion topic. Some developers tend to gravitate to a particular workflow because it probably worked well for them in the past. This is fine, but when fixing merge conflicts and release planning starts to take a good portion of developer’s day then it’s worth asking whether the headache of managing branches is worth it.

Branching isn’t bad, it’s actually good — it isolates your changes from the rest of the team. Problems usually arise when you have long lived branches that have plenty of check-ins. Git makes the merging process quite easy. The issue, however, is that your work could be so far behind (or ahead) from the rest of the team. If you use no branches or short-lived branches your work will be much closer to everyone else’s.

You might be wondering what this have to do with feature flags. CI/CD is probably a whole different topic some of which is covered in Feature Flags: Release Small and Often, and in much more detail in TrunkBasedDevelopment.com. It’s definitely worth mentioning though that if you are thinking of trunk based development or something close to it then feature flags are going to be a life saver.

Final Thoughts …

• Feature flags have been around for a long time because they are simple to use and developers felt productive when using them. We’ve introduced sophisticated tools and processes to improve velocity and quality. With this, we’ve also added complexity. It might be a time to simplify and bring the flags back.
• Feature flags can work well for companies but in general it’s best to keep feature flags short-lived and small in number. Too many flags or long-lived flags add complexity and are deliberate tech debt. Cleaning up feature flags is important to keep a codebase sane.
• Using feature flags can help companies to ship more often, minimize risk, increase productivity, and even help with targeting users and A/B test features. Evaluating a company’s branching strategy with feature flags can be very worth it, especially if you need to ship frequently.
• Keep coupling with feature flags service to minimum, create a config abstraction layer and try to avoid having flags in your core.

References

[1] — Preparing the Netflix API for Deployment — neflix.com
[2] — Developing Gmail’s new Look — Official Gmail Blog
[3] — Flipping Out — code.flickr.net