Blue-Green, Canary and Feature Flags: Three Ways to Ship, Three Ways to Roll Back

The question at the core of all three rollback strategies is "How soon can you break the change?" Feature flags offer fast rollbacks: flick a switch and nullify the latest code commit. Canary and blue-green offer nothing quite as quick, but they can reroute traffic from the new version back to the previous quickly. So far, each technique gives you an answer, if you measure rollback in traffic and code.

They fall down when it comes to the database. Latency is the limit when you need to switch code, but time itself cannot undo a destructive schema change. A feature flag toggle may nullify the software rollout, but what if the new logic has already modified or deleted data that would no longer plot back to the old schema?

Canary and blue-green deployments solve the problem of software exposure, by shielding the majority of customers until you are sure the changes have not broken the live site. But what happens when the database change carries destructive logic that no canary or blue-green deployment can shield you from? Switching the router doesn't undrop a column.

Traffic Layer and Code Path

At their core, blue-green and canary deployments offer to reroute live traffic. A blue-green deployment maintains near-duplicate production environments: one is the current stable build, and the other is the new release candidate. When it is time to deploy, the old build keeps running, but you swap the load balancer or router to direct live traffic into the new environment.

Canary deployments save costs on infrastructure by not needing a complete duplicate, but they expose yourself to the danger of a new bug..

With a canary, the stable and new versions run in parallel on the same environment, but with live traffic directed in a ratio that de-risks the exposure, such as 5% on new. Your aim with canary is to expose the new version incrementally, so that a mistake in the deployment does not crash your whole user base.

Feature flags are an orthogonal play to either of those. Blue-green and canary care about traffic exposure; they want to rollout the software with as little risk as possible. Feature flags care about the visible effects on the user; they write the feature into the app, but hide it with a runtime boolean toggle. That option to hide does let you instantaneously alter the blend of client configurations with different flags on, but it does nothing to migrate or rollback the database.

Infrastructure vs Flags

The infrastructure cost of blue-green deployments has a floor - everything the new production environment touches needs. A full switch to the new environment wouldn't divide that load. Duplicating the environment is the secret to launching into a new environment, but unnecessary if the bug has already crashed your new database behind the old tag.. Canary operators often solve that by only deploying a fraction of the environment that the stable path needs, within the same cluster.

Feature flags, despite being a code change, are land-and-expand. Old app code stays in the old path, under the same old schema on the database, with the new feature introduced into the code.

Rollback doesn't touch the database

In an age where app outcomes are API-based, you cannot afford to neglect any progression that changes on the back-end. A breaking rollout should be impossible once the database state constrains the data model.

The expand-contract pattern rolls forward and backward across commits..

If you have ever heard of the expand-contract migration pattern, there are two moves: expand exposes your program to a new schema that exists alongside the old one, and contract removes the old schema path. But you have to expand before you contract, and only when both your code and data have caught up.

Conclusion

These three strategies cover traffic, configuration, and code. But none keep your data safe through rollbacks. Better be sure your database migration could rollback, or prepare to fail at something that none of them will save you from: a single row of corrupted data.