Release Process
Overview
Fiddler bridges the cloud and on-prem worlds by adopting the principles of continuous deployment, but only actually deploying to cloud customers every two weeks (or to airgapped customers every month).Core Principles
- Simplify: Fewer steps, more frequently.
- Highest leverage: Automate everything. No excuses.
- Tolerate failure: If something breaks, (a) disable its feature flag, (b) revert it, (c) cancel the release and move to the next one.
- Engineer not Operate: Proper test coverage is more important than the feature itself.
- Service Ownership: Engineers own delivering their code, in working order, into production. This includes all necessary testing, documentation, alerts/runbooks, upgrade plans, etc.
Release Schedule
Releases are time-based, not feature based. They ship when the clock strikes Tuesday, regardless of whether or not some particular work item is ready. New versions are deployed to SaaS customers on a biweekly schedule, and to on-prem customers on a monthly basis. Cadence EnvironmentsWhy Tuesday?
We release and deploy on Tuesdays for several reasons:- To have a predictable day of the week where everyone knows what will happen.
- Tuesday allows us several days to deal with any incidents that arise without paging out mental context over the weekend.
- Deploying or releasing on Monday would sometimes be pushed to Tuesday because of holidays.
- Monday gives the team time to prepare for the week ahead, should that be needed.
- Tuesday is a well-known deployment day in the industry, in the form of Patch Tuesday.
FAQ
What happens if I can't get my code into main by the cutoff?
What happens if I can't get my code into main by the cutoff?
If your code is not ready by the release branch cutoff it will not make it into the release (even if it’s just a day late!).If your code is not ready by the cutoff date, it will be included in the following release. It is each engineer’s responsibility to honor this process, even if it means your code is delivered to customers later than promised.Please be sure to follow our guide to making ambitious changes safely. Pay special attention to the part about database upgrades being 100% backwards compatible.
I need to get my code in by Monday?
I need to get my code in by Monday?
We call for all completed code to be merged to main by EOD, Monday. That is so we can have a successful run of nightly tests before we cut the branch at 9:00 Pacific. Get your code in by then! If you miss the branch cut, your code will go out in the following release (which is only two weeks off!).
What happens if we fail to qualify a release candidate?
What happens if we fail to qualify a release candidate?
If qualification cannot be completed successfully, despite disabling feature flags, we will cancel the release and spend the remainder of the release cycle addressing any remaining release blockers.
What happened to the "validation phase"?
What happened to the "validation phase"?
Instead of manual validation steps, automated testing must be used. Feature and Product owners must sign off their features — this is a pre-requisite for validation. Product owners must work closely with engineers throughout the development process.
What happened to "release blockers"?
What happened to "release blockers"?
Because our releases are time-based and not feature-based, there is no longer a concept of a release blocker. The release is either ready to go or not.
How do I <anything about feature flags>?
How do I <anything about feature flags>?
Common Q&A relating to feature flags:
| Question | Answer |
|---|---|
| How do we handle feature flags for features that also include database/schema changes? | Backwards compatibility — both old and new state need to work. |
| When do we settle on the feature flags for a release? | Feature flags should be turned on in code before the branch cut, like any other code. |
| Will we ever have feature flags turned on for only a few customers? | No — there should not be any specific feature flag setting per customer. |
| When do we clean up feature flags? | Feature flags should be deleted one release after they roll out. |
| How do we avoid accidental release of a feature? | It is the responsibility of the TL/developer to validate state of their feature flags. |
Why do we never decommission deprecated features or services?
Why do we never decommission deprecated features or services?
We absolutely should. We must. Carrying dead weight around slows us down. Please plan this accordingly in your sprints.The timeline for deprecation should be 6 months, after which the feature is removed. This is the direction, but must be communicated to our customers first.
How are docs aligned with this process?
How are docs aligned with this process?
We use GitBook to host our documentation. The docs process aligns exactly with the code development process because GitBook syncs directly with GitHub — docs will be written with PRs. GitBook also supports WYSIWYG editing from the UI that commits back to GitHub.
What about the Python client?
What about the Python client?
The client can be released separately from the main product because the product’s APIs must be backward compatible. The customer migrating from one version of the API to another must be time-independent from the upgrade of our product.Therefore, the client lives in its own repo, has its own version numbering scheme, and does not need to conform to the process documented here. However, as a matter of convenience, we will follow the same cadence most of the time.
Where do I merge my change?
Where do I merge my change?
Merge flow diagram coming soon.
Who deploys patch releases?
Who deploys patch releases?
Any engineer making a fix for a customer is fully empowered — and expected — to deploy that fix to the customer(s) themselves. This is the fastest and most reliable way of ensuring the fix is delivered. It’s faster because additional overhead is not needed between two people, and it is more reliable because the person who understands the problem is able to validate that it is working as intended.To build a release, see:To deploy a specific release to a customer, you can either edit files in the old
fiddler-labs/deployments repo or in the new fiddler-labs/cloud-deployments repo. Examples of how to prepare the PR can be found in the respective repos.When do docs cut over?
When do docs cut over?
Docs for the upcoming release are added to GitBook immediately after the release branch is cut. Docs for a new release are made public on Release Tuesday. While available on the site, they are not made the default until the deploys are about halfway through our Cloud customers. That timing is currently a judgement call made by the field team.
When do release plan pages get updated?
When do release plan pages get updated?
The release plan page is updated by the owner (engineer or PM) before the release is cut at 9 AM PT on Tuesday.
When are field announcements made about new features?
When are field announcements made about new features?
Field enablement messages come in Slack from Product on the Thursday or Friday after the release is cut.