Code Freeze…but the application is still warm!

Photo by Ben Kolde on Unsplash

Yes, I know, I know… especially if you are working into Agile Framework, you are thinking that after each sprint your master stream contains already shippable software with added business value, and that code freeze strategy is an “Agile anti-pattern”.

At this point, are you really really sure that code freeze is not worth it?

If you work in a high tech company, with a complex application with several dependencies that require alignments across different groups, how many of you really think your master branch can be released every 2 weeks to customers, guaranteeing the same high quality? If so, hit your feet on the ground, I challenge you to do that, and I will use my superhero powers to hear and count them!

Wait.. wait… someone is hitting his floor… so this can only mean that in your process you have set automated tests with high coverage, executed as gate quality for each delivery, and regression test executed regularly. Well done! But…if the sky is not so clear and blue like this, then, keep reading. You might get inspired!

What is a stabilization branch?

Stabilizing a release means that at a certain point, a branch from the current development stream is created (yellow), becoming the release branch. The team has the «mission» to turn it into a releasable state.

Concept of stabilization branch (yellow) in which only critical bug fixes are delivered (blue) keeping the develop branch separated (purple)

Some elements that can identify “the releasable state” are:

  • the documentations are up-to-date (release Notes, user manual)
  • all critical bugs are fixed
  • all risks are mitigated
  • all regression tests have passed and testers have given their «benediction» (Haaaallelluuiaaa!!) for proceeding to share the product with the rest of the world.

The Dilemma: ‘To change not to change’?

The dilemma

Talking about stabilization is equivalent to talking about making decisions.

As soon as a release date is approaching, developers might experience a last-minute feature rush to finish their changes and deliver in time. This is against the result all aim to achieve. The more changes one tries to cram into a release at the last minute, the more the code is destabilized. This is equivalent to say that more new bugs are usually introduced.

On the other hand, if we don’t fix bugs, the customers will suffer from those, and the quality and usability of the product, along with its reputation, are gone. So the dilemma.

Well well well, please stop crying, and let’s think about some solutions and approaches on how to face it.

Ground Rules for saying NO!

There are some ground rules that developers can agree, in theory, about the type of changes allowed in the release stream during its stabilization:

  • Severe bugs without workarounds
  • Low risk and non-core changes that follow formalized guidelines for measuring risk
  • Documentation update or error messaging (except when they are considered part of the interface and must remain stable)

But in some cases, the formalization cannot replace the judgment from human, and… ah! ah! The challenge starts here…! Each developer is aiming to fix bugs, optimize the last-minute delivered feature, and see their own best and favorite changeset into the release.

The risk is to have only people motivated to change code and not enough people motivate to bar them! Considering this, the stabilization process becomes a process of creating a mechanism for saying “NO”.

Pay attention because the mechanism needs to be in place in a way that:

  • does not “hurt feelings” and demotivate developers
  • accomplishes the priority defined by the Product Owner and expectations from stakeholders
  • won’t prevent deserving changes from getting into the release

Strategies for Saying NO!

  1. ‘Heroe-ship’ by release owner

The group nominates a “release owner” that has the final say over the changes for the release. Not an easy responsibility, because there might be discussions and arguments for having such a change during the stabilization phase, but the group must grant to the release owner enough authority to make the final decision.

Because of this, the release owner shall have technical skills to understand the impact of the changeset, but he/she also shall have the soft skill to deal with people to avoid causing hurt feelings.

The release owner gives mainly reasons related to testing effort and coverage, which can degrade confidence after delivery, and the risk becomes out of control.

As a good practice, the release owner shall not be the same person as the project leader or the product owner to avoid ‘conflicts of interest’.

2. Voting on Changes

This is in opposition to the previous strategy. In this case, developers and the team have the last say, by voting, on what changes are reasonable to include in the release.

Wait wait wait… wait! Are we now saying that developers can decide what to change in the stabilization branch when we stated at the beginning of this article that developers are mainly ‘motivated’ to accept changes?

Since the most important function of release stabilization is to exclude changes, it’s important to design the voting system in such a way that getting a change into the release involves agreement by all team members, so that including a change shall need more than just a simple majority.

In order for a change to be applied to the release branch, in a small team, at least three developers must vote in favor of it, and none against. A single “no” vote is enough to stop the change from being included; a “no” vote in a release context is equivalent to a veto. Naturally, any such vote must be accompanied by a justification, and in theory, the veto could be overridden if enough people feel it is unreasonable and force a special vote over it. In practice, this never happens. People are conservative around releases anyway, and when someone feels strongly enough to veto the inclusion of a change, there’s usually a good reason for it.

Because the release procedure is deliberately biased toward conservatism, the justifications offered for vetoes are sometimes procedural rather than technical. For example, a person may feel that a change is well-written and unlikely to cause any new bugs, but he votes against its inclusion in release simply because:

  • it’s too big
  • had a gut feeling that the change needed more testing, even though they couldn’t spot any bugs in it by inspection.

Validate stability — KPI &Metrics

Photo by Markus Winkler on Unsplash

One “democratic” way to create a mechanism to “say no” is to define data-driven decisions.

Developers estimate the impact of their change based on T-Shirt size (S/L/XL) magnitude. After any delivery, there will be uncertainty about how much the stabilization stream is effectively stable.

Based on actual data, I will propose a strategy to enable a trigger to question deliveries and analyze and quantify the overall stability. First of all, we identify which data are correlated to our stability. The repository/history of the source code is the answer. In fact, putting hands-on scripting activity, and prettifying delivery’s logging, is easy to obtain data that provide meaningful information in a real dev working environment.

Threshold as trigger

In the following graphs each line represents a component, the x-axis values are time in days, and the y-axis is the count of the delivered changes in lines of code.

Use Case 1: Big delivery over a threshold

Big Bang Delivery changes

Green, red and purple components are not affected by changes. The blue seems to have some constant bug fixes, with limited changes approaching the release date thus acceptable.

The yellow is an interesting case. Two big jumps take place in the first phase of stabilization. Definitely, for the sake of stability, this should not be allowed and more people should be involved to agree to accept the risk. The second part of the stabilization validates that the big changes did not introduce instability, so far, this mitigates the risk to keep the big changes, but it is a bad practice if the team aim to limit the risk and stabilize the stream.

If the team agreed on a threshold for the max number of changes allowed in a delivery, the spikes in yellow would represent a trigger to put in the discussion, and so, to stop/continue/revert the delivery.

This is the moment to act and agree “sorry, the risk is not acceptable, not for this release!”.

Use Case 2: Daily constant changes under a defined threshold

The following graph shows incremental daily changes until the end of the first part of the stabilization phase. The continuous delivery of changes, in the first phase, indicates mainly a behavior associated with bug fixes for the feature delivered just before the stabilization phase started.

Stability reached mid stabilization phase

When the delta of changes is below a certain threshold, most likely this is associated with critical bug fixes and, the risk is contained and mitigated by the associated business value. The team should question the reason for changes in the initial days, and evaluate the root cause of that.

My 2 cents… it is due to the rush to deliver before the feature-complete date when the chance to lose the train to have their changes included in the release.

After the initial “instability” all components were not affected anymore by notable changes, so this is an example of how effectively the team stabilize the stream and gave time to testers to perform extra regression tests. Well done guys!

Put in place in the process a threshold that enables/triggers a mechanism to question the entity of the delivery.

KPI: Consecutive days of stability before release

We cannot rely on feelings to measure stability. This is the reason why we should provide a number that is representative of our stability. In other words, I want to define a measurable parameter (KPI) that enables tracking stability release after release.

From the graph we have analyzed before, a certain degree of stability is reached if limited changesets are delivered. This brings to confirm that:

the number of consecutive days in which changes are zero or below a very few limited numbers of changes is a KPI that will provide a reference number of each release, representative of one aspect of the release stability.

If KPI starts to degrade, people require to analyze the reason for this:

  • too many changes introduced
  • feature not completed once started stabilization
  • late testing
  • etc. etc. etc. …

Take-away

I would like to conclude this article with some takeaway messages that summarize in a few lines the essence of this article that can be seeded in you.

Photo by Bernard Hermant on Unsplash
  • There is a risk to have only people motivated to change code and not enough people motivate to bar them.
  • The stabilization process is a process of creating a mechanism for saying “NO”.
  • One “democratic” way to create a mechanism to “say no” is to define data-driven decisions.
  • Define a threshold that enables/triggers a mechanism to question the entity of the delivery.
  • The number of consecutive days in which changes are zero or below a very few limited numbers of changes is a KPI that will provide a reference number of each release, representative of one aspect of the release stability.

Sw Architect — Curious about new technologies! I don’t like to only watch them… I need to touch them!

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Develop your first Spring Boot app in 5 mins

Django REST Framework AS Fast AS Possible

K8s Use Case

[String] LC 680. Valid Palindrome II

My Week-3 at JADU a Full Stack Development Program.

How to Run Selenium Tests in Docker

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
GIOVANNI MARIOTTA

GIOVANNI MARIOTTA

Sw Architect — Curious about new technologies! I don’t like to only watch them… I need to touch them!

More from Medium

What is Change Tracking In SQL Server?

Do You Know Git?

The PEMDAS of T-SQL

What is SQL?