Unity: release early, release often… release daily! (part 1)

This post is part of the Unity daily release process blog post suite. This is part one, you can find:

• part 2 on upstream merge process
• part 3 on the daily release machinery
• part 4 on how dependencies are handled between stacks
• part 5 for a FAQ and conclusion

For almost the past 2 weeks (and some months for other part of the stacks), we have automated daily release of most of the Unity components directly delivered to Ubuntu raring. And you can see those different kinds of automated uploads published to the ubuntu archive.

I'm really thrilled about this achievement that we discussed and setup as a goal for the next release at past UDS.

Why?

The whole Unity stack has grown tremendously in the past 3 years. At the time, we were able to release all components, plus packaging/uploading to ubuntu in less than an hour! Keeping the one week release cadence by then was quite easy, even though a lot of work. The benefit was that it enabled us to ensure that we have a fluid process to push what we developped upstream to our users.

As of today, teams have grown by quite some extends, and if we count everything that we develop for Unity nowadays, we have more than 60 components. This covers from indicators to the theme engine, from the open input framework to all our tests infrastructure like autopilot, from webapps to web credentials, from lenses to libunity, and finally from compiz to Unity itself without forgetting nux, bamf… and the family is still growing rapidly with a bunch of new scopes coming down the pipe through the 100 scopes project, our own SDK for the Ubuntu phone, the example applications for this platform we are about to upload to Ubuntu as well… Well, you got it, the story is far from ending!

So, it's clear that those numbers of components that we develop and support will only go higher and higher. The integration team already scaled by large extends their work hours and rush to get everything delivered timely to our user base^[1]. However, it's with no question that we won't be able to do that forever. We don't want as well introducing artificial delays to our own upstream on when we are delivering stuff to our users. We needed to solve that issue while not paying any price on quality, nor balancing the experience we deliver to our users. We want to keep high standards, and even, why not allying this need while providing an even a better, more reliable, and better evaluation before releasing of what we eventually upload to Ubuntu from our upstreams. Getting our cake and eat it too! :)

Trying to change for the better

What was done in the last couple of cycles was to separate between 2 groups the delivery of those packages to the users. There was an upstream integration team, which will hand over to the Ubuntu platform team theorically ready-to-upload packages, making the reviews, helping them, fixing some issues and finally sponsoring their work to Ubuntu. However, this didn't really work for various reasons and we quickly realized that this ended up just complexifying the process instead of easing it out. You can see a diagram of where we ended up when looking back at the situation:

Seems easy isn't it? ;) Due to those inner loops and gotchas, in addition to the whole new set of components, we went from a weekly release cadence to doing 4 to 5 solid releases during a whole cycle.

Discussing about it with Rick Spencer, he gave me a blank card on thinking how we can make this more fluid to our users and developers. Indeed, with all the work piling up, it wasn't possible to release immediatly the good work that upstream did in the past days, which can lead to some frustration as well. I clearly remember that Rick used the term "think about platform^[2] as a service". This kind of immediatly echoed in me, and I thought, "why not trying that… why not releasing all our code everyday and delivering a service enabling us to do that?"

Even if not planned from the beginning (sorry, no dark, hidden, evil plan here), thinking about it, this makes sense as part of some kind a logical progression from where we started since Unity exists:

• Releasing every week, trying manually to get the release in a reasonable shape before uploading to Ubuntu
• Raise the quality bar, put some processes for merge reviewing.
• Adding Acceptance Criterias thanks to Jason Warner, ensuring that we are getting some more and more good tests and formal conditions on doing a release
• Automate those merges through a bot, ensuring that every commits in trunk builds fine, that unit tests are passing
• Raise again the quality bar, adding more and more integration tests

Being able and ensuring we are able to release daily seems really, looking at it, par of the next logical step! But it wouldn't have been possible without all those past achievements.

Advantages of daily releases

It's quite immediate to see a bunch of positive aspects of doing daily releases:

• We can spot way faster regressions. If a new issue arose, it's easier to bisect through packages and find the day when the new regression or incorrect behavior started to happen, then, looking at the few commits to trunk (3-5?) that was done this day and pinpoint what introduced it.
• This enables us to deliver everything in a rapid, reliable, predictable and fluid process. We won't have crazy rushes as we had in the past cycles around goal dates like feature freezes to get everything in the hand of the user by then. This will be delivered automatically the day after to everyone.
• I see also this as a strong motivation for the whole community who contributes to those projects. Not having to wait a random date hidden in a wiki for a "planned release date" to see the hard work you put into your code to be propagated to the user's machines. You can immediately see the effect of your hacking on the broader community. If it's reviewed and approved for merging (and tests passes, I'll come back to that later), it will be in Ubuntu tomorrow, less than 24 hours after your work reached trunk! How awesome is that?
• This also means that developers will only need to build the components they are working on. No need for instance to rebuild compiz or nux to do an Unity patch because the API changed and you need "latest everything" to build. Lowering the entry for contributing and chances that you have unwanted files in /usr/local staying around conflicting with the system install.

Challenges of daily release

Sure, this comes with various risks that we had to take into account when designing this new process:

• The main one is "yeah, it's automated, how can you be sure you don't break Ubuntu pushing blindly upstream code to it?". It's a reasonable objection and we didn't ignore it at all (having the history of years of personal pain on what it takes to get a release out in an acceptable shape to push to Ubuntu) :)
• How to interact properly with the Ubuntu processes? Only core developpers, motus, and per-package uploaders have upload rights to Ubuntu. Will this new process in some way give our internal upstream the keys to the archives, without having them proper upload rights?
• How ensuring packaging changes and upstream changes are in sync, when preparing a daily release?
• How making useful information in the changelog so that someone not following closely upstream merges but only looking at the published packages in raring-changes mailing list or update-manager can see what mainly changed in an upload?
• How to deal with ABI breaks and such transitions, especially when they are cross-stacks (like a bamf API change impacting both indicators and Unity)? How to ensure what we deliver is consistent across the whole stacks of components?
• How do we ensure that we are not bindly releasing useless changes (or even an upload with no change at all) and so, using more bandwith, build time, and so on, for nothing?

How then?

I'll detail much of those questions and how we try to address those challenges in the subsequent suite of blog posts. Just for now, to sum up, we have a good automated test suite, and stacks are only uploaded to Ubuntu if:

1. their internal and integration tests are passing above a certain theshold of accepted failures.
2. they don't regress other stacks

Stabilizing those tests to get reliable result was a tremendous work for a cross-team effort, and I'm really glad that we are confident in them to finally enable those daily release.

On another hand, additional control is made to ensure that packaging changes are not committed without someone with upload right being in the loop for acking the final change before the copy to the distribution happens.

Of course, the whole machinery is not limited to this and is in fact way more complicated, I’ll have the pleasure the write about those in separate blog posts in the following days. Stay tuned!