# Unity: release early, release often… release daily! (part 5 and conclusion)

This post is part of the Unity daily release process blog post suite.

After a week to let people ask questions about the daily release process, I guess it's time to catch up and conclude this serie with a FAQ and some thoughts for the future.

## FAQ

The FAQ is divided in multiple sequences depending on your role in the development of ubuntu, with the hope that you will be able to find what you are looking for quicker this way. If you have any question that isn't addressed below, please ping didrocks on IRC freenode (#ubuntu-unity) and I'll complete this page.

### Upstream developers

#### What's needed to be done when proposing a branch or reviewing?

As discussed in this part, when proposing a branch or reviewing a peer's work, multiple rules have been established. The developers needs to:

• Design needs to acknowledge the change if there is any visual change involved
• Both the developer, reviewer and the integration team ensure that ubuntu processes are followed (UI Freeze/Feature Freeze for instance). If exceptions are required, they check before approving for merging that they are acknowledged by different parts. The integration team can help smooth to have this happened, but the requests should emerge from developers.
• Relevant bugs are attached to the merge proposal. This is useful for tracking what changed and for generating the changelog as we'll see in the next part
• Review of new/modified tests (for existence)
• They ensure that it builds and unit tests are passing automated
• Another important one, especially when you refactor, is to ensure that integration tests are still passing
• Review of the change itself by another peer contributor
• If the change seems important enough to have a bug report linked, ensure that the merge request is linked to a bug report (and you will get all the praise in debian/changelog as well!)
• If packaging changes are needed, ping the integration team so that they acknowledge them and ensure the packaging changes are part of the merge proposal.

When you approve a change, do not forget to check that there is a commit message in launchpad provided and to set the global status to "approved".

#### When do changes need to land in a coherent piece?

Some changes can happen and touch various components (even across different stacks). To avoid loosing a daily release because only half of transition landed in some upstream projects (and so, tests will fail because they catch this issues, isn't it? ;)), it's asked to ensure that all transitions are merged by 00 UTC. Then, you can start approving again transition starting from 06 UTC. We can try to make this window shorter in the future, but let's see first how this work in practice. For any big transition, please coordinate with the ubuntu-unity team team so that they can give a hand.

#### I have a change involving packaging modifications

As told in a previous section, just ask for the ubuntu-unity team to do some review or assisting in doing those changes.

#### Rerun a partial stack daily release, rebuilding only some components

$cu2d-run -R <stack_name> -r <release> <components> /!\ this will keep the state and take into account (failure to build state for instance) and the binaries packages for the other components which were part of this daily release, but not rebuilt. for instance, keeping dee and libunity, but rebuilding compiz and unity from the unity, head release stack:$ cu2d-run -R unity compiz unity

The integration tests will still take everything we have (so including dee and libunity) at the latest stack with the new trunk content for compiz and unity when we relaunched this command.

$cu2d-run -R <stack_name> -r <release> --check-with-whole-ppa This command doesn't rebuild anything, just run the "check" step, but with the whole ppa content (basically doing a dist-upgrading instead of selecting only desired components from this stack). This is handy when for instance a transition involved the indicator and unity stack. So the indicator integration tests failed (we don't have the latest unity stack built yet). Unity built and tests pass. To be able to release the indicator stack as well as the unity stack, we need the indicator stack to publish, for that, we relaunch only the indicator tests, but with the whole ppa content, meaning last Unity. In that case, the publish for the indicators will be manual (you will need to force the publication) and as well, you should ensure that you will publish the unity stack at the same time as well to have everything in coherence copied to the -proposed pocket. More information on the stack dependency in this section. #### Adding/removing components to a stack The stack files are living in jenkins/etc/. This yaml file structure should be straightforward. Note that the tuple component: branch_location is optional (it will be set to lp:component if not provided). So add/change what you need in those file. Be aware that all running jobs are stopped for this stack.$ cu2d-update-stack -U <path_to_stack_file>

This will reset/reconfigure as well all branches, add -S if you don't have access to them (but ensure someone will configure them still at least once)

#### Different level of failures acceptance.

We do accept some integration tests failing (no unit tests should fail though as ran as part of package building). The different level of those triggers are defined in a file like jenkins/etc/autopilot.rc. Those are different level of failures, regressions, skipped and removed tests per stack. The real file depends on stackname and are found in \$jenkins/cu2d/<stack_name>.autopilotrc as described in the -check template.

#### Monitoring upstream uploads and merges

Remember to regularly look at the -changes mailing list to pick if a manual upload has been done and port the changes back (if not proposed by the ubuntu developer making the upload) to the upstream branch. The component which had a manual upload is ignored for daily release until this backport is done, meaning that no new commits will be published until then (you can see the prepare job for that component will turn unstable).

Also, ensure as you are watching all upstream merges to follow our acceptance criterias that the "latestsnapshot" branches generated automatically by the daily release process has been merged successfully by the upstream merger.

#### Boostrapping a new component for daily release

A lot of information is available on this wiki page.

## Conclusion

Phew! This was a long serie of blog post. I hoped that you appreciated to have a deeper look of what daily release means, how this work, and also that the remaining questions have been answered by the above FAQ. I've just ported all those information on the https://wiki.ubuntu.com/DailyRelease.

Of course, this is not the end of the road, I have my personal Santa Claus wishlist like UTAH being more stable (this is under work), the daily image provisionning with success more often than now, refactorisation work taking integration tests into account, less Xorg crashes when completing those tests… This is currently adding a little bit of overhead as of course, any of this will reject the whole test suite and thus the whole stack. We prefer to always be on the safe side and not accepting something in an unknown state!

Also, on the upstream stack and daily release process as well, there are rooms for improvments, like having less upstream integration tests failing (despite awesome progress already there), I wish I can soon turn that down to 3% of accepted failure at most for unity instead of the current 7% one. Dedicated integrations tests for indicators would be awesome as well (stealing the unity ones for now and only running those involving indicators, but of course, not everything is covered). Having also some oif, webapps and wecreds integration tests is one of my main priority, and we are working with the various upstream developers to get this done ASAP so that we have a better safety net than today. On the daily release process itself, maybe freezing all trunks in a go would be a nice improvement, more tests for the system itself as well is under work and my personal goal to reach 100% test coverage for the scripts before making any additional changes. Finally, some thoughts for later but a dashboard would enable to have a higher view than just using jenkins directly for reporting, to get a quicker view on all stack statuses.

Of course, I'm sure that we'll experience some bumpy roads with this daily release process through the cycle, but I'm confident we'll overcome them and I am really enthousiast about our first releases and how smoothly they went until now (we didn't break the distro… yet :p[2]. I'm sure this is a huge change for everyone and I want again give a big thank you to everyone involved into that effort, from the distro people to upstream and diverse contributors on the QA team. Changing to such a model isn't easy, it's not only having yet another process, but it's really a change of mindset in the way we deliver new features and components to Ubuntu. This might be frightening, but I guess it's the only way we have to ensure a systematic delivery, continuous and increasing quality while getting well a really short feedback loop to be able to react promptly to the changes we are introducing to our audience. I'm confident we are already collecting all the fruits of this effort and going to expand this more and more in the future.

For all those reasons, thanks again to all parties involved and let's have our daily shot of Unity!

I couldn’t finish without a special thanks to Alan Pope for being a patient first reader, and fixing tediously various typos that I introduced into the first draft of those blog posts. ;)