This post is part of the Unity daily release process blog post suite.
You hopefully recovered from your migraine on reading yesterday's blog post on the insight of daily release and are hungry for more. What's? That's not it? Well, mostly, but we purposely dismissed one of the biggest point and consequences on having stacks: they depends on each other!
Illustrating the problem
Let's say that Mr T. has an awesome patch for the indicator stack, but this one needs as well some changes in Unity and is not retro-compatible. Ok, indicators integration tests should fail, isn't it? as long as we don't have the latest version of Unity rebuilt in the ppa… But then, the indicator stack is not published (because of integration tests failing), but Unity, with its integration tests will pass as they are run against the latest indicator stack built in the same ppa. Consequently, we may end up publishing only half of what's needed (the Unity stack, without the indicator requirement).
Another possible case: let's imagine that yesterday's daily-build failed and so we have a new unity staging in the ppa (containing the indicator-compatible code), so the next day, the indicator integration tests will pass (as it's installing Unity from the ppa), but will only publish the indicator components, not the Unity ones. Bad bad bad :)
We clearly see we need to declare dependencies between stacks and handle them smartly.
What's the stack dependency as of today?
Well, a diagram is worth 1 000 words:
We can clearly see here that stacks are depending on each other, this is specified in the stack configuration file, each stack telling on what other they depend on.
This has multiple consequences.
Waiting period
First, a stack will start building only once all each stack it depends on finished to build. This enables us to ensure we are in a predictable state and not in a in-between with half of the components building against the old upstream stack and the other half against the new upstream stack because it finished to build in the same ppa.
The previous diagram needs to be then modified for the unity stack with this "wait" optional new step. This step is only deployed with cu2d-update-stack if we declared some dependent stacks.
The wait script is really simple to achieve that task. However, waiting is not enough as we've seen above, we need to take the test result into consideration, which we'll do later on during the publisher step.
Running integration tests in isolation
Ok, that's answering some part of the consistency issue, however, imagine now the second case we mentionned where yesterday's daily build failed, it means that if we take the whole ppa content, we'll end up testing the new indicator integration with a version of Unity which isn't the distro one, nor the one we'll publish later on. We need to ensure we only tests what will make sense.
The integration jobs are then not taking the whole ppa content, but only the list of components generated by the current stack. It means that we are explicitely only installing components from this stack and nothing else. Then, we log the result of the install command and filters against it. If a soname change for instance (like libbamf going from 3.0 to 3.1), ending up in a package renaming will grab a new Unity to be able to run, we'll then bail out.
Here is a real example where we can find here the list of packages we are expecting to install, then here is the list of what happened when we asked to install only those packages. The filtering happening as the first test will then exit the job in failure, meaning we can't publish that stack.
So we know if the we are ensuring we know the retro-compatibility status this way. If retro-compatibility is achieved, the tests are passing right away, and so the stack is published regardless other stacks, as we tested against Unity distro's version. This ensure we are not regressing anything then. In other case when retro-compatibility seems to fails, as in the previous example, this means that we would eventually have to publish both stacks at the same time, which is what we can do.
Ensuring that the leaf stack is not published
In a case like the previous one, the Unity integration tests will run against the latest indicator stack, and then they pass (having latest everything), but wait! We shouldn't publish without the indicator components (and those failed, we are not even sure the fixes in Unity will make them pass and we haven't tested againt the whole PPA).
Here comes the second "manual publishing" case from the publisher script I teased about yesterday. If a stack we depend on didn't finish with success, the current stack is put in manual publication. The xml artefacts giving the reason is quite clear:
indicators-head failed to build. Possible causes are:
* the stack really didn't build/can be prepared at all.
* the stack have integration tests not working with this previous stack.
What's need to be done:
* The integration tests for indicators-head may be rerolled with current dependent stack. If they works, both stacks should be published at the same time.
* If we only want to publish this stack, ensure as the integration tests were maybe run from a build against indicators-head, that we can publish the current stack only safely.
So from there, we can be either sure that the upstream stack has no relationship with the current downstream one, and so, people with credentials can force a manual publishing with cu2d-run as indicated in the previous blog post. However, if there is a transition impacting both stacks, the cleverest way is to relaunch the upstream stack tests with the whole ppa to validate that hypothesis.
For that, cu2d-run has another mode, entitled "check with whole ppa". This option will rerun a daily-release, but won't prepare/rebuild any component. It will though still takes their build status into account for the global status. The only difference is that the integration tests will take into account this time the whole ppa (and so no more filtering on what we install if it's part of the current stack or not), doing a classic dist-upgrade. This way, we validate with "all latest" that the tests are passing.
Note that even if tests pass, as we'll probably need to publish at the same time than the dependent stack, we'll automatically pass in manual publishing mode. Then, once all resolved, we can publish all stacks we want or need with cu2d-run.
Manual publish cascades to manual publish
Another case when a dependent stack can fallback to manual publishing mode doesn't necessarily imply tests failing. Remember the other case when we fallback in manual publishing mode? Exactly, packaging changes! As the dependent stack is built against the latest upstream stack, and as it's what we tested, we probably need to ensure both are published (or force a manual publishing on only one if we are confident that there is no interleaved depends in the current change).
Here is what the artefacts will provide as infos on that case:
indicators-head is in manually publish mode. Possible causes are:
* Some part of the stack has packaging changes
* This stack is depending on another stack not being published
What's need to be done:
* The other stack can be published and we want to publish both stacks at the same time.
* If we only want to publish this stack, ensure as the integration tests were run from a build against indicators-head, that we can publish the current stack only safely.
So, here, same solutions than in the previous case, with cu2d-run, we either publish both stack at the same time (because the packaging changes are safe and everything is according to the rules) or just the leaf one if we are confident there is no impact.
Note that we do handle as well if something unexpected happened on a stack and that we can't know its status.
In a nutshell
Stacks don't live in their own silos, they are high dependency behavior between them which are ensured by the fact that they live and are built against each other in the same ppa. We as well ensure the retro-compatibility mode, which is what most of daily release will be, than only publishing some stacks. We also handle that way transitions impacting multiple stacks and can publish everything in one shot.
That's it for the core of the daily release process (phew, isn't it?). Of course this is largely simplified and we have a lot of corner cases I didn't discuss about, but this should give enough high coverage on how the general flow is working out. I'll wrap it up in a week with some kind of FAQ from the most common questions that are still maybe pending (do not hesitate to comment here) and give this time to collect feedbacks. I will publish this FAQ then and at the same time will draw a conclusion from there. See you in a week. ;)