Month: December 2011

Recently I’ve encountered a problem when deploying a web application using our CI environment.

The probl… challange
Sometimes the deployed website was not the latest version (with latest changes).

I hope this blog post can save you some time, while figuring out what went wrong.

A little background
In this case, our CI environment is Jenkins, running on a Windows server. The artifact produced (a WAR) is uploaded to a Linux server where a deploy script is being ran, taking care of shutting down tomcat’s, cleaning work directories, and eventually copy the new WAR, then start everything again.

First checks
First thing I have checked: Do my dependencies compile at the right order? Are they being deployed into our own artifactory? Is the task, responsible for creating the WAR, using the latest dependencies?

All these things where setup ok.

So why does it go wrong?

Deployment strategy
In our case, we use the SVN revision nr as a version for our webapp. It is used by the deployment script, and by the uploading scripts as well. So this means, that when we are deploying revision 1000 of the website, a directory is created like : revision_1000 and the WAR file is uploaded into that directory. After that the deploy script is ran with the revision parameter.

This works well. It is used for several clients, and it has proved its worth thousand times. So this can’t be the problem right?

Branching strategy
There are a few branching strategies you can use within your projects. In this case, there is for each artifact an own trunk/branches/tags. This means that we can develop each component independently from each other. We can also branch versions of components. The website decides which versions of the components it uses.

This has several advantages. This particular website runs against a webservice that could change quite rapidly. We could be developing against version 1.0. Then later it would be 2.0, but when we had to go live we had to switch back to 1.0. Because of the branching strategy we have different branches for each webservice version, this mainly contains generated code from the WSDL’s and a thin layer. The website could easily switch between versions. Since we’re using Maven it was a matter of changing a version number of the dependency in the POM.

So everything went fine, but it still did not work? How is that possible?
Well.. thats where the real digging begins. It seemed that Jenkins reported that the task has completed with succes. However, when looking at the log, the script seemed to have failed. It failed because it could not create the revision folder to upload the WAR file to.

But how is that possible?

In this case, it is because of the branching policy.

The effect of the used branching policy
Given each module a seperate trunk/branches/tags. When a change is made at component A, then it will not automatically raise the revision nr in component B. Only when a change is made at component A again, the revision will be raised there.

Meaning. When website (component A), is at revision 1000. And it uses a dependency (Component B), at revision 999. When deploying the website, a revision_1000 directory is created.

The culprit
When a bug is found in Component B, and it is fixed. This will make Component B revision 1001. However, the website remains revision 1000. When the deploy task is checking out the website repository, it will find that the current revision is 1000 for that trunk. And, it will work from there. The deploy script wants to create a directory revision_1000 again, which will fail and will abort the script. It does use the latest dependency though, but since the creation of the deploy directory failed, it never uploaded the latest WAR (it did not overwrite this).

“The” fix
I see two ways to get out of this situation. Fix the deploy script, or change the branching policy.

Fixing the script…?
Some may argue that it is correct, that the revision remains unchanged for the website, and that the deploy script should know how to deal with this. One way would be to not only use the revision number as deploy directory, but also a timestamp. Ie, revision_1000_DD-MM-YYYY-HH-MM. For instance: revision_1000-12-1-2011-21.

Change the branching policy
Another way of fixing this issue, is to make sure that every time a change is made the revision number is being raised. The only way to be able to do this, is to let every component be beneath a branch or trunk, instead of having their own branches.

The components that are bound to the 3rd party webservices remain their branching policy. The downside would be that any fixes in these components still have this issue, but this time deliberately.