bookmark_borderChallenges when deploying your webapp, with multiple dependencies using Continous Integration. (Jenkins)

Recently I’ve encountered a problem when deploying a web application using our CI environment.

The probl… challange
Sometimes the deployed website was not the latest version (with latest changes).

I hope this blog post can save you some time, while figuring out what went wrong.

A little background
In this case, our CI environment is Jenkins, running on a Windows server. The artifact produced (a WAR) is uploaded to a Linux server where a deploy script is being ran, taking care of shutting down tomcat’s, cleaning work directories, and eventually copy the new WAR, then start everything again.

First checks
First thing I have checked: Do my dependencies compile at the right order? Are they being deployed into our own artifactory? Is the task, responsible for creating the WAR, using the latest dependencies?

All these things where setup ok.

So why does it go wrong?

Deployment strategy
In our case, we use the SVN revision nr as a version for our webapp. It is used by the deployment script, and by the uploading scripts as well. So this means, that when we are deploying revision 1000 of the website, a directory is created like : revision_1000 and the WAR file is uploaded into that directory. After that the deploy script is ran with the revision parameter.

This works well. It is used for several clients, and it has proved its worth thousand times. So this can’t be the problem right?

Branching strategy
There are a few branching strategies you can use within your projects. In this case, there is for each artifact an own trunk/branches/tags. This means that we can develop each component independently from each other. We can also branch versions of components. The website decides which versions of the components it uses.

This has several advantages. This particular website runs against a webservice that could change quite rapidly. We could be developing against version 1.0. Then later it would be 2.0, but when we had to go live we had to switch back to 1.0. Because of the branching strategy we have different branches for each webservice version, this mainly contains generated code from the WSDL’s and a thin layer. The website could easily switch between versions. Since we’re using Maven it was a matter of changing a version number of the dependency in the POM.

Example

So everything went fine, but it still did not work? How is that possible?
Well.. thats where the real digging begins. It seemed that Jenkins reported that the task has completed with succes. However, when looking at the log, the script seemed to have failed. It failed because it could not create the revision folder to upload the WAR file to.

But how is that possible?

In this case, it is because of the branching policy.

The effect of the used branching policy
Given each module a seperate trunk/branches/tags. When a change is made at component A, then it will not automatically raise the revision nr in component B. Only when a change is made at component A again, the revision will be raised there.

Meaning. When website (component A), is at revision 1000. And it uses a dependency (Component B), at revision 999. When deploying the website, a revision_1000 directory is created.

The culprit
When a bug is found in Component B, and it is fixed. This will make Component B revision 1001. However, the website remains revision 1000. When the deploy task is checking out the website repository, it will find that the current revision is 1000 for that trunk. And, it will work from there. The deploy script wants to create a directory revision_1000 again, which will fail and will abort the script. It does use the latest dependency though, but since the creation of the deploy directory failed, it never uploaded the latest WAR (it did not overwrite this).

“The” fix
I see two ways to get out of this situation. Fix the deploy script, or change the branching policy.

Fixing the script…?
Some may argue that it is correct, that the revision remains unchanged for the website, and that the deploy script should know how to deal with this. One way would be to not only use the revision number as deploy directory, but also a timestamp. Ie, revision_1000_DD-MM-YYYY-HH-MM. For instance: revision_1000-12-1-2011-21.

Change the branching policy
Another way of fixing this issue, is to make sure that every time a change is made the revision number is being raised. The only way to be able to do this, is to let every component be beneath a branch or trunk, instead of having their own branches.

The components that are bound to the 3rd party webservices remain their branching policy. The downside would be that any fixes in these components still have this issue, but this time deliberately.

bookmark_borderPrevent cross-site scripting when using JSON objects using ESAPI and Jackson framework 1.7.x

Recently I have had the opportunity to fix a cross-site-scripting problem.

The problem: a lot of JSON objects are being sent over the wire and the data is not being html escaped. This means that anyone who would put html data IN would get it out and make any user vulnerable for XSS attacks.

In this case, JSON objects are being created by using the MappingJacksonHttpMessageConverter. This is deliverd by the Spring framework. Normally it is instantiated when you use spring-mvc (using the mvc-annotation tag). This allowed us to just return an object and the MappingJacksonHttpMessageConverter would take care of translating it into a JSON object.

In order to influence the creation of the JSON object and make it encode HTML for String values we need to do a few things:

  1. Create a Custom Object Mapper and wire into Spring
  2. Create a JsonSerializer that encodes HTML (using ESAPI)
  3. Register the JsonSerializer in the CustomObjectMapper

Create a Custom Object Mapper and wire into Spring

However, now we want to change the value serialized by the MappingJacksonHttpMessageConverter. So how do we do that? The MappingJacksonHttpMessageConverter uses an ObjectMapper. So in order to get the ObjectMapper map the values to an encoded one we need to create a custom Object mapper.

public class CustomObjectMapper extends org.codehaus.jackson.map.ObjectMapper {
}

At first we leave it empty. We want to make Spring make use of our CustomObjectMapper first. In order to do that you need to write out the mvc-annotation tag, because we need to inject our CustomObjectMapper in the the MappingJacksonHttpMessageConverter. This is a bit difficult to figure out at first, but for your convinience I have proveded the configuration you need.

<bean id="CustomObjectMapper" class="your custom object mapper"/>

<bean id="MappingJacksonHttpMessageConverter" class="org.springframework.http.converter.json.MappingJacksonHttpMessageConverter">
	<property name="objectMapper" ref="CustomObjectMapper"/>
</bean>

<bean class="org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter">
	<property name="order" value="1" />
	<property name="customArgumentResolver" ref="sessionParamResolver"/>
	<property name="webBindingInitializer">
	<bean class="org.springframework.web.bind.support.ConfigurableWebBindingInitializer">
		<!-- <property name="conversionService" ref="conversionService" />  -->
		<property name="validator" ref="validator" />
	</bean>
	</property>
	<property name="messageConverters">
		<list>
			<bean class="org.springframework.http.converter.ByteArrayHttpMessageConverter" />
			<bean class="org.springframework.http.converter.StringHttpMessageConverter" />
			<bean class="org.springframework.http.converter.ResourceHttpMessageConverter" />
			<bean class="org.springframework.http.converter.FormHttpMessageConverter" />
			<ref bean="MappingJacksonHttpMessageConverter"/>
		</list>
	</property>
</bean>

As you can see, the AnnotationMethodHandlerAdapter is the bean where everything is getting wired into. The messageConverters property contains a list of converters. The last entry is where the MappingJacksonHttpMessageConverter is referenced. Normally it is defined like the beans above it. But since we want to inject our own CustomObjectMapper it is done this way.

Create a JsonSerializer that encodes HTML (using ESAPI)

Now we know our CustomObjectMapper is used, we can start the part where we want to influence the way the jackson framework serializes our objects to JSON. In order to do that we create our own JsonSerializer. Then, we override the serialize method and make sure the provided jsonGenerator is writing our encoded value.

Code:

public class JsonHtmlXssSerializer extends JsonSerializer {

   public void serialize(String value, JsonGenerator jsonGenerator, SerializerProvider serializerProvider) throws IOException, JsonProcessingException {
      if (value != null) {
         String encodedValue = encodeHtml(value);
         jsonGenerator.writeString(encodedValue);
      }
   }

}

So this class extends from org.codehaus.jackson.map.JsonSerializer, and implements its own serialize method. The method encodeHtml will use the ESAPI encoding functions to escape HTML.

 

Implementing this is very easy:

   protected String encodeHtml(String html) {
      DefaultEncoder encoder = new DefaultEncoder();
      return encoder.encodeForHTML(html);
   }

The defaultEncoder is from ESAPI allowing us to encode the HTML.

 

Register the JsonSerializer in the CustomObjectMapper

The final step is to register the JsonSerializer within the ObjectMapper used by Spring. Doing that using Jackson 1.7.0 is very easy. Using the provided SimpleModule class we can easily add our own Serializer to the ObjectMapper. We do that by creating our own default constructor, which will register our serializer. Like so:

@Component
public class CustomObjectMapper extends org.codehaus.jackson.map.ObjectMapper {

   public CustomObjectMapper() {
      SimpleModule module = new SimpleModule("HTML XSS Serializer", new Version(1, 0, 0, "FINAL"));
      module.addSerializer(new JsonHtmlXssSerializer());
      this.registerModule(module);
   }
}

bookmark_borderThe need for quality of Web Applications

I’ve been working on my thesis (about meassuring Software Quality) and its reaching its final state. I will blog some posts about my thoughts and findings about Software Quality from here. This post was a bit old, I dusted it off and actualised it a bit. More to come!

Here it comes:

Today, it is unthinkable to not use an internet related service in your daily life. Think about staying in touch with family and friends using email (gmail, yahoo, hotmail, etc), social networking (twitter, facebook, hyves, etc) or more.

If you want to book a trip, you arrange this using the internet. When you want to buy a new laptop, you find the specs and reviews on the internet. You use a comparison site to find the cheapest (web)store. And if you want to buy it, you use the internet.

It is fair to say that the internet has become more and more important to our lives. Even more than we imagined 5 years ago. How would we deal when our mail provider is down? How would we react when facebook is down? What about twitter? How do we do our bank business when we cannot reach the online-banking site?

I think you got the point.

Companies developing and maintaining (ecommerce) websites or services are constantly pressured to deliver high quality software in the shortest possible amount of time. Having these factors, time and quality; one cannot exclude the other. They do influence each other though. When you want to deliver fast, you ought to make sacrifices in terms of designing and testing the product. This will eventually lead to a lower quality. The opposite is true as well, if you spend more time in the design and testing, you will likely deliver higher quality. Also, unlike with traditional software, it is not always the first to deliver that is the best.

In the land of the internet, it is not only the case to get users to your website, but also to keep them revisiting. If you deliver a fancy new feature, you might get the attention. But if you did not put enough effort to make it high quality, you might lose them to your competitor who delivers a similiar feature a few days later, but twice as good. Remember, your visitors can easily switch if you cannot deliver what they expect.

So what does a visitor expect from a website? Looking around on the net you can find some definitions of quality of good websites. That is the biggest question of all. How can you deliver fast, and yet have some sort of quality to rely on?

You could use a Software Quality Model for that, like the ISO-9126. I would say, you’d use a model like the ISO-9126 and create/use a Web Application Quality Standard.

So what is this standard? I haven’t found a definite model, although I have found a good resource here.
In my opinion, the following Software Quality Attributes would belong in such a model:
performance. Your visitors will leave if loading a page takes longer than 2 seconds. Directly influencing possible sales!
security. Atleast be aware of the top ten of OWASP. Better yet, make sure you are not vulnerable to these exploits.
maintainability. Developers should be able to easily extend code, or fix bugs without causing too much regression.
reliability. Should your website simply shut down once the load is too high? How much ‘page faults’ may appear? One of thousand requests? One of hundred?.

Each of these attributes can be split into several measurable units. For example, performance wise you could measure the amount of time it takes to get a page loaded in the browser. Maintainability has some known measures (like McCabe’s Cyclomatic Complexity).

Note: Recently I figured that the ISO-9126 is actually not a model, but a ‘standard’. It really does not tell you *how* to do things, but it does give guidelines. The Software Quality model I’ve made does tellĀ  you how to do things.

bookmark_borderDon’t forget your resources directory! (Maven/Eclipse/Sysdeo-tomcat)

Recently I had to set up a XMLRPC Server. Using Apache’s XMLRPC it should not be that hard to set up.

It wasn’t. But it did not work the first time.

The XMLRPC Serlvet would not be initialized properly because it could not find the XmlRpcServlet.properties file. I was a bit suprised, because I had it in my src/main/org/apache/xmlrpc/webserver path… so why did it not work?

The reason is that whenever you run your Tomcat webserver, configured to look into your project directory (using the tomcat-sysdeo maven plugin) which is set up using maven; you’ll be using your classpath as set up there. If you look into your project->build path settings (tab source), you’ll see that everything under src/main is only including **/*.java, meaning it will not find the properties file at all!

Once I moved the XmlRpcServlet.properties to src/main/resources/org/apache/xmlrpc/webserver (where it belongs), it worked fine.

So remember, classpath, classpath, classpath!