Monday, October 15, 2007

Taking Continuous Integration to a Greater Height Using Hudson




At JBoss, the quality engineering team though relatively smaller in size, is involved with a variety of projects, the details of which can be found at JBoss labs page. Since all of these are open source projects, we follow the well known mantra "Release early. Release often" which often results into several branches of code base. Add to these the fact that some of these projects continuously need to be run against different versions of JDK, databases , LDAP servers, OS and application servers, you have a complex continuous integration (CI) project at hand.

We have been using CruiseControl until recently which is a good CI tool but it lacks some of the features that we really wished for. We were primarily looking for a CI tool that would
  1. provide a web UI to configure and manage builds of different projects
  2. run multiple builds at the same time in a distributed environment
  3. integrate with usual suspects: CVS, SVN, ant, maven, junit, code coverage etc.
Having a web UI was to avoid a resource bottleneck. Once a project has been setup, developers could use it to manage their projects without submitting a request to QA. A win win situation for both parties. Running multiple builds at the same time was to avoid a linear increase in build time when projects got added to the matrix. Since most of projects require running an instance of JBoss Application Server, having distributed build would allow us to run many builds at the same time without worrying about resource (port, file) conflict etc. Most of other open source CI tools namely CruiseControl and Continuum met our 3rd requirement but only Hudson had a very intuitive web UI and capability to manage builds in a distributed environment through its master/slave feature. A detailed information about Hudson can be found at it's homepage.

We did some proof of concept projects in Hudson some six months back when it was still version number 1.20. After several bug fixes, thanks to a very vibrant community, specially its lead developer Kohsuke, it's currently at version 1.146 which is much more mature and stable with several newer plugins.

We have migrated almost all projects from CruiseControl to Hudson which can be seen in action here. This instance of hudson is our public Hudson instance which acts as sort of reporting tool which also is a testimony to a flexible architecture of Hudson. To make a long story short, we needed a way to utilize servers which are inside VPN but make results available to community on a server outside VPN. We developed couple of in-house plugins that allowed us to publish build results from one Hudson instance to another. As I write, these plugins have been given back to Hudson community. May be it's a bit of exaggeration, but it's a sight to behold when Hudson manages 15 slaves of varied architecture to manage multiple builds simultaneously.

Here is my wish list for Hudson and I am sure it will not be too long before I have my wishes granted as well:
  1. Per project customization for email notification
  2. A shell script box to do some custom post build actions
  3. Configuration for post build action so that it can be configured when to run and otherwise.
  4. Has an option to provide a priority for builds. It theoretically could lead to starvation for some projects but so be it. :-)

5 comments:

Jeff said...

Great post, Prabhat. We are also starting down the path of using hudson to multi-environment, distributed builds. Hudson seems like the perfect choice.

Do you have any job dependencies defined within hudson? For example, do you have a unit test job that when complete will kick off integration or tck tests on a hudson slave? If so how do you ensure the artifacts being tested in the integration job are correct and consistant? For example, do you publish snapshot version binaries to a central maven repo so they availble to the downstream builds?

I'm curious about your insights here.

Prabhat Jha said...

Thanks for your comment Jeff.

Yes, we once had job dependencies defined within hudson. We currently do for hibernate where completion of build triggers downstream projects that runs hibernate against different database. The downstream projects are configured in such a way that they get binaries from parent project.

Regarding snapshot and maven, we also have couple of projects configured to publish snapshot to jboss maven repo. But for this, project's pom was configured to do that. All that needed to be done on hudson side was update .m2/settings.xml with URL, uid and password info.

Larry S. said...

Prabhat, looks like you have been using Hudson for some time. Can you comment, if it has been better than Maven, or will you use it in conjugation with Maven? Thanks, Larry

Unknown said...

Hi Prabhat,
I am new to Hudson, and i have TortoiseHg as the repository. How to use Hudson for integration, since TortoiseHg is a distributive repository. How can i setup hudson for this and then be able to merge dat from other repositories and push a build out.

I am really a new bee at understanding and using Hudson. Any help would be highly appreciative!!!

Thanks and Regards,
Jigar Mehta

rajesh said...

Thanks for the post.
I am new to hudson and i just want to develop a project which will work in different environments as we are having two environments like DEV and QA.How can i integrate my project so that i can build on the Environment i want.