Building URU under Continuous Integration
Posted: Sat Jan 17, 2009 10:35 pm
This topic will get a little technical...
I've been working on and off for a couple of weeks to put together an environment for building URU code in a Continuous Integration process. While we don't have any URU code yet, I have been working to put together a demonstration setup with some other URU related open source software to show how it would work.
Goals: Continuous integration is a paradigm for revealing problems with code that can be discovered as soon as possible after a change has been made to the code. Assuming the code is under some for of Version Control System such as Subversion, a CI environment would monitor the code base for changes. As soon as changes are put into the code repository, the CI system would notice the change and initiate a series of tasks to build and test the code. Any errors that were discovered during either the build or testing tasks would be noted in a historical record (and if desired, notification can be performed via email, IM, etc). If the build was successful, the results of the build are then made available for a larger audience to test or use.
How it is done. Usually a CI process is governed by an engine, which is a program designed to monitor one or more source repositories, obtain code from the repositories, and initiate the tasks needed to build and test. In my environment I've chosen Hudson as the CI engine driving the URU build and test processes. Hudson sits in the background periodically checking the status of a source repository (it is compatible with a large number of VCS systems). When it notices changes it will check out the changes to a local copy of the repository. It then schedules tasks to build that code on the appropriate types of machines and operating systems (using a slave process on a machine running that environment to do the actual work). After the slave(s) complete Hudson collates the results and prepares a web page documenting the tasks performed and where the results are.
The environment I've set up makes liberal use of Virtual Machines so it all can be run on one physical server. I'll put together a drawing later giving a pretty picture of the current setup, but we start with a server-class machine running a flavor of Unix called Solaris. Within this machine an instance of Hudson is running. It has a set of projects configured that is instructed to monitor, build and test. Each project has one or more repositories associated where the code is running. A local copy of the code is then kept on this server.
In addition to Hudson, on this machine there are two Virtual Machines running VirtualBox. Each of these "virtual" machines has it's own copy of an operating system running, along with a set of build and test tools. (In my environment these virtual machines run Windows and Ubuntu Linux.) Also on each virtual machine is a slave hudson process which is listening for instructions from the master Hudson on the enclosing Unix environment.
In the demonstration setup, I've used the new libPlasma library that Zrax has made available over at GoW. libPlasma (and it's associated PlasmaShop tool) are C++ programs that Zrax has stored in a subversion repository. I've configured a project which knows where the libPlasma repository is, and what steps are needed to build a copy. At this point there is no test code for checking libPlasma so there are no test tasks involved in a build.
The Hudson engine polls Zrax's repository about once an hour. Any time it discovers a change to the master repository, it will pull the changes into the local Hudson repository. It then will create two seperate build areas with the code from the local repository, and tell the slave hudson processes on the two virtual machines to start building libPlasma. After the build finishes, the state (success or failure) of the build is recorded in a database and the results are available for viewing at a web page. If so instructed the binary programs produced by the build can be made accessible through the web interface.
Current status: The fundamental infrastructure is up and running. You can view the Hudson status webpage here (either browse anonymously or log in with user guest password guest). Current builds of libPlasma are failing due to my still discovering tool settings and prerequisites needed for building libPlasma. The same will have to be done once we get URU code. But you should be able to see progress as I fix the remaining issues so that every time a change is made to the libPlasma code, a new version automatically will be built.
I'll be happy to entertain requests to build other URU related code in this foundry environment I've put together. Just send me a PM.
I'll post more as I finish getting the demo running reliably. Then we'll just have to wait from word from Cyan on what is needed to build the URU code.
I've been working on and off for a couple of weeks to put together an environment for building URU code in a Continuous Integration process. While we don't have any URU code yet, I have been working to put together a demonstration setup with some other URU related open source software to show how it would work.
Goals: Continuous integration is a paradigm for revealing problems with code that can be discovered as soon as possible after a change has been made to the code. Assuming the code is under some for of Version Control System such as Subversion, a CI environment would monitor the code base for changes. As soon as changes are put into the code repository, the CI system would notice the change and initiate a series of tasks to build and test the code. Any errors that were discovered during either the build or testing tasks would be noted in a historical record (and if desired, notification can be performed via email, IM, etc). If the build was successful, the results of the build are then made available for a larger audience to test or use.
How it is done. Usually a CI process is governed by an engine, which is a program designed to monitor one or more source repositories, obtain code from the repositories, and initiate the tasks needed to build and test. In my environment I've chosen Hudson as the CI engine driving the URU build and test processes. Hudson sits in the background periodically checking the status of a source repository (it is compatible with a large number of VCS systems). When it notices changes it will check out the changes to a local copy of the repository. It then schedules tasks to build that code on the appropriate types of machines and operating systems (using a slave process on a machine running that environment to do the actual work). After the slave(s) complete Hudson collates the results and prepares a web page documenting the tasks performed and where the results are.
The environment I've set up makes liberal use of Virtual Machines so it all can be run on one physical server. I'll put together a drawing later giving a pretty picture of the current setup, but we start with a server-class machine running a flavor of Unix called Solaris. Within this machine an instance of Hudson is running. It has a set of projects configured that is instructed to monitor, build and test. Each project has one or more repositories associated where the code is running. A local copy of the code is then kept on this server.
In addition to Hudson, on this machine there are two Virtual Machines running VirtualBox. Each of these "virtual" machines has it's own copy of an operating system running, along with a set of build and test tools. (In my environment these virtual machines run Windows and Ubuntu Linux.) Also on each virtual machine is a slave hudson process which is listening for instructions from the master Hudson on the enclosing Unix environment.
In the demonstration setup, I've used the new libPlasma library that Zrax has made available over at GoW. libPlasma (and it's associated PlasmaShop tool) are C++ programs that Zrax has stored in a subversion repository. I've configured a project which knows where the libPlasma repository is, and what steps are needed to build a copy. At this point there is no test code for checking libPlasma so there are no test tasks involved in a build.
The Hudson engine polls Zrax's repository about once an hour. Any time it discovers a change to the master repository, it will pull the changes into the local Hudson repository. It then will create two seperate build areas with the code from the local repository, and tell the slave hudson processes on the two virtual machines to start building libPlasma. After the build finishes, the state (success or failure) of the build is recorded in a database and the results are available for viewing at a web page. If so instructed the binary programs produced by the build can be made accessible through the web interface.
Current status: The fundamental infrastructure is up and running. You can view the Hudson status webpage here (either browse anonymously or log in with user guest password guest). Current builds of libPlasma are failing due to my still discovering tool settings and prerequisites needed for building libPlasma. The same will have to be done once we get URU code. But you should be able to see progress as I fix the remaining issues so that every time a change is made to the libPlasma code, a new version automatically will be built.
I'll be happy to entertain requests to build other URU related code in this foundry environment I've put together. Just send me a PM.
I'll post more as I finish getting the demo running reliably. Then we'll just have to wait from word from Cyan on what is needed to build the URU code.