The LSB uses buildbot for software builds and certain tests. In the past, we also built certain Moblin projects using it.
Buildbot works in a "master/slave" setup, where a master node directs various slave nodes to run builds as appropriate. All of the interesting configuration happens on the master; the slaves are intentionally kept as dumb as possible, and can be blown away and re-created at any time.
Builds are done via configuration entities called "builders"; each builder controls the build of one project on one architecture. We name these using the "project-arch" convention; for example, building the SDK for x86_64 is done with the builder "build-sdk-x86_64". Each builder in buildbot is an independent entity, so to build a project for all architectures, you have to trigger 7 different builders (as of this writing), one per architecture.
In our setup, we receive notifications from Bazaar when changes are checked in, which trigger builds after an appropriate delay to gather up multiple commits in a short period of time.
Working with Buildbot
Buildbot typically works behind the scenes, and doesn't normally need direct interaction. But we do have several ways to control it.
The results of buildbot builds are placed on our FTP server once they are available. Most builds end up in the snapshots area on a per-project basis. Production-level builds are instead put in our staging area, separated out by branch, build type, and project.
We expose a Web front-end to Buildbot here. From that page, you can get the most recent status.
Several pages let you control buildbot by triggering builds, refreshing slave status, and even shutting down the master. These actions typically require a username and password. Right now, there is a single user set up; if the need arises, we may move to a setup where multiple users can be given access.
Buildbot also supports an IRC bot, which is configured to lurk in the LSB channel (currently irc.freenode.net, #lsb). Besides reporting on the status of builds as that status changes, the bot can control buildbot with a few commands:
force build (builder) -- cause buildbot to start a build
status (builder) -- get the status of the last build completed for that builder
Command Line and Job Queue
There is a method of submitting jobs for buildbot via the command line, which uses a spool directory to create job description files that buildbot can parse.
The command-line utility is called "start_lsb_build". It takes the branch as its first argument ("devel" vs. "4.1", etc.), and the projects to build as the following arguments after that. It does the work of triggering multiple builders itself, so asking for "lsblibchk" to be built will trigger the builders for the default set of architectures. That default includes all architectures except s390 and s390x. There are options for overriding the architecture list, as well as the build type; see the help ("start_lsb_build --help") for details.
Buildbot is controlled from two Bazaar repositories. The repository "buildbot-config" contains the configuration and scripts for the master node, and "puppet-lsb" contains the magic to deploy buildbot, both for the master and slave nodes.