Writing a Build Bot using SVN and Python

Mar

Writing a Build Bot using SVN and Python

Filed in Code, How-To's, Python |

Today I went head and finalized our build bot some of our public scripts. These scripts are mostly JavaScript based. I already had the build script zipping the file for tags and trunks of projects within the repository (we store all projects in one repository, instead of one repo per project). There were still a couple things on my TODO though. So let’s first list off what we needed out of the build bot.

JS and CSS compressed builds.
Replacing certain keyword variables within files.
GZipping or Zipping the final results.
Automatic building on SVN commit hooks.

The application turned out to be pretty straight forward, although a lot of problems came up with various code bits, and using the subprocess methods. In reality it could be converted to completely use Python libraryies instead of subprocess calls, but I didn’t have time to find any useful docs for the SVN Python Bindings or to read up more on the zipfile module. The project itself consists of four files and the YUI compressor.

Handling JS/CSS Compression

For JS and CSS compression I decided to go with YUI’s compressor. It by far has the most hype around it, and seemed like a great choice especially since it supported CSS. The alternative, for JavaScript, was JSMin. YUI uses Java, so we had to install that, but the rest was cake to setup.

COMPRESSOR_LINE = 'java -jar /usr/bin/yuicompressor.jar %s -o %s'
COMPRESSOR_FILE_TYPES = ('.js', '.css')
 
def compress_files(path, base_path):
    ext = os.path.splitext(path)[1]
    """Compresses JavaScript and CSS files."""
    if ext in COMPRESSOR_FILE_TYPES:
        orig = os.path.getsize(path)
        status = os.system(COMPRESSOR_LINE % (path, path))
        if status != 0:
            raise BuildError("Failed compression")
        after = os.path.getsize(path)
        return "Compressed %s to %.2f%% of normal" % (path.replace(base_path, ''), after/float(orig)*100,)

Replacing Variables

The next thing on our TODO list was to add replacement variables. SVN supports svn:keywords but they’re extremely limited. To do this we first had to run an svn info call, and then grep out the revision number. This could become more complex for your own purposes, but for us all we wanted was the version name (trunk, or the tags name), and the revision number.

Let me also say, that the following code was designed for speed, and could be improved upon a LOT, but I do not know Python file manipulation very well (so feel free to help me out!).

KEYWORD_FILETYPES_EXCLUDE = ('.jpg', '.png', '.gif', '.jar')
 
def replace_keywords(path, keywords={}):
    ext = os.path.splitext(path)[1]
    if ext not in KEYWORD_FILETYPES_EXCLUDE:
        f = open(path)
        output = []
        try:
            for line in f:
                for k, v in keywords.iteritems():
                    line = line.replace(k, v)
                output.append(line)
        finally:
            f.close()
 
        f = file(path, 'w')
        f.writelines(output)
        f.close()

The Code

build.py

This is the core file which handles exporting from SVN, compression, variable replacement, and zipping the release.

import os
import os.path
import hashlib
import subprocess
import shutil
import zipfile
 
__all__ = ('build_tag', 'build_trunk', 'BASE_CHECKOUT', 'BuildError')
 
PATH_OVERRIDES = {
    'ibox_wordpress': 'ibox/wordpress',
    'share_wordpress': 'share/wordpress',
}
 
KEYWORD_FILETYPES_EXCLUDE = ('.jpg', '.png', '.gif', '.jar')
 
COMPRESSOR_LINE = 'java -jar /usr/bin/yuicompressor.jar %s -o %s'
COMPRESSOR_FILE_TYPES = ('.js', '.css')
 
BASE_CHECKOUT = 'https://svn.ibegin.com/ibegin/'
 
# No trailing slash
RELEASE_PATH = '/home/ibegin/releases'
TEMP_PATH = '/var/tmp'
 
class BuildError(Exception): pass
class NoCompressionError(BuildError): pass
 
def recurse_path(path, func, **kwargs):
    names = os.listdir(path)
    errors = []
    returnvals = []
    for name in names:
        pathname = os.path.join(path, name)
        try:
            if os.path.isdir(pathname):
                returnvals.extend(recurse_path(pathname, func, **kwargs))
            else:
                returnval = func(pathname, **kwargs)
                if returnval:
                    returnvals.append(returnval)
        except (IOError, os.error), why:
            errors.append((pathname, why))
    if errors:
        raise os.error, errors
    return returnvals
 
def compress_files(path, base_path):
    ext = os.path.splitext(path)[1]
    """Compresses JavaScript and CSS files."""
    if ext in COMPRESSOR_FILE_TYPES:
        orig = os.path.getsize(path)
        status = os.system(COMPRESSOR_LINE % (path, path))
        if status != 0:
            raise BuildError("Failed compression")
        after = os.path.getsize(path)
        return "Compressed %s to %.2f%% of normal" % (path.replace(base_path, ''), after/float(orig)*100,)
 
def replace_keywords(path, keywords={}):
    ext = os.path.splitext(path)[1]
    if ext not in KEYWORD_FILETYPES_EXCLUDE:
        f = open(path)
        output = []
        try:
            for line in f:
                for k, v in keywords.iteritems():
                    line = line.replace(k, v)
                output.append(line)
        finally:
            f.close()
 
        f = file(path, 'w')
        f.writelines(output)
        f.close()
 
def _build(svn_path, output_file, output=True, compress=False, use_keywords=True, version_name=None):
    if not version_name: version_name = ''
    msgs = []
    # Grab revision number
    p1 = subprocess.Popen(["svn", "info", BASE_CHECKOUT], stdout=subprocess.PIPE)
    p2 = subprocess.Popen(["grep", "Revision:"], stdin=p1.stdout, stdout=subprocess.PIPE)
    returnmsg = p2.communicate()
    revision = returnmsg[0].split(' ')[1].strip()
 
    tmp_path = "%s/%s/" % (TEMP_PATH, hashlib.md5(svn_path).hexdigest(),)
 
    # Export the build first.
    p1 = subprocess.Popen(["svn", "export", "--force", "-q", "-r%s" % (revision,), "%s%s" % (BASE_CHECKOUT, svn_path), tmp_path])
    if p1.wait() != 0:
        raise BuildError("SVN failed to export. Does %s exist?" % (svn_path,))
 
    if use_keywords:
        recurse_path(tmp_path, replace_keywords, keywords={
            '$$SVN:REVISION$$': revision,
            '$$SVN:VERSION$$': version_name
        })
 
    if compress:
        cmsgs = recurse_path(tmp_path, compress_files, base_path=tmp_path)
        if not cmsgs:
            raise NoCompressionError("No files were compressed.")
        else:
            msgs.extend(cmsgs)
 
    # Make sure output doesnt exist first
    try:
        os.remove("%s/%s" % (RELEASE_PATH, output_file,))
    except OSError:
        pass
 
    # Zip the archive in its release path.
    p1 = subprocess.Popen("zip -qr %s/%s *" % (RELEASE_PATH, output_file,), cwd=tmp_path, shell=True)
    if p1.wait() != 0:
        raise BuildError("Failed to create output ZIP")
 
    # Cleanup
    shutil.rmtree(tmp_path)
 
    if output:
        print "Created %s" % (output_file,)
        if msgs:
            for m in msgs:
                print "...%s" % (m,)
 
def build_tag(project_name, version_name):
    output_path = PATH_OVERRIDES.get(project_name, project_name)
    in_file = '%s/tags/%s' % (project_name, version_name)
    _build(in_file, '%s/%s.%s.zip' % (output_path, project_name, version_name), version_name=version_name)
    try:
        _build(in_file, '%s/%s.%s.compressed.zip' % (output_path, project_name, version_name,), version_name=version_name, compress=True)
    except NoCompressionError:
        pass
 
def build_trunk(project_name):
    output_path = PATH_OVERRIDES.get(project_name, project_name)
    in_file = '%s/trunk' % (project_name,)
    _build(in_file, '%s/%s.current.zip' % (output_path, project_name,), version_name="trunk")
    try:
        _build(in_file, '%s/%s.current.compressed.zip' % (output_path, project_name,), compress=True, version_name="trunk")
    except NoCompressionError:
        pass

The output looks something like this:

Created ibox/ibox.current.zip
Created ibox/ibox.current.compressed.zip
...Compressed ibox.js to 59.62% of normal
...Compressed readme/global.css to 80.34% of normal
...Compressed skins/lightbox/lightbox.css to 83.12% of normal
...Compressed skins/darkbox/darkbox.css to 87.80% of normal
Created ibox/ibox.2.17c.zip
Created ibox/ibox.2.17c.compressed.zip
...Compressed ibox.js to 59.59% of normal
...Compressed readme/global.css to 80.34% of normal
...Compressed skins/lightbox/lightbox.css to 83.12% of normal
...Compressed skins/darkbox/darkbox.css to 87.80% of normal

build_auto.py

Usage: python build_auto.py <revision>

This file handles the post commit hook from SVN and accepts the revision number as the argument.

"""
On a successful commit this will build zip versions
for any project.
"""

import sys
import build
import re
import subprocess

trunk_match = re.compile(r'/(?P
[^/]+)/trunk/')
tag_match = re.compile(r'/(?P
[^/]+)/tags/(?P[^/]+)/')

def main(revision=None):
    if not revision: revision = "HEAD"
    p1 = subprocess.Popen(["svn", "log", build.BASE_CHECKOUT, "-q", "-v", "-r%s" % (revision,)], stdout=subprocess.PIPE)
    p2 = subprocess.Popen(["grep", "/"], stdin=p1.stdout, stdout=subprocess.PIPE)
    p3 = subprocess.Popen(["cut", "-b 6-"], stdin=p2.stdout, stdout=subprocess.PIPE)
    returnmsg = p3.communicate()
    paths = returnmsg[0].strip().split("\n")
    trunk_builds = []
    tag_builds = {}
    for path in paths:
        path = path.split(' ')[0]
        match = trunk_match.search(path)
        if match:
            project = match.group('project')
            if project not in trunk_builds:
                build.build_trunk(project)
                trunk_builds.append(project)
            continue
        match = tag_match.search(path)
        if match:
            project = match.group('project')
            tag = match.group('tag')
            if tag not in tag_builds.get(project, []):
                build.build_trunk(project)
                if project not in tag_builds:
                    tag_builds[project] = []
                tag_builds[project].append(tag)
                build.build_tag(project, tag)
            continue

if __name__ == '__main__':
    if len(sys.argv) == 1:
        print "Usage: python build_auto.py "
        sys.exit(-1)
    else:
        main(sys.argv[1])

build_trunk.py

Usage: python build_trunk.py <project> ...

Builds the trunk release of N projects.

import sys
import build
 
def main(*projects):
    for p in projects:
        build.build_trunk(p)
 
if __name__ == '__main__':
    if len(sys.argv) == 1:
        print "Usage: python build_trunk.py <project> ..."
        sys.exit(-1)
    else:
        main(*sys.argv[1:])

build_tag.py

Usage: python build_tag.py <project> <tag>

Passing first the project name, and second the tag name, it will build a release for that specified tag.

import sys
import build
 
def main(project, tag):
    build.build_tag(project, tag)
 
if __name__ == '__main__':
    if len(sys.argv) != 3:
        print "Usage: python build_tag.py <project> <tag>"
        sys.exit(-1)
    else:
        main(*sys.argv[1:])

Debugging Django Errors (January 6th, 2010)
Big Companies Make Me Laugh (December 24th, 2009)
Jinja2 and django-registration (September 30th, 2009)
Using Jinja2 with Django (via Coffin) (September 14th, 2009)
Dynamic Keyword Arguments in Django URLs (July 15th, 2009)

3 Responses to "Writing a Build Bot using SVN and Python"

Subscribe to this topic with RSS or get the Trackback URL

Rich (Mar 28th):

That’s sweet as a nut! Thanks, Dave for sharing.

David (Mar 28th):

I was informed by my crazy C++ friend that most people use Makefile’s… but I just got down to actually learning bash scripting recently, so I don’t look forward to learning another mini language

jmhunter (Mar 28th):

heheehe or make?

David Cramer's Blog