Converting from Make to Paver

By Doug Hellmann
January 18, 2009

As I briefly mentioned in an earlier post, I recently moved the PyMOTW build from make to Kevin Dangoor's Python-based build tool Paver. I had been wanting to try Paver out for a while, especially since seeing Kevin's presentation at PyWorks in November. As a long time Unix/Linux user, I didn't have any particular problems with make, but it looked intriguing. PyMOTW is one of the few projects I have with a significant build process (beyond simply creating the source tarball), so it seemed like a good candidate for experimentation.

Concepts:

The basic concepts with Paver and make are the same. Make "targets" correspond roughly to Paver "tasks". Paver places less emphasis on file modification time-stamps, though, so tasks are all essentially "PHONEY" targets. As with make, Paver keeps track of which dependencies are executed so they are not repeated while building any one target.

Tasks are implemented as simple Python functions. Paver starts by loading pavement.py, and tasks can be defined inline there or you can import code from elsewhere if needed. According to Kevin, once the main engine settles down enough to reach a 1.0 release, he doesn't anticipate a lot of active development on the core. Recipes for extending Paver can be added easily through external modules which would be distributed separately.

Building a Source Distribution:

The most important target from the old PyMOTW Makefile was "package". It ran sphinx-build to create the HTML version of the documentation then produced a versioned source distribution with distutils. The whole thing was bundled up and dropped on my computer desktop, ready to be uploaded to my web site.

package: clean html
	rm -f setup.py
	$(MAKE) setup.py
	rm -f MANIFEST MANIFEST.in
	$(MAKE) MANIFEST.in
	python setup.py sdist --force-manifest
	mv dist/*.gz ~/Desktop/

Paver sits on top of distutils, so one of the pre-defined tasks it has built-in is "sdist" (similar to python setup.py sdist, for producing source distributions of Python apps or libraries). In my case, I extended the task definition to perform some pre-requisite tasks and move the tarball out of the build directory onto the desktop of my computer, to make it easier to upload to my web site.

Let's look at the task definition:

@task
@needs(['generate_setup', 'minilib', 'html_clean', 
        'setuptools.command.sdist'
        ])
def sdist():
    """Create a source distribution.
    """
    # Copy the output file to the desktop
    dist_files = path('dist').glob('*.tar.gz')
    dest_dir = path(options.sdist.outdir).expanduser()
    for f in dist_files:
        f.move(dest_dir)
    return

The @task decorator registers the function as a task, tying it in to the list of options available from the command line. The docstring for the function is included in the help output (paver help or paver --help-commands).

$ paver help
---> help
Paver 0.8.1

Usage: paver [global options] [option.name=value] task [task options] [task...]

Run 'paver help [section]' to see the following sections of info:

options global command line options
setup available distutils/setuptools tasks
tasks all tasks that have been imported by your pavement

'paver help taskname' will display details for a task.

Tasks defined in your pavement:
blog - Generate the blog post version of the HTML for the current module
html - Build HTML documentation using Sphinx
html_clean - Remove sphinx output directories before building the HTML
installwebsite - Rebuild and copy website files to the remote server
pdf - Generate the PDF book
sdist - Create a source distribution
webhtml - Generate HTML files for website
website - Create local copy of website files
webtemplatebase - Import the latest version of the web page template from the source

To run the task, pass the name as argument to paver on the command line:

$ paver sdist

Prerequisites:

The @needs decorator specifies the prerequisites for a task, listed in order and identified by name. Paver prerequisites correspond to make dependencies, and are all run before the task, as you would expect.

In the Makefile, before building a source distribution I always ran the "clean" and "html" targets, too. That meant I had a fresh copy of the HTML version of PyMOTW, generated by sphinx. The next step was to build setup.py using a simple input template file processed by sed (so I didn't have to remember to edit the version and download URL every time).

Paver provides a task to generate a setup.py ("generate_setup"), so I no longer need to mess around with templates on my own. The "minilib" task writes a ZIP archive with enough of Paver to support installation through the usual python setup.py install or easy_install PyMOTW incantations.

Notice that "setuptools.command.sdist" is the fully qualified name to the task being redefined locally. That means in this case the standard work for sdist (producing the source distribution) is run prior to invoking my override function.

I've defined an "html_clean" task in pavement.py to take the place of the old make targets "clean" and "html":

@task
def html_clean():
    """Remove sphinx output directories before building the HTML.
    """
    remake_directories(options.sphinx.doctrees, options.html.outdir)
    call_task('html')
    return

remake_directories() is a simple Python function I've written to remove the directories passed to it and then recreate them, empty. It is the equivalent of rm -r followed by mkdir. It's not strictly necessary, but I'm paranoid about old versions of renamed files ending up in my generated output, so I always start with an empty directory.

Working with Files:

Paver's standard library includes Jason Orendorff's path library for working with directories and files. Using the library, paths are objects with methods (instead of just strings). Methods include finding the directory name for a file, getting the contents of a directory, removing a file, etc. -- the sorts of things you want to do with files and directories. One especially nice feature is the / operator, which works like os.path.join. It is simple to construct a path using components in separate variables, joining them with /.

The sdist function is responsible for copying the packaged source distribution to my desktop. It starts by using path's globbing support to build a list of the .tar.gz files created by setuptools.command.sdist. (There should only be one file, but predicting the name is more difficult than just using globbing.) The destination directory is configured through the options Bundle (a dictionary-like object with attribute lookup for the keys). Since the value of the option might include ~, I expand it before using it as the destination directory for the file move operation.

Options:

Make is usually configured via the shell environment and variables within the Makefile itself. Paver uses a collection of Bundle objects organized into a hierarchical namespace. The options can be set to static literal values, computed at runtime using normal Python expressions, or overridden from the command line.

Each task can define its own section of the namespace with options. Some underlying recipes (especially the distutils and sphinx integration) depend on a specific structure, documented with the relevant task documentation. For example, these are the settings I use when running sphinx:

    sphinx = Bunch(
        sourcedir='PyMOTW',
        docroot = '.',
        builder = 'html',
        doctrees='sphinx/doctrees',
        confdir = 'sphinx',
    ),

Tasks access the options using dot-notation starting with the root of the namespace. For example, options.sphinx.doctrees.

Running Shell Commands:

Even with the power of Python as a programming language, sometimes it is necessary to shell-out to run an external program. Paver makes that very easy. sh() wraps Python's standard library module subprocess to make it easier to work with for the sorts of use cases commonly found in a build system. Simply pass a string containing the shell command you want run, and optionally include the capture argument to have it return the output text (useful for commands like svnversion). sh() takes care of running the command, or in dry-run mode printing the command it would have run.

For example, the last step of building the PyMOTW PDF requires running a target included in the Makefile generated by Sphinx.

    latex_dir = path(options.pdf.builddir) / 'latex'
    sh('cd %s; make' % latex_dir)

Sphinx and Cog Integration:

Paver also includes built-in support for Sphinx. The standard integration with Sphinx supports producing HTML output. You can configure many of the Sphinx options you would normally put in a conf.py file directly through Paver's pavement.py. I had to override the way Sphinx is run by default, because I want to produce 3 different versions of HTML output (using different templates) and the PDF, but simpler projects won't have to do much more than set up the location of the input files.

In addition to Sphinx, Paver integrates Ned Batchelder's Cog, a templating/macro definition tool that lets you generate part of your documentation on the fly from arbitrary Python code. I've done some work to have cog run the PyMOTW examples and insert the output into the rst file before passing it to Sphinx to be converted to HTML or PDF. The process is complicated enough to warrant its own post, though, so that will have to wait for another day.

Conclusions:

Paver is a useful alternative to make, especially for Python-based packages. The default integration with distutils makes it very easy to get started. Build environments requiring a lot of external shell calls may find Makefile's easier to deal with. In my case, I was able to fold a couple of small Python scripts into the pavement.py file, so I eliminated a few separate tools.

It's hard to say whether a pavement file is "simpler" than a Makefile. Task definitions do not tend to be shorter than make targets, but the verbosity is an artifact of Python (function definitions and decorators, etc.) rather than anything inherent in the way Paver is designed.

A typical Paver configuration file is likely to be more portable than a Makefile, so that may be something to take into account. With file operations easily accessible in a portable library, it should be easy to set up your pavement.py to work on any OS.

For the complete pavement.py file used by PyMOTW, grab the latest release from the web site.


You might also be interested in:

News Topics

Recommended for You

Got a Question?