• ControlHowto

The key defining component of a job is its control file; this file defines all aspects of a jobs life cycle. The control file is a python script which directly drives execution of the tests in question.

Simple Jobs

You are automatically supplied with a job object which drives the job and supplies services to the control file. A control file can be as simple as:

job.run_test('kernbench')

The only mandatory argument is the name of the test. There are lots of examples; each test has a sample control file under tests/<testname>/control

If you're sitting in the top level of the autotest client, you can run the control file like this:

$ bin/autotest <control_file_name>

($ is just the shell prompt - you don't need that bit ;-)) You can also supply specific arguments to the test

job.run_test('kernbench', iterations=2, threads=5)

• First paramater is the test name
• The others are arguments to the test. Most tests will run with no arguments if you want the defaults

If you would like to specify a tag for the results directory for a particular test:

job.run_test('kernbench',  iterations=2, threads=5, tag='mine')

Will create a results directory "kernbench.mine" instead of the default "kernbench". This is particularly important when writing more complex control files that may run the same test multiple times, in order to properly separate the results of each of the test runs they will need a unique tag.

Flow control

One of the benefits of the use of a true programming language for the control script is the ability to use all of its structural and error checking features. Here we use a loop to run kernbench with different threading levels.

for t in [8, 16, 32]:
        job.run_test('kernbench', iterations=2, threads=t, tag='%d' % t)

Using the profilers facility

You can enable one or more profilers to run during the test. Simply add them before the tests, and remove them afterwards, eg:

job.profilers.add('oprofile')
job.run_test('sleeptest')
job.profilers.delete('oprofile')

If you run multiple tests like this:

job.profilers.add('oprofile')
job.run_test('kernbench')
job.run_test('dbench')
job.profilers.delete('oprofile')

It will create separate profiling output for each test - generally we do a separate profiling run inside each test, so as not to perturb the performance results. Profiling output will appear under <testname>/profiling in the results directory.

Again, there are examples for all profilers in profilers/<profiler-name>/control.

Creating filesystems

We have support built in for creating filesystems. Suppose you wanted to run the fsx test against a few different filesystems:

# Uncomment this line, and replace the device with something sensible
# for you ...
# fs = job.filesystem('/dev/hda2', job.tmpdir)

for fstype in ('ext2', 'ext3'):
        fs.mkfs(fstype)
        fs.mount()
        try:
                job.run_test('fsx', job.tmpdir, tag=fstype)
        finally:
                fs.unmount()

or if we want to show off and get really fancy, we could mount ext3 with a bunch of different options, and see how the performance compares across them:

fs = job.filesystem('/dev/sda3', job.tmpdir)

iters=10

for fstype, mountopts, tag in (('ext2', '', 'ext2'),
                               ('ext3', '-o data=writeback', 'ext3writeback'),
                               ('ext3', '-o data=ordered', 'ext3ordered'),
                               ('ext3', '-o data=journal', 'ext3journal')):
        fs.mkfs(fstype)
        fs.mount(args=mountopts)
        try:
                job.run_test('fsx', job.tmpdir, tag=tag)
                job.run_test('iozone', job.tmpdir, iterations=iters, tag=tag)
                job.run_test('dbench', iterations=iters, dir=job.tmpdir, tag=tag)
                job.run_test('tiobench', dir=job.tmpdir, tag=tag)
        finally:
                fs.unmount()

Rebooting during a job

Where a job needs to cause a system reboot such as when booting a newly built kernel, there is necessarily an interuption to the control script execution. The job harness therefore also provides a phased or step based interaction model.

def step_init():
        job.next_step([step_test])
        testkernel = job.kernel('2.6.18')
        testkernel.config('http://mbligh.org/config/opteron2')
        testkernel.build()
        testkernel.boot()          # does autotest by default

def step_test():
        job.run_test('kernbench', iterations=2, threads=5)
        job.run_test('dbench', iterations=5)

By defining a step_init this control script has indicated it is using step mode. This triggers automatic management of the step state across breaks in execution (such as a reboot) maintaining forward flow.

In broad brush, the control file is not simply executed once, but repeatedly executed until it indicates the job is complete. In a stand-alone context we would expect to re-start execution automatically on boot when a control file exists, in a managed environment the managing server would perform the same role.

Obviously looping is more difficult in the face of phase based execution. The state maintained by the stepping engine is such, that we can implement a boot based loop using step parameters.

def step_init():
        step_test(1)

def step_test(iteration):
        if (iteration < 5):
                job.next_step([step_test, iteration + 1])

        print "boot: %d" % iteration

        job.run_test('kernbench', tag="%d" % i)
        job.reboot()

Running multiple tests in parallel

The job object also provides a parallel method for running multiple tasks at the same time. The method takes a variable number of arguments, each representing a different task to be run in parallel. Each argument should be a list, where the first item on the list is a function to be called and all the remaining elements are arguments that will be passed to the function when it is called.

def first_task():
        job.run_test('kernbench')

def second_task():
        job.run_test('dbench')

job.parallel([first_task], [second_task])

This control file will run both kernbench and dbench at the same time. Alternatively, this could've been written as:

job.parallel([job.run_test, 'kernbench'], [job.run_test, 'dbench'])

However, if you want to so something more complex in your tasks than call a single function then you'll have to define your own functions to do it, as in the first example.

The parallel jobs are run through fork, so each task will be running in its own address space and you don't need to worry about performing any process-local synchronization between your separate tasks. However, these processes will still be running on the same machine and so still need to make certain that these tasks don't crash into each other while accessing shared resources (e.g. the filesystem). This means no rebooting during parallel tasks, and if you're running the same test in different tasks you must be sure to give each task a unique tag, and you must also be certain that the test does not need exclusive access to its testdir and srcdir (since these are shared between all runs of a test, tagged or not).

Utility Functions

There are a variety of utility functions for retrieving information about the test machine available in client/bin/autotest_utils.py. This module is pre-imported and should be available from any client-side control file. This provides a variety of capabilities (e.g. looking up disk, memory and cpu counts, loading modules, launching executables, unpacking tarballs, and so on. As of now, autotest_utils is also the only reliable mechanism for providing generic code to be reusable across a variety of tests.