Software tests

Running Tests

The test driver is installed on all CS department machines. A configuration file is used to launch tests and mail results to subscribers. You can subscribe to a test summary or receive notices when select tests fail. The default configuration file is: $G/common/build/site/test.platform.cfg

If you want to perform a private test run and suppress mail notifications run:

$ ./common/build/valid/test.platform.py -noemail

> /systest/bin/vrl_fast  -noemail

To run a set of select tests and specify alternative mail notifications (usually to yourself) you can create and run using a custom configuration file. Make a private copy of the default configuration file and modify it. The comments in the default configuration file describe the format. Then run using the new configuration:

> /systest/bin/vrl_fast  -config  <My work area>/<Custom configuration>.cfg

Here is an example of a small custom configuration file.

#########################  test.platform.fast.cfg  #############################
#
Developer:  you@cs.brown.edu
Summary:    none

nag      none  linux common/build/valid/test.project.py common/include/nag
libnr    none  linux common/build/valid/test.project.py common/gg/libnr
libgg    none  linux common/build/valid/test.project.py common/gg/libgg
libds    none  linux common/build/valid/test.project.py common/gg/libds
libggpp  none  linux common/build/valid/test.project.py common/gg/libggpp
libnum   none  linux common/build/valid/test.project.py common/gg/libnum
libcff2  none  linux common/build/valid/test.project.py common/gg/libcff2
libmri   none  linux common/build/valid/test.project.py common/mri/libmri

#
#######################  end  test.platform.fast.cfg  ##########################

Writing Tests

It is a good idea to provide one or more quick tests that can be included in the test suite we run each night. Tests need to run in batch mode and simply return a zero status if successful and non-zero if they fail.

Create a make test target that runs a test that returns a zero status for success and a non-zero status for an error. The test should be able to run as a batch job (not interactive). Projects with multiple tests can use a target named test.<name> for each individual test and a target alltest to run them all.

Test targets should limit themselves to commands supported on both linux and Windows or use Python scripts. Currently only the "rm -f" and "diff" commands are supported. As always, absolute path names are to be avoided.

Large raw input files such as MRI images should be copied to a directory named $GROOT/data/<project>/test/. Cooked data and test output should go to a test directory inside the project directory. Tests should clear output files from prior runs to avoid false positives. Regression tests that use diff to check output should comapare against a file named: <root name>_base.<suffix>

Test System Maintenance

THe directories and files manages by tStaff are:

/tstaff/bin/regtest   # tStaff manages the driver.
   /systest/bin/*     # Run only the links; excludes other files.

/systest/bin/:   Changes are reflected to all department machines.
                 The build64 and build32 machines have separate test links.

Systest commands:

> systest-info -A    # print all project info
CTime: 1235158613
Description: Test software for the VRL group.
Links: bin/vrl_test
MTime: 1236091161
Name: vrltest
Owners: bb dhl

# Select project:  To add type in a project name and hit "Create".
# Create Project:  Description:  Enter a brief description.
#                       Owners:  Enter space separated user ids.
# Prject name = vrl -> creates /systest/projects/vrl
#
> systest-req        # Interactive box to add a new project (run on Linux box).

# Run after creating a new project and set up a test script:
#  add:     /systest/projects/<project>/<test>.<suffix>
#
> systest-link  /systest/projects/<project>/<test_name>.<suffix>  /systest/bin/<test_name>

# Run after creating a new project and set up a test script:
#  add:     /systest/projects/<project>/<test>.<suffix>
#
> systest-link  /systest/projects/<project>/<test_name>.<suffix>  /systest/bin/<test_name>