The standard tests
Asserts - reliability
in the face of complexity
The "run" command
The Perl scripts
If you find
a problem
The standard test cases that are computed every night in Lexington are the set of files with names ending in ".in". Each contains the commands needed to compute a particular model. When they are computed they produce output files with the same name but ending in ".out". Additional files are created too - these mostly results of all the assert commands, ending in ".asr", and overviews of the model results ".ovr".
The purpose of each test case is given in the documentation that follows the input commands. Cloudy stops reading input commands if the end of file or a blank line is encountered. Each *.in file begins with the commands, followed by the blank line to tell the code to stop reading, then followed by a description of the purpose of the test. This description is totally ignored - the command parser stops when it encounters the first empty line.
Asserts provide the ability to automatically validate complex results. There are two types of asserts here - a set of commands that tell the code what answer to expect, and a C language command that checks internal decisions made by the code.
All of the files in the test suite include assert commands. The assert command was introduced in C94, is described in the Miscellaneous Commands section of Hazy I, and provides the infrastructure needed for complete automatic testing of the code. This command tells the code what answer to expect. If the computed results do not agree then the code exits with an error condition. These assert commands have nothing to do with the simulation itself, and would not be included in an actual calculation. You should ignore them, or even delete all of them (a perl script is provided to do this)..
The source code also includes many C assert macros that are designed to validate the code's internal decisions. The assert macro only exists if the NDEBUG macro is not set when the code is compiled. (If the NDEBUG macro is set then assert macros within the source are ignored by the compiler.) The test cases should be run at least once with the assert macro active, that is, do not include a compiler option to define the NDEBUG macro. In most compilers the NDEBUG macro is set when compiler optimization is set to a high level. In practice this means that the entire code should be compiled with only low optimization and the tests computed to validate the platform. Then recompile with higher optimization for production runs, and recompute the test cases.
When executed as a stand-alone program the code expects to read commands from
standard input and write results to standard output. I compute single models by defining a shell script that
called "run".
It contains the following line
cloudy.exe < $1.in > $1.out
The Cloudy commands needed to compute a model are placed in a text file with a name
like "orion.in", indicating that it
is the input file to compute a model of Orion. Then the shell command line
run orion
will read the contents of orion.in, compute the model, and write the results back to orion.out.
I wrote a series of Perl scripts to compute all test cases and then check for errors. These provide automatic ways to validate the code.
Each Perl script will need to be changed before it can be used since they need to know the names of directories where files are located, and how to find the Cloudy executable. Each script explains which variables need to be set for the script to run.
Perl comments start with a sharp sign "#" and end with the end of
line.
Perl variable names begin with a dollar sign "$".
A Perl script is executed by typing perl
followed by the name of the script, as in perl runall.pl
The runall.pl script will compute all the models, the files *.in, in the current directory. You need to change the variable $exe to point to the Cloudy executable on your system.
This script searches for problems in all of the test cases (the *.out files) in the current directory. It will look for all botched asserts, warnings, and models that did not end. A summary is produced in the file checkall.txt.
This is by far the most important script, and is fundamental to the code's development. This script is executed every night that the code has changed. It computes all models and then checks the results looking for failed asserts or models that did not end. It then sends email announcing success or failure. It also backs up copies of the results, the executable that produced them, and the code source.
The backup files have names, in order of increasing age, *.bak, *.bk1, *.bk2, *.bk3, *.bk4. The output, executable, data, and source files are backed up this way as well. The executable files have names like cloudy.exe, cloudy_bak.exe, cloudy_bk1.exe, etc. The data files have names like data.tar, data_bak.tar, etc.
It is possible to run old versions of the code using versions of the run command. I have scripts called runbak, runbk2, runbk3, and runbk4, that will read an input file, use the appropriate version of the backed up executable, and create output files with names like name1.bak, name2.bak, etc, where the number indicates how far back the executable file was.
This searches for all output files that did not properly end, and reruns them. You need to change the variable $exe to point to the Cloudy executable.
This script will rerun all models that had botched asserts. This is mostly used while developing the code.
This script was written by Peter van Hoof and will run the test suite using a number of processors. The beginning of the script says how to use it.
This script will rerun all models that crashed - the did not end at all. This is mostly used while developing the code.
This script will rerun some of the models - those listed in runsome.dat.
This backs up some files onto a CDRW and sends email announcing results.
This script creates two files that document the test suite. The file doc_tsuite.html is an html representation of the set of tests, while doc_tsuite.txt is a comma delimited table that can be incorporated into a word processor.
Cloudy should run on all platforms without errors. Botched asserts or outright crashes should never happen. I can't fix it if I don't know it's broken. Please let me know of any problems. My email address is gary@cloud9.pa.uky.edu
Visit http://nimbus.pa.uky.edu/cloudy for details and latest updates.
Good luck,
Gary J. Ferland
