Boris Kaminsky
README
This directory contains a suite of tests targetting the parser. Right now, the
main test is the "kitchen sink" test, which attempts to exercise every
non-terminal in the parser, to the degree that this is possible. According to
gcov, it achieves a reasonable degree of success. Gradually, I'll add tests
that cover the rest of the parser -- mostly error conditions, but generally
constructs which are incompatible in some way with constructs in the kitchen
sink test.
The primary interface to these tests is through the test_parser.py script:
1) the test scripts (and the instructions below) assume you are in the
mdl/testsuite/parser. This will not be necessary in the nearby future,
but for now...
2) create a test.cfg file with the path to the MCell executable you want to
test:
[DEFAULT]
mcellpath = /home/jed/src/mcell/3.1-pristine/build/debug/mcell
otherwise, it should assume you mean the mcell in your path.
3) Presently, you'll need to set your PYTHONPATH. I've included a script to
help with this. This will, shortly, become unnecessary, but in the
meantime, it should allow others to get in on the testing excitement. For
sh/bash/zsh/ksh users, run:
. set_python_path.sh
and for csh/tcsh users, run:
. set_python_path.csh
(Note, I've only tested the former, and you must be in the parser
directory for this to "do the right thing".) Essentially, you want your
PYTHONPATH environment variable to point to testsuite/system_tests in the
source tree, so if you'd rather set the environment variable yourself, you
can.
4) Run test_parser.py. If python is in your path, the simplest invocation
is:
./test_parser.py
which will run the complete parser test suite (currently something like
110 tests, taking on the order of 5-10 minutes to run). There are
specific subsets of the parser test you can invoke:
./test_parser.py fullsuite # all tests (same as default invocation)
./test_parser.py quicksuite # a quickly running subset of all tests
./test_parser.py errorsuite # all parser error handling tests
./test_parser.py kitchensinksuite # tests of all major features except VIZ output
./test_parser.py allvizsuite # all tests of VIZ modes
./test_parser.py vizsuite # all tests of DREAMM v3 VIZ modes
./test_parser.py oldvizsuite # all tests of non-DREAMM VIZ modes (DX, ASCII, RK)
./test_parser.py rtcheckpointsuite # all tests of realtime checkpoint parsing
rtcheckpointsuite will take around 4 minutes to run, as each run schedules
a checkpoint for approx. 30 seconds into the run, and then the timing of
the checkpointing and exiting is tested. Note that this also means that
when run on a very heavily loaded machine, these tests could falsely
report failure. The setup of the simulation, however, I think makes it
unlikely to occur in practice.
It's possible to run individual tests in the test suite as well, though
this is a more advanced usage, probably outside the scope of this
document.
I recommend redirecting the output to a file either using the shell
redirection operator '>', or using 'tee', as:
./test_parser.py | tee testlog
since it's useful to save the output when an error occurs.
You'll see one dot (.) for every test that passes, an F for every test
that fails, and an E for every error that occurs while trying to set up
the test. 'E' means that something went wrong in the test script, whereas
F means that a failure condition was detected while running the
executable.
Note that the test will also occasionally print out messages that look like:
Warning: Gaussian value outside of 95% confidence interval.
Expected to occur occasionally.
Value: 0.290164271582879 (mean=0, stdev=0.125)
They are part of the expression evaluation tests, which, aside from
analyzing all of the deterministic output, also examine the means of 64
uniform random numbers and of 64 Gaussian random numbers, checking the
means against approximate 95% confidence intervals. Once in a while, we
should see these warning messages.
The tests have been updated to include valid 95% confidence intervals.
I'm using the 2 standard deviations rule of thumb, which is slightly
generous. We have two tests per kitchen-sink-style test, one for uniform
numbers and one for gaussian. The kitchen sink suite includes 8
kitchen-sink-style tests, and the viz output suite includes another 13.
So, on a run of the kitchen sink suite, you'd expect to see each message
about confidence intervals about 40% of the time, and for a run of the
full parser suite, you'd expect to see a message about each confidence
interval about once.
If you see these message far more or less often than this rule-of-thumb
predicts, then something may have gotten jarred loose in the random number
generators.
5) Files created by the various tests will be created under individual
subdirectories under test_results. stdout/stderr are saved in the files
stdout/stderr, but all output should have been redirected to
realout/realerr in the same directory, which are the log_file and
err_file, resp. from MCell. The complete command-line is available in
cmdline.txt. All count/vol/viz output files should be under the directory
for the execution that produced them.
*****************
NOTE: These directories are mercilessly destroyed when the test suite is
run, so don't put anything you care about under test_results, and if you
want to examine/keep files produced from a run of the test suite, be sure
to do so before you run the test suite again.
*****************
* Test 01: Nearly every parser option.
* Test 02: Realtime checkpoint variants.
* invalid-*: Test error conditions.
Currently, the parser error conditions we test are:
General errors:
* invalid-001: Point is not 3D (4D point specified)
* invalid-002: Point is not 3D (2D point specified)
* invalid-003: Point is not 3D (1D point specified)
* invalid-004: Orient doesn't fit in short integer (40000)
* invalid-005: Orient doesn't fit in short integer (-40000)
* invalid-006: Orient doesn't fit in short integer (32768)
* invalid-007: Orient doesn't fit in short integer (32970 commas)
* invalid-008: Orient doesn't fit in short integer (32970 apostrophes)
* invalid-009: Range error - log [zero]
* invalid-010: Range error - log [negative]
* invalid-011: Range error - exp [overflow]
* invalid-012: Range error - log10 [zero]
* invalid-013: Range error - log10 [negative]
* invalid-014: Range error - sqrt [negative]
* invalid-015: Range error - asin(2)
* invalid-016: Range error - acos(-2)
* invalid-017: Range error - MOD(20.0, 0.0)
* invalid-018: Range error - 20 / 0
* invalid-019: Range error - 20 ^ 1e100
* invalid-020: Range error - 1e300*1e300*1e300*1e300
* invalid-049: Includes nested too deeply.
* invalid-050: Include file not found.
I/O subsystem errors:
* invalid-021: fopen fails (file doesn't exist)
* invalid-022: fopen fails (can't open in read mode)
* invalid-023: fopen fails (file is a directory)
* // Should add: fopen fails (permission denied)
// But be careful -- using a known unwritable file could be disastrous if
// anyone ever ran the test suite as root...
* invalid-025: fclose fails (closed twice)
* invalid-026: fclose fails (no such file)
Checkpoint-related errors:
* invalid-027: CHECKPOINT_ITERATIONS out of range
* // Should discuss 0 as a value for CHECKPOINT_ITERATIONS.
// Might be legitimate reasons for specifying 0 (for instance, to disable
// iteration-based checkpointing via a variable setting. Should either test
// for failure when CHECKPOINT_ITERATIONS == 0, or should test for correct
// behavior.
* invalid-029: CHECKPOINT_REALTIME multiply specified
* invalid-030: CHECKPOINT_REALTIME interval is negative
Errors in time/space step:
* invalid-031: TIME_STEP == 0
* invalid-032: TIME_STEP < 0
* invalid-033: SPACE_STEP == 0
* invalid-034: SPACE_STEP < 0
* invalid-035: CUSTOM_TIME_STEP < 0 *** not presently treated as an error
* invalid-036: CUSTOM_TIME_STEP == 0 *** not presently treated as an error
* invalid-037: CUSTOM_SPACE_STEP < 0 *** not presently treated as an error
* invalid-038: CUSTOM_SPACE_STEP == 0 *** not presently treated as an error
* invalid-039: TIME_STEP multiply specified
* invalid-040: SPACE_STEP multiply specified
* invalid-041: TIME_STEP not specified
Errors in surface initialization:
* invalid-042: Initialize effector with surface
* invalid-043: Initialize effector with volume molecule
Errors in reactions:
* invalid-044: Reaction omits forward rate.
* invalid-045: Reaction omits reverse rate.
* invalid-046: Reaction has two reverse rates
* invalid-047: Reaction has two forward rates
* invalid-048: Reaction rate is an array variable
Errors in release sites:
* invalid-051: Set density or number for list release site.
* invalid-052: Set molecule positions for non-list release site.
* invalid-053: Site radius/diameter not a number or a 3-vector.
* invalid-054: Site radius/diameter an incorrectly sized vector
* invalid-055: Concentration specified on spherical shell.
Errors in geometry:
* invalid-056: Polygon created with more than 3 vertices.
* invalid-057: Polygon created with fewer than 3 vertices.
* invalid-058: Voxel created with more than 4 vertices.
* invalid-059: Voxel created with fewer than 4 vertices.
* invalid-062: Attempt to set aspect ratio < 2.0
* invalid-084: Attempt to create a patch on a non-BOX object.
* invalid-085: Attempt to create a patch on an already triangulated BOX object.
Errors in regions:
* invalid-060: Reference to side by name in polygon
* invalid-061: Use of PATCH on a polygon
* invalid-063: Attempt to define a region on a non-polygonal object.
* invalid-064: Out-of-range element specifiers
* invalid-065: Element specifier bounds reversed.
Errors in reaction data output:
* invalid-028: Output timing omitted
* invalid-066: Count buffer specified < 1
* invalid-067: Count buffer specified > UINT_MAX
* invalid-068: TRIGGER and COUNT in the same statement
* invalid-069: Column header with wildcard expansion
Errors in VIZ output:
* invalid-070: VIZ_OUPUT block omits FILENAME option in DREAMM mode
* invalid-071: multiple VIZ output modes in one file
* invalid-072: new style molecules-spec with non-DX and non-DREAMM output
* invalid-073: multiple CUSTOM_RK output blocks in one file
* invalid-074: both binary and ascii mesh format specifiedpppp
* invalid-075: both binary and ascii molecule format specifiedpppp
* invalid-076: VIZ_MOLECULE_FORMAT specified outside of DREAMM mode
* invalid-077: VIZ_MESH_FORMAT specified outside of DREAMM mode
* invalid-078: DX-mode name list contains molecules without state values
* invalid-079: DX-mode name list contains meshes without state values
* invalid-080: DX-mode name list contains ALL_MESHES
* invalid-081: DX-mode name list contains ALL_MOLECULES
* invalid-082: In DREAMM mode, MOLECULE_FILE_PREFIX specified
* invalid-083: In DREAMM mode, OBJECT_FILE_PREFIXES specified
main test is the "kitchen sink" test, which attempts to exercise every
non-terminal in the parser, to the degree that this is possible. According to
gcov, it achieves a reasonable degree of success. Gradually, I'll add tests
that cover the rest of the parser -- mostly error conditions, but generally
constructs which are incompatible in some way with constructs in the kitchen
sink test.
The primary interface to these tests is through the test_parser.py script:
1) the test scripts (and the instructions below) assume you are in the
mdl/testsuite/parser. This will not be necessary in the nearby future,
but for now...
2) create a test.cfg file with the path to the MCell executable you want to
test:
[DEFAULT]
mcellpath = /home/jed/src/mcell/3.1-pristine/build/debug/mcell
otherwise, it should assume you mean the mcell in your path.
3) Presently, you'll need to set your PYTHONPATH. I've included a script to
help with this. This will, shortly, become unnecessary, but in the
meantime, it should allow others to get in on the testing excitement. For
sh/bash/zsh/ksh users, run:
. set_python_path.sh
and for csh/tcsh users, run:
. set_python_path.csh
(Note, I've only tested the former, and you must be in the parser
directory for this to "do the right thing".) Essentially, you want your
PYTHONPATH environment variable to point to testsuite/system_tests in the
source tree, so if you'd rather set the environment variable yourself, you
can.
4) Run test_parser.py. If python is in your path, the simplest invocation
is:
./test_parser.py
which will run the complete parser test suite (currently something like
110 tests, taking on the order of 5-10 minutes to run). There are
specific subsets of the parser test you can invoke:
./test_parser.py fullsuite # all tests (same as default invocation)
./test_parser.py quicksuite # a quickly running subset of all tests
./test_parser.py errorsuite # all parser error handling tests
./test_parser.py kitchensinksuite # tests of all major features except VIZ output
./test_parser.py allvizsuite # all tests of VIZ modes
./test_parser.py vizsuite # all tests of DREAMM v3 VIZ modes
./test_parser.py oldvizsuite # all tests of non-DREAMM VIZ modes (DX, ASCII, RK)
./test_parser.py rtcheckpointsuite # all tests of realtime checkpoint parsing
rtcheckpointsuite will take around 4 minutes to run, as each run schedules
a checkpoint for approx. 30 seconds into the run, and then the timing of
the checkpointing and exiting is tested. Note that this also means that
when run on a very heavily loaded machine, these tests could falsely
report failure. The setup of the simulation, however, I think makes it
unlikely to occur in practice.
It's possible to run individual tests in the test suite as well, though
this is a more advanced usage, probably outside the scope of this
document.
I recommend redirecting the output to a file either using the shell
redirection operator '>', or using 'tee', as:
./test_parser.py | tee testlog
since it's useful to save the output when an error occurs.
You'll see one dot (.) for every test that passes, an F for every test
that fails, and an E for every error that occurs while trying to set up
the test. 'E' means that something went wrong in the test script, whereas
F means that a failure condition was detected while running the
executable.
Note that the test will also occasionally print out messages that look like:
Warning: Gaussian value outside of 95% confidence interval.
Expected to occur occasionally.
Value: 0.290164271582879 (mean=0, stdev=0.125)
They are part of the expression evaluation tests, which, aside from
analyzing all of the deterministic output, also examine the means of 64
uniform random numbers and of 64 Gaussian random numbers, checking the
means against approximate 95% confidence intervals. Once in a while, we
should see these warning messages.
The tests have been updated to include valid 95% confidence intervals.
I'm using the 2 standard deviations rule of thumb, which is slightly
generous. We have two tests per kitchen-sink-style test, one for uniform
numbers and one for gaussian. The kitchen sink suite includes 8
kitchen-sink-style tests, and the viz output suite includes another 13.
So, on a run of the kitchen sink suite, you'd expect to see each message
about confidence intervals about 40% of the time, and for a run of the
full parser suite, you'd expect to see a message about each confidence
interval about once.
If you see these message far more or less often than this rule-of-thumb
predicts, then something may have gotten jarred loose in the random number
generators.
5) Files created by the various tests will be created under individual
subdirectories under test_results. stdout/stderr are saved in the files
stdout/stderr, but all output should have been redirected to
realout/realerr in the same directory, which are the log_file and
err_file, resp. from MCell. The complete command-line is available in
cmdline.txt. All count/vol/viz output files should be under the directory
for the execution that produced them.
*****************
NOTE: These directories are mercilessly destroyed when the test suite is
run, so don't put anything you care about under test_results, and if you
want to examine/keep files produced from a run of the test suite, be sure
to do so before you run the test suite again.
*****************
* Test 01: Nearly every parser option.
* Test 02: Realtime checkpoint variants.
* invalid-*: Test error conditions.
Currently, the parser error conditions we test are:
General errors:
* invalid-001: Point is not 3D (4D point specified)
* invalid-002: Point is not 3D (2D point specified)
* invalid-003: Point is not 3D (1D point specified)
* invalid-004: Orient doesn't fit in short integer (40000)
* invalid-005: Orient doesn't fit in short integer (-40000)
* invalid-006: Orient doesn't fit in short integer (32768)
* invalid-007: Orient doesn't fit in short integer (32970 commas)
* invalid-008: Orient doesn't fit in short integer (32970 apostrophes)
* invalid-009: Range error - log [zero]
* invalid-010: Range error - log [negative]
* invalid-011: Range error - exp [overflow]
* invalid-012: Range error - log10 [zero]
* invalid-013: Range error - log10 [negative]
* invalid-014: Range error - sqrt [negative]
* invalid-015: Range error - asin(2)
* invalid-016: Range error - acos(-2)
* invalid-017: Range error - MOD(20.0, 0.0)
* invalid-018: Range error - 20 / 0
* invalid-019: Range error - 20 ^ 1e100
* invalid-020: Range error - 1e300*1e300*1e300*1e300
* invalid-049: Includes nested too deeply.
* invalid-050: Include file not found.
I/O subsystem errors:
* invalid-021: fopen fails (file doesn't exist)
* invalid-022: fopen fails (can't open in read mode)
* invalid-023: fopen fails (file is a directory)
* // Should add: fopen fails (permission denied)
// But be careful -- using a known unwritable file could be disastrous if
// anyone ever ran the test suite as root...
* invalid-025: fclose fails (closed twice)
* invalid-026: fclose fails (no such file)
Checkpoint-related errors:
* invalid-027: CHECKPOINT_ITERATIONS out of range
* // Should discuss 0 as a value for CHECKPOINT_ITERATIONS.
// Might be legitimate reasons for specifying 0 (for instance, to disable
// iteration-based checkpointing via a variable setting. Should either test
// for failure when CHECKPOINT_ITERATIONS == 0, or should test for correct
// behavior.
* invalid-029: CHECKPOINT_REALTIME multiply specified
* invalid-030: CHECKPOINT_REALTIME interval is negative
Errors in time/space step:
* invalid-031: TIME_STEP == 0
* invalid-032: TIME_STEP < 0
* invalid-033: SPACE_STEP == 0
* invalid-034: SPACE_STEP < 0
* invalid-035: CUSTOM_TIME_STEP < 0 *** not presently treated as an error
* invalid-036: CUSTOM_TIME_STEP == 0 *** not presently treated as an error
* invalid-037: CUSTOM_SPACE_STEP < 0 *** not presently treated as an error
* invalid-038: CUSTOM_SPACE_STEP == 0 *** not presently treated as an error
* invalid-039: TIME_STEP multiply specified
* invalid-040: SPACE_STEP multiply specified
* invalid-041: TIME_STEP not specified
Errors in surface initialization:
* invalid-042: Initialize effector with surface
* invalid-043: Initialize effector with volume molecule
Errors in reactions:
* invalid-044: Reaction omits forward rate.
* invalid-045: Reaction omits reverse rate.
* invalid-046: Reaction has two reverse rates
* invalid-047: Reaction has two forward rates
* invalid-048: Reaction rate is an array variable
Errors in release sites:
* invalid-051: Set density or number for list release site.
* invalid-052: Set molecule positions for non-list release site.
* invalid-053: Site radius/diameter not a number or a 3-vector.
* invalid-054: Site radius/diameter an incorrectly sized vector
* invalid-055: Concentration specified on spherical shell.
Errors in geometry:
* invalid-056: Polygon created with more than 3 vertices.
* invalid-057: Polygon created with fewer than 3 vertices.
* invalid-058: Voxel created with more than 4 vertices.
* invalid-059: Voxel created with fewer than 4 vertices.
* invalid-062: Attempt to set aspect ratio < 2.0
* invalid-084: Attempt to create a patch on a non-BOX object.
* invalid-085: Attempt to create a patch on an already triangulated BOX object.
Errors in regions:
* invalid-060: Reference to side by name in polygon
* invalid-061: Use of PATCH on a polygon
* invalid-063: Attempt to define a region on a non-polygonal object.
* invalid-064: Out-of-range element specifiers
* invalid-065: Element specifier bounds reversed.
Errors in reaction data output:
* invalid-028: Output timing omitted
* invalid-066: Count buffer specified < 1
* invalid-067: Count buffer specified > UINT_MAX
* invalid-068: TRIGGER and COUNT in the same statement
* invalid-069: Column header with wildcard expansion
Errors in VIZ output:
* invalid-070: VIZ_OUPUT block omits FILENAME option in DREAMM mode
* invalid-071: multiple VIZ output modes in one file
* invalid-072: new style molecules-spec with non-DX and non-DREAMM output
* invalid-073: multiple CUSTOM_RK output blocks in one file
* invalid-074: both binary and ascii mesh format specifiedpppp
* invalid-075: both binary and ascii molecule format specifiedpppp
* invalid-076: VIZ_MOLECULE_FORMAT specified outside of DREAMM mode
* invalid-077: VIZ_MESH_FORMAT specified outside of DREAMM mode
* invalid-078: DX-mode name list contains molecules without state values
* invalid-079: DX-mode name list contains meshes without state values
* invalid-080: DX-mode name list contains ALL_MESHES
* invalid-081: DX-mode name list contains ALL_MOLECULES
* invalid-082: In DREAMM mode, MOLECULE_FILE_PREFIX specified
* invalid-083: In DREAMM mode, OBJECT_FILE_PREFIXES specified