mcell/testsuite/README.testutils

=========================================
0. Outline
=========================================

To aid in the development of MCell tests, I've developed a number of
utilities.  Most of these utilities are specific test types that may be
applied to specific MCell runs to analyze various aspects of the run.  Some
are infrastructure for the test suite itself.  A few are general utility
functions or classes.  I'll start with the general utilities, then I'll
discuss the infrastructure, and finally the specific test types.  All of these
are defined in Python modules in the system_tests subdirectory, and are set up
so that they may be imported into any test suite.

Most of the specific test types exist in two forms.  One will have a name
similar to:

    assertFileExists

and the other will have a name like:

    RequireFileExists

The former is a simple function call which performs the test.  The latter is
an object version which encapsulates the test.  (The reason for this will
become clear when I discuss the infrastructure, but essentially, the setup for
a test involves adding 0 or more encapsulated tests using the
'add_extra_check' method on an McellTest object.  It will automatically run
all of the tests at the end after the run completes.)  The object version does
not run the test as soon as it is constructed; instead, it stores the details
of the call and calls the corresponding assert* version when its own 'check'
method is called.  That is:

    r = RequireFileExists(filename)
    r.check()

is equivalent to:

    assertFileExists(filename)

Most tests will include a quick summary of the arguments to the test.  Within
the argument lists, I'm using the shorthand of:

    function_name(argument1,
                  argument2,
                  argument3='Argentina',
                  argument4=1764,
                  argument5=None)

to indicate that argument1 and argument2 are required, and that argument3,
argument4, and argument5 are optional and may be provided via the normal Python
mechanisms (either as keyword arguments, or as extra positional arguments), and
that if they are not provided,their respective defaults will be 'Argentina',
1764, and None.

At the very end will be a discussion of some test-specific utilities I've
developed which can be found in the individual test suite directories.  (For
instance, there is a base class which handles all of the commonalities in the
"parser" tests, and another for the macromolecule tests.)

Most of the tests will be fairly formulaic, and you may be able to get by
largely by copying existing tests and modifying them appropriately, but this
document should serve as a reasonable reference to the utilities to help write
more specialized tests.  (There may even be tests which do not fit will into
this framework, which will require some sleight-of-hand.  See test_009 in
mdl/testsuite/regression/test_regression.py for one such example.  In the
future, we should probably add slightly better support for testing situations
which require multiple runs punctuated by checkpoint/restore cycles.)

  1. General utilities
  2. Infrastructure
  3. Specific test types
  4. Test-specific utilities

=========================================
1. General utilities
=========================================

  testutils module:

      All of these utilities may be imported from testutils.

      ----
      safe_concat:      concatenate two lists or tuples, treating None as
                        an empty list

            safe_concat([1], [2])   => [1, 2]
            safe_concat((1,), (2,)) => (1, 2)
            safe_concat([1], None)  => [1]
            safe_concat(None, (2,)) => (2,)
            safe_concat(None, None) => None

      ----
      get_output_dir:   get the top-level output directory for the testsuite

            get_output_dir() => "/path/to/test_results"

      ----
      crange:           like 'range', but closed on both ends (range is open
                        on the top)

            crange(1, 10)    => [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
            crange(5, 1, -1) => [5, 4, 3, 2, 1]
            crange(0, 5, 2)  => [0, 2, 4]
            crange(9, 0, -2) => [9, 7, 5, 3, 1]

      ----
      cleandir:         nuke an entire directory subtree

            cleandir("/etc") => PANIC!  PLEASE DO NOT DO THIS!

      ----
      assertFileExists: throw an error if the specified filename doesn't
                        exist.
      RequireFileExists: object form of the same test

            assertFileExists("/etc/passwd")  => nothing happens (unless you
                                                ran the cleandir command
                                                before...)
            assertFileExists("/xxxxxxxxxx")  => exception thrown
            RequireFileExists("/etc/passwd").check() => nothing happens
            RequireFileExists("/xxxxxxxxxx").check() => exception thrown

      ----
      assertFileNotExists: throw an error if the specified filename does
                           exist.
      RequireFileNotExists: object form of the same test

            assertFileNotExists("/etc/passwd")  => exception thrown (unless
                                                   you ran the cleandir
                                                   command before...)
            assertFileNotExists("/xxxxxxxxxx")  => nothing happens
            RequireFileNotExists("/etc/passwd").check() => exception thrown
            RequireFileNotExists("/xxxxxxxxxx").check() => nothing happens

      ----
      assertFileEmpty: throw an error if the specified filename does not refer
                       to an existing, but empty file
      RequireFileEmpty: object form of the same test

            # These tests assume that /tmp/emptyfile exists and is 0 bytes long
            assertFileEmpty("/etc/passwd")  => exception thrown (hopefully)
            assertFileEmpty("/tmp/emptyfile")  => nothing happens
            assertFileEmpty("/tmp/xxxxxxxxx")  => nothing happens
            RequireFileEmpty("/etc/passwd").check() => exception thrown
            RequireFileEmpty("/tmp/emptyfile").check() => nothing happens
            RequireFileEmpty("/tmp/xxxxxxxxx").check() => nothing happens

      ----
      assertFileNonempty: throw an error if the specified filename does not
                          refer to an existing, non-empty file [optionally
                          checking that the file size exactly matches an
                          expected size]
      RequireFileNonempty: object form of the same test

            # These tests assume that /tmp/emptyfile exists and is 0 bytes long
            # These tests assume that /tmp/file_304 exists and is 304 bytes long
            assertFileNonempty("/etc/passwd")  => nothing happens (hopefully)
            assertFileNonempty("/tmp/emptyfile")  => exception thrown
            assertFileNonempty("/tmp/xxxxxxxxx")  => exception thrown
            assertFileNonempty("/tmp/file_304")  => nothing happens
            assertFileNonempty("/tmp/file_304", 304)  => nothing happens
            assertFileNonempty("/tmp/file_304", 305)  => exception thrown
            RequireFileNonempty("/etc/passwd").check()  => nothing happens
            RequireFileNonempty("/tmp/emptyfile").check()  => exception thrown
            RequireFileNonempty("/tmp/xxxxxxxxx").check()  => exception thrown
            RequireFileNonempty("/tmp/file_304").check()  => nothing happens
            RequireFileNonempty("/tmp/file_304", 304).check()  => nothing happens
            RequireFileNonempty("/tmp/file_304", 305).check()  => exception thrown

      ----
      assertFileEquals: throw an error if the specified filename does not
                        refer to an existing file, or if the contents of the
                        file do not exactly match a provided string
      RequireFileEquals: object form of the same test

            # These tests assume that /tmp/emptyfile exists and is 0 bytes long
            # These tests assume that /tmp/hw exists and has contents "hello world"
            assertFileEquals("/etc/passwd", "hello world")  => exception
            assertFileEquals("/tmp/emptyfile", "hello world")  => exception
            assertFileEquals("/tmp/xxxxxxxxx", "hello world")  => exception
            assertFileEquals("/tmp/hw", "hello world")  => nothing happens
            RequireFileEquals("/etc/passwd", "hello world").check()  => exception
            RequireFileEquals("/tmp/emptyfile", "hello world").check()  => exception
            RequireFileEquals("/tmp/xxxxxxxxx", "hello world").check()  => exception
            RequireFileEquals("/tmp/hw", "hello world").check()  => nothing happens

      ----
      assertFileMatches: throw an error if the specified filename does not
                         refer to an existing file, or if the contents of the
                         file do not match a provided regular expression.  By
                         default, the file is required to match the regular
                         expression at least once, but potentially an
                         unlimited number of times.  This may be changed by
                         providing the 'expectMinMatches' and/or
                         expectMaxMatches keyword arguments.
      RequireFileMatches: object form of the same test

            # These tests assume that /tmp/emptyfile exists and is 0 bytes long
            # These tests assume that /tmp/hw exists and has contents "hello world"
            assertFileMatches("/tmp/emptyfile", "h[elowr ]*d")  => exception
            assertFileMatches("/tmp/xxxxxxxxx", "h[elowr ]*d")  => exception
            assertFileMatches("/tmp/hw", "h[elowr ]*d")  => nothing happens
            assertFileMatches("/tmp/hw", "h[elowr ]*d", 2)  => exception
            assertFileMatches("/tmp/hw", "l", 2, 3)  => nothing happens
            assertFileMatches("/tmp/hw", "l", 2, 2)  => exception
            assertFileMatches("/tmp/hw", "l", 4, 6)  => exception
            RequireFileMatches("/etc/passwd", "h[elowr ]*d").check()  => exception
            RequireFileMatches("/tmp/emptyfile", "h[elowr ]*d").check()  => exception
            RequireFileMatches("/tmp/xxxxxxxxx", "h[elowr ]*d").check()  => exception
            RequireFileMatches("/tmp/hw", "h[elowr ]*d").check()  => nothing happens
            RequireFileMatches("/tmp/hw", "h[elowr ]*d", 2).check()  => exception
            RequireFileMatches("/tmp/hw", "l", 2, 3).check()  => nothing happens
            RequireFileMatches("/tmp/hw", "l", 2, 2).check()  => exception
            RequireFileMatches("/tmp/hw", "l", 4, 6).check()  => exception

      ----
      assertFileSymlink: throw an error if the specified filename does not
                         refer to a symlink, and optionally, if the symlink
                         does not point to a specific location.
      RequireFileSymlink: object form of the same test

            # These tests assume that /bin/sh is a symlink to "bash", as is
            # the case on many Linux machines
            assertFileSymlink("/bin/sh")  => nothing happens
            assertFileSymlink("/bin/sh", "bash")  => nothing happens
            assertFileSymlink("/bin/sh", "tcsh")  => wrath of the gods
            assertFileSymlink("/etc/passwd")  => exception
            RequireFileSymlink("/bin/sh").check()  => nothing happens
            RequireFileSymlink("/bin/sh", "bash").check()  => nothing happens
            RequireFileSymlink("/bin/sh", "tcsh").check()  => wrath of the gods
            RequireFileSymlink("/etc/passwd").check()  => exception

      ----
      assertFileDir: throw an error if the specified filename does not refer
                     to an existing directory.
      RequireFileDir: object form of the same test

            assertFileDir("/etc")  => nothing happens
            assertFileDir("/etc/passwd")  => exception
            assertFileDir("/xxxxxxxxxx")  => exception
            RequireFileDir("/etc").check()  => nothing happens
            RequireFileDir("/etc/passwd").check()  => exception
            RequireFileDir("/xxxxxxxxxx").check()  => exception

=========================================
2. Infrastructure
=========================================

  testutils module:

      All of these utilities may be imported from testutils.

      McellTest class: This class encapsulates the details of an MCell run.
                       It contains a number of methods to allow specification
                       of details of how to invoke the run, as well as
                       specification of the exact criteria for success.

          McellTest.rand:   random number stream  (instance of random.Random)

              rand_float = McellTest.rand.uniform(0, 20)   # rand float in [0, 20.0)
              rand_int   = McellTest.rand.randint(0, 200)  # rand int in [0, 200)

          McellTest.config: configuration object - gets settings from test.cfg

              McellTest.config.get("regression", "foo"):
                  get "foo" setting from "[regression]" section of test.cfg,
                  or "[DEFAULT]" section if not found in [regression section

          McellTest(cat, file, args): create a new McellTest instance.  It
              will look for its settings preferentially in the section of
              test.cfg whose name is passed as cat, but will default to the
              '[DEFAULT]' section for any settings not found in the specified
              section.  mcell will be launched using the mdl file passed as
              'file', and will be given the arguments passed in the args list.
              File is a path relative to the script (test_parser.py,
              test_regression.py, etc.) that created the McellTest instance.
              The test will not run until the 'invoke' method is called.

                mt = McellTest('foobar', '01-test_mcell_something.mdl', ['-quiet', '-logfreq', '100])

                      The above mcell run we've set up will look in the
                      [foobar] section of test.cfg for the mcell executable to
                      use, will find 01-test_mcell_something.mdl in the same
                      directory as the test script which created this
                      McellTest instance, and it will get (in addition to the
                      -seed, -logfile, and -errfile arguments) -quiet and
                      -logfreq 100.

              A default constructed mcell test expects to send nothing to
              stdin, to receive nothing from stdout/stderr, and expects that
              the executable will exit with an exit code of 0.

          set_check_std_handles(i, o, e): Sets whether each of stdin, stdout,
                       stderr should be checked.  "checking" stdin simply
                       means closing stdin immediately so that the program
                       will get a signal if it tries to read from stdin.
                       Checking stdout and stderr checks if they are empty
                       (i.e. produced no output).  Most of the runs in the
                       test suite redirect stdout and stderr to files
                       'realout' and 'realerr', so it's usually a good idea to
                       set these flags.  They are set by default, so it is
                       unnecessary to set them explicitly unless you want
                       output out stdout/stderr or want to send input to
                       stdin.

              mt = MCellTest(...)
              mt.set_check_std_handles(True, False, False)    # check stdin, don't check stdout/stderr
              mt.set_check_std_handles(1, 0, 0)    # check stdin, don't check stdout/stderr
              mt.set_check_std_handles(0, 1, 0)    # check stdout, don't check stdin/stderr
              mt.set_check_std_handles(0, 0, 1)    # check stderr, don't check stdin/stdout

          set_expected_exit_code(ec): Sets the exit code we expect from mcell
                       when it exits.  This defaults to 0, which means
                       successful exit.  When testing error handling, it may
                       be appropriate to set the expected exit code to 1.  It
                       is never expected for the process to die due to a
                       signal.

              mt = MCellTest(...)
              mt.set_expected_exit_code(1)

          add_exist_file(f): Adds to the test the criterion that upon
                        successful exit of mcell, the file (or files) f must
                        exist.  'f' must either be a string giving a single
                        filename, or an iterable (list, tuple, etc.) giving a
                        collection of filenames.

          add_empty_file(f): Adds to the test the criterion that upon
                        successful exit of mcell, the file (or files) f must
                        exist and be empty.  'f' must either be a string
                        giving a single filename, or an iterable (list, tuple,
                        etc.) giving a collection of filenames.

          add_nonempty_file(f, expected_size): Adds to the test the criterion
                        that upon successful exit of mcell, the file (or
                        files) f must exist and be nonempty.  'f' must either
                        be a string giving a single filename, or an iterable
                        (list, tuple, etc.) giving a collection of filenames.
                        expected_size is an optional argument, and if
                        specified, the file (or files) must have size in bytes
                        exactly matching expected_size, which is an integer.

          add_constant_file(f, cnt): Adds to the test the criterion that upon
                        successful exit of mcell, the file (or files) f must
                        exist and be nonempty.  'f' must either be a string
                        giving a single filename, or an iterable (list, tuple,
                        etc.) giving a collection of filenames.  cnt is a
                        string giving the verbatim contents expected in the
                        file (or files).  If the files do not match cnt byte
                        for byte, an exception will be thrown.

          add_symlink(f, target): Adds to the test the criterion that upon
                        successful exit of mcell, the file (or files) f must
                        exist and be a symlink.  'f' must either be a string
                        giving a single filename, or an iterable (list, tuple,
                        etc.) giving a collection of filenames.  target is an
                        optional string giving a required target for the
                        symlinks.  (The target must match verbatim, rather
                        than simply referring to the same file.)  If file is a
                        collection, target must either be None (unspecified)
                        or must be a collection, and they will be paired off:

                          mt.add_symlink(["foobar1.lnk", "foobar2.lnk"],
                                         ["./dat/foobar1.orig", "./dat/foobar2.orig"])

          add_extra_check(c): Adds an extra check to this test to be run upon
                        completion.  'c' must be an object with a 'check()'
                        method that can be called which throws an exception if
                        the test fails.  Most typically, the objects passed to
                        add_extra_check will be classes whose names begin with
                        Require, such as the various utility classes defined
                        in this file.

          invoke(testdir): Invokes this mcell test, creating a test directory
                        under testdir.  The test directory will be
                        sequentially named test-xxxx where xxxx is a 4-digit
                        decimal integer.  Generally, 'get_output_dir()' should
                        be passed to this:

              mt = MCellTest("foobar", "mdl_does_not_exist.mdl", [])
              mt.set_expected_exit_code(1)
              mt.invoke(get_output_dir())

                        Invoke will throw an exception if anything is amiss.

          check_output_files(): This may be overridden in McellTest subclasses
                        to insert custom logic as an alternative to writing a
                        Require class and adding it using add_extra_check.  If
                        you do override this, be sure to call the parent
                        class' version of this method:

                def check_output_files(self):
                    McellTest.check_output_files(self)
                    # now, add other tests here...

=========================================
3. Specific test types
=========================================

  reaction_output:

      All of these utilities may be imported from reaction_output.  See
      system_tests/reaction_output.py for more details.  Each test utility in
      that file includes a block comment explaining the use of the test and
      giving a simple example usage.

      ------------
      Counts (exact):
      ------------

          assertCounts
          RequireCounts

      This test type may be used when the counts must exactly match a given
      time course.  This almost never happens, but may happen in a few cases,
      for constant counts or non-reacting molecules.  Most cases are better
      handled with one of the other count mechanisms.
      
      In addition to validating the values, this test also validates that the
      data is well-formed -- that is, that it has the right number of rows and
      columns, and that it has a header if one is expected and does not have a
      header if not.

      The parameters to this type of count validation are:

          fname: the name of the reaction data output file 

          times_vals: a list of the exact rows expected in the file.  Each
                      item in the list should be a tuple of values.  For
                      instance, in a file counting a constant quantity, such
                      as produced by the count statement:

                        REACTION_DATA_OUTPUT
                        {
                          STEP = 1e-6
                          {5} => "output.txt"
                        }

                      in a run that ran for 100 iterations with a time step of
                      1e-6, times_vals might be set to:

                        [(f*1e-6, 5) for f in range(0, 101)]

          header (optional): a header to expect on the data.  If provided, it
                      must exactly match the first line of input.  If not
                      provided, all lines in the file are expected to be count
                      data.

          eps (optional): an epsilon for what is considered equality.
                      Defaults to 1e-8.

      As with the generic utilities, this may be called either as:

        assertCounts(fname, times_vals, header=None, eps=1e-8)

      or with the delayed form:

        r = RequireCounts(fname, times_vals, header=None, eps=None)
        mt.add_extra_check(r)
        mt.invoke(get_output_dir())

      ------------
      Counts (linear constraints):
      ------------

          assertCountConstraints
          RequireCountConstraints

      This test type is fairly flexible, and was already briefly described in
      the README file.  Essentially, you must provide a set of coefficients
      which, when multiplied by the columns in each row (optionally restricted
      to a given interval of time), will yield a constant value.  You may
      provide several such linear constraints.  The coefficients for all of
      the constraints are specified together in a big matrix, and the constant
      values to which they must evaluate are specified together in a list.
      Having said that, here are the parameters that may be specified:

          fname: filename to validate
          constraints (opt) : coefficients matrix
          totals (opt): result vector
          min_time (opt): time at which to start validating these constraints
          max_time (opt): time at which to stop validating these constraints
          header (opt): If None or False, no header may appear on the data; if
                        True, a non-empty header must appear on the file; if a
                        string value is provided, the header must exactly
                        match the value (except for leading and trailing
                        whitespace).
          num_vals (opt): The number of expected values within the window of
                        time.  If specified, there must be exactly this many
                        rows between min_time and max_time (or in the whole
                        file except the header if min_time and max_time are
                        not specified)
          minimums (opt): the minimum values which may appear in each column.
                        Presently, this value is only heeded if constraints
                        were specified, but that is a bug.  By default, we
                        assume that no value may be less than 0.
          maximums (opt): the maximum values which may appear in each column.
                        Presently, this value is only heeded if constraints
                        were specified, but that is a bug.  By default, we
                        assume that no value may be greater than 1e300.

      As with the generic utilities, this may be called either as:

        assertCountConstraints(fname,
                               constraints=None,
                               totals=None,
                               min_time=None,
                               max_time=None,
                               header=None,
                               num_vals=None,
                               minimums=None,
                               maximums=None)

      or with the delayed form:

        r = RequireCountConstraints(fname,
                                    constraints=None,
                                    totals=None,
                                    min_time=None,
                                    max_time=None,
                                    header=None,
                                    num_vals=None,
                                    minimums=None,
                                    maximums=None)
        mt.add_extra_check(r)
        mt.invoke(get_output_dir())

      ------------
      Counts (equilibrium):
      ------------

          assertCountEquilibrium
          RequireCountEquilibrium

      There are two ways one could imagine statstically comparing count values
      to an expected equilibrium.  One would be to repeatedly run the same
      simulation and average the same time value across multiple seeds.  The
      second would be to average the same count across a range of time values
      in a single run.  Obviously, this only works if the time values are all
      in a period of statistical equilibrium.  This comparison implements the
      second strategy.  A minimum and/or maximum time may be provided to limit
      the summation to a region which is expected to be in equilibrium, and
      expected equilibrium values may be provided (along with tolerances) for
      each column.  The test is considered to succeed if the average value of
      each column is within the given tolerance of each expected equilibrium
      value.

          fname: filename to validate
          values: expected equilibria for each column (list of floats)
          tolerances: allowable tolerances for equilibria
          min_time (opt): minimum time value at which to start averaging
          max_time (opt): maximum time value at which to stop averaging
          header (opt): If None or False, no header may appear on the data; if
                        True, a non-empty header must appear on the file; if a
                        string value is provided, the header must exactly
                        match the value (except for leading and trailing
                        whitespace).
          num_vals (opt): The number of expected values within the window of
                        time.  If specified, there must be exactly this many
                        rows between min_time and max_time (or in the whole
                        file except the header if min_time and max_time are
                        not specified)

      As with the generic utilities, this may be called either as:

        assertCountEquilibrium(fname,
                               values,
                               tolerances,
                               min_time=None,
                               max_time=None,
                               header=None,
                               num_vals=None)

      or with the delayed form:

        r = RequireCountEquilibrium(fname,
                                    values,
                                    tolerances,
                                    min_time=None,
                                    max_time=None,
                                    header=None,
                                    num_vals=None)
        mt.add_extra_check(r)
        mt.invoke(get_output_dir())

      ------------
      Counts (average event rate):
      ------------

          assertCountRxnRate
          RequireCountRxnRate

      When a reaction is at equilibrium, the forward and reverse reactions
      should proceed at roughly equal rates.  We can compute a number of
      expected events per iteration, and then validate that the average rate
      of occurrence of the reaction is within epsilon of the expected rate.
      The current cases for this start the reactions at equilibrium, and a few
      modifications will be needed to this if we want to use this on reactions
      which do not start at equilibrium.  The computation we are doing is:

          raw_count / (time - base_time)

      where base_time is the time at which the reactions began occurring, by
      default 0.  To make it work with a system which doesn't start at
      equilibrium, we should also specify an equilibrium time, and subtract
      the reaction count at the equilibrium time:

            (raw_count - counts[eq_time]) / (time - eq_time)

      Anyway, even for a system which starts at equilibrium, this is not
      immediately accurate -- we need to allow enough time to pass for several
      events to occur so that we get enough samples to account for the noise.

          fname: filename to validate
          values: expected event rate in events per microsecond for each column
          tolerances: tolerances for average rates for each column
          min_time (opt): minimum time value at which to start averaging
          max_time (opt): maximum time value at which to stop averaging
          header (opt): If None or False, no header may appear on the data; if
                        True, a non-empty header must appear on the file; if a
                        string value is provided, the header must exactly
                        match the value (except for leading and trailing
                        whitespace).
          
      As with the generic utilities, this may be called either as:

        assertCountRxnRate(fname,
                           values,
                           tolerances,
                           min_time=None,
                           max_time=None,
                           header=None)

      or with the delayed form:

        r = RequireCountRxnRate(fname,
                                values,
                                tolerances,
                                min_time=None,
                                max_time=None,
                                header=None)
        mt.add_extra_check(r)
        mt.invoke(get_output_dir())

      ------------
      Triggers:
      ------------

          assertValidTriggerOutput
          RequireValidTriggerOutput

      Trigger output may be checked for format, and to a certain degree, for
      content.  Triggers are currently checked for mechanical constraints, such
      as syntax (header, if header is expected, correct number of columns) and
      spatial localization.  In particular, you may specify the bounding box
      for a region of space within which all trigger hits in a particular file
      are expected to occur; any triggers which occur outside of this region
      will cause a test failure.

          fname: filename for trigger output
          data_cols: 0 for reaction output, 1 for hits, 2 for molecule counts
          exact_time: True if there should be an exact time column
          header: Expected header line, or None if no header is expected
          event_titles: collection of acceptable event titles if triggers
                        should include event titles, or None if they should not
                        have titles.
          itertime: the length of an iteration (only checked if exact_time is
                    True).  exact_time column is required to be within itertime
                    of the time column in each row
          xrange: a tuple of (xmin, xmax), or None to skip checking in the x
                  dimension.  An error will be signalled if any trigger in this
                  file occurs with x coordinate outside of the closed interval
                  [xmin, xmax].
          yrange: a tuple of (ymin, ymax), or None to skip checking in the y
                  dimension.  An error will be signalled if any trigger in this
                  file occurs with y coordinate outside of the closed interval
                  [ymin, ymax].
          zrange: a tuple of (zmin, zmax), or None to skip checking in the z
                  dimension.  An error will be signalled if any trigger in this
                  file occurs with z coordinate outside of the closed interval
                  [zmin, zmax].

        assertValidTriggerOutput(fname,
                                 data_cols,
                                 exact_time=False,
                                 header=None,
                                 event_titles=False,
                                 itertime=1e-6,
                                 xrange=None,
                                 yrange=None,
                                 zrange=None)

      or with the delayed form:

        r = RequireValidTriggerOutput(fname,
                                      data_cols,
                                      exact_time=False,
                                      header=None,
                                      event_titles=False,
                                      itertime=1e-6,
                                      xrange=None,
                                      yrange=None,
                                      zrange=None)
        mt.add_extra_check(r)
        mt.invoke(get_output_dir())

  -----------

  viz_output:

      All of these utilities may be imported from viz_output.  See
      system_tests/viz_output.py for more details.  Each test utility in that
      file includes a block comment explaining the use of the test and giving a
      simple example usage.

      ------------
      ASCII mode viz output:
      ------------

          assertValidVizFileAscii
          RequireVizAscii

        ASCII viz output will have 7 columns.  The first column is always a
        state value.  Columns 2-4 are the x, y, z coordinates of a molecule.
        Columns 5-7 are the normal vector for the molecule, or 0, 0, 0 if the
        molecule is a volume molecule.  This check will validate that the
        column has the right number of lines, and if state values are provided
        as parameters, will verify that only the expected states occur, and
        that volume molecules correspond only to legal states for volume
        molecules and surface molecules correspond only to legal states for
        surface molecules.  It will check that the state is an integer and that
        the other 6 columns are floating point values.  It does not presently,
        though it could, check that the normal vector, if it is not 0, has
        length 1.  Unlike most of the tests seen so far,
        RequireValidVizFileAscii works slightly differently from
        assertValidVizFileAscii.  assertValidVizFileAscii validates a single
        output file, but RequireValidVizFileAscii will validate the input file
        for each iteration number that is expected to produce output.

          assertValidVizFileAscii
              fname: filename to check
              sstates: list of legal surface molecule states (integer values)
              vstates: list of legal volume molecule states (integer values)
                 if both sstates and vstates are None, states will not be
                 checked; if only one of them is None, it is counted as an
                 empty list -- i.e. sstates=None means no surface molecules
                 should appear in the output
          
          RequireVizAscii
              basename: specified basename from MDL file for output
                        (.ascii.<iter>.dat will be appended to get the actual
                        filename)
              iters: iterations which are expected to produce ASCII mode output
              sstates: list of legal surface molecule states (as in
                       assertValidVizFileAscii)
              vstates: list of legal volume molecule states (as in
                       assertValidVizFileAscii)
              astates: list of legal states for both surface and volume
                       molecules.  If astates is not None, it is concatenated
                       to sstates and to vstates.

      It is recommended to use the Require* form of the ASCII viz output test:

        r = RequireVizAscii(basename,
                            iters,        # for instance:  range(0, 1000, step=100),
                            sstates=None,
                            vstates=None,
                            astates=None)
        mt.add_extra_check(r)
        mt.invoke(get_output_dir())

      ------------
      RK mode viz output:
      ------------

          assertValidVizFileRK
          RequireVizRK

      Only minimal checking is done for Rex's custom output format.
      Essentially, we only check that the file exists and (optionally) that it
      has the expected number of lines.

          fname: filename
          n_iters: number of expected (non-blank) lines in the file


      As with most tests, this may be invoked either immediately:

        assertValidVizFileRK(fname, n_iters=None)

      or in delayed form:

        r = RequireVizRK(name
                         n_iters=None)
        mt.add_extra_check(r)
        mt.invoke(get_output_dir())

      ------------
      DX mode viz output:
      ------------

        assertValidVizFilesDx
        RequireVizDx

      DX output is too complicated to thoroughly validate in an automated way,
      at least at present.  We can, however, validate at least that the
      expected files were all created.  It could, but does not presently
      validate that only the expected files are created (i.e. that output
      wasn't produced for iterations where we didn't expect it).  We also check
      that the files are non-empty.  (XXX: This may not be the correct thing to
      do?)

        dir: directory in which to look for the DX output
        molfile: MOLECULE_FILE_PREFIX if it was specified
        objprefixes: list of names in the OBJECT_FILE_PREFIXES if it was
                     specified
        alliters: list of iterations where we expect all types of output
        mpositers: list of iterations where we expect volume molecule position
                     output
        mstateiters: list of iterations where we expect volume molecule state
                     output
        epositers: list of iterations where we expect grid molecule position
                     output
        estateiters: list of iterations where we expect grid molecule state
                     output
        opositers: list of iterations where we expect mesh position output
        ostateiters: list of iterations where we expect mesh state output

      As with most tests, this may be invoked either immediately:

        assertValidVizFileDX(dir,
                             molfile=None,
                             objprefixes=None,
                             alliters=None,
                             mpositers=None,
                             mstateiters=None,
                             epositers=None,
                             estateiters=None,
                             opositers=None,
                             ostateiters=None)

      or in delayed form:

        r = RequireVizDX(dir,
                         molfile=None,
                         objprefixes=None,
                         alliters=None,
                         mpositers=None,
                         mstateiters=None,
                         epositers=None,
                         estateiters=None,
                         opositers=None,
                         ostateiters=None)
        mt.add_extra_check(r)
        mt.invoke(get_output_dir())

      ------------
      DREAMM V3 non-grouped viz output:
      ------------

        assertValidVizFilesDreammV3
        RequireVizDreammV3

      This validates some basic structural constraints about a non-grouped
      DREAMM V3 output.  Essentially, it validates the existence and
      (optionally) the size of the .iteration_numbers.bin and .time_values.bin
      files and the existence of the top-level .dx file.  In order to do these
      checks, it needs to know the DREAMM output directory, the output filename
      (from which the top-level DX file takes its name), and optionally, the
      number of distinct output iterations and timesteps.

        dir: output directory
        name: output set name
        n_iters: number of distinct output iterations
        n_times: number of distinct output times

      As with most tests, this may be invoked either immediately:

        assertValidVizFilesDreammV3(dir,
                                    name,
                                    n_iters=None,
                                    n_times=None)

      or in delayed form:

        r = RequireVizDreammV3(dir,
                               name,
                               n_iters=None,
                               n_times=None)
        mt.add_extra_check(r)
        mt.invoke(get_output_dir())

      ------------
      DREAMM V3 non-grouped viz output - binary mode molecule data:
      ------------

        assertValidVizFilesDreammV3MolsBin
        RequireVizDreammV3MolsBin

      In a DREAMM V3 output set which should contain binary mode molecule data,
      we can validate the existence of appropriate output files for each
      iteration.  This is a considerably more complex validation that the ones
      described thus far.

        dir: the DREAMM output directory
        alliters: sorted list of iterations where output is expected
        surfpositers: list of iters where surface molecule position output is
                expected
        surforientiters: list of iters where surface molecule orientation
                output is expected
        surfstateiters: list of iters where surface molecule state output is
                expected
        surfnonempty: flag indicating that at least one surface molecule should
                have been output
        volpositers: list of iters where volume molecule position output is
                expected
        volorientiters: list of iters where volume molecule orientation output
                is expected
        volstateiters: list of iters where volume molecule state output is
                expected
        volnonempty: flag indicating that at least one volume molecule should
                have been output

        Each of the iters lists may be set to None which indicates that the
        output is expected on every iteration in alliters.  To disable the
        check for a particular type of output, set its iters list to [] (the
        empty list).  On iterations where a particular type of output should
        not be output, but which are subsequent to an iteration where that type
        of output has been output, we will check for appropriate symbolic links.

      As with most tests, this may be invoked either immediately:

        assertValidVizFilesDreammV3MolsBin(dir,
                                           alliters,
                                           surfpositers=None,
                                           surforientiters=None,
                                           surfstateiters=[],
                                           surfnonempty=True,
                                           volpositers=None,
                                           volorientiters=None,
                                           volstateiters=[],
                                           volnonempty=True)

      or in delayed form:

        r = RequireVizDreammV3MolsBin(dir,
                                      alliters,
                                      surfpositers=None,
                                      surforientiters=None,
                                      surfstateiters=[],
                                      surfnonempty=True,
                                      volpositers=None,
                                      volorientiters=None,
                                      volstateiters=[],
                                      volnonempty=True)
        mt.add_extra_check(r)
        mt.invoke(get_output_dir())

      ------------
      DREAMM V3 non-grouped viz output - ASCII mode molecule data:
      ------------

        assertValidVizFilesDreammV3MolsAscii
        RequireVizDreammV3MolsAscii

      In a DREAMM V3 output set which should contain ASCII mode molecule data,
      we can validate the existence of appropriate output files for each
      iteration.

        dir: the DREAMM output directory
        alliters: all iterations where output is expected
        molnames: names of molecules for which output is expected
        positers: iterations where position output is expected (or None for
                  "all")
        orientiters: iterations where orientation output is expected (or None
                  for "all")
        stateiters: iterations where state output is expected (or None for
                  "all")

      As with most tests, this may be invoked either immediately:

        assertValidVizFilesDreammV3MolsAscii(dir,
                                             alliters,
                                             molnames,
                                             positers=None,
                                             orientiters=None,
                                             stateiters=[])

      or in delayed form:

        r = RequireVizDreammV3MolsAscii(dir,
                                        alliters,
                                        molnames,
                                        positers=None,
                                        orientiters=None,
                                        stateiters=[])
        mt.add_extra_check(r)
        mt.invoke(get_output_dir())

      ------------
      DREAMM V3 non-grouped viz output - binary mode mesh data:
      ------------

        assertValidVizFilesDreammV3MeshBin
        RequireVizDreammV3MeshBin

      For DREAMM output sets which should include at binary mode mesh data, we
      can verify the existence of the appropriate files and symlinks, and can
      optionally validate that at least one mesh was output.

        dir: the DREAMM output directory
        alliters: sorted list of iterations where output is expected
        positers: iteration numbers where mesh position output was produced
                  (None for all iterations in alliters)
        regioniters: iteration numbers where region data output was produced
                  (None for all iterations in alliters)
        stateiters: iteration numbers where mesh state output was produced
                  (None for all iterations in alliters)
        meshnonempty: if true, we will check that at least one mesh was output

      As with most tests, this may be invoked either immediately:

        assertValidVizFilesDreammV3MeshBin(dir,
                                           alliters,
                                           positers=None,
                                           regioniters=None,
                                           stateiters=[],
                                           meshnonempty=True)

      or in delayed form:

        r = RegionVizDreammV3MeshBin(dir,
                                     alliters,
                                     positers=None,
                                     regioniters=None,
                                     stateiters=[],
                                     meshnonempty=True)
        mt.add_extra_check(r)
        mt.invoke(get_output_dir())

      ------------
      DREAMM V3 non-grouped viz output - ASCII mode mesh data:
      ------------

        assertValidVizFilesDreammV3MeshAscii
        RequireVizDreammV3MeshAscii

      For DREAMM output sets which should include at ASCII mode mesh data, we
      can verify the existence of the appropriate files and symlinks, and can
      optionally validate that at least one mesh was output.

        dir: the DREAMM output directory
        alliters: sorted list of iterations where output is expected
        objnames: names of objects for which output should be produced
        objswithregions: names of objects which have defined regions (other
                  than ALL)
        positers: iteration numbers where mesh position output was produced
                  (None for all iterations in alliters)
        regioniters: iteration numbers where region data output was produced
                  (None for all iterations in alliters)
        stateiters: iteration numbers where mesh state output was produced
                  (None for all iterations in alliters)
        meshnonempty: if true, we will check that at least one mesh was output

      As with most tests, this may be invoked either immediately:

        assertValidVizFilesDreammV3MeshAscii(dir,
                                             alliters,
                                             objnames,
                                             objswithregions=None,
                                             positers=None,
                                             regioniters=None,
                                             stateiters=[],
                                             meshnonempty=True)

      or in delayed form:

        r = RegionVizDreammV3MeshAscii(dir,
                                       alliters,
                                       objnames,
                                       objswithregions=None,
                                       positers=None,
                                       regioniters=None,
                                       stateiters=[],
                                       meshnonempty=True)
        mt.add_extra_check(r)
        mt.invoke(get_output_dir())

      ------------
      DREAMM V3 grouped viz output:
      ------------

        assertValidVizFilesDreammV3Grouped
        RequireVizDreammV3Grouped

      A DREAMM V3 grouped file is best validated all-at-once.  Less validation
      may be done than in the ungrouped case because less of the state is
      directly accessible from the filesystem (i.e. without parsing DX files.)
      Still, a reasonable amount of validation can be done.

        dir: DREAMM output directory
        name: output set name
        cpno: checkpoint sequence number (defaults to 1)
        n_iters: total number of iterations (None to skip iteration count check)
        n_times: total number of distinct time points (None to skip time count
                  check)
        meshpos: True iff we should expect mesh positions output
        rgnindx: True iff we should expect region indices output
        meshstate: True iff we should expect mesh states output
        meshnonempty: True if we should expect at least one mesh output
        molpos: True iff we should expect molecule position output
        molorient: True iff we should expect molecule orientation output
        molstate: True iff we should expect molecule state output
        molsnonempty: True if we should expect at least one molecule output

      As with the other test types, this may be run in immediate form:

        assertValidVizFilesDreammV3Grouped(dir,
                                           name,
                                           cpno=1,
                                           n_iters=None,
                                           n_times=None,
                                           meshpos=True,
                                           rgnindex=True,
                                           meshstate=False,
                                           meshnonempty=True,
                                           molpos=True,
                                           molorient=True,
                                           molstate=False,
                                           molsnonempty=True)

      or in delayed form:

        r = RequireVizDreammV3Grouped(dir,
                                      name,
                                      cpno=1,
                                      n_iters=None,
                                      n_times=None,
                                      meshpos=True,
                                      rgnindex=True,
                                      meshstate=False,
                                      meshnonempty=True,
                                      molpos=True,
                                      molorient=True,
                                      molstate=False,
                                      molsnonempty=True)
        mt.add_extra_check(r)
        mt.invoke(get_output_dir())

=========================================
4. Test-specific utilities
=========================================

    I'm not actually going to explain how to use the existing test-specific
    utilities, but I am going to mention them as examples of how to write more
    complicated tests.  Many of the parser tests involve similar checks and
    setup, and as a result, special base classes have been created in the
    parser_test_types.py module in mdl/testsuite/parser.  This file is
    well-commented and illustrates the use of several of the test types I've
    mentioned here.

    test_parser.py includes a useful trick for batch-generating tests, which
    may be of interest if you need to add a large number of very similar tests.
    In this case, the tests are for ~80 tests for parsing malformed MDL files:

class TestParseInvalid(unittest.TestCase):

  def test025(self):
    InvalidParserTest("invalid-025.mdl").add_empty_file("invalid-025.tmp").invoke(get_output_dir())

def make_invalid_test(i):
  methname = "test%03d" % i
  filename = "invalid-%03d.mdl" % i
  func = lambda self: InvalidParserTest(filename).invoke(get_output_dir())
  setattr(TestParseInvalid, methname, func)

## Bulk generate invalid test cases 1...23, 26...27, 29...85
## 25 is specially specified, and 24 and 28 do not presently exist.
for i in crange(1, 23) + crange(26, 27) + crange(29, 85):
  make_invalid_test(i)

    Essentially, I created a test case with whatever custom tests I needed, and
    then wrote a loop to generate the other methods, which were formulaic and
    merely involved creating the appropriate test type with a formulaic mdl
    filename.  The rest of the logic is contained in the InvalidParserTest
    class which may be seen in parser_test_types.py.

    I added a similar macromol_test_types.py in the macromols subdirectory, but
    it contains very little of practical interest aside from an analogue of the
    InvalidParserTest class for catching macromolecule parser errors.  There
    are, however, a few utilities in test_macromols.py that may be of interest,
    as they show how to create custom check types to be added with
    'add_extra_check'.  In particular, CheckListReleasePositions is used to
    validate the placement locations of the macromolecule subunits based on
    their list placement.  This particular example is specific to one of the
    macromolecule tests, and I guess I must have decided it wasn't generic
    enough to merit inclusion in macromol_test_types?  test_macromols.py also
    includes a fairly frightening example of the use of RequireCountConstraints
    to validate the macromolecule counting code (see the test_surface method).