Proposed runtests testconfig.txt changes

This is a discussion page for changes to the testconfig.txt setup in tamarin.

See bug 607719 and bug 600661

  1. Split testconfig.txt into two files - we currently have all information in this one file:
    • testconfig.txt : all directives for which tests should be run for which configuration
    • failconfig.txt : specify all tests that have failures/skips due to bugs
      • I'm not sure about the name here, suggestions welcome
      • Every line in this file would be required to have an associated bug.
  2. Add the ability to have a non-regex way of specifying configurations
    • In general, using regex patterns works well for defining the affected tests.  However, when a user wants to define "all configurations except ..." we have to use negative lookahead expressions which are hard to figure out correctly and can be quite confusing.
      • e.g. (^((?!(Dinterp|diff)).)*$|.*wordcode.*)
    • The configuration field could be split into two - include config & exclude config
  3. Add "deep" as a directive.  We currently handle deep files by skipping the file for all configurations, and then "include"ing the same files using the .*deep.* as the config.  This means that all deep tests have two configuration lines in testconfig.
    • The "include" directive is only used for deep tests and two performance-acceptance tests.  It is also slightly confusing as when tests are "include"d all other tests are ignored and only the included tests are run.
    • By having "deep" be a directive, we would not have to also "skip" the tests, thereby only having one entry per deep test.
    • One small drawback is that we would have to add special logic into the runtests framework.
  4. Create an expanded set of expectedfail modes (Bug 600661) - we currently can not specify what type of failure is expected.  Propose the following:
    • expectedfail : this covers any deviation in stdout
    • expectedfail-stderr : stderr output that is expected due to bug
    • expectedfail-exitcode : exitcode that is expected due to bug
    • Not sure if we should also have a way to specify the exact expectedfail type/string. 
  5. Change the way that testconfig lines are parsed.  We currently apply the longest testname/testcase key to each test, effectively overwriting any other directives applied to the test.   What would be ideal would be the ability to specify as many directives as needed to a test.

Document Tags and Contributors

 Last updated by: Cpeyer,