cdt/tools/toolchain-tester

CDT Toolchain Testing Framework

Running Tests

Supported features:

• Running individual tests/test suites
• Parallel execution of tests
• Comparing output WASMs for correctness
• Comparing output abis for correctness
• Outputting results a human readable format
• JSON File for controlling test runs (expected output, compile flags, etc)

Types of tests

There are multiple different types of toolchain tests, each is designed to exercise a different part of the toolchain.

• compile-fail: fail to compile
• compile-pass: compile successfully
• build-fail: fail to compile and link
• build-pass: compile and link successfully
• abigen-pass: abis are generated correctly
• abigen-fail: fail to generate abi

Test organization

Tests are put into directories under tests/toolchain based on their type as seen above (ie all compile-fail tests would be in tests/toolchain/compile-fail)

If a test is intended to check the output WASM matches an expected WASM, the expected WASM is defined in the JSON file. A compressed version can be generated by running xxd -p <file_name>.

Test files should include a comment summarizing the point of the test at the start of the file. Focus on parts of the test that are more important, and what the bug was that the test is fixing. Issue/PR numbers are also helpful.

JSON File

The tests are managed by a JSON file placed next to the test file. It can define expected output, compile flags, etc.

Expected Output

Allowable type of expected output are:

• "exit-code": Checks that it exited with a given code.
• "stderr": Checks for matching stderr. Currently a non-exact match.
• "wasm": A compressed version of the hex array representing the expected WASM.
• "abi": A stringified version of the abi that is expected.

Example files:

{
    "tests": [
        {
            "compile-flags": ["-O0", "-fno-lto"],
            "expected": {
                "exit-code": 0
            }
        }
    ]
}

{
    "tests": [
        {
            "expected": {
                "stderr": "iostreams currently clash with eosio::datastream"
            }
        }
    ]
}

{
"tests": [
        {
            "expected": {
                "wasm": "0061736d01000000019c808080000660027f7e0060027f7f0060017e0060017f0060037e7e7e0060000002bb818080000903656e760f5f5f6c696e6561725f6d656d6f727902000103656e76195f5f696e6469726563745f66756e6374696f6e5f7461626c650170000003656e76087072696e74735f6c000103656e76067072696e746e000203656e76067072696e7473000303656e760c656f73696f5f617373657274000103656e76115f5f7761736d5f63616c6c5f63746f7273000503656e7611656f73696f5f6173736572745f636f6465000003656e760e5f5f6378615f66696e616c697a650003038480808000030000040aef81808000035801027f418080808000210202404180808080002d00002203450d000240034020034125460d0120024101108080808000200241016a22022d00002203450d020c000b0b2001108180808000200241016a1082808080000b0b7901027f418080808000210202404180808080002d00002203450d000240034020034125460d0120024101108080808000200241016a22022d00002203450d020c000b0b2001108180808000200241016a1082808080000b02402001428080808080c0c6d1ea00510d004100418a808080001083808080000b0b1a001084808080004100420110858080800041001086808080000b0bb580808000020041000b0a4e616d65203a20250a0000410a0b20636865636b206e616d65206e6f7420657175616c20746f206068656c6c6f600000bf80808000082e696d706f72747311656f73696f5f6173736572745f636f64650c656f73696f5f617373657274087072696e74735f6c067072696e746e067072696e7473008c808080000a2e656f73696f5f6162690000bf81808000076c696e6b696e67010885818080000c0004071a5f5a4e3568656c6c6f326869454e35656f73696f346e616d65450102062e4c2e73747200000a0010000010010010020004081d5f5a4e3568656c6c6f35636865636b454e35656f73696f346e616d65450102082e4c2e7374722e31010020001003000509125f5f696e736572745f656f73696f5f61626900100400100500100605a580808000020e2e726f646174612e2e4c2e7374720100102e726f646174612e2e4c2e7374722e31010000c7808080000a72656c6f632e434f4445030f0406010004100100002d02004803005304045f0100046901000086010200a1010300ac010404c701060000cd010700d7010900e1010a00e9010b"
            }
        }
    ]
}

{
    "tests": [
        {
            "expected": {
                "abi": "{\"____comment\": \"This file was generated with eosio-abigen. DO NOT EDIT\",\n\"version\": \"eosio::abi/1.1\",\"types\": [],\"structs\": [..."
            }
        }
    ]
}

Command API

usage: toolchain-tester [-h] [-v] [-j JOBS] [-t TESTS] [--format {human,xunit}]
               [--cdt CDT]
               test_directory

positional arguments:
  test_directory        The directory where the toolchain tests are located.

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         Print verbose output
  -j JOBS, --jobs JOBS  Number of threads to use for parallel execution
                        (defaults to half of the system max)
  -t TESTS, --tests TESTS
                        Test/Testsuite to run (defaults to all)
  --format {human,xunit}
                        Format of the test output (defaults to human)
  --cdt CDT             Path to CDT (defaults to built CDT in this repo)