Causal Testing

This quick start guide shows how to carry set up and run causal tests using the command line interface. This is the simplest way to interact with the Causal Testing Framework as you do not need to write any code. The result will be a JSON file containing a set of pass/fail test outcomes for each of the specified causal relationships that you can then use to explore the system.

Step 1: Prepare the DAG

The first step is to specify the expected causal relationships between your variables using a directed acyclic graph (DAG). To do this, we use the DOT language, which provides an intuitive text-based representation of DAGs. The syntax is very lightweight: an edge from node X to Y is specified as X -> Y;. If you would prefer to use a visual editor, you can use Dagitty and copy the Model code.

A simple example is shown below. The first line specifies that the graph is a digraph (directed graph), and names it expected_relationships. The next three lines list the variables X, Y, and Z. The next line lists a single edge X -> Y, indicating that X should cause Y. Z has no incoming or outgoing edges, so should be independent of both X and Y.

digraph expected_relationships {
 X;
 Y;
 Z;

 X -> Y;
}

Step 2: Prepare the Causal Test Cases

Having prepared the causal DAG, you can then use the Causal Testing Framework automatically convert the specified causal relationships to test cases. Each DAG implicitly encodes two types of causal relationship: causal dependences (i.e. the edges of the DAG) and causal independences (i.e. the non-edges) of the graph. Assuming you have saved your DAG in a text file called dag.dot in the current working directory, you can generate the corresponding causal test cases using the following command from your command shell:

causal-testing generate --dag-path dag.dot -output tests.json

This will output a JSON file containing the causal test cases to tests.json. While these test cases are executable “out of the box”, they can be fully customised to suit your needs. To do this, you can either manually edit tests.json (be careful as your changes will be overwritten if you regenerate the test cases), or by providing additional configuration options to the generate command above (run causal-testing generate --help to see the full list).

Step 3: Prepare the Test Data

Causal test cases are evaluated statistically with respect to a set of system executions. This set of executions is specified as a table of values in which each column corresponds to a variable in your DAG, and each row represents a valuation of those variables for a single run of the system. For an example, check out our interactive tutorial.

A major strength of the Causal Testing Framework is that the specification of the expected causal relationships is completely separate from the collection of test data, so you can evaluate the same tests on multiple different datasets with very little additional effort. This means that, if you already have some data from previous runs of the system, you can get to testing straight away without needing to run system under test again. The framework supports several file formats, including CSV, excel, parquet, and even HTML.

Step 4: Evaluate the Test Cases

We now have everything we need to evaluate the causal tests: the DAG, the data, and the test cases themselves. Assuming your data is saved in a single CSV file called data.csv, you can execute your causal tests with the following command:

causal-testing test --dag dag.dot --data-paths data.csv --test-config tests.json --output test_results.json

This will execute your causal test cases and produce a file called test_results.json that will contain your causal test results. There are various configuration options at this stage. Run causal-testing test --help to see them all.

Note

In traditional testing, when a test case fails, this means that there must be a problem with either the system or the test case. Because causal testing is a statistical technique, test outcomes depend on the data they are evaluated with. If you have insufficient data to calculate a reliable causal effect estimate, tests may fail even for fault-free systems. Check out our Test Data and Causal Test Adequacy pages for more information.