From predicting the weather to simulating disease transmission, scientific software plays an increasingly pivotal role in developing scientific understanding that informs our everyday lives.
However, they are also some of the most difficult software systems to properly test.
They have large, complex input spaces, are computationally expensive to run, often rely on stochastic black-box components, and are applied in exploratory contexts where the expected outcomes are not known.
From a practical standpoint, the time and effort that can be dedicated to testing is often limited, especially in an academic context, making it especially important to maximise the efficiency of the limited number of test runs we are able to perform.
The Causal Testing Framework has two main workflows:
Causal Testing involves specifying the expected causal relationships and testing that the data conforms to this.
Causal Discovery involves infering the causal effects from the data and checking that the model is reasonable.
Figure: Schematic diagram of the Causal Testing Workflow.
The Causal Testing Framework uses graphical causal inference to specify and validate software behaviour by estimating the causal effects between variables.
This requires three main components:
Causal Graph: This specifies the expected causal relationships between the variables in the form of a directed acyclic graph (DAG).
The nodes in your DAG represent variables in your system, and edges between the variables represent the “flow of causality” such that an edge from X to Y represents the value of Y being caused (i.e. directly affected) by the value of X.
Test Data: This is the data that will be used to estimate the causal effects between variables and evaluate your causal test cases.
This takes the form of a table in which columns represent the variables in your DAG and each row represents a run of the system.
Causal Tests: Each causal test case validates that the causal effect between the treatment and outcome variable that can be estimated from the test data is as expected.
The most basic causal test case simply validates the presence or absence of a causal effect.
The Causal Testing Framework can automatically generate a suite of such tests from the causal DAG alone.
You can then customise and refine these tests to suite your needs.
An example of this workflow can be seen in our tutorials.
You do not need to be familiar with causal inference to use the causal testing framework, since the technical parts can all be handled automatically “under the hood”.
Throughout this documentation, we will gently and informally introduce the various concepts that are necessary to understand and use the framework.
Interested readers are invited to check out the following additional resources for a more detailed introduction, as well as our paper on causal testing.
The Book of Why provides a “pop science” introduction and motivation to causal inference.
Figure: Schematic diagram of the Causal Discovery Workflow.
An alternative, more analytical approach involves using the Causal Testing Framework to automatically “discover” the causal relationships between variables from execution data.
This approach is well suited to users who do not have a concrete notion of the expected causal relationships between variables upfront, or are looking to gain an understanding of how an unfamiliar system works,
Note that we do not recommend using discovered DAGs for Causal Testing without careful manual inspection, since such graphs will trivially lead to passing test cases.
