Causal Estimate

This page provides an overview on how to choose the most appropriate estimator for your workflow. When using the generate command to generate causal tests from a causal DAG, the estimator used is chosen based on the datatype of your outcome variable:

Linear regression is used for numerical variables
Logistic regression is used for boolean variables
Multinomial regression is used for categorical variables

In general, you won’t need to change this. However, if you have variables that are not recorded in your data, you may need to use the instrumental variable estimator. This uses the concept of instrumental variables to adjust for variables without needing to know their values. To change the estimator, you will need change the name of the estimator in the JSON representation of the test cases produced by the generate command. Depending on which estimator you are using, you may also need to add values for different parameters. See below for details.

Another useful customisation option for regression estimators is to modify the formula used for estimation. By default, the linear, logistic, and multinomial regression estimators all use the formula Y ~ X + Z1 + Z2 + ..., where Y is the outcome variable, X is the treatment variable, and Z1, Z2, etc. are the variables in the adjustment set. However, if you know in advance that your causal relationship is non-linear, it may be wise to refine this equation somewhat. This is mostly quite intuitive, but does have a few quirks. See the patsy documentation for an explanation of the available operators. An example can be seen in our tutorials.

LinearRegressionEstimator

Recommended use: For continuous numerical outcomes (e.g. the number of people who are vaccinated).

class causal_testing.estimation.linear_regression_estimator.LinearRegressionEstimator(treatment_variable: str, outcome_variable: str, control_value: float = None, treatment_value: float = None, adjustment_set: set[str] = None, adjustment_config: dict[str, Any] = None, formula: str = None, alpha: float = 0.05)

Bases: RegressionEstimator

A Linear Regression Estimator is a parametric estimator which restricts the variables in the data to a linear combination of parameters and functions of the variables (note these functions need not be linear).

estimate_ate(df: DataFrame) → EffectEstimate

Estimate the average treatment effect of the treatment on the outcome. That is, the change in outcome caused by changing the treatment variable from the control value to the treatment value.

Parameters:: df – The data to use.
Returns:: The average treatment effect and the Wald confidence intervals.

estimate_ate_calculated(df: DataFrame) → EffectEstimate

Estimate the ATE of the treatment on the outcome. That is, the change in outcome caused by changing the treatment variable from the control value to the treatment value. Here, we actually calculate the expected outcomes under control and treatment and divide one by the other. This allows for custom terms to be put in such as squares, inverses, products, etc.

Parameters:: df – The data to use.
Returns:: The average treatment effect and the Wald confidence intervals.

estimate_coefficient(df: DataFrame) → EffectEstimate

Estimate the unit average treatment effect of the treatment on the outcome. That is, the change in outcome caused by a unit change in treatment.

Parameters:: df – The data to use.
Returns:: The unit average treatment effect and the Wald confidence intervals.

estimate_risk_ratio(df: DataFrame) → EffectEstimate

Estimate the risk_ratio effect of the treatment on the outcome. That is, the change in outcome caused by changing the treatment variable from the control value to the treatment value.

Parameters:: df – The data to use.
Returns:: The average treatment effect and the Wald confidence intervals.

gp_formula(df: DataFrame, ngen: int = 100, pop_size: int = 20, num_offspring: int = 10, max_order: int = 0, extra_operators: list = None, sympy_conversions: dict = None, seeds: list = None, seed: int = 0)

Use Genetic Programming (GP) to infer the regression equation from the data.

Parameters:

df – The data to use.
ngen – The maximum number of GP generations to run for.
pop_size – The GP population size.
num_offspring – The number of offspring per generation.
max_order – The maximum polynomial order to use, e.g. max_order=2 will give polynomials of the form ax^2 + bx + c.
extra_operators – Additional operators for the GP (defaults are +, *, log(x), and 1/x). Operations should be of the form (fun, numArgs), e.g. (add, 2).
sympy_conversions – Dictionary of conversions of extra_operators for sympy, e.g. "mul": lambda \*args_: "Mul({},{})".format(\*args_).
seeds – Seed individuals for the population (e.g. if you think that the relationship between X and Y is probably logarithmic, you can put that in).
seed – Random seed for the GP.

LogisticRegressionEstimator

Recommended use: For binary outcomes (yes/no, true/false, success/failure).

class causal_testing.estimation.logistic_regression_estimator.LogisticRegressionEstimator(treatment_variable: str, outcome_variable: str, control_value: float = None, treatment_value: float = None, adjustment_set: set[str] = None, adjustment_config: dict[str, Any] = None, formula: str = None, alpha: float = 0.05)

Bases: RegressionEstimator

A Logistic Regression Estimator is a parametric estimator which restricts the variables in the data to a linear combination of parameters and functions of the variables (note these functions need not be linear). It is designed for estimating categorical outcomes.

add_modelling_assumptions(): Add modelling assumptions to the estimator. This is a list of strings which list the modelling assumptions that must hold if the resulting causal inference is to be considered valid.

estimate_unit_odds_ratio(df: DataFrame) → EffectEstimate

Estimate the odds ratio of increasing the treatment by one. In logistic regression, this corresponds to the coefficient of the treatment of interest.

Parameters:: df – The data to use.
Returns:: The odds ratio with confidence intervals.

MultinomialRegressionEstimator

Recommended use: For categorical outcomes (e.g. colurs: Red, Green, Blue).

class causal_testing.estimation.multinomial_regression_estimator.MultinomialRegressionEstimator(treatment_variable: str, outcome_variable: str, control_value: float = None, treatment_value: float = None, adjustment_set: set[str] = None, adjustment_config: dict[str, Any] = None, formula: str = None, alpha: float = 0.05)

Bases: RegressionEstimator

A Logistic Regression Estimator is a parametric estimator which restricts the variables in the data to a linear combination of parameters and functions of the variables (note these functions need not be linear). It is designed for estimating categorical outcomes.

add_modelling_assumptions(): Add modelling assumptions to the estimator. This is a list of strings which list the modelling assumptions that must hold if the resulting causal inference is to be considered valid.

estimate_unit_odds_ratio(df: DataFrame) → EffectEstimate

Estimate the odds ratio of increasing the treatment by one. In logistic regression, this corresponds to the coefficient of the treatment of interest.

Parameters:: df – The data to use.
Returns:: The odds ratio with confidence intervals.

InstrumentalVariableEstimator

Recommended use: When dealing with unmeasured confounding using instrumental variables.

class causal_testing.estimation.instrumental_variable_estimator.InstrumentalVariableEstimator(outcome_variable: str, treatment_variable: str, instrument: str, alpha: float = 0.05, bootstrap_size=100)

Bases: Estimator

Carry out estimation using instrumental variable adjustment rather than conventional adjustment. This means we do not need to observe all confounders in order to adjust for them. A key assumption here is linearity.

add_modelling_assumptions(): Add modelling assumptions to the estimator. This is a list of strings which list the modelling assumptions that must hold if the resulting causal inference is to be considered valid.

estimate_coefficient(df: DataFrame) → EffectEstimate

Estimate the unit ate (i.e. coefficient) of the treatment on the outcome.

Parameters:: df – The data to use.

iv_coefficient(df) → float: Estimate the linear regression coefficient of the treatment on the outcome.

to_dict() → dict

Convert the estimator to a python dictionary for easy serialisation as JSON or CSV.

Returns:: A JSON serialisable dict representing the estimator.

IPCWEstimator

Recommended use: For handling missing data or selection bias using inverse probability of censoring weighting (e.g. time-varying data).

class causal_testing.estimation.ipcw_estimator.IPCWEstimator(timesteps_per_observation: int, control_strategy: list[tuple[int, str, Any]], treatment_strategy: list[tuple[int, str, Any]], outcome_variable: str, status_column: str, fit_bl_switch_formula: str, fit_bltd_switch_formula: str, eligibility=None, alpha: float = 0.05, total_time: float = None)

Bases: Estimator

Class to perform Inverse Probability of Censoring Weighting (IPCW) estimation for sequences of treatments over time-varying data.

Param:: df: Input DataFrame containing time-varying data.
Param:: timesteps_per_observation: Number of timesteps per observation.
Param:: control_strategy: The control strategy, with entries of the form (timestep, variable, value).
Param:: treatment_strategy: The treatment strategy, with entries of the form (timestep, variable, value).
Param:: outcome: Name of the outcome column in the DataFrame.
Param:: status_column: Name of the status column in the DataFrame, which should be True for operating normally, False for a fault.
Param:: fit_bl_switch_formula: Formula for fitting the baseline switch model.
Param:: fit_bltd_switch_formula: Formula for fitting the baseline time-dependent switch model.
Param:: eligibility: Function to determine eligibility for treatment. Defaults to None for “always eligible”.
Param:: alpha: Significance level for hypothesis testing. Defaults to 0.05.
Param:: total_time: Total time for the analysis. Defaults to one plus the length of of the strategy (control or treatment) with the most elements multiplied by timesteps_per_observation.

add_modelling_assumptions(): Add modelling assumptions to the estimator. This is a list of strings which list the modelling assumptions that must hold if the resulting causal inference is to be considered valid.

estimate_hazard_ratio(df: DataFrame) → EffectEstimate: Estimate the hazard ratio. :param df: The data to use.

preprocess_data(df: DataFrame) → DataFrame

Set up the treatment-specific columns in the data that are needed to estimate the hazard ratio.

Parameters:: df – The data to use.
Returns:: The preprocessed DataFrame.

setup_fault_t_do(individual: DataFrame)

Return a binary sequence with each bit representing whether the current index is the time point at which the event of interest (i.e. a fault) occurred.

N.B. This is rounded _up_ to the nearest multiple of self.timesteps_per_observation. That is, if the fault occurs at time 22, and self.timesteps_per_observation == 5, then fault_t_do will be 25.

setup_fault_time(individual: DataFrame, perturbation: float = -0.001): Return the time at which the event of interest (i.e. a fault) occurred.

setup_xo_t_do(individual: DataFrame, strategy_assigned: list)

Return a binary sequence with each bit representing whether the current index is the time point at which the individual diverted from the assigned treatment strategy (and thus should be censored).

Parameters:

individual – DataFrame representing the individual.
strategy_assigned – The assigned treatment strategy.

ExperimentalEstimator

Recommended use: For randomised controlled trials or experimental data where treatment assignment is randomised.: Directly runs the system under test multiple times with different configurations (e.g. you need to collect new data by executing your system multiple times).

class causal_testing.estimation.experimental_estimator.ExperimentalEstimator(treatment_variable: str, outcome_variable: str, treatment_value: float, control_value: float, adjustment_config: dict[str, Any], alpha: float = 0.05, repeats: int = 200)

Bases: Estimator

A Logistic Regression Estimator is a parametric estimator which restricts the variables in the data to a linear combination of parameters and functions of the variables (note these functions need not be linear). It is designed for estimating categorical outcomes.

add_modelling_assumptions(): Add modelling assumptions to the estimator. This is a list of strings which list the modelling assumptions that must hold if the resulting causal inference is to be considered valid.

estimate_ate() → EffectEstimate

Estimate the average treatment effect of the treatment on the outcome. That is, the change in outcome caused by changing the treatment variable from the control value to the treatment value.

Returns:: The average treatment effect and the bootstrapped confidence intervals.

estimate_risk_ratio() → tuple[Series, list[Series, Series]]

Estimate the risk ratio of the treatment on the outcome. That is, the change in outcome caused by changing the treatment variable from the control value to the treatment value.

Returns:: The average treatment effect and the bootstrapped confidence intervals.

abstract run_system(configuration: dict) → dict: Runs the system under test with the supplied configuration and supplies the outputs as a dict. :param configuration: The run configuration arguments. :returns: The resulting output as a dict.

Custom Estimators

If the above estimators are not sufficient for your needs, you can implement your own custom estimator by extending the Estimator class and implementing the abstract add_modelling_assumptions method and the estimation method for the causal effect measure you wish to calculate. For example, if you wished to estimate the ATE using the empirical mean of the recorded outcome under the control and treatment values, you would need to implement a method called estimate_ate. If you wished to estimate the risk ratio, you would need to call your method estimate_risk_ratio. The code for the EmpiricalMeanEstimator is shown below.

import pandas as pd
from causal_testing.estimation.abstract_estimator import Estimator
from causal_testing.estimation.effect_estimate import EffectEstimate

class EmpiricalMeanEstimator(Estimator):
    """
    Custom estimator class to estimate the causal effect based on the empirical mean.
    """

    def add_modelling_assumptions(self):
        """
        Add modelling assumptions to the estimator. This is a list of strings which list the modelling assumptions that
        must hold if the resulting causal inference is to be considered valid.
        """
        self.modelling_assumptions += "The data must contain runs with the exact configuration of interest."

    def estimate_ate(self, df: pd.DataFrame) -> EffectEstimate:
        """Estimate the outcomes under control and treatment.
        :param df: The data to use.
        :return: The empirical average treatment effect.
        """

        control_results = df.where(df[self.treatment_variable] == self.control_value)[self.outcome_variable].dropna()
        treatment_results = df.where(df[self.treatment_variable] == self.treatment_value)[
            self.outcome_variable
        ].dropna()

        def ate(sample1, sample2):
            return sample1.mean() - sample2.mean()

        bootstraps = bootstrap((treatment_results, control_results), ate, confidence_level=self.alpha)
        return EffectEstimate(
            type="ate",
            value=ate(treatment_results, control_results),
            ci_low=bootstraps.confidence_interval.low,
            ci_high=bootstraps.confidence_interval.high,
        )

Once you have implemented your estimator, you will need to register it as an extra entry point in your project’s pyproject.toml file so that the Causal Testing Framework can find it. For example, if you had defined your EmpiricalMeanEstimator class in a module called empirical_mean_estimator in a folder called custom_estimators, you would register it as follows. You will also need to reinstall your project, e.g. with pip install -e . each time you add a new estimator to your pyproject.toml. You do not need to reinstall each time you edit your project for source code edits.

[project.entry-points."estimators"]
CustomFlakefighter = "custom_estimators.empirical_mean_estimator:EmpiricalMeanEstimator"

Of course, for this to work, your module needs to be discoverable on your python path. That is, you should be able to execute from custom_estimators.empirical_mean_estimator import EmpiricalMeanEstimator successfully from within the current working directory.

You can also add your custom estimator to causal test cases specified in JSON. To do so, you can simply set the estimator property to the name of your estimator class and the estimate_type property to the name of your causal effect measure. In the above EmpiricalMeanEstimator example, estimator would be set to "EmpiricalMeanEstimator" and estimate_type would be set to "ate".