Test Oracle

As in traditional testing, the oracle is a procedure used to determine whether the observed behaviour is actually correct. In causal testing, this represents checking that the causal effect estimated from the test data is what was expected. The Causal Testing Framework supports several causal effects by default. The most basic oracle procedure is to simply validate the presence or absence of a causal effect. This requires nothing more than the edges of the causal DAG. If you know the direction of a causal relationship (positive or negative), you can add a little more precision to your causal tests. If you know precisely what the causal effect should be, you can check for a particular value, within a specified tolerance.

SomeEffect

Recommended use: For validating the presence of a causal effect between two variables. For additive effect measures such as ATE, CATE, this involves checking that the confidence intervals associated with a causal effect estimate do not contain zero. For multiplicative effect measures such as risk ratio this involves checking that the confidence intervals associated with a causal effect estimate do not contain one.

class causal_testing.testing.causal_effect.SomeEffect(effect_type: str = 'direct')

Bases: CausalEffect

An extension of CausalEffect representing that the expected causal effect should not be zero.

apply(effect_estimate: EffectEstimate) → bool: Abstract apply method that should return a bool representing if the result meets the outcome :param effect_estimate: EffectEstimate to be checked :return: Bool that is true if outcome is met

NoEffect

Recommended use: For validating the absence of a causal effect between two variables. For additive effect measures such as ATE, CATE, this involves checking that the confidence intervals associated with a causal effect estimate contain zero. For multiplicative effect measures such as risk ratio this involves checking that the confidence intervals associated with a causal effect estimate contain one.

class causal_testing.testing.causal_effect.NoEffect(effect_type: str = 'direct', atol: float = 0, ctol: float = 0.0)

Bases: CausalEffect

An extension of CausalEffect representing that the expected causal effect should be zero. :param atol: Arithmetic tolerance. The test will pass if the absolute value of the causal effect is less than atol. :param ctol: Categorical tolerance. The test will pass if this proportion of categories pass.

apply(effect_estimate: EffectEstimate) → bool: Abstract apply method that should return a bool representing if the result meets the outcome :param effect_estimate: EffectEstimate to be checked :return: Bool that is true if outcome is met

to_dict()

Convert the expected effect to a python dictionary for easy serialisation as JSON.

Returns:: A JSON serialisable dict representing the expected effect.

Positive

Recommended use: For validating a positive causal effect. For additive effect measures such as ATE, CATE, this involves checking that the confidence intervals associated with a causal effect estimate are both above zero. For multiplicative effect measures such as risk ratio this involves checking that the confidence intervals associated with a causal effect estimate are both above one.

class causal_testing.testing.causal_effect.Positive(effect_type: str = 'direct')

Bases: SomeEffect

An extension of CausalEffect representing that the expected causal effect should be positive. Currently only single values are supported for the test value

apply(effect_estimate: EffectEstimate) → bool: Abstract apply method that should return a bool representing if the result meets the outcome :param effect_estimate: EffectEstimate to be checked :return: Bool that is true if outcome is met

Negative

Recommended use: For validating a negative causal effect. For additive effect measures such as ATE, CATE, this involves checking that the confidence intervals associated with a causal effect estimate are both below zero. For multiplicative effect measures such as risk ratio this involves checking that the confidence intervals associated with a causal effect estimate are both below one.

class causal_testing.testing.causal_effect.Negative(effect_type: str = 'direct')

Bases: SomeEffect

An extension of CausalEffect representing that the expected causal effect should be negative. Currently only single values are supported for the test value

apply(effect_estimate: EffectEstimate) → bool: Abstract apply method that should return a bool representing if the result meets the outcome :param effect_estimate: EffectEstimate to be checked :return: Bool that is true if outcome is met

ExactValue

Recommended use: For specifying a precise value for the expected causal effect. Here, you can also specify arithmetic tolerance, categorical tollerance (the minimum proportion of categories that must exhibit the expected effect for the test to pass), and confidence interval limits.

class causal_testing.testing.causal_effect.ExactValue(value: float, effect_type: str = 'direct', atol: float = 0, ci_low: float = None, ci_high: float = None)

Bases: CausalEffect

An extension of CausalEffect representing that the expected causal effect should be a specific value.

apply(effect_estimate: EffectEstimate) → bool: Abstract apply method that should return a bool representing if the result meets the outcome :param effect_estimate: EffectEstimate to be checked :return: Bool that is true if outcome is met

to_dict()

Convert the expected effect to a python dictionary for easy serialisation as JSON or CSV.

Returns:: A JSON serialisable dict representing the expected effect.

Custom Causal Effects

As with custom estimators, you can also implement your own custom causal effects if the options above are not sufficient for your needs. To do this, you can extend the CausalEffect class and implement your own apply method that takes an effect estimate and returns boolean True if the test should pass and False otherwise.