Hypothesis Test Calculator based on the asymptotic formulae for the profile likelihood ratio.
It performs hypothesis tests using the asymptotic formula for the profile likelihood, and uses the Asimov data set to compute expected significances or limits.
See G. Cowan, K. Cranmer, E. Gross and O. Vitells: Asymptotic formulae for likelihood- based tests of new physics. Eur. Phys. J., C71:1–19, 2011. It provides methods to perform hypothesis tests using the likelihood function, and computes the \(p\)-values for the null and the alternate hypothesis using the asymptotic formulae for the profile likelihood ratio described in the given paper.
The calculator provides methods to produce the Asimov dataset, i.e. a dataset generated where the observed values are equal to the expected ones. The Asimov data set is then used to compute the observed asymptotic \(p\)-value for the alternate hypothesis and the asymptotic expected \(p\)-values.
The asymptotic formulae are valid only for one POI (parameter of interest). So the calculator works only for one-dimensional (one POI) models. If more than one POI exists, only the first one is used.
The calculator can generate Asimov datasets from two kinds of PDFs:
The fits performed by the calculator can be steered with the global default minimizer options, e.g. via ROOT::Math::MinimizerOptions::SetDefaultStrategy() and ROOT::Math::MinimizerOptions::SetDefaultTolerance() (the tolerance is clamped to a minimum value of 1). The RooFitResult objects of the fits are retrievable after calling GetHypoTest() via GetFitResultUncondObs(), GetFitResultCondObs(), GetFitResultUncondAsimov() and GetFitResultCondAsimov(), so quantities like the minimizer status or the EDM at the minimum can be inspected, for example to cross-check a fit that did not converge. Note that when the calculator is driven by the HypoTestInverter, the stored conditional fit results correspond to the last scanned point.
Definition at line 30 of file AsymptoticCalculator.h.
Public Member Functions | |
| AsymptoticCalculator (RooAbsData &data, const ModelConfig &altModel, const ModelConfig &nullModel, bool nominalAsimov=false) | |
| constructor for asymptotic calculator from Data set and ModelConfig | |
| const ModelConfig * | GetAlternateModel (void) const |
| const RooArgSet & | GetBestFitParams () const |
| return best fit value for all parameters | |
| const RooArgSet & | GetBestFitPoi () const |
| return snapshot of the best fit parameter | |
| const RooAbsData * | GetData (void) const |
| virtual const RooArgSet * | GetFitInfo () const |
| const RooFitResult * | GetFitResultCondAsimov () const |
| Result of the conditional fit to the Asimov data set with the POI fixed to the tested value, from the last call to GetHypoTest(). | |
| const RooFitResult * | GetFitResultCondObs () const |
| Result of the conditional fit to the observed data with the POI fixed to the tested value, from the last call to GetHypoTest(). | |
| const RooFitResult * | GetFitResultUncondAsimov () const |
| Result of the fit to the Asimov data set with the POI fixed to the value of the alternate-model snapshot, performed by Initialize(). | |
| const RooFitResult * | GetFitResultUncondObs () const |
| Result of the unconditional fit to the observed data, performed by Initialize() (updated if GetHypoTest() finds a better minimum). | |
| HypoTestResult * | GetHypoTest () const override |
| re-implement HypoTest computation using the asymptotic | |
| const RooRealVar * | GetMuHat () const |
| return best fit parameter (firs of poi) | |
| const ModelConfig * | GetNullModel (void) const |
| TestStatSampler * | GetTestStatSampler (void) const |
| Returns instance of TestStatSampler. | |
| bool | Initialize () const |
| initialize the calculator by performing a global fit and make the Asimov data set | |
| TClass * | IsA () const override |
| bool | IsOneSidedDiscovery () const |
| bool | IsTwoSided () const |
| void | SetAlternateModel (const ModelConfig &altModel) override |
| Set the model for the alternate hypothesis. | |
| virtual void | SetCommonModel (const ModelConfig &model) |
| Set a common model for both the null and alternate, add to the workspace if not already there. | |
| void | SetData (RooAbsData &data) override |
| Set the DataSet. | |
| void | SetNullModel (const ModelConfig &nullModel) override |
| re-implementation of setters since they are needed to re-initialize the calculator | |
| void | SetOneSided (bool on) |
| set test statistic for one sided (upper limits) | |
| void | SetOneSidedDiscovery (bool on) |
| set the test statistics for one-sided discovery | |
| void | SetQTilde (bool on) |
| set using of qtilde, by default is controlled if RoORealVar is limited or not | |
| void | SetSigned (bool on=true) |
| use the signed (uncapped) profile likelihood test statistic. | |
| void | SetTwoSided () |
| set the test statistics for two sided (in case of upper limits for discovery does not make really sense) | |
| void | Streamer (TBuffer &) override |
| void | StreamerNVirtual (TBuffer &ClassDef_StreamerNVirtual_b) |
| void | UseSameAltToys () |
| Set this for re-using always the same toys for alternate hypothesis in case of calls at different null parameter points This is useful to get more stable bands when running the HypoTest inversion. | |
Static Public Member Functions | |
| static TClass * | Class () |
| static const char * | Class_Name () |
| static constexpr Version_t | Class_Version () |
| static const char * | DeclFileName () |
| static RooAbsData * | GenerateAsimovData (const RooAbsPdf &pdf, const RooArgSet &observables) |
| generate the asimov data for the observables (not the global ones) need to deal with the case of a sim pdf | |
| static double | GetExpectedPValues (double pnull, double palt, double nsigma, bool usecls, bool oneSided=true) |
| function given the null and the alt p value - return the expected one given the N - sigma value | |
| static RooAbsData * | MakeAsimovData (const ModelConfig &model, const RooArgSet &allParamValues, RooArgSet &globObs) |
| Make a nominal Asimov data set from a model. | |
| static RooAbsData * | MakeAsimovData (RooAbsData &data, const ModelConfig &model, const RooArgSet &poiValues, RooArgSet &globObs, const RooArgSet *genPoiValues=nullptr) |
| Make Asimov data. | |
| static void | SetPrintLevel (int level) |
| set print level (static function) | |
Protected Member Functions | |
| virtual int | CheckHook (void) const |
| virtual void | PostHook () const |
| virtual int | PreAltHook (RooArgSet *, double) const |
| virtual void | PreHook () const |
| virtual int | PreNullHook (RooArgSet *, double) const |
Protected Attributes | |
| const ModelConfig * | fAltModel |
| unsigned int | fAltToysSeed |
| const RooAbsData * | fData |
| TestStatSampler * | fDefaultSampler |
| TestStatistic * | fDefaultTestStat |
| const ModelConfig * | fNullModel |
| TestStatSampler * | fTestStatSampler |
Private Member Functions | |
| SamplingDistribution * | GenerateSamplingDistribution (ModelConfig *thisModel, double obsTestStat, RooAbsPdf *impDens=nullptr, const RooArgSet *impSnapshot=nullptr) const |
| void | SetAdaptiveLimits (double obsTestStat, bool forNull) const |
| void | SetupSampler (const ModelConfig &model) const |
| common setup for both models | |
Private Attributes | |
| RooAbsData * | fAsimovData |
| asimov data set | |
| RooArgSet | fAsimovGlobObs |
| snapshot of Asimov global observables | |
| RooArgSet | fBestFitParams |
| snapshot of all best fitted Parameter values | |
| RooArgSet | fBestFitPoi |
| snapshot of best fitted POI values | |
| std::unique_ptr< RooFitResult > | fFitResultCondAsimov |
| ! result of conditional fit to Asimov data | |
| std::unique_ptr< RooFitResult > | fFitResultCondObs |
| ! result of conditional fit to observed data | |
| std::unique_ptr< RooFitResult > | fFitResultUncondAsimov |
| ! result of fit to Asimov data at the alt POI | |
| std::unique_ptr< RooFitResult > | fFitResultUncondObs |
| ! result of unconditional fit to observed data | |
| bool | fIsInitialized |
| ! flag to check if calculator is initialized | |
| double | fNLLAsimov |
| double | fNLLObs |
| bool | fNominalAsimov |
| make Asimov at nominal parameter values | |
| bool | fOneSided |
| for one sided PL test statistic (upper limits) | |
| bool | fOneSidedDiscovery |
| for one sided PL test statistic (for discovery) | |
| bool | fSigned = false |
| use signed (uncapped) PL test statistic | |
| int | fUseQTilde |
| flag to indicate if using qtilde or not (-1 (default based on RooRealVar)), 0 false, 1 (true) | |
#include <RooStats/AsymptoticCalculator.h>
| AsymptoticCalculator::AsymptoticCalculator | ( | RooAbsData & | data, |
| const ModelConfig & | altModel, | ||
| const ModelConfig & | nullModel, | ||
| bool | nominalAsimov = false ) |
constructor for asymptotic calculator from Data set and ModelConfig
Definition at line 116 of file AsymptoticCalculator.cxx.
|
inlineprotectedvirtualinherited |
Reimplemented in RooStats::HybridCalculator.
Definition at line 73 of file HypoTestCalculatorGeneric.h.
Definition at line 151 of file AsymptoticCalculator.h.
Definition at line 151 of file AsymptoticCalculator.h.
|
static |
generate the asimov data for the observables (not the global ones) need to deal with the case of a sim pdf
| a RooAbsPdf. Can be also a RooSimultaneous, or a RooProPdf of a RooSimultaneous |
Definition at line 1242 of file AsymptoticCalculator.cxx.
|
privateinherited |
|
inlineinherited |
Definition at line 57 of file HypoTestCalculatorGeneric.h.
return best fit value for all parameters
Definition at line 104 of file AsymptoticCalculator.h.
return snapshot of the best fit parameter
Definition at line 100 of file AsymptoticCalculator.h.
|
inlineinherited |
Definition at line 52 of file HypoTestCalculatorGeneric.h.
|
static |
function given the null and the alt p value - return the expected one given the N - sigma value
Definition at line 880 of file AsymptoticCalculator.cxx.
|
inlinevirtualinherited |
Reimplemented in RooStats::FrequentistCalculator.
Definition at line 54 of file HypoTestCalculatorGeneric.h.
|
inline |
Result of the conditional fit to the Asimov data set with the POI fixed to the tested value, from the last call to GetHypoTest().
Returns nullptr if the fit was skipped or has not been run yet. The calculator keeps ownership of the returned object.
Definition at line 127 of file AsymptoticCalculator.h.
|
inline |
Result of the conditional fit to the observed data with the POI fixed to the tested value, from the last call to GetHypoTest().
Returns nullptr if the fit was skipped or has not been run yet. The calculator keeps ownership of the returned object.
Definition at line 115 of file AsymptoticCalculator.h.
|
inline |
Result of the fit to the Asimov data set with the POI fixed to the value of the alternate-model snapshot, performed by Initialize().
Since the Asimov data set is generated at that POI value, this corresponds to the unconditional minimum (updated if GetHypoTest() finds a better minimum). Returns nullptr if the fit was skipped or has not been run yet. The calculator keeps ownership of the returned object.
Definition at line 122 of file AsymptoticCalculator.h.
|
inline |
Result of the unconditional fit to the observed data, performed by Initialize() (updated if GetHypoTest() finds a better minimum).
Returns nullptr if the fit was skipped or has not been run yet. The calculator keeps ownership of the returned object.
Definition at line 110 of file AsymptoticCalculator.h.
|
overridevirtual |
re-implement HypoTest computation using the asymptotic
It performs an hypothesis tests using the likelihood function and computes the p values for the null and the alternate using the asymptotic formulae for the profile likelihood ratio.
See G. Cowan, K. Cranmer, E. Gross and O. Vitells. Asymptotic formulae for likelihood- based tests of new physics. Eur. Phys. J., C71:1–19, 2011. The formulae are valid only for one POI. If more than one POI exists consider as POI only the first one
Implements RooStats::HypoTestCalculator.
Definition at line 506 of file AsymptoticCalculator.cxx.
|
inline |
return best fit parameter (firs of poi)
Definition at line 102 of file AsymptoticCalculator.h.
|
inlineinherited |
Definition at line 53 of file HypoTestCalculatorGeneric.h.
|
inlineinherited |
Returns instance of TestStatSampler.
Use to change properties of TestStatSampler, e.g. GetTestStatSampler.SetTestSize(double size);
Definition at line 63 of file HypoTestCalculatorGeneric.h.
| bool AsymptoticCalculator::Initialize | ( | ) | const |
initialize the calculator by performing a global fit and make the Asimov data set
Initialize the calculator The initialization will perform a global fit of the model to the data and build an Asimov data set.
It will then also fit the model to the Asimov data set to find the likelihood value of the Asimov data set nominalAsimov is an option for using Asimov data set obtained using nominal nuisance parameter values By default the nuisance parameters are fitted to the data NOTE: If a fit has been done before, one for speeding up could set all the initial parameters to the fit value and in addition set the null snapshot to the best fit
Definition at line 154 of file AsymptoticCalculator.cxx.
|
inlineoverridevirtual |
Reimplemented from RooStats::HypoTestCalculator.
Definition at line 151 of file AsymptoticCalculator.h.
|
inline |
Definition at line 93 of file AsymptoticCalculator.h.
|
inline |
Definition at line 92 of file AsymptoticCalculator.h.
|
static |
Make a nominal Asimov data set from a model.
| model | ModelConfig that contains the model pdf and the model parameters | |
| allParamValues | The parameters of the model will be set to the values given in this set | |
| [out] | asimovGlobObs | Global observables set to values satisfying the constraints |
The parameter values (including the nuisance parameter) can result from a fit to data or be at the nominal values.
Definition at line 1447 of file AsymptoticCalculator.cxx.
|
static |
Make Asimov data.
Make the Asimov data from the ModelConfig and list of poi.
| realData | Real data | |
| model | Model config defining the pdf and the parameters | |
| paramValues | The snapshot of POI and parameters used for finding the best nuisance parameter values (conditioned at these values) | |
| [out] | asimovGlobObs | Global observables set to values satisfying the constraints |
| genPoiValues | Optional. A different set of POI values used for generating. By default the same POI are used for generating and for finding the nuisance parameters given an observed data set, a model and a snapshot of the poi. |
Definition at line 1337 of file AsymptoticCalculator.cxx.
|
inlineprotectedvirtualinherited |
Reimplemented in RooStats::FrequentistCalculator.
Definition at line 77 of file HypoTestCalculatorGeneric.h.
|
inlineprotectedvirtualinherited |
Reimplemented in RooStats::HybridCalculator, and RooStats::FrequentistCalculator.
Definition at line 75 of file HypoTestCalculatorGeneric.h.
|
inlineprotectedvirtualinherited |
Reimplemented in RooStats::FrequentistCalculator.
Definition at line 76 of file HypoTestCalculatorGeneric.h.
|
inlineprotectedvirtualinherited |
Reimplemented in RooStats::HybridCalculator, and RooStats::FrequentistCalculator.
Definition at line 74 of file HypoTestCalculatorGeneric.h.
|
privateinherited |
|
inlineoverridevirtual |
Set the model for the alternate hypothesis.
Implements RooStats::HypoTestCalculator.
Definition at line 82 of file AsymptoticCalculator.h.
|
inlinevirtualinherited |
Set a common model for both the null and alternate, add to the workspace if not already there.
Definition at line 70 of file HypoTestCalculator.h.
|
inlineoverridevirtual |
Set the DataSet.
Implements RooStats::HypoTestCalculator.
Definition at line 86 of file AsymptoticCalculator.h.
|
inlineoverridevirtual |
re-implementation of setters since they are needed to re-initialize the calculator
Implements RooStats::HypoTestCalculator.
Definition at line 78 of file AsymptoticCalculator.h.
|
inline |
set test statistic for one sided (upper limits)
Definition at line 61 of file AsymptoticCalculator.h.
|
inline |
set the test statistics for one-sided discovery
Definition at line 68 of file AsymptoticCalculator.h.
|
static |
set print level (static function)
Definition at line 109 of file AsymptoticCalculator.cxx.
|
inline |
set using of qtilde, by default is controlled if RoORealVar is limited or not
Definition at line 97 of file AsymptoticCalculator.h.
use the signed (uncapped) profile likelihood test statistic.
When using a one-sided test statistic, the value is not set to zero when the best fit value of the POI is beyond the tested value, but the sign of the test statistic is flipped instead (see also ProfileLikelihoodTestStat::SetSigned). It has no effect for the two-sided test statistics.
Definition at line 75 of file AsymptoticCalculator.h.
|
inline |
set the test statistics for two sided (in case of upper limits for discovery does not make really sense)
Definition at line 65 of file AsymptoticCalculator.h.
|
privateinherited |
common setup for both models
Definition at line 88 of file HypoTestCalculatorGeneric.cxx.
|
overridevirtual |
Reimplemented from RooStats::HypoTestCalculator.
|
inline |
Definition at line 151 of file AsymptoticCalculator.h.
|
inherited |
Set this for re-using always the same toys for alternate hypothesis in case of calls at different null parameter points This is useful to get more stable bands when running the HypoTest inversion.
to re-use same toys for alternate hypothesis
Definition at line 276 of file HypoTestCalculatorGeneric.cxx.
|
protectedinherited |
Definition at line 80 of file HypoTestCalculatorGeneric.h.
|
protectedinherited |
Definition at line 87 of file HypoTestCalculatorGeneric.h.
|
mutableprivate |
asimov data set
Definition at line 141 of file AsymptoticCalculator.h.
|
mutableprivate |
snapshot of Asimov global observables
Definition at line 142 of file AsymptoticCalculator.h.
|
mutableprivate |
snapshot of all best fitted Parameter values
Definition at line 144 of file AsymptoticCalculator.h.
|
mutableprivate |
snapshot of best fitted POI values
Definition at line 143 of file AsymptoticCalculator.h.
|
protectedinherited |
Definition at line 82 of file HypoTestCalculatorGeneric.h.
|
protectedinherited |
Definition at line 84 of file HypoTestCalculatorGeneric.h.
|
protectedinherited |
Definition at line 85 of file HypoTestCalculatorGeneric.h.
|
mutableprivate |
! result of conditional fit to Asimov data
Definition at line 149 of file AsymptoticCalculator.h.
|
mutableprivate |
! result of conditional fit to observed data
Definition at line 147 of file AsymptoticCalculator.h.
|
mutableprivate |
! result of fit to Asimov data at the alt POI
Definition at line 148 of file AsymptoticCalculator.h.
|
mutableprivate |
! result of unconditional fit to observed data
Definition at line 146 of file AsymptoticCalculator.h.
|
mutableprivate |
! flag to check if calculator is initialized
Definition at line 136 of file AsymptoticCalculator.h.
|
mutableprivate |
Definition at line 139 of file AsymptoticCalculator.h.
|
mutableprivate |
Definition at line 138 of file AsymptoticCalculator.h.
|
private |
make Asimov at nominal parameter values
Definition at line 135 of file AsymptoticCalculator.h.
|
protectedinherited |
Definition at line 81 of file HypoTestCalculatorGeneric.h.
|
private |
for one sided PL test statistic (upper limits)
Definition at line 132 of file AsymptoticCalculator.h.
|
mutableprivate |
for one sided PL test statistic (for discovery)
Definition at line 133 of file AsymptoticCalculator.h.
use signed (uncapped) PL test statistic
Definition at line 134 of file AsymptoticCalculator.h.
|
protectedinherited |
Definition at line 83 of file HypoTestCalculatorGeneric.h.
|
mutableprivate |
flag to indicate if using qtilde or not (-1 (default based on RooRealVar)), 0 false, 1 (true)
Definition at line 137 of file AsymptoticCalculator.h.