Enhance stat vs FAR plots on the offline analysis summary page to include all event types and singles fit lines

[tdent]

At present the stat/FAR plots appear separately for each coinc or single event type, but that does not allow us to easily compare how the coinc backgrounds line up against each other, or against the singles fits / extrapolations. There is a nice multi-event-type FAR-stat plot in the methods paper https://arxiv.org/abs/2203.08545 which does so. Extend the SNRIFAR plot code to allow for this and make it part of the standard plotting workflow.

Specifically, this would allow us to diagnose possible cases where the FAR at high stat values might be dominated by singles fit extrapolation (which would hurt sensitivity at a given combined FAR for coincs) - or indeed unduly dominated by any event type.

[GarethCabournDavies]

For whoever gets to this, the code on the Caltech LVK cluster at /home/gareth.cabourndavies/paper_codes/pycbc/singles_search/plots/stat_vs_far/plot_ifar_vs_stat.py should be a good starting point (If you are looking at developing, and don't have access, I can just send it to you).

This was used to produce Figure 6 of https://arxiv.org/pdf/2203.08545, and the general codes / plots produced can be found here (LVK paywall)

[tdent]

This paper plot script used function calls like
_, ifar = significance.significance_trig_fit(
bg_ifar_nlouder, _ = significance.significance_n_louder(
which don't seem to exist in the main version of the significance module. What's the equivalent with current code?

[GarethCabournDavies]

This paper plot script used function calls like _, ifar = significance.significance_trig_fit( bg_ifar_nlouder, _ = significance.significance_n_louder( which don't seem to exist in the main version of the significance module. What's the equivalent with current code?

As the plotting uses a mix of both fitted and extrapolated IFAR estimates, we needed to use both the n_louder and trigger_fit options.

If we do that, we will want to use significance.count_n_louder and significance.n_louder_from_fit from the significance module, and then convert to far by dividing by background time - this is effectively what is done 'under the hood' in get_far

If we only care about plotting the functions actually used for the IFAR, we can use get_far directly. But I think it makes sense to plot the n_louder in the case that we are using a trigger fit, simply so that we can check that the fit looks good.

We may also want to include the IFAR limit information, as that is used by the statmap jobs - probably as a hline for each combination that uses a limit.

[tdent]

FWIW I've just used two calls to get_far above / below the fit threshold. I don't know if it makes sense to use this plot as a diagnostic of 'fit goodness' at the single ifo level as well - the main motivation is to compare the far vs stat between event types.

[GarethCabournDavies]

FWIW I've just used two calls to get_far above / below the fit threshold. I don't know if it makes sense to use this plot as a diagnostic of 'fit goodness' at the single ifo level as well - the main motivation is to compare the far vs stat between event types.

I thought that shouldn't be needed - it should fall back to n_louder below the fit threshold if using trigger_fit

[tdent]

FWIW I've just used two calls to get_far above / below the fit threshold. I don't know if it makes sense to use this plot as a diagnostic of 'fit goodness' at the single ifo level as well - the main motivation is to compare the far vs stat between event types.

I thought that shouldn't be needed - it should fall back to n_louder below the fit threshold if using trigger_fit

I haven't looked at the singles far code in that much detail, am just trying to do the minimal adaptation of the paper script. Let's consider the details in a PR.