### Spatial Multi-Event Contingency Table

Frédéric Atger
Météo-France
frederic.atger@meteo.fr

The spatial multi-event contingency table methodology is well suited for verifying high resolution forecasts since it gives credit to forecasts that are "close" to the truth in some way but need not be exactly correct.

The performance of a set of deterministic forecasts is often represented by a simple 2 x 2 contingency table that represents the joint distribution of forecasts and observations for a specified event criterion or threshold (for example, rain exceeding 1 mm/h).

 Observed yes Observed no Forecast yes hits false alarms Forecast no misses correct negatives

Now, for the same observed event criterion, consider a range of K thresholds on the forecasts (for example, forecast rain exceeding 1 mm/h, 2 mm/h, 5 mm/hr, etc). These can be viewed as possible decision thresholds for taking action, such as issuing a warning. Instead of the contingency table having only a single event category it now contains multiple categories corresponding to the K forecast thresholds.

 Observed yes Observed no Forecast >= threshold1 hits1 false alarms1 Forecast < threshold1 misses1 correct negatives1 Forecast >= threshold2 hits2 false alarms2 Forecast < threshold2 misses2 correct negatives2 ... ... ... ... ... ... Forecast >= thresholdK hitsK false alarmsK Forecast < thresholdK missesK correct negativesK

By using multiple thresholds, a deterministic forecast system can be evaluated across a range of possible decision thresholds (instead of just one) using ROC and relative value. This enables a fairer comparison against ensemble prediction systems or other probabilistic forecasts.

For an ensemble prediction system with M members, for each forecast threshold k there are now M probability categories (at least 1 member >= thresholdk, at least 2 members >= thresholdk, etc.), yielding a total of KxM categories.

An alternative to multiple intensity thresholds is multiple "closeness" thresholds, for example, forecast event within 10 km of the location of interest, within 20 km, 30 km, etc. Forecasters conceptually interpret high resolution model output in this way. The verification results can therefore be used to assess the performance of high resolution forecasts where the exact spatial matching of forecast and observed events is difficult or unimportant.

Other forecast decision criteria are possible, depending on the application.

Decision criteria can be combined to produce multi-dimensional contingency tables. The spatial multi-category contingency table described by Atger (2001) is a good example. In the case below, the number of categories would be JxK for single-model forecasts, and JxKxM for ensemble prediction systems.

 Forecast within distance1 ... Forecast within distanceJ Observed yes Observed no ... ... Observed yes Observed no Forecast >= threshold1 hits11 false alarms11 ... ... hitsJ1 false alarmsJ1 Forecast < threshold1 misses11 correct negatives11 ... ... missesJ1 correct negativesJ1 Forecast >= threshold2 hits12 false alarms12 ... ... hitsJ2 false alarmsJ2 Forecast < threshold2 misses12 correct negatives12 ... ... missesJ2 correct negativesJ2 ... ... ... ... ... ... ... ... ... ... ... ... ... ... Forecast >= thresholdK hits1K false alarms1K ... ... hitsJK false alarmsJK Forecast < thresholdK misses1K correct negatives1K ... ... missesJK correct negativesJK

Reference:

Atger, F., 2001: Verification of intense precipitation forecasts from single models and ensemble prediction systems. Nonlin. Proc. Geophys., 8, 401-417. Click here to get the PDF (295 Kb).