WGII: User’s needs and use of verification results

In order to provide efficient communication of  verification results to end users, the working group expresses the need to set a common language. From the presentations prepared for this workshop, a common set of definitions and notations will be proposed that should serve as common framework to communicate the verification results. The common framework should start from basic definitions and gradually introduce further complexity. Relations between the different quantities should be underlined, with multiple, equivalent definitions introduced when possible. Extra information brought by a given score should be detailed, as well as limitations (e.g. how the score can be “cheated” or biased).  The number of scores should be minimised in order to maximise their usability.(simple – more complex, etc…).  Information on the magnitude of errors as well as the frequency characteristics should be equally represented. Examples of when a given score should not be used (because they would provide misleading results) should also be given.

Partitions of the different measures and scores were felt particularly useful to help the interpretation of the verification data by the end user. Several examples could be found in the lectures given at this workshop, for example:
•	The partition of errors between displacement, amplitude and pattern (B. Ebert); it was suggested that information on the temporal mismatch should also be added;
•	The partition of  the Brier Score between resolution, reliability and uncertainty (L. Wilson); this is by many aspects similar to the partition of RMSE;
•	Although quantification of the partition of errors between model, observation and representativity errors are far from simple, research aiming at this type of partition (e.g. using methodologies developed for data assimilation) should be strongly supported.

Another important feature of the verification set is that confidence tests in the verification values themselves should be given to the users. This is very similar to the crucial importance of bringing information on the confidence (probability) together with the forecast as stressed by A. Murphy (1993).

The value/ decision making approach was recognised as a very important contribution to bringing the verification results into a form that users can easily understand. The limitations of the approach for a large set of practical decision making situations were however stressed.  These limitations are particularly severe for extreme events situations, when decisions have to be made without the support of a large sample of similar cases in the past on which a careful cost-loss analysis can be calibrated.  These prescriptive models should be accompanied by more descriptive models of  decision making.

It was agreed that the most important users of the verification results should be the forecasters. The delivery and feedback of verification results to the forecasters in a timely, comprehensive and clear manner on a consistent basis should be promoted. Forecasters are usually very interested in identifying and learning about key biases in the model forecasts that they can use in the process of adding value to the purely numerical forecast. Verification  results should also allow them to identify key biases in their own (final) forecasts, therefore additionally contributing  to the improvement of the quality of the end products. This however usually means a stratification of the verification in the form of conditional probabilities that cannot always provide unambiguous results due to the limited sample size and/or the weakness of the relation under investigation. It is therefore crucial that together with the conditional bias statistics, information on the statistical significance of the results be brought to the forecaster’s knowledge. More research on the forecast process and the effect of  interventions such as providing verification information should be encouraged.

Although the forecasters are a very important group of users, it was recognised that their view of the forecast quality might also have its biases with respect to an end user point of view. Their synoptic view of the forecast might very well fit with a verification using pattern matching techniques for example. A user might have a purely local point of view that will penalise phase errors much more heavily.

Training was identified as another important issue. It has been suggested that an interactive Web application be produced with WMO support. This should allow a characterisation of some typical sets of data and relate them to the different scores in a graphical as well as mathematical way. The training will explicitly include the implications of verification for different value systems. Both general users, administrators and forecasters should be targeted, maybe using different sets of interactive examples. A CD-ROM version of the software should also be made available.

It would also be very desirable in order to help the public make better use the forecast of quantitative precipitation if standard practices to report very simple verification measures (such as percentage of detection and of false alarms) could be promoted.