How to evaluate a solar forecast against persistence

Compare the model with persistence on the same targets, horizon and scope, report more than one error norm, then test the difference. The ratio of the model's error to that of persistence says what the model adds. The NICEᵏ metrics (Voyant et al., 2025) build this ratio for norms of order one, two and three.

Cyril Voyant · Mines Paris – PSL, Centre O.I.E. · 2 November 2026

Reading the quantities. MAE is mean absolute error; RMSE is root mean squared error and gives large errors more weight. nRMSE normalizes RMSE by the mean observation on the evaluated data. AR is autoregression; ELM (extreme learning machine) fits output weights while keeping a fixed random hidden layer. In the NICE formulas, the subscript p denotes persistence; the statistical p-value reported by a test is a different quantity. sCV describes signal variability and F combines variability with temporal dependence.

The steps

1. Name the reference. Say which persistence is used: last observation, clear-sky-index (smart) persistence, or a phase-aware reference such as BLEND. A skill score means little without this.
2. Use the same targets. Same site, period, horizon and forecast times for the model and the reference.
3. State the scope. Daytime only, or day and night. Including night changes the evaluated population and the normalization.
4. Report several norms. MAE and RMSE do not weight large errors alike. Keep absolute errors next to relative scores.
5. Test the difference. Compare error distributions across stations or periods, for instance with a Wilcoxon test.
6. Describe the signal. A large error does not mean the same thing under a regular sky and under strongly fluctuating irradiance. sCV and F characterize variability and forecastability before algorithms are compared.

NICE, a worked case

NICE¹ = MAE / MAE_p and NICE² = RMSE / RMSE_p, where p is simple persistence at the same horizon. NICE³ uses the cube root of the mean cubed absolute error. NICEΣ combines the three.

A useful predictor lies between 0 and 1. Above 1, the model does worse than the simplest predictor, or there is a numerical or other problem.

In the published study, at a thirty-minute horizon, an ELM and an autoregressive model share a rounded median nRMSE of 0.19. NICEΣ gives 0.75 versus 0.77. On that protocol the test gives p < 0.001 for NICEΣ and p = 0.0591 for nRMSE. Validation combines Monte Carlo simulations and 68 SIAR stations in Spain.

Limits

These values concern that protocol and those distributions. They do not measure the economic value of an improvement, and they do not rank methods in general.

To read and to use

• The NICE study, explained · theory, code and runnable example
• Compare forecasts interactively: persistence, BLEND, AR and ELM on public data
• sCV and F: how variable and how forecastable is the signal
• Other references: Yang et al. (2020) on the verification of deterministic forecasts; Marquez and Coimbra on a metric relating forecast uncertainty to variability

References

1. Cyril Voyant, Milan Despotovic, Luis Garcia-Gutierrez, Rodrigo Amaro e Silva, Philippe Lauret, Ted Soubdhan, Nadjem Bailek. NICEᵏ metrics: Unified and multidimensional framework for evaluating deterministic solar forecasting accuracy. Sustainable Energy Technologies and Assessments 83, 104588 (2025).
2. Cyril Voyant, Alan Julien, Milan Despotovic, Gilles Notton, Luis Antonio Garcia-Gutierrez, Claudio Francesco Nicolosi, Philippe Blanc, Jamie Bright. Stochastic coefficient of variation: Assessing the variability and forecastability of solar irradiance. Renewable Energy 256, 123913 (2026).
3. Dazhi Yang, Stefano Alessandrini, Javier Antonanzas, Fernando Antonanzas-Torres et al.. Verification of deterministic solar forecasts. Solar Energy 210 (2020).
4. Ricardo Marquez, Carlos F. M. Coimbra. Proposed Metric for Evaluation of Solar Forecasting Models. Journal of Solar Energy Engineering 135.