Compare forecasts: when is a more complex model worth it?
Five forecasts of a photovoltaic plant's output, from the simplest to the most elaborate, compared on the same targets. Choose the horizon and the metric: the curve shows where each model wins, and by how much.
Reading the comparison
- Persistence
- the last observed value; the minimal reference of any forecast.
- Cyclic persistence
- the value observed 24 hours earlier at the same time; it follows the daily cycle.
- BLEND
- a mix of the two persistences, with a weight learned for each time of day.
- AR
- a linear autoregression on the last 24 hours, estimated by least squares.
- ELM
- a neural network with one random hidden layer (Extreme Learning Machine), whose output weights alone are learned, by regularized least squares.
- NICE₁, NICE₂, NICE₃
- error norms divided by those of raw simple persistence (last observation, without a night mask) at the same horizon: NICE₁ = MAE/MAEₚ, NICE₂ = RMSE/RMSEₚ; NICE₃ uses the cube root of the mean cubed absolute error. Below 1 beats the reference; above 1 is worse.
- Daytime / day and night
- "daytime" counts only the times when the sun is up; predictions are set to zero at night. Including night changes the evaluated population and normalization; state the scope alongside results.
Protocol: public Palaiseau data (AC power, 30-minute steps, first two years); models trained on the first half, evaluated on the second; 24 hours of lags plus time of day and of year as inputs; forecasts set to zero at night. nRMSE and nMAE divide RMSE and MAE by the observed mean on the selected scope. The published PV_ELM_pred solvers are used, with lag scaling fitted on training rows only in this demonstration. The displayed persistence is night-masked, unlike the raw NICE reference: its day-and-night NICE may be below 1. One ELM setup (200 neurons, 8 draws, seed 42) is studied here, not a universal model ranking. A teaching calculation on public data, not the result of an article. For the uncertainty around a forecast, see complex-valued forecasting and the probabilistic animation.
Software and protocol
Martin Leiva, Xavier Silvani, Gilles Notton, Cyril Voyant, Alan Julien. PV_ELM_pred (Zenodo). Code and data : github.com/cyrilvoyant/PV_ELM_pred.