The model is not the whole story: variability, memory and forecastability

The same algorithm can produce different errors because the signal, horizon and available information differ. I therefore start with the phenomenon to be forecast. Describing its variability and memory helps choose a reference, interpret an error and decide whether additional complexity brings a useful gain.

Explore the series · Connect the three studies · Understand no free lunch · Papers and teams

1. Describe the signal

Separate astronomical cycles, departure from clear sky and changes between successive times.

2. Examine memory

Find which dependencies remain useful at the decision horizon.

3. Evaluate the model

Compare on identical test targets, with an explicit reference and consistent units.

A laboratory separating memory, noise and model choice

Rather than only shuffling a curve, generate a two-state signal with known memory. Add additive noise, multiplicative noise or both. Ask what the regression learns, how its coefficients change and what expected error would remain even with knowledge of the generator’s current state.

Main experiment: model, assumptions and equations

St is −1 or +1 with switching probability q = (1−r)/2, initialized in its equiprobable stationary distribution. The latent signal is Kt = μ + aSt, μ = 0.6, with exact autocorrelation rh when a > 0. Observations are Yt = Kt(1 + βZm,t) + σaZa,t. The Z draws are independent of each other, across times and of the state, uniform on [−√3,+√3], with zero mean and unit variance. Multiplicative noise has conditional variance β²K², additive noise σa². Control ranges guarantee positive observations without clipping. This teaching generator is not a validated weather model.

The oracle knows the state at forecast origin: ŷoracle,t+h = μ + arhSt. Its exact expected squared risk is Rh = a²(1−r2h) + σa² + β²(μ²+a²), separating future transitions, additive noise and multiplicative noise. √Rh is an expected RMSE, not a threshold every finite sample must respect. The oracle has privileged information; it is not an operational forecast. This optimistic bound is specific to the generator, neither a numerical result of the cited papers nor a physical solar limit.

The h-step regression fits ŷ = A + BYt on training pairs only. Its stationary population-optimal linear slope is B* = a²rh / [a² + σa² + β²(μ²+a²)]. Observation noise thus reduces memory accessible to a simple regression. Displayed A and B are fitted, not substituted from this formula. Training uses indices 0–599; common test targets 624–1199 follow a 24-step buffer. Every forecast origin is exactly h steps before its target and after training. Models see only Y. sCV/F are retrospective diagnostics using the whole series, Iclr = 1, d = 0.2 and conventional T = 24 for F.

MAE is mean absolute error; RMSE weights large errors more strongly. nRMSE = 100 × RMSE / mean test observation. NICE¹ and NICE² divide model MAE and RMSE by same-horizon simple-persistence errors on identical targets. NICEΣ equally averages the L¹, L² and L³ ratios, following the reference NICE code. Ratios are undefined for a perfect reference. NICE method and paper.

Three complementary studies, three questions

2015, Can we diagnose difficulty before choosing the forecaster?

In 2015, I studied this question with Ted Soubdhan, Philippe Lauret, Mathieu David and Marc Muselli. Our paper examines statistical parameters for anticipating solar forecast error. Mean absolute log return (MALR), borrowed from econometrics, describes relative changes. This empirical diagnostic does not replace testing the model at the intended site and horizon. Paper in Energy (2015).

2025, Estimate expected error from variability

With Philippe Lauret, Mathieu David, Josselin Le Gal La Salle, Elke Lorenz and Richard Perez, I studied how hourly variability σ(Δkc) relates to RMSE, using regressions calibrated on 60 sites for intra-day and day-ahead horizons. Philippe Lauret is first author; I am a co-author. The 95% intervals are statistical, not universal physical limits. Paper in Solar Energy (2025).

sCV and F, Clear-sky departure versus temporal memory

sCV normalizes the quadratic departure from clear sky. It does not by itself describe temporal ordering. F incorporates residual dependencies at lags compatible with the horizon. The same values and sCV can therefore coexist with different temporal structures and reference-model performance. Renewable Energy paper (2026) · sCV/F calculator · Reference code.

Appendix: identical values, different temporal order, including sCV/F equations

An experiment separating amplitude from temporal order

We construct a positive series with an artificial period of 24 and shuffle a copy of its values. Its histogram is unchanged. Two simple rules are compared: copy the value available at forecast origin, or use the mean estimated on the first half. The aim is to understand these rules and assumptions. These are synthetic values, not solar measurements or a new benchmark of the papers.

The equations behind the panels

The reference is constant here: Iclr = 1 and k = I/Iclr. This isolates temporal ordering; a real application needs the Sun’s cycle too. With Δkt = kt − kt−1, MALR = mean(|ln(kt/kt−1)|). All values here are positive; real-data treatment of zeros and nights must be explicit. Both σ(Δk) and MALR depend on temporal neighbours and change when values are shuffled.

In this experiment, sCV = √2/(1−d) × √mean[(k−1)²], with d = 0.2. The general formula normalizes by root-mean-square clear-sky irradiance. For residuals ε = I − Iclr, F(h) = 1 + sCV[ρmax(h) − 1], where ρmax is the maximum |ρ(ℓ)| for h ≤ ℓ ≤ T/2, here T = 24. Correlations use available pairs, as in the reference MATLAB code. The sCV normalization at 1 is a reference, not a ceiling to impose on every series.

For each test target t: ŷpers(t) = k(t−h) and ŷmean(t) = mean(k0:95). Both RMSEs use the same 72 targets, indices 120 to 191. After training on indices 0 to 95, a 24-step buffer ensures that the mean is fitted before every forecast origin, including horizon 12. The mean stays fixed; future targets do not re-estimate it. Conversely, sCV, F, MALR and σ(Δk) describe the entire constructed series retrospectively. Those are distinct uses of the data.

No free lunch: problem structure matters

Wolpert and Macready’s optimization no-free-lunch results establish equal average performance for the search algorithms considered when uniformly averaging over all functions on a finite problem. That precise setting is not the distribution of real solar forecasting tasks. It does not establish that all models are equivalent within a given domain or that simple models always win. Original reference (1997).

The connection is methodological: model assumptions must meet problem structure. Shuffling an example is not a proof of the theorem. It illustrates how the reference choice and horizon change the interpretation of error. At horizon 12, anticorrelation may carry useful information while persistence, which copies without reversing, becomes inappropriate.

The message for a real decision

Understand the signal, establish a reference, measure improvement, then add complexity when the gain justifies it. Low error under an easy sky has a different meaning from comparable error under unstable conditions. High variability alone does not establish an absence of predictable structure. Examine the horizon, information available at forecast origin, out-of-training validation and model cost.

This connects to the atmospheric neutron study: simulation quantifies what a mechanism can offer, within its limits. Continue with NICE and persistence as a reference, BLEND, solar cycles and timestamps and the research thread.

Papers and complete author teams

  1. Cyril Voyant, Milan Despotovic, Luis Garcia-Gutierrez, Rodrigo Amaro e Silva, Philippe Lauret, Ted Soubdhan, Nadjem Bailek. NICE k metrics: Unified and multidimensional framework for evaluating deterministic solar forecasting accuracy (2025).
  2. Cyril Voyant, Ted Soubdhan, Philippe Lauret, Mathieu David, Marc Muselli. Statistical parameters as a means to a priori assess the accuracy of solar forecasting models (2015).
  3. Philippe Lauret, Mathieu David, Josselin Le Gal La Salle, Elke Lorenz, Richard Perez, Cyril Voyant. A simple method for quickly estimating solar irradiance forecast errors (2025).
  4. Cyril Voyant, Alan Julien, Milan Despotovic, Gilles Notton, Luis Antonio Garcia-Gutierrez, Claudio Francesco Nicolosi, Philippe Blanc, Jamie Bright. Stochastic coefficient of variation: Assessing the variability and forecastability of solar irradiance (2026).
  5. David H. Wolpert, William G. Macready. No Free Lunch Theorems for Optimization. IEEE Transactions on Evolutionary Computation, 1(1), 67–82 (1997).