Chapter 3: correlation, linear regression and χ² tests

Covariance and Pearson correlation, the regression line and R², the significance of r, the χ² goodness-of-fit and independence tests, with exercises.

Slides (69, in French) · PDF in French (5.1 MB) · Questions

The chapter in 10 points

Key point. With 1000 pairs, r = 0.1 can be significant while explaining only 1% of the variation: significant does not mean useful.

  1. The scatter plot shows the shape of the link; covariance cov(x, y) = Σ(xi − x̄)(yi − ȳ)/(n − 1).
  2. Pearson correlation: r = cov(x, y)/(sx sy), where sx and sy are the standard deviations, between −1 and 1, to be interpreted with care (non-linear link, outliers, causation).
  3. Least-squares line: b1 = cov(x, y)/sx², b0 = ȳ − b1 x̄, slope b1 and intercept b0; it is used to predict y from x.
  4. In simple least-squares linear regression with an intercept, R² = r² measures the share of the variation of y explained in the sample; this equality does not carry over to every model.
  5. Under H0: ρ = 0, where ρ is the correlation in the population, for independent pairs from a bivariate normal distribution, t = r√(n − 2)/√(1 − r²) follows a Student distribution with n − 2 df. A non-linear link can escape Pearson's r.
  6. Residuals, tests on the slope and on the intercept, and the 95% prediction interval around a predicted value.
  7. Qualitative variables: χ² = Σ (O − E)²/E, where O are the observed counts and E the expected counts.
  8. χ² goodness of fit: k − 1 degrees of freedom (df) for k probabilities fixed in advance; estimating m parameters can bring the df down to k − 1 − m under the conditions of the test. Example from the course: 52.75 exceeds 9.488 for 4 df at the 5% level.
  9. χ² test of independence on a contingency table, with (R − 1)(C − 1) degrees of freedom for R rows and C columns: treatment and recovery, nationality and brand, sex and site of a cancer.
  10. The synthesis exercises chain comparison of means, correlation, regression line and prediction interval on data from smokers.

To check the conditions of the models: NIST: least squares and residual variance · NIST: confidence and prediction intervals · NIST: conditions of the χ² goodness-of-fit test.

To go on: the solar forecasting course, where regression is applied to a time series · how to evaluate a forecast.

Open this experiment on its own page

Significance versus explained variation

A correlation can be significant and explain very little. Vary the correlation r and the sample size, then compare the p-value with the coefficient of determination R², the share of variation explained by the line. Open this experiment on its own page.

χ²: compare observed and expected counts

Is a six-sided die fair? Are the two variables of a 2×2 table independent? See what each cell adds to χ², and the limits set by small counts. Open this experiment on its own page.

Regression: the mean or a future observation?

The mean of y is known better than a new observation. Compare the two bands, move x₀ (the value of x where you predict), see the effect of sample size and scatter, and recognize an extrapolation. Open this experiment on its own page.

These experiments extend the notions of the biostatistics course without adding a new method. Least squares and residual variance: NIST.

The slides (in French)

The 69 slides of this chapter are in French. Their text is written out and searchable in the French chapter, which shows one slide at a time (arrow keys, deep links). The summary, the experiments and the FAQ above are the English reading of the same chapter; formulas and numbers are identical.

Open the slides → · Download the chapter (PDF in French, 5.1 MB)

Outline of the chapter

  1. Introduction (slides 1–3, in French)
  2. I. Correlation (slides 4–15, in French)
  3. II. Linear regression (slides 16–24, in French)
  4. III. Significance (slides 25–38, in French)
  5. IV. χ² goodness of fit (slides 39–45, in French)
  6. V. χ² test of independence (slides 46–54, in French)
  7. VI. Exercises (slides 55–69, in French)

Going further: questions and answers

What is the difference between a confidence interval and a prediction interval?

In simple linear regression with independent normal errors of constant variance: 95% CI of the mean = ŷ0 ± t(n−2; 0.975) se √[1/n + (x0−x̄)²/Sxx]. For a future observation, add 1 under the root. Here Sxx = Σ(xi−x̄)² and se² = Σresidual²/(n−2). The prediction interval therefore keeps the individual scatter; 1.96 is only an asymptotic approximation of the quantile.

Is a significant r enough to make a prediction useful?

No: with 1000 pairs, r = 0.1 can be significant while explaining only 1% of the variation in a simple regression with an intercept. Rejecting ρ = 0 under a model proves neither causation nor predictive usefulness; the latter is also measured on data not used to fit the model.

Why can a single outlier change everything?

Least squares weigh the square of the deviations: a point far away in x has leverage that pulls the slope and inflates or creates r. Always look at the scatter plot and the residuals before concluding.

What to do when the expected counts of a χ² are small?

The asymptotic approximation can be poor; examine the size and structure of the table. A justified grouping or an exact method may be suitable, notably Fisher's test for a 2×2 table in its setting. Yates' continuity correction does not guarantee to solve the small-count problem.

← Chapter 2 Course contents

These slides are not a handout: they accompany the lectures and cannot replace them (disclaimer of the course). Teaching material, not clinical advice. The slides and the PDF are in French. License CC BY 4.0: free reuse, including modification and commercial use, with credit to the author (Cyril Voyant).