← Part 1 of MSc in high dimensional statistics Why OLS breaks in high dimensions 2026-05-21 high-dimensional-statisticsolsridgemsc A core feature of my masters research project is the high-dimensional data setting. High-dimensional data is when we have alot more predictor variables than usual, but formally it means 𝑝≈𝑛or even 𝑝>𝑛, where 𝑝 is the number of predictors and 𝑛 is thenumber of your observations.At first I thought, when on earth would that ever happen? But in ourmodern world the data coming out of domains like genetics puts thisinto perspective. It is possible to have genome data representing 500patients each measured across 20,000 genes, i.e., 𝑝>𝑛.So why does ordinary least squares (OLS) fail in the high dimension?I think there are a couple of ways you can arrive at the answer. Mypreferred framing is the rank deficiency approach.The OLS estimatorConsider the OLS estimator in matrix form,̂𝛽=(𝑋⊤𝑋)−1𝑋⊤𝑌(1)where ̂𝛽 is our estimated parameters, X is our design matrix (the onewith the predictors) and 𝑌 is the response variable we are trying topredict.The headline act of this equation is (𝑋⊤𝑋)−1. The inverse of 𝑋⊤𝑋doesn’t exist when the matrix is not of full column rank, meaning itis rank deficient. If we cannot calculate the inverse then we cannotcompute ̂𝛽.MSE and eigenvaluesTo see how badly OLS degrades before it breaks entirely, we can lookat the mean squared error (MSE) through the eigenvalues of 𝑋⊤𝑋.The MSE can be decomposed as𝐸(‖̂𝛽−𝛽‖2)=𝜎2Trace((𝑋⊤𝑋)−1)(2)where the Trace is just the sum of the diagonal values. If 𝑋⊤𝑋 haseigenvalues 𝜆1≥𝜆2≥…≥𝜆𝑝>0, thenTrace((𝑋⊤𝑋)−1)=∑𝑝𝑗=11𝜆𝑗(3)The orthonormal caseGiraud (2022), who introduces his textbook with an overview of thecurse of dimensionality, discusses the case for linear regression. Heconsiders an orthonormal design, that being the ideal case where thepredictors are uncorrelated and standardised such that 𝑋⊤𝑋=𝐼𝑝.Equation 2 then reduces to𝐸(‖̂𝛽−𝛽‖2)=𝑝𝜎2(4)Here we can see that the MSE grows linearly with dimension 𝑝, soeach additional parameter contributes an additional 𝜎2 to the error,regardless of its contribution to the model.Ill-conditioning and correlated designReal data is usually not as simple. The Ridge regression paper byHoerl & Kennard (1970) shows that for a correlated design, 𝑋⊤𝑋becomes ill-conditioned (meaning some 𝜆𝑗 are small and thecorresponding 1/𝜆𝑗 terms in the trace dominate) causing the MSE tobecome significantly larger.This is commonly known as multicollinearity, and I want tohighlight this because high-dimensional settings are not necessarilymulticollinear, but they are both sensitive to the value of theeigenvalues. Multicollinear settings have small eigenvalues, whereashigh-dimensional settings have eigenvalues exactly equal to zero.The breaking pointThis all comes to a head as 𝑝≈𝑛. The estimation error grows with𝑝, and with more predictors, the likelihood of correlation amongthem increases.Finally, when 𝑝>𝑛, 𝑋 has at most 𝑛 linearly independent columnsbut needs 𝑝 for 𝑋⊤𝑋 to be invertible. It is rank-deficient, its inversedoes not exist, and there are infinitely many solutions to the OLSequation. We cannot recover the true model parameters.Regularisation to the rescueIn a multicollinearity setting, the MSE explodes because theeigenvalues are really small so the 𝑋⊤𝑋 matrix is near singular. Inthe high-dimensional setting, some eigenvalues are exactly zero andOLS breaks entirely.Regularisation methods like Ridge and Lasso which were originallydesigned for the former problem turn out to address both problems,and this is where the next post picks up.For any errors or feedback, feel free to reach out.ReferencesGiraud, C. (2022). Introduction to High-Dimensional Statistics (2nded.). CRC Press.Hoerl, A. E., & Kennard, R. W. (1970). Ridge Regression: BiasedEstimation for Nonorthogonal Problems. Technometrics, 12(1),55–67.