---

execute: echo: true warning: false—

Covariance and Linear Estimation

Understanding Covariance Matrices

The covariance matrix captures the relationship between random variables. For two variables:

\(\mathbf{K} = \begin{bmatrix} \text{Var}(X) & \text{Cov}(X,Y) \\ \text{Cov}(X,Y) & \text{Var}(Y) \end{bmatrix}\)

Consider this visualization:

Key observations:

• Diagonal elements represent individual variances
• Off-diagonal elements capture linear relationships
• Symmetric by definition
• Positive semi-definite (all eigenvalues ≥ 0)

Eigenanalysis of Covariance

The eigendecomposition reveals principal directions of variation:

Linear MMSE Estimation

Linear estimation finds the best linear approximation:

Principal Component Analysis

Geometric Interpretation

PCA finds orthogonal directions of maximum variance:

Dimensionality Reduction

PCA allows optimal linear dimensionality reduction:

Polynomial Regression

Design Matrix Construction

The design matrix structure affects numerical stability:

Regularization Effects

L2 regularization stabilizes the solution:

Bias-Variance Tradeoff

Model complexity affects both bias and variance: