High Dimensional Statistics

When $p\gg n$ (more features than samples), classical statistics breaks down.

The Problem

In high dimensions:

• OLS doesn’t have a unique solution
• Sample covariance is singular
• Curse of dimensionality

Key Assumptions

Sparsity

Assume only $s\ll p$ coefficients are non-zero:

$\Vert {\beta }^{\ast }{\Vert }_0=s$

Restricted Eigenvalue Condition

The design matrix $X$ satisfies:

$\frac{\Vert X\delta {\Vert }_2^2}{n}\ge \kappa \Vert \delta {\Vert }_2^2-\tau \Vert \delta {\Vert }_1^2$

for some $\kappa >0$.

Main Results

LASSO achieves near-optimal rate:

$\Vert \hat{\beta }-{\beta }^{\ast }{\Vert }_2=O\left(\sqrt{\frac{s\log p}{n}}\right)$

Connections

• Regularization is the key tool
• Applications in Bioinformatics and Machine Learning