Lecture 3 Preview Report: Covering Numbers, Dudley's Entropy Integral, and Pseudo-dimension

Topic: infinite real-valued function classes, covering numbers, Dudley's entropy integral, and pseudo-dimension Course: Theoretical Machine Learning Slides: Lecture 3, Fall 2026, Haipeng Luo

1. The problem this lecture solves

The previous lecture characterized learnability for finite function classes and binary classification. This lecture turns to regression:

Y = [- 1, 1], F \subseteq [- 1, 1]^{X} .

The central goal is still to control Rademacher complexity, because the generalization error of empirical risk minimization (ERM) is bounded through the following chain:

V^{iid} (F, n) \leq 2 P sup R^{iid} (ℓ (F)) \leq 2 G P sup R^{iid} (F),

where $G$ is the Lipschitz constant of the loss; for binary classification one can take $G = 1/2$ .

For finite classes, Massart's lemma applies directly: if function values lie in $[- C, C]$ , then

P sup R^{iid} (F) \leq C \frac{2 ln ∣ F ∣}{n} .

But in regression, $F$ is often an infinite class, and its sample projection

F ∣_{x_{1 : n}} = {(f (x_{1}), \dots, f (x_{n})) : f \in F} \subseteq [- 1, 1]^{n}

is usually still an infinite set, so $∣ F ∣$ cannot simply be plugged into the finite-class bound. The unifying strategy of this lecture is: first approximate the infinite set by a finite one, then apply Massart's lemma to the finite representative set.

2. First layer: cover the whole function class

2.1 Pointwise covering and covering numbers

A finite class $H \subseteq [- 1, 1]^{X}$ is a pointwise $α$ -cover of $F$ if for every $f \in F$ there exists $h \in H$ such that

∣ f (x) - h (x) ∣ \leq α, \forall x \in X .

The size of the smallest representative set is the pointwise covering number:

N (F, α) = min {∣ H ∣ : H is a pointwise α -cover} .

The larger $α$ is, the looser the approximation requirement, so $N (F, α)$ is non-increasing in $α$ .

2.2 Theorem 1: single-scale covering bound

R^{iid} (F) \leq α \geq 0 min (α + \frac{2 ln N ( F , α )}{n}) .

This bound expresses a basic trade-off:

Small $α$ : the approximation error is small, but more representative functions are needed, so the covering number grows;
Large $α$ : the representative set is smaller, but the discretization error grows.

The skeleton of the proof writes each function as "approximation error + representative":

f = (f - h_{f}) + h_{f} .

The first term is bounded by $α$ via the pointwise error guarantee; the second term is the Rademacher complexity of the finite class $H$ , controlled by Massart's lemma.

2.3 Example: linear functions

Consider

X = B_{q}^{d}, F = {f_{θ} (x) = ⟨ θ, x ⟩ : θ \in B_{p}^{d}}, \frac{1}{p} + \frac{1}{q} = 1.

By Hölder's inequality, $∣ f_{θ} (x) ∣ \leq 1$ .

When $p = \infty, q = 1$ , the parameter space is a hypercube with side length 2. Discretizing each coordinate uniformly gives

N (F, α) \leq (\frac{1}{α})^{d},

and for $n \geq d$ ,

R^{iid} (F) = O (\frac{d ln ( n / d )}{n}) .

For general $p$ , the slides give a more elegant volume argument. Take a maximal $α$ -packing of the parameter ball $B_{p}^{d}$ : points whose pairwise distances exceed $α$ . Maximality guarantees that the packing centers also form an $α$ -cover; placing a ball of radius $α /2$ around each center produces disjoint balls, all contained in $(1 + α /2) B_{p}^{d}$ . Comparing volumes gives

N (F, α) \leq (\frac{2}{α} + 1)^{d} .

Hence again

R^{iid} (F) = O (\frac{d ln ( n / d )}{n}) .

The general technique to master here is: a maximal packing automatically yields a cover, and a volume ratio then controls the packing size.

2.4 Why pointwise covering can be too strong

Let $F$ be the class of all non-decreasing functions from $R$ to $[- 1, 1]$ . For any $α < 1$ ,

N (F, α) = \infty.

The proof only needs the infinite subclass

f_{m} (x) = sign (x - m), m \in Z .

Two functions with different jump locations differ by 2 at some point, so no single representative can cover both of them with error below 1.

Yet this class is in fact learnable. So "approximating at every point of the entire input space" is too strong a requirement; the pointwise covering number is not a tight complexity characterization.

3. Second layer: cover only the sample projection

3.1 Sample-dependent $ℓ_{p}$ covering numbers

Fix inputs $x_{1 : n}$ . A set $V \subseteq [- 1, 1]^{n}$ is an $α$ -cover of the projection $F ∣_{x_{1 : n}}$ in the $ℓ_{p}$ sense if for every projected vector $f$ there exists $v \in V$ such that

∥ f - v ∥_{p} \leq n^{1/ p} α,

equivalently,

(\frac{1}{n} t = 1 \sum n ∣ f_{t} - v_{t} ∣^{p})^{1/ p} \leq α .

The corresponding minimal size is denoted $N_{p} (F ∣_{x_{1 : n}}, α)$ . Under this normalization,

N_{1} \leq N_{2} \leq \dots \leq N_{\infty} .

Projection covering is never worse than pointwise covering, because restricting a pointwise cover to the sample naturally gives an $ℓ_{\infty}$ cover:

N_{\infty} (F ∣_{x_{1 : n}}, α) \leq N (F, α) .

3.2 Theorem 2: projection covering bound

The conditional Rademacher complexity satisfies

R^{iid} (F; x_{1 : n}) \leq α \geq 0 min (α + \frac{2 ln N _{1} ( F ∣ _{x_{1 : n}} , α )}{n}) .

The proof is again the decomposition "original vector = residual + finite representative." An $ℓ_{1}$ cover is used here because

⟨ ϵ, f - v_{f} ⟩ \leq ∥ f - v_{f} ∥_{1}

directly controls the approximation term by $n α$ .

3.3 The monotone class becomes finite again

Sort the sample as $x_{1} \leq \dots \leq x_{n}$ . Discretize the output range $[- 1, 1]$ at scale $2 α$ into a set $S$ with $∣ S ∣ \leq 1/ α$ . All non-decreasing sequences taking values in $S$ form an $ℓ_{\infty}$ cover of the projection.

Such a sequence is determined by how many times each discrete value appears. A crude count gives

N_{\infty} (F ∣_{x_{1 : n}}, α) \leq (n + 1)^{1/ α} .

Plugging into Theorem 2:

R^{iid} (F) \leq 0 \leq α \leq 1 min (α + \frac{2 ln ( n + 1 )}{α n}) = O ((\frac{ln n}{n})^{1/3}) .

The conclusion is crucial: a function class may admit no finite cover over the whole domain while its behavior on a finite sample does. For generalization analysis, the latter is exactly what symmetrization leaves us to control.

Also, the cover only appears in the analysis; the algorithm itself can still be plain ERM. For isotonic regression, ERM can even be computed efficiently.

4. Third layer: Dudley's entropy integral

4.1 Why upgrade again

Theorem 2 picks a single scale $α$ . For the monotone class it yields a rate of about $n^{- 1/3}$ , slower than the common $n^{- 1/2}$ . The problem is not that the class is genuinely harder to learn, but that single-scale analysis throws away information.

Dudley's entropy integral exploits all covering scales from coarse to fine simultaneously:

R^{iid} (F; x_{1 : n}) \leq 0 \leq α \leq 1 min (4 α + \frac{12}{n} \int_{α}^{1} ln N_{2} (F ∣_{x_{1 : n}}, δ) d δ) .

$ln N_{2}$ is often called the metric entropy, which is why this bound is called the entropy integral.

4.2 Improvements on the two examples

For the $d$ -dimensional linear class, using

N_{2} (F ∣_{x_{1 : n}}, δ) \leq (\frac{3}{δ})^{d}

and taking $α = 0$ gives

R^{iid} (F) = O (\frac{d}{n}) .

This removes the extra $ln n$ factor from the single-scale covering bound.

For the monotone class, using

N_{2} \leq N_{\infty} \leq (n + 1)^{1/ δ},

we get

\int_{α}^{1} ln N_{2} (δ) d δ \leq ln (n + 1) \int_{α}^{1} δ^{- 1/2} d δ \leq 2 ln (n + 1) .

Therefore

R^{iid} (F) = O (\frac{ln n}{n}) .

This improves the single-scale rate of about $n^{- 1/3}$ to about $n^{- 1/2}$ .

4.3 Proof intuition for chaining

Take geometrically decreasing scales

α_{j} = 2^{- j}, j = 1, \dots, M,

and pick an $ℓ_{2}$ cover $V_{j}$ at each scale. For each projected vector $f$ , select layer-by-layer representatives

v_{f}^{0}, v_{f}^{1}, \dots, v_{f}^{M} .

Then decompose telescopically:

f = (f - v_{f}^{M}) + j = 1 \sum M (v_{f}^{j} - v_{f}^{j - 1}) .

Here:

The tail term $f - v_{f}^{M}$ at the finest scale contributes at most $α_{M}$ ;
The candidate set of increments at each layer is finite, so Massart's lemma applies;
The two representatives of the same function at adjacent scales are both close to it, hence

∥ v_{f}^{j} - v_{f}^{j - 1} ∥_{2} \leq 3 n α_{j} .

Summing the per-layer contributions yields a discrete sum over scales, and monotonicity of the covering number in the scale bounds that sum by an integral.

The reason for using $ℓ_{2}$ covers is that the "radius" in Massart-type maximal inequalities is governed by the $ℓ_{2}$ norm of the vectors.

Chaining in one sentence: instead of approximating a function to its final accuracy in one shot, express it as a sequence of increasingly fine corrections, each of which is small.

5. Comparing the three bounds

Method	Information used	Linear class	Non-decreasing class
Pointwise function covering	Uniform approximation over the whole domain	$O (d ln (n / d) / n)$	$\infty$
Single-scale projection covering	One accuracy on a fixed sample	$O (d ln (n / d) / n)$	$O ((ln n / n)^{1/3})$
Dudley's entropy integral	All accuracies on a fixed sample	$O (d / n)$	$O (ln n / n)$

This table should serve as the core memory framework of the lecture:

Moving from "covering functions" to "covering projections" removes an unnecessary global requirement;
Moving from "single scale" to "all scales" removes further slack;
The analysis tools grow sharper, but the learning algorithm can remain ERM throughout.

6. Pseudo-dimension: binarizing real-valued functions

Covering numbers play the role in regression that the growth function plays in classification. In classification, VC dimension controls the growth function; correspondingly, a classical combinatorial parameter for real-valued classes is the pseudo-dimension.

For each $f : X \to [- 1, 1]$ , consider the binary classifier on $X \times [- 1, 1]$

h_{f} (x, y) = sign (f (x) - y) .

Define

Pdim (F) = VCdim ({h_{f} (x, y) = sign (f (x) - y) : f \in F}) .

Equivalently, $Pdim (F)$ is the largest integer $n$ for which there exist thresholded points $(x_{t}, y_{t})$ such that for every sign pattern $s_{t} \in {- 1, + 1}$ one can find $f \in F$ with

sign (f (x_{t}) - y_{t}) = s_{t}, t = 1, \dots, n .

Two examples:

For the $d$ -dimensional linear class, $Pdim (F) = d$ ;
The class of all non-decreasing functions has infinite pseudo-dimension.

Finite pseudo-dimension suffices for learnability. As the slides note, up to logarithmic factors, a Sauer-type result gives

ln N_{1} (F ∣_{x_{1 : n}}, α) \approx Pdim (F) ln \frac{1}{α},

which in turn implies

R^{iid} (F) = O (\frac{Pdim ( F ) ln n}{n}) .

But finite pseudo-dimension is not necessary for learnability: the monotone class is learnable yet has infinite pseudo-dimension. So pseudo-dimension is still not the correct "if and only if" complexity characterization. That is precisely the question left for the next lecture.

7. Common confusions

A cover is not a packing. A cover requires every point to be close to some center; a packing requires the centers to be pairwise separated. Proofs exploit that "a maximal packing is a cover."
Pointwise covering is not projection covering. The former approximates simultaneously at all $x \in X$ ; the latter only at the current sample points.
Do not reverse $N_{1} \leq N_{2} \leq N_{\infty}$ . This ordering is determined by the normalized empirical $ℓ_{p}$ norms used in the slides.
Covering numbers appear in proofs, not necessarily in algorithms. Do not assume ERM must explicitly construct a cover first.
Dudley does not change the function class; it changes the analysis scale. It turns a one-shot approximation into multi-layer increments.
"Finite parameter dimension" and "finite pseudo-dimension" are not necessary for all nonparametric problems. The monotone class is the counterexample.
The log covering number is the metric entropy. Dudley's integral contains $ln N_{2}$ , not the covering number itself.

8. Questions to bring to class

At which step exactly does symmetrization allow us to move from the whole $F$ to $F ∣_{x_{1 : n}}$ ?
In the proofs of Theorems 1 and 2, through which norms is the approximation error controlled?
Why is a maximal $α$ -packing necessarily an $α$ -cover?
Why can the size of the monotone-sequence cover be counted by "how many times each discrete value appears"?
Why does single-scale optimization yield $(ln n / n)^{1/3}$ for the monotone class while the integral method yields $ln n / n$ ?
In chaining, why connect representatives of adjacent layers instead of comparing every fine representative directly with the zero vector?
Why does the pseudo-dimension of the linear class equal exactly $d$ , while the monotone class has infinite pseudo-dimension?
If pseudo-dimension is not necessary, what more appropriate scale-sensitive combinatorial parameter will the next lecture introduce?

9. A 30-minute pre-lecture review order

First 5 minutes: review Rademacher complexity, Massart's lemma, symmetrization, and the Lecture 2 logic of controlling the growth function via VC dimension. Minutes 5-12: master the definition of pointwise $α$ -cover and the "residual + representative" proof of Theorem 1. Minutes 12-18: contrast the linear class with the monotone class to understand why pointwise covering fails. Minutes 18-23: learn the definition of projection covering, focusing on the $ℓ_{p}$ normalization and Theorem 2. Minutes 23-28: read Dudley's bound and the telescoping decomposition of chaining; grasp only the multi-scale intuition for now. Final 2 minutes: memorize the comparison table and restate "finite pseudo-dimension is sufficient but not necessary."

10. One-sentence summary

The point of this lecture is not to memorize three formulas, but to understand three successive refinements of complexity analysis: discretize the infinite class; discretize only what is visible on the sample; then use chaining to exploit all discretization scales at once.