Lecture 4 Detailed Explanation

Based on Haipeng Luo's nine-page Lecture 4, Fall 2026. This report covers Propositions 1–2, Theorems 1–4, Lemmas 1–2, and both figures, explaining motivation, definitions, derivations, and limitations. The scale-sensitive entropy theorem and the full multiclass margin theorem are explicitly identified as external inputs; the remaining key steps are expanded. Added results and corrections are distinguished from the handout. Original: lecture4.pdf (local source, not published online). Companion: full handout reading edition. Prerequisite: Lecture 3. Samples are iid, classes nonempty, and the required measurability and integrability are assumed. Regression functions take values in $[-1,1]$; networks here have no biases and use coordinatewise ReLU. Statistical guarantees do not guarantee efficient optimization.

Reading Map: Two Questions Joined by Scale

The first question continues Lecture 3: how can a monotone class be learnable despite infinite pseudo-dimension? Counting arbitrarily small sign changes misses their amplitude. Fat-shattering counts distinctions at a fixed margin.

The second question concerns networks: parameter count need not describe complexity at useful scales. Weight norms yield covering bounds, spectral norms track propagation through layers, and margins compare classification confidence with output scale.

1. Arbitrarily Small Changes Versus Fixed-Scale Changes

Pseudo-dimension fixes inputs and thresholds, then asks for every sign pattern to be realizable:

It does not distinguish an upward displacement of $10^{-100}$ from one of 0.5. A cover at accuracy 0.01 has no reason to remember the former distinction. Fat-shattering adds a required amplitude:

Positive labels require outputs at least $y_t+\alpha /2$; negative labels require outputs at most $y_t-\alpha /2$. The two permitted regions are separated by $\alpha$.

The quantifier order matters: fix the points and witness first, vary the entire labeling second, and choose one function for that labeling last. Thresholds cannot depend on labels, and a different function cannot be chosen independently for each coordinate.

Larger scales impose stronger requirements:

For outputs in $[-1,1]$, no point is shattered if $\alpha >2$: two outputs realizing opposite labels would need to be more than 2 apart.

With compatible threshold conventions, pseudo-dimension is the supremum over positive-scale fat dimensions. Do not simply substitute $\alpha =0$ into the non-strict inequality: a function equal to every threshold could then appear to realize every labeling, which is not the intended definition.

2. Monotone Functions: Constructions Give Lower Bounds

2.1 Infinite pseudo-dimension

Take $x_t=t-1$, $y_t=(t-1)/n$, and prescribe

The worst adjacent labels are $+1,-1$, for which

Every pattern therefore extends to a nondecreasing step function on the real line, still in $[-1,1]$. Any $n$ works, but its margin shrinks with $n$.

Requiring $ϵ\ge \alpha /2$ restricts this construction to $n\le 1/\alpha$, proving only $\mathrm{fat}\ge \lfloor 1/\alpha \rfloor$. Failure of one construction is not an upper bound, because a different construction may exist.

2.2 Why adjacent pairs prove Proposition 1

Suppose some ordered points $x_1<\cdots <x_n$ are shattered with witness $y_1,\dots ,y_n$. Every adjacent $+1,-1$ pattern must be realizable, so

Hence adjacent thresholds differ by at least $\alpha$. Different adjacent pairs may use different functions, but they constrain the same witness. Summing gives

This rules out every possible witness and is therefore a genuine upper bound.

2.3 Added result: the exact value in this example

Realizing both labels at every point also forces

Thus $y_n-y_1\le 2-\alpha$, giving $n\alpha \le 2$. Conversely, whenever $n\alpha \le 2$, take

All outputs remain in range, and the smallest adjacent increment is zero. A monotone extension exists for every labeling. Therefore

This strengthens the handout's order bound. The constant depends on its convention that the required margin is $\alpha /2$, rather than $\alpha$.

3. From Fat Dimension to Dudley's Integral

For fixed inputs, define the normalized empirical metric and complexity by

A normalized radius $\alpha$ corresponds to Euclidean radius $\alpha \sqrt{n}$. Theorem 1 is a nontrivial scale-sensitive combinatorial result. A version with safe endpoint constants is

for absolute $C>1,c>0$. The handout does not prove this theorem, and ordinary Sauer counting alone is not a complete proof. We use it as an external input and derive the consequences. Writing $\ln (C/\delta )$ also avoids a vanishing logarithm near 1.

Dudley's bound from Lecture 3 is

For the monotone class, the integrand is bounded by a constant times $\sqrt{\ln (C/\delta )/\delta }$. Substitute $\delta =e^{-u}$:

Letting $a\downarrow 0$ gives Proposition 2, uniformly over samples. Taking expectation over samples preserves the $O(n^{-1/2})$ rate.

The handout uses the simpler estimate $\ln (C/\delta )\lesssim {\delta }^{-1/2}$. Its integrand is at most a constant times ${\delta }^{-3/4}$, and

The earlier $\sqrt{\ln n}$ factor disappears because this covering estimate no longer counts sample locations through a $\ln n$ factor.

4. Learnability, Rates, and Loss Assumptions

For a regression loss that is $G$-Lipschitz in the prediction, ERM, symmetrization, and contraction yield

The monotone class consequently has expected excess risk $O(G/\sqrt{n})$. This is neither zero error on each test point nor a high-probability statement; the latter needs an additional concentration argument and its assumptions.

Even if the entropy integral diverges at zero, finiteness at each positive scale can still establish convergence. Fix $a>0$. On $[a,1]$, fat dimension is bounded by $\mathrm{fat}(\mathcal{F},ca)<\infty$, so the integral is finite. First send $n\to \infty$, leaving $4a$; then send $a\downarrow 0$. Integrability at zero gives a favorable rate, not a necessary condition for convergence.

Necessity must be stated for the relevant real-valued learning model and loss. Lipschitzness alone does not make fat dimension characterize every conceivable loss: the identically zero loss is learnable regardless of the class.

5. The Full Architecture Versus the Trained Network

Fitting random labels shows that the architecture permits a rich class. A vacuous VC upper bound still does not prove that a particular trained predictor generalizes badly on a particular distribution.

Distinguish the full architecture class, a norm-constrained subclass, and the particular predictor selected by optimization. This lecture studies the second object to help explain the third. It does not prove that SGD always selects a low-complexity solution.

For linear predictors, duality and Jensen give a useful warm-up:

Cross terms vanish because distinct Rademacher signs have zero product expectation. If $\Vert x_t{\Vert }_2\le r$, then $\Vert X{\Vert }_F\le r\sqrt{n}$, yielding $br/\sqrt{n}$.

No explicit $d$ does not mean dimension can never matter. If every input coordinate retains constant scale, the input norm can grow as $\sqrt{d}$. Dimension-free statements must be read together with fixed norm bounds.

6. Lemma 1: Sampling a Finite Cover

Here is the proof deferred to HW1, with rounding made explicit. Suppose $\Vert v_i{\Vert }_2\le 1$ and

Let $V$ equal $v_i$ with probability ${\beta }_i/B$, and zero with the remaining probability. Then $B\mathbb{E}V=z$ and $\mathbb{E}\Vert V{\Vert }_2^2\le 1$. For independent draws, define

Independence removes cross terms in the centered error:

Some realization therefore has error at most $B/\sqrt{k}$. Choose $k=\lceil B^2/{\epsilon }^2\rceil$. There are at most $(D+1{)}^k$ possible draw sequences, giving

For the metric $\Vert u-v{\Vert }_2/\sqrt{q}$, set $\epsilon =\alpha \sqrt{q}$:

If the atoms are sign-symmetric, their convex hull already contains zero and the spare mass can be redistributed. The general formulation includes a zero atom safely. When $\epsilon \ge B$, the single center zero suffices. When $\epsilon <B$, rounding changes the estimate by at most a factor of 2. Thus the source's principal order is correct, but dropping rounding does not preserve an exact constant of 1.

The mechanism is sparsification: high ambient dimension is tolerable when all vectors have bounded total coefficient mass over a finite atom set. A cover can be indexed by finitely many sampled sequences.

7. Theorem 2: Linear Predictions as Column Atoms

The sample projection is $X\theta$. Omit zero columns and define

Add negative atoms to handle signed coefficients. Cauchy–Schwarz gives

Apply Lemma 1 with $q=n$, $B=b\Vert X{\Vert }_F$, and at most $2d$ atoms:

Ignoring universal constants and rounding gives Theorem 2's form. The atom hull only needs to contain the projected class; it need not equal it.

The old $d\ln (1+2b/\alpha )$ bound has mild accuracy dependence. The new bound has mild dimension dependence but an ${\alpha }^{-2}$ term. Where both are valid, take the smaller one.

8. Theorem 3: Why a Mixed Norm Appears

For $W\in {\mathbb{R}}^{m\times d}$, projected outputs form matrices $\sigma (X{W}^{\top })\in {\mathbb{R}}^{n\times m}$. Use

Coordinatewise ReLU does not expand Euclidean distance. Applying it to both target and cover centers therefore preserves covering accuracy.

Expand

The atom puts feature column $i$ into output coordinate $j$, so $\Vert X{e}_i{e}_j^{\top }{\Vert }_F=\Vert X_{:,i}{\Vert }_2$. After normalization and sign doubling, there are at most $2dm$ atoms. Their coefficient mass is

The mixed norm is exactly what remains after grouping by input column and applying Cauchy–Schwarz. Flattening matrices into $nm$ coordinates yields, at nontrivial scales,

The denominator's $m$ reflects metric normalization, not a guarantee that more outputs make learning easier. With raw Frobenius radius $\epsilon =\alpha \sqrt{nm}$, the denominator becomes simply ${\epsilon }^2$.

Comparing mixed norms

Let $r_j=(|W_{j1}|,\dots ,|W_{jd}|)$. The triangle inequality gives the complete comparison:

This improves that specific coefficient estimate relative to summing row norms. It does not by itself prove a stronger version of every statement in the original paper.

9. Why Intermediate Centers Must Be Legal Outputs

If $M_h=\sigma (M_{h-1}{W}_h^{\top })$, then

This uses both Lipschitzness and $\sigma (0)=0$. Biases or activations nonzero at the origin require extra terms.

An external cover center need not be a valid intermediate output. To obtain an internal cover, start with external balls of radius $\epsilon /2$. From each ball meeting the set, choose a point $a_j$ in the set. Then

Internal $\epsilon$-cover size is consequently bounded by external $\epsilon /2$-cover size. An inverse-square entropy bound incurs a factor of 4; combine this with rounding into a universal constant $C$.

Use $C(M,{\mathcal{W}}_h,{\gamma }_h)$ for an internal cover at raw Frobenius radius ${\gamma }_h$, and set

Each parent center gets its own finite child cover. The norm bound iterates to $\Vert M_h{\Vert }_F\le \Vert X{\Vert }_F{\prod }_{k\le h}{s}_k$. Therefore

The union has at most the number of parents times the largest number of children. Taking logarithms turns this into addition. Large-radius cases can use the legal zero output as a single center.

10. Lemma 2: Propagating Error Through Layers

Fix a true network and write $A_h=\sigma (A_{h-1}{W}_h^{\top })$, with $A_0=X$. Suppose a center satisfies $\Vert A_{h-1}-M_{h-1}{\Vert }_F\le {\alpha }_{h-1}$. Choose a child center within ${\gamma }_h$ of $\sigma (M_{h-1}{W}_h^{\top })$. Then

The matrix norm step follows row by row from $\Vert W_{h}u{\Vert }_2\le s_h\Vert u{\Vert }_2$, followed by squaring and summing. Set ${\alpha }_0=0$ and ${\alpha }_h={\gamma }_h+s_h{\alpha }_{h-1}$. Expanding gives

and hence

New error at layer $h$ is amplified by later layers. The current spectral norm amplifies preexisting error, which is why the product starts at $k>h$, not $k\ge h$.

If normalized errors $e_h={\alpha }_h/\sqrt{n{d}_h}$ were used throughout, the recurrence would instead contain

Working with raw Frobenius errors first avoids silently losing this dimension ratio.

11. Theorem 4: Deriving the 2/3 Allocation

Let the allowed final Frobenius error be $A>0$, and define

The entropy objective is

Assign final-error shares $A{\rho }_h$, with positive ${\rho }_h$ summing to one. Then

It remains to minimize ${\sum }_h{a}_h^2/{\rho }_h^2$. With positive $a_h$, this objective diverges at the boundary and has positive second derivative $6a_h^2/{\rho }_h^4$, so its unique optimum is interior. The Lagrangian is

Stationarity gives $-2a_h^2/{\rho }_h^3+\lambda =0$, hence

Set $T={\sum }_k{a}_k^{2/3}$. Substitution yields

Consequently

Replacing $A$ with $\alpha \sqrt{n{d}_H}$ yields the denominator in Theorem 4. Universal constants account for rounding and internal recentering.

A larger $a_h$ makes fine covering more expensive, so the optimum gives that layer a larger share of final error. Its raw local radius can still be smaller because downstream amplification must also be divided out.

12. From Entropy to Spectral Complexity

Define

The squared norm factor in covering entropy becomes $R^2$. Dudley's square root explains the change from exponent 3 to $3/2$.

For a precise scalar-output illustration, let $d_H=1$, $K=\Vert X{\Vert }_{F}R\sqrt{L}/\sqrt{n}$, and assume entropy is at most a constant times $K^2/{\delta }^2$ over the relevant scales. Let $D$ bound the empirical radius. Then

The lower cutoff cannot be zero: this integral diverges. If $K/\sqrt{n}\le D$, choose $a$ of that order to obtain

Otherwise, $a=D$ supplies a trivial radius bound. Ignoring logarithms gives $\Vert X{\Vert }_{F}R/n$, or $rR/\sqrt{n}$ for bounded input norms.

For vector outputs one must specify the process and final loss. The normalized covering denominator $d_H$ does not automatically give classification an extra $1/\sqrt{d_H}$ improvement. Passing from network covers to a multiclass margin guarantee requires additional empirical-process arguments; scalar contraction alone is not a full proof.

The product of spectral norms bounds Lipschitzness. Full spectral complexity also has a mixed-norm correction. If one weight layer is zero, the bias-free network is identically zero; handle that case separately instead of evaluating a $0/0$ ratio.

13. Margin and Output Scale

For class scores, define

If the true class is 2, scores $(1,3,2)$ have margin 1; scores $(1,1.5,2)$ have margin $-0.5$. Zero margin is a tie, whose correctness depends on the tie-breaking rule. Every misclassification lies in $\{M\le 0\}$.

The ramp loss bridges classification and smooth complexity control:

It is $1/\gamma$-Lipschitz and satisfies

For a fixed class and threshold, bounded-loss empirical-process theory gives empirical margin loss plus complexity and confidence terms. Uniformity across norm scales and thresholds is an additional step needed before selecting $\gamma$ after training.

The handout displays the leading-order relationship

Paper check. The full high-probability theorem also contains a $\sqrt{\ln (1/\delta )/n}$ confidence term and uses a row-norm correction, optionally relative to reference matrices. The handout's Equation (4) is a leading-order summary, not a complete numerical probability guarantee. Original Theorem 1.1

Increasing $\gamma$ counts more training examples as small-margin but lowers the complexity penalty. A simultaneous bound permits this choice in the analysis without retraining. If all training margins are positive, choose $0<\gamma <{\min }_t{M}_t$ to eliminate the empirical term, regardless of whether an endpoint convention uses $<$ or $\le$. Zero training error with ties is insufficient.

A homogeneity check

Bias-free ReLU satisfies $\sigma (cu)=c\sigma (u)$ for $c>0$. Scaling only the last weight matrix by $c$ scales scores and margins by $c$ without changing predictions. Mixed-to-spectral norm ratios are unchanged, while their spectral product scales by $c$. Thus

A large raw margin alone is not evidence of better generalization. Rescaling adjacent layers by $c$ and $1/c$ preserves both the function and the spectral-complexity expression, another useful check.

14. Figure 1: Identical Parameter Counts, Different Outcomes

The horizontal axis is training epoch on a logarithmic scale. Crosses mark the first zero-training-error epoch. Here “excess risk” is the test-minus-training error gap, not $L(\hat{f})-{\inf }_{f}L(f)$.

Original labels reach zero training error earlier. Thereafter their test error is about 0.3, compared with about 0.9 for random labels. The parameter counts match, but norm-based curves differ. Error around 0.9 is consistent with ten independently random classes providing no predictive information for new labels.

The unnormalized norm measure can keep growing after the gap stabilizes; dividing by margin can stabilize it. This supports measuring classification relative to output scale. The plotted Lipschitz measure is the spectral-norm product, not automatically the full corrected $R(W)$.

These experiments provide qualitative support in their setting. They do not establish exact prediction of every network's gap, nor guarantee that the bound is numerically below 1.

15. Figure 2: Reading Normalized Margin Distributions

For $Q=\Vert X{\Vert }_{F}R(W)/n>0$, write $u_t=M_t/Q$. Rescaling the original threshold to $\gamma Q$ gives leading terms

The first term is the empirical CDF's left limit. The plots show smoothed distribution shapes rather than CDF curves themselves; accumulated mass to the left of a threshold corresponds to the penalty.

Under comparable normalization, a rightward distribution tends to have less mass below a fixed threshold. More precisely, compare CDFs over relevant thresholds; a larger mean or a shifted peak alone need not suffice.

The panels compare MNIST with CIFAR10, their randomized-label versions, CIFAR100 with randomized CIFAR10, and random inputs with random labels. The indicated ordering is qualitative and specific to those experiments, not an absolute ranking of datasets independent of training.

The handout's rewritten probability formula contains a $\max$/$\arg \max$ typo. Classification error compares a predicted class with a label, not a numerical score with a class index.

16. A Two-Layer Numerical Example

Let $H=2$, $s_1=2,s_2=3$, $b_1=4,b_2=3$, and final Frobenius error $A=1$. Then

Local radii are

Indeed, $3{\gamma }_1+{\gamma }_2=1$. Layer 1 has the larger final-error share but the smaller local radius because layer 2 amplifies it.

Dropping common factors, the objective becomes

Equal final-error shares instead give ${\gamma }_1=1/6,{\gamma }_2=1/2$, with objective $576+144=720$. The optimized allocation improves the bound.

17. Pitfalls and Source Corrections

Handle degenerate cases separately: zero data collapse projections; omit zero columns; a zero layer makes the bias-free network zero; a radius of zero cannot be inserted into an inverse-square bound; and normalized margins are undefined when $Q=0$.

18. Self-Check and Review Order

1. Why is monotone pseudo-dimension infinite? Witness thresholds become closer with increasing sample size, so progressively smaller perturbations realize arbitrary signs.
2. What is the exact fat dimension at $\alpha =0.5$? Four under the handout's margin convention. Witnesses $-0.75,-0.25,0.25,0.75$ work.
3. Where does inverse-square accuracy come from? Averaging $k$ atoms has mean squared error at most $B^2/k$.
4. Why column ${\ell }_1$ followed by ${\ell }_2$? Group output coefficients by input column, then apply Cauchy–Schwarz against data-column norms.
5. Why the $2/3$ exponent? Differentiating $a_h^2/{\rho }_h^2$ gives ${\rho }_h^3\propto a_h^2$.
6. Can the network entropy integral start at zero? No; its $1/\delta$ integrand requires a cutoff.
7. Does multiplying scores by ten improve classification? Predictions are unchanged, and both margin and spectral complexity scale together.
8. What question remains open in the lecture? Why the particular optimizer tends to find low-complexity, favorable-margin solutions.

For a 30-minute review, spend eight minutes reproducing the monotone bounds and integral, seven deriving atom sampling and the mixed norm, ten on the layer recurrence, KKT calculation, and numerical example, and five explaining the figures and the distinction between statistics and optimization.

Sources and Added Material

The primary source is the local Lecture 4 handout, whose numbering is retained. The exact monotone dimension, rounded sampling bound, recentering constants, expanded KKT derivation, homogeneity checks, and numerical example are added derivations. The confidence term and norm conventions were checked against Bartlett, Foster, and Telgarsky (2017). The handout's other reference is Zhang, Bengio, Hardt, Recht, and Vinyals, Understanding deep learning requires rethinking generalization, ICLR 2017.