Lecture 6 Detailed Explanation

This lecture asks which classes can learn from adaptive data. We develop finite-class concentration, zero-covers, sequential Sauer, Littlestone upper and lower bounds, threshold and monotone counterexamples, regression covers, and surrogate losses. Source: lecture6.pdf (local source, not published online); companion: full handout; previous: Lecture 5; next: Lecture 7. The deep Dudley and sequential-fat covering proofs omitted by the handout remain explicitly external; their applications are worked through.

1 Roadmap and Conditions

Lecture 5 reduced regret to sequential Rademacher complexity. For classification, the chain is finite-expert bounds, pathwise zero-covers, then Littlestone dimension. Regression replaces exact matching with approximation at a scale, leading to sequential fat dimension.

Source pages	Content	Proof status
1–2	Lemma 1, Theorem 1: finite classes	Developed below
2–4	Zero-cover, Theorem 2, Corollary 1	Developed with induction boundaries
4–5	Theorem 3, Proposition 1	Developed with a finite-dimension lower-bound discussion
6	Theorems 4, 5, 6	4 proved; 5–6 invoked
7–8	Linear, monotone, surrogates, summary	Calculations and limitations expanded

Use Lecture 5’s measurability and game conditions. Classification functions take values ±1 with zero-one loss; regression functions lie in $[- 1, 1]$ . The horizon is $n$ . In classification $d$ means Littlestone dimension, while in the linear example it means ambient dimension; do not conflate them.

2 The Maximal Inequality Does Not Require Independent Experts

Suppose every $U_{f}$ satisfies $E e^{λ U_{f}} \leq e^{λ^{2} σ^{2} /2}$ . They may be highly dependent. The maximum exponential is bounded by their sum, and logarithm is concave, so

E f max U_{f} = \frac{1}{λ} E ln e^{λ m a x_{f} U_{f}} \leq \frac{1}{λ} ln E f \sum e^{λ U_{f}} \leq \frac{ln K}{λ} + \frac{λ σ ^{2}}{2} .

Optimize with $λ = 2 ln K / σ$ to obtain $σ 2 ln K$ . For $K = 1$ , the variables used here have mean zero, giving zero. If “sub-Gaussian” means centered around each variable’s own mean, an additional maximum-mean term is required; the handout uses the zero-centered MGF convention.

3 Theorem 1: Concentration of Tree-Indexed Sums

Fix a tree and function. Write $a_{t} = f (x_{t} (ϵ_{< t}))$ and $S_{t} = \sum_{s \leq t} ϵ_{s} a_{s}$ . Given the past, $a_{t}$ is known and only the current fair sign is new, hence

E [e^{λ ϵ_{t} a_{t}} ∣ ϵ_{< t}] = \frac{e ^{λ a_{t}} + e ^{- λ a_{t}}}{2} = cosh (λ a_{t}) \leq e^{λ^{2} a_{t}^{2} /2} .

One cannot multiply unconditional moment-generating functions: future $a_{s}$ may depend on the current sign. The handout takes successive maxima of future square sums. A shorter supplementary proof uses a nonnegative exponential supermartingale:

M_{t} = exp (λ S_{t} - \frac{λ ^{2}}{2} s = 1 \sum t a_{s}^{2}), E [M_{t} ∣ ϵ_{< t}] \leq M_{t - 1} .

Iterating from $M_{0} = 1$ gives $E M_{n} \leq 1$ . Let $σ^{2} = max_{f} max_{ϵ} \sum_{t} a_{t}^{2}$ , a deterministic uniform bound. Then

E e^{λ S_{n}} = E [M_{n} e^{\frac{λ ^{2}}{2} \sum_{t} a_{t}^{2}}] \leq e^{λ^{2} σ^{2} /2} E M_{n} \leq e^{λ^{2} σ^{2} /2} .

Apply the preceding section and divide by $n$ to obtain Theorem 1. If absolute function values are at most $C$ , then $σ^{2} \leq n C^{2}$ :

R^{seq} (F; x) \leq \frac{2 σ ^{2} ln ∣ F ∣}{n}, R^{seq} (F) \leq C \frac{2 ln ∣ F ∣}{n} .

Example. For $K = 100, n = 10000, C = 1$ , the bound is approximately 0.03035. Expert count costs only a logarithm; more rounds improve the bound by a square root. This worst-tree upper bound is not the exact realized regret of a particular sample.

4 Why Full-Tree Projection Overcounts

A depth- $n$ tree has $2^{n} - 1$ nodes. Full-tree projection records each function’s predictions on every branch. For singleton indicators with distinct node inputs, this gives the all-negative tree plus one positive-node tree per node, totaling $2^{n}$ . Yet a run visits only $n$ nodes, and its singleton labeling contains at most one +1.

We need not preserve one function’s complete responses on all unvisited branches simultaneously. Swapping two quantifiers yields an exponential saving:

Projection: Zero-cover: \forall f \exists v \forall ϵ \forall t, f (x_{t} (ϵ)) = v_{t} (ϵ), \forall f \forall ϵ \exists v \forall t, f (x_{t} (ϵ)) = v_{t} (ϵ) .

The representative may depend on the function and full path but must match every round along that path. It cannot be changed each round, and each representative tree remains predictable. The collection $V$ is fixed before drawing the path. Pathwise,

f sup t \sum ϵ_{t} f (x_{t} (ϵ)) \leq v \in V max t \sum ϵ_{t} v_{t} (ϵ) .

Take expectations and use the finite-tree-family bound. Cover trees need not be members of the original class, so equality need not hold.

5 Singleton Covers with Repeated Inputs

The handout first gives $n + 1$ trees for paths without repeats. Here is an explicit construction allowing repeats. Expert 0 always outputs −1. Expert k, for $k = 1, \dots, n$ , outputs −1 before round k, remembers the input $a = x_{k}$ at round k, and thereafter outputs +1 exactly when the input equals a.

For a target $f_{θ}$ and path, expert 0 matches if $θ$ never appears. Otherwise choose k as the first occurrence of $θ$ ; expert k matches every round, including later repeats. It uses only past/current inputs and defines a valid tree. The worst-case $n + 1$ is attained on a tree with a distinct constant input at each level: one fixed path has $n + 1$ possible label sequences and each cover tree matches only one.

For inputs a,b,a and target $θ = a$ , expert 1 outputs +1,−1,+1. The simpler “positive at one level only” construction fails here.

6 Littlestone Dimension: Realizing Every Path Labeling

\forall ϵ \in {- 1, + 1}^{n}, \exists f \in F, f (x_{t} (ϵ_{< t})) = ϵ_{t} \forall t .

Here $ϵ$ both determines branches and prescribes labels. In zero-covering it only determines the path. Different paths may use different functions, but one path must use a single function. Littlestone dimension is the largest shattered depth. A fixed input sample is a tree constant at each level, so VC dimension is no larger.

For singleton indicators, any root can be labeled positive by its own singleton or negative by another. After a positive root, exactly one function survives; whatever the next input is, it cannot realize both labels there. Hence the dimension is 1. A nonempty zero-dimensional class agrees at every input and contains one distinct function. Assigning −1 to the empty class is useful for algorithms.

7 Sequential Sauer: Why Maximum Rather Than Product

Split a dimension- $d$ class at the root into $F_{+}, F_{-}$ . Both cannot retain dimension $d$ : attach their shattered depth- $d$ trees below this root and obtain a shattered depth- $d + 1$ tree. Thus one side has dimension at most $d - 1$ , the other at most $d$ .

For the positive-label subclass, suppose the original input tree’s left subtree needs a representatives and its right subtree b. Every representative on each side only needs to appear in some pair. Repeat the shorter list and use $max (a, b)$ pairs. Once a path goes left, the right representative is irrelevant, and conversely. Simultaneously matching every branch would instead call for a Cartesian product of size $ab$ .

g (d, n) \leq g (d, n - 1) + g (d - 1, n - 1), g (0, n) = 1, g (d, 0) = 1.

The choice $g (d, n) = \sum_{i = 0}^{m i n (d, n)} (i n)$ satisfies the boundaries and Pascal recursion. When $d \geq n$ , the sum is $2^{n}$ , closing a boundary omitted in the source induction. This proves the first part of Theorem 2.

The exponential bound follows directly. Set $x = d / n \in (0, 1]$ . For $i \leq d$ , $x^{i} \geq x^{d}$ , so

x^{d} i = 0 \sum d (i n) \leq i = 0 \sum d (i n) x^{i} \leq (1 + x)^{n} \leq e^{n x} = e^{d} .

Dividing by $x^{d}$ gives $(e n / d)^{d}$ . For $d = 2, n = 10$ , the binomial sum is 56 and the exponential bound is approximately 184.7. The latter sacrifices constants for simplicity. Substitution yields Corollary 1’s $2 d ln (e n / d) / n$ .

8 Necessity: Fresh Labels Defeat Every Learner

If arbitrary depths can be shattered, choose a depth- $n$ tree and let the environment provide $(x_{t}, y_{t}) = (x_{t} (ϵ_{< t}), ϵ_{t})$ . Even after seeing the current input, the learner faces a fresh fair label, so conditional binary error probability is exactly $1/2$ . Expected total loss is $n /2$ . Shattering provides one perfect function on each realized path, giving zero comparator loss. Average regret is at least $1/2$ . Together with the upper bound, finite Littlestone dimension characterizes binary online learnability.

A finite-dimension lower bound also has a clear construction. As a supplementary derivation, first take $n = d m$ . Use a shattered depth- $d$ tree, repeat each visited node m times, and draw fresh fair labels. At the end of a block, use its majority label to select the next branch, with a fixed tie rule. Shattering provides a function predicting every block majority. If the block sum is $S_{m}$ , then

learner expected block loss = \frac{m}{2}, comparator block loss = \frac{m - ∣ S _{m} ∣}{2} .

Expected regret is $d E ∣ S_{m} ∣/2$ . A Rademacher sum satisfies $E ∣ S_{m} ∣ \geq m /2$ , obtainable from its central-binomial-coefficient formula or Khintchine’s inequality. Hence

\frac{E Reg}{n} \geq \frac{d}{2 n} \frac{m}{2} = \frac{d}{8 n} (n = d m) .

The handout leaves the precise constant for all $d \leq n$ to HW2; the divisible case above is not presented as that full proof. Unequal blocks of lengths differing by at most one already give the same order, for example $d / n /4$ . This explains that the gap is logarithmic, not a different power of dimension or horizon.

9 Thresholds: VC Dimension One Is Not Enough Online

Take $f_{θ} (x) = + 1$ exactly when $x \leq θ$ . Two fixed points cannot realize arbitrary labels, so VC dimension is 1. An online adversary can instead keep bisecting the remaining threshold interval:

Text

                       1/2
            label −1 /     \ label +1
                   1/4     3/4
                 /    \   /    \
               1/8   3/8 5/8   7/8

A positive label requires the threshold to be at least the current input, so probe the right half next; a negative label sends the probe left. Each step leaves a nonempty interval, so no finite number of rounds exhausts real thresholds.

For a rigorous ordering argument, after going right at round $t$ , the first increase is $2^{- (t + 1)}$ . The total possible subsequent leftward movement is strictly smaller, being a finite geometric sum. Thus every later node remains larger than the round- $t$ node; the left case is symmetric. Every positive node lies below every negative node. Choose

θ \in [t : ϵ_{t} = + 1 max x_{t}, t : ϵ_{t} = - 1 min x_{t}),

with defaults 0 and 1 for empty sides. For labels +1,−1,+1 the visited points are $1/2, 3/4, 5/8$ , and any threshold in $[5/8, 3/4)$ works. This proves shattering for every finite depth, without asserting that every infinite path has a positive-width interval.

10 Regression Covers and Normalization

d_{p, ϵ} (f, v) = (\frac{1}{n} t \sum ∣ f (x_{t} (ϵ)) - v_{t} (ϵ) ∣^{p})^{1/ p} .

Division by $n$ means normalized radius $α$ in the two-norm corresponds to ordinary Euclidean error $α n$ . The quantifiers remain “a representative for each function and path.” Normalized power means satisfy $d_{1} \leq d_{2} \leq d_{\infty}$ , giving the same ordering of covering numbers. Increasing the radius weakens the requirement and cannot increase the cover size.

Theorem 4 follows by adding and subtracting a representative. For a fixed function and path,

\frac{1}{n} t \sum ϵ_{t} (f (x_{t}) - v_{t}^{f, ϵ}) \leq \frac{1}{n} t \sum ∣ f (x_{t}) - v_{t}^{f, ϵ} ∣ \leq α .

The remaining representative term is bounded by the maximum over the finite cover, then Theorem 1 applies. Representatives may depend on the full path because the maximum already ranges over every representative.

R^{seq} (F; x) \leq α \geq 0 in f {α + \frac{2 ln N _{1} ( F ∣ _{x} , α )}{n}} .

This compresses at only one scale. Dudley uses progressively finer covers so small details incur small-amplitude costs. Theorem 5 states

R^{seq} (F; x) \leq 0 \leq α \leq 1 in f {4 α + \frac{12}{n} \int_{α}^{1} ln N_{2} (F ∣_{x}, δ) d δ} .

The cutoff $4 α$ avoids resolving scales with enormous or infinite covering numbers and controls the residual by approximation error. The square-root logarithm comes from the finite maximum inequality at each scale. Full sequential chaining must handle predictability and localized errors after pathwise representative selection; path-dependent representatives cannot simply be treated as fixed processes. This intuition is not a full proof, and Theorem 5 remains an invoked result.

11 Sequential Fat Dimension: The Witness Is a Tree

\exists y, \forall ϵ, \exists f \in F, ϵ_{t} (f (x_{t} (ϵ_{< t})) - y_{t} (ϵ_{< t})) \geq α /2 \forall t .

Witness thresholds may depend on the past but not the current label; otherwise moving a threshold afterward would trivialize the definition. The dimension is the maximum depth permitting such fixed positive-margin branching and is nonincreasing in scale. Ordinary fat on fixed points is a special case with level-constant inputs and witnesses and can be smaller.

ln N_{\infty} (F ∣_{x}, α) \leq sfat (F, α) ln (2 e n / α), 0 < α \leq 1.

This is Theorem 6, stated without proof. It requires discretization and scale-sensitive sequential combinatorics, not an unproved direct substitution into binary Sauer. Combine it with the ordering of covers and Dudley to recover the final chain. Use the uniform fat bound over all trees and $sup in f \leq in f sup$ ; this is not equality from freely exchanging the two operators.

For learnability characterizations and the external results, see Rakhlin, Sridharan, and Tewari (2015). Conditions depend on the loss and decision protocol. A logarithmic upper bound alone does not establish the same rate for every class finite at each scale without checking scale growth.

12 Linear Example: Evaluating the Entropy Integral

Hölder bounds linear functions on dual unit balls in $[- 1, 1]$ . Parameter distance at most $α$ in the $p$ -norm gives prediction error at most $α$ at every allowed input. A volume cover of size $(1 + 2/ α)^{d}$ therefore works pointwise and on every tree. Thus

R^{seq} (F) \leq \frac{12 d}{n} \int_{0}^{1} ln (1 + 2/ δ) d δ .

For $0 < δ \leq 1$ , $1 + 2/ δ \leq 3/ δ$ . Subadditivity of square root and substitution $u = ln (1/ δ)$ give

\int_{0}^{1} ln (1 + 2/ δ) d δ \leq ln 3 + \int_{0}^{1} ln (1/ δ) d δ = ln 3 + \int_{0}^{\infty} u e^{- u} d u = ln 3 + \frac{π}{2} .

The integral is a constant, yielding $O (d / n)$ . This bounds complexity; general-loss contraction may add logarithms. As a supplement, Euclidean unit balls admit a stronger dimension-free calculation:

R^{seq} (F) = \frac{1}{n} x sup E t \sum ϵ_{t} x_{t} (ϵ)_{2} \leq \frac{1}{n} x sup E t \sum ϵ_{t} x_{t} (ϵ)_{2}^{2} \leq \frac{1}{n} .

Cross terms vanish by conditioning on $ϵ_{< t}$ for $s < t$ : both vectors and $ϵ_{s}$ are known, while $ϵ_{t}$ has mean zero. Squared norms sum to at most $n$ . This foreshadows Lecture 7’s dimension-free Perceptron bound.

13 Why the Monotone Counterexample Reverses the Path

The threshold tree with positive labels branching right places positive points left of negative points, corresponding to a nonincreasing classifier. For a nondecreasing regressor, negatives must lie left of positives. Traverse according to $- ϵ$ but label with $ϵ_{t}$ . Then

(x_{t}, y_{t}) = (x_{t} (- ϵ_{< t}), ϵ_{t}), E [(a_{t} - y_{t})^{2} ∣ past, a_{t}] = a_{t}^{2} + 1 \geq 1.

Every algorithm incurs expected cumulative squared loss at least $n$ . On each path, a step function from −1 to +1 fits every point with zero loss, forcing linear regret. This step-function subclass also shatters arbitrary depths with witness zero for every $0 < α \leq 2$ , so sequential fat is infinite, unlike the ordinary $O (1/ α)$ fat dimension.

The transductive setting changes information by revealing the input set in advance; the handout states learnability there. That statement is not proved here, and knowing the set does not mean knowing its order or labels.

14 Surrogates Change the Comparator Loss

1 {y s \leq 0} \leq max (1 - y s, 0), 1 {y s \leq 0} \leq \frac{ln ( 1 + e ^{- y s} )}{ln 2} .

If $y s \leq 0$ , hinge is at least 1 and natural-log logistic at least $ln 2$ ; otherwise the left side is zero. Thus the displayed inequalities hold pointwise. Unnormalized natural-log logistic does not upper-bound zero-one loss: $s = 0$ gives $1 > ln 2$ .

t \sum 1 {y_{t} f_{t} (x_{t}) \leq 0} \leq f in f t \sum ℓ^{'} (f, (x_{t}, y_{t})) + Reg_{ℓ^{'}} (F, n) .

Small regret approaches the best surrogate loss, not the best zero-one error count, so it does not contradict threshold impossibility. The bound is useful when a good-margin comparator has small hinge loss. Normalized natural-log logistic has Lipschitz constant $1/ ln 2$ , not 1. Counting zero margins as errors also differs from a fixed sign tie rule.

15 Self-Test and Reading Route

Why does path-dependent representative selection not reveal the future? The cover is fixed; the maximum is taken afterward in analysis, not by an online decision.
Which expert covers a,b,a with target a? The first-positive-at-round-one expert, predicting +,−,+.
Why use a maximum when pairing same-label subcovers? A path visits only one subtree.
What happens at $d = 0$ ? A nonempty class has one function, cover size 1, and complexity 0. Do not evaluate $0 ln (e n /0)$ literally.
What threshold interval realizes +,−,+? $[5/8, 3/4)$ .
What factor is missing for natural-log logistic? $1/ ln 2$ .
Why does linear regression learnability not imply linear zero-one classification learnability? The loss and comparator have changed.

First read Sections 4–9 to master trees and quantifiers. Next rederive Sections 2–3 and 7 for concentration and recursion. Then read Sections 10–14 and compute the integral and zero-score logistic value. Proceed to Lecture 7: dimensions become realizable mistake budgets, and Hedge aggregates experts in the agnostic case.