Overfitting

Consider error of hypothesis h over

• training data: error_train(h)
• entire distribution $\cal{D}$ of data: $error_{\cal{D}}(h)$

Hypothesis $h \in H$ overfits training data if there is an alternative hypothesis $h' \in H$ such that

error_train(h) < error_train(h')

and

$$error_{\cal{D}}(h) > error_{\cal{D}}(h')$$