Overfitting

<p>Jul 03, 2011 10:24 AM

&#160;

OVERFITTING &#160;... when statistical model describes random error or noise instead of the underlying relationship.

&#160;

OVERFITTING occurs when a model is EXCESSIVELY COMPLEX, such as having too many parameters

relative to the number of observations.

&#160;

Why is it bad? &#160;A model which has been overfit will have poor predictive performance.

&#160;

a model is typically <b>trained by maximizing its performance on some set of training data</b>.&#160;

&#160;

However, its efficacy is determined not by its performance on the training data&#160;

but by its ability to perform well on unseen data.&#160;

&#160;

<b>Overfitting occurs when a model begins to memorize the training data&#160;</b>

<b>rather than learning to generalize from the model</b>.&#160;

&#160;

As an extreme example, if the number of parameters is the same as or greater than the number of observations, a simple model can learn to perfectly predict the training data<b> simply by memorizing the training data in its entirety</b>. <b>Such a model will typically fail drastically on unseen data, as it has not learned to generalize at all</b>.

&#160;

MACHINE LEARNING

&#160;

Generally, a learning algorithm is said to overfit relative to a simpler one <b>if it is more accurate in fitting known data (hindsight) but less accurate in predicting new data (foresight)</b>.&#160;

&#160;

One can intuitively understand overfitting from the fact that information from all past experience&#160;

can be divided into two groups:

&#160;<b>information that is relevant for the future and irrelevant information (“noise”)</b>.</p>