OpenAI Blog · Dec 5, 2019
Deep double descent
Reviewed by Errol Vogt, Site support technician & online learning analyst · original summary · editorial policy
Deep double descent. We show that the double descent phenomenon occurs in CNNs, ResNets, and transformers: performance first improves, then gets worse, and then improves again with increasing model size, data size, or training time. This effect is often avoided through careful regularization. While this behavior appears to be fairly universal, we don’t yet fully understand why it happens, and view further study of this phenomenon as an important research direction. This update is relevant for small-office operators tracking changes in their tools.
Operator takeaway: For operators: review whether 'Deep double descent' affects your current setup before relying on it in production.
ai