OpenAI Blog · Dec 14, 2018
How AI training scales
Reviewed by Errol Vogt, Site support technician & online learning analyst · original summary · editorial policy
How AI training scales. We’ve discovered that the gradient noise scale, a simple statistical metric, predicts the parallelizability of neural network training on a wide range of tasks. Since complex tasks tend to have noisier gradients, increasingly large batch sizes are likely to become useful in the future, removing one potential limit to further growth of AI systems. More broadly, these results show that neural network training need not be considered a mysterious art, but can be rigorized and systematized. This update is relevant for small-office operators tracking changes in their tools.
Operator takeaway: For operators: review whether 'How AI training scales' affects your current setup before relying on it in production.
ai