OpenAI Blog · Apr 23, 2019
Generative modeling with sparse transformers
Reviewed by Errol Vogt, Site support technician & online learning analyst · original summary · editorial policy
Generative modeling with sparse transformers. We’ve developed the Sparse Transformer, a deep neural network which sets new records at predicting what comes next in a sequence—whether text, images, or sound. It uses an algorithmic improvement of the attention mechanism to extract patterns from sequences 30x longer than possible previously. This update is relevant for small-office operators tracking changes in their tools.
Operator takeaway: For operators: review whether 'Generative modeling with sparse transformers' affects your current setup before relying on it in production.