Our largest model, GPT-2, is a 1.5B parameter Transformer that achieves state of the art results on 7 out of 8 tested lan-guage modeling datasets in a zero-shot setting but still underfits WebText. Samples from the model reflect these improvements and contain co-herent paragraphs of text.

Code and models from the paper "Language Models are Unsupervised Multitask Learners". You can read about GPT-2 and its staged release in our original blog post, 6 month follow-up post, and final post. We have also released a dataset for researchers to study their behaviors.

2020年5月28日 · GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic.

GPT-2 is a Transformer architecture that was notable for its size (1.5 billion parameters) on its release. The model is pretrained on a WebText dataset - text from 45 million website links.

2024年1月22日 · In this work, we study the universality of individual neurons across GPT2 models trained from different initial random seeds, motivated by the hypothesis that universal neurons are likely to be interpretable.

In this paper, we explore a semi-supervised approach for language understanding tasks using a combination of unsupervised pre-training and supervised fine-tuning. Our goal is to learn a universal representation that transfers with little adaptation to a …

Our largest model, GPT-2, is a 1.5B parameter Transformer that achieves state of the art results on 7 out of 8 tested language modeling datasets in a zero-shot setting but still underfits …

2024年12月16日 · We introduce Krony-PT, a compression technique of GPT2 \citep {radford2019language} based on Kronecker Products. We specifically target the MLP layers of each transformer layer, and systematically compress the …

Language Models are Unsupervised Multitask Learners

2019年8月12日 · The GPT2 was, however, a very large, transformer-based language model trained on a massive dataset. In this post, we’ll look at the architecture that enabled the model to produce its results. We will go into the depths of its self-attention layer.