Our largest model, GPT-2, is a 1.5B parameter Transformer that achieves state of the art results on 7 out of 8 tested lan-guage modeling datasets in a zero-shot setting but still underfits WebText. Samples from the model reflect these improvements and contain co-herent paragraphs of text.

Code and models from the paper "Language Models are Unsupervised Multitask Learners". You can read about GPT-2 and its staged release in our original blog post, 6 month follow-up post, and final post. We have also released a dataset for researchers to study their behaviors.

2020年5月28日 · GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic.

gpt2: Language Models are Unsupervised Multitask Learners. gpt3: Language Models are Few-Shot Learners. GPT系列和 BERT系列 的模型在今天的自然语言处理界已经可以说是无人不知无人不晓。 尤其是GPT2出来的时候，openai放话说因为该模型的功能太强大，担心被有心之人滥用所以选择不开源，炒足了噱头，引起了巨大的媒体轰动。 虽然过了几年回头看，觉得该团队对该模型有些过于自信，但无可否认的是该系列的模型在刚刚发布的时候，对于各项任务的处理都有优 …

2024年1月22日 · In this work, we study the universality of individual neurons across GPT2 models trained from different initial random seeds, motivated by the hypothesis that universal neurons are likely to be interpretable.

2022年11月1日 · In this work, we bridge this gap by presenting an explanation for how GPT-2 small performs a natural language task called indirect object identification (IOI). Our explanation encompasses 26 attention heads grouped into 7 main classes, which we discovered using a combination of interpretability approaches relying on causal interventions.

Our largest model, GPT-2, is a 1.5B parameter Transformer that achieves state of the art results on 7 out of 8 tested language modeling datasets in a zero-shot setting but still underfits …

GPT-2 is a Transformer architecture that was notable for its size (1.5 billion parameters) on its release. The model is pretrained on a WebText dataset - text from 45 million website links. It largely follows the previous GPT architecture with some modifications:

In this paper, we explore a semi-supervised approach for language understanding tasks using a combination of unsupervised pre-training and supervised fine-tuning. Our goal is to learn a universal representation that transfers with little adaptation to a …

2024年6月15日 · GPT-2模型来源于OpenAI 在2019年2月发布的论文《Language Models are Unsupervised Multitask Learners》，其模型参数多达15亿，它对下游任务不再需要微调，可以直接将模型应用于下游任务（所以是zero-shot)_gpt2 paper