Alphabet's AI research division, DeepMind, has introduced Gopher, a new language model that it claims achieves reading comprehension close to that of a high school student. The announcement, accompanied by a research paper, positions Gopher as a significant step in natural language processing, though its mathematical and reasoning abilities remain less advanced.
Gopher is an "ultra-large language model" with 280 billion parameters, a metric that indicates its size and complexity. This places it between OpenAI's GPT-3, which has 175 billion parameters, and Microsoft and NVIDIA's Megatron, which boasts 530 billion parameters. The model was trained on vast repositories of online text, which DeepMind says enabled the improvement in reading comprehension.
In a test, Gopher scored sufficiently high on a high school reading comprehension exam to approach human-level performance, according to DeepMind's paper. However, its math and reasoning skills showed "less of an improvement," leaving room for development. The company suggests that such systems could eventually "safely and efficiently summarize information, provide expert advice, and follow instructions via natural language."
Dialogue and Practical Applications
Beyond comprehension tests, Gopher demonstrated a "surprising" level of coherence in a full dialogue with a human, DeepMind reported. Historically, language models like Gopher have been used in commercial products such as digital assistants and translators. However, DeepMind's vice president of research, Koray Kavukcuoglu, told Fortune that commercialization is not the current focus.
The model's size is part of a broader trend where increasing parameters generally leads to greater accuracy. Yet, challenges like reading comprehension and the perpetuation of harmful stereotypes persist, despite the models' scale. These issues are proving more difficult to overcome, as the sheer volume of training data can embed biases.
Ethical Considerations and Bias Mitigation
To address potential criticism about ethnic or gender stereotypes, DeepMind published an accompanying paper detailing steps taken to maintain ethical integrity. The team developed a tool called Retrieval-Enhanced Transformer, which uses a two-trillion-word database to cross-reference sources. Despite these efforts, the DeepMind team admitted that research on how language models perpetuate harmful stereotypes "is still in early stages."
As AI systems become more adept at interpreting text, researchers are shifting focus to stickier problems, such as the potential for spreading misinformation or propaganda. Helping models like Gopher understand context and nuance—reading between the lines—remains a formidable challenge, one that many AI researchers acknowledge is far from solved.
The release of Gopher underscores the rapid progress in AI language models, but also highlights the ethical and technical hurdles that remain. While the model's reading comprehension is a notable achievement, its limitations in reasoning and the unresolved bias issues serve as reminders of the work ahead.
