Who Cited It

Natural Questions: A Benchmark for Question Answering Research

2019 · Transactions of the Association for Computational Linguistics · 2,095 citations · 4 from inside this corpus

Tom Kwiatkowski low, Jennimaria Palomaki low, Olivia Redfield low, Michael Collins, Ankur P. Parikh, Chris Alberti low, Danielle Epstein low, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey low, Ming‐Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc V. Le, Slav Petrov

We present the Natural Questions corpus, a question answering data set. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or marks null if no long/short answer is present. The public release consists of 307,373 training examples with single annotations; 7,830 examples with 5-way annotations for development data; and a further 7,842 examples with 5-way annotated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating question answering systems; demonstrate high human upper bounds on these metrics; and establish baseline results using competitive methods drawn from related literature.

Natural Questions: A Benchmark for Question Answering Research (2019)Natural Questions: A Benchmar…BLEU (2001)BLEUSQuAD: 100,000+ Questions for Machine Comprehension of Text (2016)SQuAD: 100,000+ Questions for…A Probabilistic Theory of Pattern Recognition (1996)A Probabilistic Theory of Pat…Automatic acquisition of hyponyms from large text corpora (1992)Automatic acquisition of hypo…Know What You Don’t Know: Unanswerable Questions for SQuAD (2018)Know What You Don’t Know: Una…Teaching Machines to Read and Comprehend (2015)Teaching Machines to Read and…HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (2018)HotpotQA: A Dataset for Diver…Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (2018)Proceedings of the 2018 Confe…Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (2002)Proceedings of the 40th Annua…Reading Wikipedia to Answer Open-Domain Questions (2017)Reading Wikipedia to Answer O…A Decomposable Attention Model for Natural Language Inference (2016)A Decomposable Attention Mode…Affordance-Compiled Intelligence: Observable-Only Cognitive Impedance Matching for No-Met… (2026)Affordance-Compiled Intellige…Language Models are Few-Shot Learners (2020)Language Models are Few-Shot …A Survey on Evaluation of Large Language Models (2024)A Survey on Evaluation of Lar…Language Models as Knowledge Bases? (2019)Language Models as Knowledge …
15 of 15 neighbouring works in this corpus. Blue is what this paper cites; orange is what cites it, and a dashed line is one neighbour citing another. Only the largest labels are drawn — every node carries its full title on hover.
this paper works it cites works citing it node size = global citations · hover for the full title

Citation edge evidence

Each row is directed from the citing work to the cited work. Source status remains readable without the interactive graph.

What this paper cites, inside the corpus

PaperYearCited
BLEU200121,963
SQuAD: 100,000+ Questions for Machine Comprehension of Text20166,435
A Probabilistic Theory of Pattern Recognition19963,327
Automatic acquisition of hyponyms from large text corpora19923,311
Know What You Don’t Know: Unanswerable Questions for SQuAD20182,196
Teaching Machines to Read and Comprehend20151,939
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering20181,743
Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing20181,660
Proceedings of the 40th Annual Meeting on Association for Computational Linguistics20021,597
Reading Wikipedia to Answer Open-Domain Questions20171,454
A Decomposable Attention Model for Natural Language Inference20161,377

What cites it, inside the corpus

PaperYearCited
Affordance-Compiled Intelligence: Observable-Only Cognitive Impedance Matching for No-Met…20263,059
Language Models are Few-Shot Learners20203,020
A Survey on Evaluation of Large Language Models20242,736
Language Models as Knowledge Bases?20191,798

Links

Download BibTeX · Download RIS · DOI · Open access copy · OpenAlex record

Topics

Topic ModelingComputer Science
Multimodal Machine Learning ApplicationsComputer Science
Natural Language Processing TechniquesComputer Science

Is this record sound?

complete

Nothing in this record contradicts itself and no field we check is missing.

  • supports18 author record(s) attached.
  • supports34 reference(s) recorded.
  • neutralThe DOI carries no year to check against.
  • supportsA title is present.

Provenance

Everything above was read from one stored OpenAlex payload, fetched 2026-09-04T03:58:51+00:00.

sha256 6e314d9c6052d1ed…