arXiv Corpus as One Machine-readable Dataset

In August 2020, arXiv announced its entire corpus consisting 1.7 million scholarly articles is available as a free dataset on Kaggle.

arXiv and the Dataset

arXiv is a well-known open access repository and knowledge sharing platform for physics, computer science, and related disciplines such as mathematics. Yet, it can be more than just a repository where scientists go and search to get the articles they need.

In the blog post that announced the launch of the corpus, the arXiv Executive Director Eleonora Presani said:

“By offering the dataset on Kaggle we go beyond what humans can learn by reading all these articles and we make the data and information behind arXiv available to the public in a machine-readable format.”

Additional Value the Open Corpus Can Bring

The dataset contains relevant features including article titles, authors, DOIs, abstracts, full text PDFs, and categories of all the 1.7 million articles on arXiv.

https://www.kaggle.com/Cornell-University/arxiv?select=arxiv-metadata-oai-snapshot.json

Such a sizable and well-structured dataset is an excellent resource for machine learning and deep learning research, which require vast amounts of data to develop new models, novel algorithms, etc. Besides, making the whole corpus as a single machine-readable dataset can facilitate data analysis for various purposes. Scientists can use it to trace research trends, discover relationships, develop new insights, and many more. arXiv as a repository alone would not be able to support all these usages.

What is Kaggle?

Kaggle, where the arXiv dataset resides, claims to be the largest data science and machine learning community in the world. It holds over 50,000 public datasets and 400,000 public notebooks for users to do their data science work. It also offers free online courses on machine learning, natural language processing, Python, data cleaning, and other related topics as well.

Reference
A Dataset of 1.7 Million ArXiv Articles Available on Kaggle by Soner Yildirim, 6 August 2020.

— By Poon Sau Ping, Research Support Services, Library

Hits: 6762