GitSchemas: A Schema Dataset for Automating Relational Data Preparation Tasks


The preparation of relational data for machine learning (ML) has largely remained a manual, labor-intensive process, while automated machine learning has made great strides in recent years. Long-standing challenges, such as reliable foreign key detection still pose a major hurdle towards more automation of data integration and preparation tasks. We created a new dataset aimed at increasing the level of automation of data preparation tasks for relational data. The dataset, called GitDBSchemas, consists of schema metadata for almost 50k real-world databases, collected from public GitHub repositories. To our knowledge, this is the largest dataset of such kind, containing approximately 300k table names, 2M column names including data types, and 100k real (not semantically inferred) foreign key relationships. In this paper, we describe how GitDBSchemas was created, and provide key insights into the dataset. Furthermore, we show results from a preliminary experiment that leverages GitDBSchemas to find relevant tables in an AutoML setting.

Databases for Machine Learning workshop at ICDE