Data Wrangling

Data wrangling is the process by which the data required by an application is identified, extracted, cleaned and integrated, to yield a data set that is suitable for exploration and analysis. Although there are widely used Extract, Transform and Load (ETL) techniques and platforms, they often require significant manual work from technical and domain experts at different stages of the process. When confronted with the 4 V's of big data (volume, velocity, variety and veracity), manual intervention may make ETL prohibitively expensive. 

As a result, we are interested in enabling cost-effective approaches to data wrangling, typically through automation or suggestion. In automation, individual or multiple steps within the data wrangling process are carried out by software, using evidence about what the user requires [1]. In suggestion, given a current situation, the user is informed of possible next steps from which to choose. In both cases, it is necessary to explain the proposed actions to the user, and allow additional information from the user to steer the steps that are followed [2].  

While we have recently worked on the development of end-to-end automation for data preparation [1], we are also interested in developing techniques that integrate with the notebook environments that are widely used by data scientists.