Automated Big data analysis - Exploiting artificial intelligence techniques in order to allow the automation of data analysis and feature extraction tasks

The amount of data in today’s world is ever increasing, and has even led to new terms describing this as in Big Data. Fully automated data mining technologies that can be used to understand this data currently do not exist apart from very expensive data mining software suites – which in any case require large amounts of human direction and interaction. The training for these suites is also very expensive resulting in the majority of companies being unable to afford either the time or the expense in using these software packages.

Technology developed at the University of Aberdeen over a number of years and resulting in a spinout company could help in solving this problem. This technology is based on the application of artificial intelligence techniques, as well as exploiting the power of statistical methods alongside modern computational power. It is specifically designed to allow fully automated data analysis and can potentially be applied to any type of problem.

This technology has general applicability and can be used on problems as varied as traditional condition monitoring type problems to financial instrument analysis to genomics analysis to the analysis of social media data. This project will look at improving these techniques and comparing them against other currently easily available methods. In particular, the project will examine the process of determining how important features can be determined, and how data can be automatically transformed to give the features of interest.
This work has great commercial value and will likely be of interest to companies in the data analysis field.

This project will focus on the task of automating data analysis tasks, particularly for numerical data but the techniques developed could be applied to other domains such as textual data analysis. The data mining process currently requires human interaction and guidance throughout the process, and this project will look to exploit artificial intelligence techniques in order to allow the automation of the data analysis and feature extraction tasks.

The project will build on existing research work in this area, and the problem can be broken down into a number of distinct but important parts. The identification of important and irrelevant features in a dataset is an important task and one that cannot currently be undertaken in an automated manner. This task can be complicated when the features for a given class are different for different classes. The process developed needs to be able to take this into account and automatically identify the features for the given classes. In addition the process needs to be able to automatically classify the samples into the correct classes. This classification needs to be undertaken using artificial intelligence methods so that when classification is not successful, the decisions that the method has made can be investigated in order to change the input features in some way so that the next iteration will result in a different classification. The purpose therefore is to identify an iterative process that will learn which features are important in an automated manner.

The overall methodology developed will be tested against synthetic and also real world datasets.

The successful candidate should have, or expect to have, an Honours Degree at 2.1 or above (or equivalent) in Engineering, Physics or Computational Science.

Knowledge of: Computer coding, algorithms, data analysis