Starting an eResearch revolution with deep learning

Manaaki Whenua Landcare Research (MWLR), like most research institutes, both consumes and generates an ever-increasing amount of data. In particular, spatial data (images, hyperspectral data, spatial samples) are central to much of what MWLR does. MWLR has a strong track record of producing spatial data for consumption by research, including cleaned satellite imagery and GIS layers such as the Land Cover Database (LCDB)1 . MWLR also holds many nationally and internationally significant physical collections of flora and fauna. As well as the physical samples themselves, the metadata associated with these specimens is invaluable for further analysis, such as species distribution2 .

In recent years, machine learning (ML) has increasingly matured from a branch of computer science to a respected tool in the researchers’ toolbox. Most recently, deep learning has revolutionised computer vision, unlocking new opportunities to extract knowledge from images and other spatial data. For example, whereas ten years ago it was considered reasonable to be able to identify pollen grain species from images with 65-70% accuracy by both humans and computers, it is now straightforward to achieve accuracies exceeding 95% using deep learning3 .

At MWLR, we have begun a journey to dramatically increase the impact of our research by consuming/reconsuming data using machine learning, with a particular focus on deep learning. This is being achieved through a collaboration between the Informatics research team and the wider scientific cohort at MWLR. This process began with a small number of projects where the potential benefit was clear. Early results of these projects have been disseminated internally through webinars, leading to further projects being identified. In this initial stage, the emphasis is on rapidly achieving results and developing broad knowledge of tools and techniques. To date we have focussed on image classification through feature extraction4 , segmentation (U-net5 ) and object detection (Mask R-CNN6 ).

Through this initial exploratory phase, we have identified two fundamental barriers to uptake. First, researchers distrust “black box” models that do not add to our understanding of “why”, and that cannot explain how they reach a conclusion. We are addressing this first concern by exploring how classification decisions can be visualised to show what contributed to the outcome. For example, we have shown that a deep learning model can classify beech pollen species from images to over 80% accuracy, a task considered too difficult for even specialist humans. Because the researchers have been sceptical of this outcome, we have used occlusion sensitivity visualisations to demonstrate that for correct classifications the deep learning network is focussing on expected areas of the image, such as the pollen grains’ edge or texture, unlike for incorrect classifications. We are now investigating whether similar techniques exist that are suitable for image segmentation and object detection tasks. For segmentation problems, we can also manually investigate differences between the training data and predictions; in some cases the error may be inaccuracies in the training data, highlighting the potential for deep learning models to augment manual processes as a further benefit.

The second barrier to uptake is the quantity and quality of data needed. For image classification tasks, we have developed a novel method of utilising deep learning models for feature extraction that dramatically reduces both the number of training examples required as well as the processing requirements7 ; we have successfully built species identification models with good accuracy from as little as a few hundred images for domains such as fungal spores, coprosmas, moths and beech pollen. For segmentation tasks, we are experimenting with methods for bootstrapping imperfect training data through an iterative process of training weak models and using them to refine the training data with some additional manual correction where required8 . We are experimenting with this technique for identifying tree species from UAV orthomosaics where the class polygons are weakly inferred from tree stem positions obtained through ground-based surveys, and then subsequently refined based on the segmentation suggested by the model. It is hoped that such techniques will dramatically lower the effort required to build training sets for such tasks, increasing the value obtained from localised ground surveys by using the data to make inferences at regional or national scale. Finally, we are also exploring the impact of resolution on accuracy, to quantify the limits of scaling up small-scale surveys to be repeatable at the national level from more easily available spatial data such as hyperspectral satellite imagery

We have so far identified 12 projects, half of which are being actively pursued. We have also organised an internal “mini-symposium” which will present two case studies, as well as discussing machine learning and deep learning techniques. A “panel” session will then discuss further potential project ideas submitted by the audience. This approach has been successful in engaging further researchers; to date six further projects have been identified ranging from counting manuka flowers in images to extracting text from historical documents, and it is anticipated that the panel discussion will generate significant further interest.

Brent is a machine learning specialist at Manaaki Whenua Landcare Research. His career has spanned both academic research as a senior lecturer at Canterbury University, as well as software engineering and R&D roles in various commercial companies. Brent’s research in AI and machine learning includes developing new ML classification algorithms; applying ML to real-world problems such as electricity demand forecasting; research and development in Intelligent Tutoring Systems; developing social network analysis techniques for criminal investigation. Brent holds a PhD in Computer Science from the University of Canterbury, New Zealand focussing on artificial intelligence in education.

Aleksandra is an eResearch capability specialist at Manaaki Whenua Landcare Research, where she is assisting with the development of strategy, procedures and tools that promote data-driven science and research data management. She also organises and instructs workshops assisting researchers to develop their research data skills. In her career Aleksandra has been active in the UK’s Software Sustainability institute, where she led the institute’s training activities. Outside of academia, Aleksandra has worked as a Research Community Manager for the New Zealand eScience Infrastructure (NeSI), as a researcher for NHS Lothian projects and as a freelance IT consultant in the commercial sector. She is also an instructor for the Software Carpentry Foundation. Aleksandra holds a PhD in Computing from the Open University focussing on documentation in scientific software.