Reaching into the past: Deep learning and historic aerial imagery
Deep learning has revolutionised the field of computer vision, enabling imagery to be analysed quickly and efficiently to extract valuable data. At Manaaki Whenua we are increasingly using deep learning to extract features from aerial and satellite imagery for generating maps. As increasing amounts of historic aerial imagery are digitised, it becomes possible to use deep learning to extract features for snapshots in time, allowing changes to be detected and trends to be analysed. In one such study, we are measuring the changing amount of available public and private urban green space in three of New Zealand’s cities: Auckland, Wellington and Hamilton.
One of the steps is to extract building footprints from historic aerial images taken in the 1940s and 1980s. Our approach was to train a deep learning encoder-decoder network (Unet641) on recent (monochromed) imagery using the LINZ building layer2. Whilst this yielded excellent results for modern imagery, performance was unacceptably poor when applying the model to the historic aerial images. First, the historic imagery was taken using photographic film rather than digital capture, resulting in a completely different noise (graininess) profile. Second, the lighting conditions varied enormously across the three time periods, with both shadowing (from varying light angle) and contrast varying substantially both across the image sets and within the earlier sets.
Finally, the image quality for the 1940s imagery was particularly low, with poor (and variable) focus and high levels of lens dropoff. However, there were no training labels available to train specific models for the historic image sets because of a lack of label data. To overcome these issues, we tried a variety of methods for dealing with imperfect data. First, the historic images were pre-processed to overcome some of the shortfalls, such as histogram normalisation, noise reduction and sharpening. We then trained a model for the first city (Hamilton), using the 1980s imagery but the current building labels.
This training set had substantial imperfections: many of the buildings in the current labels did not exist in 1980 or had been modified. Further, the orthomosaic created from the historic imagery was not perfectly registered because of unpredictable distortions in the images; buildings were often displaced by up to several metres. Nonetheless, by including only image tiles where there was reasonable agreement between the historic image and the modern labels, we succeeded in training a model that reliably detected buildings in the raw 1980s imagery. We then retrained the model using the pre-processed images, and used this model to detect buildings in the 1940s imagery with impressive accuracy, despite the substantial differences between the image sets.
Finally, we demonstrated that the Hamilton models gave excellent prediction for Auckland city across all tree time periods, demonstrating the generality of the approach. This approach has the potential to be applied to other domains to unlock valuable insights from the growing collections of historic aerial imagery.
Ronneberger, Olaf; Fischer, Philipp; Brox, Thomas (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. arXiv:1505.04597
Brent is a senior data scientist and machine learning specialist at Manaaki Whenua – Landcare Research. His 35-year career spans both academic research as a senior lecturer at Canterbury University, as well as software engineering and R&D roles in various commercial companies from local software house Jade to Google NY. Brent’s research in AI and machine learning includes developing new ML classification algorithms, applying ML to real-world problems such as electricity demand forecasting and internet search engines, research and development in Intelligent Tutoring Systems, developing social network analysis and anomaly detection techniques for criminal investigation, and applying deep learning to environmental problems. Brent holds a PhD in Computer Science (artificial intelligence) from the University of Canterbury, New Zealand.