Using Arrow, with Numba KerneIs, to Generate AI Workflows
Speaker: John Murray, Director, Fusion Data Science and Visiting Professor, Data Science Lab, University of Liverpool
In this session, John demonstrates the use of the Numba Python compiler, to create custom kernel functions, on top of Arrow tables, to generate end to end AI workflows with TensorFlow (training), TensorRT (inference), and RAPIDS cuml (clustering). The talk is based on his group's recently published academic paper: 'Estimating generalized measures of local neighbourhood context from multispectral satellite images using a convolutional neural network' https://doi.org/10.1016/j.compenvurbs...
Links for talk:
Software libraries used:
Numba JIT High Performance Compiler https://numba.pydata.org/
CuPy GPU-accelerated Array Python Library https://cupy.dev/
Rasterio Geospatial Image Processing Library https://github.com/rasterio/rasterio
DuckDB Fast SQL Engine for Arrow https://duckdb.org/
TensorFlow Machine Learning Library https://www.tensorflow.org/
TensorRT High Performance Deep Learning Inference Runtime https://developer.nvidia.com/tensorrt
Our recent paper from the lab describing the project in more detail:
Estimating Generalized Measures Of Local Neighbourhood Context From Multispectral Satellite Images Using A Convolutional Neural Network’ (Singleton, Murray, Aribas-Bel, Fleishmann 2022) https://doi.org/10.1016/j.compenvurbs...
Source code (open source): https://github.com/GDSL-UL/sat_cnn
Datasets used in the process including the output tensorset and tables (open source): https://dataverse.harvard.edu/dataver...
Paper defining high precision Gauss-Krüger equations for coordinate conversion used to crop the satellite images:
‘Transverse Mercator With An Accuracy Of A Few Nanometers’ (Karney 2011) https://doi.org/10.48550/arXiv.1002.1417
Paper describing the application of a convolutional neural network to ESA Sentinel 2 satellite data which provided background to the project
‘A CNN-Based Fusion Method for Super-Resolution of Sentinel-2 Data’ (Gargiulo, Mazza, Gaetano, Ruello, Scarpa 2018) https://doi.org/10.1109/IGARSS.2018.8...
Our lab
Geographic Data Science Lab at University of Liverpool https://www.liverpool.ac.uk/geographi...