Sparsearray Objects
Hervé Pagès, Software Dev Engineer IV at Fred Hutchinson Cancer Research Center, gave a short talk at the BioConductor Conference 2022. Pagès gave his presentation on Sparsearray Objects: A New Container For Efficient In-Memory Representation Of Multidimensional Sparse Arrays.
“SparseArray objects use an innovative internal representation, called Sparse Vector Tree or SVT layout, to store the sparse data in memory. This layout allows compact representation as well as efficient access to the data. SparseArray objects support the traditional array API from base R, that is, the end user can operate on them via standard array operations like [ (subsetting), [- (subassignment), dim(), dimnames(), t(), etc... Comparison, arithmetic, and other mathematical operations will be supported (some of them already are), including row and column summarization methods as defined in the matrixStats package from CRAN. In this short talk, Pagès introduced the SVT layout and quickly discuss its differences from the more traditional CSC layout used by dgCMatrix objects from the Matrix package. Furthermore, Pagès presented some typical Bioconductor use cases where we believe that using SparseArray instead of dgCMatrix objects will offer significant benefits. Finally, he showed a roadmap towards feature completeness and where we stand on that roadmap. SparseArray objects are implemented in the upcoming SparseArray package: https://github.com/Bioconductor/S4Arrays. This is still work-in-progress (work on the package started in Fall 2021).”
Main Sections
0:00 Introduction
1:36 De facto standard for in-memory sparse data representation: dcCMatrix objects
2:32 matrixStats/sparseMatrixStats
4:08 Some limitations of dgCMatrix objects
6:48 SparseArray objects
9:41 SparseArray objects: internal representation
10:56 SparseArray objects: performance
11:56 SparseArray objects: still a lot to do…
12:22 Closing and Q&A
More Resources
Bioconductor Conference Site: https://bioc2022.bioconductor.org/
BioC2022 Github: https://github.com/Bioconductor/BioC2022
Main Site: https://www.r-consortium.org/
News: https://www.r-consortium.org/news
Twitter: / rconsortium
LinkedIn: / r-consortium