New approaches for analysis and visualization of point cloud and image stack data

Single molecule localization microscopy (SMLM) can reveal the molecular dynamics and molecule arrangements inside the cell at the nanometer scale. SMLM relies on the localization of the center of individual single molecules with a precision well below the diffraction limit of an optical microscope. The localization process repeated on the different acquisition frames results in data sets consisting of high-dimensional information of each molecule e.g.  spatial and temporal coordinates among others. Yet, visualization and analysis of the recovered data remains a complex endeavor, in part because of the intricate 3-dimensional cellular architecture or dynamics across time. Tasks, such as segmentation and analysis, that rely on interaction with the 3D representation of the data at arbitrary angles are impractical and time consuming. Moreover, retrieving physical parameters from the recorded point cloud such as shapes and curvatures is hindered by noise dues to variable localization precisions and false detections.

The team has previously developed the software Genuage¹² to provide innovative ways to visualize and analyze 3D and 4D point cloud representations of single molecule localization data, in both desktop and virtual reality environments. In this internship we propose to extend the functionality of this software to loading, storing and representation of microscopy pixel-based image stack using advanced volume rendering. The goal is to develop a new type of representation of single molecule data overlaying point cloud localizations with the microscopy images in both desktop and VR interfaces. Further work will aim to efficiently render very large datasets, as well as developing new tools to analyze image stacks in Genuage. We also propose to explore a new analysis approach using Graph neural networks to infer shapes and physical parameters out of point of clouds that can run within software.

The internship’s goals are:

• Creating a new visualization pipeline integrated within Genuage, to perform volume rendering of single molecule microscopy image stacks.
• Developing a new mode of representation for 3D single molecules data overlaying the 3D volumetric representation from image stacks to a point cloud representation for individually localized molecules.
• Building systems to render massive datasets (e.g dozens of millions of points for point clouds) with comfortable experience for desktop and VR display and interaction.
• Development of innovative virtual reality tools useable in Genuage’s interface for efficient quantification, segmentation and analysis of volume data.
• Use Graph neural networks to infer shapes from point clouds.

1. Blanc, T., El Beheiry, M., Caporal, C. et al. Genuage: visualize and analyze multidimensional single-molecule point cloud data in virtual reality. Nat Methods 17, 1100–1102 (2020). https://doi.org/10.1038/s41592-020-0946-1 ↩︎
2. Blanc, T. Verdier, H., Regnier L. et al. Towards human in the loop analysis of complex point clouds ; advanced visualizations, quantifications and communication features in virtual reality. Front. Bioinform., 20 January 2022 Sec. Data Visualization https://doi.org/10.3389/fbinf.2021.775379 ↩︎