Nuclei Instance Segmentation of Cryosectioned H&E Stained Histological Images using Deep Learning

Abstract

Nuclei instance segmentation is an important step for oncological diagnosis and pathology research of cancer. HE stained images are considered the gold standard for medical diagnosis. But before being used for segmentation, it is required to pre process them. There are two principle methods to preprocess them formalin-fixed paraffin-embedded samples (FFPE) and frozen tissue samples (FS). Even though FFPE is widely used, it is a time consuming process whereas FS samples can be processed very quickly. But analysis of FS-derived HE stained images can be more difficult as rapid preparation, staining, and scanning of FS sections results in degradation of image quality. Therefore, in this thesis, we explored various state of the art segmentation architectures to create a model that will segment nuclei of FS-derived HE stained images with a high quality feature extraction.

Here, we have been working on a novel dataset called CryoNuSeg that contains 30 FS-sectioned images of 10 human organs. It has a benchline score of DICE 80.3 ±4.3, AJI 52.5 5.0, PQ 47.7 6.1. U-Net is the first and most prominent architecture for biomedical image segmentation. We are exploring various U-Net architectures. We have trained Triple U-NET on the dataset using binary masks in place of U-NET keeping all other parts of the instance segmentation algorithm same such as Gaussian Filtering and Watershed Post processing. The results using Triple U-NET crossed all the benchline scores. The triple U-Net architecture gives a score of DICE 80.33, AJI 67.41 and PQ 50.56.

We have developed a deep learning model that performs highly accurate nuclei segmentation of FS sections despite degraded image quality for fast oncological diagnosis.