In recent years, deep learning techniques have shown great potential in microscopy image analysis, especially in the segmentation of high-resolution transmission electron microscopy (TEM) images. Deep learning methods can automatically extract useful information and effectively reveal the connection between microstructure and material properties Horwath et al., 2020. As Ge et al. (2020) point out in their review, deep learning applied to microscopy image analysis establishes a generalized workflow, derived from the development of research in computer vision. Not only for electron microscopy but also for a wider range of analytical and imaging techniques can be applied to the same generalized workflow, such as light microscopy, scanning electron microscopy, atom probe tomography, and scanning probe microscopy.

Traditional analysis suffers from low efficiency. Machine learning models have faster processing speeds and handle larger amounts of data with impressive accuracy Horwath et al., 2020Ge et al., 2020. For example, traditional analytical methods cannot handle the large amount of data generated by advanced microscopy methods such as in situ electron microscopy Ge et al., 2020. For different types of microscopic images (morphology, size, distribution, intensity, etc.) at different scales, it is difficult to generalize traditional methods to the new data Ge et al., 2020, such as sorting filters, Gaussian smoothing filters, histogram thresholding Tobias & Seara, 2002, and edge detection Senthilkumaran & Rajesh, 2009, region growing Freixenet et al., 2002. Often, the information extraction of these traditional algorithms tends to rely on specific features of a particular dataset Ge et al., 2020. Furthermore, these methods rely on human experts to extract and set features based on their understanding of the dataset, thus being constrained by human limitations Ghosh et al., 2019.

In the field of image processing and computer vision, image segmentation is the process of dividing a digital image into segments to make it more meaningful and easier to analyze, as the representation of the image is simplified by the division of the segments Shapiro & Stockman, 2001. The entire image is segmented into mutually overlapping segments, and each pixel within these regions is similar in some feature or computational attribute (e.g., color, intensity, or texture), whereas neighboring regions are significantly different in the same feature or computational attribute Nielsen & Nock, 2003Shapiro & Stockman, 2001. Image segmentation is one of the common computer vision tasks that has been applied in fields such as self-driving cars, video surveillance, machine perception, biomedical image analysis, and image understanding Minaee et al., 2022Zhang et al., 2022. For example, in medicine, it is used to recognize anatomical structures such as lesions or organs Minaee et al., 2022. In video surveillance, it can be used for land and city management Minaee et al., 2022.

In the field of materials science, it can be used to identify and analyze the microstructure of materials, such as defect analysis or component identification in ceramics or metals Ge et al., 2020Horwath et al., 2020. Azimi et al. (2018) used a convolutional neural network called max-voted FCNN to recognize phases with different compositions in mild steel. Computer vision techniques can also be used to recognize two-dimensional materials in microscope images. In the study of Ramezani et al., 2023, hexagonal boron nitride (hBN) flakes were automatically detected and classified, greatly improving analysis efficiency and reducing human intervention. These approaches are often quite generalizable, allowing the extensibility to other imaging modalities. In another study, Masubuchi et al. (2020) demonstrated a deep learning-based image segmentation algorithm for microscopy that automatically searches for 2D materials and achieves robust detection under multiple microscope conditions. Other types of microscopes, such as atomic force microscopes, can use image segmentation to extract valid information. In a research paper Zhu et al., 2022, deep learning frameworks such as Mask R-CNN were able to automatically perform scanning probe microscopy image analysis, which improved the accuracy and efficiency of molecule recognition.

The main goal of microscopy image segmentation is to accurately separate different regions or phases of an image, which is essential for further analysis and research. Conventional algorithms require tuning the parameters of the algorithm, which are often related to specific microscopy dataset properties. When measuring under different conditions, or switching to other microscopes, the parameters may need to be retuned, which is not generalizable Masubuchi et al., 2020. For example, when the differences in the images are not obvious, the classical thresholding method makes it difficult to adjust the parameters to obtain accurate segmentation Sterbentz et al., 2021. The quality of each segmentation depends on the specific experimental conditions at hand. Another problem is that noise and artifacts often accompany microscopic images, and traditional algorithms may lead to segmentation errors Chuang & Comer, 2010.

The main contributions of this paper are as follows: 1. Four advanced segmentation models were compared by applying to material microscopy. Various neural network architectures, models, and pre-trained resources from the computer vision (CV) field were applied to material microscopy images. 2. We introduce the analysis of the time dimension. While existing microscope image segmentation methods for material science focus on the processing of static images, this paper introduces the analysis of the time dimension and realizes the tracking of different phases in microscope images in the time dimension.

This paper is divided into two parts. In the first part, we segment the different phases in a microscope image into masks. In the second part, we track the change of segmented masks in the time dimension. The first part applies state-of-the-art architectures and algorithms in computer vision to microscope image segmentation for materials science research. It compares four different deep neural models in computer vision. These include YOLOv8 Solawetz & Francesco, 2023 based on Convolutional Neural Networks (CNN) architecture Lecun et al., 1998, EfficientSAM Xiong et al., 2023 based on Vision Transformer (ViT) Dosovitskiy et al., 2021, Swin-UNet Cao et al., 2021Zhu & Lu, 2022 based on Swin-Transformer Liu et al., 2021, and VMamba-UNET Zhang et al., 2024 based on selective state spaces Gu & Dao, 2023. In the second part, the result of segmentation will be used as the input of the tracking model, which can realize efficient and accurate tracking for different phases and ensure the consistency of phase IDs through the EfficientSAM and DeAOT models.