An overview of industrial image segmentation using deep learning models

3.1.1. Encoder-decoder-based models

Convolutional networks have been effectively applied in computer vision tasks, especially when initially used for object recognition and classification such as LeNet ^[107]. These images are processed by the input layer, hidden layer and output layer step by step, and the conception has also been applied in most subsequent computer vision tasks. Moreover, hidden layers comprise convolution layers, fully connected layers, and pooling layers, and CNNs tend to become an influential framework of image segmentation due to their translation invariance, parameter sharing, and sparse connections with improvement of computing resource level.

Different from treating semantic segmentation as a region classification, fully convolutional networks (FCNs) from Long et al. consider it as a pixel-level classification problem, and only convolutional layers are included in FCN^[108]. As depicted in Figure 3, FCNs are encoder-decoder-based models, and their encoder is designed for downsampling to extract high-level semantic features, and then deconvolution is used for upsampling, enabling a segmentation map to be the same size as its input image. Combining FCN and structured forests with wavelet transform (SFW), FCN-SFW ^[65] is a fusion segmentation algorithm to effectively detect tiny cracks in steel beams, achieving superior segmentation performance against existing methods. The result resolution from FCN is too low due to 16 times upsampling rate, and its segmentation boundaries between objects tend to appear incorrectly. Without considering the correlation between pixels, spatial information is prone to be lost in FCN.

Figure 3. The network structure of FCN. FCN: Fully convolutional network.

To address above two issues, DeepLab V1 ^[109] introduces dilated convolution that enlarges dilation in the convolutional kernel to capture a larger range of contextual information without increasing computational complexity, improving CNN receptive field while ensuring parameter quantity. To effectively learn relationships between pixels, DeepLab V1 only improves accuracy compared to FCN, whereas the overall accuracy is still not high. Both SegNet ^[110] and U-Net ^[111] utilize a similar encoder-decoder structure as FCNs, and fine-grained upsampling to recover to original maps. In SegNet, unpooling in the decoder to upsample feature maps, which reduces the number of parameters and computations compared to deconvolution in FCN. As shown in Figure 4, U-Net improves a long skip connection, different from DenseNet ^[112], to connect feature maps at different levels and enable the model to simultaneously focus on low-level and high-level semantic features. Hence, U-Net is used to obtain preliminary insulator masks by training insulator images ^[4]. Nevertheless, the skip connection imposes unnecessary restrictive fusion schemes, forcing fusion only on feature maps of the same proportion in the encoder and decoder. Li et al. improved U-Net combining VGG16 and hybrid attention module to achieve high-precision segmentation of PCB welding defects and meet real-time detection requirements^[4]. In accordance with U-Net, a multi-scale attention DL model called MDOAU-Net is designed by Wang et al. with extended convolution and offset convolution for monitoring aquaculture rafts with SAR image segmentation^[113].

Figure 4. The network structure of U-Net.

Moreover, DeepLab V2 ^[114], DeepLab V3 ^[115], DeepLab V3+ ^[116], and PSPNet ^[117] adopt a spatial pyramid pooling (SPP) structure with granularly multi-receptive field learning the relationships between pixels, which extracts feature maps and convert them into fixed-size feature vectors for subsequent classification and regression tasks. SPP makes the model handle input images of different sizes and improves robustness to changes in object position and size by pooling feature maps at different scales. Inspired by PSPNet, SE-PSPNet ^[118] accurately segments defects in 3D braided composites, enhancing small object detection and addressing class imbalance for improved performance. Due to multi-scale input, RefineNet ^[119] can obtain global contextual information of small-sized targets. Although these models enhance their ability to learn relationships between pixels, their parameter quantity, and inference speed cannot meet real-time demands. Due to actually high real-time requirement, Mobile-Deeplab ^[92] is a lightweight pixel segmentation-based fabric defect detection strategy with 87.11 frames per second (FPS) on a $$ 256 \times 256 $$ size image. The improved PCB-DeepLabV3 model ^[93] combined with Mobilenetv2 and multi-scale attention pooling achieves high-precision intelligent segmentation of voids inside chip solder joints, improving industrial detection efficiency.

Instead of upsampling to restore resolution in the decoder, HRNet ^[71] parallelly connects high-resolution to low-resolution convolutional streams and exchanges information across resolutions with multi-scale feature fusion to remain high-resolution expression. Wang et al. achieved a mean intersection to union ratio of 70.60% with HRNet by denoising Sentinel-1 SAR radar images and enhancing them with deep neural networks in industrial applications^[120]. Besides, some DL-based models efficiently learn contextual connections between pixels from a spatial perspective. DANet ^[121] simultaneously introduces spatial attention and channel attention to capture contextual information between any two positions and channel dimensions. Given that Transformer is able to capture global information of input data effectively with multiple Transformer blocks in Figure 5, ABFormer ^[80] improves the accuracy of defect semantic segmentation and solves the problems of intra-class differences and inter-class ambiguity in industrial products by introducing boundary perception modules and attention mechanisms. Due to these drawbacks, it is essential to utilize image features from multiple scales containing features from shallow and deep layers in the backbone network for accuracy.

Figure 5. The standard network block of Transformer.

For real-time industrial demand, professional lightweight architecture is essential for practical application efficiency. ERFNet ^[122], ESPNetV2 ^[123], and other lightweight models adopt an asymmetric encoder-decoder structure, removing low-level feature extraction results via exchanging accuracy for speed. With ESPNetV2, a high-precision surface crack detection architecture for steel rolling equipment is presented by Peng et al., which utilizes coordinate attention-deep convolution generative adversarial networks (CA-DCGANs) for data augmentation, MLESPNetV2 for accurate detection, and MLESP-GAN for semi-supervised learning, enhancing real-time performance significantly^[124]. Shallow and deep features respectively contain more spatial information and contextual information for images and they are indispensable in DL models. Therefore, some researchers have utilized a lightweight network to achieve multi-scale input while avoiding large-sized convolutional kernels similar to approaches in RefineNet ^[119]. These typical models include ICNet ^[125], etc., while their segmentation accuracy is acceptable with reluctantly real-time speed.

Moreover, the feature extraction network should be as lightweight as possible for less occupied computing resources. In glaucoma diagnosis, Mallick et al. presented a ConvNext encoder and decoder with skip connections realizing light weight, and a novel attention gate module was designed for smoothing^[126]. Besides, this method improves its loss function for accurate model training. ConvNeXt-SCC ^[94] visualizes tire crown defects and enhances defect detection and localization performance in tire production despite challenges such as limited datasets and small defect sizes by a modified convolutional block attention module and ConvNext.

3.1.2. Region proposal based methods

Region proposal based methods are generally identified as the combination of semantic segmentation and object detection tasks, for which they are known as a two-stage method. In accordance with the series sequence of object detection and semantic segmentation, two-stage approaches are further divided into top-down and bottom-up schemes. The top-down DL segmentation technique adopts the notion of detection before segmentation, predicting a bounding box for each object first and then generating an instance mask within each bounding box. By contrast, the bottom-up approach utilizes a semantic segmentation model for pixel embedding projection, and then uses post-processing to project pixels onto each instance, i.e., completes segmentation before detection.

In top-down DL image segmentation, Mask Region-based CNN (R-CNN) proposed by He et al. adopts ResNet as its backbone, adds a branch to predict segmentation masks in each region of interest (RoI), constructs a feature pyramid network (FPN) to process features at different scales, and utilizes non-maximum suppression (NMS) to remove masks with excessive intersections^[127]. Since Mask R-CNN outperforms previous benchmarks in standard datasets such as COCO, an intelligent welding quality detection model ^[81] called ABC Mask R-CNN using enhanced Mask R-CNN with attention mechanisms, achieves 98.20% accuracy, improves efficiency over traditional expert-based methods, and supports intelligent maintenance in metro train body manufacturing. However, this mask R-CNN evaluation function only scores bounding boxes outputted by its target detector instead of mask prediction, disrupting segmentation results.

Mask Scoring R-CNN ^[128] adds MaskIoU head to learn intersection over union (IoU) scoring templates, and obtains more accurate masks. BlendMask ^[66] locks in the approximate range of instance pixels by object detection, and within this range, mask refinement is applied to the instance pixels to ensure the quality of the instance mask. Later, Geng et al. combined BlendMask with top-down and bottom-up structures to detect tunnel defects^[129]. Although bottom-up methods such as deep metric learning (DML) ^[130] conform to human intuition, they generally depend on post-processing methods such as performance, causing over- or under-segmentation. Hence, DL-based bottom-up image segmentation has generally been less mainstream than top-down methods.

If the order of detection and segmentation is only distinguished, it is difficult to bring performance gains to two-stage image segmentation schemes. Therefore, the multi-stage approach has emerged, where segmentation and detection no longer have a sequential order. In the typical multi-stage Cascade Mask R-CNN ^[131], each cascade has an independent RPN to produce proposed boxes, and each detector filters bounding boxes given by its previous detector for boxes with high confidence so that this more accurate box proposal is used for detection and segmentation. By cascade, it enhances accuracy for segmentation with inconspicuously increased computational burden. In accordance with Cascade Mask R-CNN, Diaz et al. presented a fast wind turbine defect detection model with depthwise separable convolutions, enhanced by image augmentation and transfer learning, with a superior performance of 82.42% mAP^[82].

Although two-stage and multi-stage DL approaches bring object instances of different scales into a standard scale and improve segmentation accuracy, they lack sufficient representational power and low efficiency. As demonstrated in the experimental literature ^[132], abundant training resources are needed in both two-stage and multi-stage DL schemes. Therefore, the one-stage segmentation model has emerged that consolidates detection and segmentation into a single network. Based on the presence or absence of a preset fixed-size object detection bounding box, the one-stage DL model is divided into two categories, i.e., anchor-based or anchor-free schemes. Anchor-based image segmentation mainly develops from you only look once (YOLO) ^[133] series. Deeplab-YOLO ^[95] is a lightweight hot spot defect detection model in infrared photovoltaic (PV) panels with MobileNetV3 to replace the YOLOv5 backbone, and MobileNetV2 was introduced into Deeplabv3+ for segmentation. SSA-YOLO ^[134] is an enhanced YOLO by a convolution squeeze-and-excitation module, Conv2d-BatchNorm-SiLU with Swin transformer, and adaptive spatial feature fusion module, thereby advancing quality control in steel production.

The anchor-free segmentation detector is mainly built on fully convolutional one-stage object detectors (FCOS) ^[135], including models that classify masks only by location and use contour regression to segment labels. Explicit shape encoding-based segmentation (ESE-Seg) ^[136] and polar coordinate-based instance segmentation (PolarMask) ^[68] predict contour regression to output instance masks. They typically use 20 to 40 coefficients to parameterize a mask contour, have fast inferring speed and are easy to optimize. However, they cannot accurately depict masks and objects with holes in their centers. Segmenting objects by locations (SOLO) ^[67] and conditional convolutions for instance segmentation (CondInst) ^[69] define instance categories by position and size. SOLO sets instance segmentation as a classification problem and removes any regression-dependent problem, making SOLO naturally independent of object detection. CondInst is prone to being affected by imprecise contour regression methods. By contrast, the performance of anchor-based segmentation is affected by detection, and an effective anchor is relatively time-consuming to obtain. Moreover, these anchor-free segmentation models serve as a comparative model to evaluate the performance of designed models for industrial applications due to their fast inferring speed.

3.2.1. Image-level weakly-supervised image segmentation

Image-level supervision refers to using image-level labels such as category labels of objects contained in each image to train a model, which ignores the object position and size in an image and poses obstacles to learning object boundaries. Compared to pixel-level annotation, image-level annotation greatly reduces the manpower, time, and capital costs of data annotation. Mayr et al. designed $$ L_{p} $$ normalization with ResNet50 for defect detection in the solar cell by image-level supervision^[137]. Shi et al. improved defect detection in diverse scenarios by a semi-supervised semantic segmentation with more reliable pseudo labels from dynamic threshold strategy^[96]. However, due to lacking the position, shape, and contour information of targets in an image, training segmentation networks through image level annotation is a highly challenging problem, to which the key lies in how to convert image category information into corresponding target location information.

The segmentation process of most existing image-level semantic models mainly includes three stages. Object features are firstly preliminarily learned with coarse-grained annotation information. Since the class activation map (CAM) ^[138] cannot cover the entire corresponding target, different CAMs need integration to obtain a more complete target activation region. CAM-based models such as gradient-weighted class activation mapping (Grad-CAM) ^[139] enable classification-trained CNNs to learn object localization without the need for any bounding boxes. LayerCAM ^[72] combines activation maps from different layers to simultaneously obtain fine-grained object details and accurate spatial localization, achieving a segmentation effect of 27.26% mean IoU (mIoU) on the industrial product defect dataset Deutsche Arbeitsgemeinschaft fuer Muster-erkennung (DAGM)-2007. Wu et al. introduced a weakly-supervised defect segmentation algorithm with image-level labels and CAMs, enhancing performance with a Siamese network and improved modules^[75]. He et al. proposed a weakly-supervised pavement crack segmentation method using CAM based on a GAN (U-GAT-IT) model to generate and refine CAMs, effectively bridging the gap between unsupervised and fully supervised approaches^[97]. A foreground-background separation transformer (FBSFormer) ^[98], a weakly-supervised pixel-level defect detection method, enhances CAMs through a foreground-background separation module and attention-map refinement, achieving superior performance on industrial defect datasets.

Introducing the prompt paradigm from natural language processing into computer vision for segmentation, the segment anything model (SAM) is applied into semi-/weakly-supervised models to obtain exact segmentation outputs. SAM is Meta's open source generic model for image segmentation tasks. It utilizes a combination of CNNs and Transformer architectures to process images in a hierarchical and multi-scale manner, and prompt engineering idea is introduced to realize prompt segmentation based on point, box, mask and even free-form text. In addition, SAM employs a large dataset (SA-1B) containing at least 1 billion masks and 11 million images for model pre-training, enabling powerful generalization capabilities.

CS-WSCDNet ^[140] adopts CAM with localization capability and SAM to localize changed regions of image pairs and generate pixel-level pseudo labels to train a model for high-resolution scenes. The self-attention and model global correlation are integrated for optimized initial CAMs, a Siamese structure and generic SAM for a joint optimization strategy are used to produce high-precision target boundaries in remote sensing images at different scales ^[141]. In PV power generation, Yang et al. strategically combined both SAM and CAM to refine efficient pseudo-labels, and a boundary-aware loss function was employed to manage error features derived from generated pseudo-labels^[142].

3.2.2. Bounding-box weakly-supervised image segmentation

Weakly-supervised schemes based on bounding boxes adopt bounding boxes surrounding an object to train their segmentation models, allowing the model to generate segmentation results based on produced bounding box range. Bounding boxes provide category labels for objects, and include the quantity and rough location information of objects. The bounding box-based segmentation methods utilize the position and size information of object bounding boxes instead of pixel-level annotation information inside the object, making it a more powerful supervision signal than image-level annotation. Thereupon, such methods tend to achieve better performance than image-level annotation methods. Because all regions outside bounding boxes are divided into background, and both foreground and background regions exist inside the minimum bounding box, it is essential for bounding box weakly supervision to accurately distinguish foreground objects and background regions decided by bounding boxes.

In 2015, Dai et al. proposed BoxSup, the first segmentation network based on bounding box annotation^[143]. BoxSup uses border annotation and multi-scale combinatorial grouping (MCG) ^[144] to generate rough segmentation outputs of target classes, and uses them as supervised information to train parameters in VGG. Then, based on the current network, the predicted target segmentation results are used as supervision to continue training a segmentation network, iteratively optimizing segmentation results as supervisory information and updating network parameters. Due to category information annotated with borders and foreground segmentation results from MCG, BoxSup achieves good segmentation accuracy.

In Box2seg ^[145], pixel-level cross entropy loss is optimized by predicting attention maps of each object type, and the filling rate of each object kind in the minimum bounding box is calculated to constrain the generation of attention maps, thereby making the attention maps better focus on the foreground region and reducing incorrect gradient propagation. Zhang et al. trained Cut-Cascade R-CNN with simple bounding box annotations, obtaining high accuracy on welding defect detection in radiographic images^[83]. CapNet ^[99] is a novel insulator self-blast defect detection network using normal samples and bounding box annotations, featuring a memory mechanism, polar alignment loss, and MirrorFill algorithm to enhance defect detection without additional manual effort. These schemes guide the segmentation process by calculating the fill rate of each foreground object in the minimum bounding box, improving the accuracy of pixel-level pseudo labels.

Some weakly-supervised schemes leverage various bounding boxes to guide network training for effective segmentation results and obtain object instances. In OBBInst ^[100], oriented bounding box annotations with compact object expression have a lower annotation burden than horizontal bounding box annotations, which are combined with corresponding loss function for oriented projection. Besides, OBBInst introduces Canny edge prior into DL to identify more precise edges for instance segmentation. Moreover, Jiang et al. presented single object box-supervised segmentation for insulator images using horizontal bounding boxes, generating segmentation maps to synthesize high-quality images for weakly-supervised segmentation mix and providing pseudo-labels for weakly-supervised instance segmentation^[101].

Compared with weakly-supervised semantic segmentation, the supervised signal used in weakly-supervised instance segmentation is directly related to target instances. There is relatively less general prior information that can be learned and utilized compared to semantics, making it more sensitive to changes in the shape, size, quantity, and other aspects of the target instance. However, coarse-grained labels in weakly-supervised image segmentation models have inaccuracies, missing labels, or mutual interference between labels, resulting in lower accuracy and stability of the training model. Therefore, further research and improvement are needed to enhance the accuracy and robustness of weakly-supervised semantic segmentation with more accurate bounding box prior and labels.

3.3.1. Pixel-level feature relation as prior information

Traditional unsupervised segmentation schemes are mainly realized through artificially designed image features and low-level image features, such as Gaussian mixture models, spectral clustering and K-means clustering, which are adopted to divide images into multiple regions with high self-similarities or discrepancies. In view of the abstract and high-level feature representation capacity of DL, unsupervised feature representation learning combined with DL develops rapidly. Unsupervised intensive feature representation learning, as indicated in SimCLR ^[146] and MoCo ^[147], substantially promotes unsupervised DL image segmentation strategies. In unsupervised image segmentation, it is essential for an effective network to learn a dense feature map for a given image without any annotation. An efficient network analyzes and models the similarity between pixels based on features and these learning feature maps represent pixels in the same semantic region (object/stuff) with similar feature descriptions, while pixels from different semantic regions are represented with various feature representations. Since the network learning process is not guided by labeled data, unsupervised segmentation schemes are constructed by extracting pixel-level features and grouping pixels into different semantic groups according to a variety of prior rules independent of training data.

Kanezaki ^[148] used a classical machine learning unsupervised segmentation algorithm for input image pre-classification to assign identical semantic labels to small areas where the semantic information is clearly identical, and CNN was adopted to process fine-grained pre-classification results from machine learning and merge these small semantic pixel regions for expected segmentation results step by step in iteration. Wang et al. designed an unsupervised CNN model with feature learning and self-attention modules to capture features and this proposed model clustered labels according to feature similarity to segment surface defects from components fabricated by selective laser melting with various materials^[84]. A multi-stage unsupervised framework was proposed by Wei et al. based on DCGAN, and completed defect detection by comparing differences between test and reconstructed images from minimal normal samples^[85]. In these models, superpixel segmentation and image reconstruction are combined into a unified network model and similarities or discrepancies between superpixel feature blocks are utilized to upgrade network parameters. Pixels extracted from randomly initialized CNNs are grouped on feature maps to minimize the distance between similar pixels while maximizing the distance between different features to maintain feature diversity.

There are proposed models to learn feature representation of repeated semantic objects in the image via maximizing mutual information between pixel features from two views of the same input image ^[149]. Information maximization and adversarial regularization segmentation (InMARS) ^[150] clusters generated superpixel regions with mutual information for maximizing the dependency between output and input, and then an adversarial regularization is combined with data augmentation strategy for training and optimization. Li et al. designed a PCA stretch alignment operation and loss function for fusing infrared and visible image features, and utilized mutual information maximization between visible feature maps and infrared feature maps to obtain these interimage relations among pixels, which is unified with loss functions to optimize computing and fusion representation^[102]. A mixed noise-guided mutual constraint (MNMC) ^[103], an unsupervised anomaly model, utilizes mixed noise generation and mutual constraints to enhance feature learning, effectively addressing challenges in anomaly detection, and demonstrating competitive performance on MVTec dataset. Midwinter et al. proposed an unsupervised semantic segmentation method leveraging pose information to enhance visual inspection, validated through a spalling quantification task^[86].

Furthermore, other models generate dense self-supervised information including cross-view consistency, cross-pixel similarity and cross-image relation according to heuristic priors to complete unsupervised image segmentation tasks. Despite different views, the same target object is considered to be consistent, which is so-called cross-view consistency. SimSiam ^[151] is a simple Siamese network model, where neither large batches nor negative sample pairs and momentum encoders are needed. It employs cross-view consistency between two randomly enhanced images by two decoder branches in the same input image. A self-supervised learning approach ^[104] for hotspot detection in thermal images, utilizing a SimSiam-based ensemble classifier, achieves high accuracy and precise hotspot isolation, addressing industrial safety concerns.

Cross-pixel similarity is that the same semantic region in an image contains pixels with highly similar cues such as color, texture or brightness. FreeSOLO ^[76] is an unsupervised framework to study instance segmentation combining segmentation and dense self-supervised learning for generated class-agnostic masks via cross-view consistency. Pixels from the same category target across images possess semantic relations. Due to the lack of supervised information, extracting semantically similar pixel features across images is challenging. Zhang et al. introduced an implicitly cross-image relation unsupervised segmentation scheme with pixel-wise contrastive learning^[152]. This model assigns pseudo labels to all pixels in training images by clustering image features and utilizes these pseudo labels to choose appropriate positive or negative pairs with contrastive learning.

It is a preliminary stage for unsupervised DL to explore how to produce intensive representation information. Different from pixel-level feature learning, dense representation requires regional priors to extract the relationship between pixels, which is used to determine whether these pixels are subordinate to the same semantic space. However, prior rules are generally unable to handle complex scenes. Objects of different categories are likely to possess similar appearances, while it is considerably possible for objects of the same category to have significant intra-class appearance variations, causing algorithms to be unable to effectively distinguish the differences in semantic features for each pixel in the feature space. Therefore, it is worth further exploration on how to introduce more accurate region priors and refine them for a more efficient segmentation learning process.

3.3.2. Intermediate features as additional clues

Instead of directly processing pixel-level features, a portion of unsupervised learning approaches adopt intermediate features as additional clues to guide semantic feature grouping, including saliency maps, superpixels, and shape priors. Saliency is a pattern of image partitioning, while saliency maps represent features on the uniqueness of each pixel. Saliency maps are extracted to simplify or change the representation of general images into a style that is appropriate for DL strategies to discover and analyze the salient and common foreground objects with the same semantic class. Superpixels are generated by directly clustering spatially connected groups of pixels instead of pixels, increasing the reliability of statistics when increasing batch size for training, and decreasing operation cost. Different from texture and color, the shape prior representation is more applicable to application scenarios with limited data scale and large shape changes but insignificant color features such as organ or tumor region extraction.

Utilizing features from a set of pixel groups, namely superpixels, transforms unsupervised image segmentation from pixel-level classification into unsupervised sub-region classification. MaskContrast ^[153] employs two additional unsupervised saliency models to generate pseudo labels for foreground objects in each image, and trains its final segmentation model a two-step bootstrap mechanism. Subsequently, collected foreground masks are clustered to form semantic concepts via contrastive learning. An unsupervised PolSAR image classification segmentation framework ^[105] utilizes high-confidence superpixel pseudo-labels and semi-supervised methods, achieving significant accuracy improvements over existing methods, as demonstrated on three real PolSAR datasets. A saliency-based multitarget detection and segmentation framework ^[77] utilizes a multibranch convolutional encoder-decoder network for circular-scan synthetic aperture sonar imagery. Although saliency maps are obtained to reduce the complexity of pixel-level feature expression, it is difficult to capture effective semantic information without the guidance of common semantic category information.

It is inadvisable for DL schemes to merely use low-level image information; the segmentation accuracy at object edge details is limited. Superpixels not only make for retaining the object edge information, but also represent image features instead of a large number of pixels, hence decreasing the noise interference for model training. Chen et al. utilized superpixel segmentation results to supervise the gradient descent direction for better FCN training, which realizes pixel-level segmentation for remote sensing images with various scales^[78]. S³Net ^[87] is a deep Siamese network guided by superpixels and inspired by transfer learning, and it generates pseudocolor maps to obtain homogeneous superpixel objects for desirable boundary adherence and multi-scale object-level difference features. Methods based on superpixels use a small region as a computing unit to independently extract features and determine saliency values, while it is possible for superpixels to generate salient object feature maps with fuzzy boundaries, incurring training instability.

Explicit shape priors are conducive to mitigating the dependence on the color features and learnable prototype. SegSort ^[154] is a pixel-level end-to-end unsupervised model with spherical K-means clustering to obtain pseudo segments and create segments aligned with semantic boundaries, which is crucial for subsequent clustering, prototype extraction and segmentation. For ultrasound vessel segmentation, an unsupervised and model-based domain adaptation segmentation with two-stage training strategy was suggested to achieve in-vivo data segmentation and the prior information from the elliptical shape of segmentation masks was applied to recognize uncontemplated outputs ^[155]. In rail surface defect segmentation, Ma et al. used prior information on content features and style features as input of a transfer module to obtain shape-consistent style features, and inputted constructed images using multitask learning to avoid distortion with SegFormer ^[156] as a feature extractor^[88].

By implementing intermediate feature transformation, consistency information comparison and other schemes according to saliency maps, superpixels, and shape priors, the semantic discrimination of features is enhanced, promoting the separation of heterogeneous pixels. It is effective to adopt intermediate features as guidance to train segmentation for lower computing cost and complexity than pixel-level feature map classification. However, these methods assume that the objects in the image have sufficient saliency, which is not always true in some complex scenario-based situations and is likely to cause segmentation inaccuracy and instability.

3.3.3. Post-processing as a refinement

In the image segmentation process, an unsupervised model needs to model and analyze the similarity between pixels according to features such as color, texture and edge. Subsequently, this unsupervised model divides all pixels in an image into different groups, and different groups represent various object categories to complete segmentation tasks. However, since the network learning process is not guided by labeled data, segmentation results are prone to be undesirable. As unsupervised learning algorithms involve the processing and learning of unlabeled data, it is essential to be fully aware of the prior structure regarding the data and the model for expected convergence. Moreover, unsupervised learning algorithms usually require more domain knowledge to select and tune models, and further post-processing or manual intervention is hence needed to improve segmentation accuracy and efficiency.

Upon careful examination of the predicted mask, small prediction edges or regions with incorrectly predicted pixels are segmented out. Therefore, a step to finer adaptation and segmentation boundaries is essential to be added into unsupervised learning. An unsupervised recurrent scheme for DL and its corresponding loss function were suggested by Wang et al. to generate optimized pixel-level feature expression and semantic labels^[89]. An attention-gate-based U-shaped reconstruction network (AGUR-Net) ^[157] achieves 59.38% precision on YDFID-1 dataset for unsupervised defect detection in color-patterned fabrics, and a dual-threshold segmentation post-processing optimizes detection results. In this recurrent framework, an over-segmentation method was implemented into model training, and the post-processing steps regarding clustering, threshold limitations, etc. were applied to coalesce over-segmented model outputs into the expected clustering number, further improving segmentation effectiveness.

Moreover, conditional random fields (CRFs) tend to be employed as a post-processing tool to improve the algorithm performance in that they are able to consider the feature information of the entire sequence and better capture the dependencies between sequences. HOTCRF-PCANet ^[73], combining high-order triplet CRFs and unsupervised principal component analysis network (PCANet), was proposed for effective segmentation of nonstationary SAR images, enhancing label consistency and feature representation. GAN continuously learns from the generative network and discriminative network through game theory in the framework to achieve better distribution convergence, where the one forges while the other performs detection, allowing models to train each other and achieve equilibrium. Although GAN achieves high performance in the game shown in Figure 8, the relationships between pixels are easily ignored, resulting in feature loss and segmentation results with discontinuity or significant deformation. Therefore, CRFs serve as a post-processing approach to capture entire sequence features and better-structured segmentation.

Figure 8. The model structure of GAN. GAN: Generative adversarial network.

3.4.1. Frameworks with improved theory

Some frameworks improve the construction mechanism of DL model in accordance with traditional image segmentation conception. Some schemes generate a joint distribution of two adjacent pixels based on similar semantic information of the same object, and obtain the segmentation result according to the similarity distribution on the centroid of a series of clusters. The segmentation route of ACMs is applied into DL image segmentation, which includes determining an initial contour, calculating the offset of the control point on the contour, shrinking the contour curve according to this offset to approximate the target boundary, and then segmenting out contours. Moreover, there are models to blend the concept that pixels with similar features are iteratively fitted to a region from GrabCut into DL-based segmentation frameworks, such as DeepCut ^[159].

Unsupervised clustering is applied to aggregate similar pixel-level features into a certain number of clusters, and each cluster represents a kind of semantic object. DeepClustering ^[160] exploits a simple process to generate the semantic distribution. In DeepClustering, K-means algorithm is used to cluster the eigenvalues of the model output for generated classification pseudo-labels. Thereafter, artificial labeling of general classification tasks is replaced by pseudo-labels generated by K-means to train neural network classifiers. As one of the earliest self-supervised representation learning methods, this paradigm has not achieved considerably acceptable performance, while its ideas have inspired numerous later works. Online deep clustering (ODC) ^[70] further improves the updating scheme of cluster centers in each iteration. The original updating strategy of resetting cluster centers in iteration is improved into an iterative updating method, and the two adjacent cluster centers are updated using the optimal matching algorithm, alleviating convergence instability to a certain extent.

The segmentation train of thought regarding Snake model ^[19] is fused into DL for accurate contour segmentation. Within the bounding box provided by the object detector, DeepSnake first generates an initial contour, and then deforms the contour by predicting vertex offsets to accurately match the object shape, implying the possibility of applying structured contour feature learning to instance segmentation. It uses cyclic convolution to efficiently learn contour features, reducing reliance on precise bounding boxes and enhancing the ability to handle object localization errors. Compared with DeepSnake, an end-to-end contour-based method (E2EC) ^[79] firstly learns the initialization contour by performing backbone operations on the image to generate a heatmap for the location of instance target centers and regresses the initial bias based on the center point features. Then, the generated initialization contour is deformed through a global deformation module to obtain a coarse contour, and finally two deformations are implemented to obtain a fine contour. These models pay special attention to boundary detection and segmentation of complex objects. They effectively handle objects with irregular shapes and fuzzy boundaries, and understand the shape and structure of objects in different contexts, thereby improving segmentation accuracy. Although the specific implementation and optimization strategies differ, they focus on real-time performance, especially in industrial application scenarios that require rapid response.

3.4.2. Frameworks with designed loss functions

In DL, all algorithms rely on minimizing or maximizing a function, called loss function. The loss function is used to measure the error between predicted values and their corresponding true values. When the model is trained, the loss function is minimized or maximized to make the model reach the convergence state and update model parameters to reduce the error of model prediction values. Therefore, the impact of different loss functions on the model is crucial. With help of additional prior information, such as shape, region and edge, traditional image segmentation algorithms achieve promising performance.

LevelSet R-CNN ^[161] is a deep variational instance segmentation framework, combining the architecture of CV model and Mask R-CNN, and its loss functions are designed corresponding to CV model and Mask R-CNN. Mask R-CNN is intended to categorize all objects in an image, and then a RoI is obtained by convolution operation to initialize a truncated signed distance function (TSDF), and instance hyperparameters. TSDF, hyperparameters and feature tensors are inputted into a CV optimization model to output a target TSDF, and a mask is produced by applying Heaviside function to this TSDF. Accordingly, a loss function on the basis of final and initial TSDFs is established to calculate the error between ground truth and prediction for joint end-to-end model training. Later, Nouri et al. utilized an original image and its encoded image generated by local word directional pattern texture descriptor as inputs to CNN for parameter maps and GVF maps, and introduced the loss expression concerning predicted ACM parameter and GVF maps for more accurate and robust fuzzy boundary segmentation^[90].

The loss function according to traditional image segmentation algorithms is defined to optimize segmentation results or constraint prediction contour length for less non-target prediction results. Due to the striking similarity between the softmax layer in those networks and these feature functions in MS model, the LSM is naturally incorporated into CNN. The weakly-supervised deep level set model ^[91] combines a multibranch structure with a self-supervised objective, and presents an adaptive algorithm with level set function as a self-supervised loss term, effectively segmenting aeroengine defects in videoscope images and achieving real-time processing on Turbo19 dataset. These designed loss functions can not only serve as a regularization term for supervised neural network segmentation, but also as a semi-supervised or unsupervised segmentation since it is difficult to obtain the dataset required for training supervised segmentation networks.

3.4.3. Frameworks with pre-processing and post-processing

DL is suitable for high-dimensional data processing to solve problems that traditional image algorithms find difficult to solve, such as diverse defect segmentation and detection. However, DL devices have high-performance requirements, high hardware consumption, and high model training costs. In addition, DL segmentation has a dependency on training data, and generally speaking, the larger the amount of data, the better its performance. Therefore, some researchers, starting from optimizing model training efficiency and smoothing segmentation boundaries, utilize traditional image segmentation as part of model pre-processing or post-processing. These approaches rely on traditional image processing in data pre-processing to greatly reduce the learning pressure of the model and noise interference.

Due to sample distribution imbalance and interference such as noise and lighting changes in image datasets, there are DL-based approaches to utilize traditional strategies such as pre-processing for noise reduction and smoothing. Lin et al. proposed ScribbleSup that iteratively trains segmentation networks and uses graph cuts to correct pixel labels^[162]. ScribbleSup first divides the image into superpixel forms, and then corresponds scribble annotations to the superpixel annotations, executing the graph-cuts algorithm on superpixels. Feng et al. adopted the graph cut scheme to obtain more effective supervision information according to activation seeds from a classification network, and employed fully-connected CRF for segmentation smoothing^[74].

Moreover, DL-based unsupervised algorithms generally require more domain knowledge to optimize models, and post-processing is hence adopted to improve segmentation efficiency. Raja et al. exploited a non-local mean filter to denoise in the pretreatment stage, and then Bayesian fuzzy clustering, deep autoencoder and softmax regression were jointly suggested to complete brain tumor segmentation and classification^[163]. S2VNet ^[106] is a general purpose framework intended to address medical image segmentation problems, using clustering techniques to initialize cluster centers from prior slices and perform fast inference speed and less memory consumption with 2D networks compared to mainstream 3D solutions.

3.4.4. Discussion on traditional and DL-based segmentation

Traditional methods are relatively easy to implement and suitable for rapid development and prototype design. In industrial scenarios with small data volumes and distinct features, traditional methods can achieve good results. Due to their clear features, the results are usually easy to interpret, making it easier for engineers to understand and debug. However, traditional methods are sensitive to image noise and lighting changes, which may result in poor segmentation performance and affect the accuracy of industrial detection. In complex scenes, selecting appropriate features and parameters requires experience and is not suitable for diverse industrial images. When dealing with complex or diverse images, the performance of traditional methods often declines, making it difficult to adapt to different industrial applications.

On large-scale datasets, DL-based methods typically achieve higher segmentation accuracy and have better robustness to noise and lighting changes. These models are able to adapt to different industrial environments and are suitable for complex industrial inspection tasks. Additionally, they automatically learn optimal features, reducing manual intervention without manually selecting features. However, training DL models requires a significant amount of computing resources and time, which is not suitable for resource-limited industrial environments. The demand for annotated data is high, and performance may decrease when there is insufficient data, especially in specific industrial scenarios. The poor interpretability of DL models affects engineers' trust in detection results.

Traditional methods are suitable for scenarios that are simple, feature-rich, and require high real-time and interpretability, while DL methods perform well in applications with complex backgrounds, high-precision requirements, and large-scale datasets. It is crucial to choose appropriate segmentation strategies based on specific industrial needs, which may require comprehensive consideration of characteristics from practical applications and resource limitations. Therefore, designing pre-processing, post-processing or loss functions using traditional methods is an effective component for weakly-supervised or unsupervised segmentation, when training supervised segmentation networks is challenging for lack of uniform and efficient datasets.

4.2.1. Dataset for daily scenarios

COCO dataset ^[164] created by Microsoft covers 80 categories of common objects and scenes, such as humans, animals, vehicles, furniture, etc. Each image is equipped with dense pixel-level segmentation annotations to identify boundaries of each object. COCO dataset consists of three subsets: training, validation, and test sets. The training set contains approximately 118, 000 images, the validation set has approximately 5, 000 images, and the test set includes approximately 20, 000 images.

Pascal VOC dataset ^[165] comes from the Pascal VOC Challenge and is a popular benchmark dataset used to evaluate image segmentation algorithms. It contains images of 20 object categories such as humans, vehicles, animals, and 10 action categories. Moreover, pixel-level annotations are provided in Pascal VOC for each image. This dataset contains 11, 540 images for object detection and classification recognition, and 6, 929 images for image segmentation.

4.2.2. Dataset for surface defect detection

KolektorSDD includes defect images of electronic commutators. Specifically, there are minor damages or cracks on the plastic embedded surface of the electronic commutator. The image is collected under controlled conditions such as uniform lighting. Provided pixel-level annotation of defects for defect images. This dataset includes eight non-overlapping images collected from each surface of 50 defective electronic commutators, resulting in a total of 399 images, including 52 defect images and 347 defect-free images.

NEU surface defect database contains 1, 800 grayscale images about surface defects in hot-rolled strip steel. Six defect categories including scale, patches, cracking, pitted surface, inclusion, and scratches are collected in NEU. Each category contains 300 images with a size of $$ 200 \times 200 $$. The characteristics of the NEU dataset are significant differences in intra-class defects and certain similarities between inter-class defects, which pose difficulties and challenges for achieving accurate classification in defect detection.

MVTec Anomaly Detection (MVTec AD) dataset contains 5, 354 high-resolution color images of 15 different objects and texture categories, where 3, 629 images were used for training and validation, and 1, 725 images were used for testing. Among 15 categories, five texture categories cover different types of regular textures such as carpets, grids and random textures such as leather, tiles, wood. The remaining ten object categories include rigid objects with a fixed appearance such as bottles, metal nuts and non-rigid and deformable objects such as cables, and some contain natural objects. All image resolutions are between $$ 700 \times 700 $$ and $$ 1, 024 \times 1, 024 $$ pixels, and pixel-level ground truth annotations are provided for all anomalies in MVTec AD.

RSDDs dataset is designed specifically for detecting surface defects in steel rails and includes two types of surface defect datasets: Type-I and Type-II. This Type-I dataset is collected from the fast lane and contains 67 images with $$ 160\times1, 000 $$ pixels. This Type-II dataset is collected from regular/heavy transport tracks and contains 128 images with $$ 55\times1, 250 $$ pixels. Each image in both datasets contains at least one defect, with a complex and noisy background. Defects in RSDDs include various common rail surface defects such as cracks, wear, grooves, and foreign object embeddings, which have been pixel level annotated by some professionals in rail surface detection.

4.2.3. Dataset for other industrial scenes

DAGM-2007 dataset is designed specifically for detecting miscellaneous defects on textured backgrounds in industrial optical inspection. It contains ten sub-datasets, with the first six being training datasets and the last four being testing datasets. The characteristic of DAGM 2007 is that it provides images in grayscale 8-bit PNG format. Each dataset contains 1, 000 defect-free images and 150 defective images, where the "defect free" images display texture backgrounds without miscellaneous defects, while defective images have exactly one marked defect on the texture background.

The high-resolution ship dataset FUSAR Ship is a unified and standardized dataset for ship target recognition. This dataset slice contains 15 key ship targets with 98 subcategories and non-ship interference targets. A total of 16, 144 slices have been accumulated, including 6, 252 ships that match AIS information, 2, 045 strong false alarms such as bright spots resembling ships, 1, 461 bridges and coastlines, 1, 010 coastal areas and islands, 1, 967 complex sea waves and clutter, 1, 785 ordinary sea surface images, and 1, 624 land images. The annotation method for ship slices in the dataset is to expand 256 pixels outward with the center of the minimum outer circle of the target as the midpoint, and store them in a fixed size of $$ 512 \times 512 $$ pixels.

Aitex fabric image dataset contains seven different fabrics with a total of 245 images, each with a resolution of $$ 4, 096\times256 $$. The fabrics in this dataset are mainly in plain colors. Among them, there are 140 flawless images, with 20 images for each type of fabric. There are a total of 105 defect images, including 12 common types of fabric defects in the textile industry. The large size of the image allows users to use different window sizes, thereby increasing the sample size. All defects are annotated at the pixel level, with white pixels representing the defect area and the remaining pixels being black.

Magnetic-tile defect dataset was automatically collected by the Chinese Academy of Sciences, and six common defects of the magnetic tile including crack, blowhole, break, fray, uneven and free images were marked by image acquisition and semantic segmentation. The dataset contains 1, 344 images, each cropped to a RoI with different sizes. Suitable for studying the detection and segmentation techniques of small defects in complex backgrounds, especially those defects with similar colors, textures, and backgrounds.