Intelligent manufacturing refers to a new production model. Manufacturing machines are fully connected through the network, monitored by sensors, and intelligently controlled by advanced computing. It improves product quality, system productivity, and sustainability  . As a powerful tool for product quality control, machine vision systems have been widely studied and applied. Machine vision is a technique measured and judged by machines instead of human eyes. However, it is not only limited to human eyes, but also the function of the human’s brain to extract information from the image of the target object, and the information is to be understood, analyzed and processed, what’s more, making feedbacks from the results and then applied on detection, location, and control. Today its applications include object measurement  , target recognition , vision-based control   and defective product inspection  .
Surface defect detection is a key step in many industrial processes. With the requirement of high-quality inspection, the advanced computer vision technology has developed into the mainstream, and it has replaced the conventional manual inspection. Manual detection methods have disadvantages such as low sampling rate, low accuracy, poor real-time performance, low efficiency, high labor intensity, and large influence by the competency of employees. In recent years, with the continuous development of computer technology and image processing technology, the disadvantage of manual detection is gradually overcome by the detection method based on machine vision. Among them, the online detection technology based on image processing  has outstanding advantages such as non-contact, fast speed, and good flexibility, and has been more and more widely studied and applied in modern industry.
Surface defects in the industry are usually defined as a local anomaly embedded in uniform texture. To realize automatic and non-destructive inspection, the visual inspection system (VISs) has been widely used in the surface inspection of steel , slabs , rails , structures , and glass ampoule packaging  application. Generally, these inspection techniques can be roughly divided into the following three categories: spectral, statistical, and model-based methods . The basic idea of the spectral methods is to transform the images into the frequency domain. In the frequency domain, the responses of the defects and the defect-free background are expected to be different, so the defects can be identified. Obviously, this method is suitable for surfaces with regular patterns and periodic textures, such as fabrics , but the choice of filters will affect its performance to a certain extent. Statistical methods usually use co-occurrence matrix , histogram , and other texture statistics to detect surface defects. Unfortunately, both spectral methods and statistical methods are difficult to deal with random changes on random texture surfaces . Based on the Markov random field model, a supervised method  was developed to monitor and diagnose stochastic textured surfaces. Typically, model-based approaches tend to require high computational complexity .
In recent years, well-designed deep convolutional neural networks have become powerful tools in various computer vision tasks. As a result, some studies have tried to use deep learning (DL) methods for defect inspection  , but these methods still face huge challenges. As mentioned in , training a DL model usually requires quite a lot of samples, and collecting and labeling a large number of defective images can be expensive. In addition, DL methods rely on data sets with specific supervision information. Therefore, the learning model is tightly coupled with a specific data set and may not perform well on other data sets .
However, most of these methods are task-specific, so it is difficult to convert them into similar detection schemes for expanded thermoplastic polyurethane (E-TPU) midsole. So, we propose a defect detection method for E-TPU midsole. The seven kinds of defects, including scorch, color pollution, hole, impurity, dirty, indentation and convex marks, were divided into three kinds of color defects, block defects and linear defects. Different detection methods were designed for different kinds of defects.
2. Materials and Methods
2.1. E-TPU Shoe Midsole
E-TPU is a new type of thermoplastic polyurethane (TPU) foam material made by the physical foaming of thermoplastic polyurethane particles, as shown in Figure 1. The polymer material is formed by changing and compounding high resilience foam particles through the structure of TPU particles, and is composed of numerous elastics and light TPU foam particles. It is a non-toxic, biodegradable, and recyclable new environmentally friendly material. It has the advantages of low density, high resilience, wear resistance, corrosion resistance, and good low-temperature performance. It can be widely used in insoles, cushions, pads, solid tires, marine floats, packaging, and shipping.
In this research, the hardware environment is Intel(R) Core (TM) i5-6600K @ 3.50 GHz * 4 CPU, 8 GB of memory, and the development environment is MATLAB 2018b. The image acquisition equipment is two strip light sources and an industrial camera, as shown in Figure 2. The pictures used in this paper have been processed by previous work.
2.2. The Method of This Paper
2.2.1. Image Preprocessing
Surface defects are generally defined as a local anomaly embedded in a uniform texture, and the defect area is smaller than the background area. Compared with
Figure 1. E-TPU material.
Figure 2. Image acquisition equipment.
other midsole defect, the scorch defect is a relatively special defect. It has wide distribution range and large area. Using other detected defects method in this paper, it will be recognized as a texture, resulting in missed inspection. But it is special in color. The E-TPU midsole is white, and its defects are white or black. There are only scorch and color pollution with different colors. At the same time, these two types of defects have obvious characteristics. The scorch accounts for a relatively large proportion, while the color pollution accounts for a relatively small proportion. It is easy to detect and classify these two kinds of defects in the preprocessing process.
Each pixel in an RGB image is composed of R, G, and B components, where R, G, and B are described by different gray levels. Black is composed of (0, 0, 0), and white is composed of (255, 255, 255). In the actual collection process of image, due to different reasons such as light source and material, this effect cannot be achieved. But in general, the gray values of the three channels are similar, as shown in Figure 3.
, , and represent the R, G, B components of respectively. The variance can be calculated by the following formula:
The image preprocessing flow chart is shown in Figure 4. Where N represents the total number of pixels in the detected image, num represents the total number of pixels that do not meet the judgment conditions. So, represents the proportion of defect area in the detected image.
The threshold value of is 30, it is used to determine whether the pixel for defects. The threshold value of θ is 0.01, it is used to distinguish scorch and color pollution. However, if the θ is less than 0.001, such defect size is acceptable, and conducts the next test it is also judged as a qualified product for the next step of testing. However, if B is less than 0.001, such defect size is acceptable, the midsole image will be tested next step.
Figure 3. The three-component gray value of each area. (a) Schematic diagram of the same three-component gray value; (b) Three-component gray value of scorch defect; (c) Three-component gray value of dirt defect.
Figure 4. Image preprocessing flow chart.
2.2.2. Block Defect Detection
After the image preprocessing, the image is then partitioned into compact and perceptually homogeneous superpixels. There are two reasons for this process. Since the number of superpixels in an image is much smaller than the number that of pixels, it is desirable to use superpixels as the smallest processing unit to reduce the computational burden. More importantly, some defects may occupy contiguous areas. When a defect image is segmented into superpixels, there are only few superpixels for corresponding to the defect .
1) Superpixel segmentation
Algorithms for generating superpixels can be roughly classified into graph-based methods or gradient ascent methods. The graph-based superpixel generation method regards each pixel as a node in the graph. The edge weight between two nodes is proportional to the similarity between adjacent pixels. Superpixels are created by minimizing a cost function defined over the graph. The gradient ascent-based algorithms start from the rough initial clustering of pixel until certain convergence criteria are met to form superpixels. This paper chooses Simple Linear Iterative Clustering (SLIC)  because it exhibits good boundary dependence.
We estimate that the number of pixels contained in each superpixel is 32 * 32. This is because the pixels number of the circular texture diameter in the qualified midsole image is 32, as shown in Figure 6(a). We hope that these textures can be segmented in the same superpixel. After the superpixel segmentation, the average gray value of the pixel contained in the superpixel is used as the value of the superpixel. And the value of each superpixel can be expressed as
where is the number of pixels included in the i-th superpixel, is the gray value of each pixel included in the i-th superpixel.
The image after superpixel segmentation can be expressed as
where C represents the number of the superpixel. and is a matrix composed of the gray value of each superpixel.
Figure 6(a) shows an image of qualified midsole. The superpixel segmentation effect is shown in Figure 6(b). The curve of is shown in Figure 6(c), the gray value of superpixel is increasing, indicating that the illumination is not uniform when image acquisition.
2) Judge the defective superpixel
In a superpixel segmentation image, the adjacent elements of each superpixel are different from the adjacent pixel of the original image. It is more irregular and the number of adjacent elements is not fixed, as shown in Figure 5.
Because the number of superpixel adjacent elements is not fixed, it is easy to cause errors in defect detection. In this paper, the background prior (BP) mining method is used to estimate the background mean. In , Wang et al. extracted the 15 dimensions (15-D) statistical features, so they used ||*||2 to calculate BP. In this paper, the gray value was only used as the calculation parameter, and the calculation method was redesigned as follows:
M superpixels are randomly selected to express as , , ∙∙∙, , and the background mean can be expressed as the following formula:
For each superpixel, BP can be expressed as the following formula:
The curve of BP is shown in Figure 6(d). The threshold value of 40 is selected. If the BP exceeds the threshold value, this superpixel is considered to be a defect.
Figure 5. The adjacent elements of individual superpixels after superpixel segmentation.
Figure 6. Superpixel segmentation. (a) Original image; (b) Superpixel Segmentation; (c) FC curve; (d) BP curve.
2.2.3. Linear Defect Detection
In the research of E-TPU midsole, there are two types of linear defects: indentation and convex marks. After the image registration, the linear defects are only distributed at about 0 degrees and 90 degrees, that is horizontal straight lines and longitudinal straight lines. In the detection of linear elements, the Hough transform has excellent effects, but the Hough transform can only process binary images. Generally, it should be the output image after edge detection. Therefore, the Gabor filter is preferentially selected for feature extraction in these two directions, and then the Hough transform is used to perform the straight-line inspection.
1) Gabor filter
Gabor wavelet is an important tool in the field of computer vision, and very similar to the visual stimulus-response of simple cells in the human visual system . It has good characteristics in extracting the local space and frequency domain information of the target. Although Gabor wavelet itself cannot form an orthogonal basis, it can form a tight frame under certain parameters. Gabor wavelet is sensitive to the edge of the image, can provide good direction selection and scale selection characteristics, and is not sensitive to changes in illumination, and can provide good adaptability to changes in illumination.
Two-dimensional Gabor wavelet transform is an important tool for signal analysis and processing in the time-frequency domain . Its transform coefficients have good visual characteristics and biological background, so they are widely used in image processing, pattern recognition, and other fields. Compared with the traditional Fourier transform, the Gabor wavelet transform has good time-frequency localization characteristics. In terms of feature extraction, Gabor wavelet transform is compared with other methods: on the one hand, it processes fewer data and can meet the real-time requirements of the system; on the other hand, wavelet transform is insensitive to changes in illumination and can tolerate a certain degree of image rotation and deformation. Figure 7 shows the feature extraction results of the E-TPU midsole.
2) Hough transform
In 1962, Huff obtained a method that can effectively identify lines in an image, called Hough transform . Even now, it is still a very important tool . In addition, Kalviainen et al.  proposed a Random Hough Transform (RHT). RHT selects n pixels from the edge image by random sampling to solve the n parameters of the curve, and then only accumulates 1 pixel in the parameter space. Compared with other Hough transforms, this method has the advantages
Figure 7. Feature extraction of Gabor operator. (a) Original image; (b1) Gabor kernel function (90˚) renderings; (b2) Gabor kernel function (0˚) renderings; (c1) Longitudinal straight line test results; (c2) Horizontal straight line detection result.
Figure 8. Hough transform detection process. (a) Original image; (b1) Binarize the Gabor operator’s longitudinal line detection result; (b2) Binarize the Gabor operator’s horizontal line detection result; (c1) Morphological removal of interference; (c2) Morphological removal of interference; (d1) Hough space; (d2) Hough space; (e1) Longitudinal straight line test results; (e2) Horizontal straight line detection result.
of high-speed expansion and low memory usage. Figure 8 shows the straight-line test results of convex marks.
2.2.4. Inspection Process
In the preparation for defect detection, we have performed edge detection and image registration on the collected E-TPU midsole images.
In this paper, image preprocessing is used to detect the scorch and color pollution defects. The main purpose is to prioritize the detection and classification of scorch defects. Because it will cause the wrong straight lines to be detected during linear defect detection, and do not have a simple method to judge whether it is a defect during block defect detection.
In the process of the block defect detection, superpixel segmentation cannot effectively segment linear defects, and linear defect detection also cannot identify block defects. Therefore, parallel computing based on the graphics processing unit (GPU)  is used to perform two types of detection methods at the same time, as shown in Figure 9. It increases the operating speed, maximizes the function of the hardware equipment, and detects whether there are multiple types of defects in a midsole.
3. Results and Analysis
3.1. Image Preprocessing
The two types of defects, scorch, and color pollution, are respectively tested, and the detection results are shown in Figure 10. Intuitively represent Dc as an image as shown in Figure 10(b), and use Otsu to binarize it as shown in Figure 10(c). It can be seen that scorch and color pollution can be easily identified by setting a threshold according to the experience.
It can be seen from Table 1 that max(Dc) can effectively distinguish scorch, color pollution and other types of defects. It is appropriate to select a threshold of 30. Obviously, the proportion of scorch defect areas is much larger than the percentage of color pollution defects. So the threshold value of θ is selected as
Figure 9. Flow chart of the proposed method in this paper.
Figure 10. The detection results of scorch and color pollution. (a) Scorch; (b) Three-component variance Dc; (c) Dc binary image and detection results; (d) Color pollution; (e) Magnified view of color pollution; (f) Dc binary image and detection results
Table 1. Three-component variance DC and the proportion of defects.
0.01. If the proportion of defect areas is greater than 0.01, it is judged as a scorch defect, otherwise, it is judged as color pollution. But when the color pollution part is particularly small, such a defect is considered acceptable.
3.2. Superpixel Segmentation
After the superpixel segmentation of midsole, the difference of local regions can be distinguished by calculating the superpixel neighborhood difference (SND). The average gray values difference between a superpixel and its neighboring superpixel is expressed as the following formula
where represents the superpixel adjacent to the i-th superpixel, represents the number of adjacent superpixels.
It can be seen from Figure 11 that SND will also determine the superpixels around the defect areas as defects. It can be used only to detect the existence of defects, but the specific location of defects cannot be determined.
It can be analyzed from Figure 11 and Figure 12 that the larger the proportion of defects in the entire image, the worse the SND detection effect after superpixel segmentation. The reasons are that the defect is also segmented into multiple superpixels, which leads to the superpixels of defects center will be judged as background areas, as shown in Figure 12(c4).
In fact, superpixel segmentation can also better segment scorch defects, as shown in Figure 13(a) and Figure 13(b), but the defect areas of the midsole are larger and the defects distribution is relatively homogeneous. There is no simple criterion to judge whether a superpixel is defect after segmentation. Even if a specific judgment method is used to determine that a superpixel is defect, it still requires other judgment conditions to distinguish the type of defects. As shown in Figure 13(c) and Figure 13(d), Superpixel segmentation cannot achieve the effect with linear defects on the midsole surface, so a separate method is used to
Figure 11. Analysis of BP and SND detection results for impurities defect. (a) Impurities; (b) Superpixel segmentation results; (c) BP curve of Superpixel; (d) SND curve of Superpixel; (e) BP detection results; (f) SND detection results.
Figure 12. Superpixel segmentation results and BP and SND detection results of three kinds of defect. (a1) Hole; (a2) Superpixel segmentation results; (a3) BP detection results; (a4) SND detection results; (b1) Impurity; (b2) Superpixel segmentation results; (b3) BP detection results; (b4) SND detection results; (c1) Dirt; (c2) Superpixel segmentation results; (c3) BP detection results; (c4) SND detection results.
Figure 13. Scorch and convex marks by superpixel segmentation. (a) Scorch; (b) Superpixel segmentation results; (c) Convex marks; (d) Superpixel segmentation results.
detect linear defects.
3.3. Linear Defect Detection
There are only two types of linear defects: indentation and convex marks. The defect detection result is shown in Figure 14. Whether the detected straight line is a defect is judged based on the length. Figure 14(a1)-(a4) list the inspection results of convex marks, and Figure 14(b1)-(b4) lists the inspection results of indentation. The test results show that the proposed method can effectively and accurately identify linear defects on the midsole surface.
3.4. Analysis of the Accuracy and Operating Speed
The inspection time of each part is shown in Table 2. Among them, the scorch and color pollution defects can be identified during image preprocessing, and the latter two steps are only to test the detection effect. It can be seen from Table 2
Figure 14. Test results for indentation and convex marks. (a1) Convex marks; (a2) Test results of longitudinal straight line; (a3) Test results of horizontal straight line; (a4) Final test result; (b1) Indentation; (b2) Test results of longitudinal straight line; (b3) Test results of horizontal straight line; (b4) Final test result.
Table 2. Detection time.
Table 3. Detection accuracy.
that the surface defect detection of the E-TPU midsole can be completed in 1.6 s.
In this paper, the false positive rate (FPR) and the false negatives rate (FNR) are calculated for performance evaluation. FPR and FNR are defined as follows:
That is, FPR is defined as the proportion of complete qualified samples that are falsely detected as defects, and FNR is the proportion of defective samples that cannot be accurately detected defect.
It can be seen from Table 3 that the FPR of this method is 8.3%, the FNR for indentation defect is 2.7%, and the FNR for hole defect is 4.1%. This is because some midsole, though containing indentation and hole, are less damaged and not well recognized by machine vision. But overall, the proposed method in this paper has a higher accuracy rate.
This paper proposes a surface defect detection method of E-TPU midsole based on machine vision, including image preprocessing, block defect detection, and linear defect detection. Image preprocessing can detect scorch and color pollution defects. Block defect detection can detect holes and dirt. Linear defect detection can detect indentation and convex marks. The FAR is 8.3%, the DR is 6.8%, using parallel computing to perform block defect detection and linear defect detection at the same time, and the total running time does not exceed 1.6 s, which can meet the industrial production application of shoe midsole.
Although the proposed method in this paper can accurately identify all kinds of defects in the E-TPU midsole, it can only classify the seven kinds of defects into three categories in terms of defect classification. For example, indentation and convex marks can only be judged as linear defects, and cannot accurately distinguish specific defect type. So, the defects classification still needs further study.
This work was supported by the Key Laboratory of Green Manufacturing of Super-light Elastomer Materials of State Ethnic Affairs Commission (Hubei Minzu University) PT092004.
 Wang, J.J., Ma, Y.L., Zhang, L.B., Gao, R.X. and Wu, D.A. (2018) Deep Learning for Smart Manufacturing: Methods and Applications. Journal of Manufacturing Systems, 48, 144-156.
 Luo, W.C., Hu, T.L., Zhang, C.R. and Wei, Y.L. (2019) Digital Twin for CNC Machine Tool: Modeling and Using Strategy. Journal of Ambient Intelligence and Humanized Computing, 10, 1129-1140.
 Ma, Y.P., Li, Q.W., Xing, J., Huo, G.Y. and Liu, Y. (2019) An Intelligent Object Detection and Measurement System Based on Trinocular Vision. IEEE Transactions on Circuits and Systems for Video Technology, 30, 711-724.
 Tootooni, M.S., Liu, C., Roberson, D., Donovan, R., Rao, P.K., Kong, Z.J. and Bukkapatnam, S.T. (2016) Online Non-Contact Surface Finish Measurement in Machining Using Graph Theory-Based Image Analysis. Journal of Manufacturing Systems, 41, 266-276.
 Fernández-Robles, L., Azzopardi, G., Alegre, E., Petkov, N. and Castejon-Limas, M. (2017) Identification of Milling Inserts in Situ Based on a Versatile Machine Vision System. Journal of Manufacturing Systems, 45, 48-57.
 Wang, T.J., Kwok, T.H., Zhou, C. and Vader, S. (2018) In-Situ Droplet Inspection and Closed-Loop Control System Using Machine Learning for Liquid Metal Jet Printing. Journal of Manufacturing Systems, 47, 83-92.
 Zhao, Y.S., Gong, L., Huang, Y.X. and Liu, C.L. (2016) A Review of Key Techniques of Vision-Based Control for Harvesting Robot. Computers and Electronics in Agriculture, 127, 311-323.
 Kuo, C.F.J., Lo, W.C., Huang, Y.R., Tsai, H.Y., Lee, C.L. and Wu, H.C. (2017) Automated Defect Inspection System for CMOS Image Sensor with Micro Multi-Layer Non-Spherical Lens Module. Journal of Manufacturing Systems, 45, 248-259.
 Wang, J.J., Fu, P.L. and Gao, R.X. (2019) Machine Vision Intelligence for Product Defect Inspection Based on Deep Learning and Hough Transform. Journal of Manufacturing Systems, 51, 52-60.
 Liu, K., Wang, H.Y., Chen, H.Y., Qu, E.Q., Tian, Y. and Sun, H.X. (2017) Steel Surface Defect Detection Using a New Haar-Weibull-Variance Model in Unsupervised Manner. IEEE Transactions on Instrumentation and Measurement, 66, 2585-2596.
 Ramana, L., Choi, W. and Cha, Y.J. (2019) Fully Automated Vision-Based Loosened Bolt Detection Using the Viola-Jones Algorithm. Structural Health Monitoring, 18, 422-434.
 Li, Q.Y., Shi, Z.P., Zhang, H.Y., Tan, Y.Q., Ren, S.W., Dai, P. and Li, W.Y. (2018) A Cyber-Enabled Visual Inspection System for Rail Corrugation. Future Generation Computer Systems, 79, 374-382.
 Yang, Y., Hirose, S., Debenest, P., Guarnieri, M., Izumi, N. and Suzumori, K. (2016) Development of a Stable Localized Visual Inspection System for Underwater Structures. Advanced Robotics, 30, 1415-1429.
 Rawashdeh, N.A., Abu-Khalaf, J.M., Khraisat, W. and Al-Hourani, S.S. (2018) A Visual Inspection System of Glass Ampoule Packaging Defects: Effect of Lighting Configurations. International Journal of Computer Integrated Manufacturing, 31, 848-856.
 Wang, J.Z., Li, Q.Y., Gan, J.R., Yu, H.M. and Yang, X. (2019) Surface Defect Detection via Entity Sparsity Pursuit with Intrinsic Priors. IEEE Transactions on Industrial Informatics, 16, 141-150.
 Kumar, A. and Pang, G.K.H. (2002) Defect Detection in Textured Materials Using Optimized Filters. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 32, 553-570.
 Halfawy, M.R. and Hengmeechai, J. (2014) Automated Defect Detection in Sewer Closed Circuit Television Images Using Histograms of Oriented Gradients and Support Vector Machine. Automation in Construction, 38, 1-13.
 Chandra, J.K. and Datta, A.K. (2013) Detection of Defects in Fabrics Using Subimage-Based Singular Value Decomposition. Journal of the Textile Institute, 104, 295-304.
 Cao, J.J., Zhang, J., Wen, Z.J., Wang, N.N. and Liu, X.P. (2017) Fabric Defect Inspection Using Prior Knowledge Guided Least Squares Regression. Multimedia Tools and Applications, 76, 4141-4157.
 Esteva, A., Robicquet, A., Ramsundar, B., Kuleshov, V., DePristo, M., Chou, K., et al. (2019) A Guide to Deep Learning in Healthcare. Nature Medicine, 25, 24-29.
 Ren, R.X., Hung, T. and Tan, K.C. (2017) A Generic Deep-Learning-Based Approach for Automated Surface Inspection. IEEE Transactions on Cybernetics, 48, 929-940.
 Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P. and Süsstrunk, S. (2012) SLIC Superpixels Compared to State-of-the-Art Superpixel Methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34, 2274-2282.
 Yang, D.D., Jiao, L.C., Gong, M.G. and Liu, F. (2011) Artificial Immune Multi-Objective SAR Image Segmentation with Fused Complementary Features. Information Sciences, 181, 2797-2812.
 Yuen, H.K., Illingworth, J. and Kittler, J. (1989) Detecting Partially Occluded Ellipses Using the Hough Transform. Image and Vision Computing, 7, 31-37.
 Zana, F. and Klein, J.C. (1999) A Multimodal Registration Algorithm of Eye Fundus Images Using Vessels Detection and Hough Transform. IEEE Transactions on Medical Imaging, 18, 419-428.