Using moles and freckles for facial recognition
Every time we open our favorite photo app on our smartphone we are amazed by the fact that Google and Apple are able to accurately identify the people in our photographs. This shows the incredible accuracy that facial recognition of these days has to offer. Recent advances in deep learning, and specifically within the field of convolutional neural networks (CNNs) have significantly improved image recognition and object detection capabilities. Nevertheless, each system has its Achilles heel, and in the case of facial recognition software, a weak spot is one that we may all have encountered at some point ourselves; the difficulty to distinguish between twins.
To tackle the problem of differentiating between twins, we can involve other biometrics in order to nudge the facial recognition system into the right direction. One biometric that may be of specific interest is facial marks, such as moles and freckles. While twins often share similar primary facial features, their facial mark patterns can differ vastly. It even turns out that when humans attempt to quickly judge if two pictures are of the same person, or of a pair of twins, using facial marks as an indicator improves their ability to do so. As such, facial marks clearly offer a possible solution, and exploring the ability to involve them in facial recognition is a direction worth exploring.
The next challenge that we face is, how do we actually detect these facial marks? To do that we can look into two concepts regarding image processing and facial marks, namely:
- Facial mark recognition
- Facial mark detection
Facial mark recognition is the process of identifying whether an image is, or is not, a facial mark. The system receives a small image, which can be a skin patch with a facial mark, a skin patch without a facial mark, or any other picture. It is then the system’s task to determine if the image does feature a facial mark. The system will determine the probability that the image is indeed a facial mark. This process is visualized in figure 1.
Figure 1: Process of facial mark recognition. The model assigns a probability score to an image, and a threshold decides if it is ‘True’ or ‘False’
In addition to recognition, facial mark detection is the process of finding out where in an image facial marks can be found. The systems is given the image of a face, and the output of the system should be a list of coordinates, pointing out where facial marks can be found on the face. Each of these coordinates is also linked to a probability that the coordinate does feature a facial mark. Facial mark detection can be implemented using facial mark recognition models in a framework, however, it can also be done using more complex dedicated architectures.
Experiments
This bachelor thesis first explored the effectiveness of facial mark recognition, using deep neural networks. Advanced convolutional neural networks are trained and tested on an extensive dataset of facial mark images. The performance of the advanced network is compared to the performance of a more simple ‘shallow’ neural network. The result of this experiment will give an indication of the viability of using deep neural networks to recognize facial marks.
Recent advances in deep learning, and specifically within the field of convolutional neural networks (CNNs) have significantly improved image recognition and object detection capabilities.
To explore the ability to detect facial marks, the previously trained recognition models are implemented in a sliding window system. This system uses a sliding window along with multiple recognition models, to detect where facial marks can be found on a face. Aside from this sliding window system, other state-of-the-art object detection models will be trained for the same task, to detect facial marks. The effectiveness of the systems can be compared.
Once multiple facial mark detection systems are set up, they are implemented for facial recognition. The facial mark detection systems will be applied to a dataset of facial images of twins. For each image in the dataset, facial mark detection is performed. The facial mark patterns that are detected will be combined with a conventional facial recognition system. When adding the information derived from the facial mark based approach into the results, the final facial recognition quality may improve.
Results
To interpret the results of these experiments, it is important to be familiar with a couple of concepts. False positive rate (FPR) and true positive rate (TPR) are measures of accuracy for a biometric system. FPR indicated how prone the system is to labelling an image as a positive, whilst it should be a negative. TPR indicated to what extend the system is able to correctly classify positive images as positive. The TPR and FPR both depend on a variable, which is the threshold at which we label a score as being positive or negative. A threshold of 0.5 means that we classify an image as positive when the system gives a >0.5 probability. By varying the threshold, we obtain a range of TPR and FPR values. The resulting curve of TPR and FPR is called the ROC curve.
Figure 2 show the ROC curve for facial mark recognition using a deep convolutional neural network, called MobileNet V2. This curve is plotted alongside the ROC curve for a shallow convolutional neural network. From the figure it becomes clear that the deep network outperforms the shallow network significantly. It displays a greater TPR and a lower FPR than the shallow network.
Figure 2: ROC curve for facial mark recognition comparing a deep CNN (MobileNet V2) to shallow CNN’s.
This experiment shows the potential of using deep neural networks for the purpose of facial mark recognition.
Using the deep models trained in the previous experiments, an implementation for facial mark detection is drawn up. For this, a nested sliding window approach is implemented. The nested sliding window approach slides a window over a facial image, checking if each iteration of the window contains a facial mark. It initially does this using a relatively large window. After the first stage, each positive window is again analyzed using a sliding window, but this time using a much smaller window. This stage is repeated with a third, even smaller, window. The result is fine-grained detection of facial marks. Figure 3 shows the process of the three stages of detection.
Figure 3: The 3 consecutive stages of the nested sliding window facial mark detection process.
This custom approach to facial mark detection is able to accurately and consistently detect facial marks. To compare, and to further explore the ability of deep networks to detect facial marks, other object detection systems are also deployed for facial mark detection. These systems are trained and tested, and can be compared side-by-side. Figure 3 shows a side-by-side comparison of the facial marks detected by three different systems.
Figure 4: Facial mark detection results from three architectures: Sliding window (left), FRCNN (center), EfficientDet (right).
Each system detects many of the same apparent facial marks, with some minor differences. Now that the facial mark detection systems are in places, the facial mark detection can be combined with facial recognition, to see if the results improve. The combined system is drawn up in figure 4.
Figure 5: Architecture of the combined facial recognition system, implementing facial mark detection.
By testing for the overlap between facial mark patterns of facial images, a facial mark based similarity score is assigned. This score is combined with the score of the conventional facial recognition, which used facial feature extraction, as seen in figure 4. Figure 5 shows the ROC curve for facial recognition between the faces of twins using only the facial recognition system, and using the facial recognition system in combination with various facial mark detection systems. It is obvious that the performance of the facial recognition system on its own is rather poor, this is due to the anticipated difficulty the system has with differentiating between twins. When including the facial mark data, the performance increases significantly.
Figure 6: ROC curve for facial recognition between twins, with and without implementing facial mark detection. No FM indicates no facial mark detection.
Conclusively, we can state that using facial mark detection significantly improves the ability of a facial recognition system to differentiate between faces. The facial mark detection algorithms are at this point much slower that the facial recognition system on its own, making it an application that would not be directly feasible to be used in a real-world application. Nevertheless, this shows the great potential of facial marks, and is a step into the right direction for twin facial recognition.
