Micro-expression recognition algorithm based on separate long-term recurrent convolutional network
-
摘要: 基于面部动态表情序列，针对静态表情缺少时间信息等问题，将空间特征与时间特征融合，利用神经网络在图像分类领域良好的特征，对需要进行细节分析的表情序列进行处理，提出基于分离式长期循环卷积网络(Separate long-term recurrent convolutional networks, S-LRCN)的微表情识别方法。首先选取微表情数据集提取面部图像序列，引入迁移学习的方法，通过预训练的卷积神经网络模型提取表情帧的空间特征，降低网络训练中过拟合的危险，并将视频序列的提取特征输入长短期记忆网络(Long short-team memory, LSTM)处理时域特征。最后建立学习者表情序列小型数据库，将该方法用于辅助教学评价。Abstract: With the rapid development of machine learning and deep neural network and the popularization of intelligent devices, face recognition technology has rapidly developed. At present, the accuracy of face recognition has exceeded that of the human eyes. Moreover, the software and hardware conditions of large-scale popularization are available, and the application fields are widely distributed. As an important part of face recognition technology, facial expression recognition has been a widely studied subject in the fields of artificial intelligence, security, automation, medical treatment, and driving in recent years. Expression recognition, an active research area in human–computer interaction, involves informatics and psychology and has good research prospect in teaching evaluation. Micro-expression, which has great research significance, is a kind of short-lived facial expression that humans unconsciously make when trying to hide some emotion. Different from the general static facial expression recognition, to realize micro-expression recognition, besides extracting the spatial feature information of facial expression deformation in the image, the temporal-motion information of the continuous image sequence also needs to be considered. In this study, given that static expression features lack temporal information, so that the subtle changes in expression cannot be fully reflected, facial dynamic expression sequences were used to fuse spatial features and temporal features, and neural networks were used to provide good features in the field of image classification. Expression sequences were processed, and a micro-expression recognition method based on separate long-term recurrent convolutional network (S-LRCN) was proposed. First, the micro-expression data set was selected to extract the facial image sequence, and the transfer learning method was introduced to extract the spatial features of the expression frame through the pre-trained convolution neural network model, to reduce the risk of overfitting in the network training, and the extracted features of the video sequence were inputted into long short-term memory (LSTM) to process the temporal-domain features. Finally, a small database of learners’ expression sequences was established, and the method was used to assist teaching evaluation.
-
Key words:
- micro-expression recognition /
- spatial-temporal features /
- LRCN /
- LSTM /
- education evaluation
-
表 1 划分情况
Table 1. Dataset classification
Classify CASME-Ⅱ Samples Happiness Happiness (32) 32 Surprise Surprise (28) 28 Disgust Disgust (63) 63 Repression Repression (27) 27 Others Others (99) 105 Sadness (4) Fear (2) 
下载: 导出CSV
表 3 不同算法识别准确率
Table 3. Recognition accuracy of different algorithms
Methods Accuracy/% F1-Score/% LBP-TOP 52.6 42.6 STCLQP 58.6 58.0 CNN+LSTM 61.0 58.5 HOOF+LSTM 59.8 56.0 S-LRCN 65.7 60.8
下载: 导出CSV
表 4 不同序列长度实验效果
Table 4. Experimental results of different sequence lengths
Sequence length Accuracy/% F1-Score/% 6 62.0 56.6 10 65.7 60.8 15 63.1 58.6 30 56.5 49.6
下载: 导出CSV
-
参考文献
[1] Mehrabian A. Nonverbal communication. New York: Routledge, 2017 [2] Ekman P. Facial expression and emotion. Am Psychol, 1993, 48(4): 384 doi: 10.1037/0003-066X.48.4.384 [3] Ekman P, Friesen W V. Nonverbal leakage and clues to deception. Psychiatry, 1969, 32(1): 88 doi: 10.1080/00332747.1969.11023575 [4] Yan W J, Wang S J, Liu Y J, et al. For micro-expression recognition: Database and suggestions. Neurocomputing, 2014, 136(136): 82 [5] Wang S Y. CNN-RNN Based Micro-Expression Recognition [Dissertation]. Harbin: Harbin Engineering University, 2018王思宇. 基于CNN-RNN的微表情识别[学位论文]. 哈尔滨: 哈尔滨工程大学, 2018 [6] Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. Commun ACM, 2017, 60(6): 84 doi: 10.1145/3065386 [7] Donahue J, Hendricks L A, Guadarrama S, et al. Long-term recurrent convolutional networks for visual recognition and description // 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, 2015: 2625 [8] Ekman P, Rosenberg E L. What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). 2nd Ed. USA: Oxford University Press, 2005 [9] Gunes H, Pantic M. Automatic, dimensional and continuous emotion recognition. Int J Synthetic Emotions, 2010, 1(1): 68 doi: 10.4018/jse.2010101605 [10] Song Y L, Morency L P, Davis R. Learning a sparse codebook of facial and body microexpressions for emotion recognition // Proceedings of the 15th ACM International Conference on Multimodal Interaction. New York, 2013: 237 [11] Li D, Xie L, Lu T, et al. Capture of microexpressions based on the entropy of oriented optical flow. Chin J Eng, 2017(11): 1727李丹, 解仑, 卢婷, 等. 基于光流方向信息熵统计的微表情捕捉. 工程科学学报, 2017(11):1727 [12] Polikovsky S, Kameda Y, Ohta Y. Facial micro-expression detection in Hi-speed video based on facial action coding system (FACS). IEICE Trans Inform Syst, 2013, E96-D(1): 81 doi: 10.1587/transinf.E96.D.81 [13] Shreve M, Godavarthy S, Goldgof D, et al. Macro- and micro-expression spotting in long videos using spatio-temporal strain // 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG). Santa Barbara, 2011: 51 [14] Pfister T, Li X B, Zhao G Y, et al. Recognising spontaneous facial micro-expressions // 2011 International Conference on Computer Vision. Barcelona, 2011: 1449 [15] Liang J, Yan W J, Wu Q, et al. Recent advances and future trends in micro-expression research. Bull Natl Nat Sci Foundation China, 2013(2): 75梁静, 颜文靖, 吴奇, 等. 微表情研究的进展与展望. 中国科学基金, 2013(2):75 [16] Wang Y D, See J, Phan R C W, et al. LBP with six intersection points: Reducing redundant information in LBP-TOP for micro-expression recognition // Asian Conference on Computer Vision ‒ ACCV2014. Switzerland, 2015: 525 [17] Liong S T, See J, Phan R C W, et al. Spontaneous subtle expression detection and recognition based on facial strain. Signal Process Image Commun, 2016, 47: 170 doi: 10.1016/j.image.2016.06.004 [18] Le Ngo A C, See J, Phan R C W. Sparsity in dynamics of spontaneous subtle emotion: analysis & application. IEEE Trans Affective Comput, 2017, 8(3): 396 doi: 10.1109/TAFFC.2016.2523996 [19] Xu F, Zhang J P, Wang J Z. Microexpression identification and categorization using a facial dynamics map. IEEE Trans Affective Comput, 2017, 8(2): 254 doi: 10.1109/TAFFC.2016.2518162 [20] Patel D, Hong X P, Zhao G Y. Selective deep features for micro-expression recognition // 2016 23rd International Conference on Pattern Recognition (ICPR). Cancun, 2016: 2258 [21] Kim D H, Baddar W J, Ro Y M. Micro-expression recognition with expression-state constrained spatio-temporal feature representations // Proceedings of the 24th ACM international conference on Multimedia. Amsterdam, 2016: 382 [22] Khor H Q, See J, Phan R C W, et al. Enriched long-term recurrent convolutional network for facial micro-expression recognition // 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2018). Xi’an, 2018: 667 [23] Verburg M, Menkovski V. Micro-expression detection in long videos using optical flow and recurrent neural networks // 2019 14th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2019). Lille, 2019: 1 [24] Itti L, Koch C. Computational modelling of visual attention. Nat Rev Neurosci, 2001, 2(3): 194 doi: 10.1038/35058500 [25] Wang C Y, Peng M, Bi T, et al. Micro-attention for micro-expression recognition [J/OL]. arXiv Preprint (2019-8-27) [2020-04-21]. https://arxiv.org/abs/1811.02360. [26] Parkhi O M, Vedaldi A, Zisserman A. Deep face recognition // Proceedings of the British Machine Vision Conference (BMVC). Swansea, 2015: 45 [27] Hu J, Shen L, Albanie S, et al. Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell, 2020, 42(8): 2011 doi: 10.1109/TPAMI.2019.2913372 [28] Cao Q, Shen L, Xie W D, et al. VGGFace2: A dataset for recognising faces across pose and age // 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2018). Xi’an, 2018: 67 [29] He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, 2016: 770 [30] Cootes T F, Edwards G J, Taylor C J. Active appearance models. IEEE Trans Pattern Anal Mach Intell, 2001, 23(6): 681 doi: 10.1109/34.927467 [31] Peng M. Dual Temporal Scale Convolutional Neural Network for Micro-Expression Recognition [Dissertation]. Chongqing: Southwest University, 2017彭敏. 基于双时间尺度卷积神经网络的微表情识别[学位论文]. 重庆: 西南大学, 2017 [32] Yan W J, Li X B, Wang S J, et al. CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE, 2014, 9(1): e86041 doi: 10.1371/journal.pone.0086041 [33] Zhou Z H, Zhao G Y, Pietikinen M. Towards a practical lipreading system // The 24th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2011). Providence, RI, 2011: 137 [34] Zhao G Y, Pietikainen M. Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell, 2007, 29(6): 915 doi: 10.1109/TPAMI.2007.1110 [35] Huang X H, Zhao G Y, Hong X P, et al. Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing, 2016, 175: 564 doi: 10.1016/j.neucom.2015.10.096 -
点击查看大图
图(11) / 表(4)
计量
- 文章访问数: 1043
- HTML全文浏览量: 426
- PDF下载量: 50
- 被引次数: 0
