• 《工程索引》(EI)刊源期刊
  • 中文核心期刊(综合性理工农医类)
  • 中国科技论文统计源期刊
  • 中国科学引文数据库来源期刊

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于S-LRCN的微表情识别算法

李学翰 胡四泉 石志国 张明

李学翰, 胡四泉, 石志国, 张明. 基于S-LRCN的微表情识别算法[J]. 工程科学学报. doi: 10.13374/j.issn2095-9389.2020.06.15.006
引用本文: 李学翰, 胡四泉, 石志国, 张明. 基于S-LRCN的微表情识别算法[J]. 工程科学学报. doi: 10.13374/j.issn2095-9389.2020.06.15.006
LI Xue-han, HU Si-quan, SHI Zhi-guo, ZHANG Ming. Micro-expression recognition algorithm based on separate long-term recurrent convolutional network[J]. Chinese Journal of Engineering. doi: 10.13374/j.issn2095-9389.2020.06.15.006
Citation: LI Xue-han, HU Si-quan, SHI Zhi-guo, ZHANG Ming. Micro-expression recognition algorithm based on separate long-term recurrent convolutional network[J]. Chinese Journal of Engineering. doi: 10.13374/j.issn2095-9389.2020.06.15.006

基于S-LRCN的微表情识别算法

doi: 10.13374/j.issn2095-9389.2020.06.15.006
基金项目: 国家自然科学基金资助项目(61977005);四川省科技计划资助项目(2018GZDZX0034);北京科技大学顺德研究生院科技创新专项资助项目(BK19CF003);北京市科技计划资助项目(Z201100004220010)
详细信息
    通讯作者:

    E-mail: husiquan@ustb.edu.cn

  • 中图分类号: TP391.4

Micro-expression recognition algorithm based on separate long-term recurrent convolutional network

More Information
  • 摘要: 基于面部动态表情序列,针对静态表情缺少时间信息等问题,将空间特征与时间特征融合,利用神经网络在图像分类领域良好的特征,对需要进行细节分析的表情序列进行处理,提出基于分离式长期循环卷积网络(Separate long-term recurrent convolutional networks, S-LRCN)的微表情识别方法。首先选取微表情数据集提取面部图像序列,引入迁移学习的方法,通过预训练的卷积神经网络模型提取表情帧的空间特征,降低网络训练中过拟合的危险,并将视频序列的提取特征输入长短期记忆网络(Long short-team memory, LSTM)处理时域特征。最后建立学习者表情序列小型数据库,将该方法用于辅助教学评价。

     

  • 图  1  动态表情识别流程

    Figure  1.  Dynamic expression-recognition process

    图  2  LRCN结构

    Figure  2.  LRCN structure

    图  3  SENet模块

    Figure  3.  SENet

    图  4  双向循环网络

    Figure  4.  Bidirectional LSTM

    图  5  LSTM神经元

    Figure  5.  LSTM neurons

    图  6  实现方法

    Figure  6.  Implementation method

    图  7  训练曲线

    Figure  7.  Training curve

    图  8  5种表情分类结果

    Figure  8.  Classification results of five expressions

    图  9  不同LSTM模型实验结果

    Figure  9.  Experimental results of different LSTM models

    图  10  数据分类

    Figure  10.  Data classification

    图  11  实验结果

    Figure  11.  Experimental result

    表  1  划分情况

    Table  1.   Dataset classification

    ClassifyCASME-ⅡSamples
    HappinessHappiness (32)32
    SurpriseSurprise (28)28
    DisgustDisgust (63)63
    RepressionRepression (27)27
    OthersOthers (99)105
    Sadness (4)
    Fear (2)
    下载: 导出CSV

    表  2  训练结果

    Table  2.   Training results %

    Test1Test2Test3Test4Test5
    64.966.265.265.866.4
    下载: 导出CSV

    表  3  不同算法识别准确率

    Table  3.   Recognition accuracy of different algorithms

    MethodsAccuracy/%F1-Score/%
    LBP-TOP52.642.6
    STCLQP58.658.0
    CNN+LSTM61.058.5
    HOOF+LSTM59.856.0
    S-LRCN65.760.8
    下载: 导出CSV

    表  4  不同序列长度实验效果

    Table  4.   Experimental results of different sequence lengths

    Sequence lengthAccuracy/%F1-Score/%
    662.056.6
    1065.760.8
    1563.158.6
    3056.549.6
    下载: 导出CSV
  • [1] Mehrabian A. Nonverbal communication. New York: Routledge, 2017
    [2] Ekman P. Facial expression and emotion. Am Psychol, 1993, 48(4): 384 doi: 10.1037/0003-066X.48.4.384
    [3] Ekman P, Friesen W V. Nonverbal leakage and clues to deception. Psychiatry, 1969, 32(1): 88 doi: 10.1080/00332747.1969.11023575
    [4] Yan W J, Wang S J, Liu Y J, et al. For micro-expression recognition: Database and suggestions. Neurocomputing, 2014, 136(136): 82
    [5] Wang S Y. CNN-RNN Based Micro-Expression Recognition [Dissertation]. Harbin: Harbin Engineering University, 2018

    王思宇. 基于CNN-RNN的微表情识别[学位论文]. 哈尔滨: 哈尔滨工程大学, 2018
    [6] Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks. Commun ACM, 2017, 60(6): 84 doi: 10.1145/3065386
    [7] Donahue J, Hendricks L A, Guadarrama S, et al. Long-term recurrent convolutional networks for visual recognition and description // 2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, 2015: 2625
    [8] Ekman P, Rosenberg E L. What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). 2nd Ed. USA: Oxford University Press, 2005
    [9] Gunes H, Pantic M. Automatic, dimensional and continuous emotion recognition. Int J Synthetic Emotions, 2010, 1(1): 68 doi: 10.4018/jse.2010101605
    [10] Song Y L, Morency L P, Davis R. Learning a sparse codebook of facial and body microexpressions for emotion recognition // Proceedings of the 15th ACM International Conference on Multimodal Interaction. New York, 2013: 237
    [11] Li D, Xie L, Lu T, et al. Capture of microexpressions based on the entropy of oriented optical flow. Chin J Eng, 2017(11): 1727

    李丹, 解仑, 卢婷, 等. 基于光流方向信息熵统计的微表情捕捉. 工程科学学报, 2017(11):1727
    [12] Polikovsky S, Kameda Y, Ohta Y. Facial micro-expression detection in Hi-speed video based on facial action coding system (FACS). IEICE Trans Inform Syst, 2013, E96-D(1): 81 doi: 10.1587/transinf.E96.D.81
    [13] Shreve M, Godavarthy S, Goldgof D, et al. Macro- and micro-expression spotting in long videos using spatio-temporal strain // 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG). Santa Barbara, 2011: 51
    [14] Pfister T, Li X B, Zhao G Y, et al. Recognising spontaneous facial micro-expressions // 2011 International Conference on Computer Vision. Barcelona, 2011: 1449
    [15] Liang J, Yan W J, Wu Q, et al. Recent advances and future trends in micro-expression research. Bull Natl Nat Sci Foundation China, 2013(2): 75

    梁静, 颜文靖, 吴奇, 等. 微表情研究的进展与展望. 中国科学基金, 2013(2):75
    [16] Wang Y D, See J, Phan R C W, et al. LBP with six intersection points: Reducing redundant information in LBP-TOP for micro-expression recognition // Asian Conference on Computer Vision ‒ ACCV2014. Switzerland, 2015: 525
    [17] Liong S T, See J, Phan R C W, et al. Spontaneous subtle expression detection and recognition based on facial strain. Signal Process Image Commun, 2016, 47: 170 doi: 10.1016/j.image.2016.06.004
    [18] Le Ngo A C, See J, Phan R C W. Sparsity in dynamics of spontaneous subtle emotion: analysis & application. IEEE Trans Affective Comput, 2017, 8(3): 396 doi: 10.1109/TAFFC.2016.2523996
    [19] Xu F, Zhang J P, Wang J Z. Microexpression identification and categorization using a facial dynamics map. IEEE Trans Affective Comput, 2017, 8(2): 254 doi: 10.1109/TAFFC.2016.2518162
    [20] Patel D, Hong X P, Zhao G Y. Selective deep features for micro-expression recognition // 2016 23rd International Conference on Pattern Recognition (ICPR). Cancun, 2016: 2258
    [21] Kim D H, Baddar W J, Ro Y M. Micro-expression recognition with expression-state constrained spatio-temporal feature representations // Proceedings of the 24th ACM international conference on Multimedia. Amsterdam, 2016: 382
    [22] Khor H Q, See J, Phan R C W, et al. Enriched long-term recurrent convolutional network for facial micro-expression recognition // 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2018). Xi’an, 2018: 667
    [23] Verburg M, Menkovski V. Micro-expression detection in long videos using optical flow and recurrent neural networks // 2019 14th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2019). Lille, 2019: 1
    [24] Itti L, Koch C. Computational modelling of visual attention. Nat Rev Neurosci, 2001, 2(3): 194 doi: 10.1038/35058500
    [25] Wang C Y, Peng M, Bi T, et al. Micro-attention for micro-expression recognition [J/OL]. arXiv Preprint (2019-8-27) [2020-04-21]. https://arxiv.org/abs/1811.02360.
    [26] Parkhi O M, Vedaldi A, Zisserman A. Deep face recognition // Proceedings of the British Machine Vision Conference (BMVC). Swansea, 2015: 45
    [27] Hu J, Shen L, Albanie S, et al. Squeeze-and-excitation networks. IEEE Trans Pattern Anal Mach Intell, 2020, 42(8): 2011 doi: 10.1109/TPAMI.2019.2913372
    [28] Cao Q, Shen L, Xie W D, et al. VGGFace2: A dataset for recognising faces across pose and age // 2018 13th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2018). Xi’an, 2018: 67
    [29] He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition // 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, 2016: 770
    [30] Cootes T F, Edwards G J, Taylor C J. Active appearance models. IEEE Trans Pattern Anal Mach Intell, 2001, 23(6): 681 doi: 10.1109/34.927467
    [31] Peng M. Dual Temporal Scale Convolutional Neural Network for Micro-Expression Recognition [Dissertation]. Chongqing: Southwest University, 2017

    彭敏. 基于双时间尺度卷积神经网络的微表情识别[学位论文]. 重庆: 西南大学, 2017
    [32] Yan W J, Li X B, Wang S J, et al. CASME II: An improved spontaneous micro-expression database and the baseline evaluation. PLoS ONE, 2014, 9(1): e86041 doi: 10.1371/journal.pone.0086041
    [33] Zhou Z H, Zhao G Y, Pietikinen M. Towards a practical lipreading system // The 24th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2011). Providence, RI, 2011: 137
    [34] Zhao G Y, Pietikainen M. Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell, 2007, 29(6): 915 doi: 10.1109/TPAMI.2007.1110
    [35] Huang X H, Zhao G Y, Hong X P, et al. Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing, 2016, 175: 564 doi: 10.1016/j.neucom.2015.10.096
  • 加载中
图(11) / 表(4)
计量
  • 文章访问数:  1043
  • HTML全文浏览量:  426
  • PDF下载量:  50
  • 被引次数: 0
出版历程
  • 收稿日期:  2020-06-15
  • 网络出版日期:  2020-07-23

目录

    /

    返回文章
    返回