A Survey on Deep Learning Algorithms in Facial Emotion Detection and Recognition

  • Prince Awuah Baffour Department of Computer Engineering, Kwame Nkrumah University of Science and Technology
  • Henry Nunoo-Mensah Department of Computer Engineering, Kwame Nkrumah University of Science and Technology, Kumasi
  • Eliel Keelson Department of Computer Engineering, Kwame Nkrumah University of Science and Technology, Kumasi
  • Benjamin Kommey Department of Computer Engineering, Kwame Nkrumah University of Science and Technology, Kumasi
https://doi.org/10.25139/inform.v7i1.4563
Abstract views: 1066 , PDF downloads: 780
Keywords: Artificial Intelligence, Emotion Detection, Machine Learning, Facial Recognition, Datasets

Abstract

Facial emotion recognition (FER) forms part of affective computing, where computers are trained to recognize human emotion from human expressions. Facial Emotion Recognition is very necessary for bridging the communication gap between humans and computers because facial expressions are a form of communication that transmits 55% of a person's emotional and mental state in a total face-to-face communication spectrum. Breakthroughs in this field also make computer systems (robotic systems) better serve or interact with humans. Research has far advanced for this cause, and Deep learning is at its heart. This paper systematically discusses state-of-the-art deep learning architectures and algorithms for facial emotion detection and recognition. The paper also reveals the dominance of CNN architectures over other known architectures like RNNs and SVMs, highlighting the contributions, model performance, and limitations of the reviewed state-of-the-art. It further identifies available opportunities and open issues worth considering by various FER research in the future. This paper will also discover how computation power and availability of large facial emotion datasets have also limited the pace of progress.

References

B. Balasubramanian, P. Diwan, R. Nadar, and A. Bhatia, 'Analysis of Facial Emotion Recognition, in 2019 3rd International Conference on Trends in Electronics and Informatics (ICOEI), Tirunelveli, India, Apr. 2019, pp. 945–949. doi: 10.1109/ICOEI.2019.8862731.

W. Mellouk and W. Handouzi, 'Facial emotion recognition using deep learning: review and insights', Procedia Comput. Sci., vol. 175, pp. 689–694, 2020, doi: 10.1016/j.procs.2020.07.101.

L. Stanciu and A. Albu, 'Analysis on Emotion Detection and Recognition Methods using Facial Microexpressions. A Review', in 2019 E-Health and Bioengineering Conference (EHB), Iasi, Romania, Nov. 2019, pp. 1–4. doi: 10.1109/EHB47216.2019.8969925.

A. Hassouneh, A. M. Mutawa, and M. Murugappan, 'Development of a Real-Time Emotion Recognition System Using Facial Expressions and EEG based on machine learning and deep neural network methods', Inform. Med. Unlocked, vol. 20, p. 100372, Jan. 2020, doi: 10.1016/j.imu.2020.100372.

E. Pranav, S. Kamal, C. Satheesh Chandran, and M. H. Supriya, 'Facial Emotion Recognition Using Deep Convolutional Neural Network', in 2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS), Coimbatore, India, Mar. 2020, pp. 317–320. doi: 10.1109/ICACCS48705.2020.9074302.

R. Ranjan and B. C. Sahana, 'An Efficient Facial Feature Extraction Method Based Supervised Classification Model for Human Facial Emotion Identification', in 2019 IEEE International Symposium on Signal Processing and Information Technology (ISSPIT), Ajman, United Arab Emirates, Dec. 2019, pp. 1–6. doi: 10.1109/ISSPIT47144.2019.9001839.

T. Wu, S. Fu, and G. Yang, 'Survey of the Facial Expression Recognition Research', in Advances in Brain Inspired Cognitive Systems, Berlin, Heidelberg, 2012, pp. 392–402. doi: 10.1007/978-3-642-31561-9_44.

J. L. Tracy and D. Randles, 'Four Models of Basic Emotions: A Review of Ekman and Cordaro, Izard, Levenson, and Panksepp and Watt', Emot. Rev., vol. 3, no. 4, pp. 397–405, Oct. 2011, doi: 10.1177/1754073911410747.

P. Ekman, Emotions Revealed: Recognizing Faces and Feelings to Improve Communication and Emotional Life. Henry Holt and Company, 2004. [Online]. Available: https://books.google.com.gh/books?id=AoIUp5fJkIcC

R. Plutchik, 'The Nature of Emotions: Clinical Implications', in Emotions and Psychopathology, M. Clynes and J. Panksepp, Eds. Boston, MA: Springer US, 1988, pp. 1–20. doi: 10.1007/978-1-4757-1987-1_1.

A. R. Dores, F. Barbosa, C. Queirós, I. P. Carvalho, and M. D. Griffiths, 'Recognizing Emotions through Facial Expressions: A Largescale Experimental Study', Int. J. Environ. Res. Public. Health, vol. 17, no. 20, p. 7420, Oct. 2020, doi: 10.3390/ijerph17207420.

Dasharath. K. Bhadangkar, J. D. Pujari, and R. Yakkundimath, 'Comparison of Tuplet of Techniques for Facial Emotion Detection', in 2020 Fourth International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), Palladam, India, Oct. 2020, pp. 725–730. doi: 10.1109/I-SMAC49090.2020.9243439.

J. Muhammad, Y. Wang, C. Wang, K. Zhang, and Z. Sun, 'CASIA-Face-Africa: A Large-scale African Face Image Database', ArXiv210503632 Cs, May 2021, Accessed: Jul. 07, 2021. [Online]. Available: http://arxiv.org/abs/2105.03632

A. Jaiswal, A. Krishnama Raju, and S. Deb, 'Facial Emotion Detection Using Deep Learning', in 2020 International Conference for Emerging Technology (INCET), Belgaum, India, Jun. 2020, pp. 1–5. doi: 10.1109/INCET49848.2020.9154121.

I. Lasri, A. R. Solh, and M. E. Belkacemi, 'Facial Emotion Recognition of Students using Convolutional Neural Network', in 2019 Third International Conference on Intelligent Computing in Data Sciences (ICDS), Marrakech, Morocco, Oct. 2019, pp. 1–6. doi: 10.1109/ICDS47004.2019.8942386.

N. Naik and M. A. Mehta, 'An Improved Method to Recognize Hand-over-Face Gesture based Facial Emotion using Convolutional Neural Network', in 2020 IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT), Bangalore, India, Jul. 2020, pp. 1–6. doi: 10.1109/CONECCT50063.2020.9198376.

S. Begaj, A. O. Topal, and M. Ali, 'Emotion Recognition Based on Facial Expressions Using Convolutional Neural Network (CNN)', in 2020 International Conference on Computing, Networking, Telecommunications & Engineering Sciences Applications (CoNTESA), Tirana, Albania, Dec. 2020, pp. 58–63. doi: 10.1109/CoNTESA50436.2020.9302866.

M. S. Bouzakraoui, A. Sadiq, and A. Y. Alaoui, 'Appreciation of Customer Satisfaction Through Analysis Facial Expressions and Emotions Recognition', in 2019 4th World Conference on Complex Systems (WCCS), Ouarzazate, Morocco, Apr. 2019, pp. 1–5. doi: 10.1109/ICoCS.2019.8930761.

D. K. Jain, P. Shamsolmoali, and P. Sehdev, 'Extended deep neural network for facial emotion recognition', Pattern Recognit. Lett., vol. 120, pp. 69–74, Apr. 2019, doi: 10.1016/j.patrec.2019.01.008.

S. Liu, D. Li, Q. Gao, and Y. Song, 'Facial Emotion Recognition Based on CNN', in 2020 Chinese Automation Congress (CAC), Shanghai, China, Nov. 2020, pp. 398–403. doi: 10.1109/CAC51589.2020.9327432.

S. Engoor, S. SendhilKumar, C. Hepsibah Sharon, and G. S. Mahalakshmi, 'Occlusion-aware Dynamic Human Emotion Recognition Using Landmark Detection', in 2020 6th International Conference on Advanced Computing and Communication Systems (ICACCS), Coimbatore, India, Mar. 2020, pp. 795–799. doi: 10.1109/ICACCS48705.2020.9074318.

A. N. Navaz, S. M. Adel, and S. S. Mathew, 'Facial Image Preprocessing and Emotion Classification: A Deep Learning Approach', in 2019 IEEE/ACS 16th International Conference on Computer Systems and Applications (AICCSA), Abu Dhabi, United Arab Emirates, Nov. 2019, pp. 1–8. doi: 10.1109/AICCSA47632.2019.9035268.

A. John, A. Mc, A. S. Ajayan, S. Sanoop, and V. R. Kumar, 'Real-Time Facial Emotion Recognition System With Improved Preprocessing and Feature Extraction', in 2020 Third International Conference on Smart Systems and Inventive Technology (ICSSIT), Tirunelveli, India, Aug. 2020, pp. 1328–1333. doi: 10.1109/ICSSIT48917.2020.9214207.

M. M. Taghi Zadeh, M. Imani, and B. Majidi, 'Fast Facial emotion recognition Using Convolutional Neural Networks and Gabor Filters', in 2019 5th Conference on Knowledge Based Engineering and Innovation (KBEI), Tehran, Iran, Feb. 2019, pp. 577–581. doi: 10.1109/KBEI.2019.8734943.

A. Ghofrani, R. M. Toroghi, and S. Ghanbari, 'Realtime Face-Detection and Emotion Recognition Using MTCNN and miniShuffleNet V2', in 2019 5th Conference on Knowledge Based Engineering and Innovation (KBEI), Tehran, Iran, Feb. 2019, pp. 817–821. doi: 10.1109/KBEI.2019.8734924.

L. Cai, J. Dong, and M. Wei, 'Multimodal Emotion Recognition From Speech and Facial Expression Based on Deep Learning', in 2020 Chinese Automation Congress (CAC), Shanghai, China, Nov. 2020, pp. 5726–5729. doi: 10.1109/CAC51589.2020.9327178.

G. Wei, L. Jian, and S. Mo, 'Multimodal(Audio, Facial and Gesture) based Emotion Recognition challenge', in 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020), Buenos Aires, Argentina, Nov. 2020, pp. 908–911. doi: 10.1109/FG47880.2020.00142.

I. Oztel, G. Yolcu, and C. Oz, 'Performance Comparison of Transfer Learning and Training from Scratch Approaches for Deep Facial Expression Recognition', in 2019 4th International Conference on Computer Science and Engineering (UBMK), Samsun, Turkey, Sep. 2019, pp. 1–6. doi: 10.1109/UBMK.2019.8907203.

Published
2022-06-08
How to Cite
Prince Awuah Baffour, Henry Nunoo-Mensah, Eliel Keelson, & Benjamin Kommey. (2022). A Survey on Deep Learning Algorithms in Facial Emotion Detection and Recognition. Inform : Jurnal Ilmiah Bidang Teknologi Informasi Dan Komunikasi, 7(1), 24-32. https://doi.org/10.25139/inform.v7i1.4563
Issue
Vol. 7 No. 1 (2022)
Section
Articles

Copyright (c) 2022 Prince Awuah Baffour, Henry Nunoo-Mensah, Eliel Keelson, Benjamin Kommey

Creative Commons License

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Authors who publish with Inform: Jurnal Ilmiah Bidang Teknologi Informasi dan Komunikasi agree to the following terms:

  1. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License (CC BY-SA 4.0) that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal. 

  2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.

  3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.

Most read articles by the same author(s)