End-To-End Deep Learning for pNN50 Estimation Using a Spatiotemporal Representation

Sayyedjavad Ziaratnia, Peeraya Sripian, Tipporn Laohakangvalvit, Kazuo Ohzeki, Midori Sugaya

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)


Various industries widely use emotion estimation to evaluate their consumer satisfaction towards their products. Generally, emotion can be estimated based on observable expressions such as facial expression, or unobservable expressions such as biological signals. Although used by many research, the Facial Expression Recognition has a lack of precision for expressions that are very similar to each other or a situation where the shown expression differs from the real subject’s emotion. On the other hand, biological signal indexes such as pNN50 can act as a supportive mechanism to improve emotion estimation from observable expressions such as facial expression recognition method. pNN50 is a reliable index to estimate stress-relax, and it originates from unconscious emotions that cannot be manipulated. In this work, we propose a method for pNN50 estimation from facial video using a Deep Learning model. Transfer learning technique and a pre-trained Image recognition Convolutional Neural Network (CNN) model are employed to estimate pNN50 based on a spatiotemporal map created from a series of frames in a facial video. The model which trained on low, middle, and high pNN50 values, shows an accuracy of about 80%. Therefore, it indicates the potential of our proposed method, and we can expand it to categorize the more detailed level of pNN50 values.

Original languageEnglish
Title of host publicationHCI International 2021 - Posters - 23rd HCI International Conference, HCII 2021, Proceedings
EditorsConstantine Stephanidis, Margherita Antona, Stavroula Ntoa
PublisherSpringer Science and Business Media Deutschland GmbH
Number of pages6
ISBN (Print)9783030786410
Publication statusPublished - 2021
Event23rd International Conference on Human-Computer Interaction, HCII 2021 - Virtual, Online
Duration: 2021 Jul 242021 Jul 29

Publication series

NameCommunications in Computer and Information Science
ISSN (Print)1865-0929
ISSN (Electronic)1865-0937


Conference23rd International Conference on Human-Computer Interaction, HCII 2021
CityVirtual, Online


  • Convolutional Neural Network
  • Deep Learning
  • pNN50 Estimation
  • rPPG

ASJC Scopus subject areas

  • Computer Science(all)
  • Mathematics(all)


Dive into the research topics of 'End-To-End Deep Learning for pNN50 Estimation Using a Spatiotemporal Representation'. Together they form a unique fingerprint.

Cite this