Multimodal Affective Recognition
The research project aims to study the multi-modal recognition of human affect, specifically frustration. Prior work reported that frustration is one of the negative emotions commonly experienced by users when working with technology [1]. This emotion can hinder users’ primary tasks, resulting in inefficient time usage and poses risks in task executions. It has been observed that users spend a significant portion of their computer usage time dealing with frustrating events [2,3]. Frustration is a concept that encompasses various disciplines, including psychology and cognitive science. There exist positive frustration which caused by intrinsic factors such as cognitively demanding tasks and negative frustration induced by external factors (e.g., system errors). Frustration can manifest through different channels, such as audiovisual cues, behavioral and physiological biometrics. This research project aims to detect and recognize frustration by drawing upon existing theoretical frameworks and leveraging multiple modalities.
The first step is to understand the relationship between frustration and task performance, including attention and perception, in cognitively demanding tasks. As a starting point, we aim to investigate the occurrence of frustration when a user’s goal is obstructed. To do so, the project employs a buggy bubble-popping game for users to play. We then measure user performance when they perform cognitively demanding tasks. Additionally, the project seeks to explore the relationships between physiological signals, cognitive workload, and frustration levels. We will also monitor user physiological signals using functional near-infrared spectroscopy (fNIRS) techniques.
References
[1] R. W. Picard, “Affective Computing for HCI,” Proceedings of the 8th HCI International on Human-Computer Interaction: Ergonomics and User Interfaces, pp. 829–833, 1999, arXiv: 742338 ISBN: 0-8058-3391-9. [Online]. Available: http://dl.acm.org/citation.cfm?id=647943.742338
[2] I. Ceaparu, J. Lazar, K. Bessiere, J. Robinson, and B. Shneiderman, “Determining Causes and Severity of End-User Frustration,” International Journal of Human–Computer Interaction, vol. 17, no. 3, pp. 333–356, Sep. 2004. [Online]. Available: https://doi.org/10.1207/s15327590ijhc1703 3
[3] J. Lazar, A. Jones, K. Bessiere, I. Ceaparu, and B. Shneiderman, “User `frustration with technology in the workplace,” in Americas Conference on Information Systems, 2003.
