Visualizing Functional Emotions: Mapping Counseling Responses from Text to Virtual Facial Expressions

This research develops an innovative virtual counseling system by integrating text-based emotion classification with visual representation to address the problem of early marriage in Lombok. The system leverages the sophisticated IndoRoBERTa model to accurately classify conselor responses into five functional emotion categories relevant to the counseling context: Enthusiasm, Gentleness, Analytical, Inspirational, and Cautionary. The limitations of conventional counseling services in rural areas serve as the primary justification for developing this responsive and accessible technological solution. Evaluation results demonstrate that the IndoRoBERTa model achieves a highly competitive accuracy rate of 89% after being trained on an expanded dataset, an achievement that significantly surpasses previous architectures. In conclusion, this IndoRoBERTa-based system is not only technically viable but also effective as a tool for providing initial empathetic support. Its capability to translate textual emotions into non-verbal visual cues makes it a promising technological solution to bridge the gap in current counseling services.

Keywords: IndoRoBERTa, Emotion Classification, Virtual Counseling, Early Marriage, Facial Expression Mapping, Natural Language Processing (NLP)

[1] M. D. H. Rahiem, “COVID-19 and the surge of child marriages: A phenomenon in Nusa Tenggara Barat, Indonesia,” Child Abuse Negl, vol. 118, p. 105168, Aug. 2021, doi: 10.1016/j.chiabu.2021.105168.

[2] M. E. Greene, M. Siddiqi, and T. F. Abularrage, “Systematic scoping review of interventions to prevent and respond to child marriage across Africa: progress, gaps and priorities,” BMJ Open, vol. 13, no. 5, p. e061315, May 2023, doi: 10.1136/bmjopen-2022-061315.

[3] A. Sampurna, H. J. Ritonga, and A. R. Matondang, “Integration of Media Literacy in Religious Counseling for Preventing Early Marriage in Nias Barat,” International Journal of Islamic Education, Research and Multiculturalism (IJIERM), vol. 6, no. 3, pp. 1205–1218, Dec. 2024, doi: 10.47006/ijierm. v6i3.392.

[4] Z. Guo, A. Lai, J. H. Thygesen, J. Farrington, T. Keen, and K. Li, “Large Language Models for Mental Health Applications: Systematic Review,” JMIR Ment Health, vol. 11, p. e57400, Oct. 2024, doi: 10.2196/57400.

[5] V. Sorin et al., “Large Language Models and Empathy: Systematic Review,” J Med Internet Res, vol. 26, p. e52597, Dec. 2024, doi: 10.2196/52597.

[6] G. Park, J. Chung, and S. Lee, “Effect of AI chatbot emotional disclosure on user satisfaction and reuse intention for mental health counseling: a serial mediation model,” Current Psychology, vol. 42, no. 32, pp. 28663–28673, Nov. 2023, doi: 10.1007/s12144-022-03932-z.

[7] H. Chin et al., “The Potential of Chatbots for Emotional Support and Promoting Mental Well-Being in Different Cultures: Mixed Methods Study,” J Med Internet Res, vol. 25, p. e51712, Oct. 2023, doi: 10.2196/51712.

[8] J. Sun, “Research and Application Analysis of Multimodal Emotion Recognition Methods Based on Speech, Text, And Facial Expressions,” Highlights in Science, Engineering and Technology, vol. 85, pp. 293–297, Mar. 2024, doi: 10.54097/agvjvq19.

[9] M. Y. Baihaqi, E. Halawa, R. A. S. Syah, A. Nurrahma, and W. Wijaya, “Emotion Classification in Indonesian Language: A CNN Approach with Hyperband Tuning,” Jurnal Buana Informatika, vol. 14, no. 02, pp. 137–146, Oct. 2023, doi: 10.24002/jbi. v14i02.7558.

[10] A. Zamsuri, S. Defit, and G. W. Nurcahyo, “Classification of Multiple Emotions in Indonesian Text Using The K-Nearest Neighbor Method,” Journal of Applied Engineering and Technological Science (JAETS), vol. 4, no. 2, pp. 1012–1021, Jun. 2023, doi: 10.37385/jaets. v4i2.1964.

[11] Y. Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” Jul. 2019.

[12] Y. O. Sihombing, R. Fuad Rachmadi, S. Sumpeno, and Moh. J. Mubarok, “Optimizing IndoRoBERTa Model for Multi-Class Classification of Sentiment & Emotion on Indonesian Twitter,” in 2024 IEEE 10th Information Technology International Seminar (ITIS), IEEE, Nov. 2024, pp. 12–17. doi: 10.1109/ITIS64716.2024.10845566.

[13] E. Bourke, C. Barker, and M. Fornells‐Ambrojo, “Systematic review and meta‐analysis of therapeutic alliance, engagement, and outcome in psychological therapies for psychosis,” Psychology and Psychotherapy: Theory, Research and Practice, vol. 94, no. 3, pp. 822–853, Sep. 2021, doi: 10.1111/papt.12330.

[14] J. Kadur, J. Lüdemann, and S. Andreas, “Effects of the therapist’s statements on the patient’s outcome and the therapeutic alliance: A systematic review,” Clin Psychol Psychother, vol. 27, no. 2, pp. 168–178, Mar. 2020, doi: 10.1002/cpp.2416.

[15] L. Del Giacco, M. T. Anguera, and S. Salcuni, “The Action of Verbal and Non-verbal Communication in the Therapeutic Alliance Construction: A Mixed Methods Approach to Assess the Initial Interactions with Depressed Patients,” Front Psychol, vol. 11, Feb. 2020, doi: 10.3389/fpsyg.2020.00234.

[16] Z. N. Maharani, A. Luthfiarta, and N. Z. Farsya, “Sentiment Analysis of the 2024 Indonesian Presidential Dispute Trial Election using SVM and Naïve Bayes on Platform X,” Building of Informatics, Technology and Science (BITS), vol. 6, no. 1, pp. 440–449, Jun. 2024, doi: 10.47065/bits. v6i1.5380.

[17] W. Suwarningsih, R. A. Pramata, F. Y. Rahadika, and M. H. A. Purnomo, “RoBERTa: language modelling in building Indonesian question-answering systems,” TELKOMNIKA (Telecommunication Computing Electronics and Control), vol. 20, no. 6, p. 1248, Dec. 2022, doi: 10.12928/telkomnika. v20i6.24248.

[18] S. William, Kenny, and A. Chowanda, “EMOTION RECOGNITION INDONESIAN LANGUAGE FROM TWITTER USING INDOBERT AND BI-LSTM,” Communications in Mathematical Biology and Neuroscience, vol. 2024, 2024, doi: 10.28919/cmbn/7858.

[19] F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, “IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP,” in Proceedings of the 28th International Conference on Computational Linguistics, Stroudsburg, PA, USA: International Committee on Computational Linguistics, 2020, pp. 757–770. doi: 10.18653/v1/2020.coling-main.66.

[20] M. Malgaroli, T. D. Hull, J. M. Zech, and T. Althoff, “Natural language processing for mental health interventions: a systematic review and research framework,” Transl Psychiatry, vol. 13, no. 1, p. 309, Oct. 2023, doi: 10.1038/s41398-023-02592-2.

[21] M. A. Witherow et al., “Customizable Avatars with Dynamic Facial Action Coded Expressions (CADyFACE) for Improved User Engagement,” Mar. 2024.

[22] S. Kang, H. Song, B. Yoon, K. Kim, and W. Woo, “The Influence of Emotion-based Prioritized Facial Expressions on Social Presence in Avatar-mediated Remote Communication,” in 2024 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), IEEE, Oct. 2024, pp. 1147–1156. doi: 10.1109/ISMAR62088.2024.00131.

[23] K. Wisnudhanti and F. Candra, “Image Classification of Pandawa Figures Using Convolutional Neural Network on Raspberry Pi 4,” J Phys Conf Ser, vol. 1655, no. 1, p. 012103, Oct. 2020, doi: 10.1088/1742-6596/1655/1/012103.

[24] D. M. Aprilla, F. Bimantoro, and I. G. P. S. Wijaya, “The Palmprint Recognition Using Xception, VGG16, ResNet50, MobileNet, and EfficientNetB0 Architecture,” JURNAL MEDIA INFORMATIKA BUDIDARMA, vol. 8, no. 2, p. 1065, Apr. 2024, doi: 10.30865/mib. v8i2.7577.

[25] S. Sathyanarayanan, “Confusion Matrix-Based Performance Evaluation Metrics,” African Journal of Biomedical Research, pp. 4023–4031, Nov. 2024, doi: 10.53555/AJBR.v27i4S.4345.

[26] R. Pereira et al., “Systematic Review of Emotion Detection with Computer Vision and Deep Learning,” Sensors, vol. 24, no. 11, p. 3484, May 2024, doi: 10.3390/s24113484.

[27] J. Terven, D.-M. Cordova-Esparza, J.-A. Romero-González, A. Ramírez-Pedraza, and E. A. Chávez-Urbiola, “A comprehensive survey of loss functions and metrics in deep learning,” Artif Intell Rev, vol. 58, no. 7, p. 195, Apr. 2025, doi: 10.1007/s10462-025-11198-7.

[28] S. Farhadpour, T. A. Warner, and A. E. Maxwell, “Selecting and Interpreting Multiclass Loss and Accuracy Assessment Metrics for Classifications with Class Imbalance: Guidance and Best Practices,” Remote Sens (Basel), vol. 16, no. 3, p. 533, Jan. 2024, doi: 10.3390/rs16030533.

[29] Y. Wang et al., “A Systematic Review on Affective Computing: Emotion Models, Databases, and Recent Advances,” Mar. 2022.

[30] G. Hu et al., “Recent Trends of Multimodal Affective Computing: A Survey from NLP Perspective,” Oct. 2024.