KSL-EMO MULTIMODAL DATASET FOR EMOTION-AWARE KAZAKH SIGN LANGUAGE RECOGNITION

Sign Language Recognition (SLR) is a main technology for bridging the communication gap between the deaf community and the hearing majority. While deep learning has advanced SLR significantly, low resource languages like Kazakh Sign Language (KSL) remain under explored due to the deficit of labeled data. In this paper, we address this limitation by establishing a novel benchmark for KSL, focusing on two distinct tasks: Isolated Sign Language Recognition (ISLR) and Emotion Recognition. We evaluate the performance of three state-of-the-art Vision Transformer architectures ViViT, VideoMAE V2, and TimeSformer on a custom collected dataset comprising 20 lexical gestures and 4 emotional states. Our experiments reveal that TimeSformer achieves superior performance, attaining a Top-1 Accuracy of 96.63% on lexical gestures and 80.87% on emotion recognition. Comparative analysis indicates that TimeSformer's "Divided Space-Time Attention" mechanism captures finegrained spatiotemporal dynamics more effectively than the factorised encoder of ViViT or the masked modeling approach of VideoMAE.

1. Adaloglou, N., Chatzis, T., Papastratis, I., Stergioulas, A., Papadopoulos, G. T., Zacharopoulou, V., Xydopoulos, G. J., Atzakas, K., Papazachariou, D., & Daras, P. (2021). A Comprehensive Study on Deep Learning-Based Methods for Sign Language Recognition. IEEE Transactions on Multimedia, 24, 1750­ 1762. https://doi.org/10.1109/tmm.2021.3070438

2. Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., & Schmid, C. (2021). ViViT: A video vision transformer. In 2021IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 6816-6826). IEEE. https://doi.org/10.1109/ICCV48922.2021.00676

3. Bertasius, G., Wang, H., & Torresani, L. (2021). Is Space-Time Attention all you need for video understanding? arXiv (Cornell University). https://doi.org/10.48550/arxiv.2102.05095

4. Camgoz, N. C., Koller, O., Hadfield, S., & Bowden, R. (2020, March 30). Sign Language Transformers: joint end-to-end sign language recognition and translation. arXiv.org. https://arxiv.org/abs/2003.13830

5. Carreira, J., & Zisserman, A. (2017). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4724-4733. https://doi.org/10.1109/cvpr.2017.502

6. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 4171-4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423

7. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv.org. https://arxiv.org/abs/2010.11929

8. Ho, J., Kalchbrenner, N., Weissenborn, D., & Salimans, T. (2019). Axial attention in multidimensional transformers. arXiv.org. https://arxiv.org/abs/1912.12180

9. Hu, H., Zhao, W., Zhou, W., & Li, H. (2023). SignBERT+: Hand-Model-Aware Self-Supervised Pre-Training for Sign Language understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9), 11221-11239. https://doi.org/10.1109/tpami.2023.3269220

10. Kimmelman V, Imashev A, Mukushev M, Sandygulova A (2020) Eyebrow position in grammatical and emotional expressions in Kazakh-Russian Sign Language: A quantitative study. PLOS ONE 15(6): e0233731. https://doi.org/10.1371/iournal.pone.Q233731

11. Koller, O., Zargaran, S., & Ney, H. (2017). Re-Sign: Re-Aligned End-to-End Sequence Modelling with Deep Recurrent CNN-HMMs. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3416-3424. https://doi.org/10.1109/cvpr.2017.364

12. Li, D., Opazo, C. R., Yu, X., & Li, H. (2020). Word-level deep sign language recognition from video: a new large-scale dataset and methods comparison. IEEE Winter Conference on Applications of Computer Vision (WACV), 1448-1458. https://doi.org/10.1109/wacv45572.2020.9093512

13. National Scientific and Practical Center for the Development of Special and Inclusive Education. (2024). Methodological guidelines for Kazakh sign language. https ://special-edu.kz/kz/news/6/single/961

14. Ong, E. J., & Ranganath, S. (2005). Automatic sign language analysis: A survey and the future beyond lexical meaning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(6), 873-891.

15. Rastgoo, R., Kiani, K., & Escalera, S. (2020). Sign Language Recognition: A deep survey. Expert Systems With Applications, 164, 113794. https://doi.org/10.1016/j.eswa.2020.113794

16. Selva, J., Johansen, A. S., Escalera, S., Nasrollahi, K., Moeslund, T. B., & Clapes, A. (2023). Video Transformers: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11), 12922­ 12943. https://doi.org/10.1109/tpami.2023.3243465

17. Starner, T., Weaver, J., & Pentland, A. (1998). Real-time American sign language recognition using desk and wearable computer based video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(12), 1371-1375. https://doi.org/10.1109/34.735811

18. Song, Y., Tong, Z., Wang, J., & Wang, L. (2022). VideoMAE: Masked Autoencoders Are DataEfficient Learners for Self-Supervised Video Pre-Training. Neural Information Processing Systems Foundation, Inc. (NeurlPS), 10078-10093. https://doi.org/10.52202/068431-0732

19. Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015). Learning Spatiotemporal Features with 3D Convolutional Networks. IEEE International Conference on Computer Vision (ICCV), 4489-4497. https://doi.org/10.1109/iccv.2015.510

20. Wang, L., Huang, B., Zhao, Z., Tong, Z., He, Y., Wang, Y., Wang, Y., & Qiao, Y. (2023). VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 14549-14560. https://doi.org/10.1109/cvpr52729.2023.01398

21. Wang, X., Girshick, R., Gupta, A., & He, K. (2018). Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 7794-7803)