koz"Вестник Северо-Казахстанского университета имени Манаша Козыбаева"Bulletin of Manash Kozybayev North Kazakhstan University2958-003X2958-0048М. Қозыбаев атындағы СҚУ10.54596/2958-0048-2026-1-263-273koz-2565Research ArticleИНФОРМАЦИОННО-КОММУНИКАЦИОННЫЕ ТЕХНОЛОГИИINFORMATION AND COMMUNICATION TECHNOLOGIESМУЛЬТИМОДАЛЬНЫЙ ДАТА СЕТ KSL-EMO ДЛЯ РАСПОЗНАВАНИЯ КАЗАХСКОГО ЖЕСТОВОГО ЯЗЫКА С УЧЁТОМ ЭМОЦИЙKSL-EMO MULTIMODAL DATASET FOR EMOTION-AWARE KAZAKH SIGN LANGUAGE RECOGNITIONhttps://orcid.org/0009-0004-9579-9288КабыкеновМ.KabykenovM.

Астана

Master student

Astana 

242890@astanait.edu.kzhttps://orcid.org/0009-0002-2051-6103НиязбекМ.NiyazbekM.

Урумчи

Associate professor, PhD

Urumqi

muheyatn@xju.edu.cnhttps://orcid.org/0000-0003-1042-0415ЖумадиллаеваА. К.ZhumadillayevaA.

Астана

Associate professor of the Department of Computer and Software engineering, candidate technical sciences

Astana

zhumadillayeva_ak@enu.kzAstana IT UniversityКазахстанAstana IT UniversityKazakhstanСиньцзянский университетКитайXinjiang UniversityChinaAstana IT University; Евразийский национальный университет имени Л.Н. ГумилеваКазахстанAstana IT University; L.N. Gumilyov Eurasian national universityKazakhstan20260804202601 (69)263273Copyright © Кабыкенов М., Ниязбек М., Жумадиллаева А.К., 20262026Кабыкенов М., Ниязбек М., Жумадиллаева А.К.Kabykenov M., Niyazbek M., Zhumadillayeva A.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://vestnik.ku.edu.kz/jour/article/view/2565

Распознавание жестового языка (SLR) является ключевой технологией для преодоления коммуникационного разрыва между сообществом глухих и слышащим большинством. Несмотря на значительный прогресс в области SLR благодаря глубокому обучению, малоресурсные языки, такие как казахский жестовый язык (KSL), остаются недостаточно изученными из-за нехватки размеченных данных. В данной работе мы решаем эту проблему, создавая новый эталонный набор данных для KSL, сосредоточенный на двух различных задачах: распознавание изолированных жестов (ISLR) и распознавание эмоций. Мы оцениваем производительность трёх современных архитектур Vision Transformer - ViViT, VideoMAE V2 и TimeSformer - на специально собранном наборе данных, включающем 20 лексических жестов и 4 эмоциональных состояния. Наши эксперименты показывают, что TimeSformer демонстрирует наилучшие результаты, достигая точности Top-1 96,63% для лексических жестов и 80,87% для распознавания эмоций. Сравнительный анализ показывает, что механизм «разделённого пространственно-временного внимания» TimeSformer более эффективно улавливает тонкую пространственно-временную динамику по сравнению с факторизованным энкодером ViViT или подходом маскированного моделирования VideoMAE.

Sign Language Recognition (SLR) is a main technology for bridging the communication gap between the deaf community and the hearing majority. While deep learning has advanced SLR significantly, low resource languages like Kazakh Sign Language (KSL) remain under explored due to the deficit of labeled data. In this paper, we address this limitation by establishing a novel benchmark for KSL, focusing on two distinct tasks: Isolated Sign Language Recognition (ISLR) and Emotion Recognition. We evaluate the performance of three state-of-the-art Vision Transformer architectures ViViT, VideoMAE V2, and TimeSformer on a custom collected dataset comprising 20 lexical gestures and 4 emotional states. Our experiments reveal that TimeSformer achieves superior performance, attaining a Top-1 Accuracy of 96.63% on lexical gestures and 80.87% on emotion recognition. Comparative analysis indicates that TimeSformer's "Divided Space-Time Attention" mechanism captures finegrained spatiotemporal dynamics more effectively than the factorised encoder of ViViT or the masked modeling approach of VideoMAE.

распознавание жестового языкаказахский жестовый языкVision Transformerраспознавание эмоцийSign Language RecognitionKazakh Sign LanguageVision TransformersEmotion RecognitionThis research has been funded by the SR-LAB-202504 Collaborative Innovation Seed Fund of Silk Road Multilingual Cognitive Computing International Cooperation Joint Laboratory, "Key Technologies for Terminology Extraction and Multidirectorial Translation in Multilingual Information Technology and Biomedical Fields in Central Asia", 2025-2027.ReferencesAdaloglou, N., Chatzis, T., Papastratis, I., Stergioulas, A., Papadopoulos, G. T., Zacharopoulou, V., Xydopoulos, G. J., Atzakas, K., Papazachariou, D., & Daras, P. (2021). A Comprehensive Study on Deep Learning-Based Methods for Sign Language Recognition. IEEE Transactions on Multimedia, 24, 1750­ 1762. https://doi.org/10.1109/tmm.2021.3070438Adaloglou, N., Chatzis, T., Papastratis, I., Stergioulas, A., Papadopoulos, G. T., Zacharopoulou, V., Xydopoulos, G. J., Atzakas, K., Papazachariou, D., & Daras, P. (2021). A Comprehensive Study on Deep Learning-Based Methods for Sign Language Recognition. IEEE Transactions on Multimedia, 24, 1750­ 1762. https://doi.org/10.1109/tmm.2021.3070438Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., & Schmid, C. (2021). ViViT: A video vision transformer. In 2021IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 6816-6826). IEEE. https://doi.org/10.1109/ICCV48922.2021.00676Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., & Schmid, C. (2021). ViViT: A video vision transformer. In 2021IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 6816-6826). IEEE. https://doi.org/10.1109/ICCV48922.2021.00676Bertasius, G., Wang, H., & Torresani, L. (2021). Is Space-Time Attention all you need for video understanding? arXiv (Cornell University). https://doi.org/10.48550/arxiv.2102.05095Bertasius, G., Wang, H., & Torresani, L. (2021). Is Space-Time Attention all you need for video understanding? arXiv (Cornell University). https://doi.org/10.48550/arxiv.2102.05095Camgoz, N. C., Koller, O., Hadfield, S., & Bowden, R. (2020, March 30). Sign Language Transformers: joint end-to-end sign language recognition and translation. arXiv.org. https://arxiv.org/abs/2003.13830Camgoz, N. C., Koller, O., Hadfield, S., & Bowden, R. (2020, March 30). Sign Language Transformers: joint end-to-end sign language recognition and translation. arXiv.org. https://arxiv.org/abs/2003.13830Carreira, J., & Zisserman, A. (2017). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4724-4733. https://doi.org/10.1109/cvpr.2017.502Carreira, J., & Zisserman, A. (2017). Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 4724-4733. https://doi.org/10.1109/cvpr.2017.502Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 4171-4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (pp. 4171-4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv.org. https://arxiv.org/abs/2010.11929Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv.org. https://arxiv.org/abs/2010.11929Ho, J., Kalchbrenner, N., Weissenborn, D., & Salimans, T. (2019). Axial attention in multidimensional transformers. arXiv.org. https://arxiv.org/abs/1912.12180Ho, J., Kalchbrenner, N., Weissenborn, D., & Salimans, T. (2019). Axial attention in multidimensional transformers. arXiv.org. https://arxiv.org/abs/1912.12180Hu, H., Zhao, W., Zhou, W., & Li, H. (2023). SignBERT+: Hand-Model-Aware Self-Supervised Pre-Training for Sign Language understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9), 11221-11239. https://doi.org/10.1109/tpami.2023.3269220Hu, H., Zhao, W., Zhou, W., & Li, H. (2023). SignBERT+: Hand-Model-Aware Self-Supervised Pre-Training for Sign Language understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9), 11221-11239. https://doi.org/10.1109/tpami.2023.3269220Kimmelman V, Imashev A, Mukushev M, Sandygulova A (2020) Eyebrow position in grammatical and emotional expressions in Kazakh-Russian Sign Language: A quantitative study. PLOS ONE 15(6): e0233731. https://doi.org/10.1371/iournal.pone.Q233731Kimmelman V, Imashev A, Mukushev M, Sandygulova A (2020) Eyebrow position in grammatical and emotional expressions in Kazakh-Russian Sign Language: A quantitative study. PLOS ONE 15(6): e0233731. https://doi.org/10.1371/iournal.pone.Q233731Koller, O., Zargaran, S., & Ney, H. (2017). Re-Sign: Re-Aligned End-to-End Sequence Modelling with Deep Recurrent CNN-HMMs. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3416-3424. https://doi.org/10.1109/cvpr.2017.364Koller, O., Zargaran, S., & Ney, H. (2017). Re-Sign: Re-Aligned End-to-End Sequence Modelling with Deep Recurrent CNN-HMMs. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3416-3424. https://doi.org/10.1109/cvpr.2017.364Li, D., Opazo, C. R., Yu, X., & Li, H. (2020). Word-level deep sign language recognition from video: a new large-scale dataset and methods comparison. IEEE Winter Conference on Applications of Computer Vision (WACV), 1448-1458. https://doi.org/10.1109/wacv45572.2020.9093512Li, D., Opazo, C. R., Yu, X., & Li, H. (2020). Word-level deep sign language recognition from video: a new large-scale dataset and methods comparison. IEEE Winter Conference on Applications of Computer Vision (WACV), 1448-1458. https://doi.org/10.1109/wacv45572.2020.9093512National Scientific and Practical Center for the Development of Special and Inclusive Education. (2024). Methodological guidelines for Kazakh sign language. https ://special-edu.kz/kz/news/6/single/961National Scientific and Practical Center for the Development of Special and Inclusive Education. (2024). Methodological guidelines for Kazakh sign language. https ://special-edu.kz/kz/news/6/single/961Ong, E. J., & Ranganath, S. (2005). Automatic sign language analysis: A survey and the future beyond lexical meaning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(6), 873-891.Ong, E. J., & Ranganath, S. (2005). Automatic sign language analysis: A survey and the future beyond lexical meaning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(6), 873-891.Rastgoo, R., Kiani, K., & Escalera, S. (2020). Sign Language Recognition: A deep survey. Expert Systems With Applications, 164, 113794. https://doi.org/10.1016/j.eswa.2020.113794Rastgoo, R., Kiani, K., & Escalera, S. (2020). Sign Language Recognition: A deep survey. Expert Systems With Applications, 164, 113794. https://doi.org/10.1016/j.eswa.2020.113794Selva, J., Johansen, A. S., Escalera, S., Nasrollahi, K., Moeslund, T. B., & Clapes, A. (2023). Video Transformers: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11), 12922­ 12943. https://doi.org/10.1109/tpami.2023.3243465Selva, J., Johansen, A. S., Escalera, S., Nasrollahi, K., Moeslund, T. B., & Clapes, A. (2023). Video Transformers: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11), 12922­ 12943. https://doi.org/10.1109/tpami.2023.3243465Starner, T., Weaver, J., & Pentland, A. (1998). Real-time American sign language recognition using desk and wearable computer based video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(12), 1371-1375. https://doi.org/10.1109/34.735811Starner, T., Weaver, J., & Pentland, A. (1998). Real-time American sign language recognition using desk and wearable computer based video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(12), 1371-1375. https://doi.org/10.1109/34.735811Song, Y., Tong, Z., Wang, J., & Wang, L. (2022). VideoMAE: Masked Autoencoders Are DataEfficient Learners for Self-Supervised Video Pre-Training. Neural Information Processing Systems Foundation, Inc. (NeurlPS), 10078-10093. https://doi.org/10.52202/068431-0732Song, Y., Tong, Z., Wang, J., & Wang, L. (2022). VideoMAE: Masked Autoencoders Are DataEfficient Learners for Self-Supervised Video Pre-Training. Neural Information Processing Systems Foundation, Inc. (NeurlPS), 10078-10093. https://doi.org/10.52202/068431-0732Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015). Learning Spatiotemporal Features with 3D Convolutional Networks. IEEE International Conference on Computer Vision (ICCV), 4489-4497. https://doi.org/10.1109/iccv.2015.510Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. (2015). Learning Spatiotemporal Features with 3D Convolutional Networks. IEEE International Conference on Computer Vision (ICCV), 4489-4497. https://doi.org/10.1109/iccv.2015.510Wang, L., Huang, B., Zhao, Z., Tong, Z., He, Y., Wang, Y., Wang, Y., & Qiao, Y. (2023). VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 14549-14560. https://doi.org/10.1109/cvpr52729.2023.01398Wang, L., Huang, B., Zhao, Z., Tong, Z., He, Y., Wang, Y., Wang, Y., & Qiao, Y. (2023). VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 14549-14560. https://doi.org/10.1109/cvpr52729.2023.01398Wang, X., Girshick, R., Gupta, A., & He, K. (2018). Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 7794-7803)Wang, X., Girshick, R., Gupta, A., & He, K. (2018). Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 7794-7803)

The authors declare that there are no conflicts of interest present.