AUTOMATED ANALYSIS OF MULTIMODAL SOCIAL MEDIA DATA BASED ON MODELS FOR UNSTRUCTURED INFORMATION PROCESSING

Social media generates massive volumes of unstructured data every day. These data contain valuable information, but their heterogeneity and complexity make analysis difficult when using standard methods.
The aim of the study is to develop and test an approach for automatic analysis of multimodal social media data using modern models for processing unstructured information. The work employs deep learning technologies, including transformers and models that combine different types of data (text and image).
A prototype was developed that integrates textual and visual features using neural network architectures. Experiments were conducted on open datasets containing user posts with images and textual captions; video content was not included in the study. The results show that the use of multimodal models improves the accuracy of sentiment analysis and enhances data interpretation.
The proposed approach can be applied in SMM analytics, marketing, user behavior prediction, and public opinion analysis. It helps automate the processing of complex data and supports decision-making based on comprehensive information.
The conclusions confirm that combining modern methods of unstructured information analysis is effective for working with multimodal data in conditions of large scale and diverse sources.

1. Baltrusaitis T., Ahuja C., Morency L. Multimodal Machine Learning: A Survey and Taxonomy // IEEE Transactions on Pattern Analysis and Machine Intelligence. - 2019. - DOI: https://doi.org/10.1109/TPAMI.2018.2798607

2. Bishop C. M. Pattern Recognition and Machine Learning [Elektronnyj resurs]. - New York: Springer, 2006. - URL: https://link.springer.com/book/10.1007/978-0-387-45528-0 - Data obrashcheniya: 12.03.2026.

3. Radford A., Kim J. W., Hallacy C. i dr. Learning Transferable Visual Models from Natural Language Supervision [Elektronnyj resurs] // arXiv preprint. - 2021. - arXiv: 2103.00020. - URL: https://doi.org/10.48550/arXiv.2103.00020 - Data obrashcheniya: 12.03.2026.

4. Russell S., Norvig P. Artificial Intelligence: A Modern Approach [Elektronnyj resurs]. - 4th ed. - Pearson, 2020. - URL: https://aima.cs.berkeley.edu/ - Data obrashcheniya: 12.03.2026.

5. Ngiam J., Khosla A., Kim M., Nam J., Lee H., Ng A. Y. Multimodal Deep Learning // Proceedings of the 28th International Conference on Machine Learning (ICML-11). - 2011. - P. 689-696. - URL: https://robotics.stanford.edu/~ang/papers/icml11-MultimodalDeepLearning.pdf - Data obrashcheniya: 12.03.2026.

6. Li H., Zhang Y., Chen X. Multi-Modal Sentiment Analysis Based on Image and Text Using CrossAttention Mechanism // Electronics. - 2024. - Vol. 13, № 11. - Art. 2069. - DOI: https://doi.org/10.3390/electronics13112069 - Data obrashcheniya: 12.03.2026.

7. Devlin J., Chang M.-W., Lee K., Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding [Elektronnyj resurs] // arXiv preprint. - 2018. - URL: https://doi.org/10.48550/arXiv.1810.04805 - Data obrashcheniya: 12.03.2026.

8. Aggarwal C. C. Neural Networks and Deep Learning [Elektronnyj resurs]. - Cham: Springer, 2018. - URL: https://link.springer.com/book/10.1007/978-3-319-94463-0 - Data obrashcheniya: 12.03.2026.

9. YAsnickij L. N. Intellektual'nye sistemy: uchebnik / L. N. YAsnickij. - Moskva: Laboratoriya znanij, 2020. - 224 s. - ISBN 978-5-00101-897-1.

10. Abdrahmanov B. S., Omarov B. B. Intellektual'nyj analiz dannyh: uchebnoe posobie. - Almaty: Қazaқ universiteti, 2021. - 240 s.

11. Jurafsky D., Martin J. H. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition [Elektronnyj resurs] - 3rd ed. - Stanford University, 2026. - URL: https://web.stanford.edu/~iurafskv/slp3/ - Data obrashcheniya: 12.03.2026.

12. He K., Zhang X., Ren S., Sun J. Deep Residual Learning for Image Recognition // Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). - 2016. - P. 770-778. - DOI: https://doi.org/10.1109/CVPR.2016.90

13. Urmashev B. A. Information-communication technology: uchebnoe posobie / B. A. Urmashev. - Almaty: Қazaқ universiteti, 2017. - 336 s.

14. Gandhi A., Adhvaryu K., Poria S., Cambria E., Hussain A. Multimodal Sentiment Analysis: A Systematic Review of History, Datasets, Multimodal Fusion Methods, Applications, Challenges and Future Directions // Information Fusion. - 2023. - Vol. 91. - P. 424-444. - DOI: https://doi.org/10.1016/i.inffus.2022.09.025 - Data obrashcheniya: 12.03.2026.

15. Horoshevskij V. F. Iskusstvennyj intellekt: uchebnik / V. F. Horoshevskij. - Moskva: Piter, 2018. - 496 s.

16. Nurtazin K. K., Sagyndykov ZH. S. Analiz dannyh i intellektual'nye sistemy: uchebnoe posobie / K. K. Nurtazin, ZH. S. Sagyndykov. - Almaty: KazNU im. al'-Farabi, 2020. - 296 s.

17. Flah P. Mashinnoe obuchenie: nauka i iskusstvo postroeniya algoritmov, kotorye izvlekayut znaniya iz dannyh / P. Flah; per. s angl. - Moskva: DMK Press, 2015. - 400 s.

18. Gavrilov A. V. Mashinnoe obuchenie: metody i algoritmy: uchebnoe posobie / A. V. Gavrilov. - Moskva: MGTU im. N. E. Baumana, 2019. - 312 s.

19. Omarov A. T., Nurpeisov D. M. Mashinnoe obuchenie i analiz dannyh: uchebnoe posobie / A. T. Omarov, D. M. Nurpeisov. - Astana: ENU im. L. N. Gumilyova, 2021. - 284 s.

20. Kozhahmetov E. B. Obrabotka bol'shih dannyh: uchebnoe posobie / E. B. Kozhahmetov. - Almaty: KazNITU im. K. I. Satpaeva, 2022. - 300 s.