Home⇛Asia Pacific Journal of Management and Sustainable Development⇛vol. 12 no. 2 Part 4 (2024)

Sign Language Translator via Smartphone Image Analysis using Convolutional Neural Network

John Carlo Torres | Roselie Alday

Discipline: Artificial Intelligence

 

Abstract:

This study presents the design and implementation of a smartphone-based image processing system aimed at translating sign language gestures into text in real-time. The primary objective was to create a platform that enhances communication between deaf individuals and non-sign language users through the accurate recognition of sign language gestures. Using Kotlin programming and MediaPipe, the system captures and processes gestures, converting them into readable text. The results demonstrated that the system is capable of recognizing and translating various sign language gestures with a high degree of accuracy and efficiency. The successful deployment of this system highlights its potential to break down communication barriers and offers a foundation for future improvements, including expanded sign language support and performance optimization for real-world applications.



References:

  1. Goyal, M., & Goyal, R. (2017). Machine translation systems and quality assessment: A systematic review. Journal of Machine Translation, 31(3), 135-166.
  2. Kahlon, K. S., & Singh, P. (2022). A systematic review of machine translation from text to sign language. International Journal of Sign Language Translation, 5(2), 45-59
  3. Seema, & Singla, P. (2023). A Comprehensive Review of CNN-Based Sign Language Translation System. Proceedings of Data Analytics and Management, 572, 347-362.
  4. Ugale, M., Rodrigues, O., Shinde, A., Desle, K., & Yadav, S. (2023). A Review on Sign Language Recognition Using CNN. ResearchGate
  5. Android Developers. (2024). Image analysis | Android media | Android Developers. Android Developers Essentials Media dev center
  6. Bhagwat, A., Gupta, P., & Kadam, N. (2022). Conversion of Sign Language to Text Using Machine Learning. SpringerLink.
  7. Camgoz, N. C., Hadfield, S., Koller, O., & Bowden, R. (2020). Sign language transformers: Joint end-to-end sign language recognition and translation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  8. Cui, R., Liu, H., & Zhang, C. (2017). A deep neural framework for continuous sign language recognition by iterative training. IEEE Transactions on Multimedia, 20(12), 3257-3269.
  9. Dabre, R., & Sumita, E. (2021). YANMTT: Yet Another Neural Machine Translation Toolkit. arXiv preprint arXiv:2108.11126. Retrieved from https://arxiv.org/abs/2108.11126
  10. David, D., Alamoodi, A. H., Albahri, O. S., Garfan, S., Albahri, A. S., Zaidan, B. B., & Chen, J. (2024). Sign language mobile apps: a systematic review of current app evaluation progress and solution framework. Evolving Systems, 15, 669-686
  11. David, D., Alamoodi, A. H., Albahri, O. S., Zaidan, B. B., Zaidan, A. A., Garfan, S., Ismail, A. R., Albahri, A. S., Alsinglawi, B., & Malik, R. Q. (2023). Landscape of sign language research based on smartphone apps: coherent literature analysis, motivations, open challenges, recommendations, and future directions for app assessment. Universal Access in the Information Society, 10(2), 87-102
  12. Rastgoo, R., Kiani, K., & Escalera, S. (2021). Sign language recognition: A deep survey. Expert Systems with Applications, 164, 113794.
  13. Koller, O., Zargaran, S., & Ney, H. (2016). Deep sign: Hybrid CNN-HMM for continuous sign language recognition. British Machine Vision Conference.
  14. Huang, J., Zhou, W., Li, H., & Li, W. (2018). Sign language recognition using 3D convolutional neural networks. 2018 IEEE International Conference on Multimedia and Expo (ICME).
  15. Park, H. J., Lee, J. S., & Ko, J. G. (2020). Achieving Real-Time Sign Language Translation Using a Smartphone’s True Depth Images. In 2020 International Conference on Communication Systems and Networks (COMSNETS 2020) (pp. 622-625).
  16. Wadhawan, A., & Kumar, P. (2019). Sign Language Recognition Systems: A Decade Systematic Literature Review. Archives of Computational Methods in Engineering, 28, 785-8133.
  17. Zhang, X., et al. (2019). Smartphone-based Sign Language Recognition System Using CNN-LSTM Model. IEEE Access, 7, 127346-127355.
  18. Li, Y., et al. (2020). Smartphone-Assisted Sign Language Translation System Using Deep Learning. IEEE Access, 8, 21337-21345

Tools


Copyright © 2025  KITE Digital Educational Solutions |  Exclusively distributed by CE-Logic