Home⇛Isabela State University Linker: Journal of Engineering, Computing and Technology⇛vol. 2 no. 1 (2025)

Real-Time Sign Language Recognition and Translation: A Mobile Solution Using Convolutional Neural Network

Jessica Reshelle B. Naraja | Client Joseph S. Leyson | Jessica Rose E. Fernandez

Discipline: Artificial Intelligence

 

Abstract:

This study presents a mobile application for sign language recognition and translation using a convolutional neural network (CNN) to overcome communication barriers for the deaf community. Unlike existing solutions, the app uses a CNN trained on a dataset of 200–450 images per sign to process hand images via preprocessing, feature extraction, and hand landmark detection, accurately recognizing sign language gestures. The application underscores a user-friendly interface and is designed for real-time mobile use. Employing CNN-based image processing, it translates hand movements into gestures with high precision, achieving 96% accuracy and a loss of 0.069 after 100 training epochs with a batch size of 40. Usability testing, conducted using the System Usability Scale (SUS) questionnaire, revealed high user satisfaction, with positive feedback on usability, functionality, maintainability, and efficiency. The average SUS score indicates an excellent usability. Further evaluation criteria included precision, recall, and F1-score, all of which demonstrated strong performance. The system effectively bridges the communication gap between the deaf and hearing communities, fostering more accessible and meaningful interactions.



References:

  1. Al-Hammadi, M., Muhammad, G., Abdul, W., Alsulaiman, M., Bencherif, M. A., & Mekhtiche, M. A. (2020). Hand gesture recognition for sign language using 3D CNN. IEEE Access, 8, 79,491–79,509. https://doi.org/10.1109/ACCESS.2020.2990191
  2. Farooqui, A., Hatekar, A., Angadi, P., & Shrikant, K. (2021). A sign translation app to convert ASL to auditory English language. i-manager’s Journal on Mobile Applications and Technologies, 8(1), 1–7. https://search.proquest.com/openview/7cf43e8d92d86c58da13b0114a302773/1
  3. Han, L. H. Y., & Murli, N. (2022). BIM sign language translator using machine learning (TensorFlow). Journal of Soft Computing and Data Mining, 3(1), 68–77. https://penerbit.uthm.edu.my/ojs/index.php/jscdm/article/view/11662
  4. Hoque, O. B., Jubair, M. I., Islam, M. S., Akash, A.-F., & Paulson, A. S. (2018, November 30). Real time Bangladeshi sign language detection using Faster R-CNN. arXiv. https://arxiv.org/abs/1811.12813
  5. Ismail, M. H. I., Dawwd, S., & Ali, F. H. (2021). Static hand gesture recognition of Arabic sign language using deep CNNs. International Journal of Computer Applications, 975, Article 8887. https://www.researchgate.net/publication/352540178
  6. Jagtap, S., Jadhav, K., Temkar, R., & Deshmukh, M. (2024). Real-time sign language recognition using MobileNetV2 and transfer learning. arXiv. https://arxiv.org/abs/2401.09251
  7. Li, D., Rodriguez Opazo, C., Yu, X., & Li, H. (2019). Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. arXiv. https://arXiv.org/abs/1910.11006
  8. Li, Y., Wang, J., & Smith, D. (2019). WLASL: A large-scale word-level American Sign Language dataset on the web. In Proceedings of the 27th ACM International Conference on Multimedia (pp. 1237–1245). https://doi.org/10.1145/3343031.3350603
  9. Li, K., Zhou, Z., & Lee, C.-H. (2017). Sign transition modeling and a scalable solution to continuous sign language recognition for real-world applications. Journal of Computing and Information Technology, 25(1), 53–63. https://doi.org/10.1145/2850421
  10. Murali, R. S. L., Ramayya, L. D., & Santosh, V. A. (2020). Sign language recognition system using convolutional neural network and computer vision. International Journal of Engineering Innovations in Advanced Technology, 4(4), 1–7. Retrieved from https://www.ijert.org/sign-language-recognition-system-using-convolutional-neural-network-and-computer-vision
  11. Nagy, D. Z., & Piller, I. (2020). An optical flow–based gesture recognition method. Acta Marisiensis, Series Technologica, 17(1), 55–60. https://doi.org/10.2478/amset-2020-0010
  12. Rastgoo, R., Kiani, K., & Escalera, S. (2021). Sign language recognition: A deep survey. Expert Systems with Applications, 164, 113,794. https://doi.org/10.1016/j.eswa.2020.113794
  13. Rathi, D. (2018). Optimization of transfer learning for sign language recognition targeting mobile platform. arXiv. https://arXiv.org/abs/1811.08421
  14. Sarma, D., Kavyasree, V., & Bhuyan, M. K. (2020). Two-stream fusion model for dynamic hand gesture recognition using 3D-CNN and 2D-CNN optical flow guided motion template. arXiv. https://arXiv.org/abs/2006.11338
  15. Selvaraj, P., Gokul, N. C., Kumar, P., & Khapra, M. (2021). OpenHands: Making sign language recognition accessible with pose-based pretrained models across languages. arXiv. https://arXiv.org/abs/2109.03480
  16. Sreemathy, R., Turuk, M., Kulkarni, I., & Khurana, S. (2023). A multimodal approach to sign language recognition using deep learning. Education and Information Technologies, 28(5), 5,259–5,278.
  17. Viswavarapu, L. K. (2018). Real-time finger spelling American sign language recognition using deep convolutional neural networks (Master’s thesis, University of North Texas). UNT Digital Library. https://digital.library.unt.edu/ark:/67531/metadc1404544/
  18. Wadhawan, A., & Kumar, P. (2020). Deep learning–based sign language recognition system for static signs. Neural Computing and Applications, 32, 7,957–7,968. https://doi.org/10.1007/s00521-019-04691-y
  19. Xia, K., Lu, W., Fan, H., & Zhao, Q. (2022). A Sign Language Recognition System Applied to Deaf-Mute Medical Consultation. Sensors, 22(23), 9107. https://doi.org/10.3390/s22239107
  20. Zhang, J., Bu, X., Wang, Y., Dong, H., Zhang, Y., & Wu, H. (2024). Sign language recognition based on dual-path background erasure convolutional neural network. Scientific Reports, 14(1). https://doi.org/10.1038/s41598-024-62008-z

Tools


Copyright © 2025  KITE Digital Educational Solutions |  Exclusively distributed by CE-Logic