Bridging Communication Gaps: Advancements, Challenges, and Future Directions in Text-to-Sign Language Translation
DOI:
https://doi.org/10.62411/faith.3048-3719-91Keywords:
Computer Vision, Machine Translation, Natural Language Processing, Sign Language, Sign Notations, Text-to-Sign ConversionAbstract
Communication is an essential part of day-to-day life. However, people who have disabling hearing loss, which in general is known as deafness, or have a severe loss of hearing use sign language for their communication. Those people learn this language through their respective schools, whereas hearing people rarely get to know this communication tool, leaving a wide communication gap. To bridge this gap, an automated communication system that can convert text to sign language without depending on human interpreters. This literature review overviews sign language and discusses the different technologies used for text-to-sign conversion systems. The paper evaluates computer-based text processing methods, machine translation techniques (rule-based, corpus-based, and neural architectures), and sign synthesis approaches (avatar animation, video concatenation, neural generation). The paper compares each method's strengths and limitations and presents quantitative performance data from recent systems. The review also provides a comprehensive summary of widely used datasets and introduces a visual taxonomy of translation approaches. Furthermore, the paper discusses the current limitations, including scalability, data limitations, the need for more expressive synthesis, and future scope for improvement. The expected outcome of this review is to find and gather information on various sign language systems worldwide, identify research gaps, and propose further enhancements in text-to-sign language conversion.
Downloads
References
M. Alaghband, H. R. Maghroor, and I. Garibay, “A survey on sign language literature,” Mach. Learn. with Appl., vol. 14, p. 100504, Dec. 2023, doi: 10.1016/j.mlwa.2023.100504.
L. S. K. Udugama, R. Nethsinghe, J. Southcott, S. Kularathna, T. D. T. L. Dhanapala, and K. A. C. Alwis, “Sign language usage of deaf or hard of hearing Sri Lankans,” J. Deaf Stud. Deaf Educ., vol. 29, no. 2, pp. 187–198, Mar. 2024, doi: 10.1093/deafed/enad055.
P. Fernando and P. Wimalaratne, “Sign Language Translation Approach to Sinhalese Language,” GSTF J. Comput., vol. 5, no. 1, p. 9, Sep. 2016, doi: 10.7603/s40601-016-0009-8.
M. De Coster, D. Shterionov, M. Van Herreweghe, and J. Dambre, “Machine translation from signed to spoken languages: state of the art and challenges,” Univers. Access Inf. Soc., vol. 23, no. 3, pp. 1305–1331, Aug. 2024, doi: 10.1007/s10209-023-00992-1.
I. Marshall and É. Sáfár, “A prototype text to British Sign Language (BSL) translation system,” in Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - ACL ’03, 2003, vol. 2, pp. 113–116. doi: 10.3115/1075178.1075194.
C. Miller, “Section I: Some reflections on the need for a common sign notation,” Sign Lang. Linguist., vol. 4, no. 1–2, pp. 11–28, Dec. 2001, doi: 10.1075/sll.4.12.04mil.
A. Othman and M. Jemni, “An XML-gloss annotation system for sign language processing,” in 2017 6th International Conference on Information and Communication Technology and Accessibility (ICTA), Dec. 2017, vol. 2017-Decem, pp. 1–7. doi: 10.1109/ICTA.2017.8336054.
S. Prillwitz, Hamburg Notation System for Sign Languages: an introductory guide: version 2.0, vol. 5. Hamburg: Signum Press, 1989, 1989. [Online]. Available: https://cnlse.es/es/recursos/biblioteca/hamburg-notation-system-sign-languages-introductory-guide-version-20
T. Hanke, “HamNoSys-Representing sign language data in language resources and language processing contexts,” in Proceedings of the LREC2004 Workshop on the Representation and Processing of Sign Languages: From SignWriting to Image Processing. Information techniques and their implications for teaching, documentation and communication, 2004, pp. 1–6. [Online]. Available: https://www.sign-lang.uni-hamburg.de/lrec/pub/04001.pdf
V. Skobov and Y. Lepage, “Video-to-HamNoSys Automated Annotation System,” in Proceedings of the LREC2020 9th Workshop on the Representation and Processing of Sign Languages: Sign Language Resources in the Service of the Language Community, Technological Challenges and Application Perspectives, 2020, pp. 11–16. [Online]. Available: https://aclanthology.org/2020.signlang-1.34/
R. Kennaway, “Synthetic animation of deaf signing gestures,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2002, vol. 2298, pp. 146–157. doi: 10.1007/3-540-47873-6_15.
R. Elliott, J. R. W. Glauert, J. R. Kennaway, I. Marshall, and E. Safar, “Linguistic modelling and language-processing technologies for Avatar-based sign language presentation,” Univers. Access Inf. Soc., vol. 6, no. 4, pp. 375–391, Feb. 2008, doi: 10.1007/s10209-007-0102-z.
K. Kaur and P. Kumar, “HamNoSys to SiGML Conversion System for Sign Language Automation,” Procedia Comput. Sci., vol. 89, pp. 794–803, 2016, doi: 10.1016/j.procs.2016.06.063.
A. H. Aliwy and A. A. Alethary, “Development of arabic sign language dictionary using 3D avatar technologies,” Indones. J. Electr. Eng. Comput. Sci., vol. 21, no. 1, p. 609, Jan. 2021, doi: 10.11591/ijeecs.v21.i1.pp609-616.
I. Zwitserlood, M. Verlinden, J. Ros, and S. Van Der Schoot, “Synthetic signing for the deaf: Esign,” 2005. [Online]. Available: https://api.semanticscholar.org/CorpusID:199412908
J. M. De Martino et al., “Neural machine translation from text to sign language,” Univers. Access Inf. Soc., vol. 24, no. 1, pp. 37–50, Mar. 2025, doi: 10.1007/s10209-023-01018-6.
B. Shi, D. Brentari, G. Shakhnarovich, and K. Livescu, “Fingerspelling Detection in American Sign Language,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun. 2021, pp. 4164–4173. doi: 10.1109/CVPR46437.2021.00415.
B. Lee and K. Secora, “Fingerspelling and Its Role in Translanguaging,” Languages, vol. 7, no. 4, p. 278, Nov. 2022, doi: 10.3390/languages7040278.
D. Waters et al., “Fingerspelling, signed language, text and picture processing in deaf native signers: The role of the mid-fusiform gyrus,” Neuroimage, vol. 35, no. 3, pp. 1287–1302, Apr. 2007, doi: 10.1016/j.neuroimage.2007.01.025.
M. Punchimudiyanse and R. G. N. Meegama, “Animation of Fingerspelled Words and Number Signs of the Sinhala Sign Language,” ACM Trans. Asian Low-Resource Lang. Inf. Process., vol. 16, no. 4, pp. 1–26, Dec. 2017, doi: 10.1145/3092743.
M. Brour and A. Benabbou, “ATLASLang MTS 1: Arabic Text Language into Arabic Sign Language Machine Translation System,” Procedia Comput. Sci., vol. 148, pp. 236–245, 2019, doi: 10.1016/j.procs.2019.01.066.
N. K. Kahlon and W. Singh, “Machine translation from text to sign language: a systematic review,” Univers. Access Inf. Soc., vol. 22, no. 1, pp. 1–35, Mar. 2023, doi: 10.1007/s10209-021-00823-1.
P. Sharma, D. Tulsian, C. Verma, P. Sharma, and N. Nancy, “Translating Speech to Indian Sign Language Using Natural Language Processing,” Futur. Internet, vol. 14, no. 9, p. 253, Aug. 2022, doi: 10.3390/fi14090253.
G. Vardan, S. Sinha, P. Bhushan, and M. Shettigar, “English Text to Indian Sign Language Translator COEN 296B-Natural Language Processing.” [Online]. Available: https://www.cse.scu.edu/~m1wang/projects/NLP_English2IndianSignLanuguage_18w.pdf
D. Klein and C. D. Manning, “Accurate Unlexicalized Parsing,” in Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, 2003, pp. 423–430. [Online]. Available: http://nlp.stanford.edu/downloads/lex-parser.shtml
P Shriya Rao, T Vamsi Rohit, B Manasa, and Prof. V Lingamaiah, “Multiple Languages to Sign Language Using NLTK,” Int. J. Sci. Res. Sci. Technol., pp. 12–17, Mar. 2023, doi: 10.32628/IJSRST2310189.
Z. Tmar, A. Othman, and M. Jemni, “A rule-based approach for building an artificial English-ASL corpus,” in 2013 International Conference on Electrical Engineering and Software Applications, Mar. 2013, pp. 1–4. doi: 10.1109/ICEESA.2013.6578458.
A. Ghosh and R. Mamidi, “English To Indian Sign Language: Rule-Based Translation System Along With Multi-Word Expressions and Synonym Substitution,” in Proceedings of the 19th International Conference on Natural Language Processing (ICON), 2022, pp. 123–127. [Online]. Available: https://aclanthology.org/2022.icon-main.17/
M. Sanaullah et al., “A Real-Time Automatic Translation of Text to Sign Language,” Comput. Mater. Contin., vol. 70, no. 2, pp. 2471–2488, 2022, doi: 10.32604/cmc.2022.019420.
Z. Dajun and W. Yun, “Corpus-based Machine Translation: Its Current Development and Perspectives,” Int. Forum Teach. Stud., vol. 11, no. 1/2, p. 2015, 2015, doi: https://americanscholarspress.us/journals/IFST/pdf/IFOTS-2-2015/v11n2-art10.pdf.
S. Morrissey and A. Way, “An Example-Based Approach to Translating Sign Language,” in Workshop on example-based machine translation, 2005. [Online]. Available: https://aclanthology.org/2005.mtsummit-ebmt.14/
L. E. Bertin-Lemée, A. Braffort, C. Challant, C. Danet, and M. Filhol, “Example-Based Machine Translation from Text to a Hierarchical Representation of Sign Languagé,” in Proceedings of the 24th Annual Conference of the European Association for Machine Translation, 2023, pp. 21–30. [Online]. Available: https://aclanthology.org/2023.eamt-1.3/
Z. Tan et al., “Neural machine translation: A review of methods, resources, and tools,” AI Open, vol. 1, pp. 5–21, 2020, doi: 10.1016/j.aiopen.2020.11.001.
S. Stoll, N. C. Camgoz, S. Hadfield, and R. Bowden, “Sign Language Production using Neural Machine Translation and Generative Adversarial Networks,” in 29th British Machine Vision Conference (BMVC 2018) (Northumbria University, Newcastle Upon Tyne, UK, 03/09/2018–06/09/2018), 2018.
A. B. Grieve-Smith, “SignSynth: A Sign Language Synthesis Application Using Web3D and Perl,” in Gesture and Sign Language in Human-Computer Interaction, 2002, pp. 134–145. doi: 10.1007/3-540-47873-6_14.
S. S. Thrimahavithana, V. N. Yasodha, M. U. Kannangara, V. W. Welgama, and A. R. Weerasinghe, “Empowering the Text-based Understandability of Students with Hearing Impairments,” in 2019 19th International Conference on Advances in ICT for Emerging Regions (ICTer), Sep. 2019, pp. 1–9. doi: 10.1109/ICTer48817.2019.9023675.
M. Delorme, M. Filhol, and A. Braffort, “Animation Generation Process for Sign Language Synthesis,” in 2009 Second International Conferences on Advances in Computer-Human Interactions, Feb. 2009, pp. 386–390. doi: 10.1109/ACHI.2009.29.
F. Quiroga, “Sign language recognition datasets,” GitHub.io. https://facundoq.github.io/guides/sign_language_datasets/slr
E.-J. Ong, N. Pugeault, and R. Bowden, “Sign Spotting Using Hierarchical Sequential Patterns with Temporal Intervals,” in 2014 IEEE Conference on Computer Vision and Pattern Recognition, Jun. 2014, pp. 1931–1938. doi: 10.1109/CVPR.2014.248.
H. Cooper, E.-J. Ong, N. Pugeault, and R. Bowden, “Sign Language Recognition Using Sub-units,” in Journal of Machine Learning Research, vol. 13, 2017, pp. 89–118. doi: 10.1007/978-3-319-57021-1_3.
O. Koller, J. Forster, and H. Ney, “Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers,” Comput. Vis. Image Underst., vol. 141, pp. 108–125, Dec. 2015, doi: 10.1016/j.cviu.2015.09.013.
A. Thangali, J. P. Nash, S. Sclaroff, and C. Neidle, “Exploiting phonological constraints for handshape inference in ASL video,” in CVPR 2011, Jun. 2011, pp. 521–528. doi: 10.1109/CVPR.2011.5995718.
M. Oszust and M. Wysocki, “Polish sign language words recognition with Kinect,” in 2013 6th International Conference on Human System Interactions (HSI), Jun. 2013, pp. 219–226. doi: 10.1109/HSI.2013.6577826.
F. Ronchetti, F. Quiroga, C. Estrebou, L. Lanzarini, and A. Rosete, “Sign Languague Recognition Without Frame-Sequencing Constraints: A Proof of Concept on the Argentinian Sign Language,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10022 LNAI, Springer Verlag, 2016, pp. 338–349. doi: 10.1007/978-3-319-47955-2_28.
C. Chen, B. Zhang, Z. Hou, J. Jiang, M. Liu, and Y. Yang, “Action recognition from depth sequences using weighted fusion of 2D and 3D auto-correlation of gradients features,” Multimed. Tools Appl., vol. 76, no. 3, pp. 4651–4669, Feb. 2017, doi: 10.1007/s11042-016-3284-7.
Y. Min, “Kinect Sign Language Working Group,” Visual Information Processing and Learning, 2025. http://vipl.ict.ac.cn/homepage/ksl/data.html#page3
A. Nandy, S. Mondal, J. S. Prasad, P. Chakraborty, and G. C. Nandi, “Recognizing & interpreting Indian Sign Language gesture for Human Robot Interaction,” in 2010 International Conference on Computer and Communication Technology (ICCCT), Sep. 2010, pp. 712–717. doi: 10.1109/ICCCT.2010.5640434.
S. M. Shohieb, H. K. Elminir, and A. M. Riad, “SignsWorld Atlas; a benchmark Arabic Sign Language database,” J. King Saud Univ. - Comput. Inf. Sci., vol. 27, no. 1, pp. 68–76, Jan. 2015, doi: 10.1016/j.jksuci.2014.03.011.
A. M. Martinez, R. B. Wilbur, R. Shay, and A. C. Kak, “Purdue RVL-SLLL ASL database for automatic recognition of American Sign Language,” in Proceedings. Fourth IEEE International Conference on Multimodal Interfaces, pp. 167–172. doi: 10.1109/ICMI.2002.1166987.
P. Dal Bianco et al., “LSA-T: The First Continuous Argentinian Sign Language Dataset for Sign Language Translation,” in Advances in Artificial Intelligence – IBERAMIA 2022, 2022, pp. 293–304. doi: 10.1007/978-3-031-22419-5_25.
P. Lu and M. Huenerfauth, “Collecting and evaluating the CUNY ASL corpus for research on American Sign Language animation,” Comput. Speech Lang., vol. 28, no. 3, pp. 812–831, May 2014, doi: 10.1016/j.csl.2013.10.004.
J. Fink, B. Frenay, L. Meurant, and A. Cleve, “LSFB-CONT and LSFB-ISOL: Two New Datasets for Vision-Based Sign Language Recognition,” in 2021 International Joint Conference on Neural Networks (IJCNN), Jul. 2021, pp. 1–8. doi: 10.1109/IJCNN52387.2021.9534336.
D. Li, C. R. Opazo, X. Yu, and H. Li, “Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison,” in 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), Mar. 2020, pp. 1448–1458. doi: 10.1109/WACV45572.2020.9093512.
UniSerj. Sign Language Recognition., “Sign Language Recognition,” Kaggle, 2023. https://www.kaggle.com/competitions/sign-language-recognition
M. Wasim, A. Ahmed, A. Shaikh, L. Ahmed, S. Faisal, and F. Saeed, “Communicator for Hearing-Impaired Persons using Pakistan Sign Language (PSL),” Int. J. Adv. Comput. Sci. Appl., vol. 9, no. 5, 2018, doi: 10.14569/IJACSA.2018.090525.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Pathumi Ahinsa, Sanduni Thrimahavithana, Kasun Karunanayaka
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
