Loading...

Development of an Interactive Iranian Sign Language Interface using Deep Learning Methods and Large Language Models

Aghakhani, Sahar | 2025

235 Viewed
  1. Type of Document: M.Sc. Thesis
  2. Language: Farsi
  3. Document No: 58703 (08)
  4. University: Sharif University of Technology
  5. Department: Mechanical Engineering
  6. Advisor(s): Taheri, Alireza; Meghdari, Ali
  7. Abstract:
  8. Iranian Sign Language (IrSL) is one of the primary means of communication for the Deaf and hard-of-hearing (DHH) community in Iran. However, the limited number of interpreters who can translate IrSL into spoken language has made access to information and public services challenging for these individuals. This issue highlights the necessity of developing intelligent systems for sign language translation and interaction. This research aims to develop a system for translating IrSL sentences and enabling text-based interaction with DHH individuals. Translation of signed sentences in other international sign languages, such as German or American Sign Language, has been investigated in previous studies. However, due to the lack of a parallel linguistic corpus for IrSL, gloss-free translation of sentences in this language has not been previously studied. In this study, gloss-free translation of sentences in Standard IrSL is examined. In addition, the first intelligent IrSL interaction framework is presented, providing experimental video-to-text interaction between a DHH user and a computer. As part of this research, the first comprehensive corpus of Standard IrSL was collected and annotated at the sentence level. This dataset, named IrSL-News, consists of 9,078 sign language videos totaling 16 hours, which were extracted from Deaf news broadcasts on Channel Two of IRIB. The model architecture used in this study was selected and implemented similar to SpaMo, one of the state-of-the-art models for gloss-free sign language translation. Evaluation results using a random data split show that the model achieved a BLEU-4 score of 28.80, indicating acceptable performance in sign language translation. It should be noted that evaluation under a signer-independent scenario (leave one signer out) shows that the model still faces challenges in generalizing to unseen individuals. Finally, the results of this research indicate that the trained model can be integrated with a large language model and deployed as an interactive system. In the experimental evaluation of the interactive framework, the response time from receiving the video to displaying the textual output was 3 minutes and 12 seconds, demonstrating the need for further optimization of the current system for real-time applications
  9. Keywords:
  10. Machine Translation ; Large Language Model ; Deep Learning ; Iranian Sign Language ; Persian Sign Language

 Digital Object List

 Bookmark

...see more