Loading...

Transformers for Bridging Persian Dialects: Transliteration Model for Tajiki and Iranian Scripts

Sadraei Javaheri, M ; Sharif University of Technology | 2024

65 Viewed
  1. Type of Document: Article
  2. Publisher: 2024
  3. Abstract:
  4. In this study, we address the linguistic challenges posed by Tajiki Persian, a distinct variant of the Persian language that utilizes the Cyrillic script due to historical “Russification”. This distinguishes it from other Persian dialects that adopt the Arabic script. Despite its profound linguistic and cultural significance, Tajiki Persian remains a low-resource language with scant digitized datasets for computational applications. To address this deficiency, we created a parallel corpus using Shahnameh, a seminal Persian epic poem. Employing optical character recognition, we extracted Tajiki Persian verses from primary sources and applied a heuristic method to align them with their Iranian Persian counterparts. We then trained and assessed transliteration models using two prominent sequence-to-sequence architectures: GRU with attention and transformer. Our results underscore the enhanced performance of our models, particularly in contrast to pre-trained large multilingual models like GPT-3.5, emphasizing the value of dedicated datasets in advancing computational approaches for underrepresented languages. With the publication of this work, we are disseminating, for the first time, a vast collection of Persian poetry spanning 1000 years, transcribed in Tajiki scripts for the benefit of the Tajiki-speaking communities. The dataset, along with the model's code and checkpoints, is accessible at https://github.com/language-ml/Tajiki-Shahname, marking a significant contribution to computational linguistic resources for Tajiki Persian. © 2024 ELRA Language Resource Association: CC BY-NC 4.0
  5. Keywords:
  6. Tajiki Persian ; Transliteration ; Large datasets ; Linguistics ; Optical character recognition ; Arabic scripts ; Computational applications ; Iranian persian ; Low resource languages ; Persian languages ; Persians ; Transliteration models ; Heuristic methods
  7. Source: 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC-COLING 2024 - Main Conference Proceedings ; 2024 , Pages 16770-16775 ; 978-249381410-4 (ISBN)
  8. URL: https://aclanthology.org/2024.lrec-main.1459