Language Modeling Using Recurrent Neural Networks

Please enable javascript in your browser.

Rahimi, Adel | 2019

1018 Viewed

Type of Document: M.Sc. Thesis
Language: Farsi
Document No: 51718 (31)
University: Sharif University of Technology
Department: Languages and Linguistics Center
Advisor(s): Sameti, Hossein
Abstract:
This thesis examines the differences and the similarities between the two famous RNN blocks the Long Short Term Memory and the Gated Recurrent Unit. It measure different aspects such as computational complexity, Word Error Rate, and subjective human evaluation in the task of text generation.In the computational complexity experiment results show that the LSTM takes more time to compute, in comparison to the GRU. Moving on into the next experiment the GRU slightly outperforms the LSTM in terms of WER but the perplexity for the language models tested was the same. This shows that slight differences in the perplexity does not drastically change the WER. Having said, the results suggest that the GRU unit will work in par with the LSTM unit and in some cases outperforms the LSTM. Having the computational complexity in mind we’d suggest using the GRU since it is faster.In the end, we have modified the GRU unit, implementing the ReLU6 activation function into it. With the new activation function of ReLU6, the GRU unit outperforms the vanilla GRU unit in terms of computational complexity and is in par with
it in terms of perplexity
Keywords:
Language Modeling ; Recurrent Neural Networks ; Speech Recognition ; Long Short Term Memory (LSTM)