Showing 1–1 of 1 results for author: Ruohe, A

Finnish Language Modeling with Deep Transformer Models

Authors: Abhilash Jain, Aku Ruohe, Stig-Arne Grönroos, Mikko Kurimo

Abstract: Transformers have recently taken the center stage in language modeling after LSTM's were considered the dominant model architecture for a long time. In this project, we investigate the performance of the Transformer architectures-BERT and Transformer-XL for the language modeling task. We use a sub-word model setting with the Finnish language and compare it to the previous State of the art (SOTA) L… ▽ More Transformers have recently taken the center stage in language modeling after LSTM's were considered the dominant model architecture for a long time. In this project, we investigate the performance of the Transformer architectures-BERT and Transformer-XL for the language modeling task. We use a sub-word model setting with the Finnish language and compare it to the previous State of the art (SOTA) LSTM model. BERT achieves a pseudo-perplexity score of 14.5, which is the first such measure achieved as far as we know. Transformer-XL improves upon the perplexity score to 73.58 which is 27\% better than the LSTM model. △ Less

Submitted 27 March, 2020; v1 submitted 14 March, 2020; originally announced March 2020.

Comments: 4 pages