Reading is one of the most common everyday activities. People read through most of their daily context such as during study or for entertainment in their spare time. Despite playing a critical role in our lives, there has been limited research on how people read and how it affects their level of understanding. The NTCIR-16 RCIR challenge is the first collaborative evaluation that aims to automatically measure the reading comprehension of a reader and integrate it as part of the information retrieval process. In this paper, we present our approach for the NTCIR-16 RCIR challenge, in which task participants are required to predict reading comprehension using eye movement signals of the readers. We utilised several conventional machine learning techniques to estimate the level of comprehension and combined it with a language model to perform text retrieval. Our extensive experiments, covering both subject-dependent and subject-independent scenarios, showed that our approach with fine-tuning obtained a Spearman’s coefficient of 0.5993 for the comprehension-evaluation task and nDCG at 0.7296 for the comprehension-based retrieval task.