Login (DCU Staff Only)
Login (DCU Staff Only)

DORAS | DCU Research Repository

Explore open access research and scholarly works from DCU

Advanced Search

Eye tracking as an MT evaluation technique

Doherty, Stephen orcid logoORCID: 0000-0003-0887-1049, O'Brien, Sharon orcid logoORCID: 0000-0003-4864-5986 and Carl, Michael (2010) Eye tracking as an MT evaluation technique. Machine Translation, 24 (1). pp. 1-13. ISSN 0922-6567

Abstract
Eye tracking has been used successfully as a technique for measuring cognitive load in reading, psycholinguistics, writing, language acquisition etc. for some time now. Its application as a technique for measuring the reading ease of MT output has not yet, to our knowledge, been tested. We report here on a preliminary study testing the use and validity of an eye tracking methodology as a means of semi-automatically evaluating machine translation output. 50 French machine translated sentences, 25 rated as excellent and 25 rated as poor in an earlier human evaluation, were selected. Ten native speakers of French were instructed to read the MT sentences for comprehensibility. Their eye gaze data were recorded non-invasively using a Tobii 1750 eye tracker. The average gaze time and fixation count were found to be higher for the “bad” sentences, while average fixation duration and pupil dilations were not found to be substantially different for output rated as good and output rated as bad. Comparisons between HTER scores and eye gaze data were also found to correlate well with gaze time and fixation count, but not with pupil dilation and fixation duration. We conclude that the eye tracking data, in particular gaze time and fixation count, correlate reasonably well with human evaluation of MT output but fixation duration and pupil dilation may be less reliable indicators of reading difficulty for MT output. We also conclude that eye tracking has promise as a semi-automatic MT evaluation technique, which does not require bi-lingual knowledge, and which can potentially tap into the end users’ experience of machine translation output.
Metadata
Item Type:Article (Published)
Refereed:Yes
Uncontrolled Keywords:MT evaluation; Eye tracking; Gaze time; Fixation count; Fixation duration; Pupil dilation; HTER
Subjects:Computer Science > Machine translating
Humanities > Linguistics
DCU Faculties and Centres:Research Initiatives and Centres > Centre for Next Generation Localisation (CNGL)
DCU Faculties and Schools > Faculty of Humanities and Social Science > School of Applied Language and Intercultural Studies
Publisher:Kluwer
Official URL:http://link.springer.com/article/10.1007/s10590-01...
Copyright Information:© 2010 Springer (Kluwer) The original publication is available at www.springerlink.com
Use License:This item is licensed under a Creative Commons Attribution-NonCommercial-Share Alike 3.0 License. View License
Funders:Science Foundation Ireland
ID Code:19471
Deposited On:02 Oct 2013 09:58 by Stephen Doherty . Last Modified 19 Jan 2022 12:28
Documents

Full text available as:

[thumbnail of 3deec524abe336a672.pdf]
Preview
PDF - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
226kB
Downloads

Downloads

Downloads per month over past year

Archive Staff Only: edit this record