This paper presents preliminary results of a study of Japanese native speakers working with the Microsoft Word application in two modalities: the released Japanese version and a machine translated (MT) version (the raw MT strings incorporated into the MS Word interface). To explore the effect of translation modality on task completion, time and satisfaction, an experiment using an eye-tracker was set up with a group of 42 users: 22 native Japanese and 20 native English speakers. The results suggest that Japanese-native speakers have higher completion scores and are more efficient when working with the released versions of the product than with the MT version, but these differences are not significant. Their self-reported satisfaction, however, is significantly higher when working with the released product as opposed to the raw MT version.