The need for accurate and effective translation cannot be overstated in an increasingly globalised world where communication is paramount. Bridging language barriers is important for promoting understanding and cooperation among diverse individuals and communities, making translation an indispensable tool. Over the past two decades, Machine Translation (MT) has undergone remarkable advancements, with significant progress attributed to the emergence of Neural Machine Translation (NMT), primarily the groundbreaking Transformer models. This rapid development in MT, which started with a focus on sentence-level translation, has not only bridged communication gaps but also brought MT systems close to delivering human-like performance on various translation tasks. While these advances are significant, focusing mainly on sentence-level modeling and evaluation, they miss the valuable contextual information around each sentence. Contextual information in document-level translation helps resolve language ambiguities and ensures consistency and coherence in the translated text, making the translation more accurate and readable. While considerable efforts have been made to incorporate context into NMT systems, the community has not reached a consensus on the most effective methods and the types of context to be integrated. In this thesis, my primary focus is on understanding document-level systems. Specifically, I explore how these systems incorporate context into their translation processes and investigate the span of context utilised by these systems. I also investigate the terminology translation mechanisms within these systems. Furthermore, with the emergence of modern-day powerful Large Language Models (LLMs), I examine their capabilities in terminology translation and propose new methodologies to improve terminology translation for these powerful models.