Explainable Artificial Intelligence (XAI) has recently become an active research field due to the need for transparency and accountability when deploying AI models for high-stake decision making. Despite the success of Deep Neural Networks (DNNs), understanding their decision processes is still a known challenge. The research direction presented in this thesis stems from the idea that combining knowledge with deep representations can be the key to more transparent decision making. Specifically, we have focused on Computer Vision tasks and Convolutional Neural Networks (CNNs) and we have proposed a graph representation, called co-activation graph, that serves as an intermediate representation between knowledge encoded within a CNN and the semantics contained in external knowledge bases. Given a trained CNN, we first show how a co-activation graph can be created and exploited to generate global insights for the model’s inner-workings. Then, we propose a taxonomy extraction method that captures how symbolic class concepts and their hypernyms in a given domain are hierarchically organised in the model’s subsymbolic representation. We then illustrate how background knowledge can be connected to the graph in order to generate textual local factual and counterfactual explanations. Our results indicate that graph analysis approaches applied to co-activation graphs can reveal important insights into how CNNs work and enable both global and local semantic explanations. Despite focusing on CNN architectures, we believe that our approach can be adapted to other architectures which would make it possible to apply the same methodology in other domains such as Natural Language Processing. At the end of the thesis we will discuss interesting research directions that are being investigated in the area of using knowledge graphs and graph analysis for explainability of deep learning models, and we outline opportunities for the development of this research area