Recent progress in artificial intelligence (AI) has been mostly due to machine learning and, in particular, deep artificial neural networks (ANNs). Deep learning has an increasing presence in everyday life, including critical applications such as medical diagnosis, transportation, and energy distribution. In response to this, the field of Explainable AI (XAI) has generated much effort in terms of techniques and algorithms to address this problem. However, there is still no consensus on a suite of technology to address these challenges, progress has been extremely limited, and the formal properties of such systems are under-studied. On the other hand, computational neuroscience (CNS) aims to discover the principles behind biological neural networks that enable the brain to support cognition, perception, and action. This project will employ the latest approaches and techniques used in the field of CNS to develop the field of XAI. Specifically, the first major goal will be to employ the methods of representational geometry and neural encoding manifolds (both proven to be effective in revealing meaningful neural relationships in previous studies) to reveal how activations of collections of artificial neurons in hidden layers are associated with the decision-making process of deep networks. Second, the same methodology will be used to reveal novel insights from a variety of existing large-scale biological datasets. Finally, we will compare and contrast the encoding strategies of neural populations found various deep learning architectures with those observed in biological networks. A better understanding of the inner-workings of biological models could directly inform researchers on how to build novel artificial models that are more accurate, robust, and even economical during both training and inference in terms of data, time, and energy consumption.