Medical imaging techniques such as X-ray, provide clinicians intensive information on the disease/condition of patients. However, clinicians have to look away from the subject to refer to medical images, thereby losing track of their work. Thus, clinicians usually study the images prior to surgery and limit reference time to images during surgery. Furthermore, unlike X-ray, novel imaging methods (such as optical ultrasound) are not taught in medical schools, so untrained clinicians face challenges in interpreting the images. These two limitations restrict the clinicians’ ability to fully utilize and adopt advanced medical imaging techniques. In this work, we explore the possibility of using Augmented and Virtual Reality (AR/VR) in the context of medical imaging. Prior applications of AR/VR technology in medicine have been limited to AR-aided training for medical students, telepresence for interaction, as well as remote therapy. We aim to use AR as a real-time diagnostic and therapeutic tool by augmenting the clinicians’ live view with various imaging modalities (such as X-ray, optical ultrasound, near-infrared). We hypothesize that providing these images in-context, and in some cases aligned with the subject, will improve the interpretation of images resulting in better guidance for diagnosis or surgery. To test this, we are creating an AR-based medical imaging/analysis application that uses techniques such as volumetric rendering and real-time image registration to augment the clinicians' view. Furthermore, clinicians can interact with the images by filtering, slicing, and reducing dimensionality, in order to better understand the images and thereby the underlying disease/condition.