After radiologists perform a set of chest-x-rays (CXRs, or radiographs) they write a short report, which is a free-text description of their observations and interpretations. Because these reports are free-text documents, there is the risk of miscommunication, which can result in reduced patient outcomes. In this study, we develop a predictive model that takes a radiology report as input and returns the probability that the report describes a positive diagnosis for pneumonia, a common respiratory condition characterized by the accumulation of fluid in the lungs. The development of such a model is challenging due to the complexity of human language (natural language). Natural language processing seeks to translate human language to a machine-understandable form. We systematically generated five predictive models. Briefly, using the R programming language, we 1) randomly assigned reports from the MIMIC-CXR database to the training set consisting of 700 reports and testing set consisting of 300 reports, 2) created a count matrix giving the frequencies of different sets of 3 consecutive words (trigram), 3) performed feature selection to identify terms that differentiate between positive and negative cases, and 4) trained the models (k nearest neighbor, random forest, gradient boosting machine, xgboost, adaboost). Our results indicate the xgboost algorithm performs the best on the testing set with a Brier Score (bs) of 0.185, but is closely followed by gradient boosting (bs=0.188), random forest (bs=0.188), adaboost (bs=0.193), and lastly, KNN (bs=0.309). These results indicate that although the xgboost model is superior, several models have similar performance. The high performance suggests machine learning models have the potential to impact patient care in radiology. Interestingly, we identified a number of reports that were consistently predicted incorrectly across all models. In collaboration with a radiologist, we plan to investigate these reports more thoroughly to improve our prediction results.