Bhayana and associates first evaluated ChatGPT based on GPT-3.5, the most widely used version, to evaluate its performance on radiology board exam questions and investigate benefits and limits. The higher-order thinking questions were further subdivided into categories (description of imaging findings, clinical care, computation and categorization, and illness links). The model struggled with questions demanding higher-order thinking (60%, 53 of 89), but did reasonably well on questions requiring lower-order thinking (84%, 51 of 61). Given its lack of pretraining in radiology, it was not surprising that it performed poorly on higher-order thinking issues. The results imply that the allegedly superior advanced reasoning abilities of GPT-4 translate to improved performance in a radiology setting.
