William Hersh, MD, Professor and Chair, OHSU
Blog: Informatics Professor
Twitter: @williamhersh
Since my own explorations with ChatGPT in informatics knowledge last month, others using the system have racked up some impressive results. Most pertinent to biomedical informatics is ChatGPT’s passing grades on medical student licensing exams. It has also performed well on Wharton MBA and multi-state bar exams.
Does this mean that machine intelligence has equaled human professionals? My subsequent encounters with ChatGPT lead me to maintain my skepticism. Not that ChatGPT is not a tremendous feat, but it is not (yet anyways) capable of performing as a physician, lawyer, or MBA.
This conclusion is evidenced from another recent interaction I had with ChatGPT, the transcript of which is at the bottom of this post. This interaction mirrors some characteristics that many others note. One property that many find is ChatGPT’s tendency to confabulate. As seen in my latest interaction, it states things that are simply false. Not that humans do not do this, but no one would rely on a physician, lawyer, or MBA known to flagrantly lie. As such, ChatGPT fails at one core task of being a competent professional in a given field, where there is an ethical responsibility to recognize and state when you do not know something. The best physician is one who knows the limits of his or her knowledge and skills. (Of course, ChatGPT lacks many other capabilities of a physician, from technical skill to empathy.)
This leads me to a thought experiment, perhaps somewhat biased by the fact I am an information retrieval (IR) researcher. If you were confronted with a highly challenging intellectual task, which would you use first, Google (or another search system, maybe PubMed for medicine) or ChatGPT? In my mind, an IR system is much more appealing, giving what IR systems provide by definition, which is pointers to information that may answer questions. Google is far from a perfect search engine, but when addressing a complicated intellectual task, I would much rather take my chances with an IR system output with primary sources listed and accessible than the narrative without citations that comes out of ChatGPT.
I have no doubt that systems like ChatGPT will continue to advance, and that human knowledge workers will find ways to use them beneficially in their work. But I cannot imagine there not being a human in the loop somewhere, especially in medicine when you might need the skills of a surgeon, not to mention a physician with empathy and grace. I am also reminded of the quote from my colleague, radiologist Curtis Langlotz, probably speaking for all of medicine, who said, “Will AI replace radiologists? is the wrong question. The right answer is: Radiologists who use AI will replace radiologists who don’t.”
Here is the transcript from my most recent interaction with ChatGPT, with my annotations in italics. I would also note, perhaps in a blow to my own ego, that ChatGPT’s answer does not include any of my own IR writings, such as the recent edition of my biomedical IR textbook, which arguably are important to this field.
I entered to Chat GPT, What are some good readings for biomedical information retrieval?
There are many resources available for learning about biomedical information retrieval, including books, articles, and online courses. Here are a few suggestions:
“Introduction to Biomedical Information Retrieval” by C. J. van Rijsbergen is a classic textbook that provides a comprehensive introduction to the field.
Keith van Rijsbergen is a retired computer science professor and IR researcher who has written several books but not one called “Introduction to Biomedical Information Retrieval.”
“Information Retrieval in Biomedicine” by Peter Willett is another comprehensive textbook that covers the basics of biomedical information retrieval as well as advanced topics.
Peter Willett is a retired computer science professor and IR researcher who has written several books but not one called “Information Retrieval in Biomedicine.”
“Medical Information Retrieval” by J. V. McInerney is a book that provides an overview of the field and covers various topics such as search engines, natural language processing, and data mining.
I am not able to find any book titled “Medical Information Retrieval” or anyone in the IR or informatics field named J. V. McInerney.
“Biomedical Information Retrieval” by J. S. Ogasawara and M. R. Al-Rawi is a book that covers various aspects of biomedical information retrieval, including text mining, ontologies, and evaluation.
I am not able to find any book titled “Biomedical Information Retrieval” or anyone in the IR or informatics fields named J. S. Ogasawara or M. R. Al-Rawi.
“Biomedical Text Mining” by Haifeng Li and Linyun Fu is a book that covers the use of text mining techniques for extracting and analyzing information from biomedical literature.
There is a book named “Biomedical Text Mining” but it does have authors Haifeng Li or Linyun Fu, neither of whom work in IR or informatics.
The Journal of Biomedical Informatics is a peer-reviewed journal that publishes research on various aspects of biomedical information retrieval.
This is true!
The International Conference on Medical Information Retrieval is an annual conference that brings together researchers and practitioners in the field to present and discuss their work.
I am not aware of any such conference by this name.
I hope these suggestions are helpful! Let me know if you have any other questions.
I will!
This article post first appeared on The Informatics Professor. Dr. Hersh is a frequent contributing expert to HealthIT Answers.