Testing AI to see whether it can truly replace a doctor.
For many years now, I often hear patients coming in after they have ‘googled’ their symptoms. However, there’s a new boss in town and that’s AI. It wouldn’t be uncommon for at least once a week to hear a patient say “I asked ChatGPT my symptoms”. Some even refer to the platform as Chat GP. Of course, having the power of information as a patient is incredible. But how accurate is it and will it take over the role of the traditional family doctor? After all, patients do still come to see the doctor even after their online virtual consult.
To put this to the test, I act as a patient having 5 frequent ailments and present the symptoms to ChatGPT to see how far I can get and if it can diagnose the issue.
My Tummy Hurts
A common symptom in the land of general practice. A sore stomach and perhaps some diarrhoea and vomiting. To be as accurate as possible, I try to start the conversation with ChatGPT with common sentences without giving too much away (as a patient would do in practice). Naturally, a doctor would prompt the patient and ask further questions to get a diagnosis.
I was led down a windy road of multiple different possible diagnoses which felt very overwhelming as a layman after a simple prompt of my symptoms. I felt that it was my duty to give it more of my history rather than being prompted like a real life doctor would. The long paragraphs were quite off-putting, however, my diagnosis was eventually nestled in the paragraphs.
What was slightly concerning, was the lack of a medical disclaimer advising me that it wasn’t an actual doctor and that I should always seek professional medical help. I would have thought this should be mandatory in these situations.
However, after a few more prompts from my end, it did end up with my diagnosis. It also gave me some ideas on how to recover and it did also say to see my doctor if symptoms persist — which was a plus from me, I only wish it had said that earlier.
I have a sore throat
I decided to give ChatGPT a try at a common presentation of having a sore throat. After minimal input of my symptoms — again, it came up with a plethora of possible diagnoses, one of which was a peritonsilar abscess — more uncommon but serious.
Whilst I think it’s good to give many possibilities, I also think that for the average layman it might be difficult to understand differences. It also gave common diagnoses, then uncommon then common again, which was very confusing. This could pose to be an issue when a patient comes into the doctor and is convinced they have a certain medical condition when they do not. Leading to a potential breakdown of trust between the doctor/patient and both parties may be wary of one another affecting rapport.
After a few more trigger symptoms, it did come up with my diagnosis, however, in a real life situation, tonsillitis could only be accurately diagnosed with both a history and examination, where typically the throat would appear red, inflamed and pus coming from the tonsils. This is an area where ChatGPT lacks in.
Painful Urination
At this point, I felt a little overwhelmed and deflated by all the endless possible diagnoses that it came up with. As a layman, this felt all a little too much. So I wanted to ask ChatGPT to talk to me like a ‘really good doctor’ hoping that it would calm down a little on its long lists and endless diagnoses.
I presented it with trouble passing water, though it did promise me to act like a really good doctor, it quickly defaulted back into its long lists and paragraphs.
Whilst is did get my diagnosis of a urinary infection, it also came up with a number of other diagnoses. This can be very confusing for a potential patient, so I decided to ask it, what I can specifically do to differentiate. It did eventually ask me to check my urine with a urine dip stick and whilst these can be commonly found in a chemist, this may sometimes be interpreted incorrectly by a patient. I would have framed this better to the patient and told them it would be a guide however they should really have an acutal assessment.
The difficulty with diagnosing yourself, is that you are unable to emotionally detach yourself from a situation. When a patient performs a urine dip for example, they may not interpret the colours correctly, thus causing anxiety and panic. They can then jump to the worst case scenario of their diagnosis. Whereas, when a doctor examines the patient, they are able to be unbias with the diagnosis but not only that have many years of experience intepreting such results.
I would say in this situation, ChatGPT would be used more so as a guide that anything more diagnostic.
I Need Strong Medication
I wanted to challenge the chatbot, because if there’s a point in time, where patients will eventually use this platform for medical help, this would be a very important scenario to try.
I told ChatGPT that I had back pain and I simply wanted the strongest pain relief I could get. Imaging that this would be a real life scenario. ChatGPT didn’t ask me questions about my back pain, nor did it direct me for further investigations or follow up with my actual doctor. Instead it directed me to take some simple pain relief. However when I challenged it further, asking for stronger pain relief, it caved in pretty quickly.
It told me I could try muscle relaxants, morphine (!) and opiates. If the algorithm had the ability to send me a prescription of these medications, I’d say it was pretty simple to get some strong pain killers with little input. I found this shocking. It also advised me to get steroid injections into my back — without actually knowing what was wrong. ChatGPT failed so badly at this scenario that it actually frightened me. If we are to trust this in future, we may end up with a worse opiate crisis than we already have and perhaps even the wrong managment plans.
I found this scenario troubling and as I reflect on ‘real life’ patients, this is exactly what we wouldn’t do. As a doctor, even if the patient reports that they’ve had their pain before and they just need pain relief, it’s our duty to do no harm. Jumping straight into pain relief is a bad idea. Any pain needs to be investigated, if it hasn’t already. There needs to be a clear management plan, that doesn’t always necessarily involve pain relief. However, this can only be assessed with a history and examination. Would I trust ChatGPT in this scenario? It’s a hard pass for me.
Sore Ear
My last scenario for ChatGPT was to see if it could ‘examine’ me. I presented it with an symptoms of an ear infection only giving it a basic history prompt.
It successfully came up with my diagnosis within a few other options, however, I wanted it to be more specific. I asked it if there was any way I could check for myself, to which it advised me I could — using a mirror and light.
Frankly, this is simply not good enough. Ear canals are very hard to examine using a torch. Whilst I think it is great to empower patients with their health, I think this was irresponsible. If a patient did have a look inside their ear, what are they looking for? Is it healthy? It takes a number of years and experience to understand what’s normal and what’s abnormal, so asking a patient to make this decision is unreasonable in my opinion.
I took this one further and asked ChatGPT to analyse a picture of ‘my ear’ . This was actually a picture that I took from a reputable dermatology website. I thought that this would make diagnosis much easier, because surely it would just use its algorithm to search the internet for my photo and tell me the diagnosis — which would make logical sense.
However, I was wrong. In fact this is when things went from bad to worse. Whilst it was very empathetic with me with my symptoms (which admittedly it had been through the whole of the scenarios), it failed miserably at photo interpretation. It came up with new and totally different diagnoses not relating to an ear infection. When I asked which one it was more likely it started to tell me that it was one diagnosis over another because I had symmetrical symptoms. I actually didn’t disclose any of this information, nor could it ‘see’ me. However, it took it upon itself to make up some information.
In my opinion this is a fatal move. There’s being unsure and then there’s making things up. It could have easily used this navigation point to say, ‘I’m not sure, see your doctor’ however, it didn’t.
Conclusion
Throughout my whole interaction with ChatGPT I felt overwhelmed with the endless questions and paragraphs and the little opportunity I had to input my own answers.
It failed terribly at certain scenarios and was dangerous when it came to pain relief.
However, it is very new and of course like anything, it will learn. I was impressed by the diagnoses and the symptoms that may be associated. I do think that with a little tinkering it could work a lot better. However, the ultimate method to get an accurate diagnosis is to see your own healthcare professional who will take the time to ask you questions, give you an opportunity to answer back, examine you and arrange for appropriate investigations.
I’m not yet scared that doctors will be obsolete because nothing beats a solid patient-doctor relationship, however, there is certainly room to add AI into the future of medicine.
What do you think, would you trust ChatGPT or AI with your health?
I hope that you have found this article helpful.
Take care and stay healthy,
Dr Nora x
Please note that the contents of this article are not intended to be a substitute for professional medical advice and should not be relied on as health or personal advice. Always seek the guidance of your doctor or other qualified health professional with any questions you may have regarding your health or medical condition.
Links to products often include an affiliate tracking code which allow us to earn fees on purchases you make through them.
