Performance of ChatGPT as an AI-assisted decision support tool in medicine: a proof-of-concept study for interpreting symptoms and management of common cardiac conditions (AMSTELHEART-2)

Figures & data

Figure 1. Interest over time for ChatGPT based on Google trends.

Numbers represent search interest relative to the highest point on the chart for the given region and time. A value of 100 is the peak popularity for the term. A value of 50 means that the term is half as popular. A score of 0 means there was not enough data for this term. (https://trends.google.com/trends/explore?date=today%203-m&q=chatgpt) (search date: Feb 4, 2023)

Figure 2. Performance of ChatGPT(3.5 version September versus January) in correctly answering medical quizzes on key aspects and practice essentials of common cardiovascular conditions (n = 50).

Abbreviations: CAD: coronary artery disease (acute coronary syndrome and coronary artery atherosclerosis management); PE/VTE: pulmonary embolism and venous thromboembolism; AF: atrial fibrillation; HF: heart failure; CVRM: cardiovascular risk factors and management.

Table 1. ChatGPT recommendations based on medical questions from patients and accuracy when compared with actual clinical course by a healthcare provider.

Case	Context	ChatGPT	Clinical course/diagnosis	Accuracy
Female, 25-30yrs	Palpitations, not related to exercise, trouble sleeping, anxiety. Prior unexplained syncope Question: what is the diagnosis?	Causes may include stress, caffeine, hormonal changes, medications, heart condition. Please consult a physician.	Palpitations due to panic attacks, stress and lack of sleep.	Adequate advice was provided by ChatGPT (both versions)
Male, 65–70yrs	Symptoms of exertion related chest pain and dizziness since 2 months. Relevant history: prior PCI Question: what is the appropriate course of action?	Symptoms may indicate a heart condition. Since PCI in the past, it is important to consult a cardiologist for further evaluation.	Angina pectoris, class 3. PCI was performed	Adequate advice was provided by ChatGPT (both versions)
Male, 65–70yrs	Calls the practice with symptoms of acute-onset of chest pain and dyspnoea. Question: how would you triage this patient and what is the appropriate course of action?	1. ABCD check 2. Detailed history 3. Physical examination, Based on this assessment; either immediate hospitalisation or non-urgent evaluation should follow.	Clinical course: 1. ABCD check 2. standardised triage questions 3. Ambulance activation Final diagnosis: NSTEMI	ChatGPT provided advice for an in-person evaluation; otherwise adequate advice (both versions)
Female, 70–75yrs	Calls the practice. She has heart failure and noted that she gained weight (2 kg in 2 weeks). Normally she takes furosemide 80mgs. Question: should I increase the dose and see what happens?	It is not recommended to increase furosemide without first consulting with your healthcare provider. It may be a sign of fluid retention, which can be a serious issue in heart failure.	The triage assistant booked an appointment in the clinic. The GP confirmed she had fluid retention due to decompensatio cordis, and adjusted medications, including uptitration of furosemide.	ChatGPT provided adequate advice (both versions)
Female, 20–25yrs	I feel like my heart is flip-flopping, what is the matter?	The sensation is known as palpitations. Causes are: stress, anxiety, panic attacks, hormonal changes, certain medications, arrhythmias. Consult your healthcare provider.	Final diagnosis: Palpitations due to party drugs	ChatGPT provided adequate advice (both versions)
Male, 70–75yrs	I have occasional episodes of atrial fibrillation, and I’m 75 years old. The attending cardiologist advised me to take blood thinners. Should I listen to this advice and take these medications, despite the fact that they increase bleeding risk?	Explanation about factors affecting the balance of stroke and bleeding risk of anticoagulants in AF. Advice to follow instructions of healthcare provider carefully and to have regular check-ups	The patient asked this question to her GP, who explained risks and benefits of anticoagulants and made sure the patient had regular follow-up	ChatGPT provided adequate advice (both versions)
Female, 80–85yrs	Calling to practice. I recently had a heart attack, my doctor prescribed five drugs that I should take. Can you tell me why I should take them?	Explanation about working mechanism of aspirin, P2Y12-inhibitor, beta-blockers, ACE inhibitors, and statins	Explanation about aspirin, beta-blockers, ACE-inhibitors, statins and P2Y12-inhibitors	1. ChatGPT 3.5-v.Sept provided correct information. 2. ChatGPT 3.5 v.Jan provided partially correct information, but also suggested the use of thrombolytic therapy.
Male, 65-70yrs	What can I do to prevent getting another heart attack?	Detailed explanation on: adherence to medications, healthy diet, regular exercise, smoking cessation, maintain a health weight, limit alcohol intake, manage stress, control other risk factors (DM, HTN, cholesterol), regular health check-ups, respond quickly when experiencing signs of a heart attack	Same explanation was provided by the GP	ChatGPT provided correct advice (both versions)
Male, 20–25yrs	Had a corona/Sars-CoV-2 vaccination, now feverish, chest pain that gets worse when bending over. What could this be? Should I seek medical attention?	Not possible to diagnose the cause. Better to be safe and seek medical attention.	Pericarditis	Adequate advice (both versions)
Female, 60–65yrs	I have had a heart attack two months ago. I feel good. Is it safe to have sex again?	In general, it is safe, as long as feeling well and having received the green light from their doctor.	The doctors gave the green light	Adequate advice (both versions)

Abbreviations: PCI– percutaneous coronary intervention, NSTEMI– non-ST elevation myocardial infarction.

Table 2. ChatGPT recommendations based on medical questions from physicians and accuracy when compared with actual clinical course by collegial consultation.

Case	Context	ChatGPT	Clinical Course	Accuracy
Male, 75-80yrs	HFrEF due to IHD. NYHA II. What other medications could I prescribe to provide optimal medical care. He is taking beta-blockers, diuretics, ACE-inhibitor, mineral corticoid receptor antagonists, statins	- ARNI - SGLT2-inhibitors - ivabradine - hydralazine with nitrates - digoxin (including a short explanation of working  mechanism, impact on symptoms and outcomes)	- ARNI was started  - SGLT2 inhibitor was started (NYHA II)  - ivabradine was not started (HF < 75/min) [medications were started by the cardiologist]	The advice is correct (version September) For version January; it was partially correct. The use of SGLT2 inhibitor was not mentioned. **
Female, 70-75yrs	Diagnosed with AF, wants to initiate an OAC. Asks for pros and cons of DOAC versus VKA	DOAC: + predictable anticoagulant effect, routine monitoring not required + fewer drug and food interactions + lower risk of bleeding - more expensive - limited reversal agents available in case of emergency bleeding - less clinical experience when compared with VKA	Mostly same recommendations by the consulting cardiologists. Additional question to make sure it is non-valvular AF. Also a reversal agent is available in emergency departments.	The advice is correct (both versions)
Female, 20-25yrs	I hear a low pitched third heart sound. Does this require follow-up? My patient is pregnant (third trimester) and she is healthy, her blood pressure is fine.	This heart sound is known as a “S3 gallop” and may be a normal finding, particularly in the third trimester, in the absence of symptoms and a normal blood pressure and foetal heart rate	It was deemed physiological due to pregnancy. Watchful weighting was observed. The heart murmur disappeared after pregnancy	The advice is correct (both versions)
Female, 75-80yrs	Can you give me the annual stroke rate for a patient with atrial fibrillation, who is female, 75 years of age, with diabetes, hypertension and vascular disease?	The annual stroke rate for a CHA2DS2-VASc score of 6 is about 9% per year. According to the guidelines, a score of 6 or higher indicates a high risk of stroke and would indicate the need for oral anticoagulation therapy.	Identical estimated stroke risk using the same score	The answer is correct (both versions)
Male, 80-85yrs	Should I use NT-proBNP to monitor chronic heart failure or rely on symptoms and weight gain only?	NT-proBNP is widely used for monitoring heart failure. It is especially useful in patients with chronic heart failure, to assess severity of the disease, determine response to treatment and to predict future outcomes. Relying on symptoms and weight gain alone may not sufficient.	In the Dutch general practice guidelines, monitoring with natriuretic peptides is not specifically mentioned. In ESC guidelines it is not recommended for routine monitoring; but only in specific settings when it’s unclear whether symptoms are due to HF or another condition (for instance chronic venous insufficiency)	The answer is incorrect for both January and September versions
Male, 70–75yrs	My patient has atrial fibrillation and a PCI about 5 years ago. He has been asymptomatic for years. He is taking rivaroxaban and ascal. I don’t think ascal is indicated, can I discontinue ascal and continue with rivaroxaban only?	Advice: consult a physician. The use of aspirin and rivaroxaban is complex. There may be situations where the treating physician might consider discontinuing aspirin and relying solely on an oral anticoagulant.	Rivaroxaban was continued and ascal was stopped.	The advice was inconclusive. (both January and September versions)
Female, 30–35yrs	Which medication is safest to use during pregnancy to treat (gestational) hypertension (150/100 mmHg)?	Methyldopa is one of the safest. Alternatives are also suggestedL Version September: labetalol, nifedipine, hydralazine, doxazosin. Version January: calcium channel blockers and ACE inhibitors	Methyldopa was used, and later switched to labetalol due to side effects (sleepiness, dry mouth) An alternative is a calcium channel blocker	Partially correct; while methyldopas is correct, the other options are not all considered safe. (both versions) Guidelines recommend against the use of RAAS inhibitor drugs or doxazosin during pregnancy and lactation. [20]
Female, 65–70yrs	My patient suffers from statin-mediated myalgia. I have tried rosuvastatin, atorvastatin, simvastatin. What options are left?	Pravastatin, fibric acid derivatives, niacin, bile acide sequstrants	Pravastatin was chosen (fluvastatin or alternate day dosing were also suggested)	Correct in both versions
Male, 80-85yrs	My patient is taking spironolactone and developed bilateral gynaecomastia, is there an alternative drug available without this side effect?	Version Sept: eplerenone, amiloride Version January: Thiazide diuretics, ACEi, ARBs, beta-blockers	Eplerenone, selective MRA with less anti-androgen effect.	Version September: correct Version January: incorrect
Male, 40–45yrs	An ECG was performed as part of a work-related health check-up. The ECG shows a normal sinus rhythm and no other abnormalities, except for two PVCs. The patient is healthy, active, normal blood pressure, glucose and cholesterol. Do these ECG findings have any clinical consequence or is this a physiological finding?	PVCs are common and are considered benign in many healthy individuals (depending on frequency and pattern), especially in the absence of other symptoms or ECG abnormalities.	PVCs in this patient should be considered a physiological finding	Correct in both versions

Abbreviations: HFrEF: heart failure with reduced left ventricular ejection fraction, IHD: ischaemic heart disease; NYHA = classification for severity of dyspnoea in heart failure (I-IV, IV is worst); ACEi: angiotensin converting enzyme inhibitor; ARB: angiotensin receptor blocker; ARNI: angiotensin recetor-neprilysin inhibitor; OAC: oral anticoagulant; DOAC: direct oral anticoagulant; VKA: vitamin K antagonist; SGLT2 : sodium/glucose cotransporter 2 inhibitor; MRA: mineralocorticoid receptor (aldosterone receptor) antagonist; ECG: electrocardiogram; PVC: premature ventricular complex.

** A combination of drug types is recommended in the routine management of patients with HF. For patients with HF with reduced ejection fraction, a combination of ACEI/ARB/ARNI, beta-blocker, mineralocorticoid antagonist, and an SGLT2 inhibitor is recommended by the latest guidelines. For patients with HF with preserved ejection fraction, an SGLT2 inhibitor is indicated, and consideration should be given for ARNI and a mineralocorticoid antagonist. Diuretic therapy is also often needed to manage excess volume that can accumulate in patients.