The NHS is facing many challenges, with rising patient demand for healthcare services, staff shortages, and limited resources putting the healthcare system under strain. This pressure has led to significant consequences for patients, including lengthy waiting lists, delayed diagnoses, and overstretched appointment systems that can leave critical conditions undetected until they become far more serious. One of the under-discussed consequences of the increasing strain on NHS services is the tendency for members of the public to turn to artificial intelligence (AI) chatbots for medical support or advice. This is a concerning phenomenon which has so far been omitted from the discussion surrounding AI and healthcare. However, in the NHS 10-Year Health Plan, the UK Government has committed to digitising triage services and transforming the NHS App, which would include an integration of AI. This presents a promising opportunity not only to improve administrative efficiency, streamline treatment plans and ease the burden on NHS staff, but also to potentially help combat the widespread use of unregulated AI chatbots for diagnostic support.
The use of ‘black-box’ AI Chatbots for health-related queries
The average waiting time for non-urgent GP appointments in 2026 is between one and three weeks. Given the exponential increase of people using AI tools in their daily lives, it is perhaps unsurprising that a large portion of the British public are side-stepping the GP waiting lists and referring to AI Chatbots instead. Recent research has found that almost two in three Brits (59%) are now using AI Chatbot tools to self-diagnose. Open access tools like ChatGPT have seen a significant increase in searches such as “what are the symptoms for […]” and “what are the side effects of […]” since January 2025. While this is a tool predominantly used by 18 to 24-year-olds, with 85% of research participants in this age group regularly using AI for self-diagnosis, this is a cross-generational phenomenon, with 35% of people over 65 doing the same. Despite the fact that a majority of Brits are now using AI Chatbots for medical purposes, this issue has been perplexingly omitted from public discussion.
Open access AI chatbots such as ‘ChatGPT’ are ‘black-box’ algorithms, which is to say that their algorithmic processes are, by nature, unknowable to the user. Their internal workings are entirely opaque; they trade off transparency and explicability to improve accuracy and algorithmic capabilities. This lack of algorithmic transparency is cause for concern, especially when they are being relied upon by patients to replace the primary care capacities of medical practitioners. The EU’s Ethics Guidelines for Trustworthy AI raises ethical and legal concerns with black-box AI algorithms. The guidelines explain that the lack of algorithmic transparency could cause issues for legal accountability if an algorithm were to generate misguided advice which causes harm.
Black-box AI tools are known to make factual mistakes, their algorithms can draw from unreliable sources of information and sometimes even fabricate sources, a phenomenon which has been described as AI ‘hallucinations’. This means there’s potential for ChatGPT to disseminate medical misinformation. Research by The Eve Appeal, a gynaecological cancers charity, investigated the responses which ChatGPT provided to questions regarding gynaecological health. They found that ChatGPT failed to recognise the potential symptoms of vulval and ovarian cancers and instead advised patients to treat them as symptoms of IBS. ChatGPT also gave advice which was not relevant to the UK health system and issued dietary advice which should only be recommended by medical professionals, regarding potentially unsuitable dietary changes. From their survey of 2000 women from the UK, they found that 9% had consulted ChatGPT about gynaecological concerns and, of those women, 1 in 4 were re-assured by the responses and chose not to consult a doctor afterwards. The charity stated their concern with these research results given the unreliability of the medical advice which ChatGPT had provided.
These black-box AI algorithms pose regulatory challenges because of their inherently inexplicable algorithmic processes, and the UK Government’s response to AI involves very limited regulations on tech companies. The involvement of these largely unregulated AI chatbots in healthcare decision-making processes could therefore pose ethical, legal and practical concerns which may not be sufficiently addressed by the existing UK regulatory frameworks.
However, the use of AI for diagnostic support is not without its advantages: it provides openly accessible and equitably distributed medical information at high speed, which could save patients from making unnecessary trips to the GP and therefore ease the burden on NHS staff. There have also been instances where AI tools have aided diagnoses. For example, ChatGPT diagnosed a young Welsh woman with a rare condition which doctors had failed to identify, leading to her receiving the treatment which she needed. The increased access to medical information which AI provides can help individuals better understand their health conditions, lead to them receiving a diagnosis, and help to recommend treatment plans. AI may also have the advantage of avoiding issues of human error or bias in medical decision making. For example, the medical testimony of women from ethnic minorities is often ignored or undermined, and consulting AI may allow patients to sidestep this discriminatory practise. Therefore, while we ought to caution against deferring advisory authority to unregulated black-box AI algorithms AI rather than experienced medical professionals, AI does have the potential to improve individual health experiences by providing medical information in a fair, accessible, and timely manner.
The NHS 10-year plan
In the third chapter of the NHS 10-Year Health Plan titled ‘From Analogue to Digital- Power in your Hands’ the UK Government committed to creating a digitally accessible health system and transforming the NHS App. This transformation will involve the creation of a virtual assistant or ‘a doctor in your pocket’ for 24/7 advice and guidance.
They hope to link each patient’s medical records and data to the NHS App, so that diagnostic care can be personalised to each patient’s individual needs. Liz Clow and Rachel Hope from NHS England explained exactly how they hope the transformed NHS app will function: AI-assisted triage tools would allow patients to ask health-related queries and receive answers based on NHS advice and their personal health records. Patients would then be prompted to either book appointments or seek further treatment based on this advice, which they could also do through AI-assisted administrative tools on the App. They said the NHS App would become a “lifelong companion” for patients, helping them better understand their health conditions and providing better access to early diagnosis and early interventions, while alleviating the strain on the NHS through reducing waiting lists, missed appointments, and referral times. They are hoping for full app integration by 2027, although this is not a concrete policy timeline.
‘Doctor in your pocket’: The future of primary care?
The ‘doctor in your pocket’ NHS AI tool seems to be a highly promising development for the future of primary care, because it would embed and institutionalise many of the benefits of diagnostic AI. The ‘chatbot’ feature in the NHS App would hopefully have the ease, speed, and accessibility of black-box AI tools, meaning patients could avoid GP waiting lists and ask non-urgent medical questions online, helping to shorten waiting lists and reduce unnecessary appointments. Patients could ask medical queries without fear of judgement or discrimination, and medical information and counsel would be accessible from anywhere, erasing travel or accessibility concerns from non-urgent primary care queries. Furthermore, an official and government regulated AI Chatbot would avoid the legal, ethical, and practical concerns surrounding black-box AI related to the dissemination of misinformation, of the lack of algorithmic transparency and the lack of regulation. The NHS chatbot could personalise its responses based on patient medical records and standard NHS advice. Therefore, it seems that a ‘doctor in your pocket’ AI chatbot could mitigate the harms which arise from unregulated black-box AI consultation, while institutionalising the benefits of accessible AI diagnosis support.
However, the success of a ‘doctor in your pocket’ will depend on the extent to which it is transparent, accountable, accurate, and trusted by patients. AI systems risk becoming a barrier between patients and healthcare professionals and undermining the very trust on which effective healthcare depends. Developers must also remain attentive to the risk of digital exclusion. While AI-assisted triage could improve access to healthcare for many patients, not everyone has equal access to smartphones, reliable internet connections, or the digital literacy required to navigate increasingly complex online healthcare systems. Elderly patients, individuals with disabilities, and those from disadvantaged socio-economic backgrounds may be disproportionately affected if digital services begin to replace rather than complement traditional routes to care. If the NHS is committed to reducing health inequalities, AI-assisted triage must remain one option among many, rather than becoming the default gateway to primary care.
The development of the NHS’s AI-assisted triage tool should therefore be accompanied by robust regulatory safeguards, independent oversight mechanisms, and ongoing public scrutiny. The coming years will reveal whether the NHS can successfully harness the benefits of AI while preserving the principles of safety, accountability, and equitable access that underpin public healthcare.
Related content
