10-Year-Old CNN Beats Frontier LLMs in Skin Cancer Diagnosis Test

In a surprising turn of events, a decade-old convolutional neural network (CNN) has outperformed the latest frontier large language models (LLMs) in diagnosing skin cancer. The test, conducted by Dr. Magnus Lynch, a consultant dermatologist at Guy's Hospital and Honorary Reader at King's College London, pitted GPT-6 Astra and Claude Fable 5.1 against a fine-tuned Inception v3 model. The results challenge the assumption that bigger, more general AI models are always better for specialized medical tasks.

The Experiment: 200 Biopsy-Proven Lesions

Dr. Lynch selected 200 skin lesions from the MILK10k dataset, a public resource released through the International Skin Imaging Collaboration in 2025. Each lesion had two images: a clinical close-up and a dermoscopic image. All were confirmed by histopathology, making this a particularly challenging test. The set included 100 lesions that should not be discharged (30 melanomas, 25 basal cell carcinomas, 25 squamous cell carcinomas, and 20 actinic keratoses) and 100 that were safe to discharge (50 benign naevi and 50 seborrhoeic keratoses).

Every lesion was biopsied, including the benign ones, meaning the dataset was deliberately harder than typical clinical presentations. As Dr. Lynch notes, "a mole that somebody decided to cut out is one that worried a clinician." This setup ensures the test reflects real-world diagnostic difficulty.

The Models: Old vs. New

The LLMs—GPT-6 Astra and Claude Fable 5.1—were given both images and asked to provide the most likely diagnosis out of six options, a confidence level, a second choice, and a recommendation (discharge, biopsy, or follow-up). Both models received identical prompts at default settings and never refused a case.

In contrast, the CNN was deliberately not state-of-the-art. Dr. Lynch used Inception v3, first published by Google in 2015, and fine-tuned it on 68,011 dermoscopic and clinical images from the ISIC Archive. The fine-tuning took only about 20 minutes on a single H100 GPU. Crucially, the CNN had never seen the 200 test lesions, nor any images of the same lesion or patient, as they were removed from its training data.

The Results: A Clear Winner

The CNN correctly identified the diagnosis in 125 out of 200 cases (62%), while GPT-6 Astra managed 84 (42%) and Claude Fable 5.1 achieved 85 (42%). When allowing a second choice, the CNN reached 173 out of 200, compared to 134 and 138 for the LLMs. The gap was most pronounced on benign lesions: the CNN recognized 37 of 50 moles and 26 of 50 seborrhoeic keratoses, whereas GPT-6 Astra identified only 16 and 13, respectively, and Claude Fable 5.1 managed 33 and 6.

Perhaps most concerning for LLM proponents is the error patterns. GPT-6 Astra called 75 lesions basal cell carcinoma when only 25 were, including 14 of the 30 melanomas. Claude Fable 5.1 favored actinic keratosis, diagnosing 53 lesions as such when only 20 were. The CNN's errors were more distributed, and it named 13 of 30 melanomas correctly, compared to 8 and 11 for the LLMs.

Safety Implications: Discharge Decisions

Beyond raw accuracy, the study examined safety-critical decisions. At its preset threshold, the CNN would have kept 77 of 100 unsafe lesions for treatment, and discharged 76 of 100 safe lesions. However, when thresholds were matched for sensitivity, the CNN could keep 98 or 99 unsafe lesions, but then it would discharge far fewer safe ones (2 or 1 melanomas). The LLMs, by contrast, kept 97 and 96 unsafe lesions but discharged only 32 and 22 safe ones—meaning they were overly cautious, likely leading to unnecessary procedures.

Why It Matters

This study highlights a critical insight: specialized, task-specific models can still outperform general-purpose AI in narrow domains. Dr. Lynch's experiment is not a condemnation of LLMs but a reminder that "frontier" does not always mean "better" for every application. The CNN's success is attributed to its focused training on dermatological images, whereas LLMs are trained on vast, diverse datasets that may not capture the subtle visual cues of skin lesions.

Moreover, the CNN is remarkably lightweight—it runs entirely in a phone browser using WebGPU, with no data upload. This accessibility could make it a practical tool for primary care settings, especially in regions with limited access to dermatologists.

Caveats and Considerations

The study has limitations. With only 200 lesions and 30 melanomas, the statistical power is limited. Each LLM was run once, and answers can vary between runs. Additionally, the MILK10k dataset has been public since 2025, so LLMs could have seen it in training—though that would have helped them, not hindered them. The CNN definitely had not seen the test images.

Dr. Lynch also notes that Inception v3 is a baseline. State-of-the-art skin cancer AI uses ensembles of image transformers trained on more data, and regulated products built on them perform considerably better. Still, the fact that a 10-year-old architecture can beat the latest LLMs is a powerful statement.

A Patient's Perspective

Interestingly, when the LLMs were asked as a patient with a clinical photograph alone and the question "do I need to see a doctor?", they advised seeing a doctor in almost every case—even for harmless lesions. For 99 and 98 of a further 100 biopsy-proven melanomas, they correctly advised a visit. This suggests LLMs may be more useful for triage than for specific diagnosis.

The Bottom Line

Dr. Lynch's experiment, available in full with data and code, serves as a wake-up call. While LLMs are impressive generalists, they are not yet ready to replace specialized diagnostic tools. The 10-year-old CNN, fine-tuned for 20 minutes, is a testament to the enduring value of purpose-built AI. As the field evolves, the lesson is clear: for high-stakes medical tasks, specialized models still hold their ground.