A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis
sungam · 596 points · 251 comments · il y a 19 heures · Open original
Comments
5 preview comments · loading full thread
Log in to use comments
Log in to h4cker, then connect Hacker News to publish comments.
SNsnittyil y a 19 heures
This is an example of why I'm a little stumped about why people are excited about Jev-style decision models. For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture (neural net, bayesian, etc.) LLMs drive the cost of making those models down to near zero.
Just use the LLM to make a thing that's actually good at what you need it to do.
SPspeereril y a 14 minutes
The part where the LLMs (actually, VLMs) outperformed the CNN was that the LLMs were better at avoiding false negatives (see the quote below). That kind of Type 2 error is the most dangerous one since it persuades people not to get advice. Since the LLMs are designed to be general-purpose and public-facing, the headline could be flipped to: "A specialised 10-year old CNN is a worse public facing tool than a general frontier LLM". Which would surprise nobody.
| GPT |Claude | CNN |
Unsafe lesions kept (of 100) | 97 | 96 | 77 at its preset [sic] threshold; 98 or 99 when matched |
TAtasukiil y a 17 heures
> I compared to a 10-year old CNN and to my surprise found that this was superior.
I'm surprised he's surprised.
> A small, old, specialised network using an outdated architecture recognises skin lesions far better than two frontier LLMs.
Yes of course. A 30m params Katago network with one playout is superhuman in playing go, while frontier LLMs have over 10,000 times more parameters and aren't any good at playing go. They're good at other things (such as helping you create a network that's good at playing go).
LELegend2440il y a 19 heures
> I used Inception v3 pretrained on everyday photographs and then fine-tuned all layers on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU.
>These 200 test lesions, and every other photograph of the same lesion or patient, were removed from its data (including the validation set) before training, so it had never seen them, and they were not used to choose the model or set its threshold.
This is just benchmaxxing. A benchmaxxed model will always beat an unbenchmaxxed model, but it will be brittle.
You can always beat a specific dataset with a small model by finetuning on a train/test split. But it doesn't generalize well to 'wild' images, because they are out-of-domain.
MIMitPittil y a 19 heures
And a calculator outperforms LLMs at multiplication and division...
Comments
5 preview comments · loading full threadLog in to h4cker, then connect Hacker News to publish comments.
This is an example of why I'm a little stumped about why people are excited about Jev-style decision models. For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture (neural net, bayesian, etc.) LLMs drive the cost of making those models down to near zero. Just use the LLM to make a thing that's actually good at what you need it to do.
The part where the LLMs (actually, VLMs) outperformed the CNN was that the LLMs were better at avoiding false negatives (see the quote below). That kind of Type 2 error is the most dangerous one since it persuades people not to get advice. Since the LLMs are designed to be general-purpose and public-facing, the headline could be flipped to: "A specialised 10-year old CNN is a worse public facing tool than a general frontier LLM". Which would surprise nobody. | GPT |Claude | CNN | Unsafe lesions kept (of 100) | 97 | 96 | 77 at its preset [sic] threshold; 98 or 99 when matched |
> I compared to a 10-year old CNN and to my surprise found that this was superior. I'm surprised he's surprised. > A small, old, specialised network using an outdated architecture recognises skin lesions far better than two frontier LLMs. Yes of course. A 30m params Katago network with one playout is superhuman in playing go, while frontier LLMs have over 10,000 times more parameters and aren't any good at playing go. They're good at other things (such as helping you create a network that's good at playing go).
> I used Inception v3 pretrained on everyday photographs and then fine-tuned all layers on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU. >These 200 test lesions, and every other photograph of the same lesion or patient, were removed from its data (including the validation set) before training, so it had never seen them, and they were not used to choose the model or set its threshold. This is just benchmaxxing. A benchmaxxed model will always beat an unbenchmaxxed model, but it will be brittle. You can always beat a specific dataset with a small model by finetuning on a train/test split. But it doesn't generalize well to 'wild' images, because they are out-of-domain.
And a calculator outperforms LLMs at multiplication and division...