A 10-year-old CNN outperforms frontier LLMs for skin cancer diagnosis
sungam · 582 points · 246 comments · vor 18 Stunden · Open original
Comments
5 preview comments · loading full thread
Log in to use comments
Log in to h4cker, then connect Hacker News to publish comments.
SNsnittyvor 18 Stunden
This is an example of why I'm a little stumped about why people are excited about Jev-style decision models. For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture (neural net, bayesian, etc.) LLMs drive the cost of making those models down to near zero.
Just use the LLM to make a thing that's actually good at what you need it to do.
TAtasukivor 16 Stunden
> I compared to a 10-year old CNN and to my surprise found that this was superior.
I'm surprised he's surprised.
> A small, old, specialised network using an outdated architecture recognises skin lesions far better than two frontier LLMs.
Yes of course. A 30m params Katago network with one playout is superhuman in playing go, while frontier LLMs have over 10,000 times more parameters and aren't any good at playing go. They're good at other things (such as helping you create a network that's good at playing go).
UNunknownvor 19 Minuten
[deleted]
LELegend2440vor 17 Stunden
> I used Inception v3 pretrained on everyday photographs and then fine-tuned all layers on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU.
>These 200 test lesions, and every other photograph of the same lesion or patient, were removed from its data (including the validation set) before training, so it had never seen them, and they were not used to choose the model or set its threshold.
This is just benchmaxxing. A benchmaxxed model will always beat an unbenchmaxxed model, but it will be brittle.
You can always beat a specific dataset with a small model by finetuning on a train/test split. But it doesn't generalize well to 'wild' images, because they are out-of-domain.
MIMitPittvor 18 Stunden
And a calculator outperforms LLMs at multiplication and division...
Comments
5 preview comments · loading full threadLog in to h4cker, then connect Hacker News to publish comments.
This is an example of why I'm a little stumped about why people are excited about Jev-style decision models. For any application at scale it'd be way more accurate to have a dedicated classifier using the right architecture (neural net, bayesian, etc.) LLMs drive the cost of making those models down to near zero. Just use the LLM to make a thing that's actually good at what you need it to do.
> I compared to a 10-year old CNN and to my surprise found that this was superior. I'm surprised he's surprised. > A small, old, specialised network using an outdated architecture recognises skin lesions far better than two frontier LLMs. Yes of course. A 30m params Katago network with one playout is superhuman in playing go, while frontier LLMs have over 10,000 times more parameters and aren't any good at playing go. They're good at other things (such as helping you create a network that's good at playing go).
[deleted]
> I used Inception v3 pretrained on everyday photographs and then fine-tuned all layers on 68,011 dermoscopic and clinical images from the ISIC Archive on a single H100 GPU. >These 200 test lesions, and every other photograph of the same lesion or patient, were removed from its data (including the validation set) before training, so it had never seen them, and they were not used to choose the model or set its threshold. This is just benchmaxxing. A benchmaxxed model will always beat an unbenchmaxxed model, but it will be brittle. You can always beat a specific dataset with a small model by finetuning on a train/test split. But it doesn't generalize well to 'wild' images, because they are out-of-domain.
And a calculator outperforms LLMs at multiplication and division...