westurner · 114 points · 38 comments · 10 godzin temu · Open original
Comments
5 preview comments · loading full thread
Log in to use comments
Log in to h4cker, then connect Hacker News to publish comments.
MAmapontosevenths5 godzin temu
The author describes the model's pairing operation as:
(zt,zt+Delta) where Delta is a positive offset.
This means the model is using the next token to predict the next token. It should have been (zt,zt-Delta), or it's cheating.
I don't know if that's just the paper, or if the code was also b0rk3d.
CHcharcircuit8 godzin temu
Scale is all you need.
That's why you see other attention free large language models work. But in a world with constraints on time and compute the architecture does matter because scaling does have a cost (which is dropping)
GNgnabgib10 godzin temu
(2025)
THthemagyar7 godzin temu
"subconscious" reasoning
CHchiharukiryu2 godziny temu
lmao seems just computing power arms race >>> specific algorithm
Comments
5 preview comments · loading full threadLog in to h4cker, then connect Hacker News to publish comments.
The author describes the model's pairing operation as: (zt,zt+Delta) where Delta is a positive offset. This means the model is using the next token to predict the next token. It should have been (zt,zt-Delta), or it's cheating. I don't know if that's just the paper, or if the code was also b0rk3d.
Scale is all you need. That's why you see other attention free large language models work. But in a world with constraints on time and compute the architecture does matter because scaling does have a cost (which is dropping)
(2025)
"subconscious" reasoning
lmao seems just computing power arms race >>> specific algorithm