westurner · 114 points · 38 comments · 10 ore fa · Open original
Comments
5 preview comments · loading full thread
Log in to use comments
Log in to h4cker, then connect Hacker News to publish comments.
MAmapontosevenths5 ore fa
The author describes the model's pairing operation as:
(zt,zt+Delta) where Delta is a positive offset.
This means the model is using the next token to predict the next token. It should have been (zt,zt-Delta), or it's cheating.
I don't know if that's just the paper, or if the code was also b0rk3d.
CHcharcircuit8 ore fa
Scale is all you need.
That's why you see other attention free large language models work. But in a world with constraints on time and compute the architecture does matter because scaling does have a cost (which is dropping)
GNgnabgib10 ore fa
(2025)
THthemagyar7 ore fa
"subconscious" reasoning
CHchiharukiryu2 ore fa
lmao seems just computing power arms race >>> specific algorithm
Comments
5 preview comments · loading full threadLog in to h4cker, then connect Hacker News to publish comments.
The author describes the model's pairing operation as: (zt,zt+Delta) where Delta is a positive offset. This means the model is using the next token to predict the next token. It should have been (zt,zt-Delta), or it's cheating. I don't know if that's just the paper, or if the code was also b0rk3d.
Scale is all you need. That's why you see other attention free large language models work. But in a world with constraints on time and compute the architecture does matter because scaling does have a cost (which is dropping)
(2025)
"subconscious" reasoning
lmao seems just computing power arms race >>> specific algorithm