westurner · 114 points · 38 comments · há 10 horas · Open original
Comments
5 preview comments · loading full thread
Log in to use comments
Log in to h4cker, then connect Hacker News to publish comments.
MAmapontoseventhshá 5 horas
The author describes the model's pairing operation as:
(zt,zt+Delta) where Delta is a positive offset.
This means the model is using the next token to predict the next token. It should have been (zt,zt-Delta), or it's cheating.
I don't know if that's just the paper, or if the code was also b0rk3d.
CHcharcircuithá 8 horas
Scale is all you need.
That's why you see other attention free large language models work. But in a world with constraints on time and compute the architecture does matter because scaling does have a cost (which is dropping)
GNgnabgibhá 10 horas
(2025)
THthemagyarhá 7 horas
"subconscious" reasoning
CHchiharukiryuhá 2 horas
lmao seems just computing power arms race >>> specific algorithm
Comments
5 preview comments · loading full threadLog in to h4cker, then connect Hacker News to publish comments.
The author describes the model's pairing operation as: (zt,zt+Delta) where Delta is a positive offset. This means the model is using the next token to predict the next token. It should have been (zt,zt-Delta), or it's cheating. I don't know if that's just the paper, or if the code was also b0rk3d.
Scale is all you need. That's why you see other attention free large language models work. But in a world with constraints on time and compute the architecture does matter because scaling does have a cost (which is dropping)
(2025)
"subconscious" reasoning
lmao seems just computing power arms race >>> specific algorithm