westurner · 114 points · 38 comments · vor 10 Stunden · Open original
Comments
5 preview comments · loading full thread
Log in to use comments
Log in to h4cker, then connect Hacker News to publish comments.
MAmapontoseventhsvor 5 Stunden
The author describes the model's pairing operation as:
(zt,zt+Delta) where Delta is a positive offset.
This means the model is using the next token to predict the next token. It should have been (zt,zt-Delta), or it's cheating.
I don't know if that's just the paper, or if the code was also b0rk3d.
CHcharcircuitvor 8 Stunden
Scale is all you need.
That's why you see other attention free large language models work. But in a world with constraints on time and compute the architecture does matter because scaling does have a cost (which is dropping)
GNgnabgibvor 10 Stunden
(2025)
THthemagyarvor 7 Stunden
"subconscious" reasoning
CHchiharukiryuvor 2 Stunden
lmao seems just computing power arms race >>> specific algorithm
Comments
5 preview comments · loading full threadLog in to h4cker, then connect Hacker News to publish comments.
The author describes the model's pairing operation as: (zt,zt+Delta) where Delta is a positive offset. This means the model is using the next token to predict the next token. It should have been (zt,zt-Delta), or it's cheating. I don't know if that's just the paper, or if the code was also b0rk3d.
Scale is all you need. That's why you see other attention free large language models work. But in a world with constraints on time and compute the architecture does matter because scaling does have a cost (which is dropping)
(2025)
"subconscious" reasoning
lmao seems just computing power arms race >>> specific algorithm