The author describes the model's pairing operation as:
(zt,zt+Delta) where Delta is a positive offset.
This means the model is using the next token to predict the next token. It should have been (zt,zt-Delta), or it's cheating.
I don't know if that's just the paper, or if the code was also b0rk3d.
CHcharcircuit8小时前
Scale is all you need.
That's why you see other attention free large language models work. But in a world with constraints on time and compute the architecture does matter because scaling does have a cost (which is dropping)
GNgnabgib10小时前
(2025)
THthemagyar7小时前
"subconscious" reasoning
CHchiharukiryu2小时前
lmao seems just computing power arms race >>> specific algorithm
评论
5 条预览评论 · 正在加载完整讨论请先登录 h4cker 账号，然后连接 Hacker News 后发表评论。
The author describes the model's pairing operation as: (zt,zt+Delta) where Delta is a positive offset. This means the model is using the next token to predict the next token. It should have been (zt,zt-Delta), or it's cheating. I don't know if that's just the paper, or if the code was also b0rk3d.
Scale is all you need. That's why you see other attention free large language models work. But in a world with constraints on time and compute the architecture does matter because scaling does have a cost (which is dropping)
(2025)
"subconscious" reasoning
lmao seems just computing power arms race >>> specific algorithm