We've been testing and developing a few openai based applications. Relatively simple stuff. I topped up my api credit with 10$ a few months ago. I just checked, we burned something like 61 cents over this period. About 126 requests over the last month. That's just us testing but it shows me that this stuff is relatively cheap. The reason we don't have a lot of requests is that most of this stuff pretty much works on the first try. This is not hard to build.
I think AI is an underused capability in applications. People come up with all these big plans to build agents. But even a simple button to "calculate a color palette that takes into account these symbolic names the user entered" is a simple prompt that comes back quickly and doesn't require a lot of reasoning. We use that with Luna and we give it some more context.
Image based stuff is more expensive but still quite reasonable. We do some things with Sol that work on images. It's not fast but the result is worth waiting for. These are features that save the user time and feel a bit magical. They just click a button, wait 20-30 seconds and then magic happens. Beats them clicking around for half an hour.
Local model hosting would force me into higher cost and overhead for less intelligence. That's only a good tradeoff if you are really going to use lots of tokens or you need this for other reasons (e.g. data security). But otherwise you just end up paying for underutilized GPU capacity.
NINitpickLawyer14小时前
> But I also think the demand for "fast/cheap/good-enough" models is just about to take off.
There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was before "thinking" models, and yet using that library I was able to "guide" the model in the required "prompt / instruct" context such that it was working towards completion, and I saw the first things like we see now in the thinking traces "oh, test x doesn't pass because blah, I need to..." and so on.
Anyway, the revelation was "even if the models never improve, I'll have years of fun finding out all the ways I can use these things". And, obviously, the models improved a lot since then. But I think that revelation can still be applied, as a sort of "truism". We have, right now, access to things that 10-20 years ago would be considered magic. We are still finding ways of cobbling together systems with glue, duct tape and prayers and find new things they can do.
I think the "good-enough" stage has come not just for API models (cheap, fast, etc) but for local as well. Even if slower, even if clunkier, but they are good enough for a set of ever increasing tasks, and what's more it's incredibly fun to work with them.
ANandsoitis5小时前
> One thing a few investors I've talked with have mentioned: "It's weird we're not seeing more consumer AI companies. Why is that?"
What would consumer AI company even be? The frontier labs have declared they will eat everything and they have a head start.
Best bet would to be a contrarian and build products and services that people actually want or need. Fine to be AI powered or augmented, but consumer companies do the hard part of understanding specific consumer needs and wants and pursuing that.
NINickNaraghi14小时前
> Across his various startups, Peter has seen two kinds of work:
> 1. the "IQ 180" work. some mad scientist genius type comes up with some crazy solution you've never thought of.
> 2. the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts.
Interesting comp to pg's Maker's Schedule, Manager's Schedule https://www.paulgraham.com/makersschedule.html
I'm curious about not only which of these roles models will fill, but also how they will empower us to be in the mode we prefer.
MImichael0church14小时前
It makes sense that we’ll see “room at the bottom” strategies. Currently, large parameter counts seem to be slush funds of world knowledge, language skills (because language’s nuances and open vocabulary make it high-dimensional), and reasoning primitives, the general belief being that the latter takes up the least space in the model.
There are many applications where world knowledge is unnecessary or even a negative, and in which only a small amount of language skill is necessary, and there we can expect small models more intelligently used to beat large ones naively used.
评论
5 条预览评论 · 正在加载完整讨论请先登录 h4cker 账号，然后连接 Hacker News 后发表评论。
We've been testing and developing a few openai based applications. Relatively simple stuff. I topped up my api credit with 10$ a few months ago. I just checked, we burned something like 61 cents over this period. About 126 requests over the last month. That's just us testing but it shows me that this stuff is relatively cheap. The reason we don't have a lot of requests is that most of this stuff pretty much works on the first try. This is not hard to build. I think AI is an underused capability in applications. People come up with all these big plans to build agents. But even a simple button to "calculate a color palette that takes into account these symbolic names the user entered" is a simple prompt that comes back quickly and doesn't require a lot of reasoning. We use that with Luna and we give it some more context. Image based stuff is more expensive but still quite reasonable. We do some things with Sol that work on images. It's not fast but the result is worth waiting for. These are features that save the user time and feel a bit magical. They just click a button, wait 20-30 seconds and then magic happens. Beats them clicking around for half an hour. Local model hosting would force me into higher cost and overhead for less intelligence. That's only a good tradeoff if you are really going to use lots of tokens or you need this for other reasons (e.g. data security). But otherwise you just end up paying for underutilized GPU capacity.
> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was before "thinking" models, and yet using that library I was able to "guide" the model in the required "prompt / instruct" context such that it was working towards completion, and I saw the first things like we see now in the thinking traces "oh, test x doesn't pass because blah, I need to..." and so on. Anyway, the revelation was "even if the models never improve, I'll have years of fun finding out all the ways I can use these things". And, obviously, the models improved a lot since then. But I think that revelation can still be applied, as a sort of "truism". We have, right now, access to things that 10-20 years ago would be considered magic. We are still finding ways of cobbling together systems with glue, duct tape and prayers and find new things they can do. I think the "good-enough" stage has come not just for API models (cheap, fast, etc) but for local as well. Even if slower, even if clunkier, but they are good enough for a set of ever increasing tasks, and what's more it's incredibly fun to work with them.
> One thing a few investors I've talked with have mentioned: "It's weird we're not seeing more consumer AI companies. Why is that?" What would consumer AI company even be? The frontier labs have declared they will eat everything and they have a head start. Best bet would to be a contrarian and build products and services that people actually want or need. Fine to be AI powered or augmented, but consumer companies do the hard part of understanding specific consumer needs and wants and pursuing that.
> Across his various startups, Peter has seen two kinds of work: > 1. the "IQ 180" work. some mad scientist genius type comes up with some crazy solution you've never thought of. > 2. the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts. Interesting comp to pg's Maker's Schedule, Manager's Schedule https://www.paulgraham.com/makersschedule.html I'm curious about not only which of these roles models will fill, but also how they will empower us to be in the mode we prefer.
It makes sense that we’ll see “room at the bottom” strategies. Currently, large parameter counts seem to be slush funds of world knowledge, language skills (because language’s nuances and open vocabulary make it high-dimensional), and reasoning primitives, the general belief being that the latter takes up the least space in the model. There are many applications where world knowledge is unnecessary or even a negative, and in which only a small amount of language skill is necessary, and there we can expect small models more intelligently used to beat large ones naively used.