In recent months as the frontier models have become more capable and more expensive, the cost of using LLMs daily has increasingly become the focus of conversations. The lauded token maxxers of a few weeks ago are now being looked at with suspicion.
"We spent $10,000 in tokens last month, but are we sure we're seeing a proportional increase in business value?"
It's a very good question and one that I think is going to decide whether agentic ai for coding remains a viable use case, it's possible that it's just too expensive to be worth it.
OpenAI and Anthropic want you to think of their agentic products as virtual employees, so when you think of it that way $10k a month becomes a single salaried employee at $120k a year, with no healthcare or other administrative costs it's practically a steal.
But on the front lines, actually working with LLMs day in and day out is like working with Amelia Bedelia (iykyk) if she were a brilliant computer science intern instead of a housekeeper. Nodding and smiling away as you feed it detailed prompts and responding convincingly as if it understands, and then goes off and thinks of a very convincing plan of action, only to continually miss the forest for the trees.
This is okay I guess if your employer is blindly footing the bill at one of the larger companies, but I don't think it's sustainable long term for companies realistic limits on their finances.
In my personal projects it's definitely caused me to change how I use agents, when Github Copilot was first released I was using it all the time and barely able to crack 50% of my monthly usage limit. After the recent changes where multipliers for usage were applied to frontier models I was barely able to use it for a few days before hitting my limit.
Their new pricing model feels even worse, $40 a month where they give you $40 worth of tokens a month doesn't seem worth it, I'm pretty sure I can blow through that in a day or two of normal use. The worst part about it is that I don't feel like I really have control over how that money is spent, if I ask an agent to do something and it spends a million tokens completely missing the point, should I have to pay for that?
Taking a step back the whole thing starts to look suspiciously like a slot machine. I pull the lever by entering a prompt, and the dials spin for a bit before producing code output that's almost a jackpot, so I refine my prompt and pull the handle again, and again, until the code does what I wanted it to. Probably a gross oversimplification since the output is not completely random, but randomness that is guided by model weights is still random.
OpenAI just demoed their latest model SoL (shit outta luck wouldn't have been my first choice) and two things stuck out to me.
First, they demoed a paper plane game where you guide a paper plane through rings as it glides down a boring hallway Pilot Wings 64 style. Right off the bat my first impressions was that it was very janky looking, graphics were clipping, objects were popping in and out of the environment. Just a poor game overall, for such a simple concept I would have expected their best model ever to be able to at least give me a clean demo.
So the next thing they demoed was using an agent with vision to fix some of the bugs in the game. I'll point out that they didn't choose one of the obvious visual glitches to fix, but they changed some text on the screen to show that the model could do things like update the text in a dialog. Okay, that gets a golf clap from me but uh, why was this simple demo so buggy in the first place?
Second, they showed off a demo where they wrote down some requirements and fed them into the new ChatGPT app, and then watched it spin up agents to write the tickets and then more agents to work on the tickets, but they didn't show us the final output which seems like...a glaring omission no? Maybe I'm just being cynical, but I feel like the folks who have the most access to the best models should be producing tighter demos.
I think the writing is clearly on the wall, I'm certainly not going to pay $100s per month for a robot to write code badly. If most people feel the way I do, it means that we're most likely going to see a shift to local models for most development unless you're working for a company that can afford to pay. I'm pretty sure that's not sustainable either though, unless the quality of the models gets dramatically better very soon I think you're going to see a falloff in adoption just due to sticker shock, and no one living off of this bubble can afford that.