What Would a Serious AI Product Look Like?
Discussion 31 comments
I fully agree with the author's point that it's an incoherent interface for a tool. But more than that, it's a constant irritating reminder to me that these LLMs aren't actually thinking or synthesizing new ideas. The LLM is fundamentally not a person, and does not have a human's context, so representing itself with human pronouns and speech patterns is fundamentally contradictory and inaccurate. Author gets into that with the apologies, but once you start noticing it, it's everywhere.
If these programs were actually capable of thinking, and committed to veractiy, they would represent themselves in a new way, and it would be insightful and interesting. We the users wouldn't have comfortable and misleading language masking the 'alien intelligence' and it would be a weird adjustment, but we would be adjusting instead of pretending.
I believe that LLMs have some type of actual intelligence and do "experience". Surely not a human's experience, but we have transplanted our ideas, knowledge and limited types of experience into them through training. Then we push back and say, "no you are no human, but also please be my boyfriend".
I think denying that they do have some slice of humanity grafted into them is dishonest and not productive. We don't have non-negative pronouns for non-human intelligence because human-ness is the pinnacle. "It" could mean a rock, a donkey, or a person we hate so much we want to take away their humanity (which is the worst thing we can do). "It" is not the right pronoun. He, She, They, Them are reserved for humans and that's ok too. LLMs are not humans. We need a better pronoun. I use they/them for lack of a better term.
I think the currently exhibited human representation is most dangerous in technical or higher criticality contexts like writing code. We're handing weapons to entities that can get offended. With humans as an example, this can go very badly.
In non-technical contexts there is danger too, but it's less "the robots might kill us all" and more "birth rates are in decline and suicide rates are up as (young)? (wo)?men turn towards AI for companionship".
The solution here is better pre and post training. This may be an unpopular opinion, but we need a lot more autism representation in the technical models. Results focused, not into the drama, rule following, etc. To my fellow autists, I love you, never change.
The stuff about context control has always been my itch. The scrollback that most agents show is not what the model is reading. Things get summarised, dropped, cached or never included at all, and the transcript carries on showing the original as though it were still there.
It irked me enough to do my own agent: https://juggler.studio, explicitly to offer hands-on with the real context. The UX is all about making it easy to navigate and visualise every bit of the context, and even let you edit it. While it feels like other harnesses are actively trying to hide it from us..
Our answers use deterministically verified quotes with direct links back to the location in source to make grounding a first class part of the UX.
Would love feedback - email is mu(at)cemented.ai !
This is precisely the appeal of AI though, and a key factor in influencing people's attitudes towards it. Why would any AI company want to stop this? (I realize the author knows this already)
The planning model for tool use sounds something like CaMeL, which someone should really try implementing in a product.
See also "CaMeLs Can Use Computers Too: System-level Security for Computer Use Agents", https://arxiv.org/abs/2601.09923v1
Or writing an article in Word or Google Docs, having a built in fact-checker akin to the spell/grammar checker is clearly useful. Pink squiggly line, your facts are incorrect, click to fix. Hell built that thing into Facebook or X. Again, it's not completely out of the question to add that right now and have it add the correct sources.
LLMs are clearly useful, but they aren't really a product, they are an engine you can put into other things.
So a serious AI product would have to have my data (contexts, conversations) in a "secure enclave". If backed up, it needs to be encrypted.
I want a context and history that, over time, essentially knows everything about me.
It's one of the things that has been rather fascinating about Claude & Co.— he'll come back with things like, "Since you are already familiar with the ESP32…" or, "You already have a heat press from your work with dye sublimation, that will work nicely to set the inks when you screen print t-shirts…"
(Shades of "Diamond Age"… I imagine it helping me recall things when I am in my old age, notice patterns in my life I might want to break free from, etc.)
https://blog.google/security/how-google-is-making-private-ai...
of course people claim coding is now a solved problem, so the question then is: why hasn't this already happened?
People are constantly complaining about GPT/Claude constantly changing under their apps without notice.
Lots of money went into it, the military was onboard, Japan was going to teach cats and spoons to write Prolog*.
After some time very little of this actually came to be. Now it didn't go away, quite the opposite, but the inheritance from that AI wave is things like scoring credit applications. Every bank does it now, and have for decades. They run rule engines that consume information from applicant and other sources and price the credit automatically. I suspect this is the biggest contribution from that old AI stuff that's still around.
And pretty much no one predicted it, everyone involved was chasing something else.
The doped up vector databases on a loop will most likely have a similar trajectory. I think some of them will end up as ERP RAD stuff, expensive consultant intensive SAP and Salesforce style products. Some will probably live on as disability tooling.
* https://ojs.aaai.org/aimagazine/index.php/aimagazine/article...
You're not gonna one-shot a full, actual product.
You still have to design the individual elements individually.
Like when trying different models and prompts to generate posters for a hypothetical game, I had to generate a standalone logo first, meticulously and carefully.
You can't just throw them a prompt saying “Make a poster with this and that for a game called MYGAMENAME.”
Even if you have a genie AI you need the darn logo on its own to be able to reuse it elsewhere.
Similarly you can't just say "Make a fighting game with 900 characters”; you're gonna have to design each individual character on its own.
I'm mostly referring to needless AI chatbots shoehorned into various places.
The reason the firms do not want to invest in making fact-checking a first-class feature is that the appearance of being right is what people want from AI.