You can’t be a skeptic without trying to also help. Being skeptical without helping is just being… cynical (and a boomer - OK Boomer). So I have been developing tooling to help. Here is my latest gadget:
As was the last post, catchy headlines with nuanced, reality-based analysis!
Having experienced this myself, I believe the enterprise “AI Committees” fail due to a poor understanding of AI being an driver of value as opposed to a tool that help people create value. Applications and patterns are vastly different depending on the use case, that it is difficult to come up with one coherent approach to these distinct problems. It’s as if one was forming a committee around using Excel: I can use it for financial modeling, edit CSVs, or a Super Mario Bros. game entirely written in VBA.
I also wonder if we should be talking about AI beyond pure text-token transformers. There is rapid advances in RL, CV in medical imaging and robotics, protein structure prediction, and teams working on world models.
The wave of investment and interest hyped up by scaled Transformers models might speed up other areas of AI research as well.
Even Sam Altman himself has said that there needs to be another breakthrough before achieving AGI - Scaling Laws and Agentic AI are simply not enough.
There are several questions I have about the future of AI algorithms. (1) Is it truly the most efficient way for a computer to learn by modeling human patterns? The push now is for multimodal models that learn video, audio, and text - as though modeling human senses will result in reasoning. A major part of the brain has little to do with the senses. Perhaps it is graph based ML like what Bronstein is studying.
(2) Two things led to massive neurological development in humans - thumbs and the use of tools. I can already see that the more tools you give a model, and the more you force the model to use the tools - the smarter the model gets. Another item that led to smarter humans was storytelling and debate. Again - having models debate yields better results - just don’t make the results of the debate be stored as a definitive fact, and you will be fine and avoid echo chamber problems. I don’t feel the current roadmaps really focus on this enough.
(3) But ultimately I wonder if the objective itself is wrong. The Industrial Revolution was not focused solely on replacement - it was focused on productivity augmentation to reduce costs and improve supply. The goal of the assembly line or the engine was not to replace a human - it was to make it so humans could do more for the same cost. But in the case of AI - it seems like we are interested in replacing humans. Humans are notoriously picky about agreeing to equivalence and a desire to always be superior. Perhaps the goal does not align to our own behaviors and further does not seem to be aligned with economics. Cost elimination is a race to zero.
It is always easy to explain to a VC that you will eliminate humans. But not always easy to explain this to a customer. It seems the objectives are skewed by a desire to justify a valuation versus actually achieve the eye popping objectives that the steam engine produced.
Great read. Could you point me to where I could find more about proper data preparation, like the one you mentioned for the financial service company and what they did? I’d like to learn more about good practices.
So what this firm did is use ODT (Oracle Data Transforms) to convert their GL data into a core data warehouse. In the case of their project, they were interested in understanding "what inventory is available that matches common customer search behaviors." They then merged the inventory data with product descriptions, which were vectorized. This enabled semantic search across their inventory, further enhanced with quantity and value information.
They were a mid-sized (~$750 million) e-commerce provider. What interested them in me was that I had accidentally recounted the decoupling of Amazon from Oracle Financials (my ex-boss was a leader on that project). They were interested in taking this decoupling further. That being said, the value of such a decoupling effort is rarely worth it - you can suck the data out of the OFA data mart.
What worked for this company was starting from the questions that they wanted to answer and working backwards. What I unfortunately see a lot of teams doing right now is going in the opposite direction - simply vectorizing data stores just for the sake of vectorizing it, in the theory that some future model will learn to take advantage of it.
However, there are some interesting ad-hoc preparation patterns worth looking into further. One interesting behavior I am interested in exploring further is a pattern I have recently seen. Use a cheaper model (e.g., Sonnet) to classify inbound questions. If this is the first time a question has been addressed (or a similar question has been addressed), route it to a higher-order model. Capture the result - and vectorize it. Now your Sonnet shortstop can benefit from the reasoned outputs, over time lowering your Opus utilization, making your Sonnet responses more intelligent, and putting you in control of model application.
But frankly, these types of approaches are interesting optimizations. For 99% of the workloads out there, the use case I just described - preparing a data mart to answer the questions your CEO wants responded to, getting the correct data elements together, vectorizing the text, making sure the extended attributes of the records have the metrics (in this case quantity and value) that the leader cares about, and only then training a large language model at the dataset at it is the pattern I would advise most customers to take. Unfortunately, the "intelligence" of the model becomes a distraction - and the enterprise ends up with dozens of cool prototypes and experiments, with limited ROI.
I hope this helps. I will ask the enterprise if they are interested in sharing the use case in public!
Thanks for the deep and comprehensive answer. It’s great to hear something true from the production side, not only power points and demos.
I am a business analyst and no matter what technology, I always look for business valye and this approach is what I prefer- like you said, start from the questions you neeed answered and work your way through. Unfortunately belief that technology will „save us” I see also with business processes and systems implementation „our processes don’t work, we need a new system to fix them”. And well I get it, there is nothing sexy in normal, smart but hard work, but implementing a new AI is very appealing.
You can’t be a skeptic without trying to also help. Being skeptical without helping is just being… cynical (and a boomer - OK Boomer). So I have been developing tooling to help. Here is my latest gadget:
https://github.com/srao-positron/camille
Thank you for reposting this!
As was the last post, catchy headlines with nuanced, reality-based analysis!
Having experienced this myself, I believe the enterprise “AI Committees” fail due to a poor understanding of AI being an driver of value as opposed to a tool that help people create value. Applications and patterns are vastly different depending on the use case, that it is difficult to come up with one coherent approach to these distinct problems. It’s as if one was forming a committee around using Excel: I can use it for financial modeling, edit CSVs, or a Super Mario Bros. game entirely written in VBA.
I also wonder if we should be talking about AI beyond pure text-token transformers. There is rapid advances in RL, CV in medical imaging and robotics, protein structure prediction, and teams working on world models.
The wave of investment and interest hyped up by scaled Transformers models might speed up other areas of AI research as well.
Even Sam Altman himself has said that there needs to be another breakthrough before achieving AGI - Scaling Laws and Agentic AI are simply not enough.
There are several questions I have about the future of AI algorithms. (1) Is it truly the most efficient way for a computer to learn by modeling human patterns? The push now is for multimodal models that learn video, audio, and text - as though modeling human senses will result in reasoning. A major part of the brain has little to do with the senses. Perhaps it is graph based ML like what Bronstein is studying.
(2) Two things led to massive neurological development in humans - thumbs and the use of tools. I can already see that the more tools you give a model, and the more you force the model to use the tools - the smarter the model gets. Another item that led to smarter humans was storytelling and debate. Again - having models debate yields better results - just don’t make the results of the debate be stored as a definitive fact, and you will be fine and avoid echo chamber problems. I don’t feel the current roadmaps really focus on this enough.
(3) But ultimately I wonder if the objective itself is wrong. The Industrial Revolution was not focused solely on replacement - it was focused on productivity augmentation to reduce costs and improve supply. The goal of the assembly line or the engine was not to replace a human - it was to make it so humans could do more for the same cost. But in the case of AI - it seems like we are interested in replacing humans. Humans are notoriously picky about agreeing to equivalence and a desire to always be superior. Perhaps the goal does not align to our own behaviors and further does not seem to be aligned with economics. Cost elimination is a race to zero.
It is always easy to explain to a VC that you will eliminate humans. But not always easy to explain this to a customer. It seems the objectives are skewed by a desire to justify a valuation versus actually achieve the eye popping objectives that the steam engine produced.
Anyway. These things bother me.
Great read. Could you point me to where I could find more about proper data preparation, like the one you mentioned for the financial service company and what they did? I’d like to learn more about good practices.
So what this firm did is use ODT (Oracle Data Transforms) to convert their GL data into a core data warehouse. In the case of their project, they were interested in understanding "what inventory is available that matches common customer search behaviors." They then merged the inventory data with product descriptions, which were vectorized. This enabled semantic search across their inventory, further enhanced with quantity and value information.
They were a mid-sized (~$750 million) e-commerce provider. What interested them in me was that I had accidentally recounted the decoupling of Amazon from Oracle Financials (my ex-boss was a leader on that project). They were interested in taking this decoupling further. That being said, the value of such a decoupling effort is rarely worth it - you can suck the data out of the OFA data mart.
What worked for this company was starting from the questions that they wanted to answer and working backwards. What I unfortunately see a lot of teams doing right now is going in the opposite direction - simply vectorizing data stores just for the sake of vectorizing it, in the theory that some future model will learn to take advantage of it.
However, there are some interesting ad-hoc preparation patterns worth looking into further. One interesting behavior I am interested in exploring further is a pattern I have recently seen. Use a cheaper model (e.g., Sonnet) to classify inbound questions. If this is the first time a question has been addressed (or a similar question has been addressed), route it to a higher-order model. Capture the result - and vectorize it. Now your Sonnet shortstop can benefit from the reasoned outputs, over time lowering your Opus utilization, making your Sonnet responses more intelligent, and putting you in control of model application.
But frankly, these types of approaches are interesting optimizations. For 99% of the workloads out there, the use case I just described - preparing a data mart to answer the questions your CEO wants responded to, getting the correct data elements together, vectorizing the text, making sure the extended attributes of the records have the metrics (in this case quantity and value) that the leader cares about, and only then training a large language model at the dataset at it is the pattern I would advise most customers to take. Unfortunately, the "intelligence" of the model becomes a distraction - and the enterprise ends up with dozens of cool prototypes and experiments, with limited ROI.
I hope this helps. I will ask the enterprise if they are interested in sharing the use case in public!
Thanks for the deep and comprehensive answer. It’s great to hear something true from the production side, not only power points and demos.
I am a business analyst and no matter what technology, I always look for business valye and this approach is what I prefer- like you said, start from the questions you neeed answered and work your way through. Unfortunately belief that technology will „save us” I see also with business processes and systems implementation „our processes don’t work, we need a new system to fix them”. And well I get it, there is nothing sexy in normal, smart but hard work, but implementing a new AI is very appealing.