So - I run a company that does AI agents for pharmacy. We are at Series B now and growing well - series C in 9-12 months given growth. We just acquired a competitor of that vendor so very much an insider.
First, The technology works, and it scales, but the whole bottleneck is domain expertise and implementation. These are expensive, and hard to scale. We hire pharmacists as project managers, that's how important domain expertise and implementations are.
Second, the amount of noise of "Voice AI for <industry>" is incredible. Most of them are completely clueless about the industry & basically "YC-striver" type who can only sell to other yc companies. They fail hard the moment they touch critical functions of the real world.
We have hundreds of pharmacies with us now and they mostly successful and happy. Our biggest friction is customers being burned by vendors like this and writing off AI altogether as "not ready".
> We have hundreds of pharmacies with us now and they mostly successful and happy. Our biggest friction is customers being burned by vendors like this and writing off AI altogether as "not ready".
This is such a great graph. Your company feels successful because your customer is happy yet the rest of the world would be just happy if your company died in a fire. This seems to also be just fine for companies like yours.
First, most tech companies of comparable size to your customers have negative margins and survive on investment cycles, and the tech companies that are bigger haven't offered human support to anyone in decades. (And indeed, everyone already resents this practice and those tech companies. Generally, sentiment is abyssmal.)
I know you're just trying to illustrate the real plight of your customers, but it's a bad illustration that confuses the discussion.
Second, out of sincere curiosity: what was the future of the industry -- and these particular pharmacies -- if AI hadn't arrived to offer salvation? The role they fill in society is non-optional, so how the market restructuring around these infeasibly narrow margins before?
Most of the support is severely binned now to edge cases. You can't just make a genius bar appointment anymore like you used to. You get triaged and most of the flow punts you to some unhelpful support or even worse, apple discussion forums. You basically have to lie to get a genius bar appointment through the system. Once you have the appointment the worker doesn't care and gets right to work same as always, but feels like the writing is on the wall where the next CEO will probably cut genius bar entirely and be rewarded with a surge in share price for the lower overhead.
Funny how you danced around google too. They are easily the worst in terms of offering support. People online in all seriousness suggest getting a google business account for 1 month and cancelling afterwards to actually get tech support.
>Apple offers online, telephone, and of course in-person support for hardware and software
The last time I waited on hold for Apple developer support, I got to enjoy two whole hours of free hold music. So they offer human support, just not very enthusiastically.
...and Coinbase directs you to send a physical letter when their voice chatbot fails. Seriously. I had a login issue preventing me from accessing tax forms and the phone tree eventually got to the point where a sane provider would hand you over to human chat.
When the Coinbase voice agent gave me an address to mail a letter to and hung up, I was so shocked I had to call back and go through the process again. I was told to mail a letter describing my issue to an address on 5th Avenue in NYC.
If my experience is common, most health plans are moving to mail-order for routine, generic, recurring prescriptions. They only want you to visit a retail pharmacy for stuff that is urgently needed or strictly controlled.
Most pharmacies are having financial issues basically due to being stuck in the middle of the battle between insurers, drug manufacturers, and PBMs over reimbursement rates, and having the weakest negotiating position of all of the parties.
AI probably will save the pharmacies some money, but IMO this will probably cause reimbursement rates to decline further, leaving the pharmacies in the same position as before.
Weakest negotiating position other than the customers.
The funny part is probably a majority of pharmacies are owned by CVS or Walgreens each of which are insurance companies in their own right, and often people with company health insurance will ONLY allow reimbursements through their company pharmacy.
Tbh, this is most industries. Very few have much of a moat and have some wedge/pressure from their suppliers (land/labor/capital), their clients and competition.
Insurance companies and PBMs would probably say the same they have competitors (ie:replaceable) and can’t just pay more for this or that, or reject any cost saving measure available.
PBMs gained control of drug pricing and squeezed out the small pharmacies that don’t offer additional services like compounding.
CVS And Walgreens both own and operate PBMs. And UHG (Optum), 19 Blue Cross plans collectively own Prime Therapeutics, Kaiser owns one, the DVA has one, Express Scripts is owned by Cigna. There’s likely more I am unfamiliar with too.
I wonder why? My pharmacy in Spain is owned by this older lady who's had this shop for life. Not too many customers but she manages and even has a few part time staff. It pays enough to make a living. And she always knows what I need and pings me on WhatsApp if something is ready for pickup. And there's 4 such neighborhood pharmacies within 200 metre walk. No kidding. All are viable.
I wonder, what changed in America? Why is the mom and pop pharmacy not viable anymore? This sounds like a much bigger problem than having to answer a couple of calls, really.
I don't really understand how everything turns into a mega corp with shareholders in the US. Sure we have some of those here too but why are small businesses not viable?
> I wonder, what changed in America? Why is the mom and pop pharmacy not viable anymore
Deregulation at every level = small/low leverage players get squeeeeezed.
Most likely in Spain (and much of the world), you have controlled pricing of drugs and fixed markups/fees at pharmacies. Quite possibly it’s illegal there for a non-pharmacist to own a pharmacy and they can’t own a bunch as a silent/relatively inactive partner either.
True all those yes. In fact the govt pays for the meds, I just pay an issuance fee of 2€ or so. You can get private health insurance but I didn't bother, the state one is fine.
And yes you need to be a licensed pharmacist obviously.
I think the pharmacists mainly make their money on value add stuff and non prescription drugs.
People here mainly pick their pharmacy by familiarity and service not price. They're all the same anyway. I've never been to the other ones close to my house :)
It's kinda weird there's so many but it works so why not. And it's nice that she knows my name and what I'm on in case of issues. I live only 5 doors down the street. I really like that about here.
> Deregulation at every level = small/low leverage players get squeeeeezed.
I love how stating a simple truth gets you downvoted. Deregulation doesn't mean "less regulation". Maybe in the abstract, but in practice after it passes through the political machine it means increasing regulation that entrenches a handful of large "private" businesses, while removing regulations that would constrain those businesses commensurate with their market power.
It's a plain fact - the players in the health care industry are closer to de facto governments rather than productive businesses in a competitive market. But yet they're allowed to squeeze pharmacy owners, patients, and anyone they can due to lack of effective regulation. That is why our small pharmacies are going away.
Note that I personally favor more market based reforms (eg the sheer majority of visits/procedures/medication should have up-front prices, and those prices should be uniform no matter who might be paying). But we've currently got the worst of both worlds and I would basically support any angle of reform as long as it actually reforms rather than resulting in more regulatory capture.
I think regulation and deregulation doesn't mean good/bad for small business anyway.
For example the EU just extended mandatory packaging regulations (PPWR and EPR) for businesses of any size, today. Meaning small artists sending as little as one package to an EU country have to register in that country (there is no central EU registry) and pay yearly fees which add up to thousands for all EU countries. In advance, no less. This even applies if you just chuck one single pendant in a tiny bubblewrap envelope, it's ridiculous. And it applies even within the EU where there is supposed to be a free open market. They have to pay the same fees as a Temu or Amazon, for whom a few thousand euros is peanuts.
So a lot of independent artists are cutting shipping to the EU or only allowing some where registration is free. This is one example where regulation is literally targeting small businesses to kill them.
Yes, you are describing the dynamic I am talking about. My point about "deregulation" was in the US context where there is a strain of people who reflexively infer that deregulation must be good, from a general idea that regulations are bad. Regulations most certainly can be bad (as is the one you're describing), but this does not imply that blindly "removing regulations" is good! Because what tends to happen is that the type of regulation you are describing (registration/compliance/etc creating a fixed costs that are hard on small businesses but easy for large businesses) never gets removed. Rather what does get removed is regulations that are inconvenient to the large businesses - things like right of withdrawal [0], liability, anticompetitive restrictions on large businesses who have now become unavoidable parts of an uncompetitive market [1], etc.
[0] IIUC "right of withdrawal", at least. It's an EU thing where you have a legal right to return products after a purchase, right? Whereas in the US it operates more on custom, customer satisfaction, and marketing.
[1] eg if artists have to start selling through Amazon (etc) rather than shipping direct
Yes but the right of withdrawal in the EU is only for things bought at distance (e.g. over the web or by mail catalog (if anyone still does this, lol)). And it's within 14 days and with some exception (hygiene-sensitive products and custom-order items etc). It is however allowed to open and use the product as long as you keep it in a pristine state. It also applies to services such as insurance. I recently cancelled my revolut package within 14 days as the benefits were not as good as claimed.
If you buy something in a physical shop it is not a right and dependent on goodwill of the seller. Similar to the US. The thinking is that with buying remotely you don't have the ability to examine the product. The supplier may charge a small restocking fee and charge for return shipping, but most don't do that in practice.
You can also use it to return for things like dead pixels, even though some suppliers try to limit that (e.g. saying you must accept x amount of dead pixels), it's not allowed to refuse that, IF you buy remote.
Funny thing is that it also applies if you order something online for pickup at a physical store. So for things like displays this is highly recommended.
Clueless? You sound tone deaf. If you came here thinking everyone would love your product becase AI, well, sorry about that.
Just because you think it is an amazing product does not mean the rest of us will agree with you. It's right up there with "I've never had a problem with it so anyone claiming to have a problem is wrong" mentality. It was your comment acknowledging the public does not like your product or products like yours. You just think your product is better than others. That may be true but it again does not mean it is so good that the public enjoys interacting with your product.
> But the real reason Walgreens, and the pharmacy business in general, is dying, is because of a failure to enforce antitrust laws against unfair business methods and illegal mergers. Elson touched on it when he mentioned lower reimbursement rates, but I don’t think people appreciate the full scope of what happened to Walgreens, and to the full pharmacy business in general. This is not a case of bad management, it’s a case of desperate management.
If you read the data you would have observed that a large chunk of those store closures were by the big retail pharmacy chains, which blows your central thesis out of the water.
Maybe in absolute numbers but the situation for the ma and pa operations is worse:
“Independent pharmacies, often excluded from networks by pharmacy benefit managers, were more than twice as likely to face closure compared to chain pharmacies.”
NPS typically improves because no one stays on hold for 20+ mins anymore.
People have no ideas have pharmacies work. Pharmacies don't control how much they pay for drug, or how much they get paid. The margins are terrible and you can't hire enough techs to answer the phone at peak call volume. So - most people experience are terrible as they wait on hold for a really long time.
EVEN if the AI is a little rough around the edges with relational calls, just solving for transactional calls with no hold is a massive win.
I'll gladly take a 20min hold after having to resort to screaming "give me a fucking human" or driving the the pharmacy JUST TO TALK TO a human in the exceptions, which seem to be about 75% of my interactions with CVS/Walgreens AI agents.
> or driving the the pharmacy JUST TO TALK TO a human
It’s nice that this remains an option. It’s a dying one. So much is better handled and more easily handled in person than over the phone/mail/web/fax/email/blah, even if they’re 1:1 with a human.
It’s why I stick with my credit union instead of moving everything over to an online bank even tho I’ve managed to get 99% of everything done with the CU without going to a branch.
I recently had an AI chatbot lie to me. SiriusXM's ai bot INSISTED that the weather notifications were NOT a siriusxm feature. But, I had called ~4 months previously and the human confirmed it WAS a feature and she very kindly turned it off
Just as you said, I spent 20 minutes YELLING at the AI before it would transfer me to a human. Oh, and the entire time it was spewing ads for their programming. When I got to the human she, once again, confirmed it WAS a feature and it re-activated during 'free weekends'.
My father had cancer. After he was diagnosed he retired. His full time job became managing his health: medicare, state benefits, doctors appointments, medications, talking with billing people. He was on firstname basis with multiple providers. He invited some nurses and their families to the first 4th of july when he was cancer-free (they politely declined to keep the relationship professional).
I just imagine my father, or my 75yr old mother (whose health is beginning to decline) trying to navigate this new AI world. It must be an absolute nightmare. I can imagine how frustrated and disempowered they'd feel after a lifetime of reasoning with other humans.
EVEN if the AI is a little rough around the edges with relational calls, just solving for transactional calls with no hold is a massive win.
You keep saying that reducing call time is the "win." It seems like that the only metric you're measuring. If so, your product is a failure, not a success.
This is HEALTHCARE. You don't get to do "rough around the edges." You don't get to dismiss edge cases.
I also work in healthcare tech. My company doesn't permit such things because people die. I'll write it again because it's a point that tech fetishes can't get into their heads:
Do you know what NPS is? That's how the comment started. Patients like it. Because the alternative is to stay on hold for 20+ minutes, and then get it wrong. (Or have no pharmacies except Optum's and Express Scripts).
Good for you - I want in the ICU delivering care during COVID as a frontline clinician. You don't get to pull this line.
I find it hard to believe that patients like it given how sales work for these sorts of services - customer satisfaction is often the lowest priority given it’s ultimately a cost cutting service.
I live in a major metro and there are no pharmacies left except the largest ones. All my problems have to do with insurance not covering things, and insurance asks me to ask the pharmacy, pharmacy asks me to ask the insurance, and often insurance then asks me to ask my doctor to adjust the dosage cause they won’t cover that many days. Customers are batted around and I don’t see any AI that will solve this problem which is structural. The only value I see in AI is cost savings to replace humans, which is to say practically none as a customer over a website to check status and try to refill things.
Not to mention my insurer will only authorize >30 day scripts if you use their own wholly owned mail order service. Want/need to get in person, locally? 30 days or less.
NPS is notoriously unreliable. When I’m frustrated with a stupid support clanker, I don’t wait for another different clanker to ask me how it did. I hang up.
If that’s how you’re measuring patient satisfaction, you’re completely clueless.
NPS is asking your customers "how likely are you recommend us to a friend or colleague?". That's definitely not an accounting number. If you care about this, you can instead ask your _new_ customers if someone recommended you. Or you could hire a third party to do an independent survey among your target demographic.
Maybe “rock solid” could be understood as slightly hyperbolic (though it is metaphorical, so hard to say whether it is really hyperbole), but, it is possible to predict some things probabilistically with a high degree of accuracy. Depending on the task, a high enough degree of accuracy sounds doable, and if it offers a sufficiently convenient fallback to an actual human when it fails, that sounds like it could be quite a bit nicer than navigating a phone tree menu?
Classical voice to text combined with NN based voice to text, I would imagine can be highly accurate, and that’s probabilistic.
Of course, you're right. I didn't intend to criticise him.
I just think that it's not a good idea to transfer this megatrend to healthcare.
Yes, those systems can be highly accurate, but as we all have experienced, this is not a stable or consistent property. A very good model can give you an ingenious answer one minute and an utterly dumb one the next. My guess is that it's simply a consequence of the extremely high complexity of the 'plant' (natural language, language interfacing with the real world), leading to some highly non linear behaviour.
Good to see info on how critical domain expertise is for success.
I've seen AI used well in a few support situations, and it is great - faster to get to the answer vs a human (usually faster than even getting to a human, which is another issue)
When my case can be handled by the automated system, it is great, when it is not, it is usually misery, leading to anger at the customer and whatever company sold them that automated isht.
Especially for intelligent knowledgeable users of anything, by the time we call, it is an issue likely to need escalation.
So, the one absolutely critical factor for me and everyone I know is: how fast and easy is it to get to a human when we find the automated assistance does not work for our particular issue? It should be "I need to speak to a human now", and the immediate response should be "OK, let me get you one... Hello, this is [person], how can I help?" with the prior transcript already on the human's screen.
If this works, it is good because the humans are operating a level up, not having to deal with endless monotonous minutiae, and dealing with interesting issues all day, which means they get good at it and have a better attitude — a win for all.
How, and how well do you handle that situation — how close are you to the proper response I described?
As someone who has worked with companies in the space more than a decade ago, I'm glad that you're doing this! Genuinely love the "embedded not installed" approach.
I suspect that applying an aerospace approach of dissimilar redundancy and root cause analysis to this will yield a much better experience for everyone (i.e. you, the pharmacies and the patients) than what would have been possible with humans alone.
I've ranted about this before, but medicine as it stands isn't a serious field. And I can say that with a straight face, because medicine has until now, been the only field of modern scientific endeavor (or rather cloaked under modern scientific endeavor) that has fought tooth and nail against gathering more data points for improving understanding.
With pharmacies, people are emotional because their perscriptions getting filled can literally mean their livelihood, their health or their life. I hoard medications because my insurance will only cover 30 day fills, and often it takes 2-4 days to call enough people for insurance to cover a refill. Its a common problem for older folks to take half of their perscribed medication from either cost, or inability to get through to the pharmacy.
You effectively have AI gatekeeping healthcare, as companies roll these out and also take away any escalation path that a human call center, or local pharmacy staff might've been able to handle.
I've been also working with AI for years now, and it is absolutely horrifying to me that any company would feel comfortable rolling it out - you're basically saying a 5% error rate is acceptable to you when you're providing a gatekeeping service that is life or death.
Yes, down to the false equivalence of "human hallucination" with AI hallucination. Except that a human who hallucinates too much will eventually be permanently discredited and especially so if that human is trying to be an expert in too many things. Yet we are asked over and over again to forgive the LLM's mistakes because of what it can do. Oh it can do this right, this was just one situation where it did something wrong. So tiresome.
> People tend to compare AI to perfection for some reasons, which is not the case with humans.
So I mostly avoid commenting on or pontificating about the value or lack thereof of AI, but the problem I have with companies rolling out "AI for whatever" isn't that sometimes AI gets "whatever" wrong, it's that the companies simultaneous gut the processes around correcting these mistakes.
I'm currently dealing with a months-long banking error where the bank has incorrectly linked my credit card account to my wife's account and/or an account of mine in a different region, so it variously doesn't show up in my online account, shows the wrong number, shows the purchases from the wrong account, etc.
Every time I call in in a 5-minute rigamarole of:
"Please describe what you're calling about."
"Representative."
"In order to connect to a representative, please describe what you're calling about."
"There's an error with my account."
"Would you like to know your account balance?"
"No, my credit card account is not correctly configured and isn't showing up in my online banking account."
"I didn't understand. Please describe what you're calling about."
Eventually the bot reaches some failure state and then allows me to wait on hold for an hour to talk to an actual rep.
When I eventually get to talk to a rep, I get "Yes, I see that your account isn't correctly configured, I'll submit a back-end ticket and it should get fixed within 5-10 business days."
The next week I check my online banking, my credit card still isn't showing up, and I call and repeat the entire process.
So yes, it's not AI that's directly causing this Kafkaesque nightmare, but it's AI that's empowered the leaders at every utility/retailer/bank/insurer/etc. that I deal with turn their support systems into this slop.
Anecdotal evidence to support this - on a much less important platform.
We have two cars from our local car dealership. When we first set up the account, we did it through what I assumed were some kind of AI voice prompts. First time we take my wifes car in? I'm getting the calls from the dealership about whats wrong. Then I start getting texts telling me the car is ready. Same thing with her, but she's getting all the calls and notifications when we take my car in.
This has gone on for THREE YEARS. Last year we just gave up after telling people in person and going online and changing it several times. It did work for one visit, then reverted right back to mixing them up.
This isn't nearly as important as your banking issues, but its in a completely different industry and the experience is nearly identical.
That's been my observation of AI customer support. They're pretty much all glorified FAQ reciters unable to handle any unusual or uncommon situation not in the FAQs.
The one AI customer support experience I’ve actually enjoyed was last winter in a cold snap when my pipes froze. Drove to a Walmart to find a space heater and of course they were sold out. Called my local Lowe’s, got an AI agent, asked if any space heaters were in stock, it told me the one model left and the aisle to find it. Drove there and got what I needed without a fuss.
That’s the exception of course, literally every other time I’ve had to deal with an AI agent it can’t deal with the slightest deviation from its assumptions. Most of the time I call support, it’s because I already tried the automated options and they didn’t work for me. The fuzzy, nonstandard situations AI customer support agents suck are the one thing customer support is most needed for. I would’ve just used the website to check space heaters in stock if I hadn’t been driving.
I had a mixed incredible/horrible experience with Claude's AI support agent. First, I suspected there was a particular problem with the API access, and I articulated it and said, "can you give me a way to test this theory and prove that the problem is on your end? Python is acceptable" and it spit out a python test program that proved the problem was on their end. Then it turned a corner and spent 30 minutes insisting that it was going to connect me with a real person who could fix the problem for me before finally admitting that the whole thing was a boondoggle and it had no way to connect me with an actual person, but maybe I should come back to the chat in a day or two and see if anyone noticed and responded.
The real interesting thing was that early on in that 30 minutes, it gave me very clear instructions about how to get a live agent (including a quoted trigger phrase like "I would like to speak to a live agent" or something), which made me believe it was possible, which ultimately made me resist the slow realization that that was a complete fabrication. It was a very effective way to waste my time.
With that experience, why have you stayed with that provider? It would seem to be no harder, and quite satisfying, to open new accounts with another provider for you and your wife, and then cancel these accounts.
Basically every provider seems to have a similar level of customer support.
For banking in particular, I already bank with a half-dozen banks because they have different products/services that are optimal for some specific use cases (i.e. I min-max my bank accounts, credit cards, and investment accounts.) — however, customer support seems to be universally bad.
If you're curious about why I'm using that particular bank, I'm forced to use them for some investment accounts because my employer matching benefits uses them. For the credit card, they're one of the few banks that offer proper co-applicants (rather than just supplemental users) which really works best for the way my wife and I have our finances organized. And flights in my city are dominated by a single airline where the only way to avoid paying for baggage is to get the co-branded credit card from this bank.
Right, of the faster/better/cheaper triad, these things are overwhelmingly being implemented towards cheaper, and the idea that owners can gut the support structure for errors and edge-cases.
I realize quoting Doctorow on this is... rather low-hanging fruit for this topic, but I like how this scenario suggests the AI company makes more money when the implementation is hostile to the consumer:
> We’ve heard some pretty credible stuff about how AI can be used to spot solid mass tumors that sometimes humans miss. And, you know, if there was a sales call right now at your local hospital, where there was a pitchman for an AI company telling the hospital administrator, “Here’s what we’re going to do. Right now you have 10 radiologists. They cost $3 million a year. They review a hundred X-rays a day each. And I tell you what: I’m going to sell you a chatbot for a million bucks a year. And it’s going to sit in the shadows, and a couple of times a day, it’s going to tap your radiologist on the shoulder and say, 'Why don't you take another look at that one? I’d be very happy,’” that would seem like a real advance on medicine. But that’s not how the pitch is going.
> [...] But what the AI companies want to sell you is: fire nine-tenths of your radiologists, save $2.7 million a year, split that between the hospital shareholders and Sam Altman, take that remaining radiologist and put them in charge of marking the AI’s homework, put them in charge of clicking OK a hundred times a minute for the radiology reports that are coming out of the chatbot, and then, when it misses something and someone dies, blame that guy, make him what Dan Davies calls the “accountability sink” for the AI.
I actually made another comment about lack of leadership causing many issues including exhausting dev teams.
What you describe is an execution failure not AI, these are basic NLP systems that could be easily configured with some dataset to make them better, but i will make a wild guess that the bank has incompetent manager who was pushing for some numbers to put in a slide to say “I” did this.
The parent commenter's experience pretty much sums up my experience with every IVR system, AI or not. The IVR cannot do anything I can't do on the website or the app. I wouldn't be calling if I could get the job done without calling them, so my experience is simply that the IVR is simply an obstacle to talking to a human. Very often the first human I talk to is simply an obstacle to talking to the second human, because the first human is so limited in their knowledge and authority that they can't do anything I can't do with the website or app either.
I don't really think this is an implementation problem or incompetence. AI is just not reliable enough yet to deal with long tail situations, especially in cases where the most common procedures don't apply. You can't give it the authority to override common-case business rules or make financially consequential decisions, because it will be manipulated/prompt injected by fraudsters.
I suspect a lot of big orgs don’t care if they lose high-need/low-margin/negative-margin clients that need a couple chains of humans to fix their issue.
If they were otherwise high profit customers, they wouldn’t/shouldn’t have put them into IVR hell in the first place.
Anywho, I’m off to acquiring a DID that suggests I’m from a high-net-worth area. Until then, I’ll forge my caller id.
> People tend to compare AI to perfection for some reasons, which is not the case with humans.
Yes, because humans have intrinsic value. Even if they make mistakes, they are still one of us and deserve to be given a chance. The same can't be said for clankers.
Basically- the same math as modern automated manufacturing. Super expensive and complex build-out - then a money printer once running and optimized.
I know there is lots of bearish sentiments here. Lots of people correctly point out that this is not the same math as FAANG products - then they make the jump that it must be bad.
But - my guess is these companies end up with margins better than Tesla (modern manufacturer), but less than 80%-90% of "pure" software. Somewhere in the middle, which is still pretty good.
Also - once the Nvidia monopoly gets broken, the initial build out becomes a lot cheaper as well.
And if you ever stop/step off the treadmill and jack up prices to reach profitability, a new upstart without your sunk costs will immediately create a 99% solution and start competing with you. Or more like hundreds of competitors. Like we've seen with Karpathy & Murati, any engineer with pedigree working on the frontline models can easily raise billions to compete with them.
Expect the trend to pick up as the pool of engineers who can create usable LLMs from scratch increases through knowledge/talent diffusion.
The LLM scene is an insane economic bloodbath right now. The tech aside, the financial moves here are historical. It's the ultimate wet dream for consumers - many competitors, face-ripping cap-ex, any missteps being quickly punished, and a total inability to hold back anything from the market. Companies are spending hundreds of billions to put the best tech in your hands as fast and as cheaply as possible.
If OpenAI didn't come along with ChatGPT, we would probably just now be getting Google Bard 1.0 with an ability level of GPT-3.5 and censorship so heavy it would make it useless for anything beyond "Tell me who the first president was".
We have been running this playbook for the last 2 years in healthcare, and we have been super successful. Doubling every quarter over the last year. 70%+ profitability, almost 7 figures of revenue. 100% bootstrapped.
People are still mentally locked in to the world where code was expensive. Code now is extremely cheap. And if it is cheap, then it makes sense that every customer gets their own.
Before - we built factories to give people heavy machinery. Now, we run a 3d printer.
Everyday I thank SV product-led growth cargo cults for telling, sometimes even forcing our competition to not go there.
One of the most pleasant experiences I had writing code, is early AI days when we did hyperscript SSE. Super locality of behavior, super interesting way of writing Server Sent Events code.
on message as string
put it into #div
end
on open
log "connection opened."
end
on close
log "connection closed."
end
on error
log "handle error here..."
end
Lots of YC companies copy each other process and selection criteria. Basically- they all have the same blind spots and look for the same type of engineer.
So, super easy to scam all of them with the same skillset and mannerism.
I send this article as part of onboarding for all new devs we hire. It is super great to keep a fast growing team from falling into the typical cycle of more people, more complexity.
Thanks for the link to the ColPali implementation - interesting! I am specifically interested in evaluation benchmarks for different image embedding models.
I see the ColiVara-Eval repo in your link. If I understand correctly, ColQwen2 is the current leader followed closely by ColPali when applying those models for RAG with documents.
But how do those models compare to each other and to the llama3.2-vision embeddings when applied to, for example, sentiment analysis for photos? Do benchmarks like that exist?
The “equivalent” here would be Jina-Clip (architecture-wise), not necessarily performance.
The ColPali paper(1) does a good job explaining why you don’t really want to directly use vision embeddings; and how you are much better off optimizing for RAG with a ColPali like setup. Basically, it is not optimized for textual understanding, it works if you are searching for the word bird; and images of birds. But doesn’t work well to pull a document where it’s a paper about birds.
Makes sense. My main takeaway from the ColPali paper (and your comments) is that ColPali works best for document RAG, whereas vision model embeddings are best used for image similarity search or sentiment analysis. So to answer my own question: The best model to use depends on the application.
I would like to through our project in the ring. We use ColQwen2 over a ColPali implementation. Basically, search & extract pipeline: https://docs.colivara.com/guide/markdown
First, The technology works, and it scales, but the whole bottleneck is domain expertise and implementation. These are expensive, and hard to scale. We hire pharmacists as project managers, that's how important domain expertise and implementations are.
Second, the amount of noise of "Voice AI for <industry>" is incredible. Most of them are completely clueless about the industry & basically "YC-striver" type who can only sell to other yc companies. They fail hard the moment they touch critical functions of the real world.
We have hundreds of pharmacies with us now and they mostly successful and happy. Our biggest friction is customers being burned by vendors like this and writing off AI altogether as "not ready".