Untitled

I worked on getting an address database into elasticsearch years ago when it was still using modified tf-idf. Customers wanted FTS where a lot of the queries would be something like "100 First Ave, NY" or "200 2nd St, MN". It was one of the most fun projects I've worked on in my career so far. I got a learn a lot about how US and international addresses worked, so many edge cases, and got to really understand how customers were using the existing search to make sure they weren't adding any duplicates to the database. Token filters and synonyms were neat and figuring out the right indexing strategy was a lot of fun. It was a lot more work to get it right for most of the use-cases our customers had than just "throw it into ES and be done". That would probably have been fine for the 80/20 case, like you said, but I agree that the bulk of the work is going to be fine-tuning the search solution, whatever technology you're using.

I think people also overestimate the need for full text search when the one doing the querying is an LLM. If your underlying data is structured records, like a customer database, while humans might not have time or skills to figure out that when they want to search by phone number they need to do a join from the contacts table to the users table and normalize the phone number to look up first, making it best to just surface phone numbers as part of the data that is full/text-indexed… an agent is quite happy to handcraft the right SQL to find records that match on a specific field, given the right SKILLS.md and schema information. Turning fuzzy searches into exact DB lookups is a great way LLMs can augment users. (Obviously this doesn’t apply to searching actual rich document data - for that, go all in on text search, embedding, etc)

I indexed thousands of documents into a SQLite database with an FTS5 index, plugged it into DeepSeek v4 Flash and got better much better results than any other commercial solutions my company has tried in the past. The trick was just to let the LLM come up with its own SQL queries for searching... and the results are impressive.

RAG is art. I have a very straight forward setup that is highly modular. RAG is routing and decision making. I found so much joy in achieving the best results given the requirements than simply hoping for the best with the cool kid called vector db and embeddings. I agree with you. Depending on the context and required output I decide how to orchestrate a multitude of specialized modules that produce the best specific result to gain a universally usable system. It maintains itself. Also live updates need reruns and rebuilding certain indexes. Everything is highly dynamic but in a deterministic way. I found my niche with RAG selling and I build them myself. I take pride in them. So many look at the technology but not on the required output. It takes hours of talking to people to get an idea of what they need. And there are regulated businesses where certain information is required to be always factual correct - pricing for example. Vector search becomes a liability for this use case. So naturally you have to reconsider your system: mixing factual with probabilistic content and how to make sure, it hits always certain quality benchmarks and on the other hand doesn’t fail others. I love this kind of stuff. And there is personal information etc. Using modules is the key. Orchestration is really fun but I have to admit, not for the faint of heart. And ever changing parts: LLMs, or restrictions to be matched liked autonomously working - I love RAG. It gave me back the joy of developing. In fact I never had so much phun before, because it is also “team work”: I am not programming, I am managing a product. I was in Senior Management of a top tier international bank and besides that build the only ever working platform or IT transformation called dbCORE and overlooked 13 teams with 120 developers. RAG gives me dbCORE vibes so to say. Good luck and fun with your RAG systems.

Re: the rube goldberg machine of diminishing returns https://www.anthropic.com/engineering/contextual-retrieval This is from two years ago, but I think it's still SotA?