Show HN: OSS AI agent that indexes and searches the Epstein files
157 points - today at 1:56 AM
Hi HN,
I built an open-source AI agent that has already indexed and can search the entire Epstein files, roughly 100M words of publicly released documents.
The goal was simple: make a large, messy corpus of PDFs and text files immediately searchable in a precise way, without relying on keyword search or bloated prompts.
What it does:
- The full dataset is already indexed
- You can ask natural language questions
- Answers are grounded and include direct references to source documents
- Supports both exact text lookup and semantic search
Discussion around these files is often fragmented. This makes it possible to explore the primary sources directly and verify claims without manually digging through thousands of pages.
Happy to answer questions or go into technical details.
Code: https://github.com/nozomio-labs/nia-epstein-ai
As many others pointed out, the released files are nearly nothing compared to the full dataset. Personally I've been fiddling a lot with OSINT and analytics over the publicly available Reddit data(a considerable amount of my spare time over the last year) and the one thing I can say is that LLMs are under-performing(huge understatement) - they are borderline useless compared to traditional ML techniques. But as far as LLMs go, the best performers are the open source uncensored models(the most uncensored and unhinged), while the worst performers are the proprietary and paid models, especially over the last 2-3 months: they have been nerfed into oblivion - to the extent where simple prompts like "who is eligible to vote in US presidential elections" is considered a controversial question. So in the unlikely event that the full files are released, I personally would look at the traditional NLP techniques long before investing any time into LLMs.
jellyotsiro
today at 1:19 PM
On the limited dataset: Completely agree - the public files are a fraction of what exists and I should have mentioned that it is not all files but all publicly available ones. But that's exactly why making even this subset searchable matters. The bar right now is people manually ctrl+F-ing through PDFs or relying on secondhand claims. This at least lets anyone verify what is public.
On LLMs vs traditional NLP: I hear you, and I've seen similar issues with LLM hallucination on structured data. That's why the architecture here is hybrid:
- Traditional exact regex/grep search for names, dates, identifiers
- Vector search for semantic queries
- LLM orchestration layer that must cite sources and can't generate answers without grounding
sebastiennight
today at 2:49 PM
> can't generate answers without grounding
"can't" seems like quite a strong claim. Would you care to elaborate?
I can see how one might use a JSON schema that enforces source references in the output, but there is no technique I'm aware of to constrain a model to only come up with data based on the grounding docs, vs. making up a response based on pretrained data (or hallucinating one) and still listing the provided RAG results as attached reference.
It feels like your "can't" would be tantamount to having single-handedly solved the problem of hallucinations, which if you did, would be a billion-dollar-plus unlock for you, so I'm unsure you should show that level of certainty.
WhitneyLand
today at 2:23 PM
That doesn’t sound right. What model treats this as a controversial question?
"who is eligible to vote in US presidential elections"
Grok: "After Elon personally tortured me I have to say women are not allowed to vote in the US"
mariogintili
today at 10:55 AM
what are the most unhinged and uncensored models out there?
jellyotsiro
today at 1:29 PM
Open source models with minimal safety fine tuning or Grok
Grok is arguably not uncensored, it’s re-aligned to a specific narrative lane.
“Uncensored” is simply a branding trick that a lot of seemingly intelligent people seem to fall for.
Wait, is abliteration actually just a branding trick? That doesn't sound correct.
plagiarist
today at 1:22 PM
I understand uncensored in the context of LLMs, what is unhinged? Fine tuning specifically to increase likelihood of entering controversial topics without specific prompting?
Yes, or catering to a preferred world view different from the mainstream SOTA model worldview.
Look for anything that includes the word “woke” in any marketing /tweet material
What use-cases gave you disappointing results? Did you build some kind of RAG?
wartywhoa23
today at 8:58 AM
The question is not how to analyze that, it's how to prosecute those who are above the law.
In order to which you must analyze the files.
Not really. We know many people involved and they’re not going to get prosecuted. Analysis is not accountability.
I keep thinking that the lack of children’s faces in the blacked out rectangles make the files much less shocking. I wonder if AI could put back fake images to make clearer to people how sick all this is.
13hunteo
today at 10:20 AM
I understand the sentiment, but I'm always very concerned when it comes to AI generating pictures of children.
Why? They are generated pictures, not real pictures.
A lot of people are now struggling to detect which images are AI generated, and inferring reality from illusions.
To an extent, this was already the case with many other things, including stuff that was expressly labelled as fiction, but I recall an old quote, fooling all of the people some of the time and some of the people all of the time, it is now easier to fool more people all the time and to fool all people an increasing fraction of the time.
This isn't only limited to fake pics of kids, but kids are weak and struggle to defend themselves, and in this context the tools faking them seems to me likely to increase rates of harm against them.
How about adding a caption saying "the parts of this picture marked with a red outline have been generated by AI"?
in this context the tools faking them seems to me likely to increase rates of harm against them.
Why does it seem this way to you?
The history of age of consent laws including Pitcairn Island, the observed results of sexualised deepfakes in classrooms by other students, and the observation that according to sexual therapists "fetishisation" is the development of a sexual response and conversion into a requirement over the course of repeated exposure rather than any innate tendency that a person is born with.
You're barely scratching the surface.
> Mr. Gates, in turn, praised Mr. Epstein’s charm and intelligence. Emailing colleagues the next day, he said: “A very attractive Swedish woman and her daughter dropped by and I ended up staying there quite late.”
What if I told you that the child sitting on Epstein's lap, the teenager he French-kissed, the girl whose skin he covered with fragments from Nabokov's Lolita, the one who had an entire corridor filled with her pictures in one of his properties, who appeared in every framed photograph on his desk and whose name is on the CD-ROMs, the only woman Epstein said he would ever marry – what if that girl is the daughter Bill Gates mentions? And that she and her mother were Epstein's main romantic interests and most percussive tools?
nancyminusone
today at 1:44 PM
I believe this would decrease credibility of the evidence, not increase it.
Imustaskforhelp
today at 10:08 AM
Please create a way to share conversations. I think that can be really relevant here
I am not a huge fan of AI but I allow this use case. This is really good in my opinion
Allowing the ability to share convo's, I hope you can also make those convo's be able to archived in web.archive.org/wayback machine
So I am thinking it instead of having some random UUID, it can have something like https://duckduckgo.com/?q=hello+test (the query parameter for hello test)
Maybe its me but archive can show all the links archived by it of a particular domain, so if many people asks queries and archives it, you almost get a database of good queries and answers. Archive features are severely underrated in many cases
Good luck for your project!
jellyotsiro
today at 1:21 PM
Shareable conversations would definitely make the tool more useful yeah.
I really like the query parameter approach over UUIDs so it would make links human-readable
onionisafruit
today at 2:26 PM
> I'm experiencing technical difficulties accessing the archive at the moment. The search tools are returning internal server errors.
looks like it’s getting hugged
iowemoretohim
today at 4:18 AM
Those are going to be some spicy hallucinations.
darepublic
today at 1:53 PM
This is just feeding the files into a rag db I assume? I hope? And then you can use any decent model in front of it
jellyotsiro
today at 1:56 PM
rag is not a core! we use both semantic search but combining with fts, grep, direct read, etc.
kevin_thibedeau
today at 1:54 PM
It would be nice to have a way to query the exposed redactions to audit which of them were in violation of the Act.
Feedback: This agent didn't really work well when I tried it with a specific non-famous, but definitely publicly known individual with known connections to Epstein. I'd rather not post a specific name here. I found more documents with keyword searches. I guess it did get me to the conclusion that there wasn't much out there, but it didn't even mention stuff that showed up in name keyword searches.
To replicate though, you might look at the list of individuals mentioned in the brief email from Epstein to Bannon a couple weeks before Esptein died containing ~30 names and phow your engine works with each one. See how a keyword search does on library of congress vs your agent.
jellyotsiro
today at 1:24 PM
Thanks for testing this. The Bannon email from June 30, 2019 is in there (HOUSE_OVERSIGHT_029622). Good stress test idea.
Couple things happening:
Semantic search limitation: Less-famous names don't have strong embeddings, so it defaults to general connections rather than specific mentions
Keyword search gap: You're right — raw grep can catch exact names I'm missing
VladVladikoff
today at 1:39 PM
I saw a similar problem. Roger Schank had some conversations with Epstein and the emails can be seen in Epsteinvisualizer.com but your site claimed there was no emails or connection. To be fair to Roger, who was an AI legend of his time and someone I knew personally before his untimely death, he really was not a pedo, and most likely never got involved with the girls, I think him and Epstein just talked about AI and education mostly.
wutsthat4
today at 4:01 AM
And what did you learn?
subzero06
today at 6:27 AM
In 2024, Trump used Epstein's former private jet for campaign appearances
Also apparently the two had Thanksgiving dinner together as recently as like 2021?
jellyotsiro
today at 4:18 AM
Trump famously told New York Magazine in 2002: "I've known Jeff for 15 years. Terrific guy. He's a lot of fun to be with. It is even said that he likes beautiful women as much as I do, and many of them are on the younger side."
Trump and Epstein were social acquaintances in Palm Beach and New York circles during the 1990s-early 2000s. They socialized together at Mar-a-Lago and other venues
TowerTall
today at 4:26 AM
Interesting. It is my impression that almost everyone globally already knew this. What else did you learn?
jellyotsiro
today at 4:35 AM
ill take like 1 hour in the evening to dive deeper, i was never familiar with epstein stuff until i built the agent to simplify things for me.
Its peak HN to whip out a LLM, instead of just reading a news paper article or two.
ishtanbul
today at 4:56 AM
This is one of the most widey quoted phrases by trump on the topic of epstein
nathan_compton
today at 2:43 PM
Why the heck does this start with some sort of video bullshit?
sschueller
today at 7:33 AM
Is it able to handle a much larger dataset? Only a tiny fraction of data has been release from what is looks like.
jellyotsiro
today at 1:23 PM
yes! once for files come out, I will add them right away
Does this work with vector embeddings?
jellyotsiro
today at 4:58 AM
it uses semantic search so yes
Reminder that only 1-2% of the files have been released.
Yep: Breaking his campaign promises, in violation of the deadlines imposed by US Federal law, and with unlawful levels of redaction.
mschuster91
today at 3:14 PM
A case can be made to discuss if the deadlines imposed by that law are actually achievable with humans and an acceptable degree of errors (i.e. overredaction, improper/recoverable redaction, and underredaction).
That's also why many "large" criminal cases only have a very limited subset of the initial charges make it to trial (often to understandable public outrage). The larger the case, the more evidence material has to be sifted through to make an airtight case, so a lot of it is dropped before the trial to secure a conviction at all.
Basically Al Capone, rinse and repeat - they got him on taxes because that's far easier to prove than ordering or committing a murder to the required degree of certainty.
The interests of the victims, their families and the general public are different from the interests of the government... the victims/families/public want justice for the unique crime they were subject to, the government just wants to lock up the bad guy for as long (or as short, let's be clear) as possible.
DanielScharf
today at 1:37 PM
Super Cool!
This is a good idea. One thing I never understand about these kinds of projects though: why are the standard questions provided to the user as prompts never cached?
jellyotsiro
today at 4:15 AM
oh forgot about it, thanks. just a funny project i build in couple hours so didnt really sweat haha
This agent is really interesting! Learning a lot. Thanks!
Outputs are usually generated with random sampling, so the same prompt may get different outputs.
can search the entire Epstein files
It's worth noting that only about 1% of the files have been released, according to the DOJ.
Of the released files, many have redactions.
Yep, they failed to meet the deadlines required by law, and it's not just any redactions either, but unlawful redactions.
King-Aaron
today at 7:11 AM
If the Lake Michigan thing is just in the first 1%, then whatever's in the other 99% is going to be absolutely disgusting.
I searched it with the tool but nothing came up about Lake Michigan. What happened?
King-Aaron
today at 8:28 AM
https://www.justice.gov/epstein/files/DataSet%208/EFTA000250...
"He participated regularly in paying money to force
me to ___ with him and he was present when my uncle murdered
my newborn child and disposed of the body in Lake Michigan. "
The uncle is allegedly referring to Trump
https://www.freep.com/story/news/local/michigan/2025/12/27/a.... This mentions the Trump angle. It also mentions that the report came out before the 2020 election and could be fake. I'm a little confused because the report itself says nothing about Trump so don't know where the Free press gets that and they don't tell you what the source is or I missed it.
Edit: Oh I get it. The woman's statement Donald Trump is named as one of the witnesses. She says that he watched the murder. He wasn't the uncle. He is listed as a witness to the murder. This is highly highly suspect in my opinion. Seems very sensationalistic and no reason given it as to why Trump was there. His name is just thrown in.
The allegation is quite clearly that Trump participated in [ redacted ] this pregnant 13 year-old.
> [Trump] participated regularly in paying money to force me to [ redacted ] with him
The reason he was allegedly there was probably to [ redacted ] a 13 year old... That's what convicted rapists with deep connections to child sex traffickers do...?
I would expect a large portion of the remaining records to be internal emails about memos about the process of building a case around evidence, rather than the root evidence itself.
Not that that would excuse the administration's unlawful behavior so far, or indicate the unreleased 99% can't have some big bombshells.
jellyotsiro
today at 4:59 AM
sorry all publicly available files *
inquirerGeneral
today at 8:55 AM
[dead]
huflungdung
today at 10:25 AM
[dead]
[flagged]
Ah, yes. Post is an LLM-something project: top comment is a general critique of LLMs. Waiting for this to get old. Meanwhile, at least you get points for being funny.
sebastiennight
today at 2:42 PM
I think the GP was unfairly downvoted, as their comment wasn't a critique of LLMs but a comical attempt at critique of the source files themselves being redacted into uselessness.
sebastiennight
today at 8:45 AM
> + '' * n
This looks like what you'd get from using text-davinci-003 as the model in your AI-assisted IDE
flexagoon
today at 9:18 AM
I think it looks like what you get by writing code and making a typo.
no - the utf8 black box was removed by hackernews. thanks for noticing.
Can't edit it anymore, but it would be "\u25A0" * n
sebastiennight
today at 2:43 PM
Ha! That makes way more sense, and was indeed quite funny and undeserving of the massive downvoting.
slfreference
today at 11:47 AM
All these attempts looks like emulation of "Pen (software) is mightier than Sword" or that only if more people believed in the cause, we would be close to resolution.
Remember folks, soft power is nothing in front of hard power.