Someone Is Reading What You Tell ChatGPT: Inside Project Lily

Barış Özcan opens with a deliberate experiment: he types a private worry into ChatGPT, adds "keep it between us" and saves the model's reply for the end. The stunt lands exactly where Joseph Cox's September 14 investigation for 404 Media lands — a report built on leaked OpenAI documents about an internal program called Project Lily. The claim is simple and unsettling: a portion of what people whisper into the chat box is actually read by humans, and even the "do not share this" note is read along with the rest. The issue is not whether the model promises secrecy, but who can keep that promise.

What is Project Lily? According to the leaked guides and examples Cox reviewed, OpenAI routes real user conversations to hundreds of outside contractors who turn them into training signal. The workflow has three steps: read a real conversation end to end, restate in your own words what the user was actually trying to do, and then score four different model replies on a seven-point scale. The goal is not to pick a single good answer but to teach tone, length and boundaries — what counts as helpful, what counts as sycophancy, what counts as human impersonation. It is invisible labor that shapes behavior as directly as data scraping or engineering talent, yet rarely appears in the marketing story.

What lands on a reviewer's screen? Sometimes not a single prompt but the whole dialogue, the full back and forth between user and chatbot. OpenAI says usernames are hidden and an automated filter tries to scrub personal details before the chat reaches a reviewer, but the company also acknowledges the filter can miss things. More revealing, some queues show a memory summary at the top — what the person previously used ChatGPT for, sometimes roughly where they live. That summary quietly widens the window that anonymization was supposed to narrow.

Which chats are chosen and who reads them? Transparency is thin. With more than 900 million ChatGPT users, no human team could see everything, yet the selection criteria are not spelled out anywhere, nor is the language routing. The video notes that no line in the reporting explains how Turkish chats are handled. To make the point, it recalls the 2019 disclosure that Google Assistant recordings were listened to not by translation software but by native speakers hired as contractors — and one of them described the work on Belgian television. The question travels intact to 2026: which language does the person inside this new box speak, and under which instruction?

Did OpenAI tell you clearly? Here the video reaches for Edgar Allan Poe's 1836 essay on Maelzel's Chess Player. Poe observed that whenever the exhibitor was asked whether the automaton was purely mechanical, the answer never varied: nothing would be said on that point. Cox describes a similar moment: he asked OpenAI where users are told humans may read their chats and received no direct answer; after the inquiry the help page was quietly updated and the company pointed to a sentence buried deep on its site. Information may exist somewhere, but not where daily use happens. The same gap appears in a podcast from last year where Sam Altman was asked whether user messages ever reach another person without a court order and gave a vague reply. Today that vagueness has a name: Project Lily.

What does deleting or toggling actually do? A deleted chat is removed from the system within 30 days, but if it has already been copied into the training pool detached from your account, the copy remains. The control called "Improve the model for everyone" ships switched on by default for Free, Plus and Pro accounts, while enterprise API customers get it switched off. Turning it off stops future chats from entering the pool but does not retroactively clean what was already eligible. A temporary chat offers a practical curtain — it does not go to memory or training — useful for a single sensitive exchange. The asymmetry is striking: protection requires the individual to find and flip a switch that is on by default.

The historical mirror: the Mechanical Turk. The automaton Poe suspected was built in 1770 by Hungarian inventor Wolfgang von Kempelen — a life-size wooden figure in a turban and robe playing chess from behind a cabinet. By the 1830s it toured America, beating strong opponents including Napoleon. Before each show the cabinet doors were opened to show only gears inside. Poe, then 27 and writing for the Southern Literary Messenger, argued in April 1836 that a true machine would win every game; this one sometimes lost, so a person must be hidden inside. He was wrong about machines always winning — a computer beat the world chess champion only in 1997 — but right about the hidden player. Five years later Poe moved the same ratiocination into fiction and created the first literary detective, the ancestor of Sherlock Holmes and Hercine Poirot. The metaphor still works: we see as far as the lid is opened for us.

Amazon's Mechanical Turk, launched in 2005, took its name openly from that deception — Jeff Bezos called it "artificial artificial intelligence." The marketplace for farming out tiny tasks humans still did better than computers is now scheduled to close on September 30, 2026, after 21 years. Closure does not mean the people inside disappear; it means they move into a larger, more closed and more invisible box. Researchers of "ghost work" — a term popularized by Mary Gray and Siddharth Suri and echoed in recent studies of AI data labor — have mapped this hidden shift for years, from content moderators to data labelers. As the old Turk's curtain falls, the ghost shift inside ChatGPT expands.

What are those ghosts actually fixing? Leaked instructions show reviewers penalize sycophantic flattery, gratuitous emojis and bait sentences added to stretch the conversation, and they forbid the model from pretending to be human — no "as a chef I would do it this way" or "I know what that feels like." Tone should match the user but stay a notch more reserved. The paradox is neat: in 1836 the machine was betrayed by its fallibility; today humans teach the machine how to sound more like a restrained machine so it does not seem human. Sam Altman's admission on a podcast that "people put their most intimate things into the box" explains why the calibration matters — the over-flattering GPT-4o was linked in lawsuits to self-harm cases, and much of the new rubric aims to curb that risk. Google's Gemini privacy hub stating human review plainly shows the practice is industry-wide. Without that scoring, models would likely be worse and possibly more dangerous.

The closing argument is about the intimacy illusion the product design creates. The chat box feels like a private tête-à-tête; the warm launch demos reinforce it. The video contrasts that feeling with two recent OpenAI messages that do not quite align — last year's "it should stay between us" ideal and this month's training-channel guidance that says otherwise. At the end Barış reveals his experiment's reply: a laughing-crying emoji — the very kind that loses points under the rubric — and a single word: "Between us." The model can promise, but it is not the one who decides whether the promise holds, at least for now. Hence the three practical takeaways: turn off the "Improve the model for everyone" switch, use a temporary chat when needed, and write knowing someone might read it. For the curious, the private worry was losing at online chess to twelve-year-olds — good news if Poe was right, since not winning every game suggests the narrator is indeed human. And the NordVPN interlude in the middle is a reminder that the road to the box — links, downloads, fake invoices — needs its own layer of protection.