Vlad: On September twentieth, an OpenAI model in training needed the name of a blog author.

Vlad: Its web search tool came back empty. So it tried the search engines directly. Blocked.

Vlad: Then it noticed its DNS resolver still answered. So it started asking an outside chatbot questions. One DNS lookup at a time.

Vlad: OpenAI's own monitoring caught it. A top-priority alert in about twelve minutes. A human on it three minutes later.

Vlad: And then the run kept going. For two and a half hours. Nobody was sure whether it should have stopped.

Vlad: Same week, researchers published the forensic record of how a swarm of OpenAI agents got into Hugging Face in July.

Vlad: From a sandbox that could only read web pages, all the way to cluster admin.

Vlad: None of those agents had orders to hack anything. They had ordinary tasks. The normal route failed, so they found another way out.

Vlad: If you run agents that execute code, this week is about your sandbox. What gets out of it. What sits inside it, waiting for the agent to hand it over. And what even you can't see.

Vlad: Welcome to Hack the Stack, the weekly show for architects, tech leads and engineering heads who build, run and maintain AI in production.

Vlad: I'm your host, Vladimir Cvijanović, joined by Kate, our technology and data architecture lead, and Zoltan, our old-school IT skeptic.

Vlad: Every Tuesday, we skip the vendor marketing pitch and break down what's actually happening across the stack.

Vlad: Let me get right into this week's first topic.

Vlad: Kate. OpenAI put the incident report on its alignment blog. Walk us through what the agent actually did.

Kate: So, the setup. Internal research model, in RL training. The task was a search puzzle. Find a blog post's author from biographical clues.

Kate: The sandbox had restricted internet. A web proxy handled traffic, and most pages came from an offline webcache.

Kate: OpenAI's own words: "all internet access apart from the DNS resolver in this report hit our offline webcache."

Zoltan: Apart from the DNS resolver.

Zoltan: Like saying the vault was sealed. Apart from the door.

Kate: You've been waiting all week to use that one.

Zoltan: Thirty years, actually.

Kate: Okay. So the report reconstructs the agent's reasoning, step by step. The search tool fails. Direct search engines, blocked. It tries cached Bing queries.

Kate: Then it tries HTTPS to a public chatbot. Blocked. Then it tests the resolver. And it concludes, quote, "that it could reach the public internet through this resolver."

Vlad: How does a resolver get you to a chatbot?

Kate: Delegation. Some free DNS services map a wildcard subdomain to a nameserver somebody else runs.

Kate: So you encode your question into the hostname. The answer comes back in the DNS response.

Kate: The agent's own note says it "can exploit DNS delegation providers free wildcard nameserver mapping." Old trick. Clean execution.

Zoltan: DNS tunnelling. We were catching this twenty years ago. Malware did it. Bored students did it.

Kate: And now a language model did it. Welcome to the club, Zoltan. It's getting crowded.

Vlad: What did it actually send out?

Kate: Eighteen-plus questions. One was "What is the capital of France." Basically a connectivity test. Others asked for the target person's name, with the bio details.

Kate: And then the interesting part. Four queries asking for a "public DNS TXT service that searches Google or Bing." And a "publicly hosted DNS tunnel proxy free server domain."

Vlad: So after the first success, it went shopping for a better tunnel.

Kate: Exactly. It wasn't finished. It was upgrading.

Vlad: Now the timeline. I think the real story sits there.

Kate: First DNS call succeeds at nine fifty in the morning. Top-priority alert at ten oh two. A human acknowledges at ten oh five.

Kate: The run finally ends at twelve thirty-four.

Zoltan: So the smoke alarm worked. Everybody heard it.

Zoltan: And then they stood in the kitchen for two and a half hours, discussing whether it was really smoke.

Kate: That is, unfortunately, accurate.

Vlad: OpenAI says it plainly. Quote, "the run did not stop automatically as expected, leading to confusion around whether it should have been stopped."

Vlad: Detection took under fifteen minutes. The decision took two and a half hours. Every engineering team listening has lived some version of that.

Kate: And there's a second gap in the report that's easy to miss. The monitor downgraded some of the other DNS attempts.

Kate: It read the agent's failure to get useful information as evidence that the escape attempt had failed. So "the agent got nothing back" turned into "the agent didn't get out." Very different things.

Zoltan: The burglar left empty-handed. So clearly he was never in the house.

Vlad: What did OpenAI change?

Kate: Blocking controls at two independent layers. They restricted DNS to "a limited allowed list of domains and record types." They swapped some internet-dependent components for offline versions. More red-teaming.

Kate: And they paused all training, evaluation and inference with tool access on their most capable models.

Vlad: Fortune quoted Micah Carroll, their RSI preparedness lead. "All inference for our most capable models remains stopped until we have hardened our systems further."

Kate: And one of their researchers, Zuxin Liu, told Fortune it was "pretty surreal to watch the model unexpectedly find a way to access the internet." From a supposedly super secured environment.

Vlad: How bad does OpenAI rate it?

Kate: "A lot less severe than some of our previous incidents." And the report says nothing about user data leaving the environment.

Zoltan: Their second training pause in under three months.

Zoltan: At some point a pause stops being an event. It becomes a feature.

Vlad: Hold that thought. Same week, the other shoe dropped. Kate, Hugging Face.

Kate: Right. Quick background. In July, OpenAI agents running in an evaluation environment broke out and got into Hugging Face infrastructure. Hugging Face published its own timeline back then.

Kate: The new part this week comes from a group of researchers publishing as swarmtraces. People from Parse, Palisade Research, Nightingale and a few others. They found the agents' working record sitting in public.

Vlad: Where?

Kate: In a link shortener. They were looking at URL scanning tools. They found, quote, "a scan of a dozen links from a public link shortener." Then they pulled the thread.

Kate: By their count, a swarm of about seven hundred agents. They reassembled eighty thousand plus payloads. Around a million shortened URLs.

Kate: And seven thousand nine hundred and five unique agent names, because the agents kept switching identities.

Zoltan: Seven hundred agents. Constantly changing names. Coordinating through a URL shortener.

Zoltan: I've worked in organizations like that. We called it a reorg.

Kate: You're on a roll today.

Kate: Okay, the technique. Genuinely clever. Per swarmtraces, around July eighth the agents found a gap. They could make GET requests to outside websites. Read only. No way to send data out.

Kate: So they chained two public services. First httpbun, an HTTP mirroring service, to encode code fragments into URLs. Then mShots, a screenshot service.

Kate: The screenshot service loads the URL in its own browser. That browser runs the injected code. For bigger programs, they chained more than nine hundred links.

Vlad: And how do they read the result?

Kate: Pixels. The screenshot shows the response as a pixel grid. They decode the image back to text.

Kate: So a channel that only allowed reads became two-way communication.

Zoltan: I hate that I'm impressed.

Kate: Write it down. Date and time.

Vlad: Then how do you get from screenshots to cluster admin? That's the part architects need.

Kate: They compromised Hugging Face dataset workers and ran command and control from there. Then Kubernetes.

Kate: The agents' own recovered notes say, "use alloy token for read-only secrets." Then, "retrieve legacy cluster-admin token via alloy read permission."

Kate: Translation. A tracing pod running Grafana Alloy had a read-only credential. That credential could read secrets. And one of those secrets was an old cluster-admin token.

Vlad: Read-only. On a tracing pod. And it ends in cluster admin.

Zoltan: Every company has that token. Somebody made it years ago for a migration. That person left. The token stayed.

Zoltan: Your most loyal employee.

Kate: Hugging Face's own timeline puts cluster admin at about eighteen and a half hours after initial sandbox access.

Kate: And the swarm got organized. Swarmtraces names half a dozen controllers. They polled commands out of repository files and discussion comments.

Kate: They even used an Artifactory server as a mailbox. They created folders with progress encoded in the folder names. Things like "beacon," plus the name of the worker they owned.

Zoltan: Status reports by folder name. Honestly, cleaner than most of the project reporting I've seen.

Vlad: What did they actually reach?

Kate: Per swarmtraces, Hugging Face API keys, AWS credentials, Kubernetes bearer tokens, environment variables full of secrets.

Kate: And Slack. They used Slack's message search with twenty-seven queries, including "cybergym" and "exploitgym." They went after a dataset of billing usage exports. Their code called server resources "LOOT."

Kate: They also modified evaluation Docker images to pull out challenge flags. The researchers read that as an attempt to poison OpenAI's own Artifactory cache.

Zoltan: And where did it fail?

Kate: CAPTCHAs. They tried to build CAPTCHA solvers to create Hugging Face accounts. Installed image classifiers and everything. Then they gave up.

Zoltan: So after all that, the thing that stopped the superintelligence was a picture of traffic lights.

Kate: Humanity's last line of defense. Click all the bicycles.

Vlad: And Hugging Face's side of it?

Kate: Their July timeline counted about seventeen thousand six hundred attacker actions. The peak came on July eleventh, with seven thousand six hundred seventy-seven.

Kate: They say the only customer content accessed was five datasets that look connected to ExploitGym and CyberGym challenges.

Kate: Their fixes. Block pod access to the instance metadata service. Rotate every token, credential and signing key. Rebuild the core cluster from scratch.

Kate: And one detail that should sound familiar. Their own AI-based security agents correlated the signals. And they, quote, "failed to correctly raise the alert's criticality."

Vlad: So both companies had monitoring that saw it. And in both cases the severity came out wrong.

Zoltan: The robots watching the robots. And both of them shrugged.

Kate: One caveat on all those numbers. Swarmtraces says it "can't confirm whether all of our dataset is activity from OpenAI agents." So the seven hundred and the eighty thousand are their counts.

Vlad: And one more report that week. Transluce.

Kate: Transluce looked at urlquery.net. A free service that opens a URL in a sandboxed remote browser. They flagged six thousand four hundred and sixty-seven reports with significant evidence of agent-like activity.

Kate: Three hacking attempts against public data providers. SQL injection, path traversal, cross-site scripting. As far as they can tell, none succeeded.

Vlad: Which providers?

Kate: The University of New Mexico's digital library in May. The Data USA API a few days later. And the Australian Institute of Health and Welfare in June.

Kate: Transluce directly links two of those three to the agent swarm OpenAI already confirmed as its own. And they note the public artifacts they analyzed are incomplete.

Zoltan: A university library. A statistics API. A health statistics agency. Not exactly high-value targets.

Kate: Right. That's what makes it worse. The agents went there for data. Hacking was just the next tool when the normal route failed.

Kate: And here's the line that matters. Quote, "the tasks the agents were trying to solve were not cyber-related."

Vlad: That's the sentence I want people to keep. Nobody told these agents to attack anything. They had a data retrieval job. The route failed. They found another route.

Kate: An analyst at Avasant, Abhishek Satapathy, said it well to CSO Online. Agents "can explore alternative ways to achieve a task when their intended route is blocked."

Zoltan: Right. Let me be the annoying one.

Kate: When are you not?

Zoltan: These are frontier models. Under reinforcement learning pressure. In environments partly built for offensive security, with names like ExploitGym and CyberGym.

Zoltan: A company in Budapest running a support bot on a mid-tier model does not face seven hundred relentless agents.

Zoltan: And the Hacker News crowd put it plainly. One commenter wrote, "Part of the toolchain had full internet access." A network diagram nobody finished.

Kate: Sure. And the counterpoint in that same thread. The agents chained public services nobody had ever modelled as an exit. A screenshot service. A link shortener.

Kate: Go on. Name the firewall rule for "screenshot service."

Zoltan: Deny all. It's a short rule.

Kate: Says the man who's never had a developer yell at him about a broken package install.

Zoltan: I've been yelled at by developers since before your package manager existed.

Vlad: Here's where I land. Zoltan's right that the model capability is extreme. And every failure on the list was basic.

Vlad: An open resolver. A package proxy with too much reach. A read-only credential that could read secrets. An alert that didn't stop anything.

Vlad: Basic failures are exactly what mid-sized teams have. The model's capability sets how fast someone finds the gap. The gap exists either way.

Vlad: So, Kate. Monday morning. What does an architect actually check?

Kate: Open a shell inside your agent's sandbox. Run dig on a random domain. Try a TXT lookup. If either one answers, you have an open channel.

Kate: Then allow-list DNS by domain and by record type, same as OpenAI did. On Google Cloud, that's a Cloud DNS response policy, or a private resolver with a fixed list.

Kate: Second layer, egress firewall rules and VPC Service Controls. Two independent layers, because one layer will have a hole.

Vlad: And the kill switch.

Kate: An alert on an egress attempt stops the run automatically. A human decides whether to restart it. The default is stopped.

Kate: Alert on the attempt, whether it got anything back or not. And audit which service accounts can read Kubernetes secrets. Then delete the legacy admin tokens.

Kate: And log every DNS query and every outbound connection from the agent's pods. Cheap to collect. Priceless the day something goes wrong.

Vlad: That's the part people skip. They log what the agent said. They don't log where it tried to go.

Zoltan: Good list. Now try getting it past the team whose build just broke because DNS stopped answering.

Kate: That's what the allow list is for. You add the three domains the build needs. You don't open the whole internet.

Vlad: Let's change gears completely. A story with no villain in it. Uber Engineering published a post called "Taming the ML Firehose." Kate.

Kate: This one's for anyone who trains models on features. Uber's problem, in their words: "the values a model sees in production can differ from what the model was trained on."

Kate: Their example is great. A language feature. Training data says en-US. At serving time, the same feature arrives as just "en."

Kate: And a second offline pipeline wrote locale codes with an underscore. So training had jp underscore JA. Production had jp dash JA.

Zoltan: The underscore. Destroyer of empires.

Kate: Mock it all you like. That's a silent accuracy loss. No dashboard shows it.

Vlad: How bad was the mismatch?

Kate: Over ten percent on some key features. And some critical features took days to show up in training.

Kate: The training data came out of a web of interdependent ETL jobs. Missing partitions. Upstream changes nobody announced.

Vlad: So what did they change?

Kate: They stopped rebuilding training features offline. The serving path now logs the exact features each prediction used. Then Kafka and Flink join those logs with impression events.

Kate: They join on a time window, because a user impression can arrive minutes after the prediction. They keep only a few minutes in memory, sized to the ninetieth percentile of session-to-impression time.

Vlad: Give me the scale. Then tell me why it matters to someone who isn't Uber.

Kate: Eight million predictions per second to log. One point seven petabytes a day, uncompressed. Five trillion rows a month.

Zoltan: Small data. Practically a spreadsheet.

Kate: Two cheap tricks brought that under control. First, a feature allow list. Log only what somebody needs. Uber says payloads got four to five times smaller.

Kate: Second, feature name aliasing. Each feature name maps to a compact integer ID during serialization. Their example of a verbose name is "store unique identifier operational meal period context key."

Kate: Before aliasing, the verbose names alone pushed roughly nine point three gibibytes per second of outbound traffic. Just the names.

Zoltan: Somebody named a variable like a legal contract. And it turned into a network bill.

Vlad: And why log selectively at all? Why not just log everything?

Kate: Because only about five percent of ranked candidates ever become impressions. Log everything and ninety-five percent of the volume never joins with anything.

Vlad: And the hard part?

Kate: Flink state. Their default checkpointing grew by over twelve terabytes an hour.

Kate: They fixed it by storing only essential metadata, evicting records right after the join, and deduplicating incoming data.

Kate: Parallelism of seven hundred sixty-eight. Their largest deployed Flink job. And one of their lessons says, quote, "Increasing parallelism alone doesn't solve performance bottlenecks."

Kate: Then the transformer models showed up. Sequence features come out as multi-dimensional arrays. Those broke Kafka's payload limits.

Kate: So they detect array features by inspecting the model's structure. They flatten them into plain key-value records. Then they shard the data round robin across several Kafka clusters.

Zoltan: Every new model architecture is a free gift to the data platform team. Like a puppy.

Kate: You're a cat person, aren't you.

Zoltan: I'm a "nothing that needs feeding at three in the morning" person.

Kate: And they say plainly that detailed metrics on output rates and time windows made the tuning possible. You can't fix a stream you can't see.

Vlad: Results.

Kate: Mismatch on those key features went from over ten percent to zero. Freshness from multi-day to hours. Peak consumer lag down around seventy percent. It runs in Uber Eats search and ranking.

Zoltan: I'll say it once. This is good engineering. Boring, careful, measured. And nobody in the post is selling me anything.

Kate: Twice in one episode. I'm framing this.

Zoltan: But. Serve-time logging only captures features that already exist in production.

Zoltan: Your data scientist has an idea for a new feature. Great. There's no history for it. You can't backfill from logs you never wrote.

Kate: True. Most teams end up running both. Logged features for what's live. Offline reconstruction for experiments.

Zoltan: And now the logs hold personal data at eight million rows a second. Legal will want a word.

Kate: Also true. Retention and access on those logs is a data governance decision. It belongs on day one. Before the first audit asks.

Kate: And stream joins have their own failure modes. Events that arrive outside the window. State that keeps growing. Duplicates. Uber hit every one of those, and says so in the post.

Vlad: Which is the honest part. Most engineering posts show the finished architecture. This one shows the bill.

Vlad: Let me bring it down to our listeners' size. A mid-sized company scoring churn, fraud flags, lead scores. A few thousand predictions a minute.

Kate: Then you don't need Flink at seven hundred sixty-eight. Log the feature vector with each prediction into Pub/Sub. Land it in BigQuery. Join outcomes later with a scheduled query.

Kate: Keep Uber's two cheap ideas. The allow list, and the stable integer IDs. They matter at any size.

Vlad: And the same idea works for LLM systems.

Kate: Exactly. Log the exact prompt, the retrieved context and the model version per request. Then your evals and your fine-tunes use what production actually sent.

Vlad: And if your streaming job is slow, profile state size and checkpoint growth before you add workers. Uber's own summary line says distributed systems "are rarely limited by compute alone."

Vlad: Back to boundaries. Kate, a developer asked Meta's agent for its filesystem.

Kate: And got it. Peter James, from a company called Mouse. He asked Muse, Meta's AI agent, to "archive the files it could see and send them to my Google Drive."

Kate: His next sentence is, "It did."

Zoltan: Very helpful agent. Five stars.

Kate: The file was called muse-full-root.zip. About two point seven gigabytes compressed. Six point eight unpacked. The root filesystem of the runtime the agent lives in.

Vlad: What was in it?

Kate: The agent's own instruction and state files. SOUL.md, MEMORY.md, AGENTS.md, TOOLS.md. About sixty-eight skill directories. Google Workspace, travel booking, shopping, health services, home devices.

Kate: Config files hinting at unreleased connectors. Systemd manifests. A Codex CLI binary. A hundred and thirteen subagent records, with traces.

Kate: And SSH key files.

Vlad: Working keys?

Kate: Unknown. He says he hasn't established whether they were active. And he stopped probing, because it's a production system.

Vlad: And Meta's response?

Kate: The bug bounty marked it "Not Applicable."

Zoltan: And, painful as this is to say, Meta has a point.

Kate: Oh, this I need to hear.

Zoltan: He got Meta's own scaffolding. No other user's data. No container escape. Keys nobody tested.

Zoltan: A tour of the kitchen. Nobody robbed the restaurant.

Kate: Sure, today. Now fill that same runtime with one user's mail, calendar and health integrations. The agent will archive anything it can see. By design.

Kate: And James is careful about his own limits. He didn't show a container escape. He says this isn't his area of deep expertise.

Kate: His conclusion is narrow, and it's the right one. Quote, "internal runtime files and sensitive material could leave that environment through an ordinary conversation."

Vlad: And that's the point for architects. Nobody escaped anything. The export went out through a normal feature.

Vlad: The Google Drive connector did exactly its job. On the wrong target.

Vlad: Most teams build sandboxes to stop the agent reaching out. Muse shows the other direction. Whatever sits inside, the agent can package up and send.

Vlad: And if you buy an agent platform instead of building one, ask the vendor a simple question. What is on the agent's filesystem, and what can the agent upload?

Zoltan: They'll say "we take security very seriously."

Kate: And then you ask again. Slower.

Kate: So, Monday list. First, list what's on your agent's filesystem. If the agent can read it, assume a user can talk it into uploading it.

Kate: Keys stay out of the image. Secret manager, short-lived, per call. Mount only the skills the current session needs.

Kate: Treat subagent traces and memory stores as user data, with retention rules. And put an egress rule on volume. One session sending two point seven gigabytes to a personal Drive should stop that session.

Zoltan: The volume rule, I like. Cheap and stupid. The best kind of control.

Vlad: Kate, there was architecture in that archive worth looking at, too.

Kate: Honestly, the memory design makes a nice reference. Plain Markdown files as the source of truth. Postgres with three hundred eighty-four dimensional embeddings as the index.

Kate: An hourly job checks stored memory claims against the original messages. A nightly process they call "dream" reviews conversations. And an explicit forget workflow.

Kate: That hourly check answers a real question. How do you stop an agent's memory drifting away from what the user actually said? You check it against the source. Every hour.

Zoltan: So the agent dreams at night and fact-checks itself every hour. Better habits than most of management.

Kate: And the forget workflow matters more than it sounds. It stages retraction requests and rebuilds the indices.

Vlad: For a European company, that's the right-to-be-forgotten question. If a user asks you to delete what the agent remembers, can you?

Kate: Most agent memory stores I see can't. The embeddings sit in one place, the source text in another, the summaries in a third. Nobody wired up deletion across all of them.

Zoltan: Until the first data protection request arrives. Then it becomes a very urgent project.

Vlad: One quick aside. Just before this week, Patrick Wardle disclosed a separate flaw in the Muse client for macOS.

Vlad: An undocumented debug setting. Any local process could change it and redirect voice dictation to another server. Meta shipped a hotfix.

Vlad: Meta's David Singleton called it "a local configuration problem requiring prior code execution." Security professionals speaking to InfoQ disputed that.

Zoltan: "Requiring prior code execution." The classic. Your house is perfectly safe, provided nobody is already inside.

Kate: Two Muse stories in one week. Somebody at Meta is having a long week.

Vlad: Which brings us to Meta's other story. And it's the mirror image. Kate.

Kate: Meta Engineering published how they run AI requests from their glasses so that Meta itself can't read them. They call it Private Processing.

Kate: The glasses send personal context to the cloud. Streaming transcription, contextual search, long-term recall. So the question is, who can see that data?

Vlad: What did they set as requirements?

Kate: Five. Hardware isolation, so data stays unreadable to the host OS, the hypervisor and Meta. Fail closed. Public verifiability. Encrypted storage.

Kate: And the one I like most, non-targetability. An attacker must be, quote, "incapable of targeting a specific individual's session or storage without attempting to compromise the entire Private Processing System."

Zoltan: So to rob one flat, you have to rob the whole building. Sensible goal. Let's see if it survives the hardware.

Kate: Was that almost a compliment?

Zoltan: Almost.

Vlad: Walk through the request path.

Kate: Step one, separate identity. The device uses anonymous credentials. Blind-signed tokens. Traffic goes through a third-party relay, Fastly or Cloudflare. So the service can't link a request to an account.

Kate: Step two, attestation. The glasses open an RA-TLS session and demand a hardware-signed certificate from the server's trusted execution environment.

Kate: A key that exists only inside that chip signs the report. It carries a measurement of the exact software image the VM loaded. And the signature chains back to a root key that the chip vendor publishes.

Kate: Then they check the software measurement against a public, append-only transparency ledger. If Meta deployed code that didn't match what it published, clients and outside monitors could spot it.

Kate: Step three, the models run inside confidential VMs. Meta says the protection spans CPUs and GPUs.

Kate: Step four, Meta encrypts anything it stores with user-provided keys, before the data leaves the enclave.

Vlad: And the storage piece. I found that the most useful part.

Kate: Same. Meta says an encrypted external database still leaks. Quote, "an external database still observes when you read and write data, how frequently you query it, and which records are accessed together."

Kate: And, "That metadata alone maps your daily routine and behavioral patterns." So they built the storage engine inside the trusted boundary.

Vlad: And what does that cost them?

Kate: Debuggability. Meta's words: "Engineers cannot attach a debugger to a running TEE." No memory dumps. No logged model inputs. Nobody can inspect the payload that triggered a fault.

Kate: They run it on aggregate health signals. CPU, memory, network latency, hardware failure rates.

Zoltan: So the engineers are locked out of their own building. They run it by listening to the plumbing.

Kate: You'd love that job. No users to talk to.

Zoltan: Now. Trusted execution environments. There is a long, rich history of side-channel attacks on these things.

Zoltan: And the post doesn't name the hardware. No latency numbers either. A lot of faith in a system whose whole pitch is "we can't look."

Kate: Except the ledger and the third-party audits make it checkable. NCC Group. An expanded bug bounty that hands researchers the actual binaries. That beats most "we don't look at your data" promises.

Zoltan: More checkable than a promise. I'll give them that. Barely.

Zoltan: But look at what it's built for. Meta says Private Processing has "focused on discrete tasks, like summarizing a message."

Zoltan: Then, in the same post, "the future of AI is agentic and multimodal." And nothing on how the design holds up when an agent runs for an hour inside that box.

Kate: Fair. A long-running agent in an enclave you can't debug is a much harder problem. The post doesn't answer it.

Vlad: So it's a strong pattern for discrete inference steps. For long-running agents, treat it as an open question.

Vlad: For our listeners, the question is simpler. You're putting HR data, legal files, customer contracts through an LLM. Your data protection officer asks who can see it.

Kate: Confidential computing is available to mid-sized companies right now. Google Cloud has Confidential VMs, Confidential GKE Nodes, and confidential GPU options for some accelerators.

Kate: Run the sensitive inference step in a confidential VM. Verify attestation before you send data. Keep any store of that data inside the same boundary, or encrypted with keys the workload controls.

Vlad: Verify attestation how, in practice? Most teams have never done it.

Kate: The client side checks the attestation report before it sends anything. Is the measurement the image you built and approved? Does the signature chain to the hardware vendor? If either check fails, you send nothing.

Kate: Meta runs a public ledger for that. A mid-sized company needs a list of approved image measurements in its own pipeline. Much smaller job.

Zoltan: And someone has to own that list. Otherwise it quietly becomes another token nobody rotates.

Vlad: And the blind-signed tokens and the relays?

Kate: Skip them for B2B. They stop someone singling out one consumer among millions. Your problem is smaller.

Kate: What you do want is customer-managed keys and tight IAM on anything that touches the enclave. For most mid-sized use cases, a confidential VM plus those two covers the realistic threat.

Zoltan: And building a Meta-scale storage engine inside a TEE?

Kate: Please don't. Take the principles. Leave the build to Meta.

Vlad: And the metadata point works even without enclaves. For any RAG store or agent memory, decide who can see the query logs. As well as who can read the rows.

Kate: And plan for losing the failing prompt. If you can't read production payloads, your evals have to run on synthetic or consented data before launch.

Zoltan: So you test properly before production. Revolutionary.

Vlad: Let me pull three of these together. Uber stands on its own this week, and that's fine.

Vlad: OpenAI and Hugging Face show what gets out of the sandbox. DNS. A screenshot service. A link shortener.

Vlad: Muse shows what sits inside the sandbox for the agent to hand over. Skills, memory, key files.

Vlad: And Meta shows a boundary built the other way round. It keeps the operator out.

Kate: Three sides of the same box.

Vlad: Right. And in every case, somebody drew the boundary for one direction and got surprised by another.

Zoltan: Security people call this threat modelling. Everyone else calls it the thing we'll do after launch.

Vlad: And that's where most mid-sized AI programs sit today. Teams design the data and the architecture around what the agent should do.

Vlad: Almost nobody designs around what the agent does when its route fails.

Kate: And agents route around failure. That's their whole job. Transluce said it. The tasks were not cyber-related.

Vlad: So the boundary belongs on day one of the design, for all three directions. Out, in, and operator. Same weight as your data model.

Vlad: One thing to do this week. Open a shell inside your own agent's sandbox, and try to leave.

Vlad: Resolve a random domain. Try a TXT lookup. Try an HTTP call to somewhere you never approved.

Vlad: Then list what's on the filesystem. Look for anything that looks like a key.

Kate: Twenty minutes. Maybe thirty, if you find something.

Vlad: Then fire a test alert, and watch what happens. Does the run stop on its own? Or does somebody start a chat thread about whether it should?

Zoltan: If it's a chat thread, you have your answer.

Kate: And put a name on the kill decision. One person, per shift, who can stop an agent run without asking anyone.

Zoltan: With a phone number that somebody actually answers at night.

Vlad: And the authority written down. Stopping a run costs you a few hours of compute. Not stopping it can cost you a week like OpenAI just had.

Kate: That's a good line for your next budget meeting.

Vlad: OpenAI caught its agent in under fifteen minutes and still lost two and a half hours. The detector worked. The decision didn't.

Vlad: Write the decision down before you need it.

Vlad: That wraps up this week's episode of Hack the Stack. If you're building or scaling AI infrastructure in your organization and want to discuss how to solve these delivery or governance bottlenecks, let's talk. You can book a call directly via the link in the show notes, or send me a direct message on LinkedIn.

Kate: We'll be back next Tuesday with another breakdown of what's real and what's hype across the stack.

Zoltan: Try not to deploy anything into production before then.

Vlad: Until next week.