Deep Hedging: Teaching AI Agents to Hedge Optimally in Discrete Markets

Why are humans still making the decisions? With news flows, spot prices, volatility gauges and even compressed histories like path signatures at hand, why aren't we just handing a state vector to a computational agent and letting it trade for us? Paolucci frames this as the frontier question. In theory, an agent fed with richer information should beat us; in practice, reinforcement learning's hunger for data and the finicky nature of approximating policy functions keep that dream in check.

What Risk-Neutral Pricing Means on the Desk

Without risk-neutral pricing, deep hedging has no anchor. Paolucci starts with the simplest question: what is the fair mid-price of a derivative? Without a crystal ball, the best we can do is an expectation — mathematically the best guess in mean-squared-error sense. In the Black-Scholes idealization the asset dynamics are cast as geometric Brownian motion and the market is taken as complete. With completeness in hand, a delta hedge can neutralize that randomness, which lets us construct the replicating portfolio whose present value defines the contract's price. That value is today's fair price, not a speculation about a 7% equity risk premium. On average, with no risk left, we must earn the risk-free rate; hence the discounted risk-neutral expected payoff.

To make it concrete, he invokes the Feynman-Kac equivalence. The Black-Scholes partial differential equation and the stochastic conditional expectation must coincide. He first leans on Monte Carlo intuition: simulate thousands of underlying paths, compute each option payoff, push it to a list and average. In the animation that average on the right converges to the red dashed theoretical Black-Scholes price. This is the stochastic proxy for fair value — not the delta-hedging argument itself, but the intuitive step that makes the latter verifiable.

Then comes the delta-hedge validation. The same underlying simulation on the left, the equity curve on the right. Buy or sell the contract at the theoretical fair price and continuously delta-hedge through its life: as more hedged paths accumulate, the average P&L converges to zero. The drift of paths centres around the risk-free rate, the hedging error vanishes, and the replicating portfolio argument becomes visual. At mid, hedged correctly, expected P&L is zero — the sar yellow line's message.

If expected P&L is zero, why would anyone provide liquidity? Here the lecture turns to market microstructure. The price you see on Yahoo Finance or a Bloomberg Terminal — say Nvidia at $220 or the SPY print — is a mid. A desk cannot live on zero expectation. Referencing his prior video on making a market on a one-run, Paolucci makes the point: without a spread, the desk makes nothing. The fix is two-way quoting — buy at the bid, sell at the ask; knowing the fair price makes systematically buying low and selling high feasible.

Edge, Casino, and Two Elephants

To explain edge, he goes to the casino. Roulette gives players a negative edge and the house a positive one. Absent an absorbing ruin state at zero wealth, the house is guaranteed to accumulate infinite wealth with enough plays — the government's tax dream. On the left, player wealth paths all bleed to zero despite variance; on the right, the house soaks up the entire pie. It's zero-sum: sum the gains and losses and you get zero, but the house takes it all. He deliberately sidesteps arithmetic versus geometric compounding, Kelly, and ergodicity — arithmetic keeps the illustration clean.

A market-making desk is the same casino logic. The risk-neutral construction gives the theoretical mid, the spread injects an artificial statistical edge. If you can buy low and sell high around a known fair value and hedge effectively, you accumulate that edge over time just as the house extracts wealth from players. Real risks remain — model risk, inventory risk, sequence-of-returns risk — but the skeleton is this: hedge to zero, collect the spread.

Two elephants fill the room: discretization and model risk. Continuous hedging is ideal in theory — easy to collect the spread — but impossible in practice because it would accumulate infinite frictions through transaction costs. Hence we must hedge discretely, say every 0.02 in the one-year simulation. The chart shows the consequence: lower expectation and greater variance versus the near-continuous case; hedging error and frictions bite. On the model side, no one knows the true dynamics. The real world is not geometric Brownian motion; volatility is rougher than Heston assumes, non-Markovian, path-dependent. Paolucci flags path signatures and rough volatility literature as attempts to shrink that gap, but model risk is unavoidable on both buy and sell sides.

When to Hedge and How to Learn It

The better question is when to hedge, not how much. Fixed-time re-hedging is not optimal: imagine a massive gap down, you re-hedge, then a massive gap up — you didn't need to, but you couldn't know. Gamma bands, no-trade regions, sticky and shadow Greeks are practical guide rails, not prescriptions. Paolucci's student anecdote is telling: asking his professor ‘when is it best to delta-hedge?’ yielded a vague answer that pushed him toward generative methods and eventually toward Hans Buehler's approach. Should new information dissemination enter the state for re-hedging, or should we stick to pure statistical dynamics? That is the whole debate, and any choice assumes model risk.

Every choice assumes model risk, even handing the hedge to an AI agent. Maximizing expected P&L alone would be psychotic — otherwise we would all go triple-levered long on Dogecoin or Pepecoins for their higher expectation. Reward must be risk-constrained: entropic risk measures, expected shortfall, concave utilities that penalize tail exposure and inventory. Paolucci stresses that P&L is not everything; tail risk is.

Here the reinforcement literature scaffold appears. Markov decision processes: a sequence of rewards from states and actions, policy functions mapping state to action iteratively, maximizing expected reward under stochastic transitions. He name-checks MDPs, SARSA, Q-learning, Bellman equations, deterministic versus quasi-deterministic settings, game trees, and Monte-Carlo tree search as foundations to explore separately — too deep for this video but essential. The biological parallel is striking: you are presented a state, you pick an action, you get a new state; you are a composite policy function, except biology feeds you a continuous sensory vector while AI must be fed a discrete, enumerated state — a friction with computational tractability.

To ground it, a gridworld animation: a +10 star and a -10 skull. The agent walks, touching both, while a narrow neural network tries to learn the right mapping from state features to actions. By episode 90-100 it has learned to avoid the skull and chase the star, weights adjusting, expected reward shaping behavior. Paolucci then detours to signal versus noise: a good decision can yield a bad outcome and vice versa in a random environment. He cites ‘Thinking in Bets’ and poker theory — discerning signal from noise after one hand is non-trivial, otherwise everyone would be a multi-time World Poker Tour champion.

Deep hedging is defined: the Hans Buehler–led framework (Buehler, Gonon, Teichmann, Wood, JP Morgan / ETH Zurich, 2018-2019) that replaces arbitrary hedging rules with a learned policy. State is not just price. It is inventory, current positions, underlying levels, implied volatility surfaces for multi-asset books, path-dependent features, realized volatility, elapsed time — anything that explains cross-sectional variation, even tokenized news. Neural networks, as universal approximators, learn the map from state vector to hedge action. Model risk persists: use a classical simulator like stochastic volatility or jump-diffusion, or a data-driven generator like VAEs or GANs to simulate tail events never seen empirically (just as no one trained on LLM-era data before LLMs changed the world); there is no escape from assuming a model and confronting data scarcity.

Training is shown on a single stochastic market path: underlying on the left, the European option position in the middle, and two hedges on the right — the learned deep hedge versus the classical delta hedge rule. As more trades are simulated, the tail risk of the deep hedge shrinks relative to the fixed delta-hedge distribution. That's the whole idea: more effective hedging on average in-model, with out-of-sample performance as the judge. The algorithm's structure is model-free, generalizes across hedging instruments (including liquid derivatives), scales with the number of hedging instruments rather than portfolio size, and can be implemented efficiently with modern ML tooling like TensorFlow. Paolucci jokes a dashboard this good will stay proprietary, then closes with the invitation to like, comment, and request deeper dives into MDP foundations.