18. Rough Volatility for Option Pricing

In this project we move from stochastic volatility to rough volatility. The goal is to understand why the volatility process can look much more irregular than a Brownian path, how we can estimate that roughness from realized variance and option skew, and how rough models change pricing, calibration, and hedge sensitivities.

We focus on five main layers:

• roughness as a statistical property of log realized variance, measured with fractional Brownian motion and moment scaling,
• rough-kernel forecasting, compared with HAR and GARCH-style variance forecasts,
• short-maturity skew power laws, where options reveal roughness through the shape of the implied-volatility surface,
• rough Bergomi and rough Heston, two different ways to put rough volatility into option pricing,
• model comparison and secondary BTC implementation, where we test whether the same logic survives in a much noisier crypto option market.

We won’t re-explain option payoffs, implied volatility, Greeks, local volatility, Heston, COS pricing, or American-option numerics from scratch. Those are already covered in Project 4, Project 8, Project 11, Project 13, and Project 14. Here the new story is the regularity of volatility paths and what happens when pricing models respect that regularity.

The main datasets are reused. To reproduce them, check the relevant folders in the data reproducibility layer:

• underlying_yfinance for SPX and other underlying price histories,
• spx_optionsdx for SPX option chains,
• us_treasury_yields for the risk-free curve,
• btc_yfinance for BTC spot history,
• btc_optionsdx for BTC/Deribit option chains.

The empirical literature gives us a starting point for the models we use here. The rough-volatility idea became central after empirical work showing that log volatility behaves close to fractional Brownian motion with \(H < 1/2\), often much lower than the Brownian value \(H = 1/2\). Option-price evidence then connected this to short-maturity implied-volatility skew. Rough Bergomi and rough Heston are two model families that translate this empirical fact into pricing engines.

1) Setup, Reused Data, and the New Object

We use the option, surface, Fourier, and volatility-forecasting tools from the earlier projects. Alongside them we need the numerical methods for rough realized-variance diagnostics, rough-kernel forecasts, rBergomi simulation, rough-Heston calibration, and fractional Riccati stability.

The SPX side uses three inputs:

• SPX spot history, used to build realized variance and log-variance proxies,
• SPX option chains, used to reconstruct implied-volatility surfaces and short-end skew,
• Treasury yields, used to attach discounting and forward/carry estimates to option prices.

Our new focus is the regularity of the volatility path alongside the price \(S_t\) and implied volatility \(\sigma_{\text{imp}}(K,T)\). We want to understand how jagged the process is when we zoom into time. We’ll work with realized variance as an observable proxy for the unobserved variance path, and with option prices when we study how the market prices short-horizon volatility.

A useful way to phrase the problem is:

\[ \text{how fast do volatility increments shrink when the time step shrinks?} \]

For a process \(X_t\), the size of a small increment over a time step \(\Delta\) is roughly controlled by a power law:

\[ |X_{t+\Delta} - X_t| \sim C \Delta^H \]

Here \(H\) is the Hurst exponent. If \(H\) is high, increments shrink quickly as \(\Delta\) gets smaller, so paths look smoother. If \(H\) is low, increments shrink slowly, so the path stays jagged even at small time scales.

Brownian motion has \(H = 1/2\). Rough volatility means the volatility-related process has \(H < 1/2\). A value like \(H = 0.1\) means the path is much rougher than Brownian motion. This is already a strong shift from the classical stochastic-volatility models in Project 11, where variance is usually driven by Markovian Brownian shocks with much smoother short-time behavior.

If volatility is rough, very short-dated option smiles can move and steepen much faster than classical Markovian diffusion models naturally expect.

1.1 Physical Roughness and Risk-Neutral Roughness

There are two roughness stories running in parallel.

The first is physical roughness, estimated from realized returns. It uses the historical probability measure \(\mathbb{P}\) because it’s based on what actually happened:

\[ X_t^{\mathbb{P}} = \log RV_t \]

When we estimate \(H\) from this series, we’re asking how the realized variance process behaves in historical data. This connects naturally to forecasting because a forecast is also a physical-measure object:

\[ \mathbb{E}^{\mathbb{P}}_t[RV_{t,t+h}] \]

The second is risk-neutral roughness, estimated from option prices. It uses the pricing measure \(\mathbb{Q}\) because option prices are expectations under risk-neutral pricing plus discounting:

\[ C_t(K,T) = D(t,T)\mathbb{E}^{\mathbb{Q}}_t[(S_T-K)^+] \]

When we estimate \(H\) from ATM skew, we’re not estimating the same thing as historical log variance. We’re estimating the roughness that the option market is pricing into short-maturity smiles:

\[ |\partial_k \sigma_{\text{imp}}(0,\tau)| \sim c\tau^{H-\frac{1}{2}} \]

These two values of \(H\) can differ. A historical \(H\) comes from realized variance. An option-implied \(H\) comes from option prices, risk premia, hedging flows, and market demand for convexity. In SPX, short-dated downside protection is a structural market object. In BTC, short-dated option skew can also reflect leverage and liquidation risk. So we don’t force the two \(H\) estimates to be equal. We compare them and use each one where it belongs.

We use each measure for a different part of the analysis:

Object	Measure	Used for	Main diagnostic
Log realized variance	\(\mathbb{P}\)	roughness estimation and forecasting	moment scaling
ATM implied skew	\(\mathbb{Q}\)	option-implied roughness	skew power law
rBergomi	\(\mathbb{Q}\)	Monte Carlo smile generation	IV fit and path behavior
Rough Heston	\(\mathbb{Q}\)	Fourier-style pricing and calibration	price/IV residuals and Riccati convergence

We’ll estimate roughness from realized variance and option prices, then compare model fits and hedge sensitivities. Each estimate keeps its own interpretation under the measure from which it was obtained.

Imports and plotting settings

import time
import math
from pathlib import Path

import numba
from numba import njit
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from cycler import cycler
from IPython.display import display
from scipy import optimize, stats

from quantfinlab.dataio import load_ohlcv, load_spx_option_pairs, load_option_chain, load_par_yield_curve
from quantfinlab.options.greeks import compute_greeks_numpy
from quantfinlab.options.iv import implied_vol, implied_vol_table
from quantfinlab.options.parity import infer_forwards_from_parity, infer_forwards_from_paired_quotes
from quantfinlab.options.quote_cleaning import (attach_spot_from_series, clean_option_quotes,
    convert_quotes_to_usd_equivalent, surface_ready_quotes, wide_option_chain_to_long)
from quantfinlab.options.rates_dividends import attach_rates, add_discount_factors, infer_carry_from_forward, infer_dividend_yield_from_forward
from quantfinlab.options.surface import fit_surface_panel, surface_grid, surface_iv, surface_iv_grid, surface_residuals
from quantfinlab.options.fourier import cos_prices
from quantfinlab.options.bsm import black76_price, black76_vega, forward_bsm_delta
from quantfinlab.calibration.fft_cos import calibration_weights, compare_fourier_models, fit_daily_models, residual_by_bucket
from quantfinlab.volatility.forecasting import DEFAULT_ARCH_MODEL_SPECS, future_realized_variance, rolling_arch_forecasts_weekly, score_forecasts_by_model
from quantfinlab.volatility.har import rolling_har_forecasts
from quantfinlab.volatility.realized import log_returns
from quantfinlab.plotting.curves import LAB_COLORS, set_plot_style

palette = ["#069AF3", "#FE420F", "#00008B", "#008080", "#CC79A7", "#9614fa", "#DC143C", "#7BC8F6", "#0072B2", "#04D8B2", "#800080", "#FF8072"]
plt.rcParams["axes.prop_cycle"] = cycler(color=palette)
plt.rcParams.update({"figure.figsize": (7, 3.2), "figure.dpi": 150, "savefig.dpi": 300, "axes.grid": True,
                     "grid.alpha": 0.22, "axes.spines.top": False, "axes.spines.right": False,
                     "axes.titlesize": 11, "axes.labelsize": 10, "xtick.labelsize": 8, "ytick.labelsize": 8,
                     "legend.fontsize": 7})
set_plot_style(palette)

data_dir = Path("../data")
if not data_dir.exists():
    data_dir = Path("data")

cache_dir = data_dir / "cache" / "project18"
cache_dir.mkdir(parents=True, exist_ok=True)

seed = 7
ann_days = 365.25
ann_trading = 252
q_values = [0.5, 1.0, 1.5, 2.0, 3.0]
lags = [1, 2, 4, 8, 16, 32]
horizons = [1, 5, 10, 21, 42, 63]
rng = np.random.default_rng(seed)

spx_spot = load_ohlcv(
    data_dir / "spx_index_ohlcv.csv",
    source="yfinance_csv",
    fields=("close", "raw_close", "open", "high", "low", "volume"),
)
spx_spot = spx_spot.loc["1990":].copy()

spx_pairs_raw = load_spx_option_pairs(data_dir / "spx_options_chain.parquet", annualization_days=ann_days,
                                      max_rel_spread=0.60, tau_min_days=7.0, tau_max_days=180.0,
                                      k_over_s_range=(0.62, 1.62), top_n_per_expiry=90, min_pairs_per_expiry=5)

us_par = load_par_yield_curve(
    data_dir / "us_treasury_yields.csv",
    source="us_treasury",
)

We start by loading the SPX spot series, the SPX option pairs, and the Treasury yield curve. We reuse the data prepared earlier and check that the time coverage is sufficient for two separate tasks: long realized-volatility roughness estimation and shorter option-surface calibration.

The realized-variance side starts in 1990 and gives us thousands of observations. The option calibration side is shorter because it uses the SPX option-chain panel. This split matters. Historical roughness estimation benefits from a long time series, while option-implied roughness depends on surface quality across strikes and maturities. These inputs give us the two necessary views of the SPX market.

2) Realized Variance and Log-Variance Proxies

We now build realized-variance proxies from SPX log returns. We already used realized variance in Project 5, so the definition doesn’t need to be repeated too much. The new point is that realized variance becomes a path whose roughness we estimate.

The daily log return is:

\[ r_t = \log S_t - \log S_{t-1} \]

A one-day realized variance proxy is:

\[ RV_t^{1d} = r_t^2 \]

This is extremely noisy because one squared return is a very weak estimate of the latent variance on that day. So we also use short rolling sums and averages:

\[ RV_t^{(m)} = \frac{1}{m}\sum_{j=0}^{m-1} r_{t-j}^2 \]

where \(m\) can be 5, 10, or 21 trading days. The code uses the 10-day proxy as the main realized-variance path:

\[ v_t = RV_t^{(10)} \]

and then works with log variance:

\[ X_t = \log(v_t) \]

The log transformation is useful because variance is positive and very skewed. Log variance compresses huge volatility spikes and makes changes closer to additive. When we later compute increments \(X_{t+\ell}-X_t\), we are asking how the log variance state moves across different lags.

For example, a move from annualized volatility 10% to 20% is a much bigger variance move than it looks in volatility units:

\[ 0.20^2 - 0.10^2 = 0.03 \]

but in log variance it becomes a more balanced relative movement:

\[ \log(0.20^2) - \log(0.10^2) = \log(4) \]

That relative interpretation is closer to how volatility shocks behave: a volatility doubling is meaningful whether it happens from 8% to 16% or from 20% to 40%.

2.1 Variance Proxy Design

The choice of variance proxy matters more in this project than in most earlier notebooks. In a GARCH or HAR forecast, a noisy realized variance proxy can still be useful if the model smooths it. For roughness estimation, the proxy directly changes the slope of the scaling regression.

A one-day squared return is:

\[ RV_t^{1d} = r_t^2 \]

This is unbiased in a loose daily sense if the return has conditional variance \(v_t\), but it has huge noise:

\[ r_t^2 = v_t \varepsilon_t^2 \]

where \(\varepsilon_t^2\) can be very small or very large even when \(v_t\) is unchanged. Taking logs makes this worse when \(r_t^2\) is near zero:

\[ \log(r_t^2) \]

can jump to very negative values because the return happened to be tiny. This is why the raw daily squared-return proxy has weak scaling quality.

A rolling \(m\)-day proxy reduces noise:

\[ RV_t^{(m)} = \frac{1}{m}\sum_{j=0}^{m-1}r_{t-j}^2 \]

but introduces overlap. For example, \(RV_t^{(10)}\) and \(RV_{t+1}^{(10)}\) share 9 of 10 squared returns. This creates mechanical persistence. If we estimate roughness from a heavily overlapping proxy, we may estimate smoothness created by the estimator rather than the latent variance process.

We handle this by showing multiple proxies and adding non-overlapping block checks. Comparing these proxies makes it easier to see whether a roughness estimate survives both measurement noise and the smoothing caused by overlap.

A compact way to think about the bias tradeoff is:

\[ \text{raw daily proxy} = \text{low smoothing} + \text{high noise} \]

\[ \text{rolling proxy} = \text{lower noise} + \text{overlap-induced persistence} \]

The useful estimate is the one that remains plausible after comparing both sides.

spx_ret = log_returns(spx_spot["close"]).dropna()
spx_var_1d = spx_ret.pow(2).rename("rv_1d")
spx_var_5d_sum = spx_var_1d.rolling(5).sum().rename("rv_5d_sum")
spx_var_10d_sum = spx_var_1d.rolling(10).sum().rename("rv_10d_sum")
spx_var_21d_sum = spx_var_1d.rolling(21).sum().rename("rv_21d_sum")
spx_var_5d = (spx_var_5d_sum / 5.0).rename("rv_5d_daily")
spx_var_10d = (spx_var_10d_sum / 10.0).rename("rv_10d_daily")
spx_var_21d = (spx_var_21d_sum / 21.0).rename("rv_21d_daily")
spx_var = spx_var_10d.dropna().rename("daily_variance_proxy")
spx_log_var_1d = np.log(spx_var_1d.clip(lower=1e-12)).rename("log_rv_1d")
spx_log_var_5d = np.log(spx_var_5d.dropna().clip(lower=1e-12)).rename("log_rv_5d")
spx_log_var_10d = np.log(spx_var_10d.dropna().clip(lower=1e-12)).rename("log_rv_10d")
spx_log_var_21d = np.log(spx_var_21d.dropna().clip(lower=1e-12)).rename("log_rv_21d")
spx_log_var = spx_log_var_10d.rename("log_variance")
spx_targets = future_realized_variance(spx_ret, horizons=tuple(horizons), annualization=ann_trading)

rv_summary = pd.DataFrame([{"start": spx_ret.index.min().date(), "end": spx_ret.index.max().date(),
                            "observations": int(spx_ret.shape[0]),
                            "annualized_mean_variance_1d": float(ann_trading * spx_var_1d.mean()),
                            "median_annualized_volatility_10d": float(np.sqrt(ann_trading * spx_var).median()),
                            "p95_annualized_volatility_10d": float(np.sqrt(ann_trading * spx_var).quantile(0.95))}])
display(rv_summary)

	start	end	observations	annualized_mean_variance_1d	median_annualized_volatility_10d	p95_annualized_volatility_10d
0	1990-01-03	2026-02-24	9102	0.032614	0.125971	0.312628

fig, axes = plt.subplots(1, 2, figsize=(14, 3.5), constrained_layout=True)
axes[0].plot(spx_spot.index, spx_spot["close"], lw=1.0, color=palette[0], label="SPX")
axes0b = axes[0].twinx()
axes0b.plot(spx_var_1d.index, np.sqrt(ann_trading * spx_var_1d), lw=0.45, alpha=0.25, color=palette[1], label="daily RV")
axes0b.plot(spx_var.index, np.sqrt(ann_trading * spx_var), lw=0.95, alpha=0.85, color=palette[3], label="10-day Project-5 RV")
axes[0].set_title("SPX Price and Project-5 Realized Variance Since 1990")
axes[0].set_ylabel("index")
axes0b.set_ylabel("annualized volatility")
axes[1].plot(spx_log_var_1d.index, spx_log_var_1d, lw=0.45, alpha=0.25, color=palette[1], label="raw daily")
axes[1].plot(spx_log_var.index, spx_log_var, lw=0.85, color=palette[2], label="10-day aggregated")
axes[1].set_title("SPX Log Realized Variance Proxies")
axes[1].set_ylabel("log variance")
axes[1].legend()
plt.show()

From the summary table we can see that the SPX return panel runs from 1990 to 2026 with about 9,100 observations. The median annualized 10-day volatility proxy is around 12.6%, while the 95th percentile is around 31.3%. This already tells us that volatility is highly state-dependent: the upper tail of volatility is more than twice the median level.

Looking at the three plotted series together gives us more detail. The price series trends upward over decades, but realized volatility appears as a sequence of sharp bursts. The 10-day realized-variance proxy smooths the raw daily squared return series enough to be readable, but it still has clear volatility clustering around crisis periods. The log-variance panel keeps the same burst structure while reducing the extreme visual dominance of the largest spikes.

The same variance history therefore gives us a useful setting for measuring roughness across several time scales. The volatility path is persistent enough to form regimes, but it’s also jagged enough that a smooth mean-reverting diffusion can miss the short-time behavior.

3) Fractional Brownian Motion as the Roughness Benchmark

To understand rough volatility, we first need a clean mathematical object that can represent different levels of path roughness. That object is fractional Brownian motion, written \(B_t^H\).

A fractional Brownian motion is a centered Gaussian process with covariance:

\[ \mathbb{E}[B_t^H B_s^H] = \frac{1}{2}\left(t^{2H} + s^{2H} - |t-s|^{2H}\right) \]

The parameter \(H \in (0,1)\) controls the path regularity.

The variance of an increment is:

\[ \operatorname{Var}(B_{t+\Delta}^H - B_t^H) = \Delta^{2H} \]

So the typical size of the increment is:

\[ |B_{t+\Delta}^H - B_t^H| \sim \Delta^H \]

This is the power-law scaling we care about. Brownian motion is the special case \(H = 1/2\) because Brownian increments scale like \(\sqrt{\Delta}\).

The sign of increment dependence also changes with \(H\):

• \(H < 1/2\): increments are anti-persistent. A positive increment is more likely to be followed by a negative one. The path looks jagged and choppy.
• \(H = 1/2\): increments are independent. This is ordinary Brownian motion.
• \(H > 1/2\): increments are persistent. Positive increments tend to be followed by positive increments. The path looks smoother and more trend-like.

The covariance of fractional Gaussian noise, which is the increment process of fBM, is:

\[ \gamma_H(k) = \frac{1}{2}\left(|k+1|^{2H} - 2|k|^{2H} + |k-1|^{2H}\right) \]

Here \(k\) is the lag between increments. We use these lag covariances to build the covariance matrix of the simulated increments. Once the covariance matrix is known, the simulation draws correlated Gaussian increments using a Cholesky factorization:

Because the increment covariance depends on the distance \(k\) between observations, each diagonal of that matrix contains the covariance at one lag. Its diagonal entries give the unit-step increment variances.

\[ \Delta B^H = LZ \]

where \(Z\) is a vector of independent standard normal draws and \(LL^\top\) is the fractional Gaussian noise covariance matrix. After that we cumulatively sum the increments to obtain the fBM path.

The Cholesky approach is easy to understand and accurate for teaching, although it becomes expensive for large simulations because the covariance matrix is dense. That cost is one reason rough models become computationally heavier than classical Markovian models.

3.1 Path Regularity by Scale

The easiest way to understand \(H\) is to compare the size of increments at a small time scale. Suppose we shrink the time step from one month to one day. If the step size is roughly divided by 21, the typical increment scales by:

\[ 21^{-H} \]

For \(H=0.5\):

\[ 21^{-0.5} \approx 0.218 \]

For \(H=0.1\):

\[ 21^{-0.1} \approx 0.738 \]

This is the whole rough-volatility intuition in one calculation. Under Brownian scaling, daily increments are much smaller than monthly increments. Under rough scaling with \(H=0.1\), the daily increment is still a large fraction of the monthly increment. The process remains jagged when we zoom in.

That is exactly what the option market cares about. A short-dated option is highly sensitive to what happens over the next few days. If volatility can move violently on short horizons, then short-maturity skew should be steep. Classical Heston can generate stochastic variance, but its short-time regularity is tied to ordinary Brownian diffusion. Rough models change the short-time scaling itself.

There is also a common confusion between roughness and long memory. Low \(H\) in rough volatility doesn’t mean volatility is a slow, persistent long-memory process. It means local paths are irregular. A process can look highly persistent in autocorrelation tests because it has clustered volatility, but its local regularity can still be rough. So we focus on increment scaling rather than only autocorrelation.

3.2 Moment Scaling on Simulated Paths

The moment-scaling method estimates \(H\) from how increments behave across lags. For a process \(X_t\), define the \(q\)-moment of increments at lag \(\ell\):

\[ m_q(\ell) = \mathbb{E}\left[|X_{t+\ell} - X_t|^q\right] \]

For fractional Brownian motion, increments scale as \(\ell^H\), so the \(q\)-moment scales as:

\[ m_q(\ell) \propto \ell^{qH} \]

Taking logs gives a line:

\[ \log m_q(\ell) = a_q + qH \log \ell \]

where \(a_q\) is a constant that depends on \(q\) but not on the lag. This is the core estimation equation. The intercept controls the vertical level of each moment curve, while the coefficient of \(\log \ell\) measures how quickly the moment changes across scales. For each \(q\), we regress \(\log m_q(\ell)\) on \(\log \ell\):

\[ \log m_q(\ell) = \beta_{0,q} + \beta_{1,q}\log \ell + \varepsilon_{\ell,q} \]

Then:

\[ \widehat{H}_q = \frac{\widehat{\beta}_{1,q}}{q} \]

If the process really behaves like fractional Brownian motion over the tested scales, then \(\widehat{H}_q\) should be fairly stable across different \(q\) values.

For example, if \(q=2\) and the regression slope is \(0.20\), then:

\[ \widehat{H} = \frac{0.20}{2} = 0.10 \]

That means a four-times larger lag increases the typical squared increment by approximately \(4^{0.20}\) rather than by \(4^1\) as we would see for a much smoother process in variance units. The power law is the signal.

3.3 The Meaning of \(H\) in Volatility Modeling

The Hurst exponent is the number that controls how quickly increments shrink as we look at shorter time intervals. For a process \(X_t\), the scaling idea is:

\[ \mathbb{E}\left[|X_{t+\Delta}-X_t|^q\right] \propto \Delta^{qH} \]

Each part has a role:

• \(X_{t+\Delta}-X_t\) is the move over a time step of length \(\Delta\),
• \(q\) chooses which moment we measure,
• \(H\) controls the slope of the log-log scaling relation,
• the proportionality sign means we care about the power-law slope, not the exact level.

Taking logs gives a regression-style expression:

\[ \log \mathbb{E}\left[|X_{t+\Delta}-X_t|^q\right] = a_q + qH\log \Delta \]

So if we plot log-moment against log-time-step, the slope should be close to \(qH\). Dividing the slope by \(q\) gives an estimate of \(H\).

For Brownian motion, \(H=0.5\). That means typical increments shrink like \(\sqrt{\Delta}\). If we cut the time interval from one day to one quarter of a day, the typical move is cut roughly in half. For rough volatility, \(H<0.5\). This means the path is less smooth at short horizons. Very small time intervals can still contain relatively sharp changes.

This matters for option pricing because short-maturity options are almost entirely about short-horizon behavior. A one-week option doesn’t care much about the average volatility over the next two years. It cares about whether variance can jump or roughen over the next few trading days. If volatility has \(H=0.1\), the short-term variance path is much more jagged than a Brownian volatility path with \(H=0.5\). That jaggedness is one reason rough models can generate steeper short-end smiles and skews.

There’s also a practical warning. Estimated \(H\) depends heavily on the object we measure. Raw squared returns are noisy. Rolling realized variance is smoother but overlap can inflate persistence. Option-implied skew measures risk-neutral roughness, not historical realized roughness. So we shouldn’t expect one universal \(H\) number to appear in every diagnostic. We compare several estimates so that we can see how the chosen proxy and the probability measure affect our view of the volatility process.

The regression form also gives a visual diagnostic. If we plot \(\log \widehat m_q(\ell)\) against \(\log \ell\), a power law appears as a straight line. The slope of that line is \(qH\).

The steps are:

1. choose a moment order \(q\),
2. choose lags \(\ell \in \{1,2,4,8,16,32\}\),
3. compute increments at each lag,
4. compute the average absolute \(q\)-power of those increments,
5. take logs,
6. fit a line.

For a single \(q\):

\[ \widehat{\beta}_{1,q} = \frac{\sum_\ell (\log \ell - \overline{\log \ell})(\log \widehat m_q(\ell)-\overline{\log \widehat m_q})}{\sum_\ell(\log \ell-\overline{\log \ell})^2} \]

and:

\[ \widehat H_q = \frac{\widehat{\beta}_{1,q}}{q} \]

The \(R^2\) of this regression matters. A nice-looking \(H\) estimate with poor \(R^2\) is not useful. It means the power-law relationship is weak. So we report both \(H\) and \(R^2\). We need the estimate and the quality of the scaling relationship.

For example, if \(q=1\) gives \(H=0.12\) with \(R^2=0.98\), and \(q=2\) gives \(H=0.13\) with \(R^2=0.97\), that is strong evidence of stable rough scaling. If \(q=1\) gives \(H=0.02\) with \(R^2=0.10\), that is just noise.

def fgn_covariance(h, n):
    k = np.arange(int(n), dtype=float)
    return 0.5 * (np.abs(k + 1.0) ** (2.0 * h) - 2.0 * np.abs(k) ** (2.0 * h) + np.abs(k - 1.0) ** (2.0 * h))


def fbm_cholesky(h, n_steps, n_paths, seed):
    cov = fgn_covariance(float(h), int(n_steps))
    idx = np.abs(np.subtract.outer(np.arange(int(n_steps)), np.arange(int(n_steps))))
    mat = cov[idx] + 1e-12 * np.eye(int(n_steps))
    z = np.random.default_rng(int(seed)).standard_normal((int(n_steps), int(n_paths)))
    increments = np.linalg.cholesky(mat) @ z / (int(n_steps) ** float(h))
    paths = np.vstack([np.zeros((1, int(n_paths))), np.cumsum(increments, axis=0)])
    return pd.DataFrame(paths, index=np.linspace(0.0, 1.0, int(n_steps) + 1))


def path_moment_scaling(x, q_values, lags):
    values = np.asarray(x, dtype=float)
    rows = []
    for q in q_values:
        for lag in lags:
            lag = int(lag)
            if lag >= values.shape[0]:
                continue
            inc = values[lag:] - values[:-lag]
            moment = float(np.nanmean(np.abs(inc) ** float(q)))
            rows.append({"q": float(q), "lag": lag, "moment": moment, "log_lag": np.log(lag), "log_moment": np.log(moment)})
    return pd.DataFrame(rows)

fbm_h_values = [0.05, 0.10, 0.20, 0.50, 0.80]
fbm_paths = {}
fbm_scaling_rows = []
fbm_h_rows = []

for i, h0 in enumerate(fbm_h_values):
    fbm_paths[h0] = fbm_cholesky(h0, 512, 4, seed + i)
    scaling_h = path_moment_scaling(fbm_paths[h0].iloc[:, 0], q_values, lags)
    scaling_h["h_true"] = h0
    fbm_scaling_rows.append(scaling_h)
    for q, g in scaling_h.groupby("q"):
        beta, cov = np.polyfit(g["log_lag"], g["log_moment"], 1, cov=True)
        yhat = np.polyval(beta, g["log_lag"])
        ss_res = float(np.sum((g["log_moment"] - yhat) ** 2))
        ss_tot = float(np.sum((g["log_moment"] - g["log_moment"].mean()) ** 2))
        fbm_h_rows.append({"h_true": h0, "q": q, "slope": beta[0], "h_estimate": beta[0] / q,
                           "standard_error": float(np.sqrt(cov[0, 0])), "r2": 1.0 - ss_res / ss_tot})

fbm_scaling = pd.concat(fbm_scaling_rows, ignore_index=True)
fbm_h_table = pd.DataFrame(fbm_h_rows)
display(fbm_h_table.groupby("h_true", as_index=False).agg(median_h=("h_estimate", "median"), mean_r2=("r2", "mean")))

	h_true	median_h	mean_r2
0	0.05	0.037069	0.976740
1	0.10	0.123190	0.993578
2	0.20	0.176987	0.990194
3	0.50	0.522184	0.997473
4	0.80	0.809222	0.999785

fig, axes = plt.subplots(2, 2, figsize=(13, 7), constrained_layout=True)
for i, h0 in enumerate(fbm_h_values):
    axes[0, 0].plot(fbm_paths[h0].index, fbm_paths[h0].iloc[:, 0], lw=1.0, color=palette[i], label=f"H={h0:.2f}")
    inc = fbm_paths[h0].iloc[:, 0].diff().iloc[1:90]
    axes[0, 1].plot(np.arange(len(inc)), inc, lw=0.9, color=palette[i], label=f"H={h0:.2f}")
for i, (h0, g) in enumerate(fbm_scaling[fbm_scaling["q"].eq(2.0)].groupby("h_true")):
    axes[1, 0].plot(g["log_lag"], g["log_moment"], marker="o", ms=3, lw=1.0, color=palette[i], label=f"H={h0:.2f}")
axes[1, 1].scatter(fbm_h_table["q"], fbm_h_table["slope"], c=fbm_h_table["h_true"], cmap="viridis", s=35)
for h0 in fbm_h_values:
    x = np.asarray(q_values, dtype=float)
    axes[1, 1].plot(x, h0 * x, lw=0.8, alpha=0.45)
axes[0, 0].set_title("fBM Path Comparison")
axes[0, 1].set_title("Zoomed fBM Increments")
axes[1, 0].set_title("Simulated q=2 Moment Scaling")
axes[1, 1].set_title("Slope Equals qH")
for ax in axes.ravel():
    ax.legend(fontsize=6, ncol=2)
plt.show()

C:\Users\Lenovo\AppData\Local\Temp\ipykernel_16732\2139515692.py:17: UserWarning: No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
  ax.legend(fontsize=6, ncol=2)

We can check the scaling estimator on simulated paths whose \(H\) values are already known. We generate fBM paths with known \(H\) values and then estimate \(H\) back from moment scaling. The recovered medians are close to the true values: the \(H=0.10\) simulation is estimated around 0.123, the Brownian-like \(H=0.50\) case is estimated around 0.522, and the smoother \(H=0.80\) case is estimated around 0.809.

The simulated paths and scaling lines make the differences between the estimated values easier to see. The \(H=0.05\) and \(H=0.10\) paths have very jagged increments. The \(H=0.80\) path looks much more persistent. In the bottom-left panel, we can compare the log-moment lines across the simulated paths. In the bottom-right panel, we compare the slopes across \(q\) with straight reference lines. For fBM, the scaling slope is approximately \(qH\).

The moment-scaling regression therefore connects a property of the path to a slope that we can estimate and compare across moment orders.

4) Estimating Roughness from SPX Realized Variance

Now we apply the same scaling logic to SPX log realized variance. The object is:

\[ X_t = \log RV_t \]

and the increments are:

\[ \Delta_\ell X_t = X_{t+\ell} - X_t \]

For each moment order \(q\) and each lag \(\ell\), we estimate:

\[ \widehat{m}_q(\ell) = \frac{1}{N_\ell}\sum_t |\Delta_\ell X_t|^q \]

Then we fit:

\[ \log \widehat{m}_q(\ell) = \beta_{0,q} + \beta_{1,q}\log \ell + e_{\ell,q} \]

and use:

\[ \widehat{H}_q = \frac{\widehat{\beta}_{1,q}}{q} \]

There are two implementation details that matter here.

First, we estimate \(H\) separately for different realized-variance proxies: raw daily squared returns, 5-day rolling RV, 10-day rolling RV, and 21-day rolling RV. This helps separate true roughness from measurement noise. A one-day squared return is too noisy and can make the scaling estimate meaningless. Longer rolling RV proxies are smoother, but overlapping windows can artificially increase persistence.

Second, we also use a pooled regression across the selected moment orders. Instead of estimating \(H\) separately for each \(q\), the pooled version stacks all moment orders and estimates one shared \(H\):

\[ \log \widehat{m}_q(\ell) = a_q + Hq\log \ell + e_{q,\ell} \]

Here each \(q\) gets its own intercept \(a_q\), but the slope coefficient is constrained to be \(Hq\). In matrix form, if each row is one pair \((q,\ell)\), the first column of the design matrix is \(q\log \ell\):

\[ y = X\theta + e \]

with:

\[ y_{q,\ell} = \log \widehat{m}_q(\ell), \qquad X_{q,\ell,1} = q\log \ell \]

The coefficient on this first column is the pooled estimate of \(H\). The remaining columns give each moment order its own intercept, so the regression compares the changes across lags without forcing all moment curves to have the same starting level. This is useful because individual \(q\) estimates can be noisy, especially at high moment orders where extreme volatility jumps dominate the moment.

4.1 Separate-\(q\) and Pooled-\(q\) Estimation

The separate-\(q\) estimator is easy to read:

\[ \widehat H_q = \frac{\widehat\beta_{1,q}}{q} \]

but every \(q\) sees the data differently. Low \(q\) values treat ordinary fluctuations more evenly. High \(q\) values give much more weight to extreme volatility jumps. In financial variance data, \(q=3\) can be strongly influenced by crisis observations.

The pooled estimator tries to stabilize this by estimating one \(H\) across all \(q\) values:

\[ \log \widehat m_q(\ell) = a_q + Hq\log \ell + e_{q,\ell} \]

The intercept \(a_q\) is allowed to change with \(q\) because the level of the moment is different for \(q=0.5\), \(q=1\), or \(q=2\). But the slope structure is constrained: the lag slope must be proportional to \(q\).

To see how the common slope and separate intercepts fit together, we can write the pooled regression in matrix notation:

\[ \mathbf{y} = \mathbf{X}\boldsymbol{\theta} + \boldsymbol{\varepsilon} \]

where the first coefficient of \(\boldsymbol{\theta}\) is \(H\). A row corresponding to moment order \(q_i\) and lag \(\ell_i\) has:

\[ X_{i,1}=q_i\log \ell_i \]

and then dummy/intercept columns for each \(q\). The pooled OLS estimate is:

\[ \widehat{\boldsymbol{\theta}} = (\mathbf{X}^\top \mathbf{X})^{-1}\mathbf{X}^\top \mathbf{y} \]

We also use a robust option when available. The reason is that volatility data has extreme jumps. A Huber-style regression reduces the influence of huge observations without throwing them away entirely. The non-overlapping block estimates group squared returns into distinct blocks, which lets us compare the rolling estimates with a proxy that doesn’t share observations across adjacent blocks. That fits the financial problem: crisis spikes are real, but we don’t want one crash week to determine the roughness exponent alone.

def power_variation(x, q_values, lags):
    s = pd.Series(x).dropna()
    values = s.to_numpy(dtype=float)
    rows = []
    for q in q_values:
        for lag in lags:
            lag = int(lag)
            inc = values[lag:] - values[:-lag]
            inc = inc[np.isfinite(inc)]
            moment = float(np.mean(np.abs(inc) ** float(q)))
            rows.append({"q": float(q), "lag": lag, "moment": moment, "log_lag": np.log(lag), "log_moment": np.log(moment), "n": int(inc.size)})
    return pd.DataFrame(rows)


def hurst_ols(scaling):
    rows = []
    for q, g in scaling.groupby("q", sort=True):
        x = g["log_lag"].to_numpy(dtype=float)
        y = g["log_moment"].to_numpy(dtype=float)
        design = np.column_stack([np.ones(len(x)), x])
        beta, _, _, _ = np.linalg.lstsq(design, y, rcond=None)
        fitted = design @ beta
        resid = y - fitted
        ss_res = float(np.sum(resid * resid))
        ss_tot = float(np.sum((y - np.mean(y)) ** 2))
        sigma2 = ss_res / max(len(x) - 2, 1)
        cov = sigma2 * np.linalg.pinv(design.T @ design)
        rows.append({"q": float(q), "slope": float(beta[1]), "h": float(beta[1] / q),
                     "standard_error": float(np.sqrt(max(cov[1, 1], 0.0))), "r2": float(1.0 - ss_res / ss_tot),
                     "n": int(len(g))})
    return pd.DataFrame(rows)


def hurst_pooled(scaling, q_exclude=(3.0,), use_huber=True):
    data = scaling.copy()
    data["log_lag_fit"] = np.log(pd.to_numeric(data["lag"], errors="coerce").replace(0, np.nan))
    data["log_moment_fit"] = np.log(pd.to_numeric(data["moment"], errors="coerce").replace([0, np.inf, -np.inf], np.nan))
    data = data.replace([np.inf, -np.inf], np.nan).dropna(subset=["q", "log_lag_fit", "log_moment_fit"])
    if q_exclude:
        data = data[~data["q"].isin(q_exclude)].copy()
    if len(data) < 4:
        return {"H": np.nan, "se": np.nan, "r2": np.nan, "n": len(data), "method": "pooled"}
    q_unique = sorted(data["q"].unique())
    n, n_q = len(data), len(q_unique)
    q_idx = {q: i for i, q in enumerate(q_unique)}
    X = np.zeros((n, 1 + n_q), dtype=float)
    q_arr = data["q"].to_numpy(float)
    ll_arr = data["log_lag_fit"].to_numpy(float)
    X[:, 0] = q_arr * ll_arr
    for i, rq in enumerate(q_arr):
        X[i, 1 + q_idx[rq]] = 1.0
    y = data["log_moment_fit"].to_numpy(float)
    h_est, se, r2, method_used = np.nan, np.nan, np.nan, "ols"
    if use_huber:
        try:
            from sklearn.linear_model import HuberRegressor
            mdl = HuberRegressor(epsilon=1.35, max_iter=500, fit_intercept=False)
            mdl.fit(X, y)
            h_est = float(mdl.coef_[0])
            y_hat = mdl.predict(X)
            ss_res = float(np.sum((y - y_hat) ** 2))
            ss_tot = float(np.sum((y - np.mean(y)) ** 2))
            r2 = float(1.0 - ss_res / ss_tot) if ss_tot > 0 else np.nan
            method_used = "huber"
        except Exception:
            use_huber = False
    if not use_huber or not np.isfinite(h_est):
        beta, *_ = np.linalg.lstsq(X, y, rcond=None)
        h_est = float(beta[0])
        y_hat = X @ beta
        ss_res = float(np.sum((y - y_hat) ** 2))
        ss_tot = float(np.sum((y - np.mean(y)) ** 2))
        r2 = float(1.0 - ss_res / ss_tot) if ss_tot > 0 else np.nan
        dof = max(n - 1 - n_q, 1)
        sigma2 = ss_res / dof
        try:
            cov = sigma2 * np.linalg.pinv(X.T @ X)
            se = float(np.sqrt(max(cov[0, 0], 0.0)))
        except Exception:
            se = np.nan
        method_used = "ols"
    return {"H": h_est, "se": se, "r2": r2, "n": n, "method": method_used}


def hurst_multiscale(returns, windows=(1, 2, 3, 5, 10, 21), q_values=(0.5, 1.0, 1.5, 2.0), lags=(1, 2, 4, 8, 16, 32), eps=1e-12, use_huber=True):
    rv_1d = (returns * returns).clip(lower=float(eps))
    rows = []
    for w in windows:
        w = int(w)
        if w == 1:
            log_rv = np.log(rv_1d.clip(lower=float(eps))).rename(f"log_rv_{w}d")
        else:
            rv_w = rv_1d.rolling(w).sum() / float(w)
            log_rv = np.log(rv_w.dropna().clip(lower=float(eps))).rename(f"log_rv_{w}d")
        sc = power_variation(log_rv, list(q_values), list(lags))
        h_per_q = hurst_ols(sc)
        h_pooled = hurst_pooled(sc, use_huber=use_huber)
        h_q1 = float(h_per_q[h_per_q["q"].between(0.9, 1.1)]["h"].mean()) if not h_per_q.empty else np.nan
        rows.append({
            "window_days": w,
            "n_obs": int(log_rv.shape[0]),
            "H_pooled": float(h_pooled["H"]) if np.isfinite(float(h_pooled["H"])) else np.nan,
            "H_q1": h_q1,
            "H_median_per_q": float(h_per_q["h"].median()) if not h_per_q.empty else np.nan,
            "r2_pooled": float(h_pooled["r2"]) if h_pooled["r2"] is not None and np.isfinite(float(h_pooled["r2"])) else np.nan,
            "method": str(h_pooled["method"]),
        })
    df = pd.DataFrame(rows)
    short_mask = df["window_days"].le(10)
    df["main_H"] = float(df.loc[short_mask, "H_pooled"].median()) if short_mask.any() else float(df["H_pooled"].median())
    return df

log_var_proxies = {"raw daily squared-return RV": spx_log_var_1d, "rolling Project-5 RV 5d": spx_log_var_5d,
                   "rolling Project-5 RV 10d": spx_log_var_10d, "rolling Project-5 RV 21d": spx_log_var_21d}
empirical_scaling_by_proxy = {}
empirical_hurst_by_proxy = {}
proxy_rows = []
for proxy_name, proxy_series in log_var_proxies.items():
    scaling_proxy = power_variation(proxy_series, q_values, lags)
    hurst_proxy = hurst_ols(scaling_proxy)
    empirical_scaling_by_proxy[proxy_name] = scaling_proxy.assign(proxy=proxy_name)
    empirical_hurst_by_proxy[proxy_name] = hurst_proxy.assign(proxy=proxy_name)
    proxy_rows.append({"source": proxy_name,
                       "sample": f"{proxy_series.index.min().date()} to {proxy_series.index.max().date()}",
                       "h": float(hurst_proxy["h"].median()),
                       "standard_error": float(hurst_proxy["standard_error"].median()),
                       "r2": float(hurst_proxy["r2"].mean()), "n": int(hurst_proxy["n"].sum()),
                       "interpretation": "noisy diagnostic" if "raw daily" in proxy_name
                       else "aggregated Project-5 RV proxy"})

roughness_proxy_table = pd.DataFrame(proxy_rows)
emp_scaling = empirical_scaling_by_proxy["rolling Project-5 RV 10d"]
emp_hurst = empirical_hurst_by_proxy["rolling Project-5 RV 10d"]

emp_pooled = hurst_pooled(emp_scaling, use_huber=True)
multi_window_table = hurst_multiscale(spx_ret, windows=(1, 2, 3, 5, 10, 21), q_values=tuple(q_values), lags=tuple(lags), use_huber=True)
h_rv_pooled = float(emp_pooled["H"]) if np.isfinite(float(emp_pooled["H"])) else float(emp_hurst["h"].median())
h_rv_multi = float(multi_window_table["main_H"].iloc[0])
h_rv = float(np.clip(h_rv_multi if np.isfinite(h_rv_multi) else h_rv_pooled, 0.03, 0.48))

def _block_rv_series(ret, block_size):
    r2 = (ret * ret).to_numpy()
    nb = len(r2) // block_size
    return pd.Series([r2[i * block_size:(i + 1) * block_size].sum() for i in range(nb)])

block_rv_rows = []
for block_size in [5, 10, 21]:
    block_rv = _block_rv_series(spx_ret, block_size)
    block_log_rv = np.log(block_rv.clip(1e-12))
    block_scaling = power_variation(block_log_rv, q_values, lags)
    block_hurst = hurst_pooled(block_scaling, use_huber=True)
    rolling_match = roughness_proxy_table[roughness_proxy_table["source"].str.contains(f"{block_size}d")]
    block_rv_rows.append({"block_days": block_size, "n_blocks": len(block_rv),
                          "H_pooled": float(block_hurst["H"]),
                          "H_rolling": float(rolling_match["h"].iloc[0]) if len(rolling_match) > 0 else np.nan,
                          "r2": float(block_hurst["r2"])})
block_rv_table = pd.DataFrame(block_rv_rows)

increment_acf = pd.DataFrame({
    "lag": np.arange(1, 31),
    "acf": [spx_log_var.diff().autocorr(lag=i) for i in range(1, 31)],
})

display(roughness_proxy_table.round(4))
display(emp_hurst.round(4))
display(pd.DataFrame([emp_pooled]).round(5))
display(multi_window_table.round(4))
display(block_rv_table.round(4))
print(f"h_rv = {h_rv:.4f}  (pooled 10d: {h_rv_pooled:.4f})")

	source	sample	h	standard_error	r2	n	interpretation
0	raw daily squared-return RV	1990-01-03 to 2026-02-24	0.0018	0.0055	0.0442	30	noisy diagnostic
1	rolling Project-5 RV 5d	1990-01-09 to 2026-02-24	0.2878	0.0748	0.8911	30	aggregated Project-5 RV proxy
2	rolling Project-5 RV 10d	1990-01-16 to 2026-02-24	0.4569	0.0652	0.9635	30	aggregated Project-5 RV proxy
3	rolling Project-5 RV 21d	1990-01-31 to 2026-02-24	0.6039	0.0310	0.9944	30	aggregated Project-5 RV proxy

	q	slope	h	standard_error	r2	n	proxy
0	0.5	0.2732	0.5463	0.0321	0.9476	6	rolling Project-5 RV 10d
1	1.0	0.4957	0.4957	0.0517	0.9584	6	rolling Project-5 RV 10d
2	1.5	0.6853	0.4569	0.0652	0.9650	6	rolling Project-5 RV 10d
3	2.0	0.8501	0.4250	0.0751	0.9697	6	rolling Project-5 RV 10d
4	3.0	1.1223	0.3741	0.0869	0.9766	6	rolling Project-5 RV 10d

	H	se	r2	n	method
0	0.47349	NaN	0.95963	24	huber

	window_days	n_obs	H_pooled	H_q1	H_median_per_q	r2_pooled	method	main_H
0	1	9102	0.0048	0.0023	0.0018	0.9997	huber	0.1313
1	2	9101	0.0447	0.0976	0.0815	0.9349	huber	0.1313
2	3	9100	0.1313	0.1923	0.1669	0.8613	huber	0.1313
3	5	9098	0.2881	0.3211	0.2878	0.8882	huber	0.1313
4	10	9093	0.4735	0.4957	0.4569	0.9596	huber	0.1313
5	21	9082	0.6068	0.6488	0.6039	0.9926	huber	0.1313

	block_days	n_blocks	H_pooled	H_rolling	r2
0	5	1820	0.0944	0.2878	0.9971
1	10	910	0.1183	0.4569	0.9690
2	21	433	0.1572	0.6039	0.9636

h_rv = 0.1313  (pooled 10d: 0.4735)

fig, axes = plt.subplots(2, 2, figsize=(13, 7), constrained_layout=True)
for i, (q, g) in enumerate(empirical_scaling_by_proxy["raw daily squared-return RV"].groupby("q", sort=True)):
    axes[0, 0].plot(g["log_lag"], g["log_moment"], marker="o", ms=2, lw=0.65, alpha=0.28, color=palette[i])
for i, (q, g) in enumerate(emp_scaling.groupby("q", sort=True)):
    axes[0, 0].plot(g["log_lag"], g["log_moment"], marker="o", ms=3, lw=1.0, color=palette[i], label=f"q={q:g}")
axes[0, 1].scatter(emp_hurst["q"], emp_hurst["slope"], s=35, color=palette[0])
x = np.linspace(0.0, max(q_values), 80)
axes[0, 1].plot(x, h_rv * x, color=palette[1], lw=1.4, label=f"median H={h_rv:.3f}")
axes[1, 0].plot(emp_hurst["q"], emp_hurst["h"], marker="o", lw=1.0, color=palette[2])
axes[1, 0].axhline(h_rv, color=palette[1], ls="--", lw=1.0)
for i, row in roughness_proxy_table.iterrows():
    axes[1, 1].bar(i, row["h"], color=palette[i % len(palette)], alpha=0.80)
axes[1, 1].set_xticks(range(len(roughness_proxy_table)))
axes[1, 1].set_xticklabels(["1d", "5d", "10d", "21d"])
axes[0, 0].set_title("Raw Daily RV Diagnostic vs 10-Day Project-5 RV")
axes[0, 1].set_title("Slope(q) and qH Linearity")
axes[1, 0].set_title("H Estimates by Moment Order")
axes[1, 1].set_title("H by Realized-Variance Proxy")
for ax in axes.ravel():
    ax.legend(fontsize=6)
plt.show()

C:\Users\Lenovo\AppData\Local\Temp\ipykernel_16732\1299751826.py:20: UserWarning: No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
  ax.legend(fontsize=6)

The different proxies give us quite different roughness estimates, even though they use the same underlying return history. The raw daily squared-return proxy produces an \(H\) estimate close to zero, but the \(R^2\) is terrible. That means the raw proxy is too noisy for a clean scaling relationship. The 5-day, 10-day, and 21-day rolling proxies give much stronger \(R^2\), but their \(H\) estimates increase as the aggregation window gets longer: about 0.288 for 5-day, 0.457 for 10-day, and 0.604 for 21-day.

That doesn’t mean volatility suddenly becomes smooth at 21 days. It means the measurement process has changed. A 21-day rolling window overlaps heavily with the previous 21-day window, so consecutive observations share most of their returns. This overlap mechanically smooths the series and inflates apparent persistence.

The multi-window diagnostic is useful because it doesn’t blindly trust the smoothest proxy. The selected main \(H\) is around 0.131, closer to the rough-volatility range. For the non-overlapping block checks, block RV estimates around 0.094 to 0.157 are much closer to rough behavior than the overlapping 10-day and 21-day rolling values.

We can see the same contrast in the moment-scaling plots. The raw daily diagnostic is messy, while the 10-day Project-5 RV proxy gives clean lines. The clean lines are helpful, but the proxy comparison warns us not to treat one smooth proxy as the true latent roughness. We therefore use \(H\) alongside the fit quality and the other proxy estimates when judging the evidence for roughness.

The contrast between rolling and non-overlapping proxies changes how we interpret the estimated roughness. If we only looked at the 21-day rolling proxy, we would conclude that \(H\) is around 0.60, which is smoother than Brownian motion and therefore not rough. But the 21-day rolling proxy is constructed by overlapping windows, so it has built-in smoothness.

The non-overlapping block estimates are much closer to the rough range. When we remove the overlap effect, the block estimates of \(H\) come back toward 0.10–0.16. That’s much more consistent with the rough-volatility literature and with the project’s final selected \(h_rv \approx 0.131\).

The lesson here is methodological:

• raw squared returns are too noisy,
• rolling realized variance is cleaner but can over-smooth,
• non-overlapping checks help detect artificial persistence,
• option-implied skew gives a separate risk-neutral roughness estimate.

We interpret \(H\) together with its measurement method and scaling diagnostics. Roughness estimation is an inference problem with measurement error, and the range across proxies tells us more than any single estimate on its own.

5) Rough-Kernel Forecasting

After estimating roughness from realized variance, we use it in a forecasting model. On the physical-measure side, we ask whether a rough-memory kernel can help us forecast future realized variance.

The rough-kernel forecast is built from past realized variance values:

\[ \widehat{v}_{t,h} = \sum_{j=1}^{L} w_{j,h}(H)v_{t-j+1} \]

where:

• \(v_{t-j+1}\) is a past daily variance proxy,
• \(L\) is the lookback length,
• \(h\) is the forecast horizon,
• \(w_{j,h}(H)\) is the weight assigned to that lag.

We use horizon-adjusted power-law weights in the forecast:

\[ w_{j,h}(H) = \frac{(j+h)^{H+1/2} - j^{H+1/2}}{\sum_{\ell=1}^{L}\left[(\ell+h)^{H+1/2} - \ell^{H+1/2}\right]} \]

These weights are positive and normalized:

The numerator measures the change in the integrated power kernel over a horizon of \(h\) days. Dividing by the sum across all \(L\) lags makes the forecast a weighted average, so the daily variance inputs and the forecast remain in the same units.

\[ \sum_{j=1}^{L} w_{j,h}(H) = 1 \]

The finance meaning is clear: we forecast future variance as a weighted average of past variance, but the weights decay in a fractional way rather than an exponential way. GARCH-like models usually rely on recursive exponential-type updating. HAR-RV uses a few block averages such as daily, weekly, and monthly variance. The rough kernel uses a smoother continuum of memory weights.

The estimated \(H\) changes the shape of the weights. When \(H\) is low, the near past receives more weight, but the decay isn’t just a short exponential cutoff. The model still keeps a long fractional tail. That’s the rough-volatility compromise: volatility reacts sharply at short horizons while still carrying memory across longer horizons.

The evaluation uses variance losses, not just volatility errors. A key loss is QLIKE:

\[ QLIKE(v,\widehat{v}) = \frac{v}{\widehat{v}} - \log\left(\frac{v}{\widehat{v}}\right) - 1 \]

where \(v\) is realized variance and \(\widehat{v}\) is forecast variance. Lower is better. QLIKE is useful because it penalizes variance forecasts in a scale-aware way and is commonly used for volatility-forecast evaluation.

The forecast scores below use the equivalent ranking form \(\log(\widehat v)+v/\widehat v\). For a given realized \(v\), the displayed expression differs from this score by \(-\log(v)-1\), which doesn’t depend on the forecast. The score can therefore be negative even though the normalized formula above is nonnegative; model rankings agree when they are evaluated on the same realized observations.

5.1 Relation to HAR and GARCH

The comparison models are reused from Project 5, so we don’t need to re-derive them deeply. But it helps to understand what the rough kernel is competing against.

HAR-RV uses a small set of block averages:

\[ RV_{t+1} = \beta_0 + \beta_d RV_t^{(d)} + \beta_w RV_t^{(w)} + \beta_m RV_t^{(m)} + \varepsilon_{t+1} \]

It approximates heterogeneous market horizons: daily traders, weekly traders, and monthly investors. It is simple, stable, and very hard to beat.

GARCH updates variance recursively:

\[ \sigma_{t+1}^2 = \omega + \alpha r_t^2 + \beta \sigma_t^2 \]

This is a compact exponential-memory model. Shocks decay through the \(\beta\) term. Extensions such as GJR and EGARCH add asymmetry and log-variance behavior, but the basic idea is still recursive updating.

The rough kernel takes a different route. It doesn’t use a few block averages and doesn’t use a one-step recursion. It uses many lags with a power-law memory shape:

\[ \widehat v_{t,h} = \sum_{j=1}^{L}w_{j,h}(H)v_{t-j+1} \]

If HAR is a three-bucket approximation and GARCH is an exponential-recursion approximation, the rough kernel is a fractional-memory approximation. This makes it a natural forecast model after we estimate \(H\).

def kernel_weights(h, lookback):
    j = np.arange(1, int(lookback) + 1, dtype=float)
    weights = j ** (float(h) - 0.5)
    weights = weights / weights.sum()
    return weights[::-1]


def kernel_weights_horizon(h, lookback, horizon):
    j = np.arange(1, int(lookback) + 1, dtype=float)
    weights = (j + float(horizon)) ** (float(h) + 0.5) - j ** (float(h) + 0.5)
    weights = np.maximum(weights, 0.0)
    weights = weights / weights.sum()
    return weights[::-1]


def kernel_forecasts(rv, targets, h, horizons, train_window, signal_step, annualization):
    v = pd.Series(rv).dropna().clip(lower=1e-12)
    values = v.to_numpy(dtype=float)
    records = []
    for pos in range(int(train_window), len(v) - int(max(horizons)), int(signal_step)):
        hist = values[pos - int(train_window) + 1:pos + 1]
        if len(hist) != int(train_window):
            continue
        for horizon in horizons:
            horizon = int(horizon)
            weights = kernel_weights_horizon(h, train_window, horizon)
            forecast_daily = float(np.sum(hist * weights))
            realized_sum = float(targets.loc[v.index[pos], f"realized_var_sum_{horizon}"])
            records.append({"date": v.index[pos], "model": "rough_kernel", "horizon": horizon,
                            "forecast_var_daily": forecast_daily, "forecast_var_sum": forecast_daily * horizon,
                            "forecast_var_ann": annualization * forecast_daily,
                            "forecast_vol_ann": np.sqrt(annualization * forecast_daily),
                            "realized_var_sum": realized_sum,
                            "realized_var_ann": annualization * realized_sum / horizon,
                            "realized_vol_ann": np.sqrt(annualization * realized_sum / horizon) if realized_sum >= 0 else np.nan})
    return pd.DataFrame(records)

arch_specs = [
    {"name": "garch11_normal", "vol": "GARCH", "p": 1, "o": 0, "q": 1, "dist": "normal"},
    {"name": "gjr11_student", "vol": "GARCH", "p": 1, "o": 1, "q": 1, "dist": "t"},
    {"name": "egarch11_normal", "vol": "EGARCH", "p": 1, "o": 0, "q": 1, "dist": "normal"},
]

rough_fc = kernel_forecasts(spx_var, spx_targets, h_rv, horizons, train_window=1500, signal_step=21, annualization=ann_trading)
har_fc = rolling_har_forecasts(spx_ret, horizons=tuple(horizons), train_window=1500, refit_every=21,
                               annualization=ann_trading, use_log=True, forecast_start="2018-01-01")

try:
    arch_fc = rolling_arch_forecasts_weekly(spx_ret, specs=arch_specs, horizons=tuple(horizons),
                                            train_window=1500, signal_step=42, forecast_start="2020-01-01",
                                            annualization=ann_trading, maxiter=120, simulations=100, seed=seed)
except Exception as exc:
    arch_fc = pd.DataFrame([{"date": pd.NaT, "model": "arch_unavailable", "horizon": np.nan, "error": str(exc)}])

forecast_data = pd.concat([rough_fc, har_fc, arch_fc.dropna(subset=["horizon"], how="any")], ignore_index=True, sort=False)
forecast_scores = score_forecasts_by_model(forecast_data)
score_rank = forecast_scores.sort_values(["horizon", "qlike_var"]).groupby("horizon", as_index=False).head(1)
win_counts = pd.concat([
    forecast_scores.sort_values(["horizon", "qlike_var"]).groupby("horizon", as_index=False).head(1).assign(metric="qlike_var"),
    forecast_scores.sort_values(["horizon", "rmse_var"]).groupby("horizon", as_index=False).head(1).assign(metric="rmse_var"),
    forecast_scores.sort_values(["horizon", "mae_var"]).groupby("horizon", as_index=False).head(1).assign(metric="mae_var"),
]).groupby(["metric", "model"], as_index=False).size()

display(forecast_scores.sort_values(["horizon", "qlike_var"]).round(6))
display(win_counts)

	model	horizon	qlike_var	rmse_var	mae_var	rmse_vol	mae_vol	corr_vol	n_obs
0	rough_kernel	1	-8.105938	0.000199	0.000133	0.126905	0.106926	0.119574	359
1	garch11_normal	1	-8.016799	0.000198	0.000135	0.108184	0.091938	0.563862	36
2	egarch11_normal	1	-8.016272	0.000203	0.000137	0.107815	0.093576	0.573411	36
3	gjr11_student	1	-7.896466	0.000203	0.000141	0.111980	0.094171	0.536213	36
4	har_rv	1	-6.097290	0.000222	0.000104	0.112719	0.080295	0.575256	94
5	har_rv	5	-6.577817	0.001143	0.000374	0.074411	0.047946	0.693915	94
6	egarch11_normal	5	-6.287714	0.001787	0.000627	0.101513	0.059138	0.680662	36
7	garch11_normal	5	-6.278677	0.001704	0.000609	0.097704	0.056859	0.699585	36
8	rough_kernel	5	-6.265616	0.001468	0.000631	0.109714	0.078851	0.263733	359
9	gjr11_student	5	-6.088571	0.001727	0.000645	0.101089	0.060236	0.648584	36
10	har_rv	10	-5.563481	0.001979	0.000805	0.081296	0.050138	0.571335	94
11	garch11_normal	10	-5.562647	0.007861	0.001918	0.138243	0.062742	0.732434	36
12	egarch11_normal	10	-5.521373	0.008071	0.001936	0.144567	0.063965	0.702403	36
13	gjr11_student	10	-5.449142	0.007909	0.001977	0.142149	0.065394	0.641501	36
14	rough_kernel	10	-5.395537	0.003573	0.001330	0.113320	0.078765	0.189375	359
15	garch11_normal	21	-4.875284	0.010264	0.002930	0.106452	0.055895	0.707850	36
16	egarch11_normal	21	-4.841467	0.010715	0.003007	0.112858	0.058316	0.664192	36
17	gjr11_student	21	-4.799418	0.010320	0.003047	0.110002	0.060324	0.633449	36
18	rough_kernel	21	-4.727213	0.006020	0.002551	0.103491	0.074559	0.164289	359
19	har_rv	21	-4.277254	0.007111	0.002159	0.105588	0.055500	0.361614	94
20	garch11_normal	42	-4.185706	0.011229	0.004456	0.082794	0.054314	0.692892	36
21	egarch11_normal	42	-4.151026	0.012176	0.004742	0.090557	0.059255	0.619306	36
22	gjr11_student	42	-4.147192	0.011219	0.004690	0.085982	0.058387	0.633643	36
23	rough_kernel	42	-3.987199	0.010545	0.004954	0.100020	0.072731	0.108303	359
24	har_rv	42	-3.523940	0.012050	0.004225	0.103395	0.055707	0.212146	94
25	rough_kernel	63	-3.555048	0.014413	0.007344	0.098078	0.072428	0.065140	359
26	egarch11_normal	63	-3.336884	0.017894	0.007798	0.102952	0.063815	0.390479	36
27	garch11_normal	63	-3.317586	0.017100	0.007231	0.098356	0.057540	0.486140	36
28	gjr11_student	63	-3.284346	0.017103	0.007733	0.100495	0.062584	0.445684	36
29	har_rv	63	-3.091842	0.015895	0.006354	0.102130	0.057994	0.062348	94

	metric	model	size
0	mae_var	har_rv	6
1	qlike_var	garch11_normal	2
2	qlike_var	har_rv	2
3	qlike_var	rough_kernel	2
4	rmse_var	garch11_normal	1
5	rmse_var	har_rv	2
6	rmse_var	rough_kernel	3

fig, axes = plt.subplots(1, 3, figsize=(15, 3.6), constrained_layout=True)
heat = forecast_scores.pivot_table(index="model", columns="horizon", values="qlike_var", aggfunc="mean")
heat_norm = heat.div(heat.min(axis=0), axis=1)
im = axes[0].imshow(heat_norm.to_numpy(dtype=float), aspect="auto", cmap="viridis")
axes[0].set_xticks(range(len(heat.columns)))
axes[0].set_xticklabels(heat.columns)
axes[0].set_yticks(range(len(heat.index)))
axes[0].set_yticklabels(heat.index)
axes[0].set_title("Normalized QLIKE by Model and Horizon")
fig.colorbar(im, ax=axes[0], fraction=0.046, pad=0.04)
path21 = forecast_data[(forecast_data["model"].eq("rough_kernel")) & (forecast_data["horizon"].eq(21))].tail(180)
axes[1].plot(path21["date"], path21["forecast_vol_ann"], lw=1.0, label="rough forecast")
axes[1].plot(path21["date"], path21["realized_vol_ann"], lw=1.0, label="realized")
axes[1].set_title("21-Day Rough-Kernel Forecast")
axes[1].legend()
rank_frame = forecast_scores.copy()
rank_frame["qlike_rank"] = rank_frame.groupby("horizon")["qlike_var"].rank(method="dense")
for i, (model, g) in enumerate(rank_frame.groupby("model", sort=True)):
    axes[2].plot(g["horizon"], g["qlike_rank"], marker="o", lw=1.0, color=palette[i], label=model)
axes[2].invert_yaxis()
axes[2].set_title("QLIKE Rank by Horizon")
axes[2].set_xlabel("horizon")
axes[2].set_ylabel("rank")
axes[2].legend(fontsize=6)
plt.show()

The forecast comparison is mixed across the models, horizons, and losses that we use to evaluate them. The rough-kernel model is competitive, especially at very short horizons. In the SPX forecast table, the rough kernel ranks best by QLIKE at the 1-day horizon, while HAR and GARCH variants remain strong at some longer horizons and under other error metrics.

We can compare model behavior across horizons and losses in the heatmap and rank plot alongside the underlying forecast results. The rough-kernel forecast tracks volatility bursts better than a flat historical mean would, but it isn’t a universal winner. HAR is hard to beat because it’s a very strong low-parameter benchmark for realized variance. GARCH variants also do well in some horizons because they adapt quickly to recent shocks.

A roughness estimate alone doesn’t guarantee the best variance forecast at every horizon or under every loss. It gives us a way to encode the empirical scaling of variance paths, but forecast performance still depends on the proxy, the horizon, the loss function, and the market sample. The rough-kernel result is strongest when the problem is short-horizon variance timing, which is exactly where rough volatility should matter most.

The model ranking also depends on the loss function. QLIKE is variance-focused and strongly penalizes under-forecasting during high-volatility periods. RMSE can be dominated by crisis spikes. MAE is more robust but less sensitive to getting the volatility scale right in stress. Correlation tells us whether the forecast moves in the same direction as realized volatility, but a forecast can have decent correlation and still be badly scaled.

So we report several metrics:

\[ RMSE_v = \sqrt{\frac{1}{N}\sum_t(\widehat v_t-v_t)^2} \]

\[ MAE_v = \frac{1}{N}\sum_t|\widehat v_t-v_t| \]

\[ \rho_{\sigma} = \operatorname{Corr}(\widehat\sigma_t,\sigma_t) \]

The rough-kernel model is strongest when the market’s variance dynamics are shaped by rough short-time scaling. HAR remains a robust benchmark because it captures slow variance persistence very well. GARCH remains useful because it reacts recursively to recent shocks. Each forecast assigns memory differently, so the ranking also depends on which parts of the variance history help at a particular horizon.

6) Option Chain Cleaning and Implied-Volatility Surface Support

We now move from realized variance to option-implied information. The data and basic IV extraction are repeated from Projects 4 and 8, so we keep the discussion short. Our option-based test of rough volatility will use the short-maturity skew of the implied-volatility surface.

The cleaned option chain must support three things:

1. enough quotes across dates,
2. enough expiries per date,
3. enough moneyness range around the money to estimate ATM skew.

The moneyness coordinate is log moneyness:

\[ k = \log\left(\frac{K}{S}\right) \]

The implied-volatility surface is written as:

\[ \sigma_{\text{imp}} = \sigma_{\text{imp}}(k,\tau) \]

where \(\tau\) is time to maturity in years. We already built surfaces in Project 8, but here we use the surface for a different reason. Project 8 used the surface mainly for smooth interpolation, local volatility, and surface diagnostics. Here we need a smooth enough surface to estimate the derivative:

\[ \partial_k \sigma_{\text{imp}}(0,\tau) \]

That derivative is the ATM skew at maturity \(\tau\). We estimate it separately at each selected maturity. If the surface is noisy, the skew estimate becomes noisy too. Quote cleaning and surface smoothing directly affect the stability of the derivative and, in turn, the roughness estimate that we obtain from options.

6.1 From Clean Quotes to Surface Targets

We move from the raw option quotes to the implied volatilities needed for the surface fit in these steps:

\[ \text{raw quotes} \rightarrow \text{clean prices} \rightarrow \text{forward/rate/carry} \rightarrow \text{IV} \rightarrow \text{surface fit} \]

The forward is inferred from put-call parity where possible:

\[ C - P = D(F-K) \]

so:

\[ F = K + \frac{C-P}{D} \]

Once the forward and discount factor are available, the Black-76 implied volatility solves:

\[ C^{\text{Black76}}(F,K,\tau,\sigma,D) = C^{\text{market}} \]

We already solved IV in Project 4, but the rough-volatility use is stricter. A noisy IV quote away from the money may not hurt a broad surface visualization, but it can distort skew derivatives and calibration residuals. We filter extreme IV values, low weights, and poor quote regions before fitting the implied-volatility surface.

The option-surface section also prepares calibration grids. Instead of calibrating to every raw quote directly, we build a controlled surface target grid across moneyness and maturity. With the same grid, we can evaluate Heston and rough Heston across comparable moneyness and maturity regions without allowing the densest part of the raw chain to dominate the comparison.

option_dates = pd.DatetimeIndex(pd.to_datetime(spx_pairs_raw["date"].dropna().unique())).sort_values()
monthly_dates = pd.Series(option_dates).groupby(pd.Series(option_dates).dt.to_period("M")).last().to_list()
vol_on_option_dates = np.sqrt(ann_trading * spx_var).reindex(option_dates, method="nearest")
stress_dates = list(vol_on_option_dates.sort_values(ascending=False).head(8).index)
control_surface_dates = list(vol_on_option_dates.sort_values(ascending=True).head(4).index)
spaced_surface_dates = list(option_dates[np.unique(np.round(np.linspace(0, len(option_dates) - 1, min(24, len(option_dates)))).astype(int))])
surface_dates = pd.DatetimeIndex(sorted(set(monthly_dates).union(spaced_surface_dates).union(stress_dates).union(control_surface_dates)))
surface_dates = surface_dates[surface_dates.isin(option_dates)]

spx_pairs = spx_pairs_raw[spx_pairs_raw["date"].isin(surface_dates)].copy()
spx_pairs["dte_days"] = spx_pairs["tau"] * ann_days
spx_pairs["moneyness"] = spx_pairs["strike"] / spx_pairs["spot"]

rate_start = spx_pairs["date"].min() - pd.Timedelta(days=30)
rate_end = spx_pairs["date"].max()
us_par_options = us_par.loc[(us_par.index >= rate_start) & (us_par.index <= rate_end)].copy()

spx_pairs = attach_rates(spx_pairs, us_par_options, date_col="date", tau_col="tau", out_col="rate")
spx_pairs = add_discount_factors(spx_pairs, rate_col="rate", tau_col="tau", out_col="discount_factor")
forward_table = infer_forwards_from_paired_quotes(spx_pairs)
spx_pairs = spx_pairs.merge(forward_table[["date", "expiry", "forward", "implied_carry",
                                           "n_pairs", "parity_error_mad"]], on=["date", "expiry"], how="left")
spx_pairs = infer_dividend_yield_from_forward(spx_pairs, rate_col="rate", carry_col="implied_carry", out_col="implied_dividend_yield")

spx_long = wide_option_chain_to_long(spx_pairs, underlying_default="SPX")
for col in ["rate", "discount_factor", "forward", "implied_carry", "implied_dividend_yield", "n_pairs", "parity_error_mad"]:
    if col not in spx_long.columns and col in spx_pairs.columns:
        spx_long = spx_long.merge(spx_pairs[["date", "expiry", "strike", col]], on=["date", "expiry", "strike"], how="left")

clean_quotes, clean_steps = clean_option_quotes(spx_long, min_dte=7, max_dte=180, moneyness_range=(0.62, 1.62),
                                                max_relative_spread=0.60, closest_atm_pairs=70,
                                                min_pairs_per_expiry=5, annualization_days=ann_days)

for col in ["rate", "discount_factor", "forward", "implied_carry", "implied_dividend_yield", "n_pairs", "parity_error_mad"]:
    if col not in clean_quotes.columns and col in spx_pairs.columns:
        clean_quotes = clean_quotes.merge(spx_pairs[["date", "expiry", "strike", col]], on=["date", "expiry", "strike"], how="left")

display(clean_steps)
display(clean_quotes[["date", "expiry", "option_type", "spot", "strike", "tau", "mid", "forward", "rate"]].head(10))

	step	rows	removed
0	raw rows	269286	NaN
1	after schema normalization	269286	0.0
2	after positive price filter	269286	0.0
3	after bid/ask/spread filter	269286	0.0
4	after DTE filter	269286	0.0
5	after moneyness filter	269286	0.0
6	after ATM-pair selection	214444	54842.0
7	final rows	214444	0.0

	date	expiry	option_type	spot	strike	tau	mid	forward	rate
0	2022-01-03	2022-01-12	call	4795.57	4625.0	0.024641	171.85	4794.051044	0.0005
1	2022-01-03	2022-01-12	put	4795.57	4625.0	0.024641	3.95	4794.051044	0.0005
2	2022-01-03	2022-01-12	call	4795.57	4630.0	0.024641	168.25	4794.051044	0.0005
3	2022-01-03	2022-01-12	put	4795.57	4630.0	0.024641	4.15	4794.051044	0.0005
4	2022-01-03	2022-01-12	call	4795.57	4635.0	0.024641	163.45	4794.051044	0.0005
5	2022-01-03	2022-01-12	put	4795.57	4635.0	0.024641	4.35	4794.051044	0.0005
6	2022-01-03	2022-01-12	call	4795.57	4640.0	0.024641	158.70	4794.051044	0.0005
7	2022-01-03	2022-01-12	put	4795.57	4640.0	0.024641	4.60	4794.051044	0.0005
8	2022-01-03	2022-01-12	call	4795.57	4645.0	0.024641	153.95	4794.051044	0.0005
9	2022-01-03	2022-01-12	put	4795.57	4645.0	0.024641	4.85	4794.051044	0.0005

iv_quotes = implied_vol_table(clean_quotes, price_cols=("bid", "mid", "ask"), option_type_col="option_type",
                              spot_col="spot", forward_col="forward", strike_col="strike", tau_col="tau",
                              rate_col="rate", discount_col="discount_factor", model="black76", engine="auto")

iv_quotes = compute_greeks_numpy(iv_quotes.dropna(subset=["iv_mid"]).copy(), iv_col="iv_mid")

surface_quotes = surface_ready_quotes(iv_quotes, min_iv=0.02, max_iv=2.50, min_tau=7 / ann_days,
                                      max_tau=180 / ann_days, min_weight=0.05, max_weight=20.0)

coverage = surface_quotes.groupby("date").agg(quotes=("iv_mid", "size"), expiries=("expiry", "nunique"),
                                              median_spread=("relative_spread", "median"),
                                              median_iv=("iv_mid", "median")).reset_index()
display(coverage)
display(surface_quotes[["date", "expiry", "option_type", "strike", "k", "tau", "iv_mid", "surface_weight"]].head(10))

	date	quotes	expiries	median_spread	median_iv
0	2022-01-03	2939	21	0.014455	0.139032
1	2022-01-31	2800	20	0.014723	0.205393
2	2022-02-02	2940	21	0.016520	0.180581
3	2022-02-28	2908	21	0.015115	0.240580
4	2022-03-04	3183	23	0.026125	0.259087
5	2022-03-31	3075	22	0.023807	0.184119
6	2022-04-05	3189	23	0.013304	0.179744
7	2022-04-29	3026	22	0.015739	0.275974
8	2022-05-05	3204	23	0.020534	0.272724
9	2022-05-06	3176	23	0.020867	0.262823
10	2022-05-09	3167	23	0.015024	0.296987
11	2022-05-10	3172	23	0.008307	0.291013
12	2022-05-18	3220	23	0.013223	0.271109
13	2022-05-31	3237	24	0.036241	0.242981
14	2022-06-07	3335	24	0.008639	0.227736
15	2022-06-20	3256	24	0.027792	0.277284
16	2022-06-21	3148	23	0.007775	0.265804
17	2022-06-30	3339	24	0.009569	0.252772
18	2022-07-08	3451	25	0.007365	0.232176
19	2022-07-29	3330	24	0.010431	0.204510
20	2022-08-09	3329	24	0.008602	0.205141
21	2022-08-31	3296	24	0.013642	0.232431
22	2022-09-08	3499	25	0.010409	0.220674
23	2022-09-30	3874	28	0.010384	0.281878
24	2022-10-10	3944	29	0.013793	0.286277
25	2022-10-31	3849	28	0.020408	0.255909
26	2022-11-09	3995	29	0.011620	0.249936
27	2022-11-10	3991	29	0.009913	0.233744
28	2022-11-30	3990	29	0.010791	0.209797
29	2022-12-09	4133	30	0.010382	0.220349
30	2022-12-30	4128	30	0.021418	0.211735
31	2023-01-13	4129	30	0.012392	0.173259
32	2023-01-31	3861	28	0.010320	0.189199
33	2023-02-15	3965	29	0.011050	0.168052
34	2023-02-28	4106	30	0.017074	0.188681
35	2023-03-20	4114	30	0.008511	0.219341
36	2023-03-31	4292	31	0.011525	0.181047
37	2023-04-20	3995	29	0.012005	0.159342
38	2023-04-28	4010	29	0.018765	0.154835
39	2023-05-22	4126	30	0.009662	0.152818
40	2023-05-31	4017	29	0.028846	0.153285
41	2023-06-23	4149	30	0.016083	0.119761
42	2023-06-30	4028	29	0.034456	0.119387
43	2023-07-25	3969	29	0.015195	0.125571
44	2023-07-31	3944	29	0.015590	0.120372
45	2023-08-24	3951	29	0.017544	0.149737
46	2023-08-31	4087	30	0.017021	0.122703
47	2023-09-26	4481	33	0.027130	0.166632
48	2023-09-29	4348	32	0.015158	0.157248
49	2023-10-26	4337	32	0.012158	0.185806
50	2023-10-31	4242	32	0.011079	0.168573
51	2023-11-28	4362	33	0.015628	0.117922
52	2023-11-29	4412	33	0.017021	0.121320
53	2023-11-30	4483	33	0.028490	0.121778
54	2023-12-05	4539	34	0.027397	0.123207
55	2023-12-06	4574	34	0.017094	0.124338
56	2023-12-29	4486	33	0.026977	0.113624

	date	expiry	option_type	strike	k	tau	iv_mid	surface_weight
0	2022-01-03	2022-01-12	call	4625.0	-0.035899	0.024641	0.147003	0.115824
1	2022-01-03	2022-01-12	put	4625.0	-0.035899	0.024641	0.158925	0.057651
2	2022-01-03	2022-01-12	call	4630.0	-0.034819	0.024641	0.157572	0.143638
3	2022-01-03	2022-01-12	put	4630.0	-0.034819	0.024641	0.157107	0.064338
4	2022-01-03	2022-01-12	call	4635.0	-0.033740	0.024641	0.155625	0.139801
5	2022-01-03	2022-01-12	put	4635.0	-0.033740	0.024641	0.155177	0.071405
6	2022-01-03	2022-01-12	call	4640.0	-0.032661	0.024641	0.153994	0.139752
7	2022-01-03	2022-01-12	put	4640.0	-0.032661	0.024641	0.153565	0.163813
8	2022-01-03	2022-01-12	call	4645.0	-0.031584	0.024641	0.152229	0.132149
9	2022-01-03	2022-01-12	put	4645.0	-0.031584	0.024641	0.151817	0.090400

fig, axes = plt.subplots(1, 3, figsize=(15, 3.5), constrained_layout=True)
axes[0].bar(coverage["date"], coverage["quotes"], width=8, color=palette[0])
axes[0].set_title("Surface Quote Coverage by Date")
axes[1].bar(coverage["date"], coverage["expiries"], width=8, color=palette[3])
axes[1].set_title("Expiry Coverage by Date")
axes[2].scatter(surface_quotes["k"], surface_quotes["iv_mid"], s=4, alpha=0.25, color=palette[1])
axes[2].set_title("IV Quality and Smile Support")
axes[2].set_xlabel("log moneyness")
axes[2].set_ylabel("implied volatility")
plt.show()

The SPX option panel has enough coverage across the displayed dates, maturities, and moneyness regions for this exercise. Quote coverage is broad across selected dates, expiry coverage is stable enough, and the IV cloud has the usual equity-index shape: implied volatility is higher for negative log moneyness, especially on the put side.

We see the familiar equity-index skew structure across the quoted option maturities. Investors pay more for downside protection, dealers manage crash risk, and index returns have asymmetric tail risk. For rough volatility, the important part is how that skew changes as maturity becomes short. If short-dated skew explodes as \(\tau\) gets smaller, the model must explain a strong maturity-dependent slope near ATM.

7) Smooth Surface Fit and ATM Skew Power Law

We fit two versions of the surface: a visual/smoothed fit and a more regularized version for derivative-sensitive tasks. We don’t need to re-teach spline surfaces here. ATM skew is a derivative, so taking it across the surface can amplify noise in the fitted implied volatilities. A small wiggle in implied volatility can become a large error in \(\partial_k \sigma\).

The ATM skew at maturity \(\tau\) is estimated with a centered finite difference:

\[ \psi(\tau) = \partial_k \sigma_{\text{imp}}(0,\tau) \approx \frac{\sigma_{\text{imp}}(+\Delta k,\tau) - \sigma_{\text{imp}}(-\Delta k,\tau)}{2\Delta k} \]

In equity indices, \(\psi(\tau)\) is usually negative. The implied-volatility smile is higher for downside strikes, so as \(k\) increases from negative to positive, IV tends to fall.

Rough-volatility theory gives a short-maturity power law:

\[ |\psi(\tau)| \approx c \tau^{H - 1/2} \]

Since \(H < 1/2\), the exponent \(H - 1/2\) is negative. That means the skew becomes steeper as maturity goes to zero.

We fit the magnitude of the skew in the equivalent form:

\[ |\psi(\tau)| \approx c \tau^{-\alpha} \]

Taking logs gives:

\[ \log |\psi(\tau)| = \log c - \alpha \log \tau \]

So the fitted roughness estimate is:

\[ \widehat{H} = \frac{1}{2} - \widehat{\alpha} \]

For example, if the fitted slope gives \(\alpha = 0.25\), then:

\[ \widehat{H} = 0.50 - 0.25 = 0.25 \]

This is an option-implied estimate of roughness. It doesn’t have to match the realized-variance estimate exactly because it’s risk-neutral, maturity-smoothed, and extracted from option prices rather than historical returns.

7.1 Short-Maturity Skew as a Roughness Fingerprint

The ATM skew power law is one of the main reasons rough volatility became so important in option pricing. Classical diffusive stochastic-volatility models can generate skew, but their short-time asymptotics usually don’t produce the same steep power-law explosion.

The rough-volatility relationship can be written:

\[ \psi(\tau) = \partial_k \sigma_{\text{imp}}(0,\tau) \]

and for small \(\tau\):

\[ |\psi(\tau)| \approx c\tau^{H-\frac{1}{2}} \]

If \(H=0.5\), then:

\[ H-\frac{1}{2}=0 \]

so the short-end skew is roughly flat in maturity. If \(H=0.1\), then:

\[ H-\frac{1}{2}=-0.4 \]

so the skew behaves like:

\[ |\psi(\tau)| \approx c\tau^{-0.4} \]

Now compare two maturities, 7 days and 28 days. The maturity ratio is 4. With \(H=0.1\), the skew ratio is approximately:

\[ \left(\frac{7}{28}\right)^{-0.4} = 4^{0.4} \approx 1.74 \]

So the 7-day skew can be about 74% steeper than the 28-day skew under this rough scaling. That kind of short-end steepness is hard to ignore in equity-index options.

We estimate the decay parameter \(\alpha\) from the observed term structure in:

\[ |\psi(\tau)| = c\tau^{-\alpha} \]

and then maps:

\[ H = \frac{1}{2} - \alpha \]

This mapping is the bridge from a market-observable surface derivative to a roughness parameter.

The maturities included in the skew window affect the power-law estimate and its interpretation. Very short maturities can be noisy because of microstructure, pin risk, expiry effects, and low vega. Long maturities can dilute the rough short-time signature because the asymptotic relation is about \(\tau \to 0\).

So we test several windows, such as 7–45 days, 7–60 days, 14–90 days, and 14–150 days. If the estimated \(H\) changes drastically across windows, the power-law evidence is unstable. If the estimate is consistent and the \(R^2\) is high, the skew roughness is more trustworthy.

surface_fit = fit_surface_panel(surface_quotes, date_col="date", k_col="k", tau_col="tau",
                                iv_col="iv_mid", weight_col="surface_weight",
                                visual_params=dict(n_k_basis=12, n_tau_basis=8, degree=3, lambda_k=3.0, lambda_tau=2.0),
                                dupire_params=dict(n_k_basis=10, n_tau_basis=7, degree=3, lambda_k=12.0, lambda_tau=20.0),
                                min_quotes=100, min_expiries=4)

fit_summary = surface_fit["fit_summary"].copy()
main_date = fit_summary[fit_summary["fit"].eq("visual")].sort_values(["weighted_rmse", "quote_count"], ascending=[True, False])["date"].iloc[0]
main_date = pd.Timestamp(main_date).normalize()
q_main = surface_quotes[surface_quotes["date"].eq(main_date)].copy()
fit_visual = surface_fit["visual_fits"][main_date]
k_grid = surface_grid(q_main, k_col="k", tau_col="tau", k_quantiles=(0.02, 0.98),
                      tau_quantiles=(0.02, 0.98), n_k=65, n_tau=35, annualization_days=ann_days)
sigma_grid = surface_iv_grid(fit_visual, k_grid)
surface_err = surface_residuals(q_main, fits={"smooth": fit_visual}, k_col="k", tau_col="tau", iv_col="iv_mid")

display(fit_summary.sort_values(["date", "fit"]).round(5))

C:\Users\Lenovo\AppData\Local\Temp\ipykernel_16732\1977237117.py:23: UserWarning: obj.round has no effect with datetime, timedelta, or period dtypes. Use obj.dt.round(...) instead.
  display(fit_summary.sort_values(["date", "fit"]).round(5))

	fit	target	quote_count	number_of_maturities	rmse	weighted_rmse	mae	median_abs_error	p95_abs_error	mean_residual	residual_std	k_min	k_max	tau_min	tau_max	date
0	dupire	log_total_variance	2939	21	0.00520	0.00388	0.00360	0.00256	0.00979	-0.00092	0.00512	-0.06503	0.06725	0.02464	0.48734	2022-01-03
1	visual	log_total_variance	2939	21	0.00342	0.00179	0.00192	0.00126	0.00524	-0.00011	0.00342	-0.06503	0.06725	0.02464	0.48734	2022-01-03
2	dupire	log_total_variance	2800	20	0.00557	0.00403	0.00365	0.00234	0.01322	-0.00101	0.00548	-0.09944	0.10390	0.02464	0.45175	2022-01-31
3	visual	log_total_variance	2800	20	0.00337	0.00237	0.00227	0.00132	0.00756	-0.00017	0.00336	-0.09944	0.10390	0.02464	0.45175	2022-01-31
4	dupire	log_total_variance	2940	21	0.00447	0.00366	0.00326	0.00250	0.01008	-0.00095	0.00437	-0.09105	0.08807	0.02464	0.48460	2022-02-02
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
109	visual	log_total_variance	4539	34	0.00712	0.00294	0.00257	0.00106	0.00774	0.00013	0.00712	-0.42772	0.12232	0.02190	0.48734	2023-12-05
110	dupire	log_total_variance	4574	34	0.00581	0.00275	0.00275	0.00169	0.00733	0.00049	0.00579	-0.42417	0.12588	0.02190	0.48460	2023-12-06
111	visual	log_total_variance	4574	34	0.00506	0.00210	0.00200	0.00107	0.00505	0.00027	0.00505	-0.42417	0.12588	0.02190	0.48460	2023-12-06
112	dupire	log_total_variance	4486	33	0.00900	0.00401	0.00340	0.00149	0.01093	0.00011	0.00900	-0.46941	0.10011	0.02738	0.47912	2023-12-29
113	visual	log_total_variance	4486	33	0.00869	0.00374	0.00297	0.00110	0.00871	0.00002	0.00869	-0.46941	0.10011	0.02738	0.47912	2023-12-29

114 rows × 16 columns

fig, axes = plt.subplots(1, 3, figsize=(15, 3.6), constrained_layout=True)
for i, d in enumerate([14, 30, 60, 90, 150]):
    tau = d / ann_days
    k_line = np.linspace(k_grid["k"].min(), k_grid["k"].max(), 80)
    axes[0].plot(k_line, surface_iv(fit_visual, k_line, np.full_like(k_line, tau)), lw=1.1, color=palette[i], label=f"{d}d")
axes[0].scatter(q_main["k"], q_main["iv_mid"], s=5, alpha=0.18, color="black")
axes[0].set_title(f"Smooth SPX Smiles {main_date.date()}")
axes[0].legend(ncol=2)
im = axes[1].imshow(sigma_grid, origin="lower", aspect="auto",
                    extent=[k_grid["k"].min(), k_grid["k"].max(),
                            k_grid["tau"].min() * ann_days, k_grid["tau"].max() * ann_days], cmap="magma")
axes[1].set_title("Smooth IV Surface")
axes[1].set_xlabel("log moneyness")
axes[1].set_ylabel("days")
fig.colorbar(im, ax=axes[1], fraction=0.046, pad=0.04)
axes[2].scatter(surface_err["k"], surface_err["residual_smooth"], s=7, alpha=0.35, color=palette[2])
axes[2].axhline(0.0, color="black", lw=0.8)
axes[2].set_title("Surface Residuals")
plt.show()

def atm_skew(q, fit, tau_values, dk=0.01):
    rows = []
    for tau in tau_values:
        left = float(surface_iv(fit, np.array([-dk]), np.array([tau]))[0])
        right = float(surface_iv(fit, np.array([dk]), np.array([tau]))[0])
        atm = float(surface_iv(fit, np.array([0.0]), np.array([tau]))[0])
        rows.append({"tau": float(tau), "tau_days": float(tau * ann_days), "atm_iv": atm,
                     "atm_skew": (right - left) / (2.0 * dk),
                     "n": int(len(q[np.isclose(q["tau"], tau, atol=3 / ann_days)]))})
    return pd.DataFrame(rows)


def power_law(psi, tau_min_days=None, tau_max_days=None, window=""):
    q = psi.replace([np.inf, -np.inf], np.nan).dropna(subset=["tau", "atm_skew"]).copy()
    if tau_min_days is not None:
        q = q[q["tau_days"].ge(float(tau_min_days))].copy()
    if tau_max_days is not None:
        q = q[q["tau_days"].le(float(tau_max_days))].copy()
    q = q[(q["tau"] > 0) & (q["atm_skew"].abs() > 0)]
    if len(q) < 3:
        return pd.DataFrame([{"window": window, "c": np.nan, "alpha": np.nan, "h_from_skew": np.nan, "r2": np.nan, "n": int(len(q))}])
    x = np.log(q["tau"].to_numpy(dtype=float))
    y = np.log(q["atm_skew"].abs().to_numpy(dtype=float))
    design = np.column_stack([np.ones(len(x)), x])
    beta, _, _, _ = np.linalg.lstsq(design, y, rcond=None)
    fitted = design @ beta
    ss_res = float(np.sum((y - fitted) ** 2))
    ss_tot = float(np.sum((y - y.mean()) ** 2))
    alpha = -float(beta[1])
    return pd.DataFrame([{"window": window, "c": float(np.exp(beta[0])), "alpha": alpha,
                          "h_from_skew": 0.5 - alpha, "r2": 1.0 - ss_res / ss_tot, "n": int(len(q))}])

skew_windows = {"short_1_7_45": (7, 45), "short_2_7_60": (7, 60), "medium_14_90": (14, 90),
                "old_14_150": (14, 150)}

tau_values = np.array([7, 10, 14, 21, 30, 45, 60, 90, 120, 150], dtype=float) / ann_days
psi_main = atm_skew(q_main, fit_visual, tau_values)
skew_fit_main_by_window = pd.concat([
    power_law(psi_main, tau_min_days=lo, tau_max_days=hi, window=name)
    for name, (lo, hi) in skew_windows.items()
], ignore_index=True)
skew_fit_main = skew_fit_main_by_window[skew_fit_main_by_window["window"].eq("short_2_7_60")].copy()
if skew_fit_main["r2"].fillna(0.0).iloc[0] < 0.80:
    skew_fit_main = skew_fit_main_by_window.sort_values(["r2", "n"], ascending=[False, False]).head(1).copy()

skew_rows = []
for d, fit in surface_fit["visual_fits"].items():
    qd = surface_quotes[surface_quotes["date"].eq(d)].copy()
    psi_d = atm_skew(qd, fit, tau_values)
    for name, (lo, hi) in skew_windows.items():
        fit_d = power_law(psi_d, tau_min_days=lo, tau_max_days=hi, window=name)
        row = fit_d.iloc[0].to_dict()
        row["date"] = d
        skew_rows.append(row)
skew_history_all = pd.DataFrame(skew_rows).sort_values(["date", "window"])
skew_history = skew_history_all[skew_history_all["window"].eq("short_2_7_60")].copy()
valid_skew_dates = skew_history[(skew_history["r2"].ge(0.80)) & (skew_history["n"].ge(6))].copy()
skew_quality_summary = pd.DataFrame([{"window": "short_2_7_60",
                                      "median_h_all_dates": float(skew_history["h_from_skew"].median()),
                                      "median_h_high_quality_dates": float(valid_skew_dates["h_from_skew"].median()),
                                      "dates": int(len(skew_history)),
                                      "high_quality_dates": int(len(valid_skew_dates)),
                                      "rejected_dates": int(len(skew_history) - len(valid_skew_dates))}])

display(psi_main.round(5))
display(skew_fit_main_by_window[["window", "alpha", "h_from_skew", "r2", "n"]].round(4))
display(skew_history[["date", "window", "alpha", "h_from_skew", "r2", "n"]].round(4))
display(skew_quality_summary.round(4))

	tau	tau_days	atm_iv	atm_skew	n
0	0.01916	7.0	0.12127	-1.48860	139
1	0.02738	10.0	0.10650	-1.29444	279
2	0.03833	14.0	0.10757	-1.25572	420
3	0.05749	21.0	0.11365	-1.23274	420
4	0.08214	30.0	0.12196	-1.19868	280
5	0.12320	45.0	0.13212	-1.09725	140
6	0.16427	60.0	0.14050	-0.99334	0
7	0.24641	90.0	0.15403	-0.82747	140
8	0.32854	120.0	0.16431	-0.72585	0
9	0.41068	150.0	0.17091	-0.65616	140

	window	alpha	h_from_skew	r2	n
0	short_1_7_45	0.1004	0.3996	0.9039	5
1	short_2_7_60	0.1358	0.3642	0.8779	6
2	medium_14_90	0.2731	0.2269	0.9275	5
3	old_14_150	0.3359	0.1641	0.9557	7

C:\Users\Lenovo\AppData\Local\Temp\ipykernel_16732\2677019905.py:41: UserWarning: obj.round has no effect with datetime, timedelta, or period dtypes. Use obj.dt.round(...) instead.
  display(skew_history[["date", "window", "alpha", "h_from_skew", "r2", "n"]].round(4))

	date	window	alpha	h_from_skew	r2	n
1	2022-01-03	short_2_7_60	0.1358	0.3642	0.8779	6
5	2022-01-31	short_2_7_60	0.2657	0.2343	0.9969	6
9	2022-02-02	short_2_7_60	0.2146	0.2854	0.9889	6
13	2022-02-28	short_2_7_60	0.3878	0.1122	0.9994	6
17	2022-03-04	short_2_7_60	0.3519	0.1481	0.9928	6
21	2022-03-31	short_2_7_60	0.0950	0.4050	0.8660	6
25	2022-04-05	short_2_7_60	0.0747	0.4253	0.6954	6
29	2022-04-29	short_2_7_60	0.2996	0.2004	0.9797	6
33	2022-05-05	short_2_7_60	0.1939	0.3061	0.9012	6
37	2022-05-06	short_2_7_60	0.1398	0.3602	0.7573	6
41	2022-05-09	short_2_7_60	0.1978	0.3022	0.9822	6
45	2022-05-10	short_2_7_60	0.1231	0.3769	0.8809	6
49	2022-05-18	short_2_7_60	0.2422	0.2578	0.9926	6
53	2022-05-31	short_2_7_60	0.0612	0.4388	0.8330	6
57	2022-06-07	short_2_7_60	0.1412	0.3588	0.9828	6
61	2022-06-20	short_2_7_60	0.0372	0.4628	0.3323	6
65	2022-06-21	short_2_7_60	0.0145	0.4855	0.0674	6
69	2022-06-30	short_2_7_60	0.1351	0.3649	0.9786	6
73	2022-07-08	short_2_7_60	0.0565	0.4435	0.5055	6
77	2022-07-29	short_2_7_60	-0.1142	0.6142	0.5169	6
81	2022-08-09	short_2_7_60	0.1712	0.3288	0.8493	6
85	2022-08-31	short_2_7_60	0.0560	0.4440	0.3056	6
89	2022-09-08	short_2_7_60	0.0979	0.4021	0.8451	6
93	2022-09-30	short_2_7_60	0.0172	0.4828	0.0859	6
97	2022-10-10	short_2_7_60	0.0778	0.4222	0.8261	6
101	2022-10-31	short_2_7_60	0.0683	0.4317	0.5020	6
105	2022-11-09	short_2_7_60	-0.0617	0.5617	0.8833	6
109	2022-11-10	short_2_7_60	0.0847	0.4153	0.4159	6
113	2022-11-30	short_2_7_60	0.0288	0.4712	0.3929	6
117	2022-12-09	short_2_7_60	-0.0314	0.5314	0.5765	6
121	2022-12-30	short_2_7_60	0.1282	0.3718	0.9975	6
125	2023-01-13	short_2_7_60	0.0673	0.4327	0.6343	6
129	2023-01-31	short_2_7_60	0.1539	0.3461	0.9465	6
133	2023-02-15	short_2_7_60	0.1234	0.3766	0.8716	6
137	2023-02-28	short_2_7_60	0.0573	0.4427	0.2816	6
141	2023-03-20	short_2_7_60	0.3651	0.1349	0.9999	6
145	2023-03-31	short_2_7_60	0.0319	0.4681	0.5929	6
149	2023-04-20	short_2_7_60	0.0989	0.4011	0.5843	6
153	2023-04-28	short_2_7_60	0.0769	0.4231	0.8480	6
157	2023-05-22	short_2_7_60	0.0697	0.4303	0.4168	6
161	2023-05-31	short_2_7_60	0.1133	0.3867	0.7306	6
165	2023-06-23	short_2_7_60	-0.1343	0.6343	0.8641	6
169	2023-06-30	short_2_7_60	-0.2438	0.7438	0.9756	6
173	2023-07-25	short_2_7_60	0.0420	0.4580	0.9191	6
177	2023-07-31	short_2_7_60	-0.0651	0.5651	0.7941	6
181	2023-08-24	short_2_7_60	0.0388	0.4612	0.1915	6
185	2023-08-31	short_2_7_60	-0.0803	0.5803	0.4305	6
189	2023-09-26	short_2_7_60	0.2063	0.2937	0.9975	6
193	2023-09-29	short_2_7_60	0.2217	0.2783	0.9973	6
197	2023-10-26	short_2_7_60	0.1527	0.3473	0.9569	6
201	2023-10-31	short_2_7_60	0.1959	0.3041	0.9993	6
205	2023-11-28	short_2_7_60	0.0910	0.4090	0.9104	6
209	2023-11-29	short_2_7_60	0.1111	0.3889	0.9684	6
213	2023-11-30	short_2_7_60	0.0893	0.4107	0.9578	6
217	2023-12-05	short_2_7_60	-0.0355	0.5355	0.5278	6
221	2023-12-06	short_2_7_60	0.0464	0.4536	0.5612	6
225	2023-12-29	short_2_7_60	-0.1395	0.6395	0.9730	6

	window	median_h_all_dates	median_h_high_quality_dates	dates	high_quality_dates	rejected_dates
0	short_2_7_60	0.4107	0.3645	57	34	23

fig, axes = plt.subplots(1, 3, figsize=(15, 3.6), constrained_layout=True)
axes[0].plot(psi_main["tau_days"], psi_main["atm_skew"], marker="o", lw=1.2, color=palette[0])
axes[0].set_title("ATM Skew Term Structure")
axes[0].set_xlabel("days")
axes[0].set_ylabel("d sigma / dk")
axes[1].scatter(psi_main["tau"], psi_main["atm_skew"].abs(), s=30, color=palette[1])
xs = np.linspace(psi_main["tau"].min(), psi_main["tau"].max(), 80)
axes[1].plot(xs, skew_fit_main["c"].iloc[0] * xs ** (-skew_fit_main["alpha"].iloc[0]), color=palette[2], lw=1.3,
             label=str(skew_fit_main["window"].iloc[0]))
axes[1].set_xscale("log")
axes[1].set_yscale("log")
axes[1].set_title("ATM Skew Power Law")
axes[1].legend()
axes[2].plot(skew_history["date"], skew_history["h_from_skew"], marker="o", ms=3, lw=0.8, alpha=0.45, color=palette[3], label="all dates")
axes[2].plot(valid_skew_dates["date"], valid_skew_dates["h_from_skew"], marker="o", ms=4, lw=1.1, color=palette[0], label="R2>=0.80")
axes[2].axhline(h_rv, color=palette[1], lw=1.1, ls="--", label="RV roughness")
axes[2].set_title("Quality-Filtered Skew-Implied H")
axes[2].legend()
plt.show()

In the smooth SPX smiles we can see the familiar pattern: short maturities have the steepest left skew, while the longer-maturity smiles flatten. The residual panel is important too. The surface fit is close enough for extracting a rough skew slope, but there are still residuals, especially where quote quality is weaker or smile curvature changes quickly.

The ATM skew term structure falls in absolute value as maturity increases. The log-log power-law fit is visually close on high-quality dates. For the main SPX date, the fitted short-window skew implies a roughness estimate around the mid-to-high 0.3 range depending on the window. The high-quality filter over the full sample gives a median \(H\) around 0.365.

This is higher than the realized-variance roughness estimate around 0.13, and that difference is meaningful. Option-implied skew isn’t spot log variance observed at daily frequency. It’s an implied quantity averaged over maturity and shaped by risk premia, dealer positioning, downside demand, and smoothing across strikes. Option-implied roughness often comes out less rough than high-frequency realized-volatility estimates.

The reported high-quality date filter keeps dates with enough maturities and strong power-law fit. Comparing the retained dates lets us check whether the power-law fit appears consistently across the sample instead of relying on a single selected surface. Across the sample-level results, we can check whether the roughness signature appears repeatedly on the retained option dates.

8) Heston as the Markovian Control Model

Before going into rough models, we fit Heston as the classical stochastic-volatility control model. Heston was already explained in Project 11, so here we only need the structure that matters for comparison.

Under a risk-neutral measure, the Heston model is:

\[ dS_t = (r-q)S_t\,dt + \sqrt{V_t}S_t\,dW_t^S \]

\[ dV_t = \kappa(\theta - V_t)\,dt + \sigma_v\sqrt{V_t}\,dW_t^V \]

with:

\[ d\langle W^S, W^V\rangle_t = \rho\,dt \]

The parameters mean:

• \(V_0\): current instantaneous variance,
• \(\kappa\): speed of mean reversion,
• \(\theta\): long-run variance level,
• \(\sigma_v\): volatility of variance,
• \(\rho\): leverage correlation between spot shocks and variance shocks.

The model is Markovian because the state at time \(t\) is just \((S_t,V_t)\). Once we know those two values, the future distribution doesn’t need the full path history. This makes Heston fast and elegant, especially with Fourier/COS pricing from Project 14. But that Markovian structure also limits the short-time skew behavior. A finite-dimensional diffusion has a different short-maturity slope shape from rough-volatility models.

The calibration objective is a weighted price error:

\[ \min_\Theta \sum_{i=1}^{N} w_i\left(C_i^{\text{model}}(\Theta) - C_i^{\text{market}}\right)^2 \]

where \(\Theta=(V_0,\kappa,\theta,\sigma_v,\rho)\), and the weights account for quote quality and scale. We use Heston as a disciplined baseline: if rough models can’t improve on it in the right places, then their extra complexity isn’t justified.

8.1 Parameter Identifiability in Heston

Heston calibration is famous for parameter interaction. Different parameter sets can produce similar option surfaces, especially if the calibration window is short or the available maturities are limited.

For example:

• increasing \(\sigma_v\) can steepen skew and increase curvature,
• making \(\rho\) more negative also steepens downside skew,
• changing \(\kappa\) and \(\theta\) can shift how variance term structure behaves,
• \(V_0\) mainly controls the very short end.

This means a calibrated Heston parameter isn’t always a clean economic estimate. It’s partly a coordinate system for fitting the surface.

A useful way to think about Heston parameter roles is:

\[ \text{level} \approx V_0,\theta \]

\[ \text{term structure} \approx \kappa,\theta,V_0 \]

\[ \text{skew} \approx \rho,\sigma_v \]

\[ \text{curvature} \approx \sigma_v,\rho,\kappa \]

The rough-Heston model adds \(H\), which affects short-time regularity and skew scaling. This can reduce pressure on \(\rho\) and \(\sigma_v\) in some regions, but it also adds another dimension of calibration risk. We therefore compare the fitted parameters alongside pricing and skew diagnostics across strikes and maturities.

def surface_calib_grid(quotes, fits, dte_targets=(7, 10, 14, 21, 30, 45, 60, 90, 120, 150), min_quotes_per_date=45, ann=ann_days):
    fit_map = {pd.Timestamp(k).normalize(): v for k, v in fits.items()}
    x = quotes.copy()
    x["date"] = pd.to_datetime(x["date"], errors="coerce").dt.normalize()
    x["expiry"] = pd.to_datetime(x["expiry"], errors="coerce").dt.normalize()
    rows = []
    for d, day in x.groupby("date", sort=True):
        d = pd.Timestamp(d).normalize()
        fit = fit_map.get(d)
        if fit is None:
            continue
        k_support = pd.to_numeric(day["k"], errors="coerce")
        tau_support = pd.to_numeric(day["tau"], errors="coerce")
        ok = np.isfinite(k_support) & np.isfinite(tau_support)
        if ok.sum() < 20:
            continue
        k_lo, k_hi = np.nanquantile(k_support[ok], [0.02, 0.98])
        tau_lo, tau_hi = np.nanmin(tau_support[ok]), np.nanmax(tau_support[ok])
        spot = float(pd.to_numeric(day["spot"], errors="coerce").median())
        if not np.isfinite(spot) or spot <= 0:
            continue
        buf = 0.025
        k_low_d = max(k_lo + buf, -0.35)
        k_high_d = min(k_hi - buf, 0.25)
        if k_high_d <= k_low_d:
            k_low_d, k_high_d = max(k_lo, -0.35), min(k_hi, 0.25)
        spread = k_high_d - k_low_d
        n_k_nodes = 11 if spread > 0.30 else (9 if spread > 0.15 else 7)
        date_k_targets = np.linspace(k_low_d, k_high_d, n_k_nodes).tolist()
        if k_low_d < 0.0 < k_high_d and 0.0 not in date_k_targets:
            date_k_targets = sorted(set(date_k_targets) | {0.0})
        date_rows = []
        for dte in dte_targets:
            tau = float(dte) / float(ann)
            if tau < tau_lo - 4.0 / ann or tau > tau_hi + 4.0 / ann:
                continue
            near = day.assign(_gap=(pd.to_numeric(day["tau"], errors="coerce") - tau).abs()).sort_values("_gap").head(max(12, min(40, len(day) // 6)))
            rate = float(pd.to_numeric(near.get("rate", pd.Series(0.0, index=near.index)), errors="coerce").median())
            div = float(pd.to_numeric(near.get("implied_dividend_yield",
                                                near.get("dividend_yield", pd.Series(0.0, index=near.index))),
                                      errors="coerce").median())
            discount = float(pd.to_numeric(near.get("discount_factor", pd.Series(np.exp(-rate * tau), index=near.index)), errors="coerce").median())
            forward = float(pd.to_numeric(near.get("forward", pd.Series(spot * np.exp((rate - div) * tau), index=near.index)), errors="coerce").median())
            if not np.isfinite(forward) or forward <= 0:
                forward = spot * np.exp((rate - div) * tau)
            if not np.isfinite(discount) or discount <= 0:
                discount = np.exp(-rate * tau)
            iv_unc = pd.to_numeric(near.get("iv_uncertainty", pd.Series(np.nan, index=near.index)), errors="coerce")
            source_unc = float(np.nanmedian(iv_unc)) if iv_unc.notna().any() else 0.015
            source_unc = float(np.clip(source_unc, 0.008, 0.08))
            quality = float(np.nanmedian(pd.to_numeric(near.get("surface_weight", pd.Series(1.0, index=near.index)), errors="coerce")))
            quality = float(np.clip(quality if np.isfinite(quality) else 1.0, 0.25, 4.0))
            for kt in date_k_targets:
                if kt < k_lo - 0.025 or kt > k_hi + 0.025:
                    continue
                sigma = float(surface_iv(fit, np.array([kt]), np.array([tau]))[0])
                if not np.isfinite(sigma):
                    continue
                sigma = float(np.clip(sigma, 0.02, 4.0))
                strike = float(forward * np.exp(kt))
                option_type = "put" if kt < 0.0 else "call"
                mid = float(black76_price(option_type, forward, strike, tau, sigma, discount))
                vega = float(black76_vega(forward, strike, tau, sigma, discount))
                if not np.isfinite(mid) or not np.isfinite(vega) or mid <= 0 or vega <= 0:
                    continue
                scale = max(vega * source_unc, 1e-5 * spot, 1e-4 * mid)
                half = max(0.5 * scale, 2e-6 * spot, 1e-5 * mid)
                maturity_weight = float(np.clip(np.sqrt(30.0 / max(float(dte), 3.0)), 0.65, 1.65))
                moneyness_weight = float(np.clip(np.exp(-max(abs(kt) - 0.22, 0.0) / 0.25), 0.45, 1.15))
                date_rows.append({
                    "date": d, "expiry": d + pd.Timedelta(days=float(dte)), "target_source": "surface_grid",
                    "quote_id": f"{d.date()}_{dte}_{kt:.3f}_{option_type}", "spot": spot, "forward": forward,
                    "strike": strike, "option_type": option_type, "tau": tau, "dte_days": float(dte),
                    "rate": rate, "discount_factor": discount, "dividend_yield": div, "implied_dividend_yield": div,
                    "k": float(kt), "log_moneyness": float(np.log(strike / spot)), "moneyness": float(strike / spot),
                    "iv_mid": sigma, "iv_bid": max(sigma - source_unc, 1e-6), "iv_ask": sigma + source_unc,
                    "vega": vega, "mid": mid, "bid": max(mid - half, 0.0), "ask": mid + half,
                    "half_spread": half, "relative_spread": 2.0 * half / mid, "rel_spread": 2.0 * half / mid,
                    "iv_target_error": source_unc, "calib_scale_px": scale,
                    "quote_quality_weight": quality, "maturity_weight": maturity_weight, "moneyness_weight": moneyness_weight,
                    "obs_weight": quality * maturity_weight * moneyness_weight, "source_quote_count": int(len(day)),
                })
        if len(date_rows) >= min_quotes_per_date:
            out_day = pd.DataFrame(date_rows)
            out_day["obs_weight"] = out_day["obs_weight"] / max(float(np.nanmedian(out_day["obs_weight"])), 1e-12)
            rows.append(out_day)
    return calibration_weights(pd.concat(rows, ignore_index=True)) if rows else pd.DataFrame()


calibration_quotes = surface_calib_grid(
    surface_quotes,
    surface_fit["visual_fits"],
    min_quotes_per_date=45,
)

date_counts = calibration_quotes.groupby("date").size()
eligible_fit_dates = pd.DatetimeIndex(date_counts[date_counts >= 45].index).sort_values()
monthly_fit_dates = pd.Series(eligible_fit_dates).groupby(pd.Series(eligible_fit_dates).dt.to_period("M")).last().to_list()
vol_fit_dates = np.sqrt(ann_trading * spx_var).reindex(eligible_fit_dates, method="nearest")
stress_fit_dates = list(vol_fit_dates.sort_values(ascending=False).head(6).index)
control_fit_dates = list(vol_fit_dates.sort_values(ascending=True).head(4).index)
def evenly_spaced_dates(dates, n):
    dates = pd.DatetimeIndex(dates).sort_values()
    if len(dates) <= int(n):
        return list(dates)
    loc = np.unique(np.round(np.linspace(0, len(dates) - 1, int(n))).astype(int))
    return list(dates[loc])


spaced_fit_dates = evenly_spaced_dates(eligible_fit_dates, 24)
fit_dates = pd.DatetimeIndex(sorted(set(monthly_fit_dates).union(spaced_fit_dates).union(stress_fit_dates).union(control_fit_dates)))
fit_dates = fit_dates[fit_dates.isin(eligible_fit_dates)]
rough_priority_dates = pd.DatetimeIndex(sorted(set(stress_fit_dates).union(control_fit_dates).union([eligible_fit_dates[-1]]))).intersection(eligible_fit_dates)
rough_background_dates = [d for d in fit_dates if d not in set(rough_priority_dates)]
rough_fill_count = max(1, 18 - len(rough_priority_dates))
rough_fit_candidates = set(rough_priority_dates).union(evenly_spaced_dates(rough_background_dates, rough_fill_count))
rough_fit_dates = pd.DatetimeIndex(sorted(rough_fit_candidates))
rough_fit_dates = rough_fit_dates[rough_fit_dates.isin(eligible_fit_dates)]
heston_cache = cache_dir / "spx_heston_surface_grid_daily.parquet"
heston_fit_cache = cache_dir / "spx_heston_surface_grid_fit.parquet"

if heston_cache.exists() and heston_fit_cache.exists():
    heston_daily = pd.read_parquet(heston_cache)
    heston_prices = pd.read_parquet(heston_fit_cache)
else:
    heston_out = fit_daily_models(calibration_quotes, ["heston"], calibration_dates=fit_dates, min_quotes=80,
                                  max_nfev=80, engine="cpp", n_terms=160, truncation_width=16.0)
    heston_daily = heston_out["params"]
    heston_prices = heston_out["fit"]
    heston_daily.to_parquet(heston_cache, index=False)
    heston_prices.to_parquet(heston_fit_cache, index=False)

heston_table = compare_fourier_models(daily=heston_daily)
heston_buckets = residual_by_bucket(heston_prices)
display(heston_daily.round(5))
display(heston_table.round(5))

C:\Users\Lenovo\AppData\Local\Temp\ipykernel_16732\3151898688.py:22: UserWarning: obj.round has no effect with datetime, timedelta, or period dtypes. Use obj.dt.round(...) instead.
  display(heston_daily.round(5))

	model	date	success	nfev	runtime_sec	quotes	weighted_price_rmse	median_abs_price_error	weighted_iv_rmse	bid_ask_hit_rate	otm_put_rmse	short_maturity_rmse	p0	p1	p2	p3	p4	feller_ratio
0	heston	2022-01-03	True	37	11.11105	80	0.42381	1.25085	0.00465	0.7750	NaN	0.54331	0.01280	4.26527	0.07553	1.60128	-0.68698	0.25129
1	heston	2022-01-31	True	24	10.63544	80	0.91599	2.23194	0.00805	0.5500	NaN	1.20163	0.06693	12.00000	0.06565	3.37697	-0.60233	0.13816
2	heston	2022-02-02	True	12	4.78645	80	0.76823	1.98318	0.00736	0.5625	NaN	1.00653	0.03781	9.94753	0.06000	2.49853	-0.62883	0.19122
3	heston	2022-02-28	True	14	3.73290	80	0.85625	2.39582	0.01057	0.4750	NaN	1.10421	0.09241	12.00000	0.07146	2.85501	-0.69212	0.21039
4	heston	2022-03-31	True	12	5.39115	100	0.96335	1.86731	0.00978	0.5200	1.71895	1.14542	0.03134	10.80840	0.06538	2.25274	-0.71222	0.27851
5	heston	2022-04-29	True	10	3.05116	80	1.70479	2.92045	0.01419	0.4250	NaN	2.37252	0.14557	12.00000	0.09149	4.00000	-0.58406	0.13724
6	heston	2022-05-05	True	8	2.51283	80	0.65653	2.07177	0.01070	0.6625	NaN	0.77727	0.11850	12.00000	0.07985	2.90385	-0.60129	0.22726
7	heston	2022-05-09	True	11	3.19299	80	1.13001	2.88896	0.01027	0.4125	NaN	1.49971	0.15720	12.00000	0.09700	4.00000	-0.47022	0.14551
8	heston	2022-05-10	True	8	2.98414	80	1.21169	2.96431	0.01048	0.3375	NaN	1.57884	0.14423	12.00000	0.09431	4.00000	-0.41440	0.14147
9	heston	2022-05-18	True	9	2.85762	80	0.92275	2.62972	0.00855	0.4000	NaN	1.15213	0.10452	12.00000	0.08614	2.74838	-0.50200	0.27369
10	heston	2022-05-31	True	17	4.56420	80	0.66982	1.29917	0.01179	0.7000	1.43085	0.68363	0.06153	7.16744	0.07086	1.23619	-0.65125	0.66474
11	heston	2022-06-20	True	15	2.99098	80	0.87309	1.97848	0.01217	0.4875	NaN	1.04298	0.09922	8.94495	0.08308	2.45824	-0.44219	0.24596
12	heston	2022-06-21	True	8	2.12386	100	1.32737	2.14210	0.01143	0.3400	2.25762	1.69485	0.08815	12.00000	0.07935	2.37713	-0.42285	0.33703
13	heston	2022-06-30	True	13	4.14485	80	0.66825	1.18306	0.00740	0.6375	NaN	0.91574	0.07258	3.59569	0.08907	1.10361	-0.69052	0.52590
14	heston	2022-07-29	True	14	4.40650	80	0.50089	1.55951	0.01124	0.6500	NaN	0.59326	0.03223	3.24656	0.08641	0.86756	-0.64225	0.74544
15	heston	2022-08-31	True	11	2.77271	80	0.47652	0.81880	0.00635	0.7500	NaN	0.66437	0.05546	4.30807	0.07891	0.96048	-0.79075	0.73704
16	heston	2022-09-08	True	22	4.82800	80	0.67279	1.61955	0.00667	0.5625	NaN	0.81984	0.05250	10.32754	0.06991	2.16612	-0.43177	0.30777
17	heston	2022-09-30	True	11	2.25491	80	0.94973	1.00461	0.01700	0.7375	NaN	1.32549	0.09733	5.99728	0.08507	1.57176	-0.57003	0.41305
18	heston	2022-10-10	True	9	2.00916	80	0.54388	1.05836	0.01127	0.8625	NaN	0.72326	0.09407	5.00755	0.08139	1.13154	-0.65411	0.63662
19	heston	2022-10-31	True	10	2.10112	80	0.97689	2.08544	0.01196	0.4875	NaN	1.19827	0.08919	12.00000	0.07171	2.94425	-0.16443	0.19854
20	heston	2022-11-10	True	24	5.93728	100	1.25144	1.78161	0.00988	0.5000	1.84840	1.70270	0.05222	3.36437	0.09515	1.31165	-0.29540	0.37213
21	heston	2022-11-30	True	30	6.25448	80	0.56808	1.11838	0.00974	0.7125	NaN	0.67615	0.03771	1.30977	0.13162	0.81440	-0.54457	0.51985
22	heston	2022-12-09	True	20	5.77358	80	1.87875	3.92601	0.01978	0.2500	NaN	2.50390	0.05364	2.93181	0.09433	1.39954	-0.28106	0.28240
23	heston	2022-12-30	True	19	6.49630	100	0.95392	1.02281	0.01343	0.5700	1.36662	1.21736	0.03903	8.52937	0.06412	1.04525	-0.50656	1.00108
24	heston	2023-01-31	True	12	3.52602	80	0.88380	2.61481	0.01163	0.3500	NaN	1.03061	0.04218	8.21538	0.03901	1.60097	-0.42392	0.25005
25	heston	2023-02-28	True	11	2.06293	80	0.65127	1.09956	0.00604	0.6500	NaN	0.86047	0.03514	8.04841	0.04596	1.13284	-0.58117	0.57650
26	heston	2023-03-31	True	14	3.17551	80	0.48988	1.25790	0.01056	0.6875	NaN	0.59176	0.02532	4.92126	0.05973	1.15156	-0.63888	0.44336
27	heston	2023-04-20	True	10	3.30069	80	0.48060	1.03175	0.00600	0.7750	NaN	0.62044	0.02269	2.58897	0.08560	1.40842	-0.61862	0.22344
28	heston	2023-04-28	True	15	5.51021	80	0.50197	0.84585	0.01066	0.8125	NaN	0.66423	0.01767	1.30950	0.13668	1.21671	-0.59617	0.24181
29	heston	2023-05-22	True	32	7.11129	80	0.43856	1.00108	0.00637	0.7625	NaN	0.56161	0.02046	4.67768	0.06257	1.87794	-0.59094	0.16598
30	heston	2023-05-31	True	18	4.80706	80	0.70001	1.18615	0.00782	0.5375	NaN	0.88628	0.02247	10.31757	0.04627	2.68928	-0.48170	0.13202
31	heston	2023-06-23	True	16	4.73243	80	0.81502	1.12918	0.00850	0.6000	NaN	1.06436	0.01020	3.85046	0.05218	1.35110	-0.52133	0.22014
32	heston	2023-06-30	True	18	6.38119	80	0.49268	1.22936	0.00774	0.7250	NaN	0.56749	0.00858	6.14181	0.03421	1.32690	-0.48516	0.23869
33	heston	2023-07-31	True	18	5.11909	80	0.34638	0.83408	0.00474	0.8375	NaN	0.42752	0.01180	2.40967	0.04687	0.83537	-0.53391	0.32366
34	heston	2023-08-31	True	22	5.73149	80	0.72397	0.80200	0.00735	0.6875	NaN	0.95526	0.01060	7.79495	0.03361	1.56931	-0.40926	0.21276
35	heston	2023-09-29	True	11	4.13843	100	1.10269	1.35861	0.01442	0.3900	1.58802	1.41308	0.02781	12.00000	0.03545	2.10715	-0.34419	0.19159
36	heston	2023-10-26	True	8	3.28283	100	1.39483	1.46474	0.01556	0.3900	2.16881	1.80981	0.04315	12.00000	0.04376	2.11158	-0.52093	0.23555
37	heston	2023-10-31	True	11	3.63399	100	1.25716	1.21754	0.01509	0.3800	1.79784	1.59494	0.02996	12.00000	0.03801	1.96975	-0.48772	0.23510
38	heston	2023-11-28	True	13	4.16665	100	1.16628	0.94969	0.01408	0.3900	1.60349	1.53553	0.00943	12.00000	0.03135	2.95104	-0.16900	0.08641
39	heston	2023-11-29	True	10	3.07152	100	1.02733	1.12528	0.01372	0.3500	1.36289	1.33148	0.01217	12.00000	0.03333	3.28428	-0.11449	0.07416
40	heston	2023-11-30	True	10	2.60410	100	1.34368	1.14571	0.01294	0.3800	1.69926	1.76887	0.01130	12.00000	0.03149	2.91792	-0.17783	0.08877
41	heston	2023-12-05	True	20	8.16701	100	0.84053	1.08006	0.01035	0.4700	1.32589	1.02899	0.01308	4.03732	0.04359	1.45772	-0.21084	0.16563
42	heston	2023-12-06	True	12	4.95118	100	0.78760	1.10926	0.01155	0.3900	0.98783	0.98940	0.01307	4.85871	0.04475	1.80189	-0.18632	0.13393
43	heston	2023-12-29	True	13	3.41937	100	0.69182	0.79756	0.00981	0.5500	1.02009	0.88114	0.00834	6.48668	0.03554	1.50591	-0.25117	0.20331

	model	dates	quotes	success_rate	weighted_price_rmse	median_abs_price_error	weighted_iv_rmse	bid_ask_hit_rate	otm_put_rmse	short_maturity_rmse	runtime_sec
0	heston	44	3780	1.0	0.86366	1.2401	0.01045	0.55642	1.58404	1.10742	195.80669

fig, axes = plt.subplots(2, 3, figsize=(15, 6.4), constrained_layout=True)
axes = axes.ravel()
heston_daily["feller_ratio"] = 2.0 * heston_daily["p1"] * heston_daily["p2"] / heston_daily["p3"].pow(2).replace(0.0, np.nan)
for i, (col, label) in enumerate([("p0", "v0"), ("p1", "kappa"), ("p2", "theta"), ("p3", "vol-of-vol"),
                                  ("p4", "rho"), ("feller_ratio", "Feller ratio")]):
    axes[i].plot(heston_daily["date"], heston_daily[col], marker="o", ms=2.5, lw=1.0, color=palette[i])
    axes[i].set_title(f"Heston {label}")
heston_main_date = heston_prices.groupby("date").size().sort_values(ascending=False).index[0]
heston_fit_main = heston_prices[heston_prices["date"].eq(heston_main_date)].copy()
heston_fit_main["k"] = np.log(heston_fit_main["strike"] / heston_fit_main["spot"])
plt.show()

The Heston calibration succeeds across the selected dates. Across the fitted dates, \(V_0\), \(\kappa\), \(\theta\), vol-of-vol, and \(\rho\) all move substantially in the parameter paths. This movement is expected because every date has a different option surface and the calibration has to absorb changes in level, slope, and curvature.

The Feller ratio is also reported:

\[ \frac{2\kappa\theta}{\sigma_v^2} \]

In classical Heston, values above 1 keep the variance process away from zero more strongly. Values below 1 mean the variance boundary can be reached. In practical calibration, the Feller condition is often violated because the market smile wants more flexibility than the strict positivity condition allows. The ratio varies across the fitted dates. That variation is consistent with several Heston parameters adjusting together to fit changing option surfaces, so we should be careful when interpreting any one parameter on its own.

Heston provides a strong baseline on the full daily set, with weighted IV RMSE around 1 volatility point. That makes the later comparison harder, which is good. Rough volatility has to earn its place.

9) Forward Variance Curve and Rough Bergomi

Rough Bergomi is the first rough pricing model we use. We start with the forward variance curve, which gives the variance level expected at each future horizon, instead of describing the initial term structure with a single variance state.

The forward variance curve is:

\[ \xi_0(t) = \mathbb{E}_0[V_t] \]

It’s the market-implied expected future variance at each horizon. We estimate it from the ATM total variance curve:

\[ w(\tau) = \tau \sigma_{\text{ATM}}^2(\tau) \]

A practical approximation is:

\[ \xi_0(\tau) \approx \frac{\partial w(\tau)}{\partial \tau} \]

We take ATM implied volatility across maturities, convert it to total variance, and differentiate a smooth interpolated curve to obtain the simulation input.

Rough Bergomi then models variance as a lognormal rough process around this forward variance curve:

\[ V_t = \xi_0(t)\exp\left(\nu W_t^H - \frac{1}{2}\nu^2 t^{2H}\right) \]

The fractional driver is usually built from a Brownian motion through a kernel:

\[ W_t^H = \sqrt{2H}\int_0^t (t-s)^{H-\frac{1}{2}}\,dW_s^V \]

The spot process is:

\[ dS_t = (r-q)S_t\,dt + \sqrt{V_t}S_t\,dW_t^S \]

with correlation:

\[ d\langle W^S,W^V\rangle_t = \rho\,dt \]

The parameters are:

The normalization in \(W_t^H\) gives it variance \(t^{2H}\). For this Gaussian driver, the exponential correction has expectation one, so the expected model variance follows the input curve \(\xi_0(t)\) before numerical discretization.

• \(H\): roughness of volatility,
• \(\nu\): vol-of-vol, controlling how strongly variance fluctuates around \(\xi_0(t)\),
• \(\rho\): spot-volatility correlation, controlling the leverage effect and smile skew.

A more negative \(\rho\) usually creates a steeper equity-style downside skew. A larger \(\nu\) increases smile curvature and variance uncertainty. A smaller \(H\) makes the short-end skew steeper because variance reacts in a rougher, more irregular way near the present.

The calibration uses Monte Carlo simulation, so it’s noisier than Heston. We use variance reduction tools such as Sobol-style sampling and antithetic paths where available. Even then, rBergomi is computationally expensive because each option price depends on simulated rough variance paths.

9.1 Forward Variance Extraction

The forward variance curve gives rBergomi its initial variance term structure. We start from the ATM total variance:

\[ w(\tau)=\tau\sigma^2_{\text{ATM}}(\tau) \]

If total variance is the integral of forward variance,

\[ w(\tau)=\int_0^\tau \xi_0(u)\,du \]

then differentiating gives:

\[ \xi_0(\tau)=\frac{dw(\tau)}{d\tau} \]

We therefore need the surface fit to be smooth across maturity before differentiating total variance. A small wiggle in ATM IV becomes a larger wiggle in total variance derivative. We use a dense maturity grid and interpolation, then keep the extracted forward variance positive for the simulation.

There’s also a financial interpretation. \(\xi_0(\tau)\) is the market-implied expectation of instantaneous variance at future horizon \(\tau\). If the curve slopes upward, the market expects future variance to be higher than near-term variance. If it slopes downward, near-term risk is priced above longer-term variance. rBergomi uses this curve as the deterministic backbone, then adds rough random fluctuations around it.

We can summarize the roles of the curve and model parameters as follows:

\[ \text{forward variance curve} = \text{term structure level} \]

\[ (H,\nu,\rho) = \text{roughness, variance randomness, skew direction} \]

9.2 Simulating Rough Bergomi Paths

The rBergomi implementation is Monte Carlo based. That means the model doesn’t give one deterministic smile from a closed-form formula. It generates many joint paths for spot and variance, prices the option payoff on those paths, and then converts the average discounted payoff back into implied volatility.

The rough variance path has two moving pieces:

\[ V_t = \xi_0(t)\exp\left(\nu W_t^H - \frac{1}{2}\nu^2 t^{2H}\right) \]

The first part, \(\xi_0(t)\), is deterministic once we extract the forward variance curve from the surface. The second part is random. The term \(\nu W_t^H\) pushes variance up and down, while the correction term \(-\frac{1}{2}\nu^2t^{2H}\) keeps the lognormal multiplier centered correctly. Without that correction, the exponential transformation would bias variance upward because exponentials of normal random variables have mean greater than one.

On the simulation grid, we use the variance of the discrete rough driver for this correction. That variance is the squared kernel scale multiplied by the sum of squared discrete kernel weights; it approaches the continuous-time expression as the grid is refined.

To see the idea with a simple example, suppose two maturities have the same forward variance backbone, but one simulation path receives a positive rough-vol shock early. Because \(W_t^H\) is rough, that early shock can create a sharp near-term variance burst. A standard smooth-vol model tends to spread that shock more gently. With rBergomi, this rough variance path can create steep short-maturity smiles: option prices care a lot about what can happen over the next few days, and rough variance can move sharply over those few days.

The spot path uses a correlated Brownian shock:

\[ dS_t = (r-q)S_tdt + \sqrt{V_t}S_tdW_t^S \]

The variance driver and spot driver have correlation \(\rho\). In simulation form, we can think of the spot shock as:

\[ dW_t^S = \rho\,dW_t^V + \sqrt{1-\rho^2}\,dW_t^\perp \]

where \(dW_t^\perp\) is an independent Brownian shock. This construction makes the spot move partially with the variance shock and partially with independent price noise. If \(\rho<0\), a negative spot move tends to arrive with a positive variance shock. That’s the leverage effect behind the equity index skew.

The pricing step is:

\[ C(K,T) = e^{-rT}\mathbb{E}\left[(S_T-K)^+\right] \]

The expectation is approximated by averaging across simulation paths:

\[ \widehat{C}(K,T)=e^{-rT}\frac{1}{M}\sum_{m=1}^{M}(S_T^{(m)}-K)^+ \]

where \(M\) is the number of Monte Carlo paths. We repeat this for several strikes and maturities, then map the model prices into implied volatilities so that we can compare them with the market surface.

We use out-of-the-money puts below the forward and calls at or above it when building the smile. The log-moneyness of that smile is measured against the forward, which keeps it aligned with the fitted surface and makes IV inversion less sensitive to deep in-the-money prices.

We can check the effect of simulation noise by comparing the path counts in the convergence results. When paths increase from hundreds to thousands, the option price estimate should stabilize. If the implied vol moves too much as path count changes, the model output is still dominated by Monte Carlo noise. The rBergomi ATM call estimate moves by a few dollars across path counts. That gives us a sense of the price uncertainty alongside the substantial computational cost of Monte Carlo rough-volatility calibration.

With rBergomi we can generate rough smile shapes from joint spot and variance paths. Every parameter evaluation requires many simulated paths, so calibration also takes substantial computation.

def xi_from_surface(fit, tau_values):
    tau_arr = np.sort(np.asarray(tau_values, dtype=float))
    dense_tau = np.sort(np.unique(np.concatenate([tau_arr, np.linspace(max(tau_arr[0] * 0.5, 1e-4), tau_arr[-1] * 1.1, 60)])))
    dense_tau = dense_tau[dense_tau > 0]
    base = []
    for tau in dense_tau:
        sigma = float(surface_iv(fit, np.array([0.0]), np.array([tau]))[0])
        if np.isfinite(sigma) and sigma > 0:
            base.append({"tau": float(tau), "total_variance": sigma * sigma * float(tau)})
    try:
        from scipy.interpolate import PchipInterpolator
        if len(base) < 3:
            raise ValueError("too few points for PCHIP")
        tv_df = pd.DataFrame(base).sort_values("tau").reset_index(drop=True)
        pchip = PchipInterpolator(tv_df["tau"].values, tv_df["total_variance"].values)
        pchip_d = pchip.derivative()
        out_rows = []
        for tau in tau_arr:
            sigma = float(surface_iv(fit, np.array([0.0]), np.array([tau]))[0])
            fv = max(float(pchip_d(tau)), 1e-6)
            out_rows.append({"tau": float(tau), "total_variance": float(pchip(tau)),
                             "atm_iv": sigma if np.isfinite(sigma) else np.nan, "variance": fv})
        return pd.DataFrame(out_rows)
    except Exception:
        out_rows = []
        last_tau, last_total = 0.0, 0.0
        for tau in tau_arr:
            sigma = float(surface_iv(fit, np.array([0.0]), np.array([tau]))[0])
            total = sigma * sigma * float(tau) if np.isfinite(sigma) and sigma > 0 else last_total
            out_rows.append({"tau": float(tau), "total_variance": total,
                             "atm_iv": sigma if np.isfinite(sigma) else np.nan,
                             "variance": max((total - last_total) / max(tau - last_tau, 1e-8), 1e-6)})
            last_tau, last_total = tau, total
        return pd.DataFrame(out_rows)


@njit
def rbergomi_xi_interp(t, xi_tau, xi_var):
    if t <= xi_tau[0]:
        return xi_var[0]
    for i in range(1, len(xi_tau)):
        if t <= xi_tau[i]:
            w = (t - xi_tau[i - 1]) / max(xi_tau[i] - xi_tau[i - 1], 1e-12)
            return xi_var[i - 1] * (1.0 - w) + xi_var[i] * w
    return xi_var[len(xi_var) - 1]


@njit
def rbergomi_paths_numba(spot, xi_tau, xi_var, h, nu, rho, rate, dividend_yield, tau, z1, z2):
    n, m = z1.shape
    dt = tau / m
    out_s = np.empty((n, m + 1))
    out_v = np.empty((n, m + 1))
    kernel = np.empty(m)
    rough_var = np.empty(m)
    alpha = h + 0.5
    scale = np.sqrt(max(2.0 * h, 1e-12)) * (dt ** h) / max(alpha, 1e-12)
    for j in range(m):
        kernel[j] = ((j + 1.0) ** alpha - j ** alpha)
    for j in range(m):
        rv = 0.0
        for l in range(j + 1):
            rv += kernel[l] * kernel[l]
        rough_var[j] = scale * scale * rv
    for i in range(n):
        out_s[i, 0] = spot
        out_v[i, 0] = rbergomi_xi_interp(0.0, xi_tau, xi_var)
        for j in range(m):
            rough = 0.0
            for l in range(j + 1):
                rough += kernel[l] * z1[i, j - l]
            rough *= scale
            xi_t = rbergomi_xi_interp((j + 1.0) * dt, xi_tau, xi_var)
            v = xi_t * np.exp(nu * rough - 0.5 * nu * nu * rough_var[j])
            if v < 1e-10:
                v = 1e-10
            if v > 25.0:
                v = 25.0
            out_v[i, j + 1] = v
            dw = rho * z1[i, j] + np.sqrt(max(1.0 - rho * rho, 1e-12)) * z2[i, j]
            out_s[i, j + 1] = out_s[i, j] * np.exp((rate - dividend_yield - 0.5 * v) * dt + np.sqrt(v * dt) * dw)
    return out_s, out_v


def _next_pow2(n):
    p = 1
    while p < n:
        p <<= 1
    return p


def _crn(paths, steps, seed, use_sobol=True):
    paths, steps, half = int(paths), int(steps), int(paths) // 2
    z1_h = z2_h = None
    if use_sobol:
        try:
            from scipy.stats.qmc import Sobol
            from scipy.stats import norm as _norm
            n_sobol = _next_pow2(half)
            sob1 = Sobol(d=steps, scramble=True, seed=int(seed))
            sob2 = Sobol(d=steps, scramble=True, seed=int(seed) + 1)
            u1 = sob1.random(n_sobol)[:half]
            u2 = sob2.random(n_sobol)[:half]
            u1 = np.clip(u1, 1e-12, 1.0 - 1e-12)
            u2 = np.clip(u2, 1e-12, 1.0 - 1e-12)
            z1_h = _norm.ppf(u1).astype(float)
            z2_h = _norm.ppf(u2).astype(float)
        except Exception:
            pass
    if z1_h is None:
        rng = np.random.default_rng(int(seed))
        z1_h = rng.standard_normal((half, steps))
        z2_h = rng.standard_normal((half, steps))
    return np.vstack([z1_h, -z1_h]), np.vstack([z2_h, -z2_h])


def rbergomi_sim(spot, xi, h, nu, rho, rate, dividend_yield, tau, paths, steps, seed, antithetic=True, z1=None, z2=None):
    if z1 is not None and z2 is not None:
        pass
    else:
        rng = np.random.default_rng(int(seed))
        half = int(paths) // 2 if antithetic else int(paths)
        z1 = rng.standard_normal((half, int(steps)))
        z2 = rng.standard_normal((half, int(steps)))
        if antithetic:
            z1 = np.vstack([z1, -z1])
            z2 = np.vstack([z2, -z2])
    xi_tau = np.asarray(xi["tau"], dtype=float)
    xi_var = np.asarray(xi["variance"], dtype=float)
    return rbergomi_paths_numba(float(spot), xi_tau, xi_var, float(h), float(nu), float(rho), float(rate), float(dividend_yield), float(tau), z1, z2)


def rbergomi_mc_smile(spot, xi, h, nu, rho, rate, dividend_yield, tau, strikes, paths, steps, seed, z1=None, z2=None):
    s, v = rbergomi_sim(spot, xi, h, nu, rho, rate, dividend_yield, tau, paths, steps, seed, antithetic=(z1 is None), z1=z1, z2=z2)
    terminal = s[:, -1]
    discount = np.exp(-float(rate) * float(tau))
    forward = float(spot) * np.exp((float(rate) - float(dividend_yield)) * float(tau))
    n_k = len(strikes)
    prices = np.empty(n_k)
    price_ses = np.empty(n_k)
    ivs = np.full(n_k, np.nan)
    for i, K in enumerate(strikes):
        if K < forward:
            payoff = np.maximum(K - terminal, 0.0)
            px = discount * float(payoff.mean())
            se = discount * float(payoff.std(ddof=1) / np.sqrt(len(payoff)))
            iv_arr = implied_vol("put", np.array([px]), forward, np.array([K]), tau, discount, engine="auto")
        else:
            payoff = np.maximum(terminal - K, 0.0)
            px = discount * float(payoff.mean())
            se = discount * float(payoff.std(ddof=1) / np.sqrt(len(payoff)))
            iv_arr = implied_vol("call", np.array([px]), forward, np.array([K]), tau, discount, engine="auto")
        prices[i] = px
        price_ses[i] = se
        ivs[i] = float(iv_arr[0]) if len(iv_arr) > 0 and np.isfinite(iv_arr[0]) else np.nan
    k_arr = np.log(strikes / forward)
    return pd.DataFrame({"tau": tau, "tau_days": tau * ann_days, "strike": strikes, "k": k_arr, "price": prices, "price_se": price_ses, "iv": ivs})

xi_curve = xi_from_surface(fit_visual, np.array([14, 21, 30, 45, 60, 90, 120, 150], dtype=float) / ann_days)
spot_main = float(q_main["spot"].median())
rate_rb = float(q_main["rate"].median())
div_rb = float(q_main["implied_dividend_yield"].median())
q_main_rbergomi_target = calibration_quotes[calibration_quotes["date"].eq(pd.Timestamp(main_date).normalize())].copy()
if q_main_rbergomi_target.empty:
    q_main_rbergomi_target = q_main.copy()
rb_h = float(np.clip(skew_fit_main["h_from_skew"].iloc[0], 0.04, 0.45))
rb_nu = 1.8
rb_rho = -0.70
k_smile = np.linspace(-0.28, 0.18, 31)

_fwd_ref = spot_main * np.exp((rate_rb - div_rb) * 30.0 / ann_days)
strikes_smile = _fwd_ref * np.exp(k_smile)

def rbergomi_iv_loss(params, z1_crn, z2_crn):
    h, nu, rho = params
    smiles = pd.concat([
        rbergomi_mc_smile(spot_main, xi_curve, h, nu, rho, rate_rb, div_rb, d / ann_days, strikes_smile,
                             len(z1_crn), z1_crn.shape[1], seed, z1=z1_crn, z2=z2_crn)
        for d in [14, 30, 60, 90]
    ], ignore_index=True)
    target = q_main_rbergomi_target[
        q_main_rbergomi_target["tau"].mul(ann_days).between(10, 100) &
        q_main_rbergomi_target["k"].between(k_smile.min(), k_smile.max())
    ].copy()
    errors, weights = [], []
    for _, row in target.iterrows():
        if not np.isfinite(row["iv_mid"]):
            continue
        valid_smiles = smiles[np.isfinite(smiles["iv"])].copy()
        if valid_smiles.empty:
            continue
        tau_choice = float(valid_smiles.iloc[(valid_smiles["tau_days"] - row["tau"] * ann_days).abs().argsort()[:1]]["tau_days"].iloc[0])
        slice_smile = valid_smiles[valid_smiles["tau_days"].eq(tau_choice)].sort_values("k")
        if len(slice_smile) >= 2 and row["k"] >= slice_smile["k"].min() and row["k"] <= slice_smile["k"].max():
            model_iv = float(np.interp(row["k"], slice_smile["k"], slice_smile["iv"]))
        else:
            score = (valid_smiles["tau_days"] - row["tau"] * ann_days).abs() + 25.0 * (valid_smiles["k"] - row["k"]).abs()
            model_iv = float(valid_smiles.loc[int(score.idxmin()), "iv"])
        err = float(model_iv - row["iv_mid"])
        if np.isfinite(err):
            w = float(row.get("obs_weight", row.get("surface_weight", 1.0)))
            errors.append(err)
            weights.append(w)
    errors = np.asarray(errors, dtype=float)
    weights = np.asarray(weights, dtype=float)
    loss = float(np.average(errors * errors, weights=np.clip(weights, 0.05, 20.0))) if errors.size else np.inf
    return loss, smiles


_crn_paths_search = 2048
_crn_steps = 80
z1_crn_search, z2_crn_search = _crn(_crn_paths_search, _crn_steps, seed, use_sobol=True)

candidate_params = [(rb_h, rb_nu, rb_rho), (max(0.05, rb_h - 0.06), rb_nu, rb_rho),
                    (min(0.45, rb_h + 0.06), rb_nu, rb_rho), (0.08, rb_nu, rb_rho), (0.16, rb_nu, rb_rho),
                    (0.28, rb_nu, rb_rho), (rb_h, 1.2, rb_rho), (rb_h, 2.6, rb_rho), (rb_h, 3.6, rb_rho),
                    (rb_h, rb_nu, -0.90), (rb_h, rb_nu, -0.45), (rb_h, rb_nu, -0.20),
                    (float(np.clip(skew_fit_main["h_from_skew"].iloc[0], 0.04, 0.45)), 2.2, -0.80),
                    (max(0.05, rb_h - 0.05), 2.6, -0.90), (min(0.45, rb_h + 0.08), 1.2, -0.45)]
loss_rows = []
candidate_results = []
for i, params in enumerate(candidate_params):
    loss, smile = rbergomi_iv_loss(params, z1_crn_search, z2_crn_search)
    loss_rows.append({"restart": i, "h": params[0], "nu": params[1], "rho": params[2], "iv_mse": loss, "iv_rmse": np.sqrt(loss)})
    candidate_results.append((loss, params, smile))

z1_crn_fine, z2_crn_fine = _crn(8192, _crn_steps, seed + 99, use_sobol=True)
best_loss = np.inf
best_smile = pd.DataFrame()
best_params = (rb_h, rb_nu, rb_rho)
for _, params, _ in sorted(candidate_results, key=lambda x: x[0])[:3]:
    loss_fine, smile_fine = rbergomi_iv_loss(params, z1_crn_fine, z2_crn_fine)
    if loss_fine < best_loss:
        best_loss = loss_fine
        best_smile = smile_fine
        best_params = params

def _rb_loss_scalar(x):
    h_ = float(np.clip(x[0], 0.04, 0.40))
    nu_ = float(np.clip(x[1], 0.3, 5.0))
    rho_ = float(np.clip(x[2], -0.99, -0.05))
    return rbergomi_iv_loss((h_, nu_, rho_), z1_crn_fine, z2_crn_fine)[0]

for _, p0_, _ in sorted(candidate_results, key=lambda x: x[0])[:3]:
    p0c = [float(np.clip(p0_[0], 0.04, 0.40)), float(np.clip(p0_[1], 0.3, 5.0)), float(np.clip(p0_[2], -0.99, -0.05))]
    try:
        opt = optimize.minimize(_rb_loss_scalar, p0c, method="Nelder-Mead",
                                options={"maxiter": 200, "xatol": 1e-4, "fatol": 1e-5, "adaptive": True})
        if opt.fun < best_loss:
            best_loss = opt.fun
            best_params = (float(np.clip(opt.x[0], 0.04, 0.40)), float(np.clip(opt.x[1], 0.3, 5.0)), float(np.clip(opt.x[2], -0.99, -0.05)))
            best_smile = rbergomi_iv_loss(best_params, z1_crn_fine, z2_crn_fine)[1]
    except Exception:
        pass

rbergomi_loss_table = pd.DataFrame(loss_rows).sort_values("iv_mse")
rb_h, rb_nu, rb_rho = best_params

rbergomi_s, rbergomi_v = rbergomi_sim(spot_main, xi_curve, rb_h, rb_nu, rb_rho, rate_rb, div_rb, 90 / ann_days, 3000, 96, seed, antithetic=True)
rbergomi_iv = pd.concat([
    rbergomi_mc_smile(spot_main, xi_curve, rb_h, rb_nu, rb_rho, rate_rb, div_rb, d / ann_days, strikes_smile, 4000, 96, seed + int(d))
    for d in [14, 30, 60, 90]
], ignore_index=True)

sens_rows = []
for name, values in [("h", [0.06, rb_h, 0.30]), ("nu", [0.9, rb_nu, 3.0]), ("rho", [-0.90, rb_rho, -0.25])]:
    for value in values:
        h1 = value if name == "h" else rb_h
        nu1 = value if name == "nu" else rb_nu
        rho1 = value if name == "rho" else rb_rho
        s1 = rbergomi_mc_smile(spot_main, xi_curve, h1, nu1,
                               rho1, rate_rb, div_rb, 30 / ann_days,
                               strikes_smile, 2500, 80, seed + int(abs(value) * 1000))
        s1["parameter"] = name
        s1["value"] = value
        sens_rows.append(s1)
rbergomi_sensitivity = pd.concat(sens_rows, ignore_index=True)

conv_rows = []
for p in [800, 1600, 3200, 6400]:
    s1 = rbergomi_mc_smile(spot_main, xi_curve, rb_h, rb_nu, rb_rho, rate_rb, div_rb, 30 / ann_days, np.array([spot_main]), p, 80, seed + p)
    conv_rows.append({"paths": p, "atm_call_price": float(s1["price"].iloc[0]),
                      "price_se": float(s1["price_se"].iloc[0]), "atm_iv": float(s1["iv"].iloc[0])})
rbergomi_convergence = pd.DataFrame(conv_rows)

display(xi_curve.round(5))
display(rbergomi_loss_table.round(6))
display(rbergomi_convergence.round(6))

	tau	total_variance	atm_iv	variance
0	0.03833	0.00044	0.10757	0.01338
1	0.05749	0.00074	0.11365	0.01768
2	0.08214	0.00122	0.12196	0.02091
3	0.12320	0.00215	0.13212	0.02466
4	0.16427	0.00324	0.14050	0.02828
5	0.24641	0.00585	0.15403	0.03512
6	0.32854	0.00887	0.16431	0.03735
7	0.41068	0.01200	0.17091	0.03972

	restart	h	nu	rho	iv_mse	iv_rmse
14	14	0.444219	1.2	-0.45	0.000214	0.014614
11	11	0.364219	1.8	-0.20	0.000264	0.016236
10	10	0.364219	1.8	-0.45	0.001328	0.036443
6	6	0.364219	1.2	-0.70	0.001394	0.037341
2	2	0.424219	1.8	-0.70	0.001426	0.037757
0	0	0.364219	1.8	-0.70	0.003366	0.058015
9	9	0.364219	1.8	-0.90	0.005626	0.075010
12	12	0.364219	2.2	-0.80	0.006642	0.081497
7	7	0.364219	2.6	-0.70	0.006816	0.082559
1	1	0.304219	1.8	-0.70	0.007853	0.088618
5	5	0.280000	1.8	-0.70	0.010991	0.104839
8	8	0.364219	3.6	-0.70	0.011091	0.105314
13	13	0.314219	2.6	-0.90	0.022885	0.151279
4	4	0.160000	1.8	-0.70	0.050835	0.225465
3	3	0.080000	1.8	-0.70	0.103451	0.321639

	paths	atm_call_price	price_se	atm_iv
0	800	55.836772	3.243942	0.105047
1	1600	56.291046	2.390488	0.105876
2	3200	56.713916	1.638684	0.106648
3	6400	54.929668	1.167085	0.103391

fig, axes = plt.subplots(2, 3, figsize=(16, 7), constrained_layout=True)
time_grid = np.linspace(0.0, 90 / ann_days, rbergomi_v.shape[1])
for i in range(20):
    axes[0, 0].plot(time_grid, rbergomi_v[i], lw=0.5, alpha=0.5)
    axes[0, 1].plot(time_grid, rbergomi_s[i], lw=0.5, alpha=0.5)
for i, (d, g) in enumerate(rbergomi_iv.groupby("tau_days")):
    axes[0, 2].plot(g["k"], g["iv"], lw=1.1, color=palette[i], label=f"{d:.0f}d")
axes[0, 2].scatter(q_main["k"], q_main["iv_mid"], s=7, alpha=0.22, color="black", label="market")
for i, (value, g) in enumerate(rbergomi_sensitivity[rbergomi_sensitivity["parameter"].eq("h")].groupby("value")):
    axes[1, 0].plot(g["k"], g["iv"], lw=1.0, color=palette[i], label=f"H={value:.2f}")
for i, (value, g) in enumerate(rbergomi_sensitivity[rbergomi_sensitivity["parameter"].eq("nu")].groupby("value")):
    axes[1, 1].plot(g["k"], g["iv"], lw=1.0, color=palette[i], label=f"nu={value:.2f}")
for i, (value, g) in enumerate(rbergomi_sensitivity[rbergomi_sensitivity["parameter"].eq("rho")].groupby("value")):
    axes[1, 2].plot(g["k"], g["iv"], lw=1.0, color=palette[i], label=f"rho={value:.2f}")
axes[0, 0].set_title("rBergomi Variance Paths")
axes[0, 1].set_title("rBergomi Price Paths")
axes[0, 2].set_title("Market vs rBergomi IV Fit")
axes[1, 0].set_title("H Sensitivity")
axes[1, 1].set_title("nu Sensitivity")
axes[1, 2].set_title("rho Sensitivity")
for ax in axes.ravel():
    ax.legend(fontsize=6)
plt.show()

C:\Users\Lenovo\AppData\Local\Temp\ipykernel_16732\641628204.py:22: UserWarning: No artists with labels found to put in legend.  Note that artists whose label start with an underscore are ignored when legend() is called with no argument.
  ax.legend(fontsize=6)

We can read the forward-variance estimates from the ATM variance curve extracted from the surface. It gives rBergomi the term structure anchor it needs. This matters because rBergomi doesn’t try to learn the whole variance term structure from just three parameters. It uses \(\xi_0(t)\) as the baseline and uses \((H,\nu,\rho)\) to shape smile dynamics around that baseline.

Across the rBergomi restarts, the best SPX fit has a moderate \(H\), negative \(\rho\), and a vol-of-vol level that gives enough curvature without exploding Monte Carlo noise. The ATM price estimates stabilize as path count increases, though the reported Monte Carlo standard errors remain visible in the comparison. That’s the unavoidable cost of Monte Carlo rough-vol pricing.

The rBergomi path plot is one of the most useful teaching plots in the project. The variance paths can spike very sharply even without explicit jumps. The price paths inherit this stochastic variance behavior. The model smile tries to match the market smile, but it still carries simulation noise and parameter sensitivity.

The main interpretation is that rBergomi is excellent conceptually for rough smile generation, but calibration is heavy. It’s useful as a rough-volatility laboratory and benchmark, but less convenient for daily large-scale fitting unless the implementation is highly optimized.

We compare the rBergomi estimates across path counts to see how much Monte Carlo noise may enter a calibration. The ATM call price estimates are close as paths increase, but the standard error isn’t negligible. When the target is an implied volatility surface, price noise maps into IV noise through vega:

\[ \Delta \sigma_{\text{imp}} \approx \frac{\Delta C}{\text{Vega}} \]

For low-vega options, especially far OTM or very short maturity, a small price simulation error can become a large IV error. That’s why rBergomi calibration is much harder than Heston calibration. Heston and rough Heston use characteristic-function pricing. rBergomi here uses simulation, so the optimizer sees a noisy objective.

Across optimizer restarts, we can see that the parameter surface is irregular and contains several competing fits. Several combinations of \(H\), \(\nu\), and \(\rho\) can produce similar smile shapes, but the model still struggles to hit every maturity and moneyness bucket. The concept is still useful, but the practical cost is clear: a non-Markovian Monte Carlo model is slower and noisier.

10) Rough Heston and the Fractional Riccati Equation

Rough Heston keeps the Heston-style economic structure but replaces the ordinary mean-reverting variance diffusion with a Volterra fractional process. Instead of:

\[ dV_t = \kappa(\theta - V_t)dt + \sigma_v\sqrt{V_t}dB_t \]

we write variance in integral form:

\[ V_t = V_0 + \frac{1}{\Gamma(\alpha)}\int_0^t (t-s)^{\alpha-1}\kappa(\theta - V_s)\,ds + \frac{1}{\Gamma(\alpha)}\int_0^t (t-s)^{\alpha-1}\sigma_v\sqrt{V_s}\,dB_s \]

where:

\[ \alpha = H + \frac{1}{2} \]

and \(H \in (0,1/2)\) gives \(\alpha \in (1/2,1)\).

The kernel is:

\[ K(t-s) = \frac{(t-s)^{\alpha-1}}{\Gamma(\alpha)} \]

This kernel is singular near \(s=t\) because \(\alpha-1<0\). That singularity is the mathematical reason the process reacts sharply to recent shocks. At the same time, the kernel integrates over the entire past, so the model is non-Markovian.

A useful operator here is the fractional integral:

\[ I^\alpha f(t) = \frac{1}{\Gamma(\alpha)}\int_0^t (t-s)^{\alpha-1}f(s)\,ds \]

Rough Heston can be written compactly as:

\[ V_t = V_0 + I^\alpha\left[\kappa(\theta - V_\cdot)\right](t) + I^\alpha\left[\sigma_v\sqrt{V_\cdot}\,dB_\cdot\right](t) \]

This explains why rough Heston is harder than Heston. In Heston, the next variance step depends on the current variance. In rough Heston, the current variance contains a weighted memory of the whole past path through a fractional kernel.

The payoff is that rough Heston remains semi-tractable. Its characteristic function has a structure similar to classical Heston, but the ordinary Riccati equation becomes a fractional Riccati equation.

10.1 Memory Kernel Shape

The kernel:

\[ K(t-s)=\frac{(t-s)^{\alpha-1}}{\Gamma(\alpha)} \]

has two important properties.

First, near \(s=t\), the term \((t-s)^{\alpha-1}\) becomes large because \(\alpha-1<0\). This means very recent shocks receive a large weight. That creates rough local behavior.

Second, the kernel decays slowly enough that older shocks aren’t forgotten with a simple exponential half-life. This creates a fractional memory structure.

Compare this with ordinary Heston. Ordinary Heston mean reversion creates exponential memory:

\[ e^{-\kappa(t-s)} \]

A shock decays at a rate controlled by \(\kappa\). In rough Heston, the memory shape is power-law-like rather than exponential. The volatility path can be locally jagged and still remember past shocks in a non-exponential way. Both the recent weight and the long tail come from the same power kernel.

Exponential memory can be updated recursively from the current state. Fractional memory requires a weighted history or a numerical approximation of that history, so we include the cost of solving the fractional Riccati equation in the pricing comparison.

10.2 Fractional Riccati Structure

In classical Heston, Fourier pricing depends on a Riccati equation. Rough Heston keeps the same spirit but changes the derivative operator. In simplified form, the Riccati function \(\psi(t,u)\) satisfies:

\[ D^\alpha \psi(t,u) = F(u,\psi(t,u)) \]

where \(D^\alpha\) is a fractional derivative and \(u\) is the Fourier variable. The function \(F\) has the same economic ingredients as Heston:

\[ F(u,\psi) = \frac{1}{2}(u^2 + iu) + (\rho\sigma_v iu - \kappa)\psi + \frac{1}{2}\sigma_v^2\psi^2 \]

Different sign conventions appear depending on whether the characteristic function is written for \(X_t=\log S_t\) or for shifted Fourier variables, but the structure is the key:

• the constant term comes from return variance exposure,
• the linear term contains mean reversion and spot-vol correlation,
• the quadratic term comes from volatility-of-volatility.

The fractional derivative means \(\psi(t,u)\) at time \(t\) isn’t updated like an ordinary ODE step. It depends on a memory convolution. Equivalently, applying the corresponding fractional integral to the right-hand side accumulates contributions from earlier values of \(F\), with weights determined by their distance from the current time. Numerically, we solve it on a grid. If the grid has \(N\) time steps, each new step needs weighted contributions from previous steps. So we check Riccati convergence across different step counts.

The pricing engine then uses the characteristic function inside the Fourier/COS machinery from Project 14. So the full rough-Heston pricing pipeline is:

1. choose parameters \((H,V_0,\kappa,\theta,\sigma_v,\rho)\),
2. solve the fractional Riccati equation for each maturity and Fourier node,
3. build the characteristic function,
4. price options with Fourier/COS integration,
5. compare model prices with market prices,
6. calibrate parameters to minimize weighted pricing errors.

The rough part enters through step 2. Everything after that looks familiar from characteristic-function pricing, but the characteristic function is much more expensive to obtain.

10.3 From Fractional Riccati to Option Prices

The fractional Riccati equation is the bridge between rough Heston dynamics and option prices. The model doesn’t simulate every variance path for pricing. Instead, it computes the characteristic function of log-price and then prices options with the Fourier machinery from Project 14.

If we write the terminal log price as:

\[ X_T=\log S_T \]

The characteristic function is:

\[ \phi_T(u)=\mathbb{E}\left[e^{iuX_T}\right] \]

Once we know \(\phi_T(u)\), we can recover option prices through Fourier inversion or a COS expansion. This gives us a semi-analytical route to pricing even though the variance process keeps a weighted memory of its past.

The rough part enters through the Riccati function \(\psi(t,u)\). In an ordinary Heston model, \(\psi\) solves a standard ODE. In rough Heston, it solves a fractional equation. The numerical solver builds \(\psi\) on a time grid:

\[ 0=t_0<t_1<\cdots<t_N=T \]

At each time step, the value of \(\psi(t_n,u)\) depends on a weighted sum of previous values. The weights come from the fractional kernel. A rough way to write the discretized memory is:

\[ \psi_n \approx \sum_{j=0}^{n-1} a_{n-j}F(u,\psi_j) \]

where the weights \(a_{n-j}\) decay like a power law rather than exponentially. Recent values get large weights because the kernel is singular near the present, but older values still matter. That’s the numerical signature of rough volatility.

The model price then follows this chain:

1. choose rough-Heston parameters,
2. solve the fractional Riccati equation for each Fourier frequency \(u\),
3. assemble the characteristic function \(\phi_T(u)\),
4. pass \(\phi_T(u)\) into the Fourier/COS pricing formula,
5. compare the model price to market prices.

The convergence diagnostic checks whether this chain is stable when we increase the Riccati grid resolution. If prices change materially between 512 and 1024 steps, the calibration result can’t be trusted because the objective is reacting to numerical error. Across the test maturities, the SPX convergence check gives maximum price differences around a few cents, which is reasonable for the simplified research engine.

There’s also a practical reason we keep this diagnostic close to the model fit. Rough volatility models can look impressive because they add advanced mathematics, but the numerical layer can quietly dominate the result. For example, a bad Fourier truncation range can make a parameter set look worse than it’s. A coarse fractional grid can make short-maturity prices unstable. These convergence checks make the pricing engine part of the research result instead of treating it as a black box.

The fractional Riccati equation is the mathematical center of rough Heston. In classical Heston, the Riccati equation is an ordinary differential equation:

\[ \frac{d\psi}{dt}=F(u,\psi) \]

A numerical solver only needs the current value of \(\psi\) to step forward. In rough Heston, we replace the ordinary derivative with a fractional derivative:

\[ D^\alpha\psi=F(u,\psi) \]

This means the current update depends on a weighted history of previous \(\psi\) values. A grid solver is effectively doing something like:

\[ \psi_n \approx \sum_{j=0}^{n-1} a_{n,j}F(u,\psi_j) \]

where the weights \(a_{n,j}\) come from the fractional kernel. The exact scheme can vary, but the interpretation is stable: every new Riccati value carries memory.

We compare the Riccati solution and prices across 128, 256, 512, and 1024 steps to assess that discretization error. If the fractional Riccati solution changes too much as the grid is refined, the resulting option prices can’t be trusted. The convergence diagnostics aren’t optional polish; they are part of making rough-Heston pricing credible.

def rough_heston_params_array(params):
    if isinstance(params, pd.Series):
        h = float(params.get("h", h_rv))
        return np.array([h, params["p0"], params["p1"], params["p2"], params["p3"], params["p4"]], dtype=float)
    return np.asarray(params, dtype=float)


def riccati_rhs(u, psi, kappa, sigma_v, rho):
    return -0.5 * (u * u + 1j * u) + (1j * u * rho * sigma_v - kappa) * psi + 0.5 * sigma_v * sigma_v * psi * psi


@njit
def fractional_riccati_adams_numba(u, alpha, kappa, sigma_v, rho, tau, n_steps, gamma_alpha_1, gamma_alpha_2, gamma_beta_1):
    m = max(1, int(n_steps))
    n_u = u.size
    dt = tau / m
    dt_alpha = dt ** alpha
    psi = np.zeros((m + 1, n_u), dtype=np.complex128)
    f = np.zeros((m + 1, n_u), dtype=np.complex128)
    predictor_weights = np.empty(m, dtype=np.float64)
    for j in range(1, m + 1):
        predictor_weights[j - 1] = (j ** alpha - (j - 1.0) ** alpha) * dt_alpha / gamma_alpha_1
    for col in range(n_u):
        uu = u[col]
        f[0, col] = -0.5 * (uu * uu + 1j * uu)
    for i in range(1, m + 1):
        n = i - 1
        a0 = n ** (alpha + 1.0) - (n - alpha) * (n + 1.0) ** alpha
        for col in range(n_u):
            uu = u[col]
            pred = 0.0 + 0.0j
            for j in range(i):
                pred += predictor_weights[i - 1 - j] * f[j, col]
            f_pred = -0.5 * (uu * uu + 1j * uu) + (1j * uu * rho * sigma_v - kappa) * pred + 0.5 * sigma_v * sigma_v * pred * pred
            corr = a0 * f[0, col]
            for j in range(1, i):
                jj = float(j)
                weight = (n - jj + 2.0) ** (alpha + 1.0) + (n - jj) ** (alpha + 1.0) - 2.0 * (n - jj + 1.0) ** (alpha + 1.0)
                corr += weight * f[j, col]
            psi[i, col] = dt_alpha * (corr + f_pred) / gamma_alpha_2
            val = psi[i, col]
            f[i, col] = -0.5 * (uu * uu + 1j * uu) + (1j * uu * rho * sigma_v - kappa) * val + 0.5 * sigma_v * sigma_v * val * val
    integral = np.empty(n_u, dtype=np.complex128)
    for col in range(n_u):
        total = 0.5 * (psi[0, col] + psi[m, col])
        for j in range(1, m):
            total += psi[j, col]
        integral[col] = dt * total
    beta = 1.0 - alpha
    fractional_terminal = np.empty(n_u, dtype=np.complex128)
    if beta <= 1e-10:
        for col in range(n_u):
            fractional_terminal[col] = psi[m, col]
    else:
        dt_beta = dt ** beta
        for col in range(n_u):
            total = 0.0 + 0.0j
            for j in range(1, m + 1):
                weight = ((m - j + 1.0) ** beta - (m - j) ** beta) * dt_beta / gamma_beta_1
                total += weight * psi[j, col]
            fractional_terminal[col] = total
    return psi, integral, fractional_terminal


def riccati_terms(u, params, tau, n_steps):
    h, v0, kappa, theta, sigma_v, rho = rough_heston_params_array(params)
    alpha = np.clip(h, 1e-6, 0.5) + 0.5
    m = max(1, int(n_steps))
    tau = max(float(tau), 1e-12)
    dt = tau / m
    t = np.linspace(0.0, tau, m + 1)
    u = np.asarray(u, dtype=np.complex128).reshape(-1)
    gamma_alpha_1 = math.gamma(alpha + 1.0)
    gamma_alpha_2 = math.gamma(alpha + 2.0)
    gamma_beta_1 = math.gamma(max(1.0 - alpha, 0.0) + 1.0)
    try:
        psi, integral, fractional_terminal = fractional_riccati_adams_numba(
            u,
            float(alpha),
            float(kappa),
            float(sigma_v),
            float(rho),
            float(tau),
            int(m),
            float(gamma_alpha_1),
            float(gamma_alpha_2),
            float(gamma_beta_1),
        )
        return t, psi, integral, fractional_terminal, alpha
    except Exception:
        pass
    psi = np.zeros((m + 1, len(u)), dtype=complex)
    f = np.zeros_like(psi)
    f[0] = riccati_rhs(u, psi[0], kappa, sigma_v, rho)
    j = np.arange(1, m + 1, dtype=float)
    predictor_weights = (j ** alpha - (j - 1.0) ** alpha) * dt ** alpha / gamma_alpha_1
    for i in range(1, m + 1):
        pred = predictor_weights[:i][::-1] @ f[:i]
        f_pred = riccati_rhs(u, pred, kappa, sigma_v, rho)
        n = i - 1
        a = np.empty(i, dtype=float)
        a[0] = n ** (alpha + 1.0) - (n - alpha) * (n + 1.0) ** alpha
        if i > 1:
            jj = np.arange(1, i, dtype=float)
            a[1:] = (n - jj + 2.0) ** (alpha + 1.0) + (n - jj) ** (alpha + 1.0) - 2.0 * (n - jj + 1.0) ** (alpha + 1.0)
        psi[i] = dt ** alpha * (a @ f[:i] + f_pred) / gamma_alpha_2
        f[i] = riccati_rhs(u, psi[i], kappa, sigma_v, rho)
    integral = np.trapezoid(psi, t, axis=0)
    beta = max(1.0 - alpha, 0.0)
    if beta <= 1e-10:
        fractional_terminal = psi[-1]
    else:
        j = np.arange(1, m + 1, dtype=float)
        frac_weights = ((m - j + 1.0) ** beta - (m - j) ** beta) * dt ** beta / gamma_beta_1
        fractional_terminal = frac_weights @ psi[1:]
    return t, psi, integral, fractional_terminal, alpha


def fractional_riccati(u, params, tau, n_steps):
    t, psi, integral, fractional_terminal, alpha = riccati_terms(np.asarray([u]), params, tau, n_steps)
    path = psi[:, 0]
    return pd.DataFrame({"t": t, "real": np.real(path), "imag": np.imag(path), "abs": np.abs(path), "alpha": alpha})


def rough_heston_cf(u, params, spot, rate, dividend_yield, tau, riccati_steps=128):
    h, v0, kappa, theta, sigma_v, rho = rough_heston_params_array(params)
    u_arr = np.asarray(u, dtype=complex)
    shape = u_arr.shape
    tau = float(tau)
    t, psi, integral, fractional_terminal, alpha = riccati_terms(u_arr.reshape(-1), params, tau, riccati_steps)
    exponent = (1j * u_arr.reshape(-1) * (np.log(float(spot)) + (float(rate) - float(dividend_yield)) * tau)
                + kappa * theta * integral + v0 * fractional_terminal)
    real = np.real(exponent)
    imag = np.imag(exponent)
    finite = np.isfinite(real) & np.isfinite(imag)
    real = np.clip(np.where(finite, real, -700.0), -700.0, 0.0)
    imag = np.where(finite, imag, 0.0)
    return np.exp(real + 1j * imag).reshape(shape)


def rough_heston_cf_diag(params, spot, rate, dividend_yield, tau, riccati_steps=512):
    u = np.r_[0.0, 0.5, 1.0, 2.0, 5.0, 10.0, -1j].astype(complex)
    h, v0, kappa, theta, sigma_v, rho = rough_heston_params_array(params)
    t, psi, integral, fractional_terminal, alpha = riccati_terms(u, params, tau, riccati_steps)
    exponent = 1j * u * (np.log(float(spot)) + (float(rate) - float(dividend_yield)) * tau) + kappa * theta * integral + v0 * fractional_terminal
    real = np.real(exponent)
    imag = np.imag(exponent)
    finite = np.isfinite(real) & np.isfinite(imag)
    clip_mask = finite & ((real > 700.0) | (real < -700.0))
    safe_phi = np.exp(np.clip(np.where(finite, real, -700.0), -700.0, 700.0) + 1j * np.where(finite, imag, 0.0))
    return pd.DataFrame([{
        "tau": float(tau),
        "riccati_steps": int(riccati_steps),
        "cf_clip_count": int(np.count_nonzero(clip_mask)),
        "cf_nonfinite_count": int(np.count_nonzero(~finite)),
        "max_real_exponent": float(np.nanmax(real)),
        "min_real_exponent": float(np.nanmin(real)),
        "phi_zero_error": float(abs(safe_phi[0] - 1.0)),
        "martingale_error": float(
            abs(safe_phi[-1] - float(spot) * np.exp((float(rate) - float(dividend_yield)) * tau))
            / max(float(spot) * np.exp((float(rate) - float(dividend_yield)) * tau), 1e-12)),
    }])


def rough_heston_px(params, strike, tau, spot, rate, dividend_yield, option_type, n_terms=96, truncation_width=16.0, riccati_steps=512):
    h, v0, kappa, theta, sigma_v, rho = rough_heston_params_array(params)
    def cf(u, tau_i, spot_i, rate_i, dividend_i):
        return rough_heston_cf(u, params, spot_i, rate_i, dividend_i, tau_i, riccati_steps=riccati_steps)
    return cos_prices("custom", None, strike, tau,
                      spot, rate, dividend_yield, option_type=option_type,
                      n_terms=int(n_terms), truncation_width=float(truncation_width), engine="cpp", cf=cf,
                      variance_hint=max(v0, theta, 1e-6))

rough_test_row = calibration_quotes.iloc[0]
rough_test_heston = heston_daily[heston_daily["date"].eq(rough_test_row["date"])]
if rough_test_heston.empty:
    rough_test_heston = heston_daily.dropna().iloc[[0]]
rough_test_params = np.array([np.clip(h_rv, 0.04, 0.35), rough_test_heston["p0"].iloc[0],
                              rough_test_heston["p1"].iloc[0], rough_test_heston["p2"].iloc[0],
                              rough_test_heston["p3"].iloc[0], rough_test_heston["p4"].iloc[0]], dtype=float)

riccati_rows = []
for u_value in [0.5 + 0.0j, 1.0 - 0.5j, 2.5 + 0.0j, 5.0 - 1.0j]:
    last_abs = np.nan
    for n in [128, 256, 512, 1024]:
        ric = fractional_riccati(u_value, rough_test_params, 90 / ann_days, n)
        terminal_abs = float(ric["abs"].iloc[-1])
        riccati_rows.append({"u": str(u_value), "riccati_steps": n, "terminal_real": float(ric["real"].iloc[-1]),
                             "terminal_imag": float(ric["imag"].iloc[-1]), "terminal_abs": terminal_abs,
                             "abs_change": abs(terminal_abs - last_abs) if np.isfinite(last_abs) else np.nan})
        last_abs = terminal_abs
riccati_grid_test = pd.DataFrame(riccati_rows)

price_rows = []
spot_test = float(rough_test_row["spot"])
rate_test = float(rough_test_row["rate"])
div_test = float(rough_test_row.get("implied_dividend_yield", 0.0))
for tau_day in [7, 30, 90]:
    tau = tau_day / ann_days
    for strike_type, k_value, option_type in [("OTM put", -0.15, "put"), ("ATM", 0.0, "call"), ("OTM call", 0.15, "call")]:
        strike = np.array([spot_test * np.exp(k_value)])
        prices_by_n = {}
        for n in [128, 256, 512, 1024]:
            prices_by_n[n] = float(rough_heston_px(rough_test_params, strike, np.array([tau]), spot_test,
                                                   rate_test, div_test, option_type, n_terms=128,
                                                   riccati_steps=n)[0])
        order_estimate = (np.log2(abs(prices_by_n[256] - prices_by_n[512])
                                  / abs(prices_by_n[512] - prices_by_n[1024]))
                          if abs(prices_by_n[512] - prices_by_n[1024]) > 1e-12 else np.nan)
        price_rows.append({"strike_type": strike_type, "tau_days": tau_day, "k": k_value,
                           "option_type": option_type, "price_128": prices_by_n[128],
                           "price_256": prices_by_n[256], "price_512": prices_by_n[512],
                           "price_1024": prices_by_n[1024],
                           "richardson_512": 2.0 * prices_by_n[512] - prices_by_n[256],
                           "abs_512_1024": abs(prices_by_n[512] - prices_by_n[1024]),
                           "order_estimate": order_estimate})
price_grid_test = pd.DataFrame(price_rows)

heston_sanity_params = rough_test_params.copy()
heston_sanity_params[0] = 0.5
heston_limit_rows = []
for tau_day in [7, 30, 90]:
    tau = tau_day / ann_days
    sanity_k = np.linspace(-0.30, 0.30, 9)
    sanity_strikes = spot_test * np.exp(sanity_k)
    sanity_tau = np.full_like(sanity_strikes, tau)
    rough_sanity_prices = rough_heston_px(heston_sanity_params, sanity_strikes, sanity_tau, spot_test,
                                          rate_test, div_test, "call", n_terms=160,
                                          riccati_steps=1024)
    heston_sanity_prices = cos_prices("heston", heston_sanity_params[1:], sanity_strikes, sanity_tau,
                                      spot_test, rate_test, div_test, option_type="call",
                                      engine="cpp", n_terms=192, truncation_width=16.0)
    for kk, strike, pr, ph in zip(sanity_k, sanity_strikes, rough_sanity_prices, heston_sanity_prices):
        heston_limit_rows.append({"tau_days": tau_day, "k": kk, "strike": strike, "rough_h_0_5_price": pr,
                                  "heston_price": ph, "abs_diff": abs(pr - ph)})
heston_limit_test = pd.DataFrame(heston_limit_rows)

cf_diagnostics = pd.concat([
    rough_heston_cf_diag(rough_test_params, spot_test, rate_test, div_test, tau_day / ann_days, riccati_steps=1024)
    for tau_day in [7, 30, 90]
], ignore_index=True)

display(riccati_grid_test.round(8))
display(price_grid_test.round(8))
display(heston_limit_test.groupby("tau_days").agg(max_abs_diff=("abs_diff", "max"), median_abs_diff=("abs_diff", "median")).round(8))
display(cf_diagnostics.round(8))

	u	riccati_steps	terminal_real	terminal_imag	terminal_abs	abs_change
0	(0.5+0j)	128	-0.025575	-0.040876	0.048218	NaN
1	(0.5+0j)	256	-0.025576	-0.040877	0.048219	1.640000e-06
2	(0.5+0j)	512	-0.025576	-0.040878	0.048220	5.200000e-07
3	(0.5+0j)	1024	-0.025576	-0.040878	0.048220	1.600000e-07
4	(1-0.5j)	128	-0.095918	0.015275	0.097127	NaN
5	(1-0.5j)	256	-0.095922	0.015275	0.097131	3.880000e-06
6	(1-0.5j)	512	-0.095923	0.015275	0.097132	1.220000e-06
7	(1-0.5j)	1024	-0.095924	0.015275	0.097132	3.900000e-07
8	(2.5+0j)	128	-0.497124	0.006300	0.497164	NaN
9	(2.5+0j)	256	-0.497151	0.006296	0.497190	2.648000e-05
10	(2.5+0j)	512	-0.497159	0.006294	0.497199	8.290000e-06
11	(2.5+0j)	1024	-0.497161	0.006294	0.497201	2.630000e-06
12	(5-1j)	128	-1.087540	0.889743	1.405129	NaN
13	(5-1j)	256	-1.087603	0.889809	1.405219	9.073000e-05
14	(5-1j)	512	-1.087622	0.889830	1.405247	2.757000e-05
15	(5-1j)	1024	-1.087628	0.889837	1.405255	8.580000e-06

	strike_type	tau_days	k	option_type	price_128	price_256	price_512	price_1024	richardson_512	abs_512_1024	order_estimate
0	OTM put	7	-0.15	put	1.089565	1.067325	1.066788	1.066476	1.066251	0.000312	0.784566
1	ATM	7	0.00	call	23.496406	23.358640	23.359538	23.359063	23.360437	0.000475	0.919472
2	OTM call	7	0.15	call	0.000000	0.011191	0.011138	0.011128	0.011085	0.000010	2.483701
3	OTM put	30	-0.15	put	11.077887	10.757598	10.755387	10.754459	10.753177	0.000928	1.252374
4	ATM	30	0.00	call	65.116460	64.902948	64.907669	64.908857	64.912390	0.001188	1.990207
5	OTM call	30	0.15	call	0.122947	0.200948	0.201054	0.200955	0.201160	0.000099	0.104372
6	OTM put	90	-0.15	put	42.858203	43.359352	43.530662	43.529066	43.701972	0.001596	6.745955
7	ATM	90	0.00	call	139.678234	140.557915	140.753864	140.757659	140.949813	0.003795	5.690342
8	OTM call	90	0.15	call	2.285561	2.735065	2.900334	2.899065	3.065604	0.001269	7.024830

	max_abs_diff	median_abs_diff
tau_days
7	0.016083	0.000527
30	0.032522	0.009349
90	0.014660	0.003519

	tau	riccati_steps	cf_clip_count	cf_nonfinite_count	max_real_exponent	min_real_exponent	phi_zero_error	martingale_error
0	0.019165	1024	0	0	8.475182	-0.019529	0.0	0.0
1	0.082136	1024	0	0	8.474307	-0.089261	0.0	0.0
2	0.246407	1024	0	0	8.472027	-0.271777	0.0	0.0

The Riccati convergence table is a numerical safety check. For several Fourier arguments and maturities, the code compares terminal values and option prices as the number of Riccati steps increases. The median differences are small, especially between 512 and 1024 steps, while some short-maturity OTM prices show visible sensitivity because those prices are tiny and highly skew-sensitive.

Those short maturities and higher Fourier frequencies are where the finite grid needs the closest numerical check. Rough Heston is being asked to price short-dated wings, where the fractional kernel and skew shape matter most. A tiny absolute price error can become meaningful in implied-vol terms when vega is small.

The diagnostic table with phi_zero_error and martingale_error is also useful. A characteristic function should satisfy:

\[ \phi(0)=1 \]

and the discounted stock process should remain consistent with the martingale condition under the risk-neutral measure. Seeing zero or near-zero errors in these checks means the numerical characteristic function is internally coherent before calibration starts.

11) Rough-Heston Calibration

The rough-Heston calibration uses the same option target grid idea as the Heston calibration, but the parameter vector expands to:

\[ \Theta_{\text{rough}} = (H,V_0,\kappa,\theta,\sigma_v,\rho) \]

In this implementation, \(H\) is often anchored or constrained by the skew-estimation step rather than freely optimized every time. We make that choice to keep the calibration comparable across the selected option dates. If \(H\) is allowed to float freely with a small number of quotes and limited function evaluations, the optimizer can use it as a noise-fitting parameter. Anchoring \(H\) to the skew power law keeps the model tied to the roughness evidence.

The objective is still a weighted pricing error:

\[ \min_{\Theta_{\text{rough}}}\sum_{i=1}^{N} w_i\left(C_i^{\text{rough}}(\Theta_{\text{rough}})-C_i^{\text{market}}\right)^2 \]

but rough Heston has extra numerical layers:

• fractional Riccati grid size,
• Fourier/COS truncation and term count,
• parameter bounds,
• skew penalty if the model-implied short-end skew moves too far from the surface-implied roughness,
• finite-price and numerical stability filters.

We also compare model errors in buckets defined by moneyness and maturity:

\[ \text{scaled residual}_i = \frac{C_i^{\text{model}}-C_i^{\text{market}}}{\text{calibration scale}_i} \]

Then it groups residuals into buckets such as deep OTM puts, near-ATM options, short maturities, and medium maturities. This is more informative than a single global RMSE because rough models are mainly supposed to help in specific regions: short maturities and skew-sensitive downside strikes.

11.1 Skew Penalty and Fixed-H Design

Rough-Heston calibration can become unstable if \(H\) freely absorbs every small residual. We use a more controlled design. The roughness parameter is guided by the skew estimate, and the calibration can include a skew penalty.

A simplified objective looks like:

\[ \mathcal{L}(\Theta) = \sum_{i=1}^{N} w_i\left(C_i^{\text{model}}(\Theta)-C_i^{\text{market}}\right)^2 + \lambda_{\text{skew}}\left(\psi_{\text{model}}-\psi_{\text{surface}}\right)^2 \]

The first part fits prices across the chosen option grid. The second keeps the model’s short-end skew close to the surface-implied skew, with \(\lambda_{\text{skew}}\) controlling how much weight that match receives alongside the price errors. A larger value places more emphasis on matching that slope.

The fixed-\(H\) or skew-anchored approach is also a good research design. It separates two questions:

1. Is the market showing roughness through realized variance and ATM skew?
2. Given that roughness, does the rough model improve pricing or Greeks?

If we optimize \(H\) freely, these questions become mixed together. A free optimizer may find an \(H\) that improves price fit but has weak roughness interpretation. Anchoring \(H\) keeps the model more honest.

Reading the Rough-Heston Diagnostics

The rough-Heston comparison shouldn’t be reduced to one RMSE number. A single weighted RMSE is useful, but rough volatility has a more specific purpose: it should help with the short-end skew and the regions where Markovian Heston struggles.

We also compare errors using bucket diagnostics, grouping each quote along two dimensions:

• moneyness bucket, based on log-moneyness \(k=\log(K/F)\),
• maturity bucket, based on time to expiry \(\tau\).

The residual is scaled first:

\[ \varepsilon_i^{\text{scaled}}=\frac{C_i^{\text{model}}-C_i^{\text{market}}}{s_i} \]

The scale \(s_i\) prevents expensive high-delta options from dominating all diagnostics. We apply the same scaling when comparing models within a bucket, so the numerator and denominator describe errors on the same basis. A raw dollar error of \(1\) has a different meaning for a deep ITM option than for a cheap OTM put. Scaling makes the comparison closer to a relative pricing-quality measure.

For each bucket, the improvement is:

\[ \text{Improvement}_{b} = 1-\frac{\text{RMSE}_{b,\text{rough}}}{\text{RMSE}_{b,\text{Heston}}} \]

A positive value means rough Heston reduces the bucket error. A negative value means ordinary Heston fits that bucket better. The separate buckets let us see an improvement in downside skew alongside a possible deterioration in near-ATM, medium-maturity fit.

On the common SPX dates, Heston leads the global metrics: lower weighted price RMSE, lower weighted IV RMSE, better bid-ask hit rate, lower OTM put RMSE, and faster runtime. That would make Heston look clearly better if we only cared about global pricing loss. At the bucket level, rough Heston has strong positive improvements in some skew-sensitive zones, including several OTM and short-to-medium maturity buckets.

The bucket results give us a more detailed picture of where rough Heston helps. It improves parts of the surface connected to rough skew behavior, while the overall fit and runtime favor ordinary Heston on this dataset. We can use both sets of results when deciding which aspect of the surface matters for the application.

For this rough-Heston calibration we use a limited set of dates and iterations, a fixed or anchored \(H\), and the numerical engine built in this project. A production calibration desk would tune optimizer restarts, parameter transformations, weighting schemes, time grids, and the Fourier/COS settings much more aggressively. So the result should be read as a controlled research comparison, not as a final claim that one model dominates across all calibration choices.

def fit_rough_heston_one_date(q, start, h_fixed, max_nfev=18, n_terms=72, riccati_steps=256, surface_fit=None, lambda_skew=0.0):
    t0 = time.perf_counter()
    day = calibration_weights(q).reset_index(drop=True)
    target = day["mid"].to_numpy(float)
    scale = day["calib_scale_px"].to_numpy(float) / np.sqrt(day["obs_weight"].to_numpy(float))
    groups = []
    for _, idx in day.groupby(["date", "expiry", "option_type"], sort=False).groups.items():
        pos = np.asarray(idx, dtype=int)
        g = day.iloc[pos]
        groups.append((pos, g["strike"].to_numpy(float), g["tau"].to_numpy(float), float(g["spot"].median()),
                       float(g["rate"].median()),
                       float(g.get("implied_dividend_yield", pd.Series(0.0, index=g.index)).median()),
                       str(g["option_type"].iloc[0])))
    lo = np.array([0.035, 1e-5, 0.05, 1e-5, 0.02, -0.999])
    hi = np.array([0.45, 4.0, 12.0, 4.0, 4.0, 0.999])
    p0 = np.clip(np.asarray(start, dtype=float), lo, hi)
    h_fixed = float(np.clip(h_fixed, lo[0], hi[0]))
    riccati_inner = max(32, riccati_steps // 3)
    _spot_d = float(day["spot"].iloc[0])
    _rate_d = float(pd.to_numeric(day.get("rate", pd.Series(0.0, index=day.index)), errors="coerce").median())
    _div_d = float(pd.to_numeric(day.get("implied_dividend_yield", pd.Series(0.0, index=day.index)), errors="coerce").median())
    _tau_vals = sorted(set(day["tau"].round(5)))
    market_skew = {}
    if surface_fit is not None and float(lambda_skew) > 0.0:
        _dk = 0.01
        for _tau_s in [7, 14, 21, 30, 45, 60]:
            _t = _tau_s / ann_days
            if _t < min(_tau_vals) - 5 / ann_days or _t > max(_tau_vals) + 5 / ann_days:
                continue
            _ilo = float(surface_iv(surface_fit, np.array([-_dk]), np.array([_t]))[0])
            _ihi = float(surface_iv(surface_fit, np.array([_dk]), np.array([_t]))[0])
            if np.isfinite(_ilo) and np.isfinite(_ihi):
                market_skew[_t] = (_ihi - _ilo) / (2 * _dk)
    _skew_scale = max(float(np.mean(np.abs(list(market_skew.values())))) if market_skew else 1.0, 0.05)
    def residual(p, rc_steps):
        px = np.empty(len(day), dtype=float)
        for pos, strike, tau, spot, rate, div, option_type in groups:
            try:
                px[pos] = rough_heston_px(p, strike, tau, spot, rate, div, option_type, n_terms=n_terms, riccati_steps=rc_steps)
            except FloatingPointError:
                px[pos] = np.nan
        if not np.all(np.isfinite(px)):
            return np.full_like(target, 1e6)
        return (px - target) / scale
    def skew_residuals(p):
        sk = []
        if not market_skew:
            return np.array([], dtype=float)
        _dk = 0.01
        for _t, _psi_mkt in market_skew.items():
            try:
                _fwd = _spot_d * np.exp((_rate_d - _div_d) * _t)
                _disc = np.exp(-_rate_d * _t)
                _Klo = np.array([_fwd * np.exp(-_dk)])
                _Khi = np.array([_fwd * np.exp(_dk)])
                _pxlo = rough_heston_px(p, _Klo, np.array([_t]), _spot_d, _rate_d, _div_d, "put", n_terms=n_terms, riccati_steps=riccati_inner)
                _pxhi = rough_heston_px(p, _Khi, np.array([_t]), _spot_d, _rate_d, _div_d, "call", n_terms=n_terms, riccati_steps=riccati_inner)
                _ivlo = float(implied_vol("put", _pxlo, _fwd, _Klo, _t, _disc)[0])
                _ivhi = float(implied_vol("call", _pxhi, _fwd, _Khi, _t, _disc)[0])
                if np.isfinite(_ivlo) and np.isfinite(_ivhi):
                    sk.append(float(np.sqrt(float(lambda_skew))) * ((_ivhi - _ivlo) / (2 * _dk) - _psi_mkt) / _skew_scale)
            except Exception:
                pass
        return np.asarray(sk, dtype=float)
    def residual_fixed(x):
        p = np.r_[h_fixed, x]
        res = residual(p, riccati_inner)
        sk = skew_residuals(p) if market_skew else np.array([], dtype=float)
        return np.concatenate([res, sk]) if sk.size else res
    fixed = optimize.least_squares(residual_fixed, p0[1:], bounds=(lo[1:], hi[1:]), max_nfev=int(max_nfev), xtol=1e-7, ftol=1e-7, gtol=1e-7)
    res_x = np.r_[h_fixed, fixed.x]
    res_fun = residual(res_x, riccati_steps)
    best_loss = float(np.nanmean(res_fun * res_fun))
    px = target + res_fun * scale
    fit = day.copy()
    fit["model"] = "rough_heston"
    fit["model_price"] = px
    fit["price_residual"] = px - target
    vega = pd.to_numeric(fit.get("vega", 1.0), errors="coerce").abs().replace(0.0, np.nan).to_numpy(float)
    inside = ((px >= pd.to_numeric(fit.get("bid", fit["mid"]), errors="coerce").to_numpy(float))
              & (px <= pd.to_numeric(fit.get("ask", fit["mid"]), errors="coerce").to_numpy(float)))
    k_col = pd.to_numeric(fit.get("k", np.zeros(len(fit))), errors="coerce").to_numpy(float)
    dte_col = pd.to_numeric(fit.get("dte_days", fit.get("tau", 0.0) * ann_days), errors="coerce").to_numpy(float)
    otype_col = fit.get("option_type", pd.Series(["call"] * len(fit))).to_numpy()
    otm_put_mask = (otype_col == "put") & (k_col < -0.10)
    short_mat_mask = dte_col <= 30
    otm_put_rmse = (float(np.sqrt(np.nanmean(((px[otm_put_mask] - target[otm_put_mask])
                                      / np.maximum(scale[otm_put_mask], 1e-8)) ** 2)))
                      if otm_put_mask.any() else np.nan)
    short_maturity_rmse = (float(np.sqrt(np.nanmean(((px[short_mat_mask] - target[short_mat_mask])
                                      / np.maximum(scale[short_mat_mask], 1e-8)) ** 2)))
                      if short_mat_mask.any() else np.nan)
    row = {"model": "rough_heston", "date": pd.Timestamp(day["date"].iloc[0]).normalize(),
           "success": bool(fixed.success or np.sqrt(best_loss) < 5.0), "nfev": int(fixed.nfev),
           "runtime_sec": float(time.perf_counter() - t0), "quotes": int(len(day)),
           "weighted_price_rmse": float(np.sqrt(np.nanmean(((px - target) / scale) ** 2))),
           "median_abs_price_error": float(np.nanmedian(np.abs(px - target))),
           "weighted_iv_rmse": float(np.sqrt(np.nanmean(((px - target) / np.maximum(vega, 1e-6)) ** 2))),
           "bid_ask_hit_rate": float(np.nanmean(inside)), "otm_put_rmse": otm_put_rmse,
           "short_maturity_rmse": short_maturity_rmse,
           "stage_fixed_rmse": float(np.sqrt(np.nanmean(fixed.fun * fixed.fun))),
           "stage_free_rmse": float(np.sqrt(best_loss)), "h_mode": "fixed", "h_anchor": h_fixed,
           "h_fixed": h_fixed, "h": float(res_x[0]), "p0": float(res_x[1]), "p1": float(res_x[2]),
           "p2": float(res_x[3]), "p3": float(res_x[4]), "p4": float(res_x[5])}
    return row, fit, res_x

rough_cache = cache_dir / "spx_rough_heston_surface_grid_fixed_h_daily.parquet"
rough_fit_cache = cache_dir / "spx_rough_heston_surface_grid_fixed_h_fit.parquet"
expected_rough_dates = pd.DatetimeIndex([
    d for d in rough_fit_dates
    if len(calibration_quotes[calibration_quotes["date"].eq(d)]) >= 45
]).sort_values()

if rough_cache.exists() and rough_fit_cache.exists():
    rough_daily = pd.read_parquet(rough_cache)
    rough_prices = pd.read_parquet(rough_fit_cache)
    if "runtime_sec" not in rough_daily.columns:
        rough_daily["runtime_sec"] = 0.0
else:
    rough_daily = pd.DataFrame()
    rough_prices = pd.DataFrame()

completed_dates = set(pd.to_datetime(rough_daily["date"]).dt.normalize()) if not rough_daily.empty and "date" in rough_daily.columns else set()
if len(completed_dates.intersection(set(expected_rough_dates))) < len(expected_rough_dates):
    rows = rough_daily.to_dict("records") if not rough_daily.empty else []
    fits = [rough_prices] if not rough_prices.empty else []
    if rows:
        last_row = pd.DataFrame(rows).sort_values("date").iloc[-1]
        last = np.array([last_row["h"], last_row["p0"], last_row["p1"], last_row["p2"], last_row["p3"], last_row["p4"]], dtype=float)
    else:
        last = None
    skew_h_source = valid_skew_dates if not valid_skew_dates.empty else skew_history
    skew_h_lookup = skew_h_source.set_index("date")["h_from_skew"].dropna().sort_index()
    for d in expected_rough_dates:
        if pd.Timestamp(d).normalize() in completed_dates:
            continue
        day = calibration_quotes[calibration_quotes["date"].eq(d)].copy()
        if len(day) < 45:
            continue
        hrow = heston_daily[heston_daily["date"].eq(d)]
        if not skew_h_lookup.empty:
            nearest_h = skew_h_lookup.reindex([pd.Timestamp(d).normalize()], method="nearest").iloc[0]
        else:
            nearest_h = h_rv
        h_fixed = float(np.clip(nearest_h if np.isfinite(nearest_h) else h_rv, 0.04, 0.45))
        if last is None:
            if not hrow.empty:
                start = np.array([h_fixed, hrow["p0"].iloc[0], hrow["p1"].iloc[0], hrow["p2"].iloc[0],
                                  hrow["p3"].iloc[0], hrow["p4"].iloc[0]], dtype=float)
            else:
                atm = float(np.nanmedian(day["iv_mid"]))
                start = np.array([h_fixed, atm * atm, 2.0, atm * atm, 0.6, -0.6], dtype=float)
        else:
            start = last
            start[0] = h_fixed
        _sfit_d = surface_fit["visual_fits"].get(pd.Timestamp(d).normalize(), surface_fit["visual_fits"].get(str(pd.Timestamp(d).date())))
        row, fit, last = fit_rough_heston_one_date(day, start, h_fixed=h_fixed, max_nfev=8,
                                                   n_terms=48, riccati_steps=128, surface_fit=_sfit_d,
                                                   lambda_skew=2.0)
        rows.append(row)
        fits.append(fit)
        rough_daily = pd.DataFrame(rows).drop_duplicates(subset=["date"], keep="last").sort_values("date")
        rough_prices = pd.concat(fits, ignore_index=True) if fits else pd.DataFrame()
        rough_daily.to_parquet(rough_cache, index=False)
        rough_prices.to_parquet(rough_fit_cache, index=False)
        completed_dates.add(pd.Timestamp(d).normalize())
        print(f"rough-Heston {pd.Timestamp(d).date()} | {len(completed_dates)}/{len(expected_rough_dates)} | "
              f"{row['runtime_sec']:.1f}s | rmse={row['weighted_price_rmse']:.3f}", flush=True)

rough_buckets = residual_by_bucket(rough_prices)

common_fit_dates = sorted(set(pd.to_datetime(heston_daily["date"]).dt.normalize()).intersection(set(pd.to_datetime(rough_daily["date"]).dt.normalize())))
heston_common = heston_daily[heston_daily["date"].isin(common_fit_dates)].copy()
rough_common = rough_daily[rough_daily["date"].isin(common_fit_dates)].copy()
heston_only_table = compare_fourier_models(daily=heston_daily)
_rough_agg_cols = dict(dates=("date", "nunique"), quotes=("quotes", "sum"), success_rate=("success", "mean"),
                       weighted_price_rmse=("weighted_price_rmse", "mean"),
                       weighted_iv_rmse=("weighted_iv_rmse", "mean"),
                       bid_ask_hit_rate=("bid_ask_hit_rate", "mean"),
                       median_abs_price_error=("median_abs_price_error", "median"),
                       runtime_sec=("runtime_sec", "sum"))
if "otm_put_rmse" in rough_daily.columns:
    _rough_agg_cols["otm_put_rmse"] = ("otm_put_rmse", "mean")
if "short_maturity_rmse" in rough_daily.columns:
    _rough_agg_cols["short_maturity_rmse"] = ("short_maturity_rmse", "mean")
rough_only_table = rough_daily.groupby("model", as_index=False).agg(**_rough_agg_cols)
_rough_common_agg = rough_common.groupby("model", as_index=False).agg(**_rough_agg_cols) if not rough_common.empty else pd.DataFrame()
model_quality = pd.concat([
    compare_fourier_models(daily=heston_common),
    _rough_common_agg,
], ignore_index=True, sort=False)

bucket_improvement = heston_buckets.merge(rough_buckets, on=["moneyness_bucket", "dte_bucket"], how="inner",
                                          suffixes=("_heston", "_rough"))
bucket_improvement["heston_abs_median_scaled_residual"] = bucket_improvement["median_scaled_residual_heston"].abs()
bucket_improvement["rough_abs_median_scaled_residual"] = bucket_improvement["median_scaled_residual_rough"].abs()
bucket_improvement["improvement_percent"] = (
    100.0 * (bucket_improvement["heston_abs_median_scaled_residual"]
             - bucket_improvement["rough_abs_median_scaled_residual"])
    / bucket_improvement["heston_abs_median_scaled_residual"].replace(0.0, np.nan))
bucket_improvement = bucket_improvement[["moneyness_bucket", "dte_bucket", "heston_abs_median_scaled_residual",
                                         "rough_abs_median_scaled_residual", "improvement_percent",
                                         "rows_heston", "rows_rough"]].sort_values("improvement_percent", ascending=False)

display(rough_daily.round(5))
display(heston_only_table.round(5))
display(rough_only_table.round(5))
display(model_quality.round(5))
display(bucket_improvement.head(12).round(4))

C:\Users\Lenovo\AppData\Local\Temp\ipykernel_16732\370601643.py:89: UserWarning: obj.round has no effect with datetime, timedelta, or period dtypes. Use obj.dt.round(...) instead.
  display(rough_daily.round(5))

	model	date	success	nfev	runtime_sec	quotes	weighted_price_rmse	median_abs_price_error	weighted_iv_rmse	bid_ask_hit_rate	...	stage_free_rmse	h_mode	h_anchor	h_fixed	h	p0	p1	p2	p3	p4
0	rough_heston	2022-01-03	True	8	9.26219	80	0.64466	1.96121	0.00700	0.6375	...	0.64466	fixed	0.36422	0.36422	0.36422	0.01242	3.75937	0.07218	1.17425	-0.78668
1	rough_heston	2022-04-29	True	8	12.41891	80	1.88975	4.31087	0.01563	0.2125	...	1.88975	fixed	0.20038	0.20038	0.20038	0.19151	11.99766	0.05690	1.53982	-0.66041
2	rough_heston	2022-05-05	True	8	13.70202	80	0.71369	2.00877	0.01209	0.5875	...	0.71369	fixed	0.30609	0.30609	0.30609	0.14020	10.69967	0.06750	1.93608	-0.63853
3	rough_heston	2022-05-09	True	8	13.30466	80	1.15201	2.31670	0.01040	0.4625	...	1.15201	fixed	0.30218	0.30218	0.30218	0.19126	12.00000	0.08222	2.72313	-0.46628
4	rough_heston	2022-05-10	True	7	11.32057	80	1.22658	2.73772	0.01044	0.4125	...	1.22658	fixed	0.37688	0.37688	0.37688	0.16132	12.00000	0.08649	3.23155	-0.40036
5	rough_heston	2022-05-18	True	8	9.83128	80	1.11831	2.62367	0.00997	0.3625	...	1.11831	fixed	0.25777	0.25777	0.25777	0.14951	11.99951	0.08156	2.91805	-0.42555
6	rough_heston	2022-06-20	True	8	10.65986	80	0.86278	1.97677	0.01202	0.4625	...	0.86278	fixed	0.36485	0.36485	0.36485	0.10012	4.94621	0.07918	1.57578	-0.42528
7	rough_heston	2022-06-21	True	8	14.37742	100	1.21301	1.98424	0.01030	0.3600	...	1.21301	fixed	0.36485	0.36485	0.36485	0.09132	10.48878	0.07326	1.62545	-0.41682
8	rough_heston	2022-09-30	True	8	13.66221	80	0.94487	1.01640	0.01703	0.7750	...	0.94487	fixed	0.42220	0.42220	0.42220	0.09756	4.01612	0.08242	1.21725	-0.56095
9	rough_heston	2022-12-09	True	8	11.15326	80	1.91462	4.09715	0.01982	0.2125	...	1.91462	fixed	0.37184	0.37184	0.37184	0.05399	4.74660	0.06320	0.87672	-0.36542
10	rough_heston	2023-04-20	True	8	11.72121	80	0.55215	1.20804	0.00632	0.6500	...	0.55215	fixed	0.42306	0.42306	0.42306	0.02201	4.05474	0.05621	1.06512	-0.71134
11	rough_heston	2023-07-31	True	8	13.29233	80	0.38222	0.84590	0.00503	0.7500	...	0.38222	fixed	0.45000	0.45000	0.45000	0.01136	3.27851	0.03562	0.67815	-0.60604
12	rough_heston	2023-11-28	True	8	11.66040	100	1.55573	0.83024	0.01894	0.4100	...	1.55573	fixed	0.40901	0.40901	0.40901	0.01141	8.74711	0.02888	1.84909	-0.27756
13	rough_heston	2023-11-29	True	8	8.64790	100	1.17846	1.06392	0.01636	0.3600	...	1.17846	fixed	0.38889	0.38889	0.38889	0.01153	11.96059	0.02896	2.47012	-0.22583
14	rough_heston	2023-11-30	True	8	11.39593	100	1.64173	1.29037	0.01555	0.3900	...	1.64173	fixed	0.41075	0.41075	0.41075	0.01325	11.99379	0.02488	2.04424	-0.31114
15	rough_heston	2023-12-05	True	8	7.07544	100	1.10942	1.30134	0.01255	0.3900	...	1.10942	fixed	0.41075	0.41075	0.41075	0.01165	11.99415	0.02709	1.85588	-0.23157
16	rough_heston	2023-12-06	True	8	14.91100	100	1.03419	1.08466	0.01705	0.3500	...	1.03419	fixed	0.41075	0.41075	0.41075	0.01300	6.67084	0.03327	1.45861	-0.29365
17	rough_heston	2023-12-29	True	8	12.93330	100	0.92813	0.96901	0.01497	0.5000	...	0.92813	fixed	0.45000	0.45000	0.45000	0.00711	11.43240	0.02621	1.47157	-0.34331

18 rows × 23 columns

	model	dates	quotes	success_rate	weighted_price_rmse	median_abs_price_error	weighted_iv_rmse	bid_ask_hit_rate	otm_put_rmse	short_maturity_rmse	runtime_sec
0	heston	44	3780	1.0	0.86366	1.2401	0.01045	0.55642	1.58404	1.10742	195.80669

	model	dates	quotes	success_rate	weighted_price_rmse	weighted_iv_rmse	bid_ask_hit_rate	median_abs_price_error	runtime_sec	otm_put_rmse	short_maturity_rmse
0	rough_heston	18	1580	1.0	1.11457	0.01286	0.46028	1.63127	211.3299	1.63515	1.41136

	model	dates	quotes	success_rate	weighted_price_rmse	median_abs_price_error	weighted_iv_rmse	bid_ask_hit_rate	otm_put_rmse	short_maturity_rmse	runtime_sec
0	heston	18	1580	1.0	0.98682	1.19828	0.01124	0.49833	1.46530	1.28191	73.65275
1	rough_heston	18	1580	1.0	1.11457	1.63127	0.01286	0.46028	1.63515	1.41136	211.32990

	moneyness_bucket	dte_bucket	heston_abs_median_scaled_residual	rough_abs_median_scaled_residual	improvement_percent	rows_heston	rows_rough
14	(-0.05, 0.05]	(30, 60]	0.2234	0.0433	80.6045	542	226
3	(-0.3, -0.15]	(60, 90]	0.5611	0.1341	76.0977	10	6
2	(-0.3, -0.15]	(30, 60]	0.3903	0.1738	55.4596	22	14
6	(-0.15, -0.05]	(0, 14]	0.9078	0.4512	50.2930	294	117
17	(-0.05, 0.05]	(120, 180]	0.1535	0.1001	34.7786	286	118
8	(-0.15, -0.05]	(30, 60]	0.4050	0.3704	8.5495	188	72
23	(0.05, 0.15]	(120, 180]	1.0609	0.9793	7.6968	7	4
22	(0.05, 0.15]	(90, 120]	0.9302	0.8601	7.5305	6	4
19	(0.05, 0.15]	(14, 30]	0.6764	0.6549	3.1817	4	4
18	(0.05, 0.15]	(0, 14]	0.4484	0.4876	-8.7584	6	6
15	(-0.05, 0.05]	(60, 90]	0.1745	0.1919	-9.9598	278	114
0	(-0.3, -0.15]	(0, 14]	3.5578	3.9547	-11.1550	33	21

fig, axes = plt.subplots(2, 3, figsize=(15, 6.4), constrained_layout=True)
axes = axes.ravel()
for i, (col, label) in enumerate([("h", "H"), ("p0", "v0"), ("p1", "kappa"), ("p2", "theta"), ("p3", "vol-of-vol"), ("p4", "rho")]):
    axes[i].plot(rough_daily["date"], rough_daily[col], marker="o", ms=2.5, lw=1.0, color=palette[i])
    axes[i].set_title(f"Rough-Heston {label}")
plt.show()

selected_skew_dates = valid_skew_dates[valid_skew_dates["date"].isin(common_fit_dates)].copy()
if selected_skew_dates.empty:
    selected_skew_dates = skew_history[skew_history["date"].isin(common_fit_dates)].copy()
rough_main_date = (selected_skew_dates.sort_values(["r2", "n"], ascending=[False, False])["date"].iloc[0]
                   if not selected_skew_dates.empty
                   else rough_prices.groupby("date").size().sort_values(ascending=False).index[0])
rough_fit_main = rough_prices[rough_prices["date"].eq(rough_main_date)].copy()
heston_fit_for_rough_date = heston_prices[heston_prices["date"].eq(rough_main_date)].copy()
for frame in [rough_fit_main, heston_fit_for_rough_date]:
    frame["k"] = np.log(frame["strike"] / frame["spot"])
rough_fit_main["model_iv"] = rough_fit_main["iv_mid"] + rough_fit_main["price_residual"] / rough_fit_main["vega"].replace(0.0, np.nan)
heston_fit_for_rough_date["model_iv"] = (heston_fit_for_rough_date["iv_mid"]
                                         + heston_fit_for_rough_date["price_residual"]
                                         / heston_fit_for_rough_date["vega"].replace(0.0, np.nan))
fig, axes = plt.subplots(1, 2, figsize=(13, 3.8), constrained_layout=True)
axes[0].scatter(rough_fit_main["k"], rough_fit_main["iv_mid"], s=10, alpha=0.35, color="black", label="market")
axes[0].scatter(heston_fit_for_rough_date["k"], heston_fit_for_rough_date["model_iv"], s=8, alpha=0.45, color=palette[0], label="heston")
axes[0].scatter(rough_fit_main["k"], rough_fit_main["model_iv"], s=8, alpha=0.55, color=palette[1], label="rough heston")
axes[0].set_title("Heston versus Rough-Heston IV Fit")
axes[0].legend()
for i, row in price_grid_test.iterrows():
    if row["strike_type"] == "ATM":
        axes[1].plot([128, 256, 512, 1024],
                     [row["price_128"], row["price_256"], row["price_512"], row["price_1024"]], marker="o",
                     lw=1.0,
                     color=palette[int(row["tau_days"] in [30, 90]) + (2 if row["tau_days"] == 90 else 0)],
                     label=f"{int(row['tau_days'])}d ATM")
axes[1].set_xscale("log", base=2)
axes[1].set_title("Rough-Heston Price Convergence")
axes[1].legend(fontsize=6)
plt.show()

Across the common dates, the overall fit and the individual moneyness and maturity buckets give us different comparisons. On the 18 common SPX fit dates, Heston has a lower global weighted price RMSE than rough Heston: about 0.987 versus 1.115, and a slightly lower weighted IV RMSE. So the rough model doesn’t simply dominate the classical model on the aggregate calibration metric.

The moneyness and maturity buckets add a more detailed view of the fit. Rough Heston improves strongly in some regions: near-ATM 30–60 day options improve by about 80.6% in median scaled residual, deep downside puts at 60–90 days improve by about 76.1%, and some short downside buckets improve meaningfully too. But there are also buckets where rough Heston is worse, including some very short deep OTM puts and some 60–90 day near-ATM regions.

The overall and bucket-level measures therefore give us complementary views of where each model fits the observed option prices well. Rough Heston is adding flexibility in the short-end skew and certain moneyness/maturity regions, but that flexibility comes with calibration cost, runtime, and sometimes worse global fit. In this run, rough Heston is a specialized improvement tool rather than a universal replacement for Heston.

The runtime difference is also part of the comparison. In the common SPX sample, rough Heston takes roughly 211 seconds versus about 74 seconds for Heston on the reported comparison. For BTC, the difference is even more dramatic. A model that improves a few buckets but costs several times more runtime has to justify itself through use case.

The use cases where rough Heston is most valuable are:

• short-dated skew-sensitive options,
• downside puts in equity indices,
• scenarios where delta and skew hedging are more important than global RMSE,
• research settings where the roughness parameter itself is the subject,
• pricing engines where characteristic-function rough models can be optimized in C++.

The use cases where plain Heston can still be enough are:

• broad surface fitting,
• faster daily batch calibration,
• medium-maturity portfolios,
• cases where global RMSE matters more than short-end skew shape.

We can make the choice using the reported fit and runtime for each model across the selected SPX dates: Heston is faster and globally stronger on this SPX fit sample, while rough Heston gives targeted improvements in skew-sensitive buckets and changes Greeks materially.

12) Model-Implied ATM Skew

After calibration, we compare market, Heston, and rough-Heston ATM skew term structures directly. We can compare the maturity profile of skew directly with the short-end power law that motivated the rough-volatility models.

For each model, we compute prices at three nearby log-moneyness nodes:

\[ k \in \{-\Delta k, 0, +\Delta k\} \]

convert those model prices back into implied volatilities, and estimate:

\[ \psi_{\text{model}}(\tau) \approx \frac{\sigma_{\text{model}}(+\Delta k,\tau)-\sigma_{\text{model}}(-\Delta k,\tau)}{2\Delta k} \]

Then we fit the same power law:

\[ |\psi_{\text{model}}(\tau)| \approx c_{\text{model}}\tau^{-\alpha_{\text{model}}} \]

with:

\[ H_{\text{model}} = \frac{1}{2} - \alpha_{\text{model}} \]

We use the fitted slopes to check how closely each model reproduces the maturity behavior of the observed short-end skew. The comparison uses the same maturity window for the selected surfaces.

The selected market date has a market-implied short-end skew fit with \(R^2\) around 0.993 and \(H\) around 0.258 from the skew window. Heston also produces a declining skew term structure, but its implied power-law estimate is different. Rough Heston produces an even steeper short-end skew curve in this selected diagnostic, with a fitted exponent that can overshoot the market.

That overshoot is useful to see. Rough models can generate steep short-end skew, but calibration still has to control how steep. Adding roughness creates the right kind of shape, but it doesn’t automatically guarantee the right magnitude.

12.1 Market Skew, Model Skew, and Calibration Tension

A model can fit option prices well and still produce the wrong skew power law. The reason is that price errors are weighted across many strikes and maturities, while the skew power law focuses on a derivative at \(k=0\) and short maturities.

The market skew diagnostic uses:

\[ \psi_{\text{market}}(\tau) = \partial_k\sigma_{\text{surface}}(0,\tau) \]

The model skew diagnostic uses:

\[ \psi_{\text{model}}(\tau) = \partial_k\sigma_{\text{model}}(0,\tau) \]

where \(\sigma_{\text{model}}\) is obtained by repricing model options and inverting them back to implied volatility.

By inverting each model price back to implied volatility, we compare the implied-volatility surface generated by the model directly with the market surface at the same strikes and maturities. That keeps the diagnostic in the same units as option traders use. We can then compare slopes on a common scale.

If rough Heston generates a too-steep short-end skew, the fitted \(H_{\text{model}}\) from the power law can become lower or even unstable. If it generates a too-flat skew, \(H_{\text{model}}\) becomes too high. The goal isn’t only to make \(H\) small; it’s to match the observed maturity decay of skew.

def model_atm_skew(model_name, params, spot, rate, dividend_yield, tau_values, dk=0.01):
    rows = []
    for tau in np.asarray(tau_values, dtype=float):
        k_nodes = np.array([-dk, 0.0, dk], dtype=float)
        strikes = spot * np.exp(k_nodes)
        tau_arr = np.full_like(strikes, tau)
        if model_name == "heston":
            price = cos_prices("heston", params, strikes, tau_arr,
                               spot, rate, dividend_yield, option_type="call",
                               engine="cpp", n_terms=192, truncation_width=16.0)
        elif model_name == "rough_heston":
            price = rough_heston_px(params, strikes, tau_arr, spot, rate, dividend_yield, "call", n_terms=128, riccati_steps=512)
        else:
            continue
        forward = spot * np.exp((rate - dividend_yield) * tau)
        discount = np.exp(-rate * tau)
        iv = implied_vol("call", price, forward, strikes, tau_arr, discount, engine="auto")
        rows.append({"model": model_name, "tau": tau, "tau_days": tau * ann_days, "atm_iv": float(iv[1]),
                     "atm_skew": float((iv[2] - iv[0]) / (2.0 * dk)), "n": 3})
    return pd.DataFrame(rows)


selected_date = pd.Timestamp(rough_main_date).normalize()
selected_quotes = surface_quotes[surface_quotes["date"].eq(selected_date)].copy()
selected_fit = surface_fit["visual_fits"][selected_date]
selected_spot = float(selected_quotes["spot"].median())
selected_rate = float(selected_quotes["rate"].median())
selected_div = float(selected_quotes["implied_dividend_yield"].median())
selected_market_skew = atm_skew(selected_quotes, selected_fit, tau_values).assign(model="market")
selected_heston_row = heston_daily[heston_daily["date"].eq(selected_date)].iloc[0]
selected_rough_row = rough_daily[rough_daily["date"].eq(selected_date)].iloc[0]
selected_heston_params = np.array([selected_heston_row["p0"], selected_heston_row["p1"],
                                   selected_heston_row["p2"], selected_heston_row["p3"],
                                   selected_heston_row["p4"]], dtype=float)
selected_rough_params = rough_heston_params_array(selected_rough_row)
selected_heston_skew = model_atm_skew("heston", selected_heston_params, selected_spot, selected_rate, selected_div, tau_values)
selected_rough_skew = model_atm_skew("rough_heston", selected_rough_params, selected_spot, selected_rate, selected_div, tau_values)
model_skew_table = pd.concat([selected_market_skew[["model", "tau", "tau_days", "atm_iv",
                                                    "atm_skew", "n"]], selected_heston_skew, selected_rough_skew], ignore_index=True)
model_skew_power = pd.concat([
    power_law(model_skew_table[model_skew_table["model"].eq(model)], tau_min_days=7, tau_max_days=60, window=model).assign(model=model)
    for model in model_skew_table["model"].unique()
], ignore_index=True)

display(model_skew_table.round(6))
display(model_skew_power[["model", "alpha", "h_from_skew", "r2", "n"]].round(5))

fig, axes = plt.subplots(1, 2, figsize=(13, 3.8), constrained_layout=True)
for i, (model, g) in enumerate(model_skew_table.groupby("model", sort=False)):
    axes[0].plot(g["tau_days"], g["atm_skew"], marker="o", lw=1.2, color=palette[i], label=model)
    axes[1].plot(g["tau"], g["atm_skew"].abs(), marker="o", lw=1.2, color=palette[i], label=model)
axes[0].set_title(f"Model-Implied ATM Skew Term Structures {selected_date.date()}")
axes[0].set_xlabel("days")
axes[0].set_ylabel("d sigma / dk")
axes[1].set_xscale("log")
axes[1].set_yscale("log")
axes[1].set_title("Short-End Skew Power Shape")
axes[1].set_xlabel("tau")
axes[1].set_ylabel("|ATM skew|")
for ax in axes:
    ax.legend()
plt.show()

e:\daneshgah\quantitative-finance-lab\.venv\Lib\site-packages\numba\core\typing\typeof.py:268: FutureWarning: future versions will not create a writeable array from broadcast_array. Set the writable flag explicitly to avoid this warning.
  readonly = not val.flags.writeable

	model	tau	tau_days	atm_iv	atm_skew	n
0	market	0.019165	7.0	0.321564	-0.970961	280
1	market	0.027379	10.0	0.290005	-0.860513	420
2	market	0.038330	14.0	0.280243	-0.802222	560
3	market	0.057495	21.0	0.276439	-0.741694	420
4	market	0.082136	30.0	0.275759	-0.681182	280
5	market	0.123203	45.0	0.273102	-0.601959	280
6	market	0.164271	60.0	0.270727	-0.558568	140
7	market	0.246407	90.0	0.266568	-0.518243	140
8	market	0.328542	120.0	0.263562	-0.488308	140
9	market	0.410678	150.0	0.261370	-0.455412	0
10	heston	0.019165	7.0	0.303379	-1.073560	3
11	heston	0.027379	10.0	0.296588	-1.037622	3
12	heston	0.038330	14.0	0.289222	-0.975800	3
13	heston	0.057495	21.0	0.279944	-0.868087	3
14	heston	0.082136	30.0	0.272414	-0.754458	3
15	heston	0.123203	45.0	0.265709	-0.620172	3
16	heston	0.164271	60.0	0.262599	-0.529169	3
17	heston	0.246407	90.0	0.260724	-0.413476	3
18	heston	0.328542	120.0	0.260996	-0.342077	3
19	heston	0.410678	150.0	0.261921	-0.293018	3
20	rough_heston	0.019165	7.0	0.292295	-1.565273	3
21	rough_heston	0.027379	10.0	0.285155	-1.324675	3
22	rough_heston	0.038330	14.0	0.279112	-1.120491	3
23	rough_heston	0.057495	21.0	0.273006	-0.904077	3
24	rough_heston	0.082136	30.0	0.268896	-0.739813	3
25	rough_heston	0.123203	45.0	0.265776	-0.581053	3
26	rough_heston	0.164271	60.0	0.264547	-0.485126	3
27	rough_heston	0.246407	90.0	0.264052	-0.371242	3
28	rough_heston	0.328542	120.0	0.264416	-0.304085	3
29	rough_heston	0.410678	150.0	0.265005	-0.259002	3

	model	alpha	h_from_skew	r2	n
0	market	0.24223	0.25777	0.99258	6
1	heston	0.37881	0.12119	0.97210	6
2	rough_heston	0.56082	-0.06082	0.99845	6

Comparing the market, Heston, and rough-Heston skew term structures lets us see the maturity profiles directly. The market skew is strongly negative at the shortest maturities and becomes less negative as maturity increases. Heston captures part of this behavior but tends to be limited by its Markovian diffusion structure. Rough Heston bends the short-end shape more aggressively.

In the log-log panel, the rough-volatility signature is a near-linear relationship between \(\log|\psi(\tau)|\) and \(\log\tau\). The market curve has a clean power-law shape on the selected date. The model curves are also close to linear over the tested maturities, but their slopes differ.

This diagnostic helps separate two ideas:

• fitting option prices well in aggregate,
• reproducing the short-maturity skew scaling.

A model can do one better than the other. We compare both because rough volatility gives us a theory about the shape and scaling of volatility as well as another parameter with which to fit option prices.

13) Delta Under Rough Volatility

The last SPX modeling layer asks how rough volatility changes hedge sensitivities. We already covered Greeks in Project 4 and surface-aware Greeks in Project 8, so here we focus on the comparison.

Delta is:

\[ \Delta = \frac{\partial C}{\partial S} \]

For model prices that don’t have a simple closed-form delta, we use a finite difference:

\[ \Delta \approx \frac{C(S+\Delta S)-C(S-\Delta S)}{2\Delta S} \]

with:

\[ \Delta S = bS \]

where \(b\) is the relative bump size. We compare four different delta notions:

• flat BSM delta using one ATM volatility,
• surface BSM delta using the local implied volatility at each \((k,\tau)\),
• Heston delta from finite differences,
• rough-Heston delta from finite differences.

In a skewed stochastic-volatility model, delta reflects a different option-pricing distribution from the one used in flat BSM. When spot moves, model prices also reflect the specified variance dynamics and spot-volatility correlation. Rough volatility affects this interaction because variance shocks are more reactive near the present.

For an equity index with negative spot-vol correlation, downside moves usually come with higher variance. That makes downside option values more sensitive to spot than flat BSM suggests. We measure the finite-difference model delta by repricing the option under the two bumped spot levels.

13.1 Delta with Smile Feedback

Flat BSM delta treats volatility as constant:

\[ \Delta_{\text{flat}} = \frac{\partial C(S,K,\tau,\sigma)}{\partial S} \]

Surface-BSM delta uses the implied volatility at each strike and maturity, but it still treats the surface mostly as an input map:

\[ \Delta_{\text{surface}} \approx \frac{\partial C(S,K,\tau,\sigma_{\text{imp}}(k,\tau))}{\partial S} \]

A stochastic-volatility model delta is different because the model links spot and volatility through \(\rho\) and through the variance process. When spot moves, the model-implied distribution changes. For rough Heston:

\[ \Delta_{\text{rough}} \approx \frac{C_{\text{rough}}(S+\Delta S)-C_{\text{rough}}(S-\Delta S)}{2\Delta S} \]

The two bumped prices use the full rough-Heston pricing model, not just a BSM formula with a local IV. This captures the model’s belief about how spot and variance interact.

For downside equity puts and near-ATM short-dated calls, this can matter a lot. A negative spot shock increases the probability of higher variance in leverage-effect models. That changes the option’s value beyond the direct intrinsic-value effect. Roughness can amplify the short-horizon version of this effect.

def heston_delta_fd(params, strike, tau, spot, rate, dividend_yield, bump):
    ds = spot * bump
    p_up = cos_prices("heston", params, strike, tau,
                      spot + ds, rate, dividend_yield, option_type="call",
                      engine="cpp", n_terms=128, truncation_width=16.0)
    p_dn = cos_prices("heston", params, strike, tau,
                      spot - ds, rate, dividend_yield, option_type="call",
                      engine="cpp", n_terms=128, truncation_width=16.0)
    return (p_up - p_dn) / (2.0 * ds)


def rough_delta_fd(params, strike, tau, spot, rate, dividend_yield, bump):
    ds = spot * bump
    p_up = rough_heston_px(params, strike, tau, spot + ds, rate, dividend_yield, "call")
    p_dn = rough_heston_px(params, strike, tau, spot - ds, rate, dividend_yield, "call")
    return (p_up - p_dn) / (2.0 * ds)

delta_k = np.linspace(-0.35, 0.25, 49)
delta_days = np.array([7, 14, 30, 60, 90], dtype=float)
delta_rows = []
delta_date = pd.Timestamp(rough_main_date).normalize()
delta_quotes = surface_quotes[surface_quotes["date"].eq(delta_date)].copy()
spot_delta = float(delta_quotes["spot"].median())
heston_delta_row = heston_daily[heston_daily["date"].eq(delta_date)].iloc[0]
rough_delta_row = rough_daily[rough_daily["date"].eq(delta_date)].iloc[0]
heston_params_last = np.array([heston_delta_row["p0"], heston_delta_row["p1"], heston_delta_row["p2"],
                               heston_delta_row["p3"], heston_delta_row["p4"]], dtype=float)
rough_params_last = rough_heston_params_array(rough_delta_row)
rate_main = float(delta_quotes["rate"].median())
div_main = float(delta_quotes["implied_dividend_yield"].median())
sigma_atm = float(delta_quotes["iv_mid"].median())
surface_fit_by_date = {pd.Timestamp(k).normalize(): v for k, v in surface_fit["visual_fits"].items()}
delta_surface_fit = surface_fit_by_date.get(delta_date)

for d in delta_days:
    tau = d / ann_days
    strike = spot_delta * np.exp(delta_k)
    forward = spot_delta * np.exp((rate_main - div_main) * tau)
    discount = np.exp(-rate_main * tau)
    flat_bsm_delta = np.asarray(forward_bsm_delta("call", forward, strike, tau,
                                                  sigma_atm, discount), dtype=float) * np.exp((rate_main - div_main) * tau)
    if delta_surface_fit is not None:
        surface_sigma = np.asarray(surface_iv(delta_surface_fit, delta_k, np.full_like(delta_k, tau)), dtype=float)
        surface_sigma = np.clip(surface_sigma, 0.03, 5.0)
    else:
        surface_sigma = np.full_like(delta_k, sigma_atm, dtype=float)
    surface_bsm_delta = np.asarray(forward_bsm_delta("call", forward, strike, tau,
                                                     surface_sigma, discount), dtype=float) * np.exp((rate_main - div_main) * tau)
    heston_delta = heston_delta_fd(heston_params_last, strike, np.full_like(strike, tau), spot_delta, rate_main, div_main, 0.005)
    rough_delta = rough_delta_fd(rough_params_last, strike, np.full_like(strike, tau), spot_delta, rate_main, div_main, 0.005)
    for kk, st, db_flat, db_surface, dh, dr in zip(delta_k, strike, flat_bsm_delta, surface_bsm_delta, heston_delta, rough_delta):
        delta_rows.append({"tau_days": d, "tau": tau, "k": kk, "strike": st,
                           "bsm_delta": db_flat, "flat_bsm_delta": db_flat, "surface_bsm_delta": db_surface,
                           "heston_delta": dh, "rough_heston_delta": dr, "rough_minus_bsm": dr - db_flat,
                           "rough_minus_flat_bsm": dr - db_flat, "rough_minus_surface_bsm": dr - db_surface,
                           "rough_minus_heston": dr - dh})

delta_map = pd.DataFrame(delta_rows)
atm_delta_table = delta_map[np.isclose(delta_map["k"], 0.0, atol=0.015)].copy()
display(atm_delta_table.round(5))

	tau_days	tau	k	strike	bsm_delta	flat_bsm_delta	surface_bsm_delta	heston_delta	rough_heston_delta	rough_minus_bsm	rough_minus_flat_bsm	rough_minus_surface_bsm	rough_minus_heston
27	7.0	0.01916	-0.0125	3876.54902	0.63408	0.63408	0.61259	0.67719	0.72741	0.09333	0.09333	0.11482	0.05022
28	7.0	0.01916	0.0000	3925.31000	0.50391	0.50391	0.50583	0.56444	0.59206	0.08815	0.08815	0.08623	0.02762
29	7.0	0.01916	0.0125	3974.68432	0.37330	0.37330	0.39048	0.43061	0.41308	0.03978	0.03978	0.02259	-0.01754
76	14.0	0.03833	-0.0125	3876.54902	0.59825	0.59825	0.59329	0.67115	0.69996	0.10172	0.10172	0.10668	0.02882
77	14.0	0.03833	0.0000	3925.31000	0.50540	0.50540	0.50591	0.58274	0.59333	0.08793	0.08793	0.08742	0.01059
78	14.0	0.03833	0.0125	3974.68432	0.41224	0.41224	0.41185	0.47694	0.46169	0.04945	0.04945	0.04984	-0.01525
125	30.0	0.08214	-0.0125	3876.54902	0.57128	0.57128	0.56945	0.66269	0.66900	0.09772	0.09772	0.09955	0.00631
126	30.0	0.08214	0.0000	3925.31000	0.50761	0.50761	0.50800	0.59396	0.59146	0.08385	0.08385	0.08346	-0.00250
127	30.0	0.08214	0.0125	3974.68432	0.44373	0.44373	0.44241	0.51417	0.50204	0.05831	0.05831	0.05963	-0.01213
174	60.0	0.16427	-0.0125	3876.54902	0.55524	0.55524	0.55493	0.64633	0.64224	0.08700	0.08700	0.08731	-0.00409
175	60.0	0.16427	0.0000	3925.31000	0.51021	0.51021	0.51017	0.59449	0.58722	0.07701	0.07701	0.07705	-0.00727
176	60.0	0.16427	0.0125	3974.68432	0.46504	0.46504	0.46288	0.53691	0.52681	0.06177	0.06177	0.06392	-0.01010
223	90.0	0.24641	-0.0125	3876.54902	0.54870	0.54870	0.54871	0.63444	0.62798	0.07928	0.07928	0.07927	-0.00647
224	90.0	0.24641	0.0000	3925.31000	0.51198	0.51198	0.51133	0.59190	0.58381	0.07183	0.07183	0.07248	-0.00808
225	90.0	0.24641	0.0125	3974.68432	0.47515	0.47515	0.47199	0.54578	0.53639	0.06125	0.06125	0.06440	-0.00939

fig, axes = plt.subplots(1, 3, figsize=(15, 3.6), constrained_layout=True)
plot_delta_day = 30.0 if 30.0 in set(delta_map["tau_days"]) else float(delta_map["tau_days"].iloc[0])
delta_slice = delta_map[delta_map["tau_days"].eq(plot_delta_day)].copy()
for i, (col, label) in enumerate([("flat_bsm_delta", "flat BSM"), ("surface_bsm_delta", "surface BSM"),
                                  ("heston_delta", "Heston"), ("rough_heston_delta", "rough Heston")]):
    axes[0].plot(delta_slice["k"], delta_slice[col], lw=1.1, color=palette[i], label=label)
for ax, col, title in [(axes[1], "rough_minus_surface_bsm", "Rough-Heston minus Surface-BSM Delta"),
                       (axes[2], "rough_minus_heston", "Rough-Heston minus Heston Delta")]:
    piv = delta_map.pivot_table(index="tau_days", columns="k", values=col)
    im = ax.imshow(piv.to_numpy(dtype=float), origin="lower", aspect="auto",
                   extent=[piv.columns.min(), piv.columns.max(), piv.index.min(), piv.index.max()],
                   cmap="coolwarm")
    ax.set_title(title)
    ax.set_xlabel("log moneyness")
    ax.set_ylabel("days")
    fig.colorbar(im, ax=ax, fraction=0.046, pad=0.04)
axes[0].set_title(f"{plot_delta_day:.0f}d Delta Curves")
axes[0].set_xlabel("log moneyness")
axes[0].legend(ncol=2)
plt.show()

Rough-Heston deltas are often higher than flat BSM and surface-BSM deltas around ATM and slightly downside moneyness in the displayed results. For example, at 30 days and near ATM, flat BSM delta is around 0.508, surface-BSM delta is around 0.508, while rough-Heston delta is around 0.591. That’s a large difference for hedge sizing.

The heatmaps show where the difference is strongest. Rough-Heston minus surface-BSM delta is positive across much of the tested grid, especially on the downside and around short to medium maturities. Rough-Heston minus Heston is smaller and can change sign depending on moneyness, which means the stochastic-volatility effect is already present in Heston, while the roughness component adds a more specific adjustment.

Those differences connect the fitted option surface to the amount of the underlying needed for a delta hedge. Even if a rough model doesn’t win every calibration metric, it can change Greeks in regions where hedging is most fragile. If a trader hedges a short-dated skew position with flat-BSM delta, the hedge can be materially under-sized.

A practical hedge example makes the difference concrete. Suppose a trader is short 1,000 near-ATM 30-day SPX calls. If flat/surface BSM delta is around \(0.508\) but rough-Heston delta is around \(0.591\), the hedge difference is:

\[ 1000 \times (0.591-0.508) \approx 83 \]

That’s about 83 SPX units of delta exposure. On an index level near 3,925 in the selected example, this isn’t a small discrepancy. The direction matters as well: under the rough model the position is more spot-sensitive than under the surface-BSM calculation.

The example gives us a way to compare hedge exposure alongside calibration RMSE when judging the models. A pricing model can affect risk management through Greeks and hedge ratios, not only through the quoted model price.

14) SPX Dashboard and Evidence Summary

The SPX dashboard combines the whole workflow:

• realized-variance roughness,
• moment-scaling diagnostics,
• forecast comparisons,
• smooth IV smiles,
• ATM skew power law,
• Heston versus rough-Heston fits,
• rBergomi path/smile behavior,
• rough-Heston delta gaps.

The roughness evidence table is the main summary. It shows several estimates of \(H\):

Source	Approximate interpretation
raw daily squared-return RV	too noisy, weak scaling fit
rolling Project-5 RV proxies	smoother but overlap can inflate \(H\)
non-overlapping/block checks	closer to rough short-time behavior
ATM skew power law	risk-neutral roughness from option prices
rough-Heston calibration	model-implied roughness under pricing constraints

The table reports the high-quality ATM skew estimate around 0.365, the broad ATM skew median around 0.411, and rough-Heston calibrated median \(H\) around 0.383. The realized-variance diagnostics are lower when we account for proxy construction. That difference isn’t a contradiction. It shows that historical variance roughness and option-implied roughness are related but not identical objects.

We can also compare the model-fit metrics alongside the roughness and skew evidence in the dashboard. Heston has better global fit on the common date set, while rough Heston improves some skew-sensitive buckets. The combined evidence lets us compare the fitted surfaces, short-end skew, and hedge sensitivities alongside the model’s numerical and runtime behavior. It gives a richer way to represent short-end skew and variance roughness, but model selection still depends on the objective.

spx_tables = {
    "roughness_evidence": pd.concat([
        roughness_proxy_table,
        pd.DataFrame([{"source": "ATM skew power law",
                       "sample": f"{skew_history['date'].min().date()} to {skew_history['date'].max().date()}",
                       "h": float(skew_history["h_from_skew"].median()), "standard_error": np.nan,
                       "r2": float(skew_history["r2"].median()), "n": int(skew_history["n"].sum()),
                       "interpretation": "option-skew roughness"}]),
        pd.DataFrame([{"source": "ATM skew power law, high quality",
                       "sample": (f"{valid_skew_dates['date'].min().date()} to {valid_skew_dates['date'].max().date()}"
                                  if not valid_skew_dates.empty else "no dates pass filter"),
                       "h": float(valid_skew_dates["h_from_skew"].median()) if not valid_skew_dates.empty else np.nan,
                       "standard_error": np.nan,
                       "r2": float(valid_skew_dates["r2"].median()) if not valid_skew_dates.empty else np.nan,
                       "n": int(valid_skew_dates["n"].sum()) if not valid_skew_dates.empty else 0,
                       "interpretation": "R2>=0.80 and at least six maturities"}]),
        pd.DataFrame([{"source": "rough-Heston calibration",
                       "sample": f"{rough_daily['date'].min().date()} to {rough_daily['date'].max().date()}",
                       "h": float(rough_daily["h"].median()), "standard_error": np.nan, "r2": np.nan,
                       "n": int(rough_daily["quotes"].sum()), "interpretation": "risk-neutral calibrated H"}]),
    ], ignore_index=True).round(5),
    "rv_roughness": emp_hurst.round(5),
    "atm_skew_roughness": skew_history[["date", "alpha", "h_from_skew", "r2", "n"]].round(5),
    "fit_quality": model_quality.round(5),
    "skew_quality_filter": skew_quality_summary.round(5),
    "bucket_improvement": bucket_improvement.head(12).round(5),
    "delta_differences": atm_delta_table.round(5),
}

for name, table in spx_tables.items():
    print(name)
    display(table)

fig, axes = plt.subplots(3, 4, figsize=(24, 14), constrained_layout=True)
axes = axes.ravel()
axes[0].plot(spx_spot.index, spx_spot["close"], lw=0.8, color=palette[0])
axes0b = axes[0].twinx()
axes0b.plot(spx_var.index, np.sqrt(ann_trading * spx_var), lw=0.6, alpha=0.55, color=palette[1])
axes[0].set_title("SPX Price and Realized Variance")
for i, (q, g) in enumerate(emp_scaling.groupby("q")):
    axes[1].plot(g["log_lag"], g["log_moment"], marker="o", ms=2.5, lw=0.9, color=palette[i], label=f"q={q:g}")
axes[1].set_title("Empirical Moment Scaling")
axes[1].legend(ncol=2)
axes[2].scatter(emp_hurst["q"], emp_hurst["slope"], s=25, color=palette[0])
axes[2].plot(np.asarray(q_values), h_rv * np.asarray(q_values), color=palette[1], lw=1.2)
axes[2].set_title("qH Linearity")
heat = forecast_scores.pivot_table(index="model", columns="horizon", values="qlike_var", aggfunc="mean")
axes[3].imshow(heat.to_numpy(dtype=float), aspect="auto", cmap="viridis")
axes[3].set_yticks(range(len(heat.index)))
axes[3].set_yticklabels(heat.index)
axes[3].set_xticks(range(len(heat.columns)))
axes[3].set_xticklabels(heat.columns)
axes[3].set_title("Forecast Race")
for i, d in enumerate([14, 30, 60, 90]):
    k_line = np.linspace(k_grid["k"].min(), k_grid["k"].max(), 80)
    axes[4].plot(k_line, surface_iv(fit_visual, k_line, np.full_like(k_line, d / ann_days)), lw=1.0, color=palette[i], label=f"{d}d")
axes[4].scatter(q_main["k"], q_main["iv_mid"], s=4, alpha=0.15, color="black")
axes[4].set_title("Smooth IV Smiles")
axes[4].legend(ncol=2)
axes[5].scatter(psi_main["tau"], psi_main["atm_skew"].abs(), s=24, color=palette[1])
xs = np.linspace(psi_main["tau"].min(), psi_main["tau"].max(), 80)
axes[5].plot(xs, skew_fit_main["c"].iloc[0] * xs ** (-skew_fit_main["alpha"].iloc[0]), color=palette[2])
axes[5].set_xscale("log")
axes[5].set_yscale("log")
axes[5].set_title("ATM Skew Power Law")
axes[6].scatter(rough_fit_main["k"], rough_fit_main["iv_mid"], s=8, alpha=0.30, color="black")
axes[6].scatter(heston_fit_for_rough_date["k"], heston_fit_for_rough_date["model_iv"], s=7, alpha=0.50, color=palette[0], label="heston")
axes[6].scatter(rough_fit_main["k"], rough_fit_main["model_iv"], s=7, alpha=0.55, color=palette[1], label="rough")
axes[6].set_title("Heston vs Rough-Heston IV")
axes[6].legend()
axes[7].bar(model_quality["model"], model_quality["weighted_price_rmse"], color=palette[:len(model_quality)])
axes[7].set_title("Model Fit Quality")
axes[7].tick_params(axis="x", rotation=15)
for i, (model, g) in enumerate(model_skew_table.groupby("model", sort=False)):
    axes[8].plot(g["tau_days"], g["atm_skew"], marker="o", lw=1.0, color=palette[i], label=model)
axes[8].set_title("Market vs Model ATM Skew")
axes[8].set_xlabel("days")
axes[8].legend(ncol=1)
for i in range(12):
    axes[9].plot(time_grid, rbergomi_v[i], lw=0.5, alpha=0.5)
axes[9].set_title("rBergomi Variance Paths")
for i, (d, g) in enumerate(rbergomi_iv.groupby("tau_days")):
    axes[10].plot(g["k"], g["iv"], lw=1.0, color=palette[i], label=f"{d:.0f}d")
axes[10].set_title("rBergomi Smile Generation")
axes[10].legend(ncol=2)
piv = delta_map.pivot_table(index="tau_days", columns="k", values="rough_minus_surface_bsm")
axes[11].imshow(piv.to_numpy(dtype=float), origin="lower", aspect="auto",
                extent=[piv.columns.min(), piv.columns.max(), piv.index.min(), piv.index.max()], cmap="coolwarm")
axes[11].set_title("Rough-Heston vs Surface-BSM Delta")
fig.suptitle("Project 18 SPX Rough Volatility Dashboard", fontsize=20, y=1.02)
plt.show()

roughness_evidence

C:\Users\Lenovo\AppData\Local\Temp\ipykernel_16732\57385579.py:9: UserWarning: obj.round has no effect with datetime, timedelta, or period dtypes. Use obj.dt.round(...) instead.
  "atm_skew_roughness": skew_history[["date", "alpha", "h_from_skew", "r2", "n"]].round(5),

	source	sample	h	standard_error	r2	n	interpretation
0	raw daily squared-return RV	1990-01-03 to 2026-02-24	0.00176	0.00552	0.04415	30	noisy diagnostic
1	rolling Project-5 RV 5d	1990-01-09 to 2026-02-24	0.28778	0.07485	0.89108	30	aggregated Project-5 RV proxy
2	rolling Project-5 RV 10d	1990-01-16 to 2026-02-24	0.45690	0.06523	0.96346	30	aggregated Project-5 RV proxy
3	rolling Project-5 RV 21d	1990-01-31 to 2026-02-24	0.60387	0.03104	0.99442	30	aggregated Project-5 RV proxy
4	ATM skew power law	2022-01-03 to 2023-12-29	0.41075	NaN	0.86409	342	option-skew roughness
5	ATM skew power law, high quality	2022-01-03 to 2023-12-29	0.36454	NaN	0.96314	204	R2>=0.80 and at least six maturities
6	rough-Heston calibration	2022-01-03 to 2023-12-29	0.38288	NaN	NaN	1580	risk-neutral calibrated H

rv_roughness

	q	slope	h	standard_error	r2	n	proxy
0	0.5	0.27316	0.54632	0.03211	0.94762	6	rolling Project-5 RV 10d
1	1.0	0.49568	0.49568	0.05166	0.95836	6	rolling Project-5 RV 10d
2	1.5	0.68534	0.45690	0.06523	0.96503	6	rolling Project-5 RV 10d
3	2.0	0.85008	0.42504	0.07508	0.96974	6	rolling Project-5 RV 10d
4	3.0	1.12233	0.37411	0.08693	0.97657	6	rolling Project-5 RV 10d

atm_skew_roughness

	date	alpha	h_from_skew	r2	n
1	2022-01-03	0.13578	0.36422	0.87793	6
5	2022-01-31	0.26566	0.23434	0.99685	6
9	2022-02-02	0.21460	0.28540	0.98894	6
13	2022-02-28	0.38783	0.11217	0.99937	6
17	2022-03-04	0.35188	0.14812	0.99284	6
21	2022-03-31	0.09502	0.40498	0.86603	6
25	2022-04-05	0.07473	0.42527	0.69536	6
29	2022-04-29	0.29962	0.20038	0.97973	6
33	2022-05-05	0.19391	0.30609	0.90120	6
37	2022-05-06	0.13975	0.36025	0.75728	6
41	2022-05-09	0.19782	0.30218	0.98217	6
45	2022-05-10	0.12312	0.37688	0.88089	6
49	2022-05-18	0.24223	0.25777	0.99258	6
53	2022-05-31	0.06117	0.43883	0.83296	6
57	2022-06-07	0.14117	0.35883	0.98277	6
61	2022-06-20	0.03717	0.46283	0.33227	6
65	2022-06-21	0.01448	0.48552	0.06745	6
69	2022-06-30	0.13515	0.36485	0.97861	6
73	2022-07-08	0.05648	0.44352	0.50554	6
77	2022-07-29	-0.11417	0.61417	0.51694	6
81	2022-08-09	0.17119	0.32881	0.84931	6
85	2022-08-31	0.05600	0.44400	0.30562	6
89	2022-09-08	0.09790	0.40210	0.84513	6
93	2022-09-30	0.01719	0.48281	0.08592	6
97	2022-10-10	0.07780	0.42220	0.82608	6
101	2022-10-31	0.06830	0.43170	0.50203	6
105	2022-11-09	-0.06173	0.56173	0.88331	6
109	2022-11-10	0.08472	0.41528	0.41592	6
113	2022-11-30	0.02880	0.47120	0.39287	6
117	2022-12-09	-0.03141	0.53141	0.57646	6
121	2022-12-30	0.12816	0.37184	0.99754	6
125	2023-01-13	0.06733	0.43267	0.63428	6
129	2023-01-31	0.15387	0.34613	0.94650	6
133	2023-02-15	0.12337	0.37663	0.87162	6
137	2023-02-28	0.05734	0.44266	0.28164	6
141	2023-03-20	0.36513	0.13487	0.99986	6
145	2023-03-31	0.03189	0.46811	0.59285	6
149	2023-04-20	0.09893	0.40107	0.58431	6
153	2023-04-28	0.07694	0.42306	0.84804	6
157	2023-05-22	0.06970	0.43030	0.41679	6
161	2023-05-31	0.11327	0.38673	0.73063	6
165	2023-06-23	-0.13428	0.63428	0.86409	6
169	2023-06-30	-0.24382	0.74382	0.97560	6
173	2023-07-25	0.04198	0.45802	0.91907	6
177	2023-07-31	-0.06513	0.56513	0.79412	6
181	2023-08-24	0.03880	0.46120	0.19147	6
185	2023-08-31	-0.08030	0.58030	0.43048	6
189	2023-09-26	0.20628	0.29372	0.99755	6
193	2023-09-29	0.22174	0.27826	0.99732	6
197	2023-10-26	0.15271	0.34729	0.95690	6
201	2023-10-31	0.19585	0.30415	0.99931	6
205	2023-11-28	0.09099	0.40901	0.91038	6
209	2023-11-29	0.11111	0.38889	0.96843	6
213	2023-11-30	0.08925	0.41075	0.95785	6
217	2023-12-05	-0.03545	0.53545	0.52775	6
221	2023-12-06	0.04641	0.45359	0.56122	6
225	2023-12-29	-0.13946	0.63946	0.97303	6

fit_quality

	model	dates	quotes	success_rate	weighted_price_rmse	median_abs_price_error	weighted_iv_rmse	bid_ask_hit_rate	otm_put_rmse	short_maturity_rmse	runtime_sec
0	heston	18	1580	1.0	0.98682	1.19828	0.01124	0.49833	1.46530	1.28191	73.65275
1	rough_heston	18	1580	1.0	1.11457	1.63127	0.01286	0.46028	1.63515	1.41136	211.32990

skew_quality_filter

	window	median_h_all_dates	median_h_high_quality_dates	dates	high_quality_dates	rejected_dates
0	short_2_7_60	0.41075	0.36454	57	34	23

bucket_improvement

	moneyness_bucket	dte_bucket	heston_abs_median_scaled_residual	rough_abs_median_scaled_residual	improvement_percent	rows_heston	rows_rough
14	(-0.05, 0.05]	(30, 60]	0.22337	0.04332	80.60450	542	226
3	(-0.3, -0.15]	(60, 90]	0.56110	0.13412	76.09771	10	6
2	(-0.3, -0.15]	(30, 60]	0.39028	0.17383	55.45957	22	14
6	(-0.15, -0.05]	(0, 14]	0.90776	0.45122	50.29299	294	117
17	(-0.05, 0.05]	(120, 180]	0.15349	0.10011	34.77859	286	118
8	(-0.15, -0.05]	(30, 60]	0.40500	0.37037	8.54949	188	72
23	(0.05, 0.15]	(120, 180]	1.06092	0.97926	7.69678	7	4
22	(0.05, 0.15]	(90, 120]	0.93016	0.86012	7.53046	6	4
19	(0.05, 0.15]	(14, 30]	0.67643	0.65491	3.18165	4	4
18	(0.05, 0.15]	(0, 14]	0.44838	0.48765	-8.75838	6	6
15	(-0.05, 0.05]	(60, 90]	0.17451	0.19189	-9.95981	278	114
0	(-0.3, -0.15]	(0, 14]	3.55781	3.95468	-11.15497	33	21

delta_differences

	tau_days	tau	k	strike	bsm_delta	flat_bsm_delta	surface_bsm_delta	heston_delta	rough_heston_delta	rough_minus_bsm	rough_minus_flat_bsm	rough_minus_surface_bsm	rough_minus_heston
27	7.0	0.01916	-0.0125	3876.54902	0.63408	0.63408	0.61259	0.67719	0.72741	0.09333	0.09333	0.11482	0.05022
28	7.0	0.01916	0.0000	3925.31000	0.50391	0.50391	0.50583	0.56444	0.59206	0.08815	0.08815	0.08623	0.02762
29	7.0	0.01916	0.0125	3974.68432	0.37330	0.37330	0.39048	0.43061	0.41308	0.03978	0.03978	0.02259	-0.01754
76	14.0	0.03833	-0.0125	3876.54902	0.59825	0.59825	0.59329	0.67115	0.69996	0.10172	0.10172	0.10668	0.02882
77	14.0	0.03833	0.0000	3925.31000	0.50540	0.50540	0.50591	0.58274	0.59333	0.08793	0.08793	0.08742	0.01059
78	14.0	0.03833	0.0125	3974.68432	0.41224	0.41224	0.41185	0.47694	0.46169	0.04945	0.04945	0.04984	-0.01525
125	30.0	0.08214	-0.0125	3876.54902	0.57128	0.57128	0.56945	0.66269	0.66900	0.09772	0.09772	0.09955	0.00631
126	30.0	0.08214	0.0000	3925.31000	0.50761	0.50761	0.50800	0.59396	0.59146	0.08385	0.08385	0.08346	-0.00250
127	30.0	0.08214	0.0125	3974.68432	0.44373	0.44373	0.44241	0.51417	0.50204	0.05831	0.05831	0.05963	-0.01213
174	60.0	0.16427	-0.0125	3876.54902	0.55524	0.55524	0.55493	0.64633	0.64224	0.08700	0.08700	0.08731	-0.00409
175	60.0	0.16427	0.0000	3925.31000	0.51021	0.51021	0.51017	0.59449	0.58722	0.07701	0.07701	0.07705	-0.00727
176	60.0	0.16427	0.0125	3974.68432	0.46504	0.46504	0.46288	0.53691	0.52681	0.06177	0.06177	0.06392	-0.01010
223	90.0	0.24641	-0.0125	3876.54902	0.54870	0.54870	0.54871	0.63444	0.62798	0.07928	0.07928	0.07927	-0.00647
224	90.0	0.24641	0.0000	3925.31000	0.51198	0.51198	0.51133	0.59190	0.58381	0.07183	0.07183	0.07248	-0.00808
225	90.0	0.24641	0.0125	3974.68432	0.47515	0.47515	0.47199	0.54578	0.53639	0.06125	0.06125	0.06440	-0.00939

Reading the SPX Evidence as One Workflow

We can read the dashboard alongside the separate results to connect the evidence from each part of the SPX workflow. It links four separate kinds of evidence:

1. historical roughness from realized-variance scaling,
2. forecast usefulness from rough-kernel variance forecasts,
3. option-implied roughness from ATM skew power laws,
4. pricing and hedging consequences from rough models.

The strongest result isn’t one single table. It’s the alignment between several diagnostics. The historical realized-variance side says volatility is locally rough after controlling for proxy issues. The option side says short-end skew follows a power law. The rough models can generate steep short-end skew and materially different deltas. The calibration comparison also records the fitting and runtime costs alongside the regions where rough Heston improves the ordinary Heston fit.

That’s a good research outcome. It teaches the model’s purpose and its limitations at the same time.

Connecting the SPX Evidence

For the SPX market we can put three different roughness measurements next to each other and compare their interpretations:

\[ H_{\text{RV}}, \qquad H_{\text{skew}}, \qquad H_{\text{model}} \]

These are related, but they are estimated from different objects.

\(H_{\text{RV}}\) comes from historical realized variance. It’s a physical-measure estimate because it uses actual realized returns. The raw daily squared-return proxy gives almost no reliable scaling structure because it’s too noisy. The rolling proxies are smoother and produce larger \(H\) values, but overlap and smoothing can push estimates upward. The block/non-overlapping checks help reduce that bias.

\(H_{\text{skew}}\) comes from option prices. It’s risk-neutral because it uses the implied volatility surface. The short-end skew power law is one of the most important rough-volatility diagnostics:

\[ |\psi(\tau)| \approx c\tau^{H-\frac{1}{2}} \]

If the estimated slope is close to \(-\frac{1}{2}\), then \(H\) is close to zero and the skew explodes very sharply as maturity gets short. If the slope is less steep, \(H\) is larger. The SPX high-quality skew estimate around \(H\approx0.36\) to \(0.41\) isn’t as rough as the original \(H\approx0.1\) empirical result from high-frequency realized variance literature, but it still points to rougher-than-Brownian implied volatility because it’s below \(0.5\).

\(H_{\text{model}}\) comes from rough-Heston calibration. It’s constrained by the model’s ability to fit prices, not only by a slope regression. That makes it a hybrid diagnostic. It tells us what roughness level the model can support while fitting the surface under the chosen parameter bounds and numerical engine.

Looking across these measurements connects the different sources of evidence and the choices that enter each estimate. If we only look at realized variance, the answer depends strongly on proxy construction. If we only look at option skew, the answer depends on surface quality and maturity filtering. If we only look at model calibration, the answer depends on optimizer and numerical choices. Putting all three together gives a more stable interpretation:

• realized variance shows evidence of non-smooth volatility dynamics,
• option skew shows a short-end power-law structure,
• rough Heston can reproduce parts of that skew structure,
• the pricing comparison still has to be checked bucket by bucket,
• deltas change because the model changes the spot-volatility interaction.

This is also why the project belongs after Projects 4, 8, 11, 13, and 14. We already know how to extract IV, build surfaces, price under stochastic volatility, and use Fourier/COS methods. Here we use all of that machinery to ask a sharper question: whether volatility itself behaves like a rough process and whether that roughness matters for pricing and hedging.

The most useful practical reading is:

1. SPX volatility is rough in several diagnostics, especially when we avoid trusting only a smoothed overlapping RV proxy.
2. Option skew carries a clear roughness signature, because ATM skew follows a power-law shape across short maturities on many dates.
3. rBergomi is conceptually clean for rough smile generation, but Monte Carlo calibration is noisy and expensive.
4. rough Heston is more production-friendly than rBergomi, because it can use characteristic-function pricing, but fractional Riccati computation is still heavier than classical Heston.
5. rough models change hedge sensitivities, especially near ATM and downside short-maturity options.

This gives the project its place in the series. Project 11 compared stochastic-volatility and jump models. Project 14 built the Fourier machinery. Here we use that machinery to test a deeper question: whether the time regularity of volatility itself should be part of the model.

15) BTC Library Implementation

We now repeat the workflow on BTC using the packaged library. BTC has a very different volatility ecosystem from SPX, giving us another market in which to compare the same methods:

• spot volatility is much higher,
• option quotes are noisier,
• moneyness and maturity support can be uneven,
• crypto options are strongly shaped by leverage, funding, liquidations, and weekend trading,
• BTC trades continuously, so the annualization basis uses 365 days rather than 252 trading days.

The data comes from:

• btc_yfinance for BTC spot,
• btc_optionsdx for BTC/Deribit options,
• us_treasury_yields for discounting.

We also check the BTC option price units because Deribit-style option prices can be quoted in BTC while the underlying spot is in USD. If the option mid is in BTC and BTC spot is around 64,000, a raw mid of \(0.05\) BTC corresponds to several thousand dollars:

\[ C_{\text{USD}} = C_{\text{BTC}} \times S_{\text{BTCUSD}} \]

In the unit check, the median raw mid is about 0.0546 BTC, and the USD-equivalent median mid is about 3,369 USD. Keeping those units aligned is necessary before we compare the option prices with the USD spot and extract implied volatility. Without it, the calibration would compare option prices and spot levels in inconsistent units.

15.1 Continuous-Time Market Structure

BTC has a different time structure from SPX, not only a higher volatility level. SPX cash trading is tied to exchange hours, while BTC trades continuously. That changes the annualization convention and the meaning of weekend risk.

We use:

\[ \text{annualization}_{BTC}=365 \]

instead of:

\[ \text{annualization}_{SPX}=252 \]

This affects realized variance, forecast scaling, and option maturity conversion. A 7-day BTC option really contains 7 calendar days of trading. A 7-calendar-day SPX option contains fewer cash-market trading sessions, though options still price overnight and weekend risk indirectly.

This is one reason BTC roughness and BTC skew need separate interpretation. The time clock is different, the market microstructure is different, and the volatility risk premium is shaped by different participants.

from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import display
from cycler import cycler

from quantfinlab.dataio import load_ohlcv, load_option_chain, load_par_yield_curve
from quantfinlab.volatility.realized import log_returns
from quantfinlab.volatility.forecasting import future_realized_variance, rolling_arch_forecasts_weekly, score_forecasts_by_model
from quantfinlab.volatility.har import rolling_har_forecasts
from quantfinlab.volatility.rough import (daily_variance, log_variance, fbm_cholesky_paths, moment_scaling,
    hurst_from_moments, hurst_from_moments_pooled, rough_kernel_forecasts, rough_forecast_frame)
from quantfinlab.options.quote_cleaning import attach_spot_from_series, clean_option_quotes, convert_quotes_to_usd_equivalent, surface_ready_quotes
from quantfinlab.options.rates_dividends import attach_rates, add_discount_factors, infer_carry_from_forward, infer_dividend_yield_from_forward
from quantfinlab.options.parity import infer_forwards_from_parity
from quantfinlab.options.iv import implied_vol_table
from quantfinlab.options.surface import surface_grid, fit_surface_panel, surface_iv_grid, surface_residuals
from quantfinlab.calibration.fft_cos import surface_target_grid_quotes, fit_daily_models, compare_fourier_models, residual_by_bucket
from quantfinlab.options.rough import (atm_skew_term_structure, skew_power_law, forward_variance_curve,
    simulate_rbergomi, rbergomi_smile, rbergomi_calibration, fit_rough_heston_dates,
    compare_heston_rough_heston, rough_heston_residuals, riccati_convergence, rough_delta_grid)
from quantfinlab.plotting.volatility import (price_and_variance, fbm_roughness, moment_ladder,
    hurst_linearity, forecast_race, forecast_path, smile_slices, smooth_surface, skew_power, fit_comparison,
    maturity_error, rbergomi_paths, rbergomi_fit, riccati_stability, parameter_paths, delta_slices, delta_gap)

palette = ["#069AF3", "#FE420F", "#00008B", "#008080", "#CC79A7", "#9614fa", "#DC143C", "#7BC8F6", "#0072B2", "#04D8B2", "#800080", "#FF8072"]
plt.rcParams["axes.prop_cycle"] = cycler(color=palette)

data_dir = Path("../data")
if not data_dir.exists():
    data_dir = Path("data")

cache_dir = data_dir / "cache" / "project18" / "btc"
cache_dir.mkdir(parents=True, exist_ok=True)

ann_days = 365.0
ann_trading = 365
seed = 7
q_values = (0.5, 1.0, 1.5, 2.0, 3.0)
lags = (1, 2, 4, 8, 16, 32)
horizons = (1, 5, 10, 21, 42, 63)

btc_history_path = data_dir / "btc_usd_ohlcv.csv"
btc_spot = load_ohlcv(btc_history_path, source="yfinance_csv", fields=("close",))
btc_raw = load_option_chain(data_dir / "btc_options_chain.parquet", source="btc_deribit", annualization_days=ann_days)
us_par_btc = load_par_yield_curve(data_dir / "us_treasury_yields.csv", source="us_treasury")

btc_ret = log_returns(btc_spot["close"]).dropna()
btc_var_1d = daily_variance(btc_ret, annualization=ann_trading)
btc_var = (btc_var_1d.rolling(10).sum() / 10.0).dropna().rename("btc_project5_10d_daily_variance")
btc_log_var = log_variance(btc_var)
btc_targets = future_realized_variance(btc_ret, horizons=horizons, annualization=ann_trading)

fbm = fbm_cholesky_paths(h_values=(0.05, 0.10, 0.20, 0.50, 0.80), n_steps=384, n_paths=4, seed=seed)
scaling = moment_scaling(btc_log_var, q_values=q_values, lags=lags)
hurst = hurst_from_moments(scaling, q_col="q", lag_col="lag", moment_col="moment")
hurst_pooled = hurst_from_moments_pooled(scaling)
h = float(np.clip(float(hurst_pooled["H"].iloc[0])
                  if not hurst_pooled.empty and np.isfinite(float(hurst_pooled["H"].iloc[0]))
                  else hurst["h"].median(), 0.04, 0.35))

train_window = min(180, max(90, len(btc_var) // 2))
rough_fc = rough_kernel_forecasts(btc_var, h=h, horizons=horizons, train_window=train_window, signal_step=7, annualization=ann_trading)
har_fc = rolling_har_forecasts(btc_ret, horizons=horizons, train_window=train_window, refit_every=7, annualization=ann_trading, use_log=True)

btc_forecast_data = rough_forecast_frame(rough_fc=rough_fc, har_fc=har_fc, arch_fc=pd.DataFrame(), rv_targets=btc_targets, horizons=horizons)
forecast_scores = score_forecasts_by_model(btc_forecast_data) if not btc_forecast_data.empty else pd.DataFrame()

btc_quotes = attach_spot_from_series(btc_raw, btc_spot["close"], date_col="date", spot_col="spot", method="previous", overwrite=False)
btc_quotes = convert_quotes_to_usd_equivalent(btc_quotes, unit="base")
btc_unit_audit = pd.DataFrame([{
    "input_file": str(btc_history_path.name),
    "price_unit": str(btc_quotes["price_unit_detected"].iloc[0]),
    "spot_median": float(pd.to_numeric(btc_quotes["spot"], errors="coerce").median()),
    "raw_mid_median_btc": float(pd.to_numeric(btc_quotes["mid_raw"], errors="coerce").replace(0.0, np.nan).median()),
    "usd_mid_median": float(pd.to_numeric(btc_quotes["mid"], errors="coerce").replace(0.0, np.nan).median()),
}])
clean_quotes, clean_steps = clean_option_quotes(btc_quotes, min_dte=3, max_dte=180, moneyness_range=(0.45, 1.75),
                                                max_relative_spread=0.60, closest_atm_pairs=90,
                                                min_pairs_per_expiry=0, annualization_days=ann_days)
rate_start = clean_quotes["date"].min() - pd.Timedelta(days=30)
rate_end = clean_quotes["date"].max()
us_par_btc = us_par_btc.loc[(us_par_btc.index >= rate_start) & (us_par_btc.index <= rate_end)].copy()
clean_quotes = attach_rates(clean_quotes, us_par_btc, date_col="date", tau_col="tau", out_col="rate")
clean_quotes = add_discount_factors(clean_quotes, rate_col="rate", tau_col="tau", out_col="discount_factor")
clean_quotes = infer_forwards_from_parity(clean_quotes, date_col="date", expiry_col="expiry",
                                          strike_col="strike", option_type_col="option_type", mid_col="mid",
                                          discount_col="discount_factor", spot_col="spot", out_col="forward")
clean_quotes = infer_carry_from_forward(clean_quotes, spot_col="spot", forward_col="forward", tau_col="tau", out_col="implied_carry")
clean_quotes = infer_dividend_yield_from_forward(clean_quotes, rate_col="rate", carry_col="implied_carry", out_col="implied_dividend_yield")

iv_quotes = implied_vol_table(clean_quotes, price_cols=("bid", "mid", "ask"), option_type_col="option_type",
                              spot_col="spot", forward_col="forward", strike_col="strike", tau_col="tau",
                              rate_col="rate", discount_col="discount_factor", model="black76", engine="auto")
surface_quotes = surface_ready_quotes(iv_quotes, min_iv=0.05, max_iv=3.50, min_tau=3 / ann_days,
                                      max_tau=180 / ann_days, min_weight=0.05, max_weight=20.0)

surface_fit = fit_surface_panel(surface_quotes, date_col="date", k_col="k", tau_col="tau",
                                iv_col="iv_mid", weight_col="surface_weight",
                                visual_params=dict(n_k_basis=10, n_tau_basis=7, degree=3, lambda_k=3.0, lambda_tau=2.0),
                                dupire_params=dict(n_k_basis=9, n_tau_basis=6, degree=3, lambda_k=10.0, lambda_tau=16.0),
                                min_quotes=80, min_expiries=4)
visual_fits_by_date = {pd.Timestamp(k).normalize(): v for k, v in surface_fit["visual_fits"].items()}
btc_tau_values = np.array([7, 10, 14, 21, 30, 45, 60, 90, 120], dtype=float) / ann_days
btc_skew_rows = []
for d, fit in visual_fits_by_date.items():
    qd = surface_quotes[surface_quotes["date"].eq(d)].copy()
    psi_d = atm_skew_term_structure(qd, fit=fit, k_col="k", tau_col="tau", iv_col="iv_mid", tau_values=btc_tau_values, dk=0.01)
    for window, lo, hi in [("short_7_60", 7, 60), ("medium_14_90", 14, 90), ("old_14_150", 14, 150)]:
        sub = psi_d[psi_d["tau_days"].between(lo, hi)].copy()
        row = skew_power_law(sub, tau_col="tau", skew_col="atm_skew").iloc[0].to_dict()
        row["date"] = d
        row["window"] = window
        row["h_from_skew"] = row.get("h", np.nan)
        row["quote_count"] = int(len(qd))
        btc_skew_rows.append(row)
btc_skew_history = pd.DataFrame(btc_skew_rows)
surface_summary = surface_fit["fit_summary"]
surface_visual_summary = surface_summary[surface_summary["fit"].eq("visual")].copy() if not surface_summary.empty else pd.DataFrame()
btc_main_candidates = btc_skew_history[btc_skew_history["window"].eq("short_7_60")].copy()
if not surface_visual_summary.empty:
    btc_main_candidates = btc_main_candidates.merge(surface_visual_summary[["date", "weighted_rmse", "quote_count", "number_of_maturities"]],
                                                    on="date", how="left", suffixes=("", "_surface"))
else:
    btc_main_candidates["weighted_rmse"] = np.nan
    btc_main_candidates["number_of_maturities"] = np.nan
btc_high_quality_skew = btc_main_candidates[btc_main_candidates["r2"].ge(0.80) & btc_main_candidates["n"].ge(6)].copy()
btc_selection_pool = btc_high_quality_skew.copy()
if btc_selection_pool.empty:
    btc_selection_pool = btc_main_candidates[btc_main_candidates["n"].ge(4)].copy()
if btc_selection_pool.empty:
    btc_selection_pool = btc_main_candidates.copy()
btc_selection_pool["surface_quality_rank"] = btc_selection_pool["weighted_rmse"].rank(method="first", ascending=True, na_option="bottom")
main_date = btc_selection_pool.sort_values(["r2", "surface_quality_rank", "quote_count"], ascending=[False, True, False])["date"].iloc[0]
q_main = surface_quotes[surface_quotes["date"].eq(main_date)].copy()
fit_visual = visual_fits_by_date[pd.Timestamp(main_date).normalize()]
k_grid = surface_grid(q_main, k_col="k", tau_col="tau", k_quantiles=(0.02, 0.98),
                      tau_quantiles=(0.02, 0.98), n_k=65, n_tau=35, annualization_days=ann_days)
sigma_grid = surface_iv_grid(fit_visual, k_grid)
surface_err = surface_residuals(q_main, fits={"smooth": fit_visual}, k_col="k", tau_col="tau", iv_col="iv_mid")

psi = atm_skew_term_structure(q_main, fit=fit_visual, k_col="k", tau_col="tau", iv_col="iv_mid", tau_values=btc_tau_values, dk=0.01)
skew_fit = skew_power_law(psi[psi["tau_days"].between(7, 60)], tau_col="tau", skew_col="atm_skew")
xi = forward_variance_curve(q_main, fit=fit_visual, tau_values=np.array([7, 14, 21, 30, 45, 60, 90, 120], dtype=float) / ann_days)

calibration_quotes = surface_target_grid_quotes(surface_quotes, surface_fit["visual_fits"], min_dte=7,
                                                max_dte=150, max_abs_log_moneyness=0.45, min_quotes_per_date=45,
                                                min_source_quotes_per_date=70, annualization_days=ann_days,
                                                iv_floor=0.05, iv_cap=3.50, iv_error_floor=0.020,
                                                iv_error_cap=0.140, adaptive_k_grid=True, adaptive_k_min_nodes=7,
                                                adaptive_k_max_nodes=11)
date_counts = calibration_quotes.groupby("date").size()
eligible_btc_fit_dates = date_counts[date_counts >= 45].index.sort_values()
if len(eligible_btc_fit_dates) > 12:
    fit_dates = eligible_btc_fit_dates[np.linspace(0, len(eligible_btc_fit_dates) - 1, 12).round().astype(int)]
else:
    fit_dates = eligible_btc_fit_dates
h_skew_main = float(skew_fit["h"].iloc[0]) if not skew_fit.empty and np.isfinite(skew_fit["h"].iloc[0]) else h
btc_h_source = btc_high_quality_skew.copy()
if btc_h_source.empty:
    btc_h_source = btc_main_candidates[btc_main_candidates["r2"].ge(0.70)].copy()
btc_h_source = btc_h_source[btc_h_source["h_from_skew"].between(0.04, 0.45)].copy()
btc_h_by_date = btc_h_source.set_index("date")["h_from_skew"].dropna().sort_index()

btc_heston_cache = cache_dir / "btc_heston_surface_grid_daily.parquet"
btc_heston_fit_cache = cache_dir / "btc_heston_surface_grid_fit.parquet"
btc_rough_cache = cache_dir / "btc_rough_heston_surface_grid_skew_anchor_v2_daily.parquet"
btc_rough_fit_cache = cache_dir / "btc_rough_heston_surface_grid_skew_anchor_v2_fit.parquet"

if btc_heston_cache.exists() and btc_heston_fit_cache.exists():
    heston_daily = pd.read_parquet(btc_heston_cache)
    heston_prices = pd.read_parquet(btc_heston_fit_cache)
else:
    heston_out = fit_daily_models(calibration_quotes, ["heston"], calibration_dates=fit_dates, min_quotes=45,
                                  max_nfev=18, engine="cpp", n_terms=112, truncation_width=16.0)
    heston_daily = heston_out["params"]
    heston_prices = heston_out["fit"]
    heston_daily.to_parquet(btc_heston_cache, index=False)
    heston_prices.to_parquet(btc_heston_fit_cache, index=False)

if btc_rough_cache.exists() and btc_rough_fit_cache.exists():
    rough_daily = pd.read_parquet(btc_rough_cache)
    rough_prices = pd.read_parquet(btc_rough_fit_cache)
else:
    rough_out = fit_rough_heston_dates(calibration_quotes, calibration_dates=fit_dates, h_start=h_skew_main,
                                       h_by_date=btc_h_by_date, h_mode="fixed", min_quotes=45, max_nfev=14,
                                       engine="cpp", n_terms=88, truncation_width=16.0, riccati_steps=192,
                                       lambda_skew=3.0, surface_fit_map=surface_fit["visual_fits"])
    rough_daily = rough_out["params"]
    rough_prices = rough_out["fit"]
    rough_daily.to_parquet(btc_rough_cache, index=False)
    rough_prices.to_parquet(btc_rough_fit_cache, index=False)

model_table = compare_heston_rough_heston(heston_daily=heston_daily, rough_daily=rough_daily)
heston_table = compare_fourier_models(daily=heston_daily)
heston_buckets = residual_by_bucket(heston_prices)
rough_buckets = rough_heston_residuals(rough_prices, scale_col="calib_scale_px")
rough_valid = rough_daily[
    rough_daily["success"].astype(bool)
    & pd.to_numeric(rough_daily.get("weighted_price_rmse"), errors="coerce").replace([np.inf, -np.inf], np.nan).notna()
    & pd.to_numeric(rough_daily.get("finite_price_share", pd.Series(1.0, index=rough_daily.index)), errors="coerce").ge(0.999)
].copy()
if rough_valid.empty:
    rough_valid = rough_daily[pd.to_numeric(rough_daily.get("weighted_price_rmse"), errors="coerce").replace([np.inf, -np.inf], np.nan).notna()].copy()
if rough_valid.empty:
    raise RuntimeError("No finite BTC rough-Heston parameter row is available for diagnostics.")
display_date = pd.Timestamp(main_date).normalize()
if display_date not in set(pd.to_datetime(rough_valid["date"]).dt.normalize()):
    display_date = pd.Timestamp(rough_valid.sort_values("weighted_price_rmse")["date"].iloc[0]).normalize()
rough_fit = rough_prices[rough_prices["date"].eq(display_date)].copy()
heston_fit = heston_prices[heston_prices["date"].eq(display_date)].copy()
q_main_target = calibration_quotes[calibration_quotes["date"].eq(pd.Timestamp(main_date).normalize())].copy()
if q_main_target.empty:
    q_main_target = q_main.copy()
rough_param_row = rough_valid[rough_valid["date"].eq(display_date)].sort_values("weighted_price_rmse").iloc[0]
riccati_test = riccati_convergence(q_main_target, params=rough_param_row, n_grid_values=(128, 256, 512), n_terms=96, engine="cpp")

rbergomi = rbergomi_calibration(q_main_target, xi=xi, h_start=h_skew_main, nu_start=2.0,
                                rho_start=-0.70, paths=5200, steps=88, restarts=6,
                                seed=seed, engine="numba", use_sobol=True, lambda_skew=1.0)
rbergomi_sim = simulate_rbergomi(spot=float(q_main["spot"].median()), xi=xi, h=float(rbergomi["params"]["h"]),
                                 nu=float(rbergomi["params"]["nu"]), rho=float(rbergomi["params"]["rho"]),
                                 tau=float(np.nanmax(q_main["tau"])), paths=1200, steps=80, seed=seed,
                                 engine="numba", rate=float(q_main["rate"].median()),
                                 dividend_yield=float(q_main["implied_dividend_yield"].median()),
                                 antithetic=True)
rbergomi_iv = rbergomi_smile(q_main_target, xi=xi, params=rbergomi["params"], maturity_days=(7, 14, 30, 60, 90),
                             paths=3000, steps=88, seed=seed, engine="numba")

delta_map = rough_delta_grid(q_main, heston_params=heston_daily.dropna().iloc[-1], rough_params=rough_param_row,
                             rbergomi_params=rbergomi["params"], xi=xi, surface_fit=fit_visual,
                             k_values=np.linspace(-0.35, 0.25, 49),
                             tau_days=np.array([7, 14, 30, 60, 90], dtype=float), n_terms=96, riccati_steps=192,
                             bump=0.005, paths=2500, steps=80, seed=seed,
                             engine="cpp", mc_engine="numba")

summary = pd.DataFrame([{"asset": "BTC", "spot_start": float(btc_spot["close"].dropna().iloc[0]),
                         "spot_end": float(btc_spot["close"].dropna().iloc[-1]),
                         "return_days": int(btc_ret.shape[0]), "option_rows_clean": int(clean_quotes.shape[0]),
                         "surface_rows": int(surface_quotes.shape[0]),
                         "surface_target_rows": int(calibration_quotes.shape[0]),
                         "main_date": pd.Timestamp(main_date).date(),
                         "main_date_rule": "best short-skew R2 with surface RMSE tiebreak",
                         "h_from_rv": float(h), "h_from_skew": float(skew_fit["h"].iloc[0]),
                         "h_used_for_rough_heston": float(h_skew_main), "skew_r2": float(skew_fit["r2"].iloc[0]),
                         "skew_high_quality_dates": int(len(btc_high_quality_skew)),
                         "rough_dates": int(rough_daily["date"].nunique()),
                         "heston_dates": int(heston_daily["date"].nunique())}])

for name, table in {"summary": summary, "btc_unit_audit": btc_unit_audit, "hurst": hurst,
                    "forecast_scores": (forecast_scores.sort_values(["horizon", "qlike_var"]).head(20)
                                        if not forecast_scores.empty else forecast_scores),
                    "btc_skew_history": btc_skew_history.sort_values(["window", "r2"], ascending=[True, False]).head(12),
                    "skew_fit": skew_fit, "model_table": model_table, "heston_table": heston_table,
                    "rbergomi_params": pd.DataFrame([rbergomi["params"]]), "rbergomi_restart_loss": rbergomi["fit"],
                    "riccati_test": riccati_test,
                    "delta_sample": delta_map[np.isclose(delta_map["k"], 0.0, atol=0.015)].copy()}.items():
    print(name)
    display(table)

fig, axes = plt.subplots(4, 4, figsize=(30, 22), constrained_layout=True)
axes = axes.ravel()
price_and_variance(axes[0], btc_spot, btc_var, title="BTC Price and Project-5 Realized Variance")
fbm_roughness(axes[1], fbm, title="Fractional Brownian Roughness")
moment_ladder(axes[2], scaling, title="BTC Moment Scaling")
hurst_linearity(axes[3], hurst, title="BTC qH Linearity")
forecast_race(axes[4], forecast_scores, metric="qlike_var", title="BTC Forecast Race by QLIKE")
forecast_path(axes[5], btc_forecast_data, model="rough_kernel", horizon=21, title="BTC 21-Day Rough Forecast")
smile_slices(axes[6], q_main, fit=fit_visual, maturities_days=(7, 14, 30, 60, 90), title="BTC IV Smiles")
smooth_surface(axes[7], k_grid, sigma_grid, q_main, title="BTC Smooth IV Surface")
skew_power(axes[8], psi, skew_fit, title="BTC ATM Skew Power Law")
fit_comparison(axes[9], heston_fit, rough_fit, title="BTC Heston vs Rough Heston")
maturity_error(axes[10], heston_buckets, rough_buckets, title="BTC Error by Maturity and Moneyness")
rbergomi_paths(axes[11], rbergomi_sim, title="BTC rBergomi Paths")
rbergomi_fit(axes[12], rbergomi_iv, q_main, title="BTC rBergomi Smile Fit")
riccati_stability(axes[13], riccati_test, title="BTC Fractional Riccati Stability")
delta_slices(axes[14], delta_map, title="BTC Delta Slices")
delta_gap(axes[15], delta_map, base_col="surface_bsm_delta", title="BTC Rough-Model Delta Gap")
fig.suptitle(f"Project 18 BTC Library Implementation | {pd.Timestamp(main_date).date()}", fontsize=22, y=1.01)
plt.show()

e:\daneshgah\quantitative-finance-lab\.venv\Lib\site-packages\numba\core\typing\typeof.py:268: FutureWarning: future versions will not create a writeable array from broadcast_array. Set the writable flag explicitly to avoid this warning.
  readonly = not val.flags.writeable

summary

	asset	spot_start	spot_end	return_days	option_rows_clean	surface_rows	surface_target_rows	main_date	main_date_rule	h_from_rv	h_from_skew	h_used_for_rough_heston	skew_r2	skew_high_quality_dates	rough_dates	heston_dates
0	BTC	457.334015	73845.9375	4274	121361	120736	27106	2024-01-13	best short-skew R2 with surface RMSE tiebreak	0.35	0.121271	0.121271	0.999026	129	12	12

btc_unit_audit

	input_file	price_unit	spot_median	raw_mid_median_btc	usd_mid_median
0	btc_usd_ohlcv.csv	base	63931.0229	0.054625	3369.138183

hurst

	q	slope	h	standard_error	r2	n
0	0.5	0.301238	0.602477	0.038807	0.937748	6
1	1.0	0.516490	0.516490	0.059497	0.949596	6
2	1.5	0.679718	0.453145	0.070786	0.958423	6
3	2.0	0.809969	0.404984	0.076873	0.965222	6
4	3.0	1.013346	0.337782	0.081284	0.974909	6

forecast_scores

	model	horizon	qlike_var	rmse_var	mae_var	rmse_vol	mae_vol	corr_vol	n_obs
0	rough_kernel	1	-5.722008	0.002742	0.001501	0.531766	0.436103	0.169303	575
1	har_rv	1	3.890556	0.003282	0.001253	0.538658	0.329553	0.288779	576
2	rough_kernel	5	-4.274958	0.007103	0.004858	0.347325	0.283583	0.332557	575
3	har_rv	5	-3.548666	0.012593	0.004254	0.354016	0.228676	0.419341	576
4	rough_kernel	10	-3.506610	0.016271	0.009284	0.333014	0.259050	0.325720	575
5	har_rv	10	-2.588079	0.020115	0.008339	0.332395	0.215593	0.412994	576
6	rough_kernel	21	-2.645925	0.029726	0.018565	0.302089	0.235235	0.329372	575
7	har_rv	21	-2.329208	0.029960	0.015944	0.291803	0.207866	0.367074	576
8	rough_kernel	42	-1.953071	0.049223	0.033913	0.270705	0.214830	0.337049	575
9	har_rv	42	-1.725558	0.049959	0.032366	0.275633	0.210824	0.280480	576
10	rough_kernel	63	-1.549488	0.067374	0.047839	0.253515	0.201392	0.341580	575
11	har_rv	63	-1.457243	0.068879	0.048201	0.260865	0.205071	0.281591	576

btc_skew_history

	c	alpha	h	r2	n	date	window	h_from_skew	quote_count
37	0.044046	0.389790	0.110210	0.999342	5.0	2024-01-13	medium_14_90	0.110210	294
619	0.133062	-0.341788	0.841788	0.999070	5.0	2024-07-30	medium_14_90	0.841788	545
700	0.003856	0.766390	-0.266390	0.998537	5.0	2024-08-26	medium_14_90	-0.266390	516
49	0.015688	0.673165	-0.173165	0.998015	5.0	2024-01-17	medium_14_90	-0.173165	334
40	0.031679	0.624160	-0.124160	0.996627	5.0	2024-01-14	medium_14_90	-0.124160	318
91	0.013880	0.346300	0.153700	0.995371	5.0	2024-01-31	medium_14_90	0.153700	377
10	0.033028	0.233442	0.266558	0.991604	5.0	2024-01-04	medium_14_90	0.266558	427
466	0.176105	-0.515649	1.015649	0.989862	5.0	2024-06-05	medium_14_90	1.015649	411
55	0.015744	0.823945	-0.323945	0.988779	5.0	2024-01-19	medium_14_90	-0.323945	410
610	0.141683	-0.153852	0.653852	0.988115	5.0	2024-07-27	medium_14_90	0.653852	486
28	0.010783	0.989343	-0.489343	0.987918	5.0	2024-01-10	medium_14_90	-0.489343	358
277	0.278458	-1.485721	1.985721	0.987732	5.0	2024-04-02	medium_14_90	1.985721	569

skew_fit

	c	alpha	h	r2	n
0	0.045236	0.378729	0.121271	0.999026	6

model_table

	model	dates	quotes	success_rate	weighted_price_rmse	weighted_iv_rmse	otm_put_rmse	short_maturity_rmse	median_abs_price_error	bid_ask_hit_rate	runtime_sec
0	rough_heston	12	1229	1.0	1.744876	0.049765	1.652673	1.563994	61.248607	0.252904	332.976756
1	heston	12	1229	1.0	1.782892	0.054447	1.832712	1.825694	62.547957	0.223832	28.677019

heston_table

	model	dates	quotes	success_rate	weighted_price_rmse	median_abs_price_error	weighted_iv_rmse	bid_ask_hit_rate	otm_put_rmse	short_maturity_rmse	runtime_sec
0	heston	12	1229	1.0	1.782892	62.547957	0.054447	0.223832	1.832712	1.825694	28.677019

rbergomi_params

	h	nu	rho	boundary_hit	target_rows	iv_rmse
0	0.12127	0.8	-0.3	False	28	0.221728

rbergomi_restart_loss

	restart	h	nu	rho	iv_mse	iv_rmse
0	29	0.12127	0.80000	-0.30000	0.049163	0.221728
1	-1	0.12127	0.80000	-0.30000	0.051118	0.226093
2	4	0.12127	1.10000	-0.30000	0.071139	0.266719
3	-1	0.12127	1.10000	-0.30000	0.073416	0.270954
4	28	0.12127	0.80000	-0.50000	0.090796	0.301323
5	34	0.12127	1.40000	-0.30000	0.092699	0.304465
6	-1	0.12127	0.80000	-0.50000	0.092907	0.304806
7	27	0.12127	0.80000	-0.55000	0.101254	0.318205
8	9	0.12127	1.60000	-0.30000	0.106195	0.325876
9	50	0.30000	2.00000	-0.70000	0.113188	0.336435
10	3	0.12127	1.10000	-0.50000	0.125917	0.354848
11	14	0.12127	2.00000	-0.30000	0.130308	0.360982
12	26	0.12127	0.80000	-0.70000	0.130889	0.361786
13	2	0.12127	1.10000	-0.55000	0.138337	0.371937
14	39	0.12127	2.20000	-0.30000	0.140722	0.375129
15	33	0.12127	1.40000	-0.50000	0.155211	0.393969
16	19	0.12127	2.70000	-0.30000	0.162025	0.402524
17	25	0.12127	0.80000	-0.90000	0.164158	0.405165
18	32	0.12127	1.40000	-0.55000	0.167698	0.409509
19	1	0.12127	1.10000	-0.70000	0.170439	0.412843
20	8	0.12127	1.60000	-0.50000	0.171066	0.413601
21	7	0.12127	1.60000	-0.55000	0.182985	0.427767
22	44	0.12127	3.50000	-0.30000	0.184677	0.429740
23	46	0.20127	2.00000	-0.70000	0.186716	0.432106
24	24	0.12127	3.60000	-0.30000	0.186776	0.432176
25	49	0.20000	2.00000	-0.70000	0.187536	0.433055
26	55	0.12654	1.37845	-0.70439	0.194243	0.440730
27	13	0.12127	2.00000	-0.50000	0.195214	0.441831
28	31	0.12127	1.40000	-0.70000	0.198204	0.445201
29	38	0.12127	2.20000	-0.50000	0.204479	0.452194
30	12	0.12127	2.00000	-0.55000	0.205810	0.453663
31	0	0.12127	1.10000	-0.90000	0.213763	0.462345
32	37	0.12127	2.20000	-0.55000	0.215333	0.464040
33	6	0.12127	1.60000	-0.70000	0.223047	0.472279
34	18	0.12127	2.70000	-0.50000	0.234623	0.484379
35	30	0.12127	1.40000	-0.90000	0.252169	0.502164
36	11	0.12127	2.00000	-0.70000	0.257150	0.507100
37	48	0.12000	2.00000	-0.70000	0.257614	0.507557
38	17	0.12127	2.70000	-0.55000	0.268716	0.518378
39	5	0.12127	1.60000	-0.90000	0.270742	0.520329
40	36	0.12127	2.20000	-0.70000	0.271779	0.521325
41	51	0.12133	2.25386	-0.74112	0.284303	0.533201
42	47	0.06000	2.00000	-0.70000	0.286042	0.534829
43	52	0.07674	1.66742	-0.84875	0.286416	0.535178
44	45	0.04127	2.00000	-0.70000	0.286594	0.535345
45	43	0.12127	3.50000	-0.50000	0.290400	0.538888
46	23	0.12127	3.60000	-0.50000	0.291109	0.539545
47	42	0.12127	3.50000	-0.55000	0.293666	0.541910
48	22	0.12127	3.60000	-0.55000	0.294161	0.542366
49	54	0.09025	2.43290	-0.64647	0.294759	0.542917
50	10	0.12127	2.00000	-0.90000	0.295170	0.543295
51	16	0.12127	2.70000	-0.70000	0.295182	0.543307
52	35	0.12127	2.20000	-0.90000	0.296479	0.544499
53	41	0.12127	3.50000	-0.70000	0.297475	0.545413
54	21	0.12127	3.60000	-0.70000	0.297575	0.545505
55	53	0.12428	3.41861	-0.77383	0.297783	0.545695
56	15	0.12127	2.70000	-0.90000	0.297821	0.545730
57	40	0.12127	3.50000	-0.90000	0.298333	0.546199
58	20	0.12127	3.60000	-0.90000	0.298357	0.546221

riccati_test

	riccati_steps	terminal_abs	price	abs_change
0	128	0.048171	0.548966	NaN
1	256	0.048171	0.546154	0.002812
2	512	0.048171	0.544404	0.001750

delta_sample

	tau_days	tau	k	strike	bsm_delta	flat_bsm_delta	surface_bsm_delta	heston_delta	rough_heston_delta	rough_minus_bsm	rough_minus_flat_bsm	rough_minus_surface_bsm	rough_minus_heston
27	7.0	0.019165	-0.0125	42806.125228	0.575634	0.575634	0.576919	0.587004	0.617259	0.041626	0.041626	0.040340	0.030256
28	7.0	0.019165	0.0000	43344.560000	0.516110	0.516110	0.515554	0.505719	0.522119	0.006009	0.006009	0.006565	0.016400
29	7.0	0.019165	0.0125	43889.767448	0.456212	0.456212	0.453293	0.425018	0.424689	-0.031522	-0.031522	-0.028604	-0.000328
76	14.0	0.038330	-0.0125	42806.125228	0.564536	0.564536	0.567134	0.569402	0.590869	0.026333	0.026333	0.023735	0.021467
77	14.0	0.038330	0.0000	43344.560000	0.522448	0.522448	0.519844	0.509240	0.527881	0.005434	0.005434	0.008038	0.018642
78	14.0	0.038330	0.0125	43889.767448	0.480094	0.480094	0.471928	0.449453	0.463747	-0.016348	-0.016348	-0.008182	0.014293
125	30.0	0.082136	-0.0125	42806.125228	0.560708	0.560708	0.560518	0.557958	0.576067	0.015359	0.015359	0.015549	0.018109
126	30.0	0.082136	0.0000	43344.560000	0.532053	0.532053	0.528721	0.516963	0.537495	0.005443	0.005443	0.008774	0.020532
127	30.0	0.082136	0.0125	43889.767448	0.503217	0.503217	0.496568	0.476092	0.498406	-0.004811	-0.004811	0.001838	0.022314
174	60.0	0.164271	-0.0125	42806.125228	0.563867	0.563867	0.561582	0.556231	0.574724	0.010858	0.010858	0.013142	0.018494
175	60.0	0.164271	0.0000	43344.560000	0.543761	0.543761	0.539336	0.529349	0.550598	0.006837	0.006837	0.011261	0.021249
176	60.0	0.164271	0.0125	43889.767448	0.523528	0.523528	0.516874	0.502459	0.526219	0.002690	0.002690	0.009344	0.023760
223	90.0	0.246407	-0.0125	42806.125228	0.568338	0.568338	0.564815	0.560005	0.578923	0.010585	0.010585	0.014108	0.018918
224	90.0	0.246407	0.0000	43344.560000	0.552053	0.552053	0.546791	0.539555	0.560694	0.008640	0.008640	0.013902	0.021139
225	90.0	0.246407	0.0125	43889.767448	0.535666	0.535666	0.528600	0.519066	0.542296	0.006630	0.006630	0.013696	0.023230

15.2 BTC Roughness and Forecasting

The BTC realized-variance roughness estimates are much higher than the SPX roughness diagnostics. Across the moment orders, the estimated \(H\) values decline from about 0.60 at \(q=0.5\) to about 0.34 at \(q=3.0\), with a pooled/clipped value used around 0.35 for the realized-variance roughness side. This is already a warning that BTC variance has stronger extreme moves and stronger scale dependence than SPX.

The rough-kernel forecast performs very well against HAR in QLIKE across all shown BTC horizons. At 1 day, rough-kernel QLIKE is around -5.72 versus HAR around 3.89. At 5, 10, 21, 42, and 63 days, rough-kernel also stays better by QLIKE, though HAR is competitive in MAE and correlation. This says the rough kernel is useful for variance scaling in BTC, especially under QLIKE, but HAR can still track average volatility levels better under some simpler losses.

The forecast errors across horizons help us see how those results can both hold in the BTC sample. BTC realized volatility has violent spikes. A model can handle the variance scale well under QLIKE while still having imperfect timing and low volatility correlation. Crypto volatility is less stable than SPX volatility, so a strong variance-loss result doesn’t mean the forecast path is visually perfect.

15.3 BTC Option Skew and Rough Models

The BTC surface has a much larger option panel after cleaning: about 121k clean option rows and 120k surface-ready rows. The selected main date is 2024-01-13, chosen by a high-quality short-skew rule. The short-skew fit gives:

\[ H_{\text{skew}} \approx 0.121 \]

with \(R^2\) close to 0.999, which is a very clean power-law fit for the selected date.

The realized-variance side uses a clipped \(H\) around 0.35, while the option-skew estimate is around 0.12. The two estimates describe different data and probability measures, so we can compare their implications without forcing them to agree. For BTC, the option market’s short-end smile can be extremely steep even when historical realized variance scaling is less rough. That difference likely comes from jump risk, liquidity, crash-up/crash-down asymmetry, and the way crypto options price weekend and liquidation risk.

The BTC Heston versus rough-Heston comparison is more favorable to rough Heston than the SPX aggregate table. Rough Heston has lower weighted price RMSE (1.745 vs 1.783), lower weighted IV RMSE (0.0498 vs 0.0544), lower OTM put RMSE, lower short-maturity RMSE, and a slightly better bid-ask hit rate. Runtime is much worse: about 333 seconds versus 29 seconds. So the improvement is real but computationally expensive.

The rBergomi calibration for BTC picks \(H \approx 0.121\), \(\nu \approx 0.8\), and \(\rho \approx -0.3\), with a high IV RMSE around 0.222. That’s not surprising. BTC smiles are noisy, jumpy, and difficult to fit with a small three-parameter rBergomi layer on top of a forward variance curve. The model is still useful as a rough-smile generator, but the calibration quality isn’t as clean as an ideal textbook case.

Across the BTC skew history, the estimated power-law exponents are also highly unstable on some dates. Some dates produce impossible or economically weird \(H\) estimates, including negative values or values above 1, even with high \(R^2\) inside a narrow window. That can happen because a power law can fit a small number of maturity points very well while the fitted exponent isn’t economically plausible.

Before using the estimates in rough Heston, we filter dates and clip the parameter to the range:

\[ H \in [0.04, 0.45] \]

for rough-Heston use. With that range, noisy option-surface dates can’t force the model roughness parameter outside the region used for our calibration. For SPX, high-quality skew dates are more stable. For BTC, quote quality and surface shape vary more, so robust selection is more important.

We select the main BTC date using a quality rule based on high short-skew \(R^2\) and surface RMSE tie-breaks. The selected date is a teaching date where the rough-volatility signature is clean enough to inspect.

15.4 BTC Delta and Secondary Results

The displayed BTC rough-model delta gaps around ATM are smaller than the corresponding SPX gaps, though the pattern is still visible. At 30 days near ATM, flat BSM delta is around 0.532, surface-BSM delta around 0.529, Heston delta around 0.517, and rough-Heston delta around 0.537. On the slightly downside node, rough-Heston delta is higher than both surface-BSM and Heston. On the slightly upside node, the gap is smaller and sometimes changes sign.

This behavior makes sense. BTC options have very high implied volatilities, so the delta surface is flatter and less binary near ATM than short-dated low-vol equity options. But rough volatility still changes the hedge map, especially where skew and maturity interact.

The BTC dashboard ties the secondary workflow together: roughness estimation, forecast comparison, surface construction, skew power law, model fit, rBergomi paths, Riccati convergence, and delta gaps. We can apply the rough-volatility workflow to BTC, with the different data coverage and market structure shaping how we interpret the results. In SPX, rough volatility is strongly connected to downside index skew and variance risk premium. In BTC, roughness is mixed with crypto-specific jump risk, liquidity effects, and unit/market-structure issues.

That makes the secondary implementation valuable. The BTC application gives us a second market in which to compare realized roughness, option-implied skew, model pricing, and hedge sensitivities under the same broad workflow. The model can improve some fit metrics and capture rough skew behavior, but the market is noisy enough that diagnostics matter more than any single parameter estimate.

BTC Interpretation Across the Whole Pipeline

BTC is a stronger stress test for the workflow than SPX because the same mathematical model meets a very different market structure.

SPX options are tied to an equity index with deep liquidity, clear macro hedging demand, dealer inventory effects, and a long history of downside crash protection. BTC options are tied to a 24/7 asset with stronger jumps, exchange-specific liquidity, weekend risk, funding and leverage cycles, and a much shorter institutional history. The rough-volatility machinery can still be used, but the interpretation needs more care.

The BTC realized-variance scaling estimates show relatively high \(H\) values for lower moments and lower \(H\) values for higher moments. That pattern means the scaling isn’t perfectly monofractal. Higher moments are more sensitive to extreme moves, and BTC has many more large jumps than SPX. When the \(q=3\) estimate is much lower than the \(q=0.5\) estimate, the process is telling us that large shocks scale differently from ordinary fluctuations.

The forecast comparison also has a crypto-specific interpretation. The rough kernel wins QLIKE across the visible horizons, which means it forecasts the variance level in a way that’s especially good under a likelihood-style loss. HAR is still competitive in MAE and correlation because it captures slower realized-volatility persistence. So BTC volatility has both ingredients:

• short-memory reaction to shocks, which helps rough kernels,
• slower regime persistence, which helps HAR-style averages.

The option side is more extreme. The selected BTC skew date gives \(H_{\text{skew}}\approx0.121\) with a very high fit quality. That tells us the short-maturity implied skew decays with maturity very quickly. In simple words, the BTC option market prices very sharp near-term asymmetry. This may come from liquidation cascades, weekend gap risk, event risk around ETFs/regulation, and the fact that crypto traders use options as leveraged crash and breakout exposure.

Across the displayed global fit measures, rough Heston improves more for BTC than it did for SPX. Rough Heston improves weighted price RMSE, weighted IV RMSE, OTM put RMSE, short-maturity RMSE, median absolute price error, and bid-ask hit rate. The cost is runtime: rough Heston is more than ten times slower. This is a real tradeoff. For a research project, the extra runtime is acceptable because we care about understanding the model. For production, this would require caching, parallelization, better parameter warm starts, or a faster compiled engine.

The BTC delta comparison is also different from SPX. In SPX, negative spot-vol correlation creates a strong downside-skew hedge effect. In BTC, skew can be directionally unstable: sometimes downside protection is expensive, sometimes upside calls become extremely expensive during speculative rallies. So rough-Heston delta gaps are present, but they are less one-directional. The model changes the hedge map most where maturity is short and smile curvature is steep.

The final interpretation is that rough volatility transfers well as a modeling language, but it doesn’t transfer as a one-size-fits-all explanation. For SPX, rough volatility is mostly about equity-index downside skew, variance risk premium, and short-end implied-vol behavior. For BTC, roughness is mixed with jumps, liquidity, leverage, and quote-unit issues. The model is still useful because it gives a disciplined way to measure those effects through \(H\), skew power laws, forecast losses, pricing residuals, and hedge deltas.