Turn regime probabilities into portfolio weights through state-conditioned sleeves and risk budgets.
Compare regime-aware portfolios against repeated portfolio baselines from earlier projects.
Repeat the full idea on a sector ETF universe using the packaged library implementation.
This one should be read as two connected courses inside one finance application:
Econometrics and stochastic processes, especially Markov chains, Markov regression, and hidden Markov models.
Machine learning, including unsupervised learning, supervised classification, feature selection, model validation, and probability-driven portfolio decisions.
The financial problem is simple to state but hard to solve: markets don’t behave the same way in every environment. A portfolio that works in a calm growth regime can be completely wrong during inflation stress, credit stress, or volatility clustering. The hard part is that the regime isn’t directly visible. We don’t observe a label on the market saying “today is defensive.” We observe returns, rates, spreads, drawdowns, correlations, macro blocks, and sector behavior. The model has to infer the state from those signals.
Market behavior across equity, credit, bonds, commodities, and sectors
Returns and rolling features
Predictors \(X_t\)
Market state variables available at date \(t\)
Diagnostics
Feature quality control
Avoid unstable features, redundancy, and leakage
Clusters, HMM, classifiers
Regime inference
Estimate whether the environment is risk-on, neutral, or defensive
Month-end probabilities
Decision signal
Translate daily state estimates into rebalancing inputs
Regime-aware weights
Portfolio policy
Change risk exposure based on state probabilities
Costed backtest
Out-of-sample evaluation
Check whether the regime layer adds value after trading costs
The important design choice is that we sample information daily but trade monthly. Daily observations give the model enough data to learn regime structure. Monthly rebalancing keeps the portfolio from becoming a noisy high-turnover trading system. A regime probability doesn’t need to trigger a trade every day. It needs to make the month-end allocation more aware of the current environment.
The pipeline diagram shows this exactly. The dashed line under the features means the model only uses information that would’ve been available at the rebalance date. That matters because regime models are very easy to overstate if future outcomes accidentally leak into the features. The realized future regime labels are used only for training and evaluation, not as live inputs at the rebalance date.
The diagram also separates learning from trading. A model can learn on daily observations, but the portfolio can still rebalance monthly. This is a good compromise in finance because regimes usually persist longer than one day, while daily noise is high.
For a learner, the useful way to read the rest of the project is:
Features describe today’s market.
Labels describe what happened over the next 63 or 126 trading days.
Unsupervised models try to discover regimes from the features without seeing labels.
Supervised models learn a direct mapping from features to realized future regime labels.
Markov models add time dependence, meaning today’s state is related to yesterday’s state.
Portfolio rules convert probabilities into allocations instead of converting hard labels into all-or-nothing trades.
That last point is important. A regime model rarely knows the state with certainty. The output is usually something like \(P(\text{risk-on})=0.60\), \(P(\text{neutral})=0.25\), and \(P(\text{defensive})=0.15\). That probability vector carries much more information than a single hard label. It lets us build a portfolio that leans risk-on without pretending the defensive risk has disappeared.
The pipeline diagram is also useful because it separates three ideas that often get mixed together:
Feature learning: turning market history into a matrix \(X_t\) that describes the current state.
Regime inference: estimating probabilities for states such as risk-on, neutral, or defensive.
Portfolio action: turning those probabilities into weights, costs, and realized performance.
These three layers have different failure modes. A weak feature set can make every model bad. A good feature set can still be handled badly if the model overfits or produces unstable probabilities. A good probability model can still fail if the portfolio mapping is too aggressive, too slow, or too costly.
So the project isn’t only about asking which classifier has the highest accuracy. The real workflow is:
where \(X_t\) is the feature vector observed at time \(t\), \(\hat{p}_t(S_t=k)\) is the estimated probability of regime \(k\), \(w_t\) is the portfolio weight vector chosen at the rebalance date, and \(R_{p,t+1}\) is the next realized portfolio return.
The important design choice is that we keep the learning horizon and trading horizon separate. The model can read daily data and learn daily state changes, but the strategy still trades monthly. This avoids the common mistake of building a very active model just because daily predictions are available. In real portfolios, daily switching across ETFs creates cost, slippage, and model noise. Monthly rebalancing forces the regime signal to be economically meaningful enough to survive a practical trading schedule.
A useful way to think about the whole project is this:
We don’t try to predict tomorrow’s return directly. We try to estimate the market environment that makes some portfolio sleeves more attractive than others.
That difference matters. Predicting one-day returns is extremely noisy. Predicting whether the market is in a broad risk-on, neutral, or defensive condition over the next few months is still difficult, but it matches how allocators usually think. A portfolio manager doesn’t usually say “SPY will return 0.12% tomorrow.” They are more likely to say “credit is strong, breadth is healthy, volatility is falling, and duration isn’t being rewarded, so the portfolio can carry more risky exposure.” This project turns that kind of reasoning into a machine-learning and econometric system.
2) Data, Reused Inputs, and Portfolio Baselines
The raw data is mostly reused from earlier projects, so we keep the data explanation focused on how the data is used in this regime-learning setup.
We use a cross-asset ETF universe for the main regime model:
Risk assets: SPY, QQQ, IWM, EFA, EEM, VNQ, HYG.
Real assets and inflation-sensitive assets: DBC and GLD.
Rates and credit assets: IEF, TLT, LQD, HYG, SHY.
Context tickers: XLE and XLU are included because energy and utilities help identify inflation and defensive rotations.
The cash-like ticker is SHY, which behaves as short-duration Treasury exposure. It isn’t pure cash, but it gives a conservative low-duration place for the portfolio to move when the regime model becomes defensive.
The baseline portfolios reuse machinery from earlier projects:
Equal weight and mean-variance logic come from Project 2.
Mean-CVaR and Wasserstein robust optimization come from Project 10.
Financial conditions and macro blocks are based on Project 12.
We don’t need to re-explain Ledoit-Wolf, Bayes-Stein, CVaR, or Wasserstein DRO here. The important part is how regime probabilities change the portfolio relative to those baselines.
The coverage table confirms that the ETF panel is long enough to support a regime-learning workflow. SPY starts in 1999, QQQ in 1999, IWM in 2000, EFA in 2001, and the fixed-income ETFs in 2002. Some assets start later: EEM starts in 2003, VNQ in 2004, DBC in 2006, and HYG in 2007. Because HYG and DBC are important for credit and commodity regimes, the effective fully usable cross-asset regime sample naturally starts later than SPY’s history.
The daily panel runs from 1999-01-04 to 2026-06-17, with 6906 daily rows and 270 monthly rebalance dates after the training-history requirement. This gives enough observations for daily ML fitting, but the number of monthly backtest decisions is much smaller. That distinction matters. Classification metrics are evaluated on many daily observations, while portfolio performance is evaluated on monthly decisions and daily returns between rebalances.
The asset summary shows a wide range of behavior. GLD has the highest sample Sharpe among the listed assets, followed by LQD, SPY, QQQ, and IEF. QQQ has high annualized return but also very high volatility and a huge historical drawdown. DBC has weak Sharpe and a massive drawdown, which fits its commodity-futures behavior over long samples. TLT has a positive long-run return but large rate-cycle drawdowns, especially after the 2021-2022 rate shock.
This matters for regime learning because the labels shouldn’t just identify “stocks up” or “stocks down.” They need to distinguish between:
broad risk-on periods where equities and credit lead,
neutral periods where defensive assets may still do fine,
stress periods where defensive assets and gold can dominate risky sleeves,
inflation-rate stress where bonds don’t protect equity drawdowns well.
The model has to learn from cross-asset relationships, not just from one equity index.
The correlation heatmap after the asset summary shows the expected block structure. Equity ETFs and high-yield credit cluster together. Treasury ETFs cluster together. GLD and DBC are less tightly connected to the equity-credit block, which is exactly why they can help distinguish inflation, risk-off, and diversification regimes.
For regime learning, correlation structure isn’t only a diversification object. It also acts as a state variable. In calm markets, risky assets may rise together but correlations aren’t necessarily extreme. In panic markets, correlations across risky assets often jump because investors sell everything liquid at the same time. In inflation/rate stress, the stock-bond correlation can become less negative or even positive, reducing the protection from long duration.
So when we later build features like average rolling correlation, correlation shocks, and SPY-TLT correlation, we are turning this market structure into predictors. The regime model doesn’t only look at returns. It also asks whether the market’s internal relationships are behaving like a normal expansion, a defensive rotation, or a correlated stress event.
The baseline grid is intentionally short in this project. The mean-variance and MaxSharpe baselines are used to create comparison portfolios and an anchor for the regime allocations. The training-period selection picks MV (EWMA, BayesSteinMomentum) as the MV source in the first main backtest, and it uses a MaxSharpe variant as the source for Mean-CVaR and Wasserstein robust portfolios.
The main lesson from this output is that the baselines are already non-trivial. We aren’t comparing the regime layer only against a weak equal-weight portfolio. We also compare against:
a shrinkage-based MV portfolio,
a Mean-CVaR portfolio that explicitly cares about downside tail risk,
and a Wasserstein robust portfolio that penalizes fragile optimization under distributional uncertainty.
That makes the final comparison more serious. If regime learning only beats equal weight but loses to robust optimization, the result is still useful, but it tells us the ML layer is more of a risk-shaping overlay than a complete replacement for robust portfolio construction.
The new contribution is what we build on top of these pieces: a regime-learning layer that compares unsupervised clustering, Markov models, hidden Markov models, and supervised classifiers, then asks whether their state probabilities improve allocation decisions after costs.
This also explains why the data has to be broad. A regime model trained only on SPY returns would mostly learn equity trend and volatility. That would miss many important transition signals. Credit can weaken before equities sell off. Treasury duration can stop diversifying when inflation becomes the dominant risk. Gold can behave differently in liquidity crises and inflation shocks. International equities can reveal dollar and global-growth pressure. Macro factors add another lens by measuring inflation, growth, labor, policy, and financial conditions.
So the feature space is deliberately cross-asset and macro-aware. The goal isn’t to create a perfect return forecast. The goal is to give the model enough economically meaningful information to distinguish different market environments.
3) Feature Engineering for Financial Regimes
A regime model is only as good as its feature space. The feature vector at date \(t\) is:
where each feature is a market condition that would’ve been observable at time \(t\). The model later learns either:
\[
\hat{S}_t = f(X_t)
\]
for unsupervised/discovery models, or:
\[
P(Y_t = k \mid X_t)
\]
for supervised classifiers, where \(Y_t\) is a realized future regime label.
The feature set is designed around economic blocks, not random technical indicators. The blocks are:
Equity trend and leadership, such as SPY momentum, QQQ vs SPY, small-cap vs large-cap, international vs US, and EM vs developed ex-US.
Credit health, such as HYG vs LQD, high-yield drawdown, high-yield downside volatility, and HYG vs SHY.
Duration and rates, such as TLT vs SHY, IEF vs SHY, Treasury yields, and curve-slope changes.
Real assets and inflation, such as DBC, GLD, DBC vs IEF, GLD vs SPY, and energy vs utilities.
Cross-asset pressure, such as risk-on assets versus defensive assets, credit-equity divergence, inflation-duration spread, and duration-equity divergence.
Market internals, such as breadth, dispersion, volatility shock, drawdown level, drawdown speed, average correlation, and equity-bond correlation.
The reason we use blocks is that regimes aren’t one-dimensional. A high SPY return can happen in several different environments. It can be a broad growth rally, a narrow mega-cap rally, a relief rally after a crash, or an inflationary nominal rally. The regime label becomes more meaningful when the model sees equity, credit, rates, real assets, correlation, and macro conditions together.
3.1 Equity Leadership Features
The first feature block measures equity momentum and leadership.
A basic total return feature over a window \(L\) is:
where \(P_{t,i}\) is the adjusted price of asset \(i\) at date \(t\). This says how much the asset has gained or lost over the last \(L\) trading days.
The skip avoids letting very recent reversal noise dominate the signal. For SPY, the feature spy_252_21 measures roughly 12-month momentum excluding the latest month.
Relative return features take the difference between two total returns:
From the latest table, EEM vs EFA is strongly positive near the end of the sample, while QQQ vs SPY and IWM vs SPY are also positive. This tells us the latest equity leadership isn’t purely defensive. It has signs of EM strength, growth strength, and some small-cap participation. That doesn’t automatically mean the portfolio should be fully risk-on, but it gives the supervised and unsupervised models positive evidence on the risky side.
3.2 Credit, Duration, and Real-Asset Features
Credit features are central because credit often turns before equity. HYG is high-yield corporate credit, and LQD is investment-grade corporate credit. The spread-like relative return:
acts like a price-based credit sentiment indicator. When HYG outperforms LQD, credit investors are accepting default risk and liquidity risk. When HYG underperforms, the market is usually moving toward quality and safety.
We also use high-yield drawdown and downside semi-volatility. If \(r_t\) is the daily return, downside semi-volatility keeps only the negative return side:
When this is positive, long-duration bonds are beating short bills, usually because yields are falling or investors are paying for duration protection. When it is negative, duration is under pressure, usually because rates are rising or inflation risk is high.
Real-asset features use commodities, gold, and energy:
This feature is powerful in environments like 2022, where real assets and energy can hold up while duration bonds fall. In a normal equity crash with falling yields, this feature may behave very differently because Treasuries can rally. That difference helps the model separate classic risk-off from inflation/rate stress.
The latest credit and duration table shows high-yield credit still positive over the recent 63-day window, with HYG drawdown near zero and high-yield volatility low. That supports a calmer credit environment. At the same time, several duration-relative features can be weak when long bonds lag short duration. This combination gives the model a more nuanced state: risk appetite isn’t collapsing, but duration isn’t necessarily attractive.
This is exactly why regime learning can be useful. A simple stock-only rule might say “risk-on.” A cross-asset model can say “risky assets are strong, credit isn’t breaking, but the rate environment still makes long duration risky.” Later, the portfolio weights reflect that nuance by holding risky assets and gold/IEF together rather than simply going all in on equities.
3.3 Market Pressure and Internal Structure
The pressure features combine sleeves. We compare average risky performance against defensive performance:
A market where SPY is up but breadth is weak is more fragile than a market where most risky assets are rising together. The model also uses dispersion:
If this ratio jumps above 1, short-term volatility is rising faster than the slower volatility baseline. Drawdown level and drawdown speed measure whether SPY is already below its rolling high and whether the drawdown is worsening.
Correlation features use rolling correlations across assets:
In stress periods, \(\bar{\rho}_{t,L}\) often rises because diversification inside risky assets disappears. We also track correlation shocks, which compare short-window correlation to long-window correlation. This makes the model sensitive to sudden changes in market structure, not only levels.
def breadth(tickers, window): data = pd.concat([total_ret(t, window) for t in tickers], axis=1)return data.gt(0.0).mean(axis=1)def dispersion(tickers, window): data = pd.concat([total_ret(t, window) for t in tickers], axis=1)return data.std(axis=1, ddof=1)def realized_vol(ticker, window):return returns[ticker].rolling(int(window)).std(ddof=1) * np.sqrt(annualization)def drawdown_level(ticker, window):return close[ticker] / close[ticker].rolling(int(window), min_periods=max(63, int(window) //4)).max() -1.0def rolling_avg_corr(tickers, window): data = returns[tickers].astype(float) pairs = []for i, left inenumerate(tickers):for right in tickers[i +1:]: pairs.append(data[left].rolling(int(window)).corr(data[right]))return pd.concat(pairs, axis=1).mean(axis=1)
The internal market-state table near the end of the sample shows full breadth across risky assets, relatively moderate SPY volatility, and drawdown levels close to zero. That leans risk-on. But the average correlation level and correlation changes also matter because high correlation can turn a seemingly diversified risky sleeve into one large equity-credit bet.
The important point for the model is that features are allowed to disagree. We don’t force every signal to say the same thing. A regime classifier can learn that strong breadth with rising volatility means something different from strong breadth with falling volatility. It can learn that gold strength with credit stress means something different from gold strength during a broad commodity rally. The feature set is intentionally redundant at first because the next steps, VIF, PCA, and RandomForest importance, help us diagnose and reduce that redundancy.
A feature matrix for regime learning is different from a feature matrix for a normal cross-sectional prediction problem. In a normal stock-selection problem, each row might be a stock and each column might be a characteristic. Here each row is a date, and each column is a market-state measurement observed on that date.
The important point is that a regime is usually not visible in one variable. A market can have positive equity momentum but deteriorating credit. It can have strong breadth but rising inflation pressure. It can have high Treasury returns because growth is weakening, or because policy rates are falling in a calm disinflationary environment. The same raw asset return can mean different things depending on the rest of the state vector.
where \(A\) and \(B\) are two assets and \(L\) is the lookback window. If \(A=\text{HYG}\) and \(B=\text{LQD}\), this feature asks whether high-yield credit is outperforming investment-grade credit. If it is positive, credit markets are usually accepting more default risk. If it is negative, investors are usually moving toward safer credit quality.
This is always less than or equal to zero. A value near zero means the asset is near its local high. A value like \(-0.20\) means the asset is down 20% from its recent peak. In regime learning, drawdown is useful because it measures accumulated damage, not just recent volatility. A market can be volatile but still near highs; it can also be in a slow grind-down with moderate daily volatility but deep drawdown.
Volatility-shock features measure whether volatility itself is changing:
where \(\sigma_t^{(L)}\) is a rolling volatility estimate and \(m\) is a change window such as 21 or 63 trading days. Rising volatility often marks a transition from calm risk-on to unstable neutral or defensive states. Falling volatility after a selloff can mark recovery.
Correlation features are especially important in this project. A cross-asset portfolio usually depends on diversification. If equities, credit, commodities, and bonds start moving together, the portfolio’s true risk can rise even if individual asset volatilities don’t explode. A rolling average correlation can be written as:
where \(\rho_{ij,t}^{(L)}\) is the rolling correlation between assets \(i\) and \(j\) over window \(L\). High \(\bar{\rho}_t\) means diversification is weaker. A correlation shock can turn many separate trades into one large macro trade.
This is why the feature set starts broad. We aren’t assuming one variable defines the state. We first collect many economically meaningful measurements, then use diagnostics and selection to decide which ones are usable for the models.
Feature scaling matters more here than it might seem at first. Many models in this project are geometry-based. KMeans, GMM, SVM, and KNN all depend on distances, margins, or Gaussian densities in feature space. If one feature is measured in percentage points and another is measured in basis points, the larger-scale feature can dominate the model even if it isn’t economically more important.
So we standardize features before fitting most models:
\[
z_{t,j} = \frac{x_{t,j} - \mu_j}{s_j}
\]
where \(x_{t,j}\) is feature \(j\) at date \(t\), \(\mu_j\) is the training-window mean, and \(s_j\) is the training-window standard deviation. After this transformation, each feature is measured in “number of standard deviations from its normal level.”
In financial language, this makes features comparable as stress signals. A \(+2\) value for vol_change_63 means volatility has risen strongly relative to its own history. A \(+2\) value for fci_blend means financial conditions are meaningfully tight relative to their own history. The units are different, but the z-score puts them on a comparable abnormality scale.
There is also a subtle leakage point here. Scaling parameters must be estimated only from the training window in a walk-forward setting. If we standardize the whole sample using future means and standard deviations, the model learns from the future indirectly. For example, the volatility distribution after 2020 isn’t the same as before 2020. Using full-sample scaling tells the pre-2020 model how extreme future crisis volatility will be, which isn’t available in real time.
The same training mean and training standard deviation are used for the test observation. This is exactly the same logic as fitting a scaler in a normal ML pipeline, but in finance the chronological order is more important because market distributions change through time.
The feature matrix isn’t only a numerical input. It is the economic vocabulary that all later models are allowed to speak.
A good regime model should have features that are interpretable after the fact. This matters because portfolio decisions need explanation. If the model reduces risk, we want to know whether it is because credit weakened, volatility rose, breadth broke, macro pressure tightened, or duration stopped diversifying.
The selected feature list gives us a compact economic dashboard:
FCI and macro blocks describe the macro backdrop. Tight financial conditions usually pressure risky assets because discount rates rise, credit becomes less available, and liquidity becomes less supportive. A high fci_blend can therefore push the model toward defensive or neutral states even before equities fully sell off.
Volatility and drawdown features describe market damage. vol_change_63 captures acceleration in uncertainty. dd_level captures realized drawdown pressure. These features often separate normal pullbacks from real regime shifts.
Credit features describe balance-sheet risk. HYG downside semivariance and HYG volatility show whether lower-quality credit is becoming unstable. Credit weakness is especially important because equity markets can remain calm for a while even when credit risk starts repricing.
Duration features describe the rate side. Features such as ief_shy_126, tlt_shy_126, and tlt_ief_126 tell us whether duration is being rewarded or punished. This is crucial after the 2022 experience, where long bonds weren’t always defensive.
International leadership features describe global risk appetite. EFA vs SPY, EEM vs EFA, and IWM vs SPY help identify whether leadership is broad or narrow. EM strength can indicate global liquidity and dollar conditions are supportive. Small-cap strength can indicate domestic cyclicality and credit appetite.
Correlation and breadth features describe internal market structure. Strong breadth means many risky assets are participating. High correlation means diversification is weaker. A market with strong breadth and low correlation is healthier than a market with high correlation and falling breadth.
This feature interpretation is one of the reasons Project 16 can work as a learning source. The model isn’t a black box sitting on random numbers. Each variable is tied to a market mechanism, and the diagnostics show which mechanisms ended up mattering most.
Before we fit regime models, we inspect the feature matrix. Financial features are usually correlated because they are constructed from overlapping prices and overlapping horizons. For example, tlt_shy_126, tlt_ief_126, and ief_shy_126 are mathematically connected. If TLT beats IEF and IEF beats SHY, then TLT should also beat SHY. That creates multicollinearity.
The variance inflation factor for feature \(j\) is:
\[
\text{VIF}_j = \frac{1}{1 - R_j^2}
\]
where \(R_j^2\) comes from regressing feature \(x_j\) on all the other features. If \(R_j^2\) is close to 1, the feature is almost explained by the others, and \(\text{VIF}_j\) explodes. A high VIF doesn’t mean the feature is useless. It means linear models may have unstable coefficients because the model can’t uniquely assign credit to one correlated feature.
PCA gives another view. If \(X\) is the standardized feature matrix, PCA decomposes it into orthogonal directions:
\[
X = UDV^\top
\]
The columns of \(V\) are loading vectors, and each principal component is:
\[
PC_{t,k} = X_t v_k
\]
The explained variance ratio for component \(k\) is:
where \(\lambda_k\) is the variance explained by component \(k\). If the first few PCs explain a large share of the variance, the feature space has strong common structure. If the variance is spread across many PCs, the features contain more independent dimensions.
RandomForest permutation importance answers a more predictive question. After training a model, we shuffle one feature and see how much the score falls:
where \(X_{\pi(j)}\) is the feature matrix with column \(j\) permuted. If shuffling feature \(j\) hurts balanced accuracy, that feature carries useful predictive information.
Show code
def feature_vif_local(x): z0 = x.replace([np.inf, -np.inf], np.nan).dropna().astype(float) rows = []for col in z0.columns: other = [c for c in z0.columns if c != col]ifnot other: rows.append({"feature": col, "r2": np.nan, "vif": np.nan})continue y0 = z0[col].to_numpy(dtype=float) X0 = z0[other].to_numpy(dtype=float) r2 = LinearRegression().fit(X0, y0).score(X0, y0) vif = np.inf if r2 >=1.0-1e-12else1.0/ (1.0- r2) rows.append({"feature": col, "r2": r2, "vif": vif})return pd.DataFrame(rows).set_index("feature").sort_values("vif", ascending=False)def pca_tables_local(x, n_components=10): z0 = x.replace([np.inf, -np.inf], np.nan).dropna().astype(float) arr = StandardScaler().fit_transform(z0) n =min(int(n_components), min(arr.shape)) pca = PCA(n_components=n, random_state=42).fit(arr) cols = [f"PC{i +1}"for i inrange(n)] explained = pd.DataFrame( {"explained_variance": pca.explained_variance_,"explained_variance_ratio": pca.explained_variance_ratio_,"cumulative": np.cumsum(pca.explained_variance_ratio_), }, index=cols, ) loadings = pd.DataFrame(pca.components_.T, index=z0.columns, columns=cols)return explained, loadings
The first diagnostic output shows exactly the expected multicollinearity problem. In the full candidate set, many VIFs are infinite because several features are deterministic or near-deterministic combinations of others. This isn’t surprising. Relative-return features share the same assets, slope changes share the same Treasury yields, and macro blocks are intentionally related.
The PCA output on all 79 candidate features shows that PC1 explains about 25.1% of the variance, PC2 explains 13.2%, and the first 10 PCs explain about 77.6%. This means the feature set has strong common structure, but the regime information isn’t one-dimensional. A single “risk-on/risk-off” axis wouldn’t capture the full state. Rates, inflation, credit, equity leadership, correlation stress, and macro pressure all matter.
The correlation/PCA plot makes the same point visually. There are large blocks of correlated features, but there are also pockets of independent behavior. That is exactly the kind of setting where we shouldn’t feed every raw feature into every model. We first let a RandomForest selector identify variables that actually help predict the realized regime labels.
The VIF output looks extreme, but it is actually a useful warning. VIF, or variance inflation factor, asks how well one feature can be explained by the other features. For feature \(j\), we regress it on all other features and compute the \(R^2\) from that auxiliary regression:
\[
x_j = a + X_{-j}b + e
\]
Then:
\[
VIF_j = \frac{1}{1-R_j^2}
\]
If \(R_j^2\) is close to zero, the other features don’t explain feature \(j\), so \(VIF_j\) is close to 1. If \(R_j^2\) is close to 1, feature \(j\) is almost a linear combination of the others, so \(VIF_j\) becomes huge. If \(R_j^2=1\), the VIF is infinite.
In this project, infinite VIF is expected because many features are built from the same underlying assets. For example, tlt_shy_126, ief_shy_126, and tlt_ief_126 are mathematically related. If we know two of them, the third contains very little new information. Similarly, many macro blocks share inflation, policy, and financial-condition components. This doesn’t mean the features are useless. It means some models will over-count the same signal if we feed all features blindly.
The model impact depends on the method:
Linear models can get unstable coefficients because correlated variables compete with each other.
LDA needs covariance estimation, so correlated variables can make \(\Sigma^{-1}\) unstable unless we use shrinkage.
KMeans and KNN can over-weight repeated directions because distance counts every dimension.
GMM/HMM can overfit covariance structure if we use too flexible covariance matrices.
Tree ensembles can handle correlation better, but importance may be split across related features.
PCA gives a complementary view. Instead of asking whether each individual variable is redundant, PCA asks how many independent directions explain the feature space. For standardized feature vector \(Z_t\), PCA finds orthogonal directions \(v_1,\dots,v_p\) such that the first component explains maximum variance:
\[
PC_{1,t} = v_1^\top Z_t
\]
with:
\[
v_1 = \arg\max_{\|v\|=1} Var(v^\top Z_t)
\]
The next component is the highest-variance direction orthogonal to the first one. In matrix form, PCA is based on the eigen-decomposition of the correlation/covariance matrix:
\[
S = V\Lambda V^\top
\]
where \(V\) contains eigenvectors and \(\Lambda\) contains eigenvalues. The explained-variance ratio of component \(m\) is:
In the full candidate set, the first PC explains about 25% of the variance and the first 10 PCs explain about 78%. This tells us the features have strong common structure, but they aren’t reducible to a single risk factor. The selected set is cleaner: PC1 is about 22%, PC2 about 19%, and the first 8 PCs explain about 84%.
That is exactly the kind of structure that makes regime learning interesting. There is enough commonality for regimes to exist, but enough separate information for different models to disagree. If all features collapsed into one component, the project would only need a one-dimensional risk indicator. If every feature were independent noise, regime learning would fail. The actual result sits between those extremes: a structured but multi-dimensional market-state space.
The threshold avoids labeling tiny differences as meaningful regimes. If the risk sleeve beats the defensive sleeve by only a few basis points, that isn’t enough to call the market risk-on. A regime label should represent an economically meaningful future difference.
with wider thresholds of \(\pm 0.04\). This slower label is useful later when walk-forward classifiers need a more persistent target.
The target construction is one of the most important econometric choices in the project. A bad label can make even a strong model useless. Here the label isn’t “SPY up or down.” It is the future return difference between a risk sleeve and a defensive sleeve. That makes the target directly aligned with allocation.
The risk sleeve represents assets that usually benefit when investors accept growth, credit, and equity risk. The defensive sleeve represents assets that usually benefit when investors want duration, cash-like safety, or crisis hedges. The target spread:
is positive when risk assets beat defensive assets over the next 63 trading days. It is negative when defensive assets beat risk assets.
This target is better aligned with portfolio selection than a single-asset return label because a regime strategy is making a relative allocation decision. It doesn’t only care whether SPY is positive. It cares whether carrying risk beats holding defensive exposure. For example, SPY could rise 2% over a quarter, but if GLD and IEF rise 6%, a defensive allocation was still better. A regime model should capture that.
The neutral class is important because not every date deserves a strong risk-on or defensive label. Financial markets often sit in mixed environments where risky and defensive sleeves perform similarly. If we forced every observation into risk-on or defensive, the model would learn noisy boundaries around tiny return differences. The neutral label reduces that noise.
The 126-day label adds a slower horizon. A 63-day label captures quarter-like tactical conditions. A 126-day label captures a more persistent half-year macro regime. In practice, a strong regime model shouldn’t only recognize short-term market stress; it should also understand slower macro environments such as tightening financial conditions, persistent inflation pressure, or extended growth leadership.
Show code
def state_profile_local(features, labels, outcomes=None): lab = pd.Series(labels, name="state") parts = [features.copy()]if outcomes isnotNone: out = outcomes.copy()ifisinstance(out, pd.Series): out = out.to_frame() parts.append(out.add_prefix("outcome_")) z0 = pd.concat(parts + [lab], axis=1).replace([np.inf, -np.inf], np.nan).dropna(subset=["state"]) cols = [c for c in z0.columns if c !="state"] prof = z0.groupby("state")[cols].mean() prof.insert(0, "share", z0.groupby("state").size() /len(z0)) prof.insert(0, "observations", z0.groupby("state").size()) prof.index = prof.index.astype(int)return prof.sort_index()def sort_profile_states(profile): cols = [c for c in profile.columns if c notin {"observations", "share"}] z0 = profile[cols].astype(float) z0 = (z0 - z0.mean(axis=0)) / z0.std(axis=0, ddof=0).replace(0.0, np.nan) z0 = z0.fillna(0.0) score = pd.Series(0.0, index=profile.index)for col in z0.columns: name =str(col).lower()if"risk_defensive_spread"in name: score +=3.0* z0[col]continueifany(k in name for k in ["spy", "qqq", "iwm", "efa", "eem", "hyg", "risk_on", "breadth", "growth", "credit", "cyclical", "sector_ew"]): score += z0[col]ifany(k in name for k in ["vol", "corr", "dispersion"]): score -= z0[col]if"dd"in name or"drawdown"in name: score += z0[col]if"defensive_sleeve"in name: score -=0.5* z0[col]if"tlt"in name and"outcome"in name: score -=0.5* z0[col]return score.sort_values(ascending=False).index.astype(int).tolist()def remap_series(labels, order): mapping = {int(old): int(new) for new, old inenumerate(order)}return pd.Series(labels).map(mapping).astype(int)
The forward regime profile confirms that the labels have real economic meaning. The risk-on class has strong positive 63-day and 126-day forward outcomes for SPY, HYG, DBC, EEM, and the risk sleeve. The defensive class has negative forward SPY and EEM outcomes, strong TLT and GLD outcomes, and a negative risk-defensive spread. The neutral class sits between those two.
The class balance isn’t perfectly equal. Risk-on is the largest class, neutral is smaller, and defensive is meaningful but less frequent. This is normal for market regimes. Most days aren’t crisis days. But this creates an ML problem: raw accuracy can reward a model that mostly predicts risk-on. That is why balanced accuracy and macro F1 matter later.
This punishes models that perform well only on one class. In finance, that is crucial because the rare class is often the one we care about most. A model that misses defensive periods can still have acceptable accuracy, but it can fail as a portfolio tool.
Show code
fig, axes = plt.subplots(1, 2, figsize=(12.5, 4.3))y.value_counts(normalize=True).sort_index().plot(kind="bar", ax=axes[0])axes[0].set_title("Forward 63-day regime balance")axes[0].set_ylabel("Share")profile_cols = [ c for c in ["outcome_risk_sleeve_63","outcome_defensive_sleeve_63","outcome_risk_defensive_spread_63","outcome_SPY_63","outcome_HYG_63","outcome_TLT_63","outcome_DBC_63","outcome_GLD_63","outcome_risk_defensive_spread_126", ]if c in future_profile.columns]profile_z = future_profile[profile_cols]profile_z = (profile_z - profile_z.mean()) / profile_z.std(ddof=0).replace(0, np.nan)im = axes[1].imshow(profile_z.fillna(0.0).values, aspect="auto", cmap="coolwarm", vmin=-2.2, vmax=2.2)axes[1].set_yticks(range(len(profile_z.index)))axes[1].set_yticklabels(profile_z.index)axes[1].set_xticks(range(len(profile_z.columns)))axes[1].set_xticklabels([c.replace("outcome_", "") for c in profile_z.columns], rotation=35, ha="right")axes[1].set_title("Forward regime outcome profile")fig.colorbar(im, ax=axes[1], fraction=0.046, pad=0.04)plt.tight_layout()plt.show()
The class distribution also teaches an important ML lesson. Risk-on observations are the largest class because most market days aren’t stress days. Defensive regimes are less frequent, but they matter disproportionately for portfolio outcomes because they usually coincide with large drawdowns or rapid rotations.
This is why raw accuracy can be misleading. Suppose risk-on is 57% of observations. A lazy model that always predicts risk-on could get decent raw accuracy while being useless in crises. In a portfolio context, missing the defensive class is more costly than missing a few neutral periods.
Balanced accuracy fixes part of that problem by averaging recall across classes:
where \(TP_k\) is the number of true positives for class \(k\) and \(FN_k\) is the number of missed observations from class \(k\). This metric gives equal importance to risk-on, neutral, and defensive regimes even if they aren’t equally common.
Precision asks whether predicted class \(k\) observations really belong to class \(k\). Recall asks whether the model finds the actual class \(k\) observations. In regime switching, both matter. If the model predicts defensive too often, the portfolio may sit in safe assets and miss rallies. If it predicts defensive too rarely, it may stay risk-on into drawdowns. The metrics help us detect those two different errors.
The RandomForest importance table selects features that are economically meaningful, not just statistically convenient. The top features include:
fci_blend, linking the regime model to the macro/financial-condition framework from Project 12.
vol_change_63, showing that rising volatility carries regime information.
dd_level, because deep drawdowns often separate defensive states from normal volatility.
macro_breadth_conflict_block, which captures disagreement between macro stress blocks and market breadth.
international and cross-asset relative returns such as efa_spy_126, iwm_spy_126, and eem_efa_126.
duration and policy features such as ief_shy_126, rate_10y, tlt_shy_126, and policy_rate_pressure_block.
credit and downside risk features such as hyg_vol_63, hyg_downside_semivar_63, and hyg_lqd_change_63.
This is a good feature-selection result because it doesn’t collapse into one category. The selected set includes macro pressure, volatility, drawdown, international leadership, credit, duration, and inflation. That gives the regime models enough information to distinguish:
equity-led risk-on,
credit-supported risk-on,
neutral defensive yield environments,
inflation-rate stress,
and broad market stress.
After selection, we transform features with an expanding z-score:
where \(\bar{x}_{t-1,j}\) and \(s_{t-1,j}\) are computed only from observations before date \(t\). This avoids lookahead. The clip at \(\pm 4\) prevents extreme crisis observations from dominating the scale forever.
The selected-feature VIF table still shows high VIF for duration features, especially tlt_shy_126, tlt_ief_126, and ief_shy_126. This is expected because those spreads share assets. The model doesn’t remove all redundancy. Instead, it keeps economically useful redundancy and relies on methods like PCA-HMM, tree models, regularization, and validation to handle it.
RandomForest feature selection is used as a pragmatic filter, not as the final truth about causality. The idea is to ask: if a nonlinear tree ensemble tries to predict future regime labels, which features actually help out of sample?
Permutation importance is especially useful because it measures the loss of predictive performance when one feature is randomly shuffled. If feature \(j\) is important, shuffling it breaks its relationship with the target and model performance drops. If shuffling it barely changes performance, the feature may be redundant or uninformative.
where \(X^{\pi_j}\) is the same feature matrix after randomly permuting column \(j\). The bigger \(I_j\) is, the more the model depended on that feature.
The selected features make economic sense. fci_blend is the highest-ranked feature, which means the macro/financial-condition system from Project 12 carries information about future cross-asset regimes. vol_change_63 and dd_level are also near the top, which means recent volatility acceleration and drawdown damage matter. macro_breadth_conflict_block is important because markets often become vulnerable when price breadth and macro conditions disagree. Relative equity features such as efa_spy_126, iwm_spy_126, and eem_efa_126 show whether leadership is U.S.-centric, small-cap-led, or international/EM-led.
This feature selection stage also helps with interpretability. Instead of saying “the model saw 79 variables,” we can say the final regime models are mostly learning from:
financial conditions,
volatility acceleration,
drawdown pressure,
macro-market disagreement,
international leadership,
duration and policy pressure,
credit downside risk,
and inflation/stagflation blocks.
That is a coherent economic state space. It gives the model a chance to separate regimes in ways that match market intuition.
The selected-feature PCA is cleaner than the full candidate PCA. PC1 explains about 22.0%, PC2 about 18.7%, and the first 8 components explain about 83.8%. So the selected feature set is still strongly structured, but it is less bloated than the 79-feature candidate set.
This matters for the models:
KMeans and GMM can be sensitive to correlated dimensions because distance and Gaussian density can over-count repeated information.
PCA-HMM deliberately compresses features before fitting the sequence model.
LDA benefits from shrinkage because class covariance estimates can become unstable when predictors are correlated.
RandomForest and boosting can handle correlated features better, but importance can be split across related variables.
SVM and KNN depend on geometry, so scaling and redundancy strongly affect the margin and nearest-neighbor structure.
This diagnostic stage isn’t just a plot-making step. It tells us what kind of model risk each method faces before we compare the results.
6) Unsupervised Learning for Regime Discovery
We now split the problem into two learning styles.
Unsupervised learning sees only \(X_t\). It tries to discover structure in the feature space without using future labels. The output is a state assignment:
\[
\hat{S}_t = g(X_t)
\]
The model doesn’t know the future risk-defensive spread. After fitting, we profile each discovered state by looking at its realized future outcomes. Then we order the states as risk-on, neutral, or defensive.
Supervised learning sees pairs \((X_t, Y_t)\) during training. It directly learns:
\[
P(Y_t = k \mid X_t)
\]
and evaluates whether those probabilities predict future regimes.
The teaching plot in this section shows the difference cleanly. Discovery takes today’s features and finds latent states. Prediction uses future labels during training and then predicts the next realized state.
For finance, both styles are useful.
Unsupervised models are closer to econometric regime discovery. They ask: “What clusters or latent states exist in the market data?” Supervised models are closer to forecasting. They ask: “Given today’s features, which future sleeve outcome is most likely?”
We include both because regime modeling has two separate goals:
Interpretation, which benefits from unsupervised and Markov-style models.
Prediction, which benefits from supervised classifiers and explicit out-of-sample labels.
Unsupervised learning is the discovery layer. It doesn’t use the forward regime labels when fitting the states. It only sees the feature matrix \(X_t\) and tries to find structure. Afterward, we evaluate whether the discovered states correspond to economically meaningful future outcomes.
This gives us a useful separation:
Discovery: identify clusters or latent states from today’s market conditions.
Economic mapping: inspect future returns conditional on those states.
Portfolio usefulness: test whether the states produce useful weights after costs.
In notation, unsupervised models estimate:
\[
\hat{S}_t = g(X_t)
\]
or, for probabilistic models:
\[
\hat{p}_{t,k} = P(S_t=k \mid X_t)
\]
without directly using \(Y_t\) during fitting. That makes them appealing for regime discovery because they can find states we did not label by hand. The danger is that a statistically clean cluster can still be financially weak. A cluster might separate high volatility from low volatility, but if both clusters have similar future risk-defensive spreads, the separation doesn’t help allocation.
So every unsupervised model is judged on both statistical and economic criteria:
Silhouette: are observations geometrically well separated?
Minimum state share: is every state large enough to matter?
Average duration: are states persistent enough for monthly allocation?
Transitions per year: is the state path too jumpy?
Posterior confidence: are probabilities decisive or diffuse?
Economic separation: do states actually differ in future returns?
This is why the output table is more important than the individual model label. We aren’t looking for a pretty clustering result. We are looking for a state representation that can support a real portfolio policy.
Update step: replace each centroid with the mean of the points assigned to it.
\[
\mu_k = \frac{1}{N_k}\sum_{t:c_t=k}X_t
\]
The geometry plot shows the result in a two-dimensional teaching example. Each colored region is the area closest to one centroid. The boundaries are straight lines because the model is using Euclidean distance. That makes KMeans easy to understand, but it also creates limitations. Regime clusters in finance are rarely spherical, equally sized, or separated by straight boundaries.
For example, a defensive regime may not be a compact cloud. It can include the COVID crash, the 2022 inflation-rate shock, and smaller volatility shocks. Those episodes may share “stress” behavior, but they don’t need to occupy one neat ball in feature space. KMeans can still be useful as a baseline, but we shouldn’t expect it to be the most realistic regime model.
The KMeans objective is simple but powerful. For \(K\) clusters, it chooses centroids \(\mu_1,\dots,\mu_K\) and assignments \(c_t\) to minimize within-cluster squared distance:
Each centroid is the average feature vector of observations assigned to that cluster. The algorithm alternates between:
assigning each observation to the nearest centroid,
recomputing each centroid as the mean of assigned observations.
The financial interpretation is that each centroid becomes a typical market environment. One centroid might represent calm risk-on: positive breadth, low volatility, healthy credit, and normal financial conditions. Another might represent stress: negative breadth, rising volatility, weak credit, and tight FCI. A third might represent mixed conditions.
The main weakness is the geometry assumption. KMeans works best when clusters are roughly spherical and similar in size after scaling. Financial regimes rarely behave like clean spherical clusters. Stress states can be smaller, more extreme, and more stretched. Neutral states can be broad and overlapping. Because KMeans uses hard assignments, it also doesn’t express uncertainty. A date just on the boundary is treated as fully belonging to one state.
The teaching plot helps here: cluster boundaries are straight Voronoi regions around centroids. In a real market-state space, that means the model assigns regimes based on nearest average environment. This is clear and interpretable, but it can miss nonlinear or overlapping regimes.
Agglomerative clustering builds a hierarchy. It starts with every observation as its own cluster and then repeatedly merges the closest groups. The teaching diagram shows this as a dendrogram. Cutting the tree at a chosen height gives the number of clusters.
With Ward linkage, the merge cost between two clusters \(A\) and \(B\) is:
\(\bar{X}_A\) and \(\bar{X}_B\) are cluster centroids.
Ward linkage tries to merge clusters in a way that increases within-cluster variance as little as possible. That makes it somewhat similar to KMeans, but the hierarchical path is different. Once two observations are merged, they stay together.
The useful part for market regimes is the nested structure. A broad stress group may contain subgroups: credit stress, rate stress, and equity crash. Agglomerative clustering can reveal that hierarchy. The limitation is that it still doesn’t model sequence dynamics. It doesn’t know that regimes tend to persist over time.
Agglomerative clustering builds the state structure from the bottom up. At the start, every observation is its own cluster. Then the algorithm repeatedly merges the closest clusters until only \(K\) clusters remain. The merge rule depends on the linkage method. A common choice is Ward linkage, which merges clusters that create the smallest increase in within-cluster variance.
If \(A\) and \(B\) are two clusters, Ward linkage chooses the merge that minimizes the increase in:
\[
W = \sum_{k=1}^{K}\sum_{t\in C_k}\left\|X_t-\bar{X}_{C_k}\right\|_2^2
\]
This produces a hierarchical tree, or dendrogram. The tree diagram in we is important because it shows that clustering doesn’t have to be a flat one-step assignment. The same data can have nested structure. For example, the top-level split might separate stress from normal markets. Inside normal markets, another split might separate growth leadership from reflation leadership. Inside stress, another split might separate inflation stress from recession stress.
In portfolio terms, hierarchy is useful because regimes often have nested meanings. A two-state model might say “risk-on vs risk-off.” A three-state model might add neutral. A four-state model might split defensive into “duration works” and “duration fails.” Agglomerative clustering helps us see whether such nested structure exists.
The smoother state duration in the output suggests that hierarchical clusters are less jumpy than KMeans here. That is good for turnover, but it also means the model may be slow to detect sudden transitions. This tradeoff appears repeatedly in regime modeling: fast models adapt quickly but can whipsaw; slow models are stable but can lag.
KMeans assigns each observation to one nearest centroid. A Gaussian Mixture Model (GMM) is more flexible. It assumes the feature distribution is a weighted sum of Gaussian components:
This is a soft membership probability. Instead of saying “today is exactly state 1,” the model can say today’s feature vector is 70% close to risk-on, 20% close to neutral, and 10% close to defensive.
GMM parameters are usually estimated by expectation-maximization (EM). The EM logic is:
E-step: compute responsibilities \(\gamma_{t,k}\) using current parameters.
M-step: update \(\pi_k\), \(\mu_k\), and \(\Sigma_k\) using those responsibilities.
The mixture plot shows why this is more realistic than KMeans. The component ellipses can overlap. Financial states often overlap because the transition from expansion to stress is gradual. GMM captures that uncertainty with probabilities.
A Gaussian mixture model adds probability to clustering. Instead of saying each date belongs entirely to one cluster, it says each date has a probability of coming from each component:
This soft assignment is more useful for portfolios than KMeans hard labels. If a date is 55% risk-on and 45% neutral, the allocation can be blended. It doesn’t need to jump fully from one sleeve to another.
The diagonal covariance choice is conservative. A full covariance matrix for state \(k\) has \(p(p+1)/2\) parameters, where \(p\) is the number of features. With many correlated features and a limited number of stress observations, that can overfit badly. A diagonal covariance only estimates one variance per feature inside each state:
This assumes conditional independence across features within each state. That assumption isn’t literally true in markets, but it can be a good regularization choice. It lets each state have different feature volatility without trying to estimate every cross-feature covariance.
The Bayesian mixture goes one step further by making component weights uncertain. A Dirichlet prior gives the model a way to shrink unused components:
If the data doesn’t support a component, its posterior weight can become small. This is useful because we often don’t know the right number of regimes. Financial markets may have two dominant regimes in one era, four in another, and several small transition states around crises. A Bayesian mixture can start with more possible components and let weak ones fade.
The main implementation uses diagonal covariance for the GMM variants. A diagonal covariance means each state has its own variance per feature, but it doesn’t estimate all feature-to-feature covariance inside each component:
This is a practical design choice. With many correlated features and limited regime history, a full covariance GMM can overfit badly. Diagonal covariance sacrifices some flexibility but makes the density estimate more stable.
The Bayesian Gaussian mixture adds a prior over component weights. In we, we start with an upper bound of six candidate components, but the prior lets weak components shrink away. This is useful when we don’t know the right number of states.
If the prior concentration is small, the posterior can push unsupported components toward very low weights. In a Dirichlet-process-style mixture, this acts like “let the data decide how many components are active.”
The teaching plot shows six possible component densities, but only the supported components remain meaningfully weighted. This is helpful because market data may have a dominant risk-on regime, a neutral regime, a stress regime, and maybe a few smaller transition states. We don’t want to force every candidate component to matter.
Unsupervised labels are arbitrary. If a KMeans model returns labels 0, 1, and 2, those numbers don’t automatically mean risk-on, neutral, and defensive. The model only knows geometry. It doesn’t know finance.
So after fitting, we build an economic profile for each discovered state. For state \(k\), the profile is:
Then we order states by economic meaning. A state with high future risk-sleeve return and high risk-defensive spread becomes risk-on. A state with low future risky returns, better defensive returns, and negative spread becomes defensive. A state in the middle becomes neutral.
This step is crucial. Without economic ordering, a cluster label could flip between runs. The model could call risk-on “cluster 2” in one run and “cluster 0” in another. Portfolio rules need stable names, so we remap discovered states into a consistent economic order.
The profile table is also a guard against fake regimes. If two clusters have very similar future outcomes, they may be statistically separated but financially unimportant. A good regime isn’t only a pretty cluster. It should correspond to different future asset behavior.
7) Markov Chains and State Persistence
The next step is the stochastic-process part of the project. A regime process is a sequence:
\[
S_1, S_2, \dots, S_T
\]
where each \(S_t\) takes one of a small number of states, such as risk-on, neutral, or defensive.
A Markov chain assumes the next state depends on the current state, not the full past path:
The diagonal entries are persistence probabilities. If \(A_{00}=0.82\), then a risk-on day has an 82% probability of being followed by another risk-on day. The expected duration of state \(i\) is:
\[
E[D_i] = \frac{1}{1-A_{ii}}
\]
So if a state has a self-transition probability of 0.82, its expected duration is about:
\[
E[D] = \frac{1}{1-0.82} \approx 5.56
\]
observations, under the geometric-duration assumption. If the self-transition probability is 0.98, the expected duration becomes 50 observations. This is why small changes in diagonal probabilities can create much smoother regime paths.
Show code
fig, ax = markov_chain()plt.show()
The Markov chain diagram in we is a teaching simplification. It shows risk-on, neutral, and stress states with self-transition probabilities and cross-transition probabilities. The point is that regimes usually persist. A stress state doesn’t usually appear for one isolated day and vanish forever. A risk-on trend can persist for months. A neutral state can act as a transition zone.
We can update a state distribution forward through time using:
\[
p_t = p_{t-1}A
\]
where \(p_t\) is a row vector of state probabilities. If today’s distribution is \(p_t = [0.70, 0.20, 0.10]\), and \(A\) has strong diagonal probabilities, tomorrow’s distribution will still be mostly risk-on unless the observed features provide strong contrary evidence.
This persistence is financially useful because it reduces whipsaw. A clustering model can flip states whenever the feature vector crosses a boundary. A Markov model adds inertia. It says the market can change regimes, but the prior probability of staying in the same state is usually high.
A Markov chain is the first true stochastic-process model in the project. It models a sequence of states:
\[
S_1,S_2,\dots,S_T
\]
where each \(S_t\) belongs to a finite state space such as:
The diagonal entries are persistence probabilities. If \(A_{00}=0.95\), a risk-on state tends to remain risk-on. If \(A_{22}=0.80\), defensive states also persist, but less strongly. Off-diagonal entries describe transitions. For example, \(A_{02}\) is the probability of jumping from risk-on directly to defensive.
A state distribution evolves by multiplication:
\[
p_{t+1} = p_t A
\]
If today’s probability vector is \(p_t=[0.70,0.20,0.10]\), then tomorrow’s prior state distribution is the weighted average of the transition rows. This is the “state inertia” that clustering lacks.
The \(n\)-step transition matrix is:
\[
A^n = \underbrace{A A \cdots A}_{n\text{ times}}
\]
The entry \((A^n)_{ij}\) gives the probability of being in state \(j\) after \(n\) steps if we start in state \(i\). This is useful for thinking about multi-month regimes. If the strategy rebalances monthly, we caren’t only about tomorrow’s state but whether a state is likely to persist long enough to justify a portfolio tilt.
The expected duration of state \(i\) is approximately:
\[
E[D_i] = \frac{1}{1-A_{ii}}
\]
If \(A_{ii}=0.90\), expected duration is about 10 periods. If \(A_{ii}=0.50\), expected duration is only 2 periods. This formula comes from the geometric distribution: if the probability of leaving the state each period is \(1-A_{ii}\), the expected waiting time before leaving is the inverse of that exit probability.
This duration formula is one of the cleanest links between stochastic-process math and trading. A model with very low state duration creates high turnover and unstable allocations. A model with very high duration may fail to adapt. The transition matrix lets us quantify that tradeoff rather than describe it vaguely.
7.1 Markov Regression
The Markov regression model in this project is a regime-switching time-series model applied to the global equal-weight return series. Instead of clustering a multivariate feature vector, it models a univariate return process whose mean and variance can switch by regime.
A simple Markov-switching regression can be written as:
\(\sigma_{S_t}^2\) is the regime-specific variance,
\(S_t\) follows a Markov chain.
The likelihood must sum over unobserved state paths. Directly summing over all possible paths would be impossible for long time series because there are \(K^T\) possible state sequences. Instead, the Hamilton filter updates state probabilities recursively.
The filtered probability is:
\[
P(S_t=k \mid y_1,\dots,y_t)
\]
It combines two pieces:
the predicted probability from yesterday’s state distribution and the transition matrix,
the likelihood of today’s return under each regime.
For a regime with low volatility and positive mean, a calm positive return gets high likelihood. For a high-volatility stress regime, a large negative return gets higher likelihood. The filter updates the posterior regime probabilities accordingly.
The MarkovRegression output has strong likelihood diagnostics because it is directly a time-series likelihood model, but its economic separation in the main table isn’t the highest. That makes sense. It sees one return series, while the feature-based models see macro, credit, rates, commodities, and internal market structure.
The transition matrix gives several diagnostics that are useful beyond the basic state plot.
Stationary distribution. If a Markov chain is stable and ergodic, it has a long-run distribution \(\pi\) satisfying:
\[
\pi = \pi A
\]
with:
\[
\sum_{k=1}^{K}\pi_k = 1
\]
This means that if we keep applying the transition matrix for a very long time, the state distribution tends toward \(\pi\) regardless of the starting state. In finance, the stationary distribution can be interpreted as the model’s long-run regime mix. If \(\pi_{\text{risk-on}}\) is very high, the model believes risk-on is the dominant long-run environment. If \(\pi_{\text{defensive}}\) is too small, the model may underrepresent crises.
A stationary distribution isn’t a forecast that the future must match the historical regime mix. It is a diagnostic of the fitted transition structure. If the stationary distribution says the defensive state is basically impossible, but the realized sample contains meaningful stress periods, the transition model may be too optimistic.
Expected duration. The formula:
\[
E[D_i]=\frac{1}{1-A_{ii}}
\]
turns transition probabilities into a trading concept. If the state is expected to last only a few days, monthly allocation won’t benefit much from detecting it. If the state lasts several months, it can support a regime sleeve.
For example, if a defensive state has \(A_{22}=0.96\), then:
\[
E[D_2]=\frac{1}{1-0.96}=25
\]
If the time unit is daily observations, this is roughly one trading month. If the model is fit on monthly observations, the same number would mean 25 months, which would be far too persistent. So duration must always be interpreted together with the observation frequency.
Transition asymmetry. Market regimes are often asymmetric. Risk-on may drift slowly into neutral before turning defensive. Defensive states may recover quickly after policy support or liquidity intervention. This asymmetry appears in off-diagonal transition probabilities:
A chain with high \(A_{\text{risk-on},\text{defensive}}\) allows direct crash transitions. A chain with low direct transition but high risk-on-to-neutral and neutral-to-defensive movement implies deterioration tends to pass through a transition state.
For portfolio construction, the transition path matters. If the model often moves through neutral before defensive, the portfolio can gradually reduce risk. If direct risk-on-to-defensive transitions are common, the strategy needs faster risk controls.
Regime entropy. A Markov or HMM probability vector can be summarized by entropy:
Entropy is close to 0 when one state dominates and close to 1 when probabilities are spread evenly. This becomes a risk-control input later because uncertain probabilities shouldn’t create extreme allocations. A confident 80% defensive probability and a mixed 35/35/30 probability have different portfolio meanings even if defensive is the largest class in both.
This is why Markov models fit portfolio work so naturally. They produce probabilities, persistence, duration, transition risk, and uncertainty. Those are exactly the objects a portfolio layer can use.
A small numerical Markov example makes the persistence idea concrete. Suppose the transition matrix is:
The first row says a risk-on state stays risk-on with 92% probability, moves to neutral with 7%, and jumps directly to defensive with 1%. The third row says a defensive state stays defensive with 75%, recovers to neutral with 20%, and jumps back to risk-on with 5%.
Even before observing tomorrow’s features, the model is still mostly risk-on because the transition matrix has persistence. But defensive probability rises slightly from 5% to 6.8% because there is some chance of deterioration through neutral or direct transition.
Now suppose tomorrow’s features show a strong volatility shock and weak credit. The HMM emission likelihood for the defensive state rises, and the filtering update can move the defensive probability much more. The transition matrix gives inertia, while the emission likelihood gives evidence. The final probability is the balance between both.
This is the whole appeal of Markov regime modeling in finance. We don’t want the model to forget yesterday’s regime instantly, and we don’t want it to ignore today’s new stress evidence either.
This means each hidden state has its own average feature profile and feature volatility. A risk-on state might have high breadth, positive credit momentum, lower volatility pressure, and positive equity leadership. A defensive state might have negative breadth, higher drawdown, high volatility shock, stronger GLD/TLT behavior, and worse risk-defensive forward outcomes.
The joint probability of states and observations is:
where \(\pi_{S_1}\) is the initial state probability.
This equation is the full HMM story in one line. The state path has Markov persistence, and each state emits observations according to its own feature distribution.
8.1 Filtering, Smoothing, and State Probabilities
The HMM doesn’t observe \(S_t\) directly. It infers probabilities. The forward recursion computes:
\(\alpha_{t-1}(i)\) is yesterday’s joint probability of being in state \(i\).
\(A_{ij}\) moves probability from state \(i\) to state \(j\).
\(p_j(X_t)\) checks whether today’s features look likely under state \(j\).
For numerical stability, the probabilities are normalized at each step. The normalized version gives filtered probabilities:
\[
P(S_t=j \mid X_1,\dots,X_t)
\]
Smoothing uses the full sample:
\[
P(S_t=j \mid X_1,\dots,X_T)
\]
Filtering is closer to real-time use. Smoothing is helpful for diagnostics because it uses future observations to classify historical states more cleanly. In backtesting, we need walk-forward probabilities, so the model is refit through time and probabilities are produced using information available up to the decision date.
The HMM teaching diagram shows hidden states at the top and observed features at the bottom. The state sequence transitions through time, and each state emits feature observations. This is exactly the right structure for markets: the regime is latent, but returns, volatility, credit, rates, and macro conditions are observable emissions of that latent regime.
A Hidden Markov Model adds one more layer: the regime state isn’t directly observed. We only observe market features. The state is hidden, and the model infers it from the feature sequence.
The first equation is the transition model. It says how hidden regimes move through time. The second equation is the emission model. It says what feature vectors we expect to observe inside each regime.
The joint probability of a full state path and observation sequence is:
This formula is the backbone of the model. It combines persistence from the transition matrix with feature evidence from the emission densities.
There are three probability objects that matter:
Filtering estimates the current state using observations up to today:
\[
\alpha_t(k) = P(S_t=k \mid X_{1:t})
\]
This is what a real-time strategy can use. At date \(t\), we only know \(X_{1:t}\).
Smoothing estimates the state using the full sample:
\[
\gamma_t(k) = P(S_t=k \mid X_{1:T})
\]
This is useful for diagnosis because it uses future observations to better infer what historical states probably were. It isn’t a real-time trading probability.
Prediction projects the state distribution forward:
This update has a clear intuition. A state gets high probability if:
it was likely based on yesterday’s state distribution and transition matrix,
today’s features look likely under that state’s emission distribution.
For example, suppose the model was mostly risk-on yesterday. If today’s features show low volatility, strong breadth, and healthy credit, the risk-on probability remains high. If today’s features suddenly show volatility shock, credit stress, and falling breadth, the emission likelihood of the defensive state rises and the probability can shift.
The backward recursion computes future evidence:
\[
\beta_t(i) = P(X_{t+1:T}\mid S_t=i)
\]
Then smoothing combines forward and backward information:
This is useful for analyzing historical plots. Smoothed probabilities usually look cleaner than filtered probabilities because they know what happened afterward. But for live portfolio construction, filtered or walk-forward probabilities are the honest object.
The HMM filtering update can also be understood with a simple probability example. Suppose the prior state probability after applying the transition matrix is:
Now today’s feature vector arrives. The emission densities say how likely today’s features are under each state. Suppose the features are moderately stressful:
The result is still not fully defensive because the prior was heavily risk-on. But defensive probability rises from 10% to 25% because today’s features look more defensive than risk-on. That is the filtering logic.
This example also explains why HMM probabilities often move smoothly. The model needs enough evidence to overcome the transition prior. A single noisy feature movement may not flip the state. A persistent sequence of stressful observations can.
Show code
fig, ax = hidden_markov_model()plt.show()
8.2 HMM Training with EM
The HMM parameters are usually estimated by an EM algorithm called Baum-Welch. The observed data are \(X_{1:T}\), but the state path \(S_{1:T}\) is hidden. EM handles this by alternating between estimating state probabilities and updating parameters.
This is very similar to the GMM EM update, but with one important addition: transition probabilities \(\xi_t(i,j)\). GMM treats observations as independent. HMM treats observations as a sequence.
That difference is the main reason HMMs are so natural for market regimes. The current state isn’t only determined by today’s feature vector. It also depends on where the market probably was yesterday.
The Baum-Welch EM algorithm is important enough to unpack slowly. The difficulty is that we don’t observe \(S_t\). If we did observe the state path, estimating parameters would be easy. We would count transitions to estimate \(A\), and compute the mean/variance of features inside each state to estimate \(\mu_k\) and \(\Sigma_k\).
Because the states are hidden, EM replaces hard counts with probability-weighted counts.
The E-step computes:
\[
\gamma_t(k)=P(S_t=k\mid X_{1:T})
\]
and:
\[
\xi_t(i,j)=P(S_t=i,S_{t+1}=j\mid X_{1:T})
\]
Here \(\gamma_t(k)\) is the probability that date \(t\) belongs to state \(k\). \(\xi_t(i,j)\) is the probability that the model moved from state \(i\) to state \(j\) between \(t\) and \(t+1\).
The numerator is the expected number of transitions from \(i\) to \(j\). The denominator is the expected number of times we were in state \(i\) before a transition. So \(A_{ij}\) is an expected transition frequency.
This is a weighted average of feature vectors. Dates that strongly belong to state \(k\) get high weight. Dates that barely belong to state \(k\) get low weight.
For diagonal covariance, the variance of feature \(m\) in state \(k\) is:
So the model learns not only the average feature level in each regime but also the typical variability of each feature inside that regime.
This is exactly why HMMs are more appropriate than static clustering for market regimes. A GMM estimates similar emission distributions, but it treats observations as independent. The HMM adds the transition matrix, so a state assignment is shaped by both today’s feature vector and yesterday’s inferred state. That is the econometric structure we want for markets.
PCA-HMM first compresses the selected feature matrix into principal components, then fits an HMM on those components. This helps when features are correlated and high-dimensional.
If \(Z_t\) is the standardized feature vector and \(V_m\) contains the first \(m\) PCA loadings, then the compressed vector is:
\[
F_t = V_m^\top Z_t
\]
The HMM is then fit on \(F_t\) instead of \(Z_t\).
The advantage is that PCA removes redundant directions. Instead of fitting emissions on many correlated features, the HMM sees a smaller set of orthogonal factors. The downside is interpretability. A hidden state defined by PCs is less directly tied to original macro and market features.
The HMM and the supervised classifier answer different questions, so their disagreement is useful.
The HMM estimates:
\[
P(S_t=k\mid X_{1:t})
\]
where \(S_t\) is a latent state. It tries to explain the sequence of observed features through hidden regimes. It cares strongly about persistence because the transition matrix makes today’s state depend on yesterday’s state.
The supervised classifier estimates:
\[
P(Y_t=k\mid X_t)
\]
where \(Y_t\) is a realized future regime label created from the 63-day risk-defensive spread. It tries to predict forward outcomes directly from the current feature vector.
So the HMM is a state inference model, while the classifier is a future-label prediction model. The two objects are related, but they aren’t identical. A current latent state can be risk-on while the future label is neutral if the next 63 days are mixed. A current state can look defensive after a selloff even if the future 63-day return is positive because markets rebound.
This explains why the hybrid system is valuable. The HMM brings sequence discipline and regime persistence. The supervised model brings direct alignment with the forward allocation target. If both agree, the signal is stronger. If they disagree, the hybrid can soften the allocation instead of choosing one model blindly.
where \(\alpha_t\) is the rolling utility-based weight. This is a probability ensemble. It combines two different sources of information before the portfolio layer converts probabilities to weights.
A useful interpretation is:
HMM probability says: “given the feature path and persistence, which latent state are we in?”
ML probability says: “given today’s features, which future regime label is most likely?”
Hybrid probability says: “which probability engine has recently been more useful for allocation, and how should we blend them now?”
This keeps the workflow coherent. We don’t use ML only because it is modern, and we don’t use HMM only because it is econometric. We compare both through the same portfolio objective.
9) Supervised Learning for Future Regime Prediction
The supervised part turns the project into a classification problem.
\(Y_t\) is the realized future regime label from the 63-day risk-defensive spread.
A classifier estimates:
\[
\hat{p}_{t,k} = P(Y_t=k \mid X_t)
\]
These probabilities are more useful than hard labels because portfolio weights can be blended across states. If the model is only 45% risk-on, 28% defensive, and 27% neutral, we shouldn’t treat the market as fully risk-on.
The training/test split uses observations up to 2018-12-31 for training and later observations for testing. This is a time-series split, not a random split. Random splitting would mix future and past observations, making the test set too easy because market regimes are serially correlated.
The classifier score includes:
accuracy, raw fraction correct,
balanced accuracy, average recall across classes,
macro F1, average F1 across classes,
log loss, probability quality,
economic utility, whether the predicted probabilities align with future risk-defensive spread.
A model can have decent accuracy but bad log loss if it is confidently wrong. In portfolio allocation, confidence matters because high-confidence probabilities can move weights more aggressively.
Supervised learning changes the problem. Instead of discovering states from \(X_t\) alone, we train models to predict the realized forward label \(Y_t\).
This is closer to standard machine learning. We have features, labels, train/test splits, model metrics, and probability outputs. But finance adds three complications:
Labels are delayed. A 63-day forward label is unknown until 63 trading days later.
Classes are imbalanced. Risk-on is more common than defensive.
The cost of errors is asymmetric. Missing defensive periods can hurt more than missing normal periods.
The supervised model estimates a conditional probability:
\[
\hat{p}_{t,k}=P(Y_t=k\mid X_t)
\]
The probability vector is more valuable than a hard class prediction because portfolio weights can be blended across regimes. If the model says 45% risk-on, 30% neutral, and 25% defensive, the portfolio can hold a mixed allocation. This is much more realistic than switching 100% into a single regime sleeve.
The loss function for many probabilistic classifiers is cross-entropy:
This loss strongly punishes confident wrong predictions. If the true class is defensive and the model assigns only 1% probability to defensive, the penalty is large. That is exactly what we want for portfolio use. A model that is confidently wrong during stress can cause large allocation mistakes.
For this project, probability quality matters at least as much as class accuracy. A classifier with slightly lower accuracy but better probability calibration can be more useful in portfolio construction because weights respond to probabilities, not only hard labels.
A supervised regime model has to solve three problems at once: classification, probability estimation, and economic usefulness.
Classification asks whether the predicted class matches the realized class:
\[
\hat{Y}_t = \arg\max_k \hat{p}_{t,k}
\]
This gives accuracy, balanced accuracy, and F1 metrics.
Probability estimation asks whether the full vector \(\hat{p}_t\) is meaningful. Two models can predict the same hard class but with different confidence. A model predicting \([0.36,0.33,0.31]\) and a model predicting \([0.90,0.07,0.03]\) both choose risk-on, but their portfolio implications are very different.
Economic usefulness asks whether the probability direction aligns with the future return spread. The regime signal used in we is:
If \(s_t\) is high, the model leans risk-on. If \(s_t\) is negative, it leans defensive. A simple economic utility check multiplies this signal by the future spread \(\Delta_t^{63}\):
\[
u_t = s_t \Delta_t^{63}
\]
If the model leans risk-on and the risk sleeve beats the defensive sleeve, \(u_t\) is positive. If it leans defensive and the defensive sleeve beats the risk sleeve, \(u_t\) is also positive because both \(s_t\) and \(\Delta_t^{63}\) are negative. If the model leans the wrong way, \(u_t\) is negative.
Then the utility ratio is scaled like a Sharpe ratio:
\[
U = \sqrt{12}\frac{E[u_t]}{\operatorname{Std}(u_t)}
\]
This isn’t the final strategy return because it ignores actual weights, costs, and constraints. But it tells us whether the probability signal points in an economically useful direction.
This is why the selected classifier can differ from the highest-accuracy classifier. Accuracy treats all correct labels equally. A portfolio cares more about whether probability tilts line up with the return spread and whether the signal is stable enough to trade.
There is also a bias-variance tradeoff across the models.
A high-bias model, like simple logistic regression, may underfit because it can only draw linear boundaries. A high-variance model, like a complex boosted tree, may fit training patterns that don’t persist out of sample. LDA sits in an interesting middle ground. It is simple, but it uses class means and covariance geometry. With shrinkage, it can be stable in correlated macro/market feature spaces.
That is a key reason LDA can win in this project. The feature set is already economically engineered. We don’t necessarily need a very complex learner to discover everything from scratch. A simpler model can perform well when the features are strong and the sample is limited.
The supervised ML workflow can be summarized as a disciplined pipeline:
Build features using only information available at date \(t\).
Build labels using future outcomes, but only use labels after they become known.
Split training and testing chronologically.
Fit scalers and models only on the training window.
Evaluate both classification metrics and economic utility.
Convert probabilities into weights using rules that control turnover and risk.
Each step prevents a common finance-ML mistake.
If features use future information, the signal is leaked. If labels are used before the horizon passes, the training set is leaked. If scaling uses full-sample means and standard deviations, the model sees the future distribution. If we evaluate only accuracy, the model may ignore rare defensive states. If we convert hard predictions into all-in/all-out portfolios, transaction costs and whipsaw can destroy performance.
The whole project is built around avoiding these mistakes. The model outputs probabilities, not just labels. The portfolio layer blends probabilities, smooths weights, applies risk budgets, and anchors to a baseline. The final comparison includes costs and strong non-ML baselines.
This is also why we works as both an ML course and an econometrics course. The ML side teaches features, classifiers, validation, calibration, and model comparison. The econometrics side teaches Markov dependence, hidden states, likelihood, EM, filtering, transition probabilities, and state persistence. The finance layer connects both to allocation.
The supervised model set is deliberately diverse. Each model class has a different bias:
Model
Main structure
Useful when
Main risk
Logistic regression
linear log-odds
boundaries are simple and probabilities are needed
underfits nonlinear regime interactions
LDA
Gaussian class clouds with shared covariance
class means separate regimes cleanly
covariance assumptions are too simple
RandomForest
bagged decision trees
nonlinear interactions matter
probabilities can be overconfident or class-biased
Gradient boosting / LightGBM
sequential tree correction
tabular nonlinear signals are strong
overfitting if tuned too aggressively
SVM
margin-based boundary
separation is geometric and nonlinear kernels help
probability calibration can be weak
KNN
local neighborhood voting
similar historical states repeat
sensitive to scaling and sparse samples
The important modeling question isn’t whether one algorithm is universally better. Each one represents a different assumption about the shape of regime boundaries.
Logistic regression assumes class separation is roughly linear in the transformed feature space. A risk-on probability might rise smoothly with breadth and fall smoothly with volatility. LDA assumes the class distributions look like ellipsoids with a shared covariance matrix. Tree models assume regimes can be separated by threshold rules and interactions. SVM focuses on margins. KNN assumes similar past feature vectors imply similar future regimes.
The finance side is what makes this comparison meaningful. If regimes are mostly driven by gradual linear pressure, LDA or logistic regression can work well. If regimes depend on nonlinear combinations such as high volatility plus weak breadth plus tightening FCI, tree models can help. If regimes are local patterns that repeat, KNN can help. If class clouds overlap but have separable margins, SVM can help.
The supervised models can also be understood through their decision surfaces.
Logistic regression creates smooth probability surfaces. The log-odds of class \(k\) relative to a reference class are linear:
\[
\log\frac{P(Y=k\mid X)}{P(Y=r\mid X)} = \beta_k^\top X
\]
A one-unit increase in feature \(j\) changes the log-odds by \(\beta_{k,j}\), holding other features fixed. In regime terms, if vol_change_63 has a positive coefficient for defensive, rising volatility increases defensive probability.
LDA creates linear boundaries too, but from a generative assumption. It estimates each class mean \(\mu_k\) and shared covariance \(\Sigma\). The term \(\Sigma^{-1}\mu_k\) means class separation is measured in covariance-adjusted space. If two features are highly correlated, LDA doesn’t count them as two fully independent signals. This is one reason shrinkage LDA can work well when VIF is high.
RandomForest creates piecewise-constant regions. A tree might split on dd_level, then on fci_blend, then on hyg_vol_63. This creates rules such as: deep drawdown plus tight financial conditions plus high credit volatility implies defensive probability. The forest averages many such trees, reducing dependence on one split.
Boosting creates an additive sequence of corrections. A weak tree handles one part of the classification error, then the next tree focuses on the residual mistakes. In tabular finance data, boosting can be very strong, but it can also chase sample-specific interactions if the dataset isn’t large enough.
SVM focuses on the margin. In a binary setting, the decision boundary is chosen to maximize distance from the closest training points. With kernels, it can bend the boundary. The margin idea is attractive when classes are separable in transformed geometry, but regime classes often overlap.
KNN stores history. For a new date, it looks at nearby historical feature vectors and averages their labels. This is intuitive for regimes because “similar market environments” may have similar outcomes. But it becomes fragile in high dimensions because distances become less informative. This is the curse of dimensionality. If every observation is far from every other observation, nearest neighbors aren’t very reliable.
The final model table reflects these theoretical differences. RandomForest and LightGBM do well on raw accuracy, which means nonlinear rules are useful. LDA wins the selection score, which means simpler covariance-aware class geometry gives better balance and economic usefulness. KNN and SVM are reasonable but less compelling in this feature space.
Each class has a coefficient vector \(\beta_k\). A positive coefficient means that increasing that feature raises the log-odds of the class relative to others.
The model is estimated by minimizing regularized cross-entropy:
Logistic regression is simple, transparent, and often a good baseline. Its weakness is linear decision boundaries. If defensive regimes require nonlinear combinations, such as “rising volatility plus falling breadth plus credit underperformance,” logistic regression can miss those interactions unless features explicitly encode them.
In this project, logistic regression isn’t the strongest classifier. That doesn’t make it useless. It tells us that the regime boundary probably isn’t cleanly linear in the selected feature space.
The predicted class is the class with the largest discriminant score. Probabilities come from transforming these class scores.
The LDA implementation uses shrinkage, which is important because the selected features are correlated. Shrinkage stabilizes the covariance matrix:
\[
\Sigma_{\text{shrink}} = (1-\lambda)\hat{\Sigma} + \lambda T
\]
where \(T\) is a target covariance structure, often diagonal or scaled identity.
LDA ends up as the best supervised model by selection score in the main output. It has the highest balanced accuracy, around 0.464, and the highest macro F1, around 0.447. This is meaningful because it shows LDA is doing a better job across all regimes, not just the majority risk-on class. Its raw accuracy is lower than RandomForest and LightGBM, but the class balance problem makes raw accuracy less important.
A RandomForest is an ensemble of decision trees. Each tree partitions the feature space into regions and predicts a class distribution inside each leaf. The forest averages probabilities across trees:
Each tree is trained on a bootstrap sample and uses random subsets of features at splits. This reduces variance compared with a single tree. The forest can learn nonlinear interactions, such as:
drawdown is dangerous only when credit is weak,
high rates are dangerous only when growth is slowing,
volatility shock matters more when breadth is deteriorating.
RandomForest has the highest raw accuracy in the main classifier table, around 0.583. But its balanced accuracy is only around 0.381, and its macro F1 is around 0.330. This suggests it predicts common regimes well but isn’t as balanced across classes as LDA.
That is a common issue in imbalanced financial classification. A model that is good at identifying the dominant state can look strong by accuracy, while still being less useful for defensive detection.
Boosting builds models sequentially. Each new tree tries to correct errors left by the previous ensemble. In a simplified additive form:
\[
F_m(X) = F_{m-1}(X) + \nu h_m(X)
\]
where:
\(F_m\) is the ensemble after \(m\) trees,
\(h_m\) is the new tree,
\(\nu\) is the learning rate.
For classification, boosting minimizes a multiclass loss, usually cross-entropy. The model learns nonlinear boundaries and interactions, often with strong predictive power.
LightGBM is a high-performance gradient boosting implementation. It can fit complex nonlinear models efficiently and often works well on tabular data. In thisn’tebook, LightGBM has high raw accuracy, around 0.563, but balanced accuracy around 0.398. That is better balanced than RandomForest but still below LDA in the main comparison. Its log loss is also weaker than RandomForest in the main output.
Boosting is powerful, but in financial time series it can overfit subtle historical patterns. A complex model can learn regime quirks from one era and fail in a new era. This is why we evaluate both statistical metrics and economic utility.
The model tries to maximize the margin while allowing some violations. With kernels, SVM can create nonlinear boundaries. In we, SVC is used with probability output, so it can feed regime probabilities into portfolio weights.
SVC has balanced accuracy around 0.373 in the main table. It isn’t the selected model, but it performs reasonably. The issue is that SVM probabilities can be sensitive to calibration and class imbalance.
where \(\mathcal{N}_K(t)\) is the set of nearest neighbors. Distance-weighted KNN gives closer neighbors more influence.
KNN is intuitive: if today’s feature vector looks like past stress episodes, predict stress. But it has weaknesses in high-dimensional correlated spaces. Distance becomes less meaningful as the number of dimensions grows, and correlated features can over-weight one economic theme.
In the quality table, KMeans with 3 and 4 states has reasonable silhouette scores, around 0.18 and 0.17 in the main application. The 3-state version has a small minimum state share around 5.6%, which means one cluster is quite small. KMeans 4 improves economic separation slightly but adds another cluster. That extra cluster can be useful if the market has a distinct inflation-stress or credit-stress state, but it also makes the final mapping harder because the portfolio policy is built around risk-on, neutral, and defensive.
The model’s main strength is interpretability. You can look at centroids and see which features are high or low in each cluster. The weakness is that KMeans gives hard assignments and has no time dependence. If one day is barely on one side of a boundary and the next day is barely on the other side, the state can flip even if the market hasn’t really changed.
That is why later models use soft probabilities and Markov persistence.
The main table shows agglomerative clustering has lower transition frequency than KMeans. Agglomerative 3 has an average state duration above 250 observations and only about 0.94 transitions per year. That is much smoother than KMeans. Agglomerative 4 also has strong economic separation, around 1.45, but one state is small.
This tells us the hierarchical grouping is finding more persistent states. In a portfolio setting, persistence is useful because frequent regime switching can create turnover and unstable allocations. But too much persistence can also be a problem. If the model is slow to leave a state, it may miss sudden crashes or sudden recoveries.
So we don’t choose a model only by smoothness. We need to compare:
separation across future outcomes,
minimum state size,
state duration,
transition frequency,
and whether the probability path can be used in a real walk-forward setting.
In the quality table, GMM 3 and GMM 4 have strong posterior confidence, around 0.997 and 0.986, but their economic separation differs. GMM 4 reaches around 1.40, while GMM 3 is closer to 0.65. This suggests that adding a fourth component helps separate economically different states. The problem is that the final portfolio policy is designed around three named regimes, so a four-state unsupervised discovery model has to be mapped carefully.
This is one of the recurring tradeoffs in regime work:
more states can capture more market nuance,
fewer states are easier to interpret and use in allocation,
too many states can create unstable labels and small-sample profiles.
In the output, the Bayesian GMM has six active labels after filtering, with component weights such as 0.3174, 0.2134, 0.1479, 0.0491, and 0.2582, while one small component is near 0.0138. It also has the strongest economic separation in the main model-quality table, around 1.84. That means its states separate realized future outcomes well. Its weakness is practical: six states are harder to map cleanly into the three allocation regimes.
The output shows that Bayesian GMM finds more economic separation, but it also creates a very small state. That is an honest warning. Rare states can be meaningful, but they can also be sample artifacts. If a state has only a tiny share of observations, we shouldn’t let it dominate portfolio policy without further validation.
In the output, PCA-HMM 3 has smoother state behavior than some models, but its economic separation is only around 0.657 in the main table. That suggests compression helped stability but didn’t improve the separation of future risk-defensive outcomes enough to dominate.
HMM Quality in the Main Output
The HMM 3 model has a low silhouette score, around 0.167, but that isn’t fatal. Silhouette is a static clustering metric based on geometric separation. HMMs are sequence models, so their strength isn’t only static separation. They also use transition persistence.
The HMM 3 model has:
3 states,
minimum state share around 17.3%,
average state duration around 426 observations,
transitions per year around 0.53,
posterior confidence around 0.999,
economic separation around 0.828.
That profile is very persistent. The model doesn’t jump between states often. This makes it attractive for portfolio use because monthly allocations won’t flip too aggressively. The selection rule picks HMM 3 as the best econometric model, even though Bayesian GMM has higher economic separation. The reason is that the selection rule rewards a combination of valid state count, minimum state size, posterior confidence, duration, and lower transition frequency.
This is a reasonable choice for a regime portfolio. A model with extremely high economic separation but too many states or too much label fragmentation can be hard to use in an allocation policy. HMM 3 is less dramatic but more stable.
The HMM quality table shows why HMM 3 becomes the selected econometric model. It doesn’t dominate every metric. Bayesian GMM has higher economic separation, and some four-state models separate more nuanced regimes. But HMM 3 has a strong combination of posterior confidence, state persistence, and practical interpretability.
The high posterior confidence means that most dates are assigned with a clear state probability rather than a flat distribution. The long average duration means the state path is stable enough for monthly portfolio allocation. The economic separation isn’t the highest, but it is meaningful. In other words, HMM 3 gives a usable state process.
The convergence warnings during walk-forward fitting should be treated honestly. The warnings show that some EM fits reach tiny negative likelihood changes, often on the order of \(10^{-6}\) or \(10^{-7}\). This usually means the optimizer is extremely close to a numerical plateau. It isn’t necessarily a model failure, but it warns us that HMM likelihood surfaces can be flat, sensitive, and initialization-dependent.
For a trading model, this matters because small parameter changes can shift state probabilities. That is one reason the portfolio layer later uses smoothing, probability blending, entropy penalties, and MV anchoring. We don’t want the final allocation to overreact to small changes in the hidden-state fit.
9.6 Probability Calibration and Portfolio Use
A classifier used for portfolio construction needs usable probabilities, not only good class labels. If a model predicts risk-on with 95% probability but is only right 60% of the time in those cases, the portfolio will take too much risk. If a model is always under-confident, it may never shift weights enough to matter.
A probability model is well calibrated when:
\[
P(Y=k \mid \hat{p}_k \approx q) \approx q
\]
For example, among observations where the model says 70% risk-on, risk-on should occur about 70% of the time. In finance, calibration is hard because regimes drift and the sample is limited, but probability quality still matters.
Log loss helps check calibration because it punishes confident wrong predictions more than mild wrong predictions:
\[
-\log(0.90) \approx 0.105
\]
for a correct 90% prediction, but:
\[
-\log(0.10) \approx 2.303
\]
for a wrong 90% prediction. This asymmetry is exactly what we want. A model that is confidently wrong in stress periods can damage a portfolio.
The portfolio layer also controls this risk by using probability blending, entropy penalties, and smoothing. We don’t let a single classifier probability instantly turn into an extreme position.
Probability calibration connects ML output to allocation size. If a classifier gives probabilities, we need those probabilities to mean something. A 70% risk-on forecast should behave roughly like a 70% risk-on condition historically.
For example, among all observations where the model assigns risk-on probability around 70%, risk-on should happen around 70% of the time. In finance, this will never be perfect because regimes are nonstationary and samples are small. But wildly miscalibrated probabilities are dangerous.
where \(\hat{p}_{t,Y_t}\) is the predicted probability assigned to the true class. If the model assigns high probability to the true class, log loss is small. If it assigns tiny probability to the true class, log loss becomes large.
This matters because later the probability vector directly affects weights:
If probabilities are overconfident, the strategy will swing too hard into one sleeve. If probabilities are too flat, the strategy will barely move. The ML model isn’t only a classifier; it is a position-sizing engine.
10) Model Selection and Confusion Matrices
The model-quality comparison has two separate panels:
econometric/unsupervised/sequence models,
supervised classifiers.
The econometric table reports silhouette, state shares, average duration, transitions per year, posterior confidence, entropy, and economic separation. The supervised table reports accuracy, balanced accuracy, macro F1, log loss, and economic utility.
The final selected models are:
Best econometric model: HMM 3.
Best ML model: LinearDiscriminantAnalysis.
The HMM 3 choice is based on stability and probability quality. It has very high posterior confidence and long average state duration. It doesn’t have the highest economic separation, but it gives a clean three-state probability process with strong persistence.
The LDA choice is based on balanced classification. It has the best balanced accuracy and macro F1 in the main classifier table. RandomForest and LightGBM have higher raw accuracy, but the regime problem isn’t only about predicting the most common class.
The confusion matrix for the LDA model shows the class-level behavior:
For true risk-on observations, many are correctly predicted as risk-on, but a large number are predicted as neutral.
For true neutral observations, predictions are spread across all three classes. This confirms neutral is the hardest class.
For true defensive observations, the model correctly identifies a meaningful number as defensive, which matters for portfolio risk control.
Neutral regimes are naturally difficult because they are boundary states. If the forward risk-defensive spread is near zero, small market changes can move the label from neutral to risk-on or defensive. A model can be economically useful even when neutral classification is imperfect, as long as it gives reasonable probabilities.
The classifier selection score combines statistical and economic criteria. The economic utility component measures whether the model’s risk-on-minus-defensive probability aligns with the future risk-defensive spread:
This isn’t a full trading strategy, but it checks whether the probability direction has economic content. If a classifier is statistically accurate but its probability signal doesn’t align with the future risk-defensive spread, it may not help allocation.
LDA has the strongest final selection score. RandomForest is close because it has good log loss and high accuracy, but LDA’s better class balance matters more. This is a good example of model selection being tied to the financial task, not only machine-learning leaderboard metrics.
best econometric model: HMM 3
best ML model: LinearDiscriminantAnalysis
The output shows LDA is selected even though RandomForest has higher raw accuracy. That isn’t surprising. LDA has better overall selection score because it balances statistical performance and economic usefulness. It also produces smoother, more stable probabilities. In a portfolio setting, stable probability quality can beat higher raw hit rate.
Show code
best_pred = ml_models[best_ml].predict(x_test)best_cm = pd.DataFrame( confusion_matrix(y_test, best_pred, labels=class_labels), index=[f"true_{i}"for i in class_labels], columns=[f"pred_{i}"for i in class_labels],)display(best_cm)fig, ax = plt.subplots(figsize=(5.8, 4.8))im = ax.imshow(best_cm.values, cmap="Blues")ax.set_xticks(range(len(class_labels)))ax.set_yticks(range(len(class_labels)))ax.set_xticklabels(class_labels)ax.set_yticklabels(class_labels)for i inrange(best_cm.shape[0]):for j inrange(best_cm.shape[1]): ax.text(j, i, int(best_cm.iloc[i, j]), ha="center", va="center", fontsize=9)ax.set_xlabel("Predicted")ax.set_ylabel("Realized")ax.set_title(f"Confusion matrix: {best_ml}")fig.colorbar(im, ax=ax, fraction=0.046, pad=0.04)plt.tight_layout()plt.show()
pred_0
pred_1
pred_2
true_0
494
338
146
true_1
152
131
126
true_2
95
62
205
The confusion matrix gives a more concrete view of model behavior. In the main test period, the selected classifier correctly identifies many risk-on observations, but it still confuses a meaningful number of risk-on dates with neutral and some neutral dates with defensive. Defensive recall is better than a naive risk-on-only model, but it isn’t perfect.
That pattern is realistic. Neutral is the hardest class because it sits between two strong states. A neutral date can look risk-on by some features and defensive by others. Many regime models struggle with this middle state because markets transition gradually. The neutral label is still useful because it prevents the model from forcing every mixed environment into an extreme sleeve.
For portfolio construction, we care about the practical meaning of the errors:
Risk-on predicted as neutral usually reduces upside but doesn’t necessarily create disaster.
Neutral predicted as risk-on can add risk too early.
Defensive predicted as risk-on is the dangerous error because it keeps the portfolio exposed during future stress.
Risk-on predicted as defensive is costly through missed returns.
The model-selection score tries to balance these issues by combining classification metrics with economic utility. The output choosing LDA tells us that a simple, shrinkage-stabilized probabilistic classifier is more usable than a more complex model with slightly better raw accuracy but less balanced or less economically useful behavior.
11) Walk-Forward Regime Probabilities
The static train/test comparison is useful, but portfolio construction needs walk-forward probabilities. At each rebalance date, we fit or refit the selected model using only historical data available up to that date, then produce a probability vector for the current state.
The HMM refit produces some convergence warnings with extremely tiny likelihood deltas. These warnings aren’t unusual for EM-based models. The output says the current likelihood isn’t greater than the previous likelihood by a tiny amount, often around \(10^{-6}\) or smaller. That means the algorithm is effectively at a numerical plateau. We shouldn’t ignore convergence warnings, but these particular warnings don’t necessarily invalidate the estimates.
The latest HMM probabilities show strong regime jumps. In late 2025, the econometric model moves from neutral-heavy probabilities in July through September to risk-on-heavy probabilities in October and November. The ML probabilities are smoother and less certain, with the final observation around 45.5% risk-on, 26.1% neutral, and 28.3% defensive.
This difference is useful. The HMM gives a persistent sequence-based signal. LDA gives a softer feature-classification signal. The hybrid strategy later blends them instead of forcing one to be completely right.
Show code
fig, ax = walkforward_split()plt.show()
The walk-forward split diagram is important because it shows the real-time logic. Labels are only available after the horizon passes. If we use a 63-day forward label, then at date \(t\) we can’t train on labels for observations too close to \(t\), because those future returns aren’t known yet. We handle this by using cutoffs that respect the horizon.
This is one of the most important anti-leakage details in the project.
A clean supervised time-series training set at date \(t\) should satisfy:
\[
s + h \le t
\]
where \(s\) is the feature date and \(h\) is the forward-label horizon. If \(h=63\), then a feature observation from last week doesn’t yet have a realized 63-day label. Training on it would leak future information.
In a normal tabular ML problem, train/test split is simple. In finance, label availability depends on the prediction horizon. This is why walk-forward ML is harder than static ML.
The walk-forward design is where the project becomes realistic. A static train/test split is useful for diagnostics, but a live strategy needs probabilities at each rebalance date using only information available at that date.
At rebalance date \(t\), the training set must include only observations \(s\) whose forward label is already known:
\[
s+h \le t
\]
where \(h\) is the forward horizon, such as 63 trading days. If \(s+h>t\), then the label for date \(s\) depends on returns that have not happened yet by date \(t\). Including it would create lookahead bias.
This is one of the easiest mistakes to make in regime modeling. Feature values are known immediately, but labels are known only after the forward horizon passes. A model can look extremely good if it accidentally trains on labels that weren’t available at the rebalance date.
are therefore more important than the in-sample state histories. These are the probabilities the portfolio actually sees.
The latest probabilities show a useful disagreement. The econometric HMM moves strongly risk-on by October and November 2025, while the ML model remains more mixed. That disagreement isn’t a bug. The two engines read the same environment through different assumptions. HMM emphasizes sequence persistence and emission likelihood. LDA emphasizes current features relative to supervised class means. The hybrid portfolio later blends these signals instead of forcing one model to dominate always.
Time-series validation needs its own explanation because it is one of the biggest differences between finance ML and ordinary tabular ML.
In a normal supervised learning dataset, observations might be treated as exchangeable. If rows are independent, random train/test splitting can be reasonable. Financial rows aren’t exchangeable. Date \(t\) comes before date \(t+1\), labels are forward-looking, volatility clusters, and macro regimes persist.
\[
\max(\text{feature date in train}) + h \le \min(\text{test rebalance date})
\]
where \(h\) is the label horizon. This prevents using future returns that weren’t known at the time.
There is also a difference between expanding-window and rolling-window training.
In an expanding window, the training set grows through time:
\[
\mathcal{T}_t = \{1,2,\dots,t-h\}
\]
This uses all available history. It is stable and data-efficient, but it can be slow to forget old regimes. A model trained on 2008, 2020, and 2022 all at once may mix crisis types that are economically different.
In a rolling window, the training set keeps a fixed length:
\[
\mathcal{T}_t = \{t-h-L+1,\dots,t-h\}
\]
This adapts more quickly to recent environments, but it has fewer observations. In regime classification, fewer observations can be painful because defensive regimes are already rare.
This project leans toward using enough history for stable model fitting while still evaluating probabilities walk-forward. That is a practical compromise. Purely adaptive models can become too noisy. Fully static models can become stale.
Another validation issue is overlapping labels. A 63-day forward label for date \(t\) and a 63-day forward label for date \(t+1\) share 62 of the same future returns. That means labels are highly autocorrelated. The effective sample size is smaller than the row count. This is one reason we shouldn’t overreact to small differences in classifier accuracy.
If labels are overlapping, the effective number of independent horizon blocks is closer to:
\[
N_{\text{eff}} \approx \frac{T}{h}
\]
This is only a rough intuition, but it explains why a few thousand daily rows don’t mean we have thousands of independent regime examples. For a 63-day horizon, many adjacent labels describe almost the same future quarter.
This also explains why simple models can beat complex ones. Complex ML models like boosting and random forests can fit subtle patterns in daily overlapping labels. Some of those patterns may not survive in genuinely independent future periods. LDA can win because its lower flexibility acts like a regularizer.
wf_dates = pd.DatetimeIndex([d for d in rebalance_dates_all if d in x_raw.index and d in y.index or d in x_raw.index])wf_dates = wf_dates[wf_dates >= x_raw.index[min(train_days, len(x_raw) -1)]]refit_every =3p_eco_rows = []eco_state_cache =Nonefor i, dt inenumerate(wf_dates): hist_raw = x_raw.loc[:dt].tail(train_days)iflen(hist_raw) <756:continueif eco_state_cache isNoneor i % refit_every ==0: eco_state_cache = fit_eco_now(best_eco, hist_raw) p_now = eco_current_proba(eco_state_cache, x_raw.loc[[dt], selected_features]) p_now.name = dt p_eco_rows.append(p_now)p_eco = pd.DataFrame(p_eco_rows).sort_index()display(p_eco.tail().round(4))
Model is not converging. Current: -28416.947636196703 is not greater than -28416.94762900147. Delta is -7.195230864454061e-06
Model is not converging. Current: -26932.47641607526 is not greater than -26932.476415575435. Delta is -4.998255462851375e-07
Model is not converging. Current: -28010.585012426258 is not greater than -28010.585012276977. Delta is -1.4928082237020135e-07
Model is not converging. Current: -29249.863060795647 is not greater than -29249.863060633983. Delta is -1.6166450222954154e-07
risk_on
neutral
defensive
2025-07-31
0.09
0.82
0.09
2025-08-29
0.09
0.82
0.09
2025-09-30
0.09
0.82
0.09
2025-10-31
0.82
0.09
0.09
2025-11-28
0.82
0.09
0.09
Show code
def make_ml_model(name):if name =="LogisticRegression":return LogisticRegression(max_iter=2500, class_weight="balanced", C=0.75)if name =="LinearDiscriminantAnalysis":return LinearDiscriminantAnalysis(solver="lsqr", shrinkage="auto")if name =="RandomForestClassifier":return RandomForestClassifier(n_estimators=320, min_samples_leaf=24, class_weight="balanced_subsample", random_state=42, n_jobs=1)if name =="GradientBoostingClassifier":return GradientBoostingClassifier(n_estimators=260, learning_rate=0.035, max_depth=2, min_samples_leaf=25, random_state=42)if name =="LGBMClassifier"and lgbm_available:return LGBMClassifier(n_estimators=320, learning_rate=0.025, num_leaves=15, min_child_samples=45, subsample=0.85, colsample_bytree=0.85, objective="multiclass", random_state=42, verbosity=-1, n_jobs=1)if name =="SVC":return SVC(C=1.8, gamma="scale", probability=True, class_weight="balanced", random_state=42)return KNeighborsClassifier(n_neighbors=31, weights="distance")def make_calibrated_ml_model(name):return make_ml_model(name)def current_ml_proba(model, current_z):ifhasattr(model, "predict_proba"): raw = model.predict_proba(current_z)[0] classes = model.classes_else: pred = model.predict(current_z)[0] raw = np.array([1.0]) classes = np.array([pred]) p = pd.Series(0.0, index=regime_names)for cls, val inzip(classes, raw, strict=False): p[regime_names[int(cls)]] =float(val)return shrink_probabilities(p, base=0.06)def current_stress_proba(model, current_z):if model isNone:return0.0 raw = model.predict_proba(current_z)[0] classes =list(model.classes_)returnfloat(raw[classes.index(1)]) if1in classes else0.0def blend_horizon_and_stress(p_fast, p_slow, p_stress): p = shrink_probabilities(0.65* p_fast +0.35* p_slow, base=0.03) defensive_floor =0.85*float(p_stress)if defensive_floor > p["defensive"]: p["defensive"] = defensive_floor remaining =max(1.0- p["defensive"], 0.0) nondef = p[["risk_on", "neutral"]]if nondef.sum() >1e-12: p.loc[["risk_on", "neutral"]] = remaining * nondef / nondef.sum()else: p.loc[["risk_on", "neutral"]] = remaining /2.0return p / p.sum()
The score is transformed into top-\(n\) weights with maximum asset caps. This avoids allocating to every asset when only a few assets have strong regime-consistent scores.
The neutral sleeve blends the MV benchmark with scored assets. The defensive sleeve starts from SHY, IEF, GLD, TLT, and LQD, and can adapt based on recent defensive asset scores.
The model also uses a regime risk budget. A probability vector doesn’t only change which risky assets we hold. It also changes how much total risk we allow.
Entropy is near 0 when the model is confident in one state, and near 1 when probabilities are evenly spread. The risk budget is higher when risk-on probability is high and lower when defensive probability or uncertainty is high:
This is a nice design because it separates two decisions:
Composition, which assets inside the risky or defensive sleeve look strongest.
Risk level, how much of the portfolio should be in the risky blend at all.
A model can be risk-on but uncertain, in which case the entropy term prevents the allocation from becoming too aggressive. A model can be defensive with high confidence, in which case the risk budget falls and the defensive sleeve gets more weight.
12.1 Smoothing and Anchoring
The raw regime allocation can change abruptly when probabilities move. To reduce turnover, we smooth weights through time:
where \(\alpha\) controls how quickly the schedule reacts. If \(\alpha=1\), there is no smoothing. If \(\alpha\) is smaller, the portfolio moves more slowly toward the new target.
We also anchor regime schedules to the MV baseline:
where \(\eta\) is the signal share. This keeps the regime portfolio from becoming a pure classifier trade. The MV anchor adds a stable optimization-based structure, while the regime signal adds state-awareness.
This design is practical. Pure ML allocations can become unstable because the model’s probability surface changes with refits. Pure MV allocations can ignore regime information. The anchored schedule combines both.
The regime-to-portfolio mapping can be read as a layered control system.
The first layer is asset scoring. Inside each regime sleeve, assets are ranked using recent trend, momentum, mean return, hit rate, volatility, and drawdown. A simplified score is:
The exact weights are design choices. The main idea is to reward assets with strong trend and return quality while penalizing volatility and drawdown damage.
The second layer is state-specific selection. A risk-on state can favor equities, credit, commodities, and growth assets. A defensive state can favor GLD, IEF, TLT, SHY, and higher-quality bonds. A neutral state can spread between them. This means the same asset score can matter differently across states.
The fourth layer is risk budgeting through \(b_t\). This controls how much of the blended risky sleeve is actually allowed to enter the final portfolio.
The fifth layer is smoothing, which reduces turnover.
The sixth layer is anchoring, which keeps the allocation connected to the MV baseline.
These layers create a portfolio that is more cautious than a raw classifier. A raw classifier might say 55% risk-on and immediately jump into a risk-on sleeve. The layered system says: probability is only one input. We also care about uncertainty, asset quality, risk budget, turnover, and baseline stability.
This is why the regime strategy can be more diversified than tail-optimized baselines. Mean-CVaR and Wasserstein can concentrate strongly when their optimization objective finds attractive risk/reward pockets. The regime strategy is driven by state probabilities and sleeve logic, so it tends to rotate more gradually across a broader set of assets.
The portfolio layer is the place where regime probabilities become actual financial decisions. A probability vector alone isn’t a strategy. We still need rules for:
which assets belong in each regime sleeve,
how much risk to take,
how quickly to adjust,
how to control turnover,
how to combine econometric and ML signals.
A clean state-weighted allocation starts with state-specific portfolios:
\[
w_t^{(0)},\;w_t^{(1)},\;w_t^{(2)}
\]
for risk-on, neutral, and defensive states. The probability-weighted target is:
This is a soft regime switch. If \(p_{t,0}=1\), we hold the risk-on sleeve. If probabilities are mixed, we hold a weighted blend. This avoids a binary rule that flips the whole portfolio because one probability crosses a threshold.
This is an important practical rule. If the model is uncertain, the portfolio doesn’t aggressively choose one regime. It reduces risk slightly and lets the neutral/defensive pieces matter. If defensive probability rises, the portfolio cuts risky exposure. If risk-on probability rises and uncertainty is low, it can carry more risk.
If \(\alpha\) is high, the portfolio reacts quickly. If \(\alpha\) is low, the portfolio changes slowly. In finance, smoothing is often necessary because model probabilities can move faster than a real trading process should.
keeps the final portfolio connected to a stable optimization baseline from Project 2. This is especially important because ML probabilities can be wrong. Anchoring prevents a classifier from becoming the only source of risk allocation.
The hybrid blend between econometric and ML schedules uses a rolling utility comparison:
If the econometric schedule has recently produced better utility, the blend leans toward it. If the ML schedule has been stronger, the blend leans toward ML. The logistic transform keeps the blend weight between 0 and 1 and avoids sudden discontinuous switching.
A probability-weighted allocation example makes the portfolio rule clearer. Suppose the three regime sleeves allocate risky assets differently:
That is a meaningful risk reduction without a binary switch. The strategy doesn’t need to go from fully invested to fully defensive. It reduces risk gradually as the probability mass moves toward defensive.
This is important because regime probabilities are uncertain. A hard rule such as “if defensive probability is above 50%, sell risk assets” can create unstable behavior around thresholds. A probability-weighted rule changes smoothly. The model can express partial concern, and the portfolio can reflect partial concern.
The risk-budget layer then adjusts this again. If the same probability vector has high entropy, meaning the model is uncertain, \(b_t\) lowers the final risky exposure further. If the model is confident and risk-on, \(b_t\) allows more exposure. This creates a direct bridge from probabilistic uncertainty to position sizing.
There is also an implementation reason to keep the portfolio mapping smooth. The underlying models are estimated from noisy financial data, and their probabilities can shift when the training window changes. If every small probability change created a large trade, the strategy would become a model-noise amplifier. Smoothing and anchoring make the final allocation less sensitive to small estimation differences.
This is a common principle in applied finance ML: the prediction model can be complex, but the action rule should be stable. The action rule is where transaction costs, turnover, and risk limits enter. A model that predicts well but trades too aggressively can underperform a simpler model with steadier weights.
For this reason, the final model comparison should be read at the strategy level, not only at the classifier level. A classifier can look good on balanced accuracy and still create weak portfolio weights if its probability changes are too noisy. A classifier can look modest statistically and still help the portfolio if its probabilities are stable, directionally useful, and combined with good risk controls. Project 16 is built around this full chain from data to probability to allocation to costed return.
That chain is what makes the project more than a collection of algorithms. The same regime probability has different value depending on how it is validated, how it is translated into weights, and how it behaves after realistic trading costs. So the model layer and portfolio layer have to be evaluated together.
This final connection is the central lesson of the project.
It is also why the sector application matters: it tests whether the same learning architecture still works when the tradable universe and regime economics change.
The latest econometric weights and ML weights show the difference between the two probability engines. The econometric allocation is more defensive in July through September 2025, with large GLD and IEF weights. By October and November, as the HMM moves toward risk-on, QQQ, EEM, and SPY weights rise.
The ML allocation is smoother and more diversified. It still holds a large GLD weight, but it spreads more across SPY, QQQ, EFA, EEM, IEF, LQD, and HYG. This reflects the softer LDA probabilities.
The hybrid weight is a rolling blend of the econometric and ML schedules. The blend weight is based on rolling utility:
where \(U\) is a rolling return-to-volatility utility estimate. If the econometric schedule has recently had better utility than the ML schedule, the hybrid gives it more weight. The clip keeps \(\alpha_t\) from going to 0 or 1 too aggressively.
The latest hybrid allocation is high-risk overall, with risky allocation around 96.7%, but the composition isn’t a pure equity bet. It holds GLD around 25.8%, IEF around 14.7%, QQQ around 13.3%, EEM around 11.7%, EFA around 8.5%, HYG around 8.1%, and SPY around 7.4%. That tells us the regime model sees enough positive risk evidence to avoid cash, but it still values gold and intermediate duration as diversifiers.
13) Strategy Comparison and Portfolio Risk
The final strategy comparison includes:
Global EW,
MV,
Mean-CVaR 95,
Wasserstein DRMV,
Econometric Regime,
ML Regime,
Hybrid Regime.
The summary table gives an honest result. The strongest portfolio in the main cross-asset application is Wasserstein DRMV, with CAGR around 11.3% and Sharpe around 0.621. Mean-CVaR also performs strongly, with CAGR around 9.7% and Sharpe around 0.584. The ML Regime portfolio reaches CAGR around 9.3% and Sharpe around 0.545. Hybrid Regime reaches around 9.0% CAGR and 0.536 Sharpe. Econometric Regime reaches around 8.7% CAGR and 0.506 Sharpe.
So the regime layer improves over Global EW and MV, but it doesn’t beat the best robust optimization baseline in the main application.
That is still a useful result. The regime layer has a different purpose from Wasserstein DRMV. Wasserstein DRMV is a robust optimizer that handles estimation uncertainty. The regime layer uses market state probabilities to change allocation structure. In this sample, robust optimization is stronger on pure performance, while the regime models give interpretable dynamic exposure.
The turnover comparison is also important:
Mean-CVaR and Wasserstein DRMV have high turnover above 0.30 monthly.
ML Regime and Econometric Regime have lower turnover, around 0.11 and 0.10.
Hybrid Regime has turnover around 0.093.
The regime portfolios aren’t free to trade, but they are less turnover-heavy than the tail/robust baselines. That matters if costs are higher than the assumed 10 bps.
Show code
weights_by_strategy = {"Global EW": w_equal,"MV": w_mv,"Mean-CVaR 95": w_mean_cvar,"Wasserstein DRMV": w_wasserstein,"Econometric Regime": w_eco,"ML Regime": w_ml,"Hybrid Regime": w_hybrid,}w_max = pd.Series(0.40, index=backtest_cols)w_max[cash_ticker] =1.0results = run_many_weights_backtests( weights_by_strategy, returns=returns_bt, cost_bps=cost_bps, rf_daily=rf_daily, w_min=0.0, w_max=w_max, long_only=True, normalize=True, weight_timing="next_close",)nav = pd.concat({name: res.net_values for name, res in results.items()}, axis=1).dropna(how="all")strategy_returns = pd.concat({name: res.net_returns for name, res in results.items()}, axis=1).dropna(how="all")turnover = pd.concat({name: res.turnover for name, res in results.items()}, axis=1)costs = pd.concat({name: res.costs for name, res in results.items()}, axis=1)display(nav.tail().round(4))
The comparison table is one of the most important outputs because it prevents the project from becoming only a modeling exercise. A regime model can be intellectually impressive and still fail as a portfolio strategy. Here we compare it against strong baselines: Global EW, MV, Mean-CVaR, and Wasserstein DRMV.
The main result is balanced. Wasserstein DRMV is the strongest performer in the main cross-asset application, with CAGR around 11.3% and Sharpe around 0.62. Mean-CVaR is also strong, with CAGR around 9.7% and Sharpe around 0.58. The ML Regime, Hybrid Regime, and Econometric Regime strategies improve over Global EW and MV but don’t beat the best robust optimizer.
This is a good result because it is honest. It shows that regime learning adds value, but robust optimization remains hard to beat in this universe. The robust models from Project 10 directly optimize portfolio weights against mean/covariance uncertainty and tail behavior. Regime models add a probability overlay. They are useful, but they aren’t magic.
The model comparison has several layers:
Return and Sharpe. Robust and tail-aware baselines lead in the main universe. ML Regime and Hybrid Regime still produce better Sharpe than Global EW and MV.
Drawdown. MV has low drawdown because it is defensive. Regime portfolios sit between defensive MV and more aggressive robust strategies.
Turnover. Mean-CVaR and Wasserstein have high turnover, around 30% monthly. Regime portfolios have lower turnover, around 9% to 11%, which makes them more implementable.
Diversification. Regime portfolios have higher effective number of assets than robust/tail baselines. That means they aren’t simply concentrating into a few optimized bets.
Interpretability. Regime models produce a probability path and state interpretation. Robust optimizers produce strong portfolios but less direct macro/regime explanation.
This means Project 16’s contribution isn’t only “beat everything.” Its contribution is a full ML/econometric regime layer that can be compared honestly against optimization baselines and used to explain portfolio behavior.
Show code
rf_importance_table = importance.set_index("feature")realized_profile_show = future_profile[[c for c in future_profile.columns if c.startswith("outcome_")]]risky_hybrid =1.0- w_hybrid[cash_ticker]
Post-Model Diagnostics
The final diagnostic plots help explain the full system.
The selected-feature correlation heatmap still shows clusters of related predictors. That supports the earlier VIF results. The model isn’t working with independent variables. It is learning from a structured economic feature space.
The selected-feature PCA plot shows that a few components explain a large share of feature variance, but not enough to reduce the whole problem to one factor. This supports the use of both full-feature models and PCA-HMM. The feature space has common market pressure, but it also has credit, duration, macro, volatility, and international leadership dimensions.
The feature-importance panel shows that FCI, volatility change, drawdown level, macro conflict, and relative equity performance are important. This confirms that the regime model isn’t just a price momentum model. It is combining macro and market-internal information.
The model-quality plots compare econometric separation and classifier balanced accuracy. The econometric side shows that models like Bayesian GMM can create strong economic separation, while HMM 3 gives more usable state persistence. The supervised side shows that LDA is the most balanced classifier even if RandomForest and boosting look stronger by raw accuracy.
The probability plots show the different behavior of the probability engines. HMM probabilities can be persistent and sometimes sharp. ML probabilities are softer. The max-posterior-confidence plot shows when the model is highly confident and when the market state is ambiguous. Ambiguity is important because uncertain regimes should produce more diversified portfolios, not extreme bets.
The weight heatmaps show the practical consequence of probabilities. Econometric, ML, and Hybrid schedules don’t simply chase one asset. They rotate between GLD, duration, QQQ, EEM, SPY, HYG, and other assets depending on state probabilities and asset scores.
The hybrid risky allocation plot shows that the strategy usually keeps meaningful risk exposure, but it can reduce risk when defensive probability rises or uncertainty increases. This is a smoother way to implement regime switching than binary all-in/all-out rules.
The NAV and drawdown plots confirm the summary table: regime strategies improve over simpler baselines in some ways, but Wasserstein and Mean-CVaR remain very competitive. The drawdowns aren’t eliminated. The regime layer is a probability-based allocation overlay, not a crash predictor with perfect timing.
The turnover and cumulative cost plots show that the hybrid strategy has controlled turnover compared with the more optimization-heavy portfolios. This is a practical strength. A model that looks good before costs but trades constantly can be fragile. Here, the regime schedules are smoothed and anchored, so turnover stays moderate.
The post-model diagnostic plots tie the project together. The feature-correlation heatmap confirms that the selected predictors still contain structured dependence. This is expected because macro stress, volatility, credit weakness, and drawdown often move together. The selected-feature PCA plot confirms that a few common factors explain much of the variation, but not all of it. The feature-importance plot shows that macro/FCI, volatility change, drawdown, and market-internal conflict are central to the final regime logic.
The probability plots are especially important. They show that HMM probabilities are more state-persistent, while ML probabilities are softer and more mixed. This is exactly what we expect from the model structures. HMM has a transition matrix, so it naturally resists rapid switching. LDA reads the current feature vector relative to class means, so it can express uncertainty when the current environment lies between historical regimes.
The weight heatmaps show that this probability difference becomes an allocation difference. The econometric schedule can make stronger regime moves. The ML schedule is more diversified. The hybrid combines both, so it doesn’t become fully dependent on one modeling philosophy.
The latest hybrid snapshot is a good example. It has high risky allocation because the econometric model is strongly risk-on, but it still holds large GLD and IEF weights because the ML model is mixed and the asset scores favor some defensive/real-asset exposure. This isn’t a contradiction. It is exactly what a soft regime allocation can produce: high total risk budget, but diversified across assets that currently score well.
That kind of portfolio would be hard to obtain from a binary regime rule. A binary rule might simply choose risk-on assets or defensive assets. The probability-weighted system can hold a state-aware mixture.
The latest snapshot is economically interesting. The econometric model gives 82% risk-on probability, while the ML model is more mixed: about 45.5% risk-on, 26.1% neutral, and 28.3% defensive. The hybrid allocation responds by taking high risky allocation, around 96.7%, but with a large gold and intermediate Treasury component.
That combination can happen because the portfolio construction separates regime risk budget from asset ranking. The regime budget says risk exposure can be high, but the asset scoring still sees GLD and IEF as strong or useful. This produces a portfolio that isn’t a naive equity-only risk-on portfolio.
The latest hybrid weights are concentrated in GLD, IEF, QQQ, EEM, EFA, HYG, and SPY, with smaller SHY, IWM, TLT, and DBC. This fits a late-sample environment where equity leadership and EM strength are present, but gold and duration still carry important diversification roles.
The risk report adds another layer. MV has the lowest VaR/ES numbers because it is defensive and low-volatility. Wasserstein DRMV has higher return but also higher VaR/ES. The regime portfolios sit in the middle.
For VaR backtesting, the target breach rate is 5%. The Cornish-Fisher VaR method for the Hybrid Regime is very close, with a breach rate around 5.07%. The Econometric Regime CF method is also close, around 5.17%. That means unconditional coverage is reasonable. However, the Christoffersen p-values are often low, which indicates breach clustering. This is common in financial returns because losses cluster during stress periods rather than arriving independently.
That is a useful limitation. The regime model helps adapt weights, but it doesn’t eliminate volatility clustering. When a crisis begins, breaches can still cluster before the model fully adapts or before the portfolio can rebalance.
The stress-window table shows the same pattern. During COVID, all strategies suffer significant losses because the crash is fast. During the 2022 rates/inflation shock, robust and tail-aware models generally handle the period better than pure equal weight, while the regime strategies reduce some risk but don’t dominate. In 2023 rebound, ML and riskier strategies capture more upside than the more defensive regime variant.
The interpretation is balanced: regime learning is valuable, but it isn’t magic. It improves state awareness and gives dynamic allocation logic, but it still faces delay, label noise, model uncertainty, and regime changes that don’t look like the past.
The risk diagnostics show the same tradeoff from a tail perspective. MV has the lowest VaR and ES because it carries less volatility. Wasserstein has higher VaR/ES because it takes more return-seeking exposure. The regime models sit in the middle.
Historical VaR at 5% is the empirical 5th percentile of returns:
\[
VaR_{5\%}^{\text{hist}} = -q_{0.05}(R_p)
\]
Historical ES averages losses beyond that quantile:
Cornish-Fisher VaR adjusts the normal quantile using skewness and kurtosis. This helps when returns aren’t normal, but it can still fail in clustered crisis periods. Filtered historical simulation uses volatility-standardized residuals and then rescales them by current volatility, which makes it more adaptive to changing risk conditions.
The breach-rate target for 5% VaR is 5%. Hybrid CF VaR is very close, around 5.07%, and Econometric Regime CF VaR is also close, around 5.17%. That means unconditional coverage is reasonable. But Christoffersen tests often show low p-values, meaning breaches are clustered. This is common in financial returns. Losses aren’t independent. They arrive in stress clusters.
The regime model helps adapt exposure, but it doesn’t remove the fundamental clustering of losses. That is an important limitation. Regime probabilities can reduce risk when the model sees stress building, but sudden shocks and volatility cascades can still create clustered breaches.
The final system still has model risk, and it is important to state it inside the workflow rather than hiding it at the end.
Regime labels are constructed, not observed. There is no official “true” risk-on or defensive label. We define labels from future risk-versus-defensive sleeve performance. That target is economically sensible, but another investor could define regimes differently. A volatility-targeting investor might use realized drawdowns. A macro investor might use recession indicators. A credit investor might use spreads.
The feature set is engineered. Strong feature engineering is a strength, but it also means the model inherits the researcher’s assumptions. We decide that credit, duration, FCI, breadth, and relative momentum matter. That is defensible, but it is still a modeling choice.
The sample has few true crises. Even with data back to the late 1990s, the number of independent defensive regimes is small. Dot-com, GFC, COVID, 2022 inflation shock, and a few smaller episodes don’t cover every possible future crisis. A model can learn past stress signatures and still miss a new type of stress.
Probabilities aren’t guarantees. A 70% risk-on probability means the model sees an environment historically associated with risk-on outcomes. It doesn’t mean risk assets must outperform. The portfolio layer handles this by blending, smoothing, anchoring, and using entropy, but the uncertainty remains.
Economic performance can change. The sector application shows ML working better than in the cross-asset application. That doesn’t mean ML will always be better for sectors. It means that in this sample and design, sector rotation offered more repeatable patterns for the supervised model.
These limitations aren’t reasons to discard the project. They are reasons to evaluate it like a real model. A useful regime framework should be interpretable, tested against strong baselines, honest about uncertainty, and robust enough that one wrong state probability doesn’t destroy the portfolio.
14) Sector Regime Switching with the Library
The final cell repeats the full regime-switching workflow on a sector ETF universe using the packaged library implementation. This is the secondary application. The data again comes from:
The sector tradable universe includes XLB, XLE, XLF, XLI, XLK, XLP, XLU, XLV, XLY, IYR, and IYZ. This application is different from the main cross-asset one because all tradable assets are equity sectors. There is still a cash proxy, SHY, but the main rotation is inside equity sectors.
The sector features are built around sector breadth, dispersion, cyclicals vs defensives, technology leadership, energy/materials versus rates, real estate versus duration, sector drawdown, sector average volatility, sector correlations, leadership concentration, and rotation spread. These features are more about equity market internals than global cross-asset allocation.
The realized sector regimes are defined from future sector sleeve behavior:
growth sleeve: XLK and XLY,
cyclical sleeve: XLF, XLI, XLB, XLE, IYR, IYZ,
inflation sleeve: XLE and XLB,
defensive sleeve: XLP, XLU, XLV plus some SHY.
This setup asks a different question: given current market internals, should the sector portfolio lean into growth, cyclicals, inflation beneficiaries, defensives, or cash?
The sector output shows a more favorable result for the ML regime model. In the secondary application, ML Regime has the highest Sharpe among the listed sector strategies, around 0.676, with CAGR around 15.9%. MV is close, with Sharpe around 0.657 and CAGR around 14.5%. Sector EW has strong return but higher drawdown and lower Sharpe. Econometric Regime underperforms, with Sharpe around 0.523.
This makes sense. Sector rotation is a more supervised-learning-friendly problem than broad cross-asset crisis timing. Sector leadership patterns are often tied to features like breadth, dispersion, technology leadership, energy/materials strength, defensive relative performance, and volatility. A classifier can learn these patterns more directly.
The latest sector probabilities also show disagreement between models:
Econometric probability is almost split between risk-on and neutral, with very low defensive probability.
ML probability is strongly risk-on, around 80%.
The latest sector hybrid weights are concentrated in IYZ, XLK, XLY, XLI, XLU, XLF, XLP, XLV, and a small SHY position. This looks like a risk-on sector portfolio with technology/communication leadership, consumer discretionary, industrials, and some utilities/staples/healthcare ballast.
The stress output shows the limits. During COVID, all sector strategies fall sharply because sector portfolios are still equity portfolios. There is no full cross-asset escape unless SHY receives a large weight before the crash. During 2022, sector rotation helps because energy and materials behave very differently from growth and real estate. This is where regime-aware sector allocation can add real value.
The sector implementation closes the project nicely because it shows that regime switching isn’t one model. It is a workflow:
define the tradable universe,
build features that match that universe,
define economically meaningful forward labels,
compare discovery and prediction models,
convert probabilities into portfolio rules,
test performance and risk after costs.
In the cross-asset version, robust optimization is still the strongest benchmark. In the sector version, the ML regime strategy becomes more competitive and even leads the summary table. That tells us the usefulness of regime learning depends on the domain. Cross-asset regime timing is hard because crises are rare and different from each other. Sector rotation gives the model more repeated patterns across growth, cyclicals, defensives, energy, and rate-sensitive sectors.
The final risk report for the sector version confirms the same caution as the main version. VaR coverage can be close to target, but breaches can still cluster. Stress periods remain difficult. The regime model improves allocation awareness, but it doesn’t remove equity beta or crash risk. The model is strongest when it is used as a structured probability layer over portfolio construction, not as a promise of perfect regime detection.
The sector case also gives a clean lesson about domain fit. ML performance isn’t only about model class. It depends on whether the target contains repeated patterns that the feature set can learn.
Cross-asset regime timing includes rare events: crises, rate shocks, inflation shocks, liquidity stress, and policy reversals. These events are important, but the sample has only a limited number of them. A model trained from 1999 onward sees dot-com, GFC, COVID, 2022 inflation/rates shock, and a few smaller drawdowns. That isn’t a huge sample for learning all possible crisis types.
Sector rotation gives more repeated structure. Leadership rotates across technology, energy, financials, defensives, cyclicals, and real estate more often than full cross-asset crisis regimes occur. That gives supervised ML more examples of patterns such as:
growth leadership with XLK/XLY/IYZ strength,
inflation leadership with XLE/XLB strength,
defensive rotation into XLP/XLV/XLU,
cyclical recovery through XLI/XLF,
rate-sensitive weakness in real estate and utilities when yields rise.
This is why ML Regime can lead in the sector application even though it doesn’t beat Wasserstein in the main cross-asset application. The method fits the domain better.
The sector stress windows also show the limits. During COVID, every sector gets hit because all are equity-linked. Sector rotation can reduce damage but can’t create the same safe-haven effect as Treasury or gold exposure in the cross-asset universe. During 2022, sector rotation has more room to help because inflation and rates created large dispersion between energy/materials and long-duration growth/real-estate sectors.
So the secondary application isn’t just a repeated library demo. It shows when regime learning is more naturally useful: when the universe has recurring leadership cycles and enough cross-sectional dispersion for the model to exploit.
The sector application changes the nature of the problem. In the cross-asset universe, the model decides between equities, credit, duration, commodities, gold, and cash-like exposure. In the sector universe, most tradable assets are U.S. equity sectors. That means the model can’t escape equity beta as easily. It has to rotate inside equities.
This makes the ML regime model more useful in the secondary application. Sector leadership patterns repeat more often than large cross-asset crisis regimes. Technology, communication services, discretionary, energy, utilities, staples, healthcare, financials, industrials, and real estate all respond differently to macro and market conditions.
A few economic examples:
Technology and communication services usually benefit from growth leadership, falling discount-rate pressure, and risk appetite.
Energy and materials can lead during inflationary or commodity-supported regimes.
Utilities, staples, and healthcare often behave defensively when growth weakens or volatility rises.
Financials and industrials tend to respond to cyclical breadth, credit conditions, and yield-curve dynamics.
Real estate is equity-like but also very rate-sensitive.
The secondary results show ML Regime leading the sector strategy table, with Sharpe around 0.676 and CAGR around 15.9%. MV is close, but ML has better return. The hybrid also performs reasonably, though it trails ML in this application. Econometric Regime underperforms.
That tells us the usefulness of regime learning depends on the universe. In the broad cross-asset case, robust optimization is very strong. In sector rotation, the supervised ML layer finds more repeated patterns. This makes sense because sector cycles often have recurring leadership structures, while cross-asset crises are rarer and more heterogeneous.
The 2022 stress window is especially informative. Energy and materials behaved very differently from long-duration growth sectors. A sector-aware ML model can use macro/inflation/FCI and market-internal features to rotate toward the sectors that are less damaged by inflation and rate shocks. It can’t eliminate equity risk, but it can reduce the damage compared with a naive sector allocation.
The latest sector weights are concentrated in IYZ, XLK, XLY, XLI, and some XLU, with smaller weights in financials, staples, healthcare, SHY, and real estate. That is a growth/communication/industrial tilt with some defensive ballast. The probabilities show the ML model is much more risk-on than the econometric model, which explains why the sector ML strategy is more aggressive.
The sector version closes the project naturally because it shows that regime switching is a reusable workflow, not one fixed model. We keep the learning structure, but the feature interpretation, state mapping, and portfolio impact change with the tradable universe.
