In this project we move from forecasting assets to learning a portfolio policy. The earlier ML work predicted future returns or regimes, then converted those forecasts into weights with hand-built rules. Here we let an agent choose the weights directly, observe what happens to the portfolio, and update the policy from the realized reward.
Main parts in this notebook:
build a portfolio RL environment from weekly cross-asset decisions, previous weights, costs, volatility, drawdown, and market states,
define a continuous action space that turns neural-network outputs into valid long-only portfolio weights,
design a reward based on differential Sharpe with penalties for turnover, drawdown, volatility, concentration, excess cash, and redundant equal-weight behavior,
teach the RL path from MDPs and Bellman equations to Q-learning, DQN, policy gradients, actor-critic, PPO, recurrent PPO, and SAC,
train PPO, recurrent PPO, and SAC policies and compare them against the rule-based and ML portfolio models from earlier projects,
repeat the workflow on sector ETFs through the library implementation.
The repeated material is intentionally kept short. Mean-variance and portfolio backtesting were already covered in Project 2, Black-Litterman and learned confidence in Project 6, robust/tail portfolio models in Project 10, macro financial conditions in Project 12, regime classification in Project 16, and neural forecasting plus Kelly sizing in Project 19. We use those pieces as inputs here. The new subject is reinforcement learning as sequential portfolio control.
For the RL theory, the main backbone follows the standard MDP/value-function/policy-gradient framing from Sutton and Barto’s reinforcement learning text, plus the specific modern algorithms behind DQN, PPO, GAE, and SAC: Sutton & Barto, DQN, GAE, PPO, and SAC. We don’t turn the notebook into a literature review, but those are the concepts the implementation is built on.
A good way to read this project is to treat it as the final step in the portfolio-learning chain. In the earlier ML notebooks, we mostly asked models to predict something: a regime label, a return rank, a volatility state, or a forward alpha. In this project, we ask a more difficult question: after seeing all those signals, what should the portfolio actually do now?
That change sounds small, but it changes the learning problem completely. A forecast model is usually trained with a fixed target. The target could be next-month return, next-month rank, or whether the next period is risk-on. The model predicts the target and we evaluate prediction error. A portfolio policy has a different job. It has to pick an action that changes wealth through time. The quality of that action depends on future return, turnover, drawdown, volatility, concentration, cash use, and the next opportunity set. This is why RL is a natural final project: it lets us write the portfolio decision as a repeated control problem instead of one isolated prediction problem.
We should still be careful with this. Finance is a difficult RL domain because the market doesn’t give us millions of independent training episodes. The agent sees one noisy historical path, and every apparent edge can be a sample artifact. So the goal here isn’t to claim that RL automatically beats everything. The goal is to build a serious, transparent RL allocation framework and understand exactly what the policies learn, what they fail to learn, and how they compare with the more interpretable rule-based models from the previous projects.
Because this is the final project, the markdown also has to do more teaching than a normal research note. The reader should understand the path from the simplest RL objects to the implemented models. So the explanations don’t jump straight from “agent” to PPO or SAC. They build the bridge: state, action, reward, MDP, value, Q-value, advantage, policy gradient, actor-critic, PPO, recurrent PPO, and SAC. The portfolio details are then layered on top of that RL course path.
This is also why the markdown spends less space on repeated data and baseline explanations. The repeated pieces are only there to make the RL setup grounded. The main intellectual work is understanding how a neural policy learns sequential allocation under realistic portfolio frictions.
So this markdown treats RL as a decision framework first and a neural-network technique second. That order is important for understanding the project. Clearly.
1) Reused Inputs and the Portfolio-Control Setup
We start by importing the project dependencies, setting the random seeds, choosing the compute device, and defining the cross-asset universe. This first cell is mostly setup, so the important part is the shape of the decision problem rather than the imports.
The tradable risky assets are:
Equities: SPY, QQQ, IWM, EFA, EEM
Real assets / inflation-sensitive assets: VNQ, DBC, GLD
Rates and credit: IEF, TLT, LQD, HYG
Cash-like defensive asset: SHY
The signal-only assets AGG and UUP still help describe the market state, but they aren’t part of the risky action vector. The agent receives features from these markets, then decides how much risk to take and how to distribute it across the portfolio.
The main data sources are repeated from earlier notebooks, so we only need the reproducibility links:
The first printed output reports the compute device. Here the run used cuda, so the neural-network training was done on GPU. That matters because PPO, recurrent PPO, and SAC all repeatedly collect rollouts and update neural networks. A CPU version would still work, but the training loop would be slower.
The portfolio universe also makes the RL task meaningful. If the universe only contained highly correlated equity ETFs, the action space would mostly be sector rotation. Here the agent can rotate between broad equity, growth, small caps, international equity, emerging markets, real estate, commodities, gold, duration, credit, high yield, and cash. That means one action can represent several economic choices at the same time:
risk exposure: high risky allocation versus more SHY,
growth/cyclical tilt: QQQ and SPY versus IWM, EEM, HYG, and DBC,
inflation hedge: GLD and DBC versus duration,
rate sensitivity: TLT and IEF versus SHY,
credit beta: HYG and LQD versus Treasury exposure,
international risk: EFA and EEM versus US assets.
The RL policy therefore isn’t just choosing tickers. It is choosing a macro allocation shape. This is why the state contains both asset-level signals and global context. A high forecast for HYG means something different when NFCI is loose, VIX is falling, breadth is strong, and SPY drawdown is shallow. The same HYG forecast is more dangerous when financial conditions are tightening and equity breadth is breaking.
The weekly decision frequency is also a design choice. Daily RL allocation would create too much noise and transaction cost. Monthly allocation may be too slow to react when the state changes sharply. Weekly decisions are a middle ground: the agent can adapt more often than the monthly optimizer, but the holding period is long enough that each action has a real portfolio consequence.
Show code
panels = load_yfinance_panel( data_path, fields=("close", "volume"), tickers=all_tickers, source="yfinance_export", start="2007-01-01",)close_all = panels["close"].sort_index().ffill(limit=3)volume_all = panels["volume"].sort_index().ffill(limit=3).reindex(close_all.index)first_valid = close_all[assets + [cash_ticker]].dropna(how="any").index.min()close = close_all.loc[first_valid:, [c for c in all_tickers if c in close_all.columns]].ffill(limit=3)volume = volume_all.reindex(index=close.index, columns=close.columns).ffill(limit=3)r_d = prices_to_returns_panel(close, kind="simple").replace([np.inf, -np.inf], np.nan).fillna(0.0)coverage = pd.DataFrame({"first_valid": close[assets + [cash_ticker]].apply(pd.Series.first_valid_index),"last_valid": close[assets + [cash_ticker]].apply(pd.Series.last_valid_index),"observations": close[assets + [cash_ticker]].notna().sum(),"coverage": close[assets + [cash_ticker]].notna().mean(),})display(coverage)print(close.index.min().date(), "to", close.index.max().date(), close.shape)
The loaded cross-asset panel runs from 2007-04-11 to 2026-06-17 with 4,828 daily observations after alignment. All 13 investable assets in the main environment have full coverage over this aligned period. That is important for RL because missing data would break the state-action-transition sequence. A supervised forecasting model can sometimes drop a row or impute a missing feature; an RL environment needs every step to have a valid next state, action, reward, and next-period return.
The asset summary is also useful. QQQ has the strongest annualized return in the full panel, around 17.9%, with annualized volatility around 22.3%. SPY is lower return but also slightly lower volatility. IWM, EEM, and VNQ carry larger volatility. GLD has a strong standalone return with lower volatility than most equity ETFs. DBC has the weakest return in this window, which reminds us that commodities can be useful in specific regimes without being attractive as a permanent high-weight allocation.
We rebalance the RL policy weekly, using Friday decisions. The output reports 950 weekly decision points, starting in April 2008 and ending in June 2026. Monthly optimizer dates are also created because the earlier portfolio models, like forecast-gated MaxSharpe and Kelly, use monthly timing. RL doesn’t have to follow the monthly optimizer schedule; it can learn at a more frequent weekly control horizon while still being compared against slower rule-based strategies.
1.1 Forecasts, regimes, and portfolio priors as inputs
The next long cell builds the forecasting and rule-based inputs that the RL state will later consume. We don’t need to re-teach the forecasting models from Project 19 or the regime classifier from Project 16. The important point is how we reuse them.
The project builds a forecast table with:
forward 21-day alpha targets, measured relative to the cross-sectional median,
volatility-scaled return labels, so assets with different risk levels can be compared more fairly,
tree-based forecasts from a RandomForest-style model,
TCN rank forecasts from the sequence model developed earlier,
confidence and rank features, so the RL policy can see not only the direction of a forecast but also how strong and stable it appears,
regime probabilities, FCI variables, VIX features, and cross-asset conditions.
The validation grid for lambda_alpha is used to choose how aggressively the forecast-gated optimizer should use the forecast signal. In the output, the best validation Sharpe is reached at lambda_alpha = 1.5, which is the most forecast-sensitive value in the grid. That doesn’t mean the forecast is always strong. It means that, over the validation period used in this notebook, the forecast-gated MaxSharpe model benefited from allowing a larger forecast tilt.
The displayed weight shapes show that we now have four baseline weight frames ready:
Equal Weight: 217 monthly rows and 12 risky assets,
Forecast-Gated MaxSharpe: 217 rows and 13 columns including SHY,
RandomForest Kelly: 217 rows and 13 columns including SHY,
ML Regime-Aware: 201 rows and 12 assets.
These aren’t the new RL models. They are the reference policies. We use them in two ways: first as benchmarks for performance, and second as state information so the RL policy can see what the earlier models would have done.
The forecast construction also gives us a useful example of how a supervised model becomes an RL feature. A forecast on its own is just a number:
Here \(\hat{\alpha}_{i,t}\) is the expected relative alpha for asset \(i\) over the next horizon. It tells us whether the model thinks the asset should beat the cross-sectional median. In a supervised notebook, we might rank assets by this value and buy the top names. In the RL notebook, we don’t force that rule. We pass the forecast into the state and let the policy learn how much to trust it.
This distinction matters. A forecast can be directionally useful but still too noisy to size directly. A high forecast for EEM might deserve a small tilt in a volatile environment and a larger tilt in a low-volatility, dollar-weak environment. The RL policy can learn these interactions because it sees the forecast together with volatility, regime, previous portfolio, and cost information.
The output showing tree features and neural-network features gives us the feature scale of the problem. We aren’t giving the RL agent raw prices only. We are giving it a research stack built from:
growth, inflation, financial conditions, stress state
Regime features
probability of risk-on, neutral, or defensive market states
Prior weight features
what earlier portfolio models would allocate
Portfolio memory
previous weights, turnover context, cash level
This is one of the best parts of the project. RL doesn’t replace the earlier work; it sits on top of it. The earlier models create structured information, and the policy learns how to turn that information into sequential allocation decisions.
1.2 Forecast features as policy context
The forecast features deserve one more clarification because they are easy to misuse in an RL setting. A forecast feature shouldn’t be treated as an order. If a model ranks QQQ highly, the policy doesn’t have to buy QQQ heavily. The policy sees that rank together with everything else. The forecast is one input to the decision, not the decision itself.
The first mapping predicts. The second mapping allocates. The second one has a harder job because it must consider cost, risk, and path-dependence. That is why a weaker raw forecast can still be useful if the policy learns when to trust it. It is also why a strong raw forecast can be harmful if the policy sizes it too aggressively.
The notebook gives the RL policy access to several prior portfolio models. This is a useful design because previous model outputs contain compressed information. A forecast-gated optimizer weight tells the agent how a disciplined optimizer reacts to the forecast. A Kelly weight tells the agent how growth-optimal sizing reacts. A regime-aware allocation tells the agent how the macro classifier changes exposure. The RL policy can learn to follow, fade, smooth, or ignore those priors.
This is closer to how a human portfolio team might work. A portfolio manager doesn’t look only at one forecast. They look at signals, risk reports, prior allocations, costs, and current exposures, then decide what change is actually worth making.
The benchmark NAV plot confirms that the rule-based baselines have very different behavior. Forecast-Gated MaxSharpe and RandomForest Kelly are more active and more concentrated than equal weight, so they have larger deviations through time. Equal weight is smoother but less informed. The regime-aware strategy tends to change more around stress and recovery periods because it is responding to market-state probabilities.
This matters for RL because we aren’t training the agent in an empty world. The agent receives signals from a full research stack: forecasts, regimes, macro conditions, VIX, and previous portfolio state. The hard part is deciding whether those signals should translate into more risky exposure, more cash, higher concentration, lower turnover, or a different cross-asset allocation.
2) The Agent-Environment Loop
Reinforcement learning starts with an interaction loop. At each decision time, the agent observes a state, chooses an action, receives a reward, and moves to the next state.
\(S_t\) is the market and portfolio state at the weekly decision date,
\(A_t\) is the raw action produced by the policy network,
\(w_t\) is the valid portfolio weight vector after the raw action is mapped into constraints,
\(R_{t+1}\) is the reward earned over the next holding interval,
\(S_{t+1}\) is the updated market and portfolio state at the next decision date.
Show code
from quantfinlab.plotting.diagrams import agent_environment_loop_diagramfig, ax = agent_environment_loop_diagram()plt.show()
The first diagram shows this loop directly. The key financial difference from a normal supervised model is that the action changes the future portfolio path. A forecast model can be wrong and still leave the portfolio construction rule to handle the error. An RL agent owns the entire decision: if it chooses a high-turnover, high-concentration, high-risk allocation right before a drawdown, the reward should punish that decision.
A simple example makes the loop concrete. Suppose the state says credit is strong, equity breadth is high, volatility is low, and the previous portfolio is underweight risky assets. The policy may choose a high risky exposure and tilt toward SPY, QQQ, and HYG. The environment then applies that allocation over the next week, subtracts turnover costs, updates drawdown and realized volatility, and gives the agent a reward. If the decision raised return without creating too much risk, the reward improves. If it increased losses, turnover, drawdown, or concentration, the reward gets worse.
2.1 Supervised prediction and reinforcement learning in this project
Supervised learning and reinforcement learning look similar from far away because both train neural networks. The training signal is very different.
In supervised forecasting, we usually have pairs \((x_t, y_t)\) and minimize a prediction loss:
The model sees the input \(x_t\), predicts \(y_t\), and gets judged against the realized label. If the label is next-month return rank, the training target is fixed before the model acts. The model doesn’t change the market path by making a prediction.
In reinforcement learning, the training object is a policy:
\[
\pi_\theta(a_t \mid s_t)
\]
This means the model doesn’t only predict a label. It chooses an action \(a_t\) from state \(s_t\). In this notebook, the action becomes the portfolio weights. The policy is judged by the reward sequence generated after its actions:
The expectation is written under \(\pi_\theta\) because the policy itself influences the path of portfolio weights, costs, drawdowns, and rewards. In a market backtest, asset returns are historical and fixed, but the portfolio path isn’t fixed. The policy decides how much of each return stream it actually experiences.
This gives us the central RL difficulty: credit assignment. If a policy loses money in week 30, which earlier choices caused the problem? Was it the current allocation? A previous high-risk allocation that created drawdown pressure? A turnover decision that reduced wealth? A concentration choice that made the portfolio fragile? RL algorithms exist because this credit-assignment problem is hard.
In this notebook we use three modern continuous-control policy families:
PPO: a stable on-policy actor-critic method that updates cautiously.
Recurrent PPO: the same basic PPO idea, but with memory through an LSTM.
SAC: an off-policy actor-critic method that adds entropy to encourage exploration and robust stochastic policies.
The rest of the notebook builds the machinery needed to understand these three models from the ground up.
The important habit while reading the rest of the notebook is to always connect the algorithmic object back to portfolio behavior. A value function is about future reward. A policy is about allocation. Entropy is about exploration. A critic is about judging whether an allocation was better than expected. Keeping that mapping clear makes the RL math much easier to follow.
The agent-environment diagram should be read from left to right. The agent never sees the future return before it acts. It only sees the current state and the previous portfolio context. The environment receives the action, converts it into realized portfolio behavior, and returns the next reward.
This is also why RL can easily overfit in finance. The sample is one historical market path, not millions of independent games. If the agent memorizes that one path, it may look good during training and fail out of sample. That is why this project uses a clear train/validation/test split, heavy constraints, risk penalties, and comparisons against simple baselines.
2.2 Exploration, exploitation, and portfolio risk
The exploration/exploitation tradeoff is one of the first RL ideas that feels very different in finance. Exploration means trying actions that may not look best right now so the agent can learn whether they work. Exploitation means using the action that currently looks best.
In a game environment, exploration might mean trying a weird move to discover a hidden reward. In a live portfolio, exploration has real cost. Trying a high-HYG/high-IWM/high-EEM allocation in a stress environment can lose money. So we need exploration during training, but we also need strict constraints and risk-aware rewards so the agent doesn’t learn reckless behavior.
The policies in this notebook explore through stochastic actions. The actor outputs a Normal distribution over raw actions:
\[
a_t \sim \mathcal{N}(\mu_t, \sigma_t^2)
\]
The mean \(\mu_t\) is the policy’s preferred action. The standard deviation \(\sigma_t\) controls how much it explores around that action. A high \(\sigma_t\) means the policy samples many different allocations. A low \(\sigma_t\) means the policy becomes more deterministic.
PPO uses entropy regularization to avoid collapsing too early. SAC goes further and makes entropy part of the main objective. This is useful because portfolio reward is noisy. A policy may look good after one lucky training path, but a more stochastic policy can avoid becoming too committed to one historical pattern.
There is also a financial version of exploration that appears through the action constraints. Since the policy can never go below 55% risky exposure or above 35% in one risky asset, its exploration is bounded. The agent can explore different allocation shapes, but it can’t create absurd leverage or all-in positions. This is a strong design choice. It makes the training problem safer and the resulting policies more realistic.
2.3 Markov decision process notation
A fully observed reinforcement-learning problem is usually written as a Markov decision process:
\(\mathcal{S}\) is the set of possible states. In our case, a state contains asset features, global market features, and portfolio features.
\(\mathcal{A}\) is the action space. Here the action becomes a portfolio allocation.
\(P(s' \mid s,a)\) is the transition rule. It describes the probability of moving to next state \(s'\) after taking action \(a\) in state \(s\).
\(r(s,a,s')\) is the reward function. Here it is built from portfolio return, differential Sharpe, costs, volatility, drawdown, and concentration penalties.
\(\gamma\) is the discount factor. It controls how much the agent cares about future rewards relative to immediate rewards.
The Markov part means that the current state is supposed to contain enough information to model the next step:
This formula says that once we know the current state and action, the older history shouldn’t add extra information. In markets this assumption is only approximate. If the state vector includes momentum, volatility, drawdown, forecast features, regimes, previous weights, and recent portfolio behavior, we are trying to compress the useful part of the history into \(S_t\).
A useful way to understand the MDP components in this project is to map each symbol to something in the portfolio code.
MDP object
Portfolio meaning
Concrete example in this notebook
\(S_t\)
observed state
asset tensor, global tensor, prior weights, previous weights
\(A_t\)
raw policy action
risky scores plus exposure score
\(w_t\)
valid portfolio weights
constrained long-only weights after action mapping
\(R_{t+1}\)
reward
DSR-based reward minus penalties
\(P\)
transition law
historical next state after the decision date
\(\gamma\)
future-reward weight
SAC uses \(\gamma=0.97\)
The transition law is the least controllable object. In a game simulator we can sample many possible futures. In finance, the notebook has one realized historical path. That is why the policy must be conservative. It can’t assume it has seen every possible crisis, every possible rate shock, or every possible liquidity event.
The discount factor \(\gamma\) deserves a clear interpretation. If \(\gamma=0\), the agent only cares about immediate reward. If \(\gamma\) is close to 1, it cares about longer-term consequences. With weekly decisions and \(\gamma=0.97\), a reward one week ahead receives full weight, a reward ten weeks ahead receives about:
\[
0.97^{10} \approx 0.74
\]
So the agent still cares about what happens several months later. This is important because drawdown and turnover are path-dependent. A policy that grabs a tiny short-term reward by taking unstable risk can damage future reward through drawdown penalties and a worse wealth path.
The Markov assumption is an approximation, but the state design tries to make it less unrealistic. We add rolling windows, forecast ranks, regime probabilities, previous weights, volatility, drawdown, and macro indicators so the current observation carries information about recent history. The recurrent policy later goes one step further by learning an internal memory state.
Show code
from quantfinlab.plotting.diagrams import mdp_diagramfig, ax = mdp_diagram()plt.show()
The MDP diagram shows how state, action, reward, and transition connect. In a textbook gridworld, the state might be the square where the agent stands, the action might be up/down/left/right, and the reward might be +1 for reaching the goal. In this project, the state is a high-dimensional financial vector, the action is a constrained weight vector, and the reward is a risk-adjusted portfolio outcome.
That difference changes the modeling challenge. The transition law \(P\) isn’t known. We don’t know the true probability distribution of future market states. So we use model-free RL: the agent learns from sampled historical transitions rather than estimating a full economic transition model.
2.4 Partial observability in finance
The POMDP diagram is important because finance is rarely fully observable. We don’t see the true economic state. We observe noisy proxies:
prices and returns,
volatility and drawdown,
macro variables,
FCI and NFCI features,
VIX states,
regime probabilities,
forecast signals from the previous ML project,
previous portfolio weights and turnover.
The hidden state might be something like true growth, liquidity, inflation risk, policy reaction function, investor leverage, or market fragility. We don’t observe those directly. We observe indicators that may be correlated with them.
A partially observed problem can be written as:
\[
O_t \sim Z(\cdot \mid S_t)
\]
Here \(S_t\) is the latent state and \(O_t\) is the observation. The policy doesn’t receive \(S_t\) directly; it receives \(O_t\). In the code, the state vector is really an observation vector. We still call it state because it is the input to the policy, but conceptually it is a compressed observation of the market.
This is one reason recurrent policies can help. If the current observation doesn’t contain the full state, a recurrent network can use previous observations to build an internal memory:
\[
h_t = f_\theta(h_{t-1}, O_t)
\]
The hidden memory \(h_t\) can carry information about how the market got here, not only where it is now.
Partial observability is especially important for portfolio allocation because many of the variables that drive markets are latent. We never directly observe “risk appetite” or “liquidity preference” as clean numeric series. We observe proxies. A positive HYG-LQD signal might suggest credit risk appetite. A negative NFCI value might suggest loose financial conditions. A falling VIX might suggest less demand for crash insurance. None of these is perfect.
This means the policy shouldn’t treat one feature as truth. It has to combine imperfect signals. For example:
high equity breadth and falling VIX point toward risk-on,
rising dollar and weak EEM point against emerging markets,
positive commodity momentum and weak duration point toward an inflation-sensitive environment,
high realized correlation means diversification is weakening.
A non-recurrent policy receives this compressed state and acts immediately. A recurrent policy receives a sequence of states and can learn persistence. If financial conditions have been tightening for eight weeks, the recurrent state can carry that information even if the current observation alone looks neutral.
The hidden state in a recurrent policy is not an economic label. It is a learned vector. But if training works well, the vector can behave like a memory of recent market conditions:
\[
h_t = \text{LSTM}_\theta(h_{t-1}, o_t)
\]
Here \(o_t\) is the current observation, and \(h_t\) is the internal memory after processing it. The policy then chooses an action using \(h_t\) along with the current input. This is why recurrent PPO is a natural second model in the project.
Show code
from quantfinlab.plotting.diagrams import mdp_pomdp_diagramfig, ax = mdp_pomdp_diagram()plt.show()
3) Building the State Tensors
The RL state is split into three parts:
Asset state: a tensor with one row per asset and one column per asset-specific feature.
Global state: a vector of market-wide context variables.
Portfolio state: previous weights and recent portfolio behavior.
The asset-state tensor has shape:
\[
X^{asset}_t \in \mathbb{R}^{N \times F_a}
\]
where \(N\) is the number of risky assets and \(F_a\) is the number of asset features. In the main implementation, the output later shows:
This means we have 949 weekly decision points, 12 risky assets, and 50 asset-level features per asset. Every weekly decision is a small cross-sectional dataset: one row for SPY, one row for QQQ, one row for IWM, and so on.
The global-state tensor has shape:
\[
X^{global}_t \in \mathbb{R}^{F_g}
\]
and the output shows:
\[
X^{global} \in \mathbb{R}^{949 \times 31}
\]
So each decision also has 31 market-wide features. These are not asset-specific; they describe the whole environment.
The next-return matrix has shape:
\[
R^{next} \in \mathbb{R}^{949 \times 13}
\]
The 13 columns are the 12 risky assets plus SHY. This is what the environment uses to compute the realized portfolio return after the policy chooses weights.
The asset tensor structure is one of the cleanest ways to make the policy respect the cross-sectional nature of the problem. Instead of flattening everything into one long vector and asking the model to learn that SPY, QQQ, HYG, and GLD are different assets, we represent the input as a matrix:
\[
X^{asset}_t =
\begin{bmatrix}
\text{features of SPY at } t \\
\text{features of QQQ at } t \\
\vdots \\
\text{features of HYG at } t
\end{bmatrix}
\]
Each row is an asset. Each column is a feature type. This lets the network apply similar transformations across assets while still comparing assets to each other. That is useful for portfolio allocation because many decisions are relative: QQQ versus SPY, HYG versus LQD, EEM versus EFA, GLD versus DBC, and so on.
The global state has a different role:
\[
g_t \in \mathbb{R}^{F_g}
\]
It describes the market environment shared by all assets. A global feature like VIX, NFCI, breadth, or regime probability doesn’t belong to one asset row. It affects the interpretation of every asset feature. A strong QQQ forecast is more attractive in a loose, low-volatility environment than in a tightening, high-correlation environment.
The portfolio state adds information about the agent’s current position:
This matters because the same action has different cost depending on previous weights. If the policy already owns 25% SPY, staying at 25% has almost no turnover cost. Moving from 0% SPY to 25% SPY creates large turnover. RL has to learn this path dependence.
The feature output confirms that the raw asset feature block becomes complete on 2008-04-10. We start the RL decision sequence after that point. The raw feature table has 57,936 rows, which equals daily observations multiplied by assets, and 49 raw features before the later merged forecast features bring the asset-state dimension to 50.
The sample rows show a typical asset-state record for SPY: recent returns over 1, 5, 21, 63, and 126 days, skip-momentum, rolling volatility, group-relative features, trend consistency, residual momentum versus SPY, and TCN forecast features like tcn_alpha, tcn_rank, and tcn_confidence. These features are exactly the sort of variables that a portfolio allocator might use manually: recent strength, risk level, relative ranking, and forecast confidence.
The context-feature table at the end of the sample shows a recent risk-on environment: breadth around 0.75 to 0.83, SPY positive over 63 days, QQQ outperforming SPY, HYG slightly outperforming LQD, and NFCI negative, which means financial conditions are easier than average. The regime probabilities are available on some dates after the model updates. They don’t decide the RL action by themselves, but they are part of the state.
Show code
def align_weight_frame(weights, target_dates, columns, fallback_equal=False): dates = pd.DatetimeIndex(target_dates).sort_values().unique() W = pd.DataFrame(weights).copy()if W.empty: base = pd.DataFrame(0.0, index=dates, columns=columns)if fallback_equal: base[assets] =1.0/len(assets)return base W.index = pd.to_datetime(W.index) W = W.sort_index().reindex(columns=columns) W = W.reindex(W.index.union(dates)).sort_index().ffill().reindex(dates).fillna(0.0) zero = W.sum(axis=1).abs() <=1e-12ifbool(zero.any()) and fallback_equal: W.loc[zero, assets] =1.0/len(assets) row_sum = W.sum(axis=1).replace(0.0, np.nan)return W.div(row_sum, axis=0).fillna(0.0)def period_daily_windows(returns, decision_dates, columns): R = returns.reindex(columns=columns).fillna(0.0) idx = pd.DatetimeIndex(R.index) rows, windows = [], []for i, dt inenumerate(decision_dates): start = idx.searchsorted(pd.Timestamp(dt), side="right")if start >=len(idx):continue end = idx.searchsorted(pd.Timestamp(decision_dates[i +1]), side="right") if i +1<len(decision_dates) elsemin(start +5, len(idx))if end <= start:continue window = R.iloc[start:end].replace([np.inf, -np.inf], np.nan).fillna(0.0) rows.append(((1.0+ window).prod() -1.0).rename(pd.Timestamp(dt))) windows.append(window)return pd.DataFrame(rows).reindex(columns=columns).fillna(0.0), windowsstate_dates = pd.DatetimeIndex([d for d in decision_dates if pd.Timestamp(d) >= state_ready_start])state_columns = assets + [cash_ticker]returns_next, daily_windows = period_daily_windows(r_d, state_dates, state_columns)state_dates = returns_next.indexdaily_windows_np = [w.reindex(columns=state_columns).fillna(0.0).to_numpy(dtype=float) for w in daily_windows]
Show code
feature_cols = [ c for c in asset_x.columnsif c notin {"date", "asset"} and pd.api.types.is_numeric_dtype(asset_x[c])]asset_blocks = []asset_feature_names = []for col in feature_cols: piv = asset_x.pivot_table(index="date", columns="asset", values=col, aggfunc="last") piv = piv.reindex(piv.index.union(state_dates)).sort_index().ffill().reindex(state_dates) piv = piv.reindex(columns=assets) fill_by_asset = asset_feature_fill_by_asset[col].reindex(assets) if col in asset_feature_fill_by_asset.columns else pd.Series(np.nan, index=assets) fill_by_asset = fill_by_asset.fillna(float(asset_feature_fill_values.get(col, 0.0))) asset_blocks.append(piv.fillna(fill_by_asset).fillna(0.0)) asset_feature_names.append(col)asset_blocks.append(pd.DataFrame(0.0, index=state_dates, columns=assets))asset_feature_names.append("previous_weight")asset_state = np.stack([b.to_numpy(dtype=np.float32) for b in asset_blocks], axis=2)global_state_frame = global_x.reindex(global_x.index.union(state_dates)).sort_index().ffill().reindex(state_dates)global_state_frame = global_state_frame.apply(pd.to_numeric, errors="coerce").replace([np.inf, -np.inf], np.nan).fillna(0.0)global_state = global_state_frame.to_numpy(dtype=np.float32)shape_table = pd.DataFrame({"shape": [ asset_state.shape, global_state.shape, returns_next.shape, ]}, index=["asset_state", "global_state", "returns_next"])display(shape_table)
So at each of 949 weekly decision points, the policy sees 12 risky assets, 50 asset-level features, 31 global features, and next-period returns for 13 investable columns including SHY. These shapes are the practical version of the theoretical state/action/reward setup.
3.1 The leakage check
The state construction must avoid target leakage. We don’t want future labels like z_21 or y_alpha to sneak into the RL state. Those labels were used in Project 19 for forecasting, but here the RL policy should only see forecasts and features available at the decision time.
max absolute values that are finite and controlled after feature cleaning.
This is a small output, but it is one of the most important engineering checks. If leakage enters the state, the RL policy can appear to learn a brilliant allocation rule while really reading the answer key. The diagnostic says the state table is complete and clean for the main environment.
Leakage control is more important in this project than in a simple regression because RL can exploit tiny mistakes brutally. If a future-return label accidentally enters the state, the agent doesn’t need to understand finance; it can learn a direct shortcut. That would produce a beautiful backtest and a useless policy.
The leakage check verifies that the state doesn’t include target columns such as next-period realized return, future alpha, or future rank. The output shows missing share equal to zero and bounded feature magnitudes. This means two things:
The state tensors are complete, so the environment won’t be forced to handle missing rows during training.
The standardized features don’t contain extreme numerical explosions that would destabilize neural-network training.
The maximum absolute feature values are also useful. Asset features reach about 7.32 in standardized units, while global features reach about 3.29. That is large but still manageable. If the model saw values in the hundreds or thousands, gradient updates could become unstable, especially in PPO and SAC where policy log-probabilities and value losses interact.
For a finance ML notebook, this small diagnostic is one of the most important quality checks. It protects the credibility of everything that follows.
4) Continuous Actions as Portfolio Weights
In many RL examples the action space is discrete. An Atari agent chooses a joystick action. A gridworld agent chooses up, down, left, or right. A portfolio agent has a different problem: it needs to choose a vector of weights.
The first \(N\) elements are risky-asset logits. The last element controls total risky exposure. The code maps this raw action into valid weights through two transformations.
First, the exposure logit becomes a risky-exposure level using a sigmoid:
where \(E_{min}=0.55\) and \(E_{max}=1.00\). The sigmoid \(\sigma(e_t)\) maps any real number into \((0,1)\), so the model can output unconstrained neural-network values while the final exposure remains bounded. If \(e_t=0\), then \(\sigma(0)=0.5\) and the exposure becomes halfway between 55% and 100%:
\[
E_t = 0.55 + 0.45 \times 0.5 = 0.775
\]
Second, the risky logits become cross-sectional risky weights through a softmax:
This is a good action design because the neural network doesn’t have to learn the fully invested constraint from scratch. The transformation guarantees nonnegative weights and a controlled risky exposure. After that, the code caps each risky asset at 35% and redistributes any excess weight to assets with remaining room. Cash is the leftover:
So the final action is always a valid long-only portfolio.
4.1 From raw neural-network output to valid weights
The policy network doesn’t directly output final portfolio weights. It outputs an unconstrained action vector. That raw action must be translated into weights that obey the portfolio rules.
This creates a long-only allocation across risky assets. Every \(\tilde{w}_{i,t}\) is positive, and the risky weights sum to 1. The exposure score is passed through a sigmoid and scaled into a range:
where \(E_{min}=0.55\) and \(E_{max}=1.00\). This means the policy is always at least 55% invested in risky assets and can go up to 100%. SHY receives the leftover cash-like weight:
\[
w_{SHY,t} = 1 - E_t
\]
Then the risky weights become:
\[
w_{i,t} = E_t \tilde{w}_{i,t}
\]
A cap is applied so no single risky asset can dominate the portfolio:
\[
w_{i,t} \le 0.35
\]
If an asset hits the cap, the excess weight is redistributed across the uncapped assets. This keeps the policy from solving the problem by putting everything into one high-forecast asset.
A small example helps. If the network gives equal scores to all 12 risky assets and chooses the neutral exposure, the output shows about 6.46% in each risky asset and 22.5% in SHY. The policy starts from a diversified risky sleeve plus defensive cash-like ballast. Training then teaches it how to move away from that neutral shape.
This mapping is one of the most important implementation choices in the notebook. Instead of asking RL to learn portfolio constraints from penalties, we build the constraints into the action transformation. That makes training easier and keeps every sampled action investable.
The simplex constraint is strict. A policy can’t output weights that sum to 1.08 or -3% in an asset unless shorting/leverage is allowed. This project is long-only and fully invested, so every final action must land inside the simplex.
The raw action lives in unconstrained Euclidean space:
\[
a_t\in\mathbb{R}^{N+1}
\]
The action mapping is therefore a projection-like transformation from unconstrained scores to feasible portfolio weights. The softmax handles relative risky weights, the sigmoid handles total risky exposure, and the cap redistribution handles maximum position size.
This separation is cleaner than asking the neural network to learn feasibility by itself. If the policy output final weights directly, we would need penalties for invalid weights or a complicated constrained distribution. Here every sampled action is valid after transformation. That lets the RL algorithm focus on learning which valid portfolio shapes produce better reward.
The max-weight cap is also important for exploration. Without it, an early lucky sequence could teach the policy that one asset deserves extreme allocation. The cap forces the model to express conviction through a diversified sleeve. For example, if SAC likes risk, it can overweight SPY, IWM, QQQ, and HYG, but it can’t put 80% in one ticker.
This is a practical finance choice, not only a machine-learning choice. Real allocators usually care about concentration, liquidity, and mandate limits. Building those into the environment makes the learned policy more credible.
The neutral action output helps verify the transformation. When the raw action is all zeros, the model doesn’t choose an equal 1/13 allocation. It chooses about 6.46% in each risky asset and 22.50% in SHY.
That comes directly from the exposure formula. A zero exposure logit gives 77.5% risky exposure. Split equally over 12 risky assets, that is:
\[
\frac{0.775}{12} \approx 0.0646
\]
and the remaining 22.5% goes to cash-like SHY. This is a very useful sanity check because it tells us the default policy is neither all-cash nor fully risky. It begins from a balanced point and then learns to tilt.
5) Transaction Costs, Returns, and Portfolio Path Updates
The environment has to translate a weekly weight decision into a realized portfolio path. The realized return over the holding interval is computed from the asset returns and the chosen weights. If the action changes the portfolio from \(w_{t-1}\) to \(w_t\), the turnover is approximately:
The factor \(1/2\) avoids double-counting buys and sells. If we sell 10% of one asset and buy 10% of another, the absolute changes sum to 20%, but the traded portfolio notional is 10%.
With 10 bps cost, turnover of 0.50 costs 0.0005, or 5 basis points of portfolio value. The printed check returns exactly (0.01956, 0.5, 0.0005), meaning the test portfolio produced about 1.96% gross net step return with 50% turnover and 5 bps cost.
This is important because RL can otherwise learn unrealistic churning. A policy that changes weights aggressively every week may look good before costs and much weaker after costs. The environment makes turnover part of both the realized return and the reward penalty.
The transaction-cost formula is simple but important:
Here \(w^{pre}_{i,t}\) is the portfolio weight just before rebalancing, after market moves have changed the previous weights. The factor \(1/2\) appears because buying one asset and selling another creates two absolute weight changes for one dollar of portfolio rotation. If we sell 10% of SPY and buy 10% of GLD, the absolute changes sum to 20%, but the actual portfolio turnover is 10%.
With 10 bps cost, a turnover of 0.50 costs 0.0005, or 5 basis points of portfolio value. The sanity output (0.01956, 0.5, 0.0005) confirms exactly this logic: the example action produces a portfolio path return around 1.96%, turnover of 50%, and cost of 0.05%.
This cost term is a major reason RL policies shouldn’t chase every small forecast wiggle. A forecast must be strong enough to overcome the cost of changing weights. If the policy learns to trade too much, the reward should punish it through the cost penalty and through lower net portfolio returns.
6) Reward Design with Differential Sharpe
Reward design is where portfolio RL becomes finance-specific. If we reward only next-period return, the agent can learn to maximize upside by taking too much risk. If we reward only low volatility, it may hide in cash. The reward has to push the agent toward a usable investment policy.
The primary reward in this project is based on the differential Sharpe ratio (DSR). The idea is to approximate how a new return observation changes the running Sharpe ratio. We maintain exponentially updated moments:
\[
A_t = A_{t-1} + \eta(r_t - A_{t-1})
\]
\[
B_t = B_{t-1} + \eta(r_t^2 - B_{t-1})
\]
Here \(A_t\) is a running mean estimate, \(B_t\) is a running second-moment estimate, and \(\eta\) controls how quickly the estimates update. The variance estimate is:
\[
\sigma_t^2 \approx B_t - A_t^2
\]
A normal Sharpe ratio is roughly:
\[
SR_t = \frac{A_t}{\sqrt{B_t - A_t^2}}
\]
The differential Sharpe asks how the current return changes this running ratio. The code computes:
The numerator has two parts. The first part rewards returns above the current average. The second part penalizes returns that increase the second moment too much. So a positive return with modest volatility can have a positive DSR, while a large noisy return can be less attractive than its raw return suggests.
The reward doesn’t stop there. The full reward is:
Each penalty targets a specific portfolio failure mode:
\(P^{dd}_t\) penalizes drawdowns below the -12% floor,
\(P^{vol}_t\) penalizes realized volatility above the 18% target,
\(P^{cost}_t\) penalizes turnover cost,
\(P^{conc}_t\) penalizes excessive concentration using HHI,
\(P^{cash}_t\) penalizes cash above 20%,
\(P^{corr}_t\) penalizes policies that are too correlated with equal weight.
This reward is deliberately opinionated. We are not asking the agent to maximize raw wealth at all costs. We are training it to be a cost-aware, drawdown-aware, volatility-aware portfolio allocator.
6.1 Differential Sharpe from running moments
The reward is based on differential Sharpe ratio (DSR), which is useful when an agent receives rewards step by step. A normal Sharpe ratio is computed after a full return series:
\[
SR = \frac{\bar{r}}{\sqrt{\overline{r^2} - \bar{r}^2}}
\]
This is not directly convenient as a per-step reward because the Sharpe ratio depends on the whole history. DSR turns the idea into an incremental signal. We maintain running estimates of the first and second moments:
\[
A_t = A_{t-1} + \eta(r_t - A_{t-1})
\]
\[
B_t = B_{t-1} + \eta(r_t^2 - B_{t-1})
\]
Here:
\(A_t\) is the running mean return estimate,
\(B_t\) is the running second-moment estimate,
\(\eta\) is the update speed,
\(B_t - A_t^2\) is the running variance estimate.
The DSR approximation used in the environment measures how the new return changes the Sharpe estimate:
This formula looks dense, so let’s unpack it. The term \(r_t-A_{t-1}\) asks whether today’s return is above the current running mean. That improves the numerator. The term \(r_t^2-B_{t-1}\) asks whether today’s squared return is above the current running second moment. That increases estimated volatility, which can hurt Sharpe. So DSR rewards returns that improve mean more than they increase variance.
This is a good fit for allocation. A high-return week with huge volatility doesn’t automatically receive a huge reward. A moderate positive return with controlled volatility can be more valuable. That matches the financial objective of building a stable risk-adjusted portfolio rather than maximizing raw return.
6.2 Reward components as portfolio behavior constraints
Each penalty is connected to a real portfolio failure mode:
\(C_t\) penalizes turnover cost.
\(V_t\) penalizes realized volatility above the target.
\(D_t\) penalizes drawdown beyond the floor.
\(H_t\) penalizes concentration through HHI.
\(K_t\) penalizes excessive cash above the cap.
\(M_t\) penalizes portfolios that remain too close to equal-weight behavior when correlation is high.
The concentration penalty uses the Herfindahl-Hirschman index:
\[
HHI_t = \sum_i w_{i,t}^2
\]
If the portfolio is equally spread across 12 risky assets, HHI is low. If the portfolio puts 35% into one asset and large weights into only a few others, HHI rises. The effective number of assets is:
\[
N_{eff,t} = \frac{1}{HHI_t}
\]
This is easier to interpret. An HHI of 0.10 means the portfolio behaves roughly like 10 equally weighted assets. An HHI of 0.25 means it behaves like only 4 equally weighted assets.
The reward-settings table shows the exact values used in the environment. The daily risk-free rate is about 0.000156, corresponding to a 4% annual risk-free assumption. The reward mode is DSR, scaled by 4.0. The drawdown floor is -12%, target volatility is 18%, maximum risky exposure is 100%, minimum risky exposure is 55%, and maximum single risky asset weight is 35%.
The cash penalty matters because the action map allows cash through SHY. Without a cash penalty, a risk-aware reward might encourage the agent to hold too much SHY and avoid learning meaningful allocation. With a cash cap of 20% and a penalty above that, the policy can still de-risk, but it has to pay a reward cost for excessive defensiveness.
The environment sanity check is excellent: the path total return from the environment and the daily backtest total return both equal 6.2521, with an absolute gap of 0.0000. That tells us the RL environment is aligned with the later backtest engine. If this check failed, the agent could train on one reward accounting system and be evaluated on another, which would make the results unreliable.
The reward table for the baselines is very helpful. Equal weight has tiny turnover and no concentration penalty, but it suffers from larger drawdown and volatility penalties. Forecast-Gated MaxSharpe has lower drawdown penalty but more turnover and some concentration. RandomForest Kelly has a large cash penalty, which means it often becomes too defensive relative to the reward design. ML Regime-Aware has strong DSR but still takes drawdown penalty.
This reward decomposition is the teacher signal for the RL models. The agent learns from these tradeoffs. It isn’t being told “buy SPY” or “avoid TLT.” It is being told what kind of portfolio path is valuable.
6.3 Reward shaping without hiding the real objective
Reward shaping can make RL easier to train, but it can also create strange behavior if the shaped reward is misaligned with portfolio goals. The reward here tries to keep the main objective close to risk-adjusted wealth growth while adding penalties that represent real implementation constraints.
A pure return reward would be:
\[
R_t = r^{portfolio}_t
\]
That would push the agent toward high-risk assets because the sample contains long bull-market periods. A pure Sharpe-style reward is better, but it can still ignore costs, concentration, and drawdown path. The DSR reward plus penalties gives a more complete training signal.
Think of each penalty as a guardrail:
Guardrail
What it prevents
Cost penalty
constant reshuffling for tiny forecast changes
Volatility penalty
high-return but unstable portfolios
Drawdown penalty
accepting deep losses for short-term reward
Concentration penalty
hiding all risk in one or two assets
Cash penalty
solving the problem by staying defensive forever
Decorrelation penalty
pretending to be active while staying close to equal-weight under high correlation
The cash penalty is easy to misunderstand, so it’s worth being clear. SHY is a valid defensive asset, and the agent is allowed to hold it. But if the reward didn’t penalize excess cash at all, a defensive policy could avoid volatility and drawdown by hiding in SHY too often. That might produce stable rewards but wouldn’t be a useful allocation model. The cash penalty says: use cash when it helps, but don’t let it become the default answer.
The decorrelation penalty also has a specific purpose. If the policy behaves almost exactly like equal weight while paying turnover costs, it isn’t adding value. During high-correlation periods, many assets move together, so diversification can be weaker than the number of tickers suggests. The decorrelation term encourages the policy to make meaningful allocation choices instead of producing a noisy version of equal weight.
The reward design doesn’t guarantee success. It encodes what we care about. If the reward is too strict, the policy becomes too defensive. If it is too loose, the policy becomes too aggressive. The later results show this tension clearly: recurrent PPO is defensive, SAC is aggressive, and the best rule-based models still win on out-of-sample risk-adjusted performance.
6.4 Reward scale and learning stability
The scale of the reward matters because policy-gradient updates are sensitive to advantage magnitude. If rewards are extremely small, the policy barely receives a learning signal. If rewards are extremely large or unstable, the gradients become noisy and the policy may chase outliers.
This is why the DSR term is multiplied by 4. The raw DSR value is a small incremental statistic, so scaling it makes the reward visible to the optimizer. The penalties are then calibrated to live on a comparable scale. This doesn’t change the economic direction of the reward; it changes how strongly the agent feels that direction during training.
Return quality comes from DSR. Implementation damage comes from costs, drawdown, volatility, concentration, cash, and correlation penalties. If the implementation penalties dominate too much, the agent becomes defensive and static. If they are too weak, the agent becomes aggressive and unstable. The reward settings table is therefore not just a configuration table. It is the investor preference encoded into the RL environment.
The baseline reward decomposition is useful because it tells us whether the reward is sane before training. Equal weight receives almost no turnover or concentration penalty but suffers drawdown and volatility penalties. Forecast-Gated MaxSharpe pays more turnover but controls drawdown better. RandomForest Kelly gets punished for cash usage. These patterns make sense, so the reward is at least directionally aligned with real portfolio behavior.
The baseline reward decomposition is useful before any neural policy appears.
Equal weight has reward close to zero. It has very low turnover and no cash penalty, but it receives larger volatility and drawdown penalties. Forecast-Gated MaxSharpe has lower drawdown and volatility penalties, but higher turnover. RandomForest Kelly has a strong primary reward but pays a large cash penalty because it spends too much time in the defensive asset. ML Regime-Aware has a good primary reward but also drawdown and volatility penalties.
This table shows the reward is not just ranking strategies by CAGR. It is judging behavior. A strategy can earn high returns and still receive a weak reward if it relies on cash too much, trades too aggressively, or creates drawdown risk. That is exactly the kind of reward shaping we need before training PPO or SAC.
This split is stricter than random cross-validation. RL training has to respect time ordering. The policy learns on earlier market environments, gets selected on the validation period, and is evaluated on the later test period. The sample is still small for deep RL. A few hundred weekly steps is tiny compared with robotics or game environments. This is one of the biggest practical limitations of financial RL, and it is the reason the notebook uses constraints, priors, and compact policies instead of an unconstrained huge agent.
7) Value Functions, Bellman Equations, and Q-Learning
Before policy gradients, PPO, and SAC, we need the value-function foundation. The return from time \(t\) is the discounted sum of future rewards:
The discount factor \(\gamma \in [0,1]\) controls the future horizon. If \(\gamma\) is close to 1, the agent cares about long-term reward. If it is small, the agent becomes short-term. In this notebook, SAC uses \(\gamma=0.97\), so rewards several weeks ahead still matter.
A policy \(\pi\) maps states to actions. If the policy is stochastic, \(\pi(a\mid s)\) is the probability density of choosing action \(a\) in state \(s\). The value function is:
\[
V^\pi(s) = \mathbb{E}_\pi[G_t \mid S_t=s]
\]
This means: if we start in state \(s\) and follow policy \(\pi\), what is the expected total future reward? The action-value function is:
This means: if we start in state \(s\), take action \(a\) now, and then follow policy \(\pi\), what is the expected total future reward?
The difference between them gives the advantage:
\[
A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)
\]
The advantage tells us whether action \(a\) was better or worse than the policy’s typical action in state \(s\). In portfolio terms, if the current policy usually holds 70% risky exposure in a risk-on state, and one sampled action increases exposure to 90%, the advantage measures whether that more aggressive decision improved future reward relative to the baseline behavior.
7.1 Return, value, and the meaning of delayed reward
The return in RL is the discounted sum of future rewards:
Each term is a future reward. The discount factor \(\gamma\) controls how quickly those future rewards fade. In this project, rewards are weekly, so \(\gamma=0.97\) still gives meaningful weight to the next several months.
The state-value function is the expected return from a state under policy \(\pi\):
\[
V^\pi(s) = E_\pi[G_t \mid S_t=s]
\]
This is the value of being in state \(s\) if we keep following policy \(\pi\). In portfolio language, a state with strong breadth, low volatility, loose financial conditions, and positive forecasts should generally have higher value than a state with high volatility, negative breadth, and rising drawdown risk, but the value also depends on the policy’s ability to act.
The action-value function is:
\[
Q^\pi(s,a) = E_\pi[G_t \mid S_t=s, A_t=a]
\]
This is the value of taking action \(a\) now and then following policy \(\pi\) afterward. For portfolio allocation, \(Q(s,a)\) answers: if we choose these weights today, and then continue with our policy, what future reward should we expect?
The advantage function compares an action to the average action from the same state:
\[
A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)
\]
If \(A^\pi(s,a)>0\), the action was better than the policy’s typical action in that state. If it is negative, the action was worse. Advantage is the object PPO uses to decide which sampled actions should become more likely and which should become less likely.
A portfolio example makes this clearer. Suppose the state is a high-breadth, low-volatility, positive-forecast environment. A high-risky-exposure action may have positive advantage if it earns return without large penalties. In a high-volatility, high-drawdown state, the same high-exposure action may have negative advantage. The action isn’t good or bad by itself; it is judged relative to the state.
7.2 Monte Carlo targets and temporal-difference targets
There are two broad ways to estimate value. A Monte Carlo target waits until the end of a trajectory and uses realized future rewards:
\[
\hat{V}^{MC}(s_t)=G_t
\]
This is unbiased if the trajectory is sampled from the policy, but it can be very noisy. In finance, one realized path can contain unusual crises, rebounds, and rate shocks. A Monte Carlo return from one path may say more about that specific period than about the true value of the state.
A temporal-difference target bootstraps from the critic:
This is lower variance, but it depends on the critic being accurate. If the critic is wrong, the target is biased.
GAE blends these ideas. It doesn’t rely only on one-step TD, and it doesn’t wait for a full Monte Carlo return. It sums a decaying sequence of TD errors. This is a practical compromise, which is why PPO implementations often use it.
For portfolio RL, this compromise is valuable because rewards are noisy and path-dependent. A weekly return may be positive because the whole market went up, not because the action was smart. A drawdown penalty may appear several weeks after the risky allocation that caused the vulnerability. Multi-step advantage estimation helps the algorithm connect these delayed effects without relying entirely on long-horizon returns.
7.3 Discrete actions and continuous actions
Q-learning and DQN become easier when the action set is finite. For example, a discrete allocation agent might choose from:
Then the model only needs to learn the value of each action in each state. That can work, but it restricts the portfolio. We would have to predefine all possible allocation templates.
This notebook uses continuous control. The action lives in a real-valued space:
\[
a_t\in\mathbb{R}^{N+1}
\]
The model can choose smooth changes in exposure and relative weights. This is more flexible and closer to real portfolio management, but it makes learning harder. The agent has to learn both direction and sizing. Policy-gradient and actor-critic methods are designed for this kind of continuous decision.
The key transition in the RL course flow is therefore:
Bellman equations explain what optimal sequential value means.
Q-learning shows how value can be learned from temporal differences.
DQN shows how neural networks approximate value functions.
Policy gradients show how to optimize continuous stochastic policies directly.
Actor-critic combines policy learning with value estimation.
PPO and SAC are modern, stable versions of actor-critic learning.
7.4 The full model bridge from DQN to modern actor-critic
It helps to see the algorithm family as one continuous path instead of disconnected model names.
Step 1: value learning. We start with \(V(s)\) and \(Q(s,a)\). These functions answer how good a state or state-action pair is. The learning signal is Bellman consistency: today’s value should match today’s reward plus discounted next value.
Step 2: Q-learning. If actions are discrete, we can learn \(Q(s,a)\) and choose the action with the highest value:
\[
a^*(s)=\arg\max_a Q(s,a)
\]
This is intuitive but works poorly when the action is a high-dimensional continuous portfolio vector.
Step 3: DQN. We replace the Q-table with a neural network. The model can generalize across states, so it can handle complex observations. But it still usually assumes a discrete action set. For portfolio allocation, we would need to discretize weights into templates, and that would remove much of the flexibility.
Step 4: policy gradients. Instead of scoring every action, we directly learn a policy distribution. This fits continuous control:
Now the model can sample any raw action vector and map it to a valid portfolio.
Step 5: actor-critic. Policy gradients can be noisy, so we add a critic. The critic estimates value and produces advantages. This makes learning more stable.
Step 6: PPO. We keep actor-critic, but make policy updates conservative through clipping. PPO is popular because it is stable, simple enough to implement, and works across many continuous-control problems.
Step 7: recurrent PPO. We add memory for partial observability. This is useful when the current state doesn’t fully reveal the market regime.
Step 8: SAC. We move to off-policy actor-critic with twin Q-functions, replay buffer, and entropy maximization. This is more sample-efficient and often stronger in continuous-control tasks.
That is the complete path this notebook wants the reader to understand. We don’t use every model in the code, but each idea explains the next one. Q-learning explains value. DQN explains neural value approximation. Policy gradient explains continuous action learning. Actor-critic explains how value and policy work together. PPO and SAC are the implemented modern versions.
7.5 Bellman recursion
The Bellman equation is the recursive identity behind RL. For a fixed policy:
The formula is simple but powerful. The value of being in state \(s\) equals the immediate reward plus the discounted value of the next state. The expectation appears because both the action and the next state can be random.
The max means that after this action, we assume the agent will choose the best possible future action. This is the basis of Q-learning.
In a tiny discrete problem, we can store \(Q(s,a)\) in a table. In portfolio allocation, the state is continuous and high-dimensional, and the action is also continuous. A table is impossible. That is why we use neural networks.
The Bellman equation is the recursive structure behind value learning:
This equation says: the value of the current state equals the immediate reward plus the discounted value of the next state. The expectation appears because the policy may be stochastic and the next state may be uncertain.
This is the same idea with the current action fixed. We take action \(a\), receive reward, move to \(s'\), and then average over future actions from the policy.
In finance, this recursion is the mathematical reason RL can account for path effects. A one-week reward may look good, but if it leaves the next state in a bad position, the future value term should reduce the action’s attractiveness. For example, a policy could increase return by going almost fully into high beta assets. If this creates a deeper drawdown and worse future reward, the Bellman recursion should eventually learn that the action’s total value is lower than its one-week return suggests.
The Bellman idea also explains why value estimation is noisy. The value target includes future values, and those values are estimated by another neural network. That creates bootstrapping error. PPO controls this partly through conservative on-policy updates. SAC controls it with twin critics and target networks.
7.6 Temporal difference learning and Q-learning
Temporal-difference learning updates a value estimate using a one-step target. The TD error is:
If \(\delta_t\) is positive, the outcome was better than expected. If it is negative, the state was overvalued. A value model can update in the direction of that error.
Here \(\alpha\) is the learning rate. The term in brackets is the Q-learning TD error. The target uses the best next action according to the current Q estimate.
\[
y = r + \gamma \max_{a'} Q_{\theta^-}(s',a')
\]
The network \(Q_{\theta^-}\) is a target network, a slower-moving copy that stabilizes training.
DQN is a bridge concept here. We don’t use DQN as the final model because a portfolio action is continuous and constrained. The action isn’t one of a few buttons. It is a full vector of allocation weights. Continuous-action problems are usually easier to handle with policy-gradient or actor-critic methods, which is why the implementation focuses on PPO and SAC.
Q-learning is the classic bridge from Bellman equations to practical RL. In tabular Q-learning, we update an estimate of \(Q(s,a)\) using:
The target is \(r_{t+1} + \gamma \max_{a'}Q(s_{t+1},a')\). It says: take the reward we just saw, then assume we choose the best action next time. If the target is higher than the current estimate, raise \(Q(s_t,a_t)\). If it is lower, reduce it.
Deep Q-Networks replace the table with a neural network:
\[
Q_\phi(s,a) \approx Q^*(s,a)
\]
DQN is important historically and conceptually, but it is less natural for this notebook’s action space. DQN works best when the action set is discrete: buy, sell, hold; move left/right; choose one of a finite number of options. Portfolio weights are continuous. We could discretize the allocation space, but with 12 risky assets plus cash, the number of possible weight combinations explodes.
That is why the notebook uses policy-gradient and actor-critic methods. Instead of scoring every possible discrete action, the actor directly outputs a continuous action distribution. This fits portfolio allocation much better.
Still, Q-learning is useful to understand because SAC uses a critic that estimates \(Q(s,a)\). The difference is that SAC learns a continuous-action \(Q\) function and an actor that samples actions likely to have high \(Q\) value while maintaining entropy.
8) Policy Gradients and Actor-Critic Learning
A policy-gradient method directly parameterizes the policy:
\(\nabla_\theta \log \pi_\theta(A_t\mid S_t)\) tells us how the policy parameters affected the probability of the action we took,
\(A^\pi(S_t,A_t)\) tells us whether that action was better or worse than expected,
multiplying them increases the probability of good actions and decreases the probability of bad actions.
In this notebook, the policy outputs a Gaussian distribution over raw action vectors. So the policy can sample slightly different actions during training. That exploration is necessary. If the model always chose its current mean action, it would never learn whether nearby actions are better.
An actor-critic method combines two networks or two heads:
the actor proposes actions through \(\pi_\theta(a\mid s)\),
the critic estimates value, usually \(V_\phi(s)\) or \(Q_\phi(s,a)\).
The critic reduces variance. Instead of treating every positive return as good and every negative return as bad, we ask whether the action beat the expectation for that state. That expectation comes from the value function.
8.1 Policy gradients from the objective
Policy-gradient methods optimize the policy directly. The objective is:
Here \(\tau\) means a trajectory: states, actions, and rewards generated by policy \(\pi_\theta\). The policy-gradient theorem gives a practical gradient direction:
\(\nabla_\theta \log \pi_\theta(a_t\mid s_t)\) tells us how to change the policy parameters to make the sampled action more or less likely.
\(A^\pi(s_t,a_t)\) tells us whether the action was better or worse than expected.
If the advantage is positive, the update increases the probability of that action in similar states. If the advantage is negative, the update decreases it.
In continuous portfolio control, the policy outputs a Normal distribution over raw actions:
The action is then mapped into weights. The policy doesn’t output fixed weights deterministically during training; it samples. That sampling is exploration. Without exploration, the policy would keep repeating the same allocation and never learn whether nearby allocations are better.
The actor-critic diagram shows the two-network idea:
the actor produces actions,
the critic estimates value,
the advantage connects the two.
The critic reduces variance. Instead of comparing every reward to zero, we compare it to what the critic expected from that state. This makes policy-gradient learning much more stable.
8.2 Log probabilities and continuous actions
The term \(\log \pi_\theta(a_t\mid s_t)\) is central to policy gradients. Since the policy is stochastic, it assigns a probability density to the action it sampled. The log-probability tells us how likely that exact raw action was under the current policy.
For a Normal policy with diagonal standard deviation:
This formula says an action has high log-probability if it is close to the policy mean and the policy standard deviation is not too small. During training, the gradient of this log-probability tells the network how to shift its mean and standard deviation.
The raw action distribution is defined before the action-to-weight transformation. That is simpler because the raw action is unconstrained. After sampling, we map the raw action into a valid portfolio. The policy gradient is still computed using the raw action’s log-probability. This is common in continuous-control implementations: the network explores in an unconstrained action space, and the environment transforms the action into a legal control.
The portfolio interpretation is direct. If a sampled raw action leads to a better-than-expected reward, PPO increases the chance of similar raw actions in similar states. If the action leads to worse reward, PPO decreases it. Over many rollouts, this should shape the policy distribution toward useful allocation choices.
8.3 Actor-critic learning in portfolio language
Actor-critic can be described without losing the finance intuition:
The actor is the portfolio decision maker.
The critic is the evaluator of market states and actions.
The advantage is the critic’s correction signal.
At time \(t\), the actor samples an allocation. The environment returns a reward and a next state. The critic estimates whether that outcome was better than expected. If the action produced a positive advantage, the actor is nudged toward similar actions. If it produced a negative advantage, the actor is nudged away.
The critic is not a performance report. It is a learned function approximator. That means it can be wrong. If the critic overestimates the value of risky actions in a fragile regime, the actor can learn a bad policy. PPO reduces the damage by clipping updates. SAC reduces overestimation through twin critics and entropy. These algorithmic details matter because financial rewards are noisy and nonstationary.
The actor also doesn’t directly know the Sharpe ratio. It receives the shaped reward. So if the reward over-penalizes volatility, the actor may become too defensive. If the reward under-penalizes drawdown, the actor may become too aggressive. The reward design and the actor-critic update are connected. A good algorithm can’t save a badly aligned reward.
This is why the notebook reports reward components, policy diagnostics, stress windows, and risk reports. We need to see not only that the RL loss went down, but also what kind of portfolio behavior emerged.
Show code
from quantfinlab.plotting.diagrams import policy_gradient_diagramfig, ax = policy_gradient_diagram()plt.show()
Show code
from quantfinlab.plotting.diagrams import actor_critic_diagramfig, ax = actor_critic_diagram()plt.show()
The policy-gradient diagram shows this idea visually. The policy samples an action, the environment returns reward, and the advantage estimate tells the policy whether that action should become more or less likely. The actor-critic diagram adds the critic: the actor still chooses the action, but the critic learns a value estimate and gives the actor a lower-variance learning signal.
For portfolio allocation, this division is natural. The actor asks, “what weights should I hold?” The critic asks, “given the current market state and portfolio condition, how valuable is this situation?” If the actor makes a risky allocation in a fragile state and the next reward is poor, the critic helps convert that poor outcome into a parameter update.
8.4 Generalized advantage estimation
PPO and recurrent PPO use generalized advantage estimation. The one-step TD residual is:
\[
\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)
\]
This residual says how surprising the reward plus next-state value was relative to the current value. GAE combines many TD residuals:
The parameter \(\lambda\) controls the bias-variance tradeoff. If \(\lambda\) is close to 0, the advantage depends mostly on one-step TD errors. This is lower variance but more biased. If \(\lambda\) is close to 1, it uses longer reward information. This is less biased but noisier.
The code uses \(\gamma=0.97\) and \(\lambda=0.90\). That means the advantage estimate has memory, but it doesn’t let very distant rewards dominate. In weekly portfolio data, this is reasonable: the agent should care about multi-week consequences, but not treat rewards one year later as clean feedback about today’s action.
GAE smooths the advantage estimate across multiple temporal-difference errors. The one-step TD error is:
The parameter \(\lambda\) controls the bias-variance tradeoff. If \(\lambda=0\), the advantage is mostly one-step and low-variance but biased toward the critic’s current estimate. If \(\lambda=1\), it uses longer returns and becomes less biased but noisier. The notebook uses \(\lambda=0.90\), which gives the policy several weeks of reward information without making the advantage estimate too noisy.
This is valuable in finance because a good allocation decision may not reveal itself in one week. A move into cash before a drawdown may look unexciting at first, then become valuable later. A move into high beta may look good for a week and then hurt the portfolio two weeks later. GAE helps the policy connect actions with multi-step consequences.
9) PPO: Stable On-Policy Policy Optimization
PPO updates the policy using data collected from the current policy. The core problem is that policy gradients can be unstable. If one update changes the policy too much, the data collected under the old policy no longer represents the new policy well.
If \(r_t(\theta)=1\), the new policy gives the sampled action the same probability as the old policy. If it is larger than 1, the new policy increased the probability of that action. If it is smaller than 1, the new policy decreased it.
The clipping range in the code is 0.8 to 1.2, so \(\epsilon=0.2\). This means the policy update is discouraged from changing the probability of sampled actions by more than about 20% in one update.
The full PPO loss also includes a value loss and an entropy bonus:
The value loss trains the critic. The entropy term keeps the policy from collapsing too early. In portfolio terms, entropy means the raw action distribution stays exploratory during training, instead of immediately committing to one rigid allocation pattern.
This ratio says how much more likely the new policy makes the old sampled action. If \(r_t=1\), the new policy assigns the same probability as before. If \(r_t=1.3\), the new policy makes that action 30% more likely. If \(r_t=0.7\), it makes the action 30% less likely.
The clipping range is \([0.8,1.2]\) in this notebook. The logic is easy to read case by case:
Advantage sign
Policy change
PPO behavior
\(\hat{A}_t>0\)
action becomes slightly more likely
good update
\(\hat{A}_t>0\)
action becomes too much more likely
clipped
\(\hat{A}_t<0\)
action becomes slightly less likely
good update
\(\hat{A}_t<0\)
action becomes too much less likely
clipped
This makes PPO cautious. It can improve the policy, but it shouldn’t move too far from the policy that generated the rollout. That is important in finance because the reward signal is noisy. If the policy changes too aggressively after a lucky or unlucky sequence, it can destroy out-of-sample behavior.
\(\mathcal{L}_{policy}\) is the clipped policy-gradient loss.
\(\mathcal{L}_{value}\) trains the critic to predict returns.
\(\mathcal{H}(\pi)\) is entropy, which keeps the policy from becoming deterministic too early.
The entropy term is especially useful early in training. If the policy collapses immediately into one allocation shape, it stops exploring other possible risk profiles.
9.2 On-policy learning and rollout quality
PPO is an on-policy method. It collects a rollout using the current policy, computes advantages, then updates the policy using that same rollout. After the update, old data becomes less useful because it came from an older policy.
This matters in finance because the data sample is already small. On-policy learning can be stable but sample-hungry. Every update depends on the specific states and rewards collected by the current policy. If the rollout period contains an unusual market pattern, the update can be biased toward that pattern. PPO clipping reduces the damage by limiting how far the policy can move.
The rollout object in this notebook stores:
states,
actions,
log-probabilities under the old policy,
rewards,
done flags,
value estimates,
resulting weights,
reward components.
The old log-probabilities are essential because PPO compares old and new policy probabilities through \(r_t(\theta)\). Without them, we couldn’t compute the clipped objective.
Show code
from quantfinlab.plotting.diagrams import ppo_diagramfig, ax = ppo_diagram()plt.show()
Show code
from quantfinlab.plotting.diagrams import sac_diagramfig, ax = sac_diagram()plt.show()
9.3 Cross-asset policy architecture
The PPO policy starts with a cross-asset encoder. This is a neural architecture designed for a portfolio universe, not for a single time series.
For each decision date, the asset features form:
\[
X_t \in \mathbb{R}^{N\times F_a}
\]
The encoder maps every asset row into a hidden vector:
\[
h_{i,t} = f_\theta(x_{i,t})
\]
Then multi-head attention lets assets interact with each other. Attention creates a context-aware asset representation:
where \(H_t\) is the matrix of all asset embeddings. Financially, this matters because asset allocation is cross-sectional. The attractiveness of QQQ doesn’t only depend on QQQ features; it also depends on whether SPY, IWM, HYG, GLD, and SHY look better or worse at the same time.
The model also encodes global features and portfolio features:
\[
g_t = f_g(X^{global}_t)
\]
\[
p_t = f_p(X^{portfolio}_t)
\]
The final fused representation is:
\[
z_t = [c_t, g_t, p_t]
\]
where \(c_t\) is the pooled cross-asset context. The policy has one head for risky-asset logits, one head for total exposure, and one value head for the critic.
The policy also has an alpha boost term. If the forecast feature index is available, the asset logits get an extra learned multiple of the forecast rank signal. This means the neural policy can learn to lean into the forecast signal instead of rediscovering it from scratch.
The cross-asset actor-critic architecture is designed around the input structure. The asset encoder reads the \(N\times F_a\) asset matrix. A multi-head attention block lets assets interact with each other. This matters because allocation is relative. The model should be able to learn that SPY’s signal is strong only relative to QQQ, HYG’s signal is risky when LQD is weak, or GLD becomes useful when equity and bond features are both stressed.
\(Q\) are queries, asking what each asset wants to compare,
\(K\) are keys, describing what information each asset offers,
\(V\) are values, carrying the transformed asset information,
\(d\) scales the dot products so attention scores don’t explode.
In this project, attention is not used as a huge transformer. It is a compact way to let asset rows exchange information before the policy forms allocation scores.
The global encoder processes macro/regime/market features. The portfolio encoder processes previous portfolio context. These three streams are combined into one representation:
The actor head maps this representation to action means. The critic head maps it to a value estimate. The same encoded information therefore supports both decisions and value learning.
The PPO implementation collects rollouts, computes GAE advantages, standardizes the advantages, and updates the policy using the clipped objective. The validation step is deterministic: it uses the mean action, not random sampling. That is how we test the learned policy as an actual portfolio rule.
The PPO validation table shows:
total reward: 92.52,
mean reward: 0.8897,
return: 8.15% over the validation window,
volatility: 7.79%,
Sharpe: 0.0415 under the validation calculation,
max drawdown: -11.54%,
average risky exposure: 79.98%.
The reward is strong because the policy learned to respect the reward design, not because its validation Sharpe is high. This is a useful warning: reward and conventional performance metrics can disagree. In RL portfolio work, a policy can learn the shaped reward very well while still being mediocre on CAGR or Sharpe.
The training curves should be read as diagnostics rather than proof of success. A rising training reward is good, but it can reflect overfitting to the training market path. Validation exposure and validation reward are more informative. In the output, PPO finds a controlled allocation with moderate cash and diversified risky weights. That is stable behavior, but the final test comparison shows that stability alone doesn’t beat the stronger supervised portfolio baselines.
The validation result for PPO shows a positive total reward and controlled volatility around 7.8%, but the Sharpe is only slightly positive in the validation table. The policy average exposure is around 80%, and the weight summary shows meaningful cash around 20%. PPO learns a conservative policy with diversified risky weights, but it doesn’t look like it dominates the stronger rule-based forecast models.
Reading PPO training outputs
The PPO training plot usually contains reward and loss-style curves. These are useful, but they should be interpreted carefully.
A rising training reward means the policy is learning to score better inside the training environment. It doesn’t automatically mean the policy will perform better out of sample. The validation table is more important because it uses a separate period. The test-period strategy comparison is more important still.
The value loss tells us how well the critic is fitting return targets. If value loss is too high, the advantage estimates become unreliable. If it is too low too quickly, the critic may be overfitting the training path. The entropy curve tells us whether the policy is still exploring. If entropy collapses early, the policy may become deterministic before it has learned enough.
For this project, the practical PPO diagnostic is the validation exposure plot. PPO doesn’t become extremely aggressive. It keeps a meaningful SHY position and spreads risk across several assets. That behavior is consistent with the reward: cost, concentration, drawdown, and volatility penalties push the policy away from extreme allocations.
The limitation is that PPO’s caution also reduces upside. A policy can avoid many mistakes and still fail to beat a strong forecast-gated strategy if it doesn’t take enough profitable risk when conditions are favorable.
Neural-network outputs and portfolio interpretation
The PPO actor has two important output concepts: allocation scores and exposure. The risky scores decide the relative risky allocation. The exposure score decides how much of the portfolio goes into the risky sleeve versus SHY.
That structure is better than asking the network to output 13 unrelated weights. It matches portfolio thinking. A manager often first decides the risk budget, then decides which assets deserve that risk. For example, in a defensive environment the policy may reduce total risky exposure. Inside the remaining risky sleeve, it may still prefer GLD, IEF, or defensive equity exposure. In a risk-on environment, the policy may raise total exposure and tilt toward SPY, QQQ, HYG, or IWM.
The value head has a different job. It doesn’t choose weights. It estimates how good the current state is under the current policy. When the critic says the state is valuable but the realized reward disappoints, the advantage becomes negative. When the realized reward exceeds the critic’s expectation, the advantage becomes positive. This is how the model learns from surprise.
The architecture therefore separates three tasks cleanly:
encode asset-level relative information,
encode global market context,
encode previous portfolio state,
produce a stochastic action distribution,
estimate state value for advantage learning.
This separation helps the reader understand why the code has encoders, actor heads, exposure heads, log-standard-deviation parameters, and value heads. They aren’t arbitrary neural-network pieces. Each one corresponds to a part of the portfolio-control problem.
The PPO training plot shows noisy training rewards, noisy policy/value losses, and a gradual change in entropy. That is normal for on-policy RL with a small financial sample. The policy is learning from a single sequence of weekly market transitions, so the reward path is much more unstable than the loss curve of a supervised regression model.
The PPO validation exposure plot shows the policy staying mostly around 75% to 85% risky exposure, with cash around 15% to 25%. This is consistent with the validation weight table. PPO gives the largest average allocations to SPY, SHY, IWM, QQQ, and HYG. It gives very small weights to IEF and GLD on average, which means it doesn’t behave like a defensive bond/gold allocator. It behaves more like a diversified risky policy with a cash buffer.
10) Recurrent PPO and Memory
Feedforward PPO receives one state vector at a time. Recurrent PPO adds memory. The policy keeps an internal hidden state that updates as observations arrive:
\[
h_t = \text{LSTM}_\theta(z_t, h_{t-1})
\]
The LSTM was already covered in the forecasting project, so here the important point is its RL role. The policy isn’t forecasting an asset return sequence. It is carrying a memory of the market-control sequence: previous states, previous actions, and the reward environment implied by them.
The recurrent policy still outputs a Gaussian raw action distribution, but the mean depends on the memory:
This can help in partial observability. For example, a single state might show moderate volatility, but the last several states may show volatility rising week by week. A recurrent policy can remember that path. In finance, the path often matters because regimes evolve through sequences, not isolated observations.
The recurrent PPO training code evaluates sequence actions in chunks. It computes GAE inside subsequences and applies the same clipped PPO objective as the feedforward model:
The difference is not the PPO loss. The difference is the policy representation. Recurrent PPO can make the same observed state lead to different actions depending on the hidden memory.
10.1 Sequence memory and recurrent state
Recurrent PPO adds memory through an LSTM. This is useful because the current state vector may not fully capture the market path. The LSTM processes a sequence of observations and carries a hidden state:
\(h_t\) is the hidden state passed to the policy head,
\(c_t\) is the cell state that stores longer-term information.
The LSTM gates decide what to keep, forget, and output. A simplified view is:
\[
f_t = \sigma(W_f[x_t,h_{t-1}] + b_f)
\]
\[
i_t = \sigma(W_i[x_t,h_{t-1}] + b_i)
\]
\[
o_t = \sigma(W_o[x_t,h_{t-1}] + b_o)
\]
The forget gate \(f_t\) controls how much old memory survives. The input gate \(i_t\) controls how much new information enters memory. The output gate \(o_t\) controls how much of the memory is exposed to the policy. This gating structure is why an LSTM can remember more than a simple RNN.
For portfolio allocation, memory can represent patterns like:
volatility rising for several weeks,
equity breadth deteriorating gradually,
forecasts staying positive but realized returns weakening,
credit spreads improving persistently,
repeated failed attempts to rotate into risk assets.
10.2 Truncated backpropagation through time
Training a recurrent policy requires backpropagating through sequences. If we backpropagated through the entire 949-week history at once, training would be unstable and memory-heavy. Instead we train on chunks of fixed sequence length. This is called truncated backpropagation through time.
Let the recurrent policy process a sequence:
\[
x_{t-L+1}, x_{t-L+2}, \ldots, x_t
\]
The LSTM updates hidden memory step by step:
\[
h_{k},c_{k}=f_\theta(x_k,h_{k-1},c_{k-1})
\]
The policy loss is computed across the sequence, but gradients are only propagated across that training chunk. This gives the model enough history to learn persistent regimes without making training too expensive.
In this notebook the recurrent policy uses a sequence length of 52 weeks. That is roughly one year of weekly decisions. This is a sensible horizon for macro and allocation regimes: long enough to observe slow deterioration or recovery, short enough to keep training manageable.
The recurrent PPO validation table shows lower risky exposure than PPO, around 72%, and higher cash. It also shows lower validation total reward. In this run, the recurrent memory makes the policy more defensive, but it doesn’t clearly improve performance. That is still a useful result. Memory helps only if the training data is rich enough and the sequence signal is stable enough. If the recurrent model mostly learns to avoid risk, it can reduce drawdown but lose too much return.
The RPPO result is an important reminder that more model structure doesn’t automatically improve allocation. The recurrent model has memory, but it also has more parameters and a harder training problem. In this run it becomes more defensive, holds more SHY, and doesn’t beat the non-recurrent PPO or SAC in the final comparison. That is a real result, not a failure of the notebook. It shows that the data may not contain enough stable sequential signal for this recurrent architecture to dominate.
The recurrent PPO validation table is weaker than the feedforward PPO table. It shows total reward 55.76, return 7.10%, volatility 7.29%, validation Sharpe -0.0281, max drawdown -11.08%, and average exposure 72.14%.
The lower exposure is visible in the validation weight table. Recurrent PPO holds about 27.86% in SHY on average, compared with about 20.02% for PPO. It also spreads risk more evenly. Effective diversification later confirms that recurrent PPO is the most diversified RL policy.
This is not automatically a failure. It means the recurrent policy learned a more defensive allocation. In this validation sample, that defensiveness reduced volatility but didn’t improve Sharpe. The added memory may also be too flexible for the amount of training data. Recurrent models need enough sequential evidence to justify the extra parameters.
The recurrent training plots show the same basic pattern: noisy reward, unstable loss paths, and gradual changes in exposure. This is a good place to remember that RL training curves in finance don’t look like clean supervised-learning curves. A supervised model sees many labeled samples. An on-policy RL model repeatedly changes the policy, changes the data distribution it experiences, and then updates again. That makes the training path noisier by design.
11) SAC and Maximum-Entropy Actor-Critic
SAC is the most advanced RL model in this notebook. It is an off-policy actor-critic method with a maximum-entropy objective.
The standard RL objective maximizes expected return:
The entropy term rewards policies that keep some randomness. The coefficient \(\alpha\) controls how much the policy values exploration. In continuous control, this is very useful because the action space has many possible directions. In portfolio allocation, it means the policy doesn’t have to collapse immediately into one deterministic set of weights during training.
Minimizing this actor loss means increasing Q-value while also keeping entropy. The implementation also learns \(\alpha\) using a target entropy, so the amount of randomness adjusts during training rather than being fixed forever.
11.1 Maximum entropy objective
SAC optimizes a different objective from PPO. It doesn’t only maximize expected reward. It also rewards entropy:
The entropy term measures how random the policy is. Higher entropy means the policy keeps multiple actions plausible instead of collapsing into one deterministic allocation. The coefficient \(\alpha\) controls how much entropy matters.
In financial allocation, entropy has a practical meaning. A policy that becomes too certain too quickly may overfit one historical pattern. A stochastic policy keeps exploring different allocation shapes, which can act like a regularizer. SAC also learns \(\alpha\) automatically, which is useful because the right exploration level changes during training.
SAC learns two critics:
\[
Q_{\phi_1}(s,a), \qquad Q_{\phi_2}(s,a)
\]
The target uses the minimum of the two:
\[
\min(Q_{\phi_1},Q_{\phi_2})
\]
This reduces overestimation bias. If one critic is too optimistic about an action, the other critic can pull the target down. In noisy finance rewards, that is a big advantage.
The first part is the immediate reward. The second part is discounted future soft value. The \(\log\pi\) term is negative for low-probability actions, so subtracting \(\alpha\log\pi\) rewards entropy. The target critics \(Q_{\phi_i^-}\) are slow-moving copies of the learned critics, which stabilizes training.
Minimizing this loss pushes the actor toward actions with high critic value and enough entropy. The actor wants high \(Q\), but it is also encouraged not to become too deterministic.
This is why SAC often behaves differently from PPO. PPO collects fresh rollouts and updates cautiously around them. SAC reuses experience through a replay buffer and learns a critic over continuous actions.
11.2 Reparameterization in SAC
SAC uses the reparameterization trick for the actor update. Instead of sampling an action in a way that blocks gradients, we write the sampled action as:
Now the randomness is isolated in \(\epsilon_t\), and the action remains differentiable with respect to \(\mu_\theta\) and \(\sigma_\theta\). This lets the actor receive gradients through the critic:
\[
\nabla_\theta Q_\phi(s_t,a_t)
\]
That is one reason SAC is powerful for continuous action spaces. The actor doesn’t only learn from log-probability weighting like vanilla policy gradient. It can move directly toward actions that the critic says have high value, while the entropy term keeps it exploratory.
11.3 Automatic entropy temperature
The entropy temperature \(\alpha\) is learned by comparing the policy entropy with a target entropy. A common objective is:
If the policy entropy is too low, \(\alpha\) rises and exploration becomes more valuable. If entropy is too high, \(\alpha\) falls and the policy focuses more on reward. The notebook sets the target entropy proportional to the action dimension:
\[
\mathcal{H}_{target} = -0.75(N+1)
\]
The action dimension is \(N+1\) because we have one raw score per risky asset plus one exposure score. This scaling makes sense because a higher-dimensional action space needs more entropy than a one-dimensional action space.
The learned temperature is another stabilizer. In a noisy portfolio environment, a fixed entropy penalty might be too high early and too low later. Automatic temperature tuning gives SAC a way to adjust exploration during training.
11.4 Twin critics and pessimism
SAC’s twin critics are especially useful in finance because return noise can make value estimates too optimistic. If one critic overestimates the value of a high-risk action, the min operator reduces the target:
This is a built-in pessimism mechanism. It doesn’t make the policy conservative in every sense, but it reduces the chance that one critic’s positive error makes the actor chase a bad action.
11.5 SAC behavior in the final portfolio context
SAC’s aggressive behavior can be understood from its objective. The critic learns that many risky allocations produced strong reward over parts of the historical sample. The entropy objective keeps the policy from collapsing, but it doesn’t force the policy to hold cash. If the critic values high-exposure actions, SAC will learn high exposure.
11.6 PPO and SAC as different answers to the same control problem
PPO and SAC solve the same portfolio-control problem with different learning assumptions.
PPO says: collect data from the current policy, estimate advantages, and update the policy carefully so it doesn’t move too far. The policy is stable because each update is restricted. The downside is that PPO is sample-hungry. It uses fresh rollouts and doesn’t reuse old experience as aggressively.
SAC says: store transitions in a replay buffer, learn critics off-policy, and train an actor that maximizes both value and entropy. This is more sample-efficient and often more powerful in continuous-control environments. The downside is that critic errors can shape the actor in bad ways if the Q-function becomes miscalibrated.
In finance, neither assumption is perfect. PPO’s on-policy stability is attractive because market data is noisy, but on-policy learning wastes data. SAC’s replay buffer is attractive because data is limited, but old transitions may come from regimes that no longer apply. This is why the notebook trains both. The comparison isn’t just a leaderboard. It shows two different RL philosophies applied to the same allocation environment.
11.7 Replay buffer and off-policy updates
SAC is off-policy, so it can reuse previously collected transitions. The replay buffer stores tuples:
\[
(s_t,a_t,r_t,s_{t+1},d_t)
\]
where \(d_t\) indicates whether the episode ended. During training, SAC samples minibatches from this buffer and updates the critics and actor. This is more sample-efficient than on-policy PPO because the same transition can be used multiple times.
In this notebook, the SAC code collects a full rollout from the current policy, appends transitions to the replay buffer, and then performs minibatch critic/actor updates. The target critic is updated softly:
The code uses a slow update, with most of the target network preserved and a small amount of current critic weight mixed in. This is another stability mechanism.
The twin-critic structure is especially useful in continuous action problems. If a single critic overestimates the value of a weird action, the actor may exploit that error. Taking the minimum of two critics makes that exploitation harder.
Off-policy learning is another major SAC advantage. PPO can only update from data collected by the current policy, or almost-current policy. SAC stores transitions in a replay buffer:
\[
(s_t,a_t,r_t,s_{t+1},d_t)
\]
Then it samples mini-batches from that buffer many times. This is more sample efficient. In finance, sample efficiency matters because we have very limited historical data. However, replay also has risk: old transitions may come from policies or market periods that are no longer representative.
The target critic update uses soft target networks:
\[
\phi^- \leftarrow \rho\phi^- + (1-\rho)\phi
\]
With \(\rho=0.995\), the target critic changes slowly. This prevents the critic target from chasing the online critic too aggressively.
The SAC validation table shows the strongest validation Sharpe among the three RL policies, even though the reward is not the highest. It reports total reward 60.38, return 8.59%, volatility 9.35%, Sharpe 0.0704, max drawdown -14.43%, and average exposure 96.24%.
This makes SAC clearly more aggressive than PPO and recurrent PPO. The validation weight table shows average weights around 25.01% SPY, 15.30% HYG, 14.46% IWM, 11.35% EFA, and only 3.76% SHY. The max weights also show that SPY, EFA, and HYG can hit the 35% cap.
So SAC is learning a high-risky-exposure policy. It accepts more volatility and drawdown than the PPO policies, but it captures more upside. Later, when we compare out-of-sample performance, SAC remains the best RL policy by Sharpe, but it still doesn’t beat the stronger supervised/rule-based baselines.
The SAC loss plot has four panels: training reward, critic loss, actor loss, and soft Q-value. The critic loss falls quickly and then stays near a low region, which suggests the critic is fitting the replay-buffer targets. The Q-value plot rises over training, which means the actor-critic system is learning actions the critics value more highly.
The policy validation exposure plot shows SAC staying close to full risky exposure. Cash is consistently low. This is the opposite of recurrent PPO and very different from the cash-heavy validation behavior of RandomForest Kelly. SAC is using the action space to make aggressive allocations while respecting the per-asset cap.
In this notebook, SAC becomes the highest-risky-exposure RL policy, with average exposure around 96% and cash around 4%.
The final SAC behavior still turns out aggressive. That tells us the critic learned high value for risky exposure in many states. Since the sample includes strong equity and credit periods, that is understandable. The risk report later shows the cost of that behavior in stress windows.
The validation weight summary confirms this. SAC holds much less SHY than PPO and recurrent PPO. It puts more average weight into SPY, IWM, HYG, QQQ, and EFA. That is a risk-on cross-asset allocation. It can do well when equity and credit premia are rewarded, but it creates larger losses during broad stress.
This is exactly why SAC is interesting here. It is the most advanced RL method in the notebook, and it becomes the best RL model by final Sharpe, but it still doesn’t dominate the rule-based alternatives. That makes the empirical story stronger. We can see the algorithm’s strength and its limitation at the same time:
it learns a decisive continuous policy,
it uses off-policy sample reuse,
it handles stochastic exploration elegantly,
it still struggles against carefully designed financial baselines.
A weaker notebook would hide that comparison. This one makes it visible.
The output behavior matches that difference. PPO is controlled and diversified. SAC is more decisive and risk-seeking. Recurrent PPO becomes more defensive because the memory model appears to learn caution from sequences. These behavioral differences are just as important as the final Sharpe ratios.
The SAC validation result shows higher return than PPO and recurrent PPO, but also higher volatility and a deeper drawdown. The average exposure is near the maximum. This is exactly the kind of tradeoff we need to interpret carefully. SAC is the most aggressive RL model in this run. It learns to use the risky sleeve heavily, especially SPY, IWM, HYG, and QQQ. That can help in bull or rebound periods, but it also leaves the policy more exposed in stress windows.
12) Building the Final RL Weight Frames
After training, we convert each policy into a deterministic weight frame for the test period. PPO, recurrent PPO, and SAC each produce weekly weights. Then we also create an RL ensemble by blending the policies using validation Sharpe-like scores.
The small floor prevents a model from receiving exactly zero ensemble weight. This is useful when validation scores are noisy. If a model is weak but not completely unusable, a tiny allocation keeps it represented without letting it dominate.
The final strategy set contains:
Equal Weight,
Forecast-Gated MaxSharpe from Project 19,
RandomForest Kelly from Project 19,
ML Regime-Aware from Project 16,
PPO,
Recurrent PPO,
SAC,
Ensemble RL.
This comparison is deliberately hard for RL. It doesn’t compare RL only against naive baselines. It compares RL against strong hand-built and supervised-learning strategies that already use forecasts, Kelly sizing, and regime logic.
12.1 Validation-score ensemble
The RL ensemble combines PPO, recurrent PPO, and SAC using validation performance. The generic ensemble weight is:
where \(w^{(m)}_t\) is the weight vector from model \(m\), and \(\omega_m\) is the model’s ensemble weight. The weights are based on validation scores, so stronger validation policies receive more influence.
This kind of ensemble can reduce model-specific risk. PPO is more cautious, recurrent PPO is even more defensive, and SAC is more aggressive. Averaging them can smooth extreme behavior. The tradeoff is that an ensemble can also dilute the best model. If SAC is the strongest RL model, averaging it with weaker PPO policies may lower return while only partially improving drawdown.
13) Final Strategy Comparison
Show code
def policy_weight_frame_local(model, indices, recurrent=False): rows = collect_recurrent_rollout(model, indices, deterministic=True) if recurrent else collect_policy_rollout(model, indices, deterministic=True)return pd.DataFrame(rows["weights"], index=pd.DatetimeIndex(rows["date"]), columns=state_columns)def blend_policy_weights(weight_frames, blend_weights): total =sum(max(0.0, v) for v in blend_weights.values()) orfloat(len(weight_frames)) columns =sorted({c for f in weight_frames.values() for c in f.columns}) index =sorted({d for f in weight_frames.values() for d in f.index}) blended = pd.DataFrame(0.0, index=pd.DatetimeIndex(index), columns=columns)for name, frame in weight_frames.items(): blended = blended + (max(0.0, blend_weights.get(name, 0.0)) / total) * frame.reindex(index=blended.index, columns=columns).fillna(0.0)return blended.div(blended.sum(axis=1).replace(0.0, np.nan), axis=0).fillna(0.0)w_ppo = policy_weight_frame_local(ppo, test_idx)w_rppo = policy_weight_frame_local(rppo, test_idx, recurrent=True)w_sac = policy_weight_frame_local(sac, test_idx)w_ensemble = blend_policy_weights( {"PPO": w_ppo, "Recurrent PPO": w_rppo, "SAC": w_sac}, {"PPO": max(0.01, ppo_validation["sharpe"]),"Recurrent PPO": max(0.01, rppo_validation["sharpe"]),"SAC": max(0.01, sac_validation["sharpe"]), },).reindex(columns=state_columns).fillna(0.0)weights_by_strategy = {"Equal Weight": w_eq,"Forecast-Gated MaxSharpe": w_forecast_gated_ms,"RandomForest Kelly": w_rf_kelly,"ML Regime-Aware": w_regime,"PPO": w_ppo,"Recurrent PPO": w_rppo,"SAC": w_sac,"Ensemble RL": w_ensemble,}test_returns_daily = r_d.loc[pd.Timestamp(test_period[0]):pd.Timestamp(test_period[1]), assets + [cash_ticker]]final_backtests = run_many_weights_backtests( weights_by_strategy, returns=test_returns_daily, cost_bps=cost_bps, rf_daily=rf_daily, w_min=0.0, w_max=1.0, long_only=True, normalize=True, weight_timing="next_close",)
The final results show exactly that kind of mixed behavior. Ensemble RL sits between the individual policies. It has lower volatility than SAC but also lower return. It doesn’t beat the strongest rule-based models.
The final performance table is the most important result in the notebook. It shows that the strongest overall model is ML Regime-Aware, with Sharpe 0.6889, CAGR 10.88%, volatility 10.00%, and max drawdown -19.13%. RandomForest Kelly and Forecast-Gated MaxSharpe have higher CAGR, around 12.44% and 12.46%, but also higher volatility and larger drawdowns.
Among the RL models, SAC is the strongest, with Sharpe 0.4211, CAGR 9.03%, volatility 13.11%, and max drawdown -28.24%. Ensemble RL, recurrent PPO, and PPO are weaker. PPO has the lowest RL Sharpe at 0.3754. Recurrent PPO has lower volatility and drawdown than SAC, but it doesn’t generate enough return to compensate.
The honest interpretation is that RL is impressive as an implementation, but it doesn’t dominate the portfolio models in this sample. This is a realistic result. Financial RL has very limited independent training data, strong nonstationarity, transaction costs, and a noisy reward. The rule-based ML strategies benefit from stronger structure: they transform forecasts and regimes through explicit portfolio rules. The RL policies have to learn that mapping from a much smaller sequential training set.
The table still shows useful behavior. SAC reaches a reasonable out-of-sample CAGR and effective number around 6.01, so it isn’t just degenerating into one asset. Recurrent PPO has the lowest RL drawdown at -20.53%, suggesting the memory policy learned a more defensive style. PPO and recurrent PPO also have lower turnover than the rule-based forecasting strategies.
The final comparison table is the most important empirical output. The strongest strategies in the test period are still the earlier supervised/rule-based models: ML Regime-Aware, RandomForest Kelly, and Forecast-Gated MaxSharpe. SAC is the best pure RL model by Sharpe among the three RL policies, but it doesn’t beat the strongest non-RL models.
This is a realistic result. RL has a more flexible objective and a richer action loop, but flexibility doesn’t guarantee better out-of-sample performance. The stronger rule-based models benefit from more direct forecast-to-weight structure and stronger inductive bias. RL has to learn the whole mapping from state to action from a small historical sample.
Still, the RL models are not useless. They show different policy behavior:
PPO learns a diversified, moderately defensive allocation.
Recurrent PPO learns an even more defensive allocation with more SHY.
SAC learns a high-exposure, more concentrated policy.
The RL ensemble averages those behaviors and lands between them.
The final performance table gives us a good honest conclusion: RL is promising as a decision layer, but in this dataset and implementation, it doesn’t yet dominate the simpler forecast-gated and regime-aware systems. That is valuable because it tells us where the bottleneck is. The issue is not building an RL environment. The environment works. The issue is extracting robust signal from a small, noisy market history without overfitting.
13.1 Model comparison across learning philosophies
The final table is also a comparison of modeling philosophies:
Strategy type
Main learning idea
Main risk
Equal Weight
no forecast, stable diversification
ignores changing opportunity set
Forecast-Gated MaxSharpe
supervised alpha forecast plus optimizer
sensitive to forecast quality and optimizer assumptions
RandomForest Kelly
tree forecast plus growth-optimal sizing
sizing can become too aggressive or too defensive
ML Regime-Aware
regime probabilities guide allocation
regime model may lag transitions
PPO
cautious on-policy RL control
sample-hungry and may learn conservative behavior
Recurrent PPO
PPO plus memory
harder to train, may become overly defensive
SAC
off-policy entropy-regularized actor-critic
can become aggressive if critic values risky exposure
Ensemble RL
average RL policy behavior
may dilute the best individual policy
This table is useful because the project shouldn’t be judged by one leaderboard number only. Each model has a different bias. Forecast-Gated MaxSharpe has a strong prior: forecasts enter an optimizer. RL has a weaker prior: it learns the state-action mapping from reward. That makes RL more flexible but also easier to overfit.
The empirical result favors the structured supervised allocation models. That doesn’t reduce the value of Project 20. It actually makes the project stronger, because it shows a realistic outcome. The final project demonstrates RL engineering, reward design, policy training, and risk diagnostics, while still admitting that the best simpler models remain hard to beat.
13.2 Practical ranking of the final strategies
The final ranking can be interpreted in three layers.
First, the strongest test Sharpe belongs to ML Regime-Aware. That means the regime information from Project 16 still has strong practical value when mapped into portfolio weights. It doesn’t need to learn a full RL control policy. It uses a structured macro/market-state idea and behaves well out of sample.
Second, RandomForest Kelly and Forecast-Gated MaxSharpe have the strongest return profiles. These are more forecast-driven and more active. Their drawdowns are larger than ML Regime-Aware, but their CAGR is also higher. This is the familiar tradeoff between growth, turnover, concentration, and drawdown.
Third, the RL models occupy the middle/lower part of the table. SAC is the strongest RL model, but its Sharpe is below the main supervised baselines. PPO and recurrent PPO are more conservative but also lower return. Ensemble RL smooths the RL policies but doesn’t create a new edge.
This is a useful final lesson for the series. More flexible ML doesn’t automatically beat stronger structure. In finance, a model with a good inductive bias can outperform a larger model that has to learn the bias from limited data. RL gives us a powerful framework, but it needs enough data and a very well-aligned reward to compete with carefully designed portfolio rules.
The table also shows a practical limitation of final-project complexity. A model can be technically more advanced and still lose to a simpler structure if the simpler structure matches the data-generating problem better. In this sample, regime-aware and forecast-gated rules have strong financial priors: they already know how forecasts and regimes should affect weights. RL has to discover that mapping from reward. With only hundreds of weekly decisions, discovering a robust mapping is difficult. That is the central empirical message.
Show code
strategy_returns = pd.DataFrame({name: res.net_returns for name, res in final_backtests.items()}).dropna(how="all")nav = pd.DataFrame({name: res.net_values for name, res in final_backtests.items()}).dropna(how="all")fig, axes = plt.subplots(1, 3, figsize=(18, 4.2))plot_strategy_nav(nav, ax=axes[0], title="Final test NAV")plot_strategy_drawdowns(nav, ax=axes[1], title="Final test drawdowns")plot_rolling_sharpe(axes[2], strategy_returns, window=126, rf_daily=rf_daily, title="Rolling Sharpe")plt.tight_layout()plt.show()
The NAV and drawdown plots make the same story clearer. The forecast and regime models build larger wealth over the test period. The RL policies lag, especially after strong risk-on periods where their constraints and reward shaping keep them from fully chasing upside. SAC tracks risk assets more closely because it keeps high risky exposure, but that also gives it the largest RL drawdown.
The active-return plot versus Forecast-Gated MaxSharpe is negative for the RL policies. This is consistent with the active performance table that follows. RL is not adding active alpha relative to the strongest forecast-driven strategy. The policy is learning a valid control rule, but the existing forecast-gated optimizer remains stronger in this historical test.
Show code
best_policy_name = final_summary.loc[[x for x in final_summary.index if x in ["PPO", "Recurrent PPO", "SAC"]], "Sharpe"].idxmax()fig, ax = plt.subplots(figsize=(10, 3.8))active_return_curve(ax, strategy_returns, strategy=best_policy_name, benchmark="Forecast-Gated MaxSharpe", title=f"{best_policy_name} active return vs Forecast-Gated MaxSharpe")plt.show()print("best policy:", best_policy_name)
best policy: SAC
The best RL policy by Sharpe is SAC, and the SAC active return plot versus Forecast-Gated MaxSharpe is mostly below zero. The active curve falls sharply during parts of the test window and never fully recovers. That tells us the underperformance is not one isolated month. It is a persistent active-return gap.
This result is important for the repo. The point of a final project like this is not to pretend RL always wins. The point is to show the full machinery, train the policies correctly, compare them honestly, and learn where RL struggles in real financial allocation.
Show code
def active_performance(strategy_returns, benchmark): R = strategy_returns.copy() rows = [] b = R[benchmark]for name in R.columns:if name == benchmark:continue active = (R[name] - b).dropna() te = active.std(ddof=1) * np.sqrt(252.0) active_ann = active.mean() *252.0 active_nav = (1+ active).cumprod() rows.append({"Strategy": name,"Active CAGR": active_ann,"Tracking Error": te,"Information Ratio": active_ann / te if te >0else np.nan,"Active Max Drawdown": (active_nav / active_nav.cummax() -1).min(),"Monthly Active Hit Rate": (active.resample("ME").sum() >0).mean(), })return pd.DataFrame(rows).set_index("Strategy")active_table = active_performance(strategy_returns, "Forecast-Gated MaxSharpe")display(active_table.round(4))
Active CAGR
Tracking Error
Information Ratio
Active Max Drawdown
Monthly Active Hit Rate
Strategy
Equal Weight
-0.0225
0.0696
-0.3239
-0.2724
0.4333
RandomForest Kelly
-0.0019
0.0700
-0.0275
-0.2265
0.4444
ML Regime-Aware
-0.0201
0.0805
-0.2497
-0.2425
0.4333
PPO
-0.0484
0.0647
-0.7477
-0.3918
0.3889
Recurrent PPO
-0.0464
0.0678
-0.6838
-0.3775
0.3556
SAC
-0.0327
0.0709
-0.4615
-0.3257
0.4778
Ensemble RL
-0.0389
0.0665
-0.5851
-0.3506
0.4111
Benchmark-relative results
The active performance table confirms that every strategy underperforms Forecast-Gated MaxSharpe on active CAGR. RandomForest Kelly is close, with active CAGR -0.19% and information ratio -0.0275. The RL models are farther behind:
PPO active CAGR: -4.84%,
Recurrent PPO active CAGR: -4.64%,
SAC active CAGR: -3.27%,
Ensemble RL active CAGR: -3.89%.
SAC has the best monthly active hit rate among the RL models at 47.78%, but it is still below 50%. Recurrent PPO has the weakest active hit rate at 35.56%.
Tracking error is moderate, around 6.5% to 7.1% for the RL models. This means the RL policies are meaningfully different from Forecast-Gated MaxSharpe, but the difference is not rewarded with positive active return. That is the key benchmark-relative result.
The benchmark-relative table compares every strategy against Forecast-Gated MaxSharpe. Since Forecast-Gated is the strongest rule-based optimizer in this project, it is a tough benchmark. The RL models have negative active CAGR and negative information ratios versus it. SAC has the least bad monthly hit rate among RL policies, around 47.8%, but still negative active return.
That means the RL policy occasionally adds value but not consistently enough to beat the strong forecast-gated allocation. This is where interpretation matters. A weak IR doesn’t mean the RL machinery is wrong. It means this specific RL training setup, reward design, sample size, and policy architecture didn’t extract enough additional edge beyond the supervised forecast stack.
The policy diagnostics explain the behavior behind those numbers. PPO and recurrent PPO maintain effective diversification around 9 to 11 assets and keep substantial cash. SAC has effective diversification around 6 assets and much higher maximum weight. SAC takes more decisive bets, which helps its return but hurts drawdown. The diagnostics are therefore consistent with the performance table.
The policy diagnostics table explains the personalities of the three RL models.
PPO has average risky exposure 78.31%, average cash 21.69%, turnover 14.57%, and effective number 9.11. This is the middle policy: it takes risk, keeps a cash buffer, and stays reasonably diversified.
Recurrent PPO has average risky exposure 72.73%, average cash 27.27%, turnover 13.14%, and effective number 11.15. This is the most diversified and defensive policy. The memory model avoids concentration, but that defensive behavior limits upside.
SAC has average risky exposure 96.10%, average cash 3.90%, turnover 17.38%, and effective number 6.01. This is the aggressive policy. It takes more concentrated risk and keeps very little cash. That explains why it has the best RL CAGR but also a weaker drawdown profile.
The exposure and weight-area plots visually confirm this. PPO and recurrent PPO keep a visible SHY allocation. SAC mostly pushes toward risky assets, with higher weights in SPY, HYG, IWM, and EFA through parts of the sample.
14) Ablation and Sensitivity Checks
The ablation table tests how the best policy behaves when some state information is removed or corrupted. The three ablations are:
no_forecasts: forecast-related features are set to zero,
no_priors: prior-weight features are set to zero,
shuffled_forecasts: forecast features are shuffled through time.
The result is a little surprising. no_forecasts has Sharpe 0.4782, which is better than the base SAC Sharpe 0.4263 in this rollout diagnostic. shuffled_forecasts drops to Sharpe 0.2654, which shows that corrupted forecast timing hurts. no_priors is identical to base, suggesting the selected best policy either didn’t receive meaningful prior features in the relevant state block or didn’t use them in a way that changed the deterministic rollout.
This is a good example of why ablation is necessary. A model can include many impressive features, but the trained policy may rely on only some of them. In this case, the forecast features are not clearly beneficial to the best RL policy, even though they are very useful for the rule-based Forecast-Gated MaxSharpe strategy. The same signal can help one decision rule and confuse another.
Ablation is where we ask which inputs the best policy is actually using. The best RL policy is SAC, so the notebook tests several modified states:
no forecast features,
no prior-policy features,
shuffled forecast features,
base configuration.
The logic is straightforward. If removing forecasts doesn’t hurt, the SAC policy wasn’t using them. If shuffling forecasts hurts, the policy was relying on their time alignment. If removing priors hurts, the policy was using information from the earlier rule-based systems.
The ablation table suggests that the policy is sensitive to the forecast/prior stack, but the relationship isn’t perfectly clean. The shuffled-forecast version performs worse in Sharpe than the no-forecast version in the output, which suggests that wrong timing in forecast information can be actively harmful. This is an important practical lesson: noisy features are one problem, but misaligned features are worse. They can teach the policy a false mapping between state and action.
This is also why the anti-leakage and alignment checks matter so much. RL can learn shortcuts, but it can also learn bad shortcuts from features that look predictive inside one historical path and fail out of sample.
14.1 Forecast alignment as an RL failure mode
The shuffled-forecast ablation is one of the most important checks because RL policies can exploit forecast features in subtle ways. If forecasts are useful only because they are correctly aligned in time, shuffling them should hurt. If shuffling doesn’t hurt, the policy probably wasn’t using them meaningfully.
The output shows that shuffled forecasts weaken the SAC variant. That means the forecast timing contains information the policy was using. At the same time, the no-forecast and no-prior variants don’t collapse completely, which means SAC also learns from direct market features, previous weights, and global context.
This is the kind of diagnostic that makes the project credible. Instead of only reporting final performance, we test whether the policy depends on the information sources it is supposed to depend on. In finance ML, this is crucial because a model can look good for the wrong reason. Ablation helps separate real signal use from accidental backtest behavior.
The VaR and ES table gives another view of risk. ML Regime-Aware has the lowest historical 5% VaR and ES among the main strategies: hist VaR about 0.91% and hist ES about 1.51%. Recurrent PPO is also relatively controlled, with hist VaR 0.94% and hist ES 1.52%. Forecast-Gated MaxSharpe has larger tail numbers, especially under filtered historical simulation, with FHS VaR 2.40% and FHS ES 3.47%.
SAC has hist VaR 1.18% and hist ES 1.95%, worse than PPO and recurrent PPO but not wildly worse than the rule-based risk-taking models. This matches its aggressive exposure behavior.
The stress table is more revealing. During the 2022 rates/inflation shock, Forecast-Gated MaxSharpe loses only -10.09%, the best result in that window. SAC loses -21.35%, PPO loses -20.38%, and Ensemble RL loses -20.63%. Recurrent PPO is better than the other RL models at -16.90%, helped by its more defensive exposure.
During the COVID crash, RandomForest Kelly loses only -3.71%, ML Regime-Aware loses -6.02%, and Forecast-Gated MaxSharpe loses -8.51%. SAC loses -12.54%, which is the weakest in that stress window. Again, the high-risky-exposure policy pays for its aggressiveness.
During the 2023 rebound, RandomForest Kelly performs best with 16.61%, while SAC earns 13.46% and equal weight earns 13.98%. SAC participates in the rebound, but it doesn’t capture enough extra upside to offset its earlier stress losses.
The stress-window results show the real personality of each policy. In the 2022 rates/inflation shock, Forecast-Gated MaxSharpe has the smallest loss among the displayed strategies. ML Regime-Aware also handles the episode better than Equal Weight and most RL policies. SAC loses more than the other RL models because it carries high risky exposure and more concentration. Recurrent PPO is more defensive and loses less than SAC, but its long-run return is weaker.
During the 2023 rebound, RandomForest Kelly and Equal Weight do well because broad risk exposure is rewarded. SAC also participates, but not enough to beat the best non-RL rebound exposure. During COVID, RandomForest Kelly and ML Regime-Aware are more resilient, while SAC suffers a larger drawdown.
The risk report confirms the same pattern through VaR and ES. ML Regime-Aware has the strongest filtered historical tail behavior in the table, with lower FHS VaR/ES than the RL strategies. PPO and recurrent PPO have moderate tail risk because they keep more cash and diversification. SAC has higher tail exposure, consistent with its aggressive allocation.
This is a strong empirical ending for the main project. RL gives us a powerful framework and flexible policies, but the final risk report keeps us honest. The best-performing strategy in the test window is still the model with a clear regime-aware structure and strong supervised inputs. The RL policies are useful experiments, and SAC is the strongest of them, but they need stronger sample design, more episodes, better reward calibration, or more robust offline-RL methods before they can be treated as superior allocation engines.
Tail risk interpretation
VaR and ES are especially important for RL policies because an agent can improve average reward while creating hidden tail risk. Historical VaR asks for a loss quantile:
\[
VaR_{5\%} = -q_{5\%}(r_p)
\]
Expected shortfall asks for the average loss once returns are already in the bad tail:
\[
ES_{5\%}= -E[r_p \mid r_p \le q_{5\%}(r_p)]
\]
ES is more informative because it measures tail severity, not only the cutoff. In the table, SAC has higher historical ES than PPO and recurrent PPO, consistent with its higher risky exposure. ML Regime-Aware has much better filtered historical tail risk, which reinforces its strong final ranking.
The stress windows add economic context. The 2022 period punishes duration and broad risk assets. The 2023 rebound rewards risk-taking. COVID tests crash response and recovery behavior. Looking at these windows keeps us from over-trusting full-sample Sharpe. A strategy can have a good full-period Sharpe while failing badly in a specific macro shock.
For RL, this stress analysis is the reality check. The agent is trained on a shaped reward, but investors experience realized NAV, drawdown, and tail loss. If reward improves while stress losses worsen, the policy isn’t actually better for a real portfolio.
Reading this final project as a research result
The main result is not that RL wins. The main result is that a complete RL portfolio layer can be built, trained, diagnosed, and compared honestly inside the same research framework as the earlier models.
A weak RL project would only show a rising reward curve and a final NAV. This notebook goes further. It shows:
state construction and leakage checks,
action-to-weight feasibility,
reward decomposition,
PPO/RPPO/SAC training diagnostics,
validation exposure and weight behavior,
test-period comparison against strong baselines,
benchmark-relative metrics,
policy behavior diagnostics,
ablations,
VaR/ES and stress windows,
a sector ETF transfer through the library.
That makes the conclusion much more trustworthy. The RL models don’t beat the best rule-based systems, but we can understand why. PPO and RPPO are too defensive. SAC takes more rewarded risk but suffers in stress. The ensemble smooths but doesn’t create enough new edge. The earlier supervised allocation stack remains stronger in this sample.
For future improvement, the most promising directions are clear: better offline-RL validation, more simulated episodes, regime-conditioned reward parameters, stronger uncertainty-aware critics, and maybe pretraining the policy to imitate strong baselines before RL fine-tuning. Those directions come naturally from the diagnostics rather than from guessing.
The risk-report plots show the same relationships in visual form. The stress-window bars make it easy to see that the rule-based ML strategies handle the 2022 shock better than RL. The scatter-style diagnostics show the strategies are still highly correlated because they all allocate among the same liquid ETFs. The correlation heatmap is mostly positive and high, so the differences are not coming from completely unrelated return streams. They are coming from exposure level, concentration, cash usage, and timing.
This is one of the clearest lessons of the project. RL can learn a policy, but if the policy shares the same asset universe and state information as stronger rule-based models, it has to be genuinely better at timing and sizing. In this sample, it isn’t.
16) Sector ETF Secondary Implementation
The final cell repeats the RL workflow on sector ETFs using the library implementation. The sector data is separate from the cross-asset ETF file:
This is a different allocation problem from the main cross-asset environment. In the main universe, the agent can move across stocks, bonds, credit, gold, commodities, and cash. In the sector universe, almost everything is equity beta. The agent can rotate between cyclicals, defensives, growth, financials, energy, and materials, but it can’t fully escape the equity market except through SHY.
The sector implementation uses HistGradientBoosting forecasts instead of the main TCN/RandomForest blend. It then builds forecast-gated MaxSharpe, Kelly, and regime-aware priors, constructs an RL state with asset, global, regime, VIX, and forecast features, and trains PPO, recurrent PPO, and SAC through the packaged quantfinlab RL API.
16.1 Sector allocation as a stricter RL test
The sector version uses the same RL idea on a narrower universe. The assets are:
This universe is harder in a different way. In the cross-asset project, the agent can manage risk by moving across equities, duration, credit, commodities, gold, and SHY. In sectors, most risky assets share equity beta. A defensive sector allocation can reduce drawdown, but it can’t diversify like bonds or gold. The agent has to learn relative equity rotation:
XLK and XLY for growth leadership,
XLE and XLB for inflation and commodity-linked environments,
The sector state includes 58 asset features and 34 global features, and the leakage diagnostic reports zero leaky forecast features. This is the important confirmed output. It means the library implementation is using a clean feature set before training the sector RL policies.
Since the saved notebook output only displays the leakage diagnostic first, we shouldn’t write exact sector performance numbers here. When the notebook is executed fully, the sector cell trains PPO, recurrent PPO, and SAC through the packaged quantfinlab RL API, creates validation summaries, builds weight frames, compares strategies, and generates the risk report. The markdown interpretation should therefore focus on what the implementation is designed to test: whether the same RL control layer can transfer from cross-asset allocation to equity-sector allocation when the diversification structure is much weaker.
That transfer test is the right final note for the series. The repo has moved from risk measures, optimization, options, term structure, macro, factors, regimes, networks, forecasting, and Kelly sizing into a final RL allocation layer. The strongest conclusion is practical: RL is a flexible portfolio-control framework, but finance still rewards strong feature design, conservative validation, explicit risk controls, and honest comparison against simpler models.
16.2 Library implementation and reproducibility
The sector cell uses the packaged quantfinlab implementation instead of local one-off code. That matters because Project 20 is not only a research notebook; it is also the final demonstration that the repo now contains reusable RL portfolio tools.
The library workflow mirrors the main notebook structure:
build forecasts and state features,
check leakage,
construct the portfolio environment,
train PPO, recurrent PPO, and SAC,
convert raw actions into feasible weights,
backtest policies,
compare against baselines,
generate the risk report.
The confirmed diagnostic tells us the sector state has zero leaky forecast features. That is the minimum condition before trusting any sector RL result. The asset and global feature counts are larger than in the main notebook, which means the sector implementation includes a rich state representation.
The sector version is also a good bridge to future repo development. Once the RL environment and policy classes work for both cross-asset allocation and sector allocation, they can be reused for other universes: international ETFs, country ETFs, factor ETFs, crypto baskets, or custom institutional portfolios. The main thing that changes is the state design and the reward calibration.
For this final project, that is the real contribution. We don’t only train three RL models once. We build an allocation framework where forecasts, regimes, risk controls, and neural policies can interact inside one reusable sequential decision system.
The sector extension also protects against one narrow interpretation of the project. If the RL layer only worked in the main cross-asset notebook, it might be a one-off experiment. By repeating the workflow through the package on sector ETFs, we show that the abstraction is reusable: state builder, environment, policies, training loop, weight conversion, backtest, and risk report. The details change, but the learning framework remains the same.
That is a strong ending for the whole series because it connects research notebooks with library design. The notebook teaches the models, and the package makes them reusable.
The saved output for the sector cell shows the leakage diagnostic first:
leaky forecast features in state: 0,
asset features: 58,
global features: 34.
That tells us the sector RL state has more features than the main cross-asset state, but the target columns are still excluded. The rest of the cell trains the library versions of PPO, recurrent PPO, and SAC, creates policy weights, compares them with sector baselines, displays validation and backtest tables, and produces the risk report. Since the saved notebook output only includes the first diagnostic table, we should not invent exact sector performance numbers from this file. The important confirmed result is that the library-level sector state construction passes the same anti-leakage check before policy training begins.
Economically, the sector version is a stricter test of RL timing. Cross-asset allocation can hide in bonds or gold during bad equity regimes. Sector allocation mostly has to manage relative equity exposure: technology versus energy, defensives versus cyclicals, financials versus staples, and so on. That makes the action space narrower but the signal interpretation more subtle. A good sector RL policy has to learn when the forecast and regime information deserves a sector rotation, when to keep diversified exposure, and when to shift risk into SHY.
