One of the assumptions of Modern Portfolio Theory (MPT) is that the mean and covariance matrix of returns are known. Clearly that is not the case and we have to use some estimates. The accuracy of the estimates used has a big effect on the resulting portfolio. In the previous article I used sample mean and covariance calculated over a rolling window. In this article I will try to improve the performance of optimal portfolios by using weighted estimators. Article is based on the paper ‘Estimating Covariance Matrices’ (Litterman, Winkelmann 1998), which describes estimation framework that was used at Goldman Sachs at the time.
Section II of the paper demonstrates several empirical regularities found in financial data:
- Volatility is time-varying.
- Correlations are time-varying.
- Financial data has fat-tails.
Let’s see if those regularities are present in recent data.
First I am going to take a look at volatility. I will use returns of SPY ETF from 01–01–2018 to 2022–12–31. I will compare historical standard deviation (calculated using all available data) with 100-day rolling standard deviation. Code for doing it is provided below.
We can see on the plot above that volatility estimates calculated using rolling window deviate a lot from historical volatility calculated using all data. Let’s take a look at some descriptive statistics of rolling volatility.
Rolling volatility ranges from 7.74% to 49.69% with mean value of 19.2%. But some deviations in rolling volatility are expected even if true volatility is constant. To see what deviations are expected let’s generate a random sample of normally distributed returns with the same mean and variance as SPY returns. Then we can compare two rolling volatilities by plotting them together.
As you can see on the plot above, rolling volatility of SPY has larger deviations from historical volatility than rolling volatility of the random sample with constant volatility. Below I calculate mean absolute deviations for both rolling volatilities and plot a histogram of absolute deviations.
We can see that absolute deviations of SPY volatility are bigger than that of the random sample. But what we did above involves just one random sample and we can’t really draw any conclusions from it. To resolve this issue we can perform Monte Carlo simulation. We want to answer the following question: what is the probability of getting such big deviations in rolling volatility from normally distributed random sample with constant volatility?
We do it as follows:
- Perform 10000 simulations.
- In each simulation calculate rolling volatility and its mean absolute deviations.
- Save the results.
Code for doing it is shown below.
Maximum mean absolute deviation out of 10000 samples was approximately 2.2%, which is smaller than mean absolute deviation for SPY (7.9%). We can conclude that there is less than 0.0001 chance that SPY rolling volatility was obtained from normal distribution with constant volatility of 21.55% (historical volatility of SPY). Actually it is a lot smaller than 0.0001. We could calculate the actual probability (p-value) using mean and standard deviation of the results, but knowing that it is very small is enough for our purposes.
Now let’s perform similar studies of correlations. We need another asset for this. I am going to use GLD ETF, which represents gold market. First we download the data and estimate historical correlation coefficient.
Historical correlation coefficient between SPY and GLD is approximately 0.1055. Basically it means that there is almost no correlation between the two assets. Now let’s calculate and plot 100-day rolling correlation.
On the screenshot above we can see that rolling correlation deviates a lot from historical correlation. It ranges from about -0.5 to +0.5 and we can see that there are periods of positive correlation, negative correlation or no correlation. This, of course, will have a huge impact on the optimal portfolio. Let’s compare this result with rolling correlation of a random sample as we did for volatility.
I’m going to generate a multivariate normal random sample with mean and covariance matrix of SPY and GLD returns. Then I will add rolling correlation of that random sample to the plot above.
Again we see that rolling correlation of SPY-GLD deviates from historical correlation significantly more than rolling correlation of the random sample. SPY-GLD has mean absolute deviation of 0.212 and random sample has mean absolute deviation of 0.067.
Now let’s perform Monte Carlo simulation to estimate how likely it is to get deviations similar to that of SPY-GLD. I generate 10000 random samples, calculate its 100-day rolling correlation and its mean absolute deviation from historical value.
Maximum value of mean absolute deviation out of 10000 samples is 0.135, which is less than mean absolute deviation of SPY-GLD (0.212). This means that the probability that returns of SPY and GLD are normally distributed with constant correlation of 0.1055 is less than 0.0001.
Another test in the paper is used to show that financial data has ‘fat tails’. It is demonstrated by counting the percentage of datapoints that are more than two standard deviations away from the long-term mean. If that value is more than 4.6% then distribution has ‘fat tails’. This number comes from the normal distribution, where 4.6% of returns are beyond 2 SD from the mean.
Let’s perform this test for the returns of both assets (SPY and GLD).
On the screenshot above we can see that about 4.53% of SPY returns are more than 2 SD from the mean. It is less than the required 4.6%, so we conclude that distribution of SPY returns does not have ‘fat tails’. For GLD we have 5.32% returns in the tails of the distribution, therefore GLD has ‘fat tails’.
So far we have concluded that variances and correlations of returns are not stable and vary over time. In that case, how should we estimate them? One approach is just to use rolling window with only last N data points for each estimate. I used this approach in my previous article. In the paper authors propose using weighted estimators, which give more weight to newer datapoints and less weight to older datapoints. Formula for calculating weighted covariance is provided below. Formula in the paper is a little different — it doesn’t contain expectations of returns because they are assumed to be zero.
I will also use weighted average of returns, which is calculated as follows.
If we want to give more weight to more recent observations, w_t should be a decaying function of time. In the paper the following function is proposed.
For example, if d=0.1 then each day receives 90% of the weight of the day after it.
Let’s try to compare weighted estimators with estimators using rolling window. In the code below I calculate correlations between SPY and GLD using both methods. I’m using decay rate of 0.023 and 252-day rolling window. Function for calculating weighted covariance matrix is already implemented in numpy. We just need to provide a vector of weights as an input variable.
On the plot above you can see both correlations together. 252-day rolling correlation looks smoother and doesn’t react to changes as quickly as weighted correlation. We can regulate this behaviour by adjusting decay rate. We should choose it based on our investment horizon. As the decay rate decreases, our investment horizon increases. On the screenshot below you can see a table from the paper showing decay rates for German 10-year bonds.
In the paper authors propose a method for estimating decay rate, but it is beyond the scope of this article. Interested reader is referred to Appendix B of the paper. I am just going to use decay rate of 0.023 (one week investment horizon in the table above).
Now let’s compare the performance of trading strategies with different estimators. We expect that strategies using weighted estimators will perform better than strategies using simple estimators. I’m going to use the following ETFs, representing different asset types:
- SPY — large-cap stocks.
- VWO — small-cap stocks.
- BNDX — bonds.
- USO — oil.
- GLD — gold.
Portfolio will be rebalanced weekly. Mean and covariance matrix will be estimated using daily data. (Using higher frequency data is another way to improve the accuracy of covariance estimates.)
First we download and prepare our data.
Now let’s take a look at historical correlation matrix.
ETFs representing stock market (SPY and VWO) are highly correlated with correlation coefficient of 0.77. Other assets are relatively uncorrelated to the stock market and to each other, which is a nice property for portfolio optimization based on MPT.
Now we are ready to implement our strategies. First let’s take a look at the performance metrics of equally weighted portfolio.
Next we will test other strategies and see if they provide better performance.
Recall that when we solve portfolio optimization problem using Modern Portfolio Theory (MPT) we have two options: either minimize volatility of the portfolio or maximize portfolio returns for a given level of volatility. To make a fair comparison I will use volatility of equally weighted portfolio as my target volatility. It is shown on the screenshot below.
Now we are going to implement our first strategy. Here I am going to use 252-day rolling window to estimate mean and covariance of returns. That’s the same strategy I used in my previous article about MPT. Code for backtesting it is shown below.
You can see above that this strategy performs significantly worse than equally weighted portfolio and its volatility is higher than target volatility of 15.8%.
Next we will backtest the same strategy, but with weighted estimators. To do it we need to implement new functions for calculating volatility and negative returns, which will be used for mean-variance optimization. These function are similar to the ones used before, just with weighted estimators. Note that I’m also using weighted mean for calculating average returns. Everything else stays the same.
As you can see on the screenshots below, performance metrics of this strategy are slightly better than performance metrics of equally weighted portfolio. And significantly better than performance metrics of the strategy using rolling window. Volatility of returns is also a little closer to the target volatility. Weighted estimators seem to improve the performance of MPT-based strategies.
In the previous article I’ve mentioned that we can easily modify portfolio optimization problem according to our needs by adding more constraints or changing objective function. Here I’d like to show an example of that.
In the strategies above we used volatility of equally weighted portfolio as a target volatility. We tried to maximize returns for that level of volatility. But what if we don’t have a strict limit for portfolio volatility? What if we are willing to accept more risk in exchange for higher returns? In that case we can modify portfolio optimization problem and try maximizing Sharpe ratio (which measures risk-adjusted returns) of the portfolio.
To do that we need to implement a function to calculate negative Sharpe ratio (which we will minimize). Code of the function is shown below. For now I’m working with simple mean and covariance estimators.
Now we are ready to perform a backtest. Strategy is basically the same as before, we just change an objective function and remove a constraint on volatility.
Using this technique (even with rolling windows) allows us to achieve better performance (compared to classic MPT with rolling window). What if we use weighted estimators? To do it we need to implement a function that calculates Sharpe ratio using weighted mean and covariance estimators. Code for it is provided below.
Now we are ready to perform a backtest. Again, everything is the same, we just need to change the objective function, which we are minimizing.
Performance metrics are shown in the table below. This strategy has the best performance so far in terms of Sharpe ratio and APR. It also has the shortest maximum drawdown duration.
Recall that I didn’t implement any methods for estimating decay rate. I just used a value from the paper. I think that we can further improve performance by using better values of decay rate. The simplest approach would be just to backtest several values and choose the one with the best performance. Another approach is to estimate decay rate using likelihood function (described in Appendix B of the paper).
We’ve seen that using weighted estimators allows us to improve performance in both types of trading strategies. Note that improving covariance estimates is important in many other areas of quantitative finance (not only for portfolio optimization). For example, we can try using weighted estimators in pairs trading strategies, which use correlation coefficients for pair selection.
In conclusion, even though using weighted estimators allows us to improve the performance of optimal portfolios (compared to simple estimators with rolling window), the performance of our strategies is still quite low. In the following articles I will continue exploring the topic of portfolio optimization and try to build trading strategies with better performance.
Jupyter notebook with source code is available here.
If you have any questions, suggestions or corrections please post them in the comments. Thanks for reading.
Yuan Di prepared a Chinese adaptation of this article, which is available here.
References
[1] Estimating Covariance Matrices (Litterman, Winkelmann, 1998)
[2] Quantitative Equity Investing (Fabozzi, 2010)
[3] Introduction to Portfolio Optimization and Modern Portfolio Theory
Leave a Reply