A Unified Framework for Fast Large-Scale Portfolio Optimization

Abstract

We introduce a unified framework for rapid, large-scale portfolio optimization that incorporates both shrinkage and regularization techniques. This framework addresses multiple objectives, including minimum variance, mean-variance, and the maximum Sharpe ratio, and also adapts to various portfolio weight constraints. For each optimization scenario, we detail the translation into the corresponding quadratic programming (QP) problem and then integrate these solutions into a new open-source Python library. Using 50 years of return data from US mid to large-sized companies, and 33 distinct firm-specific characteristics, we utilize our framework to assess the out-of-sample monthly rebalanced portfolio performance of widely-adopted covariance matrix estimators and factor models, examining both daily and monthly returns. These estimators include the sample covariance matrix, linear and nonlinear shrinkage estimators, and factor portfolios based on Asset Pricing (AP) Trees, Principal Component Analysis (PCA), Risk Premium PCA (RP-PCA), and Instrumented PCA (IPCA). Our findings emphasize that AP-Trees and PCA-based factor models consistently outperform all other approaches in out-of-sample portfolio performance. Finally, we develop new ${\mathcal{l}}_1$ and ${\mathcal{l}}_2^2$ regularizations of factor portfolio norms which not only elevate the portfolio performance of AP-Trees and PCA-based factor models but they have a potential to reduce an excessive turnover and transaction costs often associated with these models.

Keywords:

• AP-trees
• IPCA
• ${\mathcal{l}}_1$ and ${\mathcal{l}}_2^2$ regularization
• RP-PCA
• mean and covariance matrix
• shrinkage estimators

Acknowledgements

We thank two anonymous referees and the Associate Editor for their insightful suggestions, which significantly enhanced the quality of our paper. We are also thankful for valuable comments by Andrew Mullhaupt, Milind Sharma, and Stan Uryasev, as well as the participants of the 2023 Winter School in Quantitative Finance at the University of Zurich, and QWAFAXNEW webinar series in New York.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Notes

1 The latest version of the code can be found at https://github.com/PawPol/PyPortOpt

2 In the empirical analysis we work with dividend and split adjusted simple returns.

3 Here, we use Dirichlet distributed random vectors to guarantee uniform sampling on the N dimensional simplex ($\mathbf{w}\prime {\mathbf{1}}_N=1$). The results for the weights sampled from uniform distribution normalized on the simplex, i.e. $\mathbf{w}=\mathbf{x}/(\mathbf{x}\prime {\mathbf{1}}_N),$ where $\mathbf{x}=[x_1,\dots ,x_N]$ and $x_i\stackrel{\mathit{\text{iid}}}{\sim }U([0,1]);$ and for the weights sampled from the absolute value of standard normal distribution normalized on the simplex, i.e. $\mathbf{w}=\left|\mathbf{x}\right|/{\Vert \mathbf{x}\Vert }_1,$ where $\mathbf{x}=[x_1,\dots ,x_N]$ and $x_i\stackrel{\mathit{\text{iid}}}{\sim }N(0,1)$ are similar.

4 We utilize Dirichlet distributed random vectors to ensure uniform sampling on the N dimensional simplex ($\mathbf{w}\prime {\mathbf{1}}_N=1$). The results from weights sampled from the uniform distribution normalized on the simplex (i.e. $\mathbf{w}=\mathbf{x}/(\mathbf{x}\prime {\mathbf{1}}_N),$ where $\mathbf{x}=[x_1,\dots ,x_N]$ and $x_i\stackrel{\mathit{\text{iid}}}{\sim }U([0,1]$); and from the weights derived from the absolute value of the standard normal distribution normalized on the simplex (i.e. $\mathbf{w}=\left|\mathbf{x}\right|/{\Vert \mathbf{x}\Vert }_1,$ where $\mathbf{x}=[x_1,\dots ,x_N]$ and $x_i\stackrel{\mathit{\text{iid}}}{\sim }N(0,1)$) align closely.

5 It is possible to further simplify the optimization problem from 3 N to 2 N variables by incorporating these constraints explicitly. But we tested this empirically, and it slows down the algorithms because one needs to use then the $2N\times 2N$ matrix instead of $\mathbf{\Sigma }$ in (8). The same argument applies to the similar optimization problems below.