Bayesian estimation and optimization for high-dimensional portfolio selection

Abstract

This research incorporates Bayesian estimation and optimization into portfolio selection framework, particularly for high-dimensional portfolio in which the number of assets is strictly larger than the number of observations. We leverage a portfolio selection model, called Linear Programming Optimal (LPO) portfolio, which utilizes a constrained l1 minimization approach to directly estimate effective parameters appearing in the optimal portfolio. We propose 2 refinements for the existing LPO strategy. First, we explore improved estimators of returns mean and covariance matrix by utilizing Bayesian estimates instead of sample estimates. Second, we introduce Bayesian optimization (BO) to replace traditional grid search cross-validation (CV) in tuning hyperparameters of LPO strategy. We further propose modifications in the BO algorithm: (1) Taking into account time-dependent nature of financial problems and (2) Extending commonly used Expected Improvement (EI) acquisition function to include a tunable trade-off with the Improvement Variance (EIVar). Allowing the general case of noisy observations, we derive sub-linear convergence rate of Bayesian optimization under the newly proposed EIVar and prove theoretically that our algorithm has no-regret. Empirical studies confirm that our modified BO result in portfolio with higher out-of-sample profitability (Sharpe ratio, CEQ) and lower Turnover compared to those tuned with CV. This superior performance is achieved through fewer function evaluations, thus addressing time consuming issues of CV. Empirical studies also suggest that LPO with certain Bayesian estimates outperform LPO with sample estimates, as well as the benchmark Equally Weighted and Plug-in strategies.