O(N2)-Operation Approximation of Covariance Matrix Inverse in Gaussian Process Regression Based on Quasi-Newton BFGS Method

Authors: Leithead, W. E.; Zhang, Yunong

Source: Communications in Statistics: Simulation and Computation, Volume 36, Number 2, March 2007 , pp. 367-380(14)

Buy & download fulltext article:

Price: $56.94 plus tax (Refund Policy)

Abstract:

Gaussian process (GP) is a Bayesian nonparametric regression model, showing good performance in various applications. However, during its model-tuning procedure, the GP implementation suffers from numerous covariance-matrix inversions of expensive O(N3) operations, where N is the matrix dimension. In this article, we propose using the quasi-Newton BFGS O(N2)-operation formula to approximate/replace recursively the inverse of covariance matrix at every iteration. The implementation accuracy is guaranteed carefully by a matrix-trace criterion and by the restarts technique to generate good initial guesses. A number of numerical tests are then performed based on the sinusoidal regression example and the Wiener-Hammerstein identification example. It is shown that by using the proposed implementation, more than 80% O(N3) operations could be eliminated, and a typical speedup of 5-9 could be achieved as compared to the standard maximum-likelihood-estimation (MLE) implementation commonly used in Gaussian process regression.

Keywords: Gaussian process regression; Matrix inverse; Optimization; O(N2) operations; Quasi-Newton BFGS method

Document Type: Research article

DOI: http://dx.doi.org/10.1080/03610910601161298

Affiliations: 1: Hamilton Institute, National University of Ireland, Maynooth, Ireland

Publication date: 2007-03-01