On spectral properties of steepest descent methods

In recent years, it has become increasingly clear that the critical issue in gradient methods is the choice of the step length, whereas using gradient as the search direction may lead to very effective algorithms, whose surprising behaviour has only been partially explained, mostly in terms of the spectrum of the Hessian matrix. On the other hand, the convergence of the classical Cauchy steepest descent (SD) method has been analysed extensively and related to the spectral properties of the Hessian matrix, but the connection with the spectrum of the Hessian has not been exploited much to modify the method in order to improve its behaviour. In this work, we show how, for convex quadratic problems, moving from some theoretical properties of the SD method, second-order information provided by the step length can be exploited to dramatically improve the usually poor practical behaviour of this method. This allows us to achieve computational results comparable with those of the Barzilai and Borwein algorithm, with the further advantage of monotonic behaviour.