Convergence analysis of online gradient method for BP neural networks

Neural Networks, Volume 24, Issue 1, January 2011, Pages 91-98

Convergence analysis of online gradient method for BP neural networks

Wei Wu (a), Jian Wang (a, b), Mingsong Chenga, Zhengxue Li (a)

wuweiw@dlut.edu.cn

a School of Mathematical Sciences, Dalian University of Technology, Dalian, 116024, PR China

b School of Mathematics and Computational Sciences, Petroleum University of China, Dongying, 257061, PR China

Abstract

Considers a class of online gradient learning methods for back propagation (BP) neural networks with a single hidden layer.

Assumes that in each training cycle, each sample in the training set is supplied in a stochastic order to the network exactly once. The stochastic learning methods can be shown to be deterministically convergent. The paper presents some weak and strong convergence results for the learning methods, indicating that the gradient of the error function goes to zero and the weight sequence goes to a fixed point, respectively. The conditions on the activation function and the learning rate to guarantee the convergence are relaxed compared with the existing results. Convergence results are valid for not only S–S type neural networks (both the output and hidden neurons are Sigmoid functions), but also for P–P, P–S and S–P type neural networks, where S and P represent Sigmoid and polynomial functions, respectively.

Full Text Source (Subscription or Fee): http://www.sciencedirect.com/science/article/pii/S0893608010001723