Back
 JAMP  Vol.6 No.4 , April 2018
A Study on the Convergence of Gradient Method with Momentum for Sigma-Pi-Sigma Neural Networks
Abstract: In this paper, a gradient method with momentum for sigma-pi-sigma neural networks (SPSNN) is considered in order to accelerate the convergence of the learning procedure for the network weights. The momentum coefficient is chosen in an adaptive manner, and the corresponding weak convergence and strong convergence results are proved.
Cite this paper: Zhang, X. and Zhang, N. (2018) A Study on the Convergence of Gradient Method with Momentum for Sigma-Pi-Sigma Neural Networks. Journal of Applied Mathematics and Physics, 6, 880-887. doi: 10.4236/jamp.2018.64075.
References

[1]   Shin, Y. and Ghosh, J. (1991) The Pi-Sigma Network: An Efficient Higher-Order Neural Network for Pattern Classification and Function Approximation. International Joint Conference on Neural Networks, 1, 13-18.
https://doi.org/10.1109/IJCNN.1991.155142

[2]   Li, C.K. (2003) A Sigma-Pi-Sigma Neural Network (SPSNN). Neural Processing Letters, 17, 1-19.
https://doi.org/10.1023/A:1022967523886

[3]   Rumelhart, D.E., Hinton, G.E. and Williams, R.J. (1986) Learning Representations by Back-Propagating Errors. Nature, 323, 533-536.
https://doi.org/10.1038/323533a0

[4]   Phansalkar, V.V. and Sastry, P.S. (1994) Analysis of the Back-Propagation Algorithm with Momentum. IEEE Transactions on Neural Networks, 5, 505-506.
https://doi.org/10.1109/72.286925

[5]   Torii, M. and Hagan, M.T. (2002) Stability of Steepest Descent with Momentum for Quadratic Functions. IEEE Transactions on Neural Networks, 13, 752-756.
https://doi.org/10.1109/TNN.2002.1000143

[6]   Bhaya, A. and Kaszkurewicz, E. (2004) Steepest Descent with Momentum for Quadratic Functions Is a Version of the Conjugate Gradient Method. Neural Networks, 17, 65-71.
https://doi.org/10.1016/S0893-6080(03)00170-9

[7]   Qian, N. (1999) On the Momentum Term in Gradient Descent Learning Algorithms. Neural Networks, 12, 145-151.
https://doi.org/10.1016/S0893-6080(98)00116-6

[8]   Shao, H. and Zheng, G. (2011) Convergence Analysis of a Back-Propagation Algorithm with Adaptive Momentum. Neurocomputing, 74, 749-752.
https://doi.org/10.1016/j.neucom.2010.10.008

[9]   Shao, H. and Zheng, G. (2011) Boundedness and Convergence of Online Gradient Method with Penalty and Momentum. Neurocomputing, 74, 765-770.
https://doi.org/10.1016/j.neucom.2010.10.005

[10]   Shao, H., Xu, D., Zheng, G. and Liu, L. (2012) Convergence of an Online Gradient Method with Inner-Product and Adaptive Momentum. Neurocomputing, 747, 243-252.
https://doi.org/10.1016/j.neucom.2011.09.003

[11]   Xu, D., Shao, H. and Zhang, H. (2012) A New Adaptive Momentum Algorithm for Split-Complex Recurrent Neural Networks. Neurocomputing, 93, 133-136.
https://doi.org/10.1016/j.neucom.2012.03.013

[12]   Zhang, N., Wu, W. and Zheng, G. (2006) Convergence of Gradient Method with Momentum for Two-Layer Feedforward Neural Networks. IEEE Transactions Neural Networks, 17, 522-525.
https://doi.org/10.1109/TNN.2005.863460

[13]   Wu, W., Zhang, N., Li, Z., Li, L. and Liu, Y. (2008) Convergence of Gradient Method with Momentum for Back-Propagation Neural Networks. Journal of Computational Mathematics, 4, 613-623.

[14]   Gori, M. and Maggini, M. (1996) Optimal Convergence of On-Line Backpropagation. IEEE Transactions on Neural Networks, 7, 251-254.
https://doi.org/10.1109/72.478415

[15]   Yuan, Y. and Sun, W. (1997) Optimization Theory and Methods. Science Press, Beijing.

[16]   Yan, X. and Chao, Z. (2008) Convergence of Asynchronous Batch Gradient Method with Momentum for Pi-Sigma Networks. Mathematica Applicata, 21, 207-212.

 
 
Top