The Asymptotic Study of Smooth Entropy Support Vector Regression

Affiliation(s)

School of Electronics and Information, Shanghai Technical Institute of Electronics and Information, Shanghai, China.

School of Electronics and Information, Shanghai Technical Institute of Electronics and Information, Shanghai, China.

ABSTRACT

In this paper, a novel formulation, smooth entropy support vector regression (SESVR), is proposed, which is a smooth unconstrained optimization reformulation of the traditional linear programming associated with an ε-insensitive support vector regression. An entropy penalty function is substituted for the plus function to make the objective function con- tinuous ,and a new algorithm involving the Newton-Armijo algorithm proposed to solve the SESVR converge globally to the solution. Theoretically, we give a brief convergence proof to our algorithm. The advantages of our presented algorithm are that we only need to solve a system of linear equations iteratively instead of solving a convex quadratic program, as is the case with a conventional SVR, and lessen the influence of the penalty parameter C in our model. In order to show the efficiency of our algorithm, we employ it to forecast an actual electricity power short-term load. The experimental results show that the presented algorithm, SESVR, plays better precisely and effectively than SVMlight and LIBSVR in stochastic time series forecasting.

In this paper, a novel formulation, smooth entropy support vector regression (SESVR), is proposed, which is a smooth unconstrained optimization reformulation of the traditional linear programming associated with an ε-insensitive support vector regression. An entropy penalty function is substituted for the plus function to make the objective function con- tinuous ,and a new algorithm involving the Newton-Armijo algorithm proposed to solve the SESVR converge globally to the solution. Theoretically, we give a brief convergence proof to our algorithm. The advantages of our presented algorithm are that we only need to solve a system of linear equations iteratively instead of solving a convex quadratic program, as is the case with a conventional SVR, and lessen the influence of the penalty parameter C in our model. In order to show the efficiency of our algorithm, we employ it to forecast an actual electricity power short-term load. The experimental results show that the presented algorithm, SESVR, plays better precisely and effectively than SVMlight and LIBSVR in stochastic time series forecasting.

Cite this paper

G. Hu and J. Zhang, "The Asymptotic Study of Smooth Entropy Support Vector Regression,"*Intelligent Information Management*, Vol. 4 No. 3, 2012, pp. 45-51. doi: 10.4236/iim.2012.43007.

G. Hu and J. Zhang, "The Asymptotic Study of Smooth Entropy Support Vector Regression,"

References

[1] C. V. Gustavo, G. Juan and G. P. Gabriel, “On the Suitable Domain for SVM Training in Image Coding,” Journal of Machine Learning Research, Vol. 9, No. 1, 2008, pp. 49-66.

[2] F. Chang, C. Y. Guo and X. R. Lin, “Tree Decomposition for Large-Scale SVM Problems,” Journal of Machine Learning Research, Vol. 11, No. 10, 2010, pp. 2935-2972.

[3] Y. H. Kong, W. Cai Wei and W. He, “Power Quality Disturbance Signal Classification Using Support Vector Ma- chine Based on Feature Combination,” Journal of North China Electric Power University, Vol. 37, No. 4, 2010, pp. 72-77.

[4] J. Zhe, “Research on Power Load Forecasting Base on Sup- port Vector Machines,” Computer Simulation, 2010, No. 8, pp. 282-285.

[5] B. Chen and P. T. Harker, “Smooth Approximations to Non- linear Complementarity Problems,” SIAM Journal of Optimization, Vol. 7, No. 2, 1997, pp. 403-420. doi:10.1137/S1052623495280615

[6] C. H. Chen and O. L. Mangasarian, “Smoothing Methods for Convex Inequalities and Linear Complementarity Problems,” Mathematical Programming, Vol. 71, No. 1, 1995, pp. 51-69. doi:10.1007/BF01592244

[7] C. H. Chen and O. L. Mangasarian, “A Class of Smoothing Functions for Nonlinear and Mixed Complementarity Problems,” Computational Optimization and Applications, Vol. 5, No. 2, 1996, pp. 97-138. doi:10.1007/BF00249052

[8] X. Chen, L. Qi and D. Sun, “Global and Superlinear Convergence of the Smoothing Newton Method and Its Application to General Box Constrained Variational Inequali- ties,” Mathematics of Computation, Vol. 67, No. 222, 1998, pp. 519-540. doi:10.1090/S0025-5718-98-00932-6

[9] Y. J. Lee and O. L. Mangasarin, “SSVM: A Smooth Support Vector Machine for Classification,” Computational Optimization and Applications, Vol. 20, No. 1, 2010, pp. 5-22. doi:10.1023/A:1011215321374

[10] Z. Q. Meng, G. G. Zhou and Y. H. Zhu, “A Smoothing Support Vector Machine Based on Exact Penalty Function,” Lecture Notes in Artificial Intelligence, 2005, Vol. 3801, pp.568-573.

[11] Y. F. Fan, D. X. Zhang and H. C. He, “Tangent Circular Arc Smooth SVM（TCA-SSVM） Research,” 2008 Congress on Image and Signal Processing, Sanya, 27-30 May 2008, pp. 646-648. doi:10.1109/CISP.2008.112

[12] Y. F. Fan, D. X. Zhang and H. C. He, “Smooth SVM Research: A Polynomial-Based Approach,” The 9th International Conference on Information and Communications Security, Singapore, 10-13 December 2007, pp. 983-988.

[13] L. K. Lao, C. D. Lin, H. Peng, et al., “A Study on Piece- wise Polynomial Smooth Approximation to the Plus Function,” The 9th International Conference on Control, Auto- mation, Robotics and Vision, Singapore, 5-8 December 2066, pp. 342-347.

[14] J. Z. Xiong, T. M. Hu and G. G. Li, “A Comparative Study of Three Smooth SVM Classifiers,” Proceedings of the 6th World Congress on Intelligent Control and Automation, Dalian, 21-23 June 2006, pp. 5962-5966.

[15] P. A. Forero, A. C. Georgios and B. Giannakis, “Consensus- Based Distributed Support Vector Machines,” Journal of Machine Learning Research, Vol. 11, No. 5, 2010, pp. 1663- 1707.

[16] C. H. Chen and O. L. Mangasarian, “Smoothing Methods for Convex Inequalities and Linear Complementarity Problems,” Mathematical Programming, Vol. 71, No. 1, 1995, pp. 51-69. doi:10.1007/BF01592244

[17] Y. J. Lee, W. F. Hsieh and C. M. Huang, “ε-SSVR: A Smooth Support Vector Machine for ε-Insensitive Regression,” Knowledge and Data Engineering, Vol. 17, No. 5, 2005, pp. 678-695. doi:10.1109/TKDE.2005.77

[18] S. H. Peng and X. S. Li, “The asymptotic analysis of quasi-exact penalty function method,” Journal of Computational Mathematics, 2007, Vol. 29, No. 1, pp. 47-56.

[19] T. Joachims, “SVMlight,” 2010. http://svmlight. joachims .org

[20] C.-C. Chang and C.-J. Lin, “LIBSVM: A Library for Support Vector Machines,” 2010. http://www.csie.ntu. edu.tw/~cjlin/libsvm

[21] R. Fletcher, “Practical methods of optimization,” John Wiley & Sons, New York, 1981.

[22] O. L. Mangasarian and D. R. Musicant, “Successive over- relaxation for support vector machines,” IEEE Transactions on Neural Networks, Vol. 10, No. 5, 2010, pp. 1032- 1037. doi:10.1109/72.788643 ftp://ftp.cs.wisc.edu/math-prog/tech-reports/98-18.ps

[23] Y. W. Chang, C. J. Hsieh and K. W. Chang, “Training and Testing Low-degree Polynomial Data Mappings via Linear SVM,” Journal of Machine Learning Research, Vol. 11, No. 4, 2010, pp. 1471-1490.

[1] C. V. Gustavo, G. Juan and G. P. Gabriel, “On the Suitable Domain for SVM Training in Image Coding,” Journal of Machine Learning Research, Vol. 9, No. 1, 2008, pp. 49-66.

[2] F. Chang, C. Y. Guo and X. R. Lin, “Tree Decomposition for Large-Scale SVM Problems,” Journal of Machine Learning Research, Vol. 11, No. 10, 2010, pp. 2935-2972.

[3] Y. H. Kong, W. Cai Wei and W. He, “Power Quality Disturbance Signal Classification Using Support Vector Ma- chine Based on Feature Combination,” Journal of North China Electric Power University, Vol. 37, No. 4, 2010, pp. 72-77.

[4] J. Zhe, “Research on Power Load Forecasting Base on Sup- port Vector Machines,” Computer Simulation, 2010, No. 8, pp. 282-285.

[5] B. Chen and P. T. Harker, “Smooth Approximations to Non- linear Complementarity Problems,” SIAM Journal of Optimization, Vol. 7, No. 2, 1997, pp. 403-420. doi:10.1137/S1052623495280615

[6] C. H. Chen and O. L. Mangasarian, “Smoothing Methods for Convex Inequalities and Linear Complementarity Problems,” Mathematical Programming, Vol. 71, No. 1, 1995, pp. 51-69. doi:10.1007/BF01592244

[7] C. H. Chen and O. L. Mangasarian, “A Class of Smoothing Functions for Nonlinear and Mixed Complementarity Problems,” Computational Optimization and Applications, Vol. 5, No. 2, 1996, pp. 97-138. doi:10.1007/BF00249052

[8] X. Chen, L. Qi and D. Sun, “Global and Superlinear Convergence of the Smoothing Newton Method and Its Application to General Box Constrained Variational Inequali- ties,” Mathematics of Computation, Vol. 67, No. 222, 1998, pp. 519-540. doi:10.1090/S0025-5718-98-00932-6

[9] Y. J. Lee and O. L. Mangasarin, “SSVM: A Smooth Support Vector Machine for Classification,” Computational Optimization and Applications, Vol. 20, No. 1, 2010, pp. 5-22. doi:10.1023/A:1011215321374

[10] Z. Q. Meng, G. G. Zhou and Y. H. Zhu, “A Smoothing Support Vector Machine Based on Exact Penalty Function,” Lecture Notes in Artificial Intelligence, 2005, Vol. 3801, pp.568-573.

[11] Y. F. Fan, D. X. Zhang and H. C. He, “Tangent Circular Arc Smooth SVM（TCA-SSVM） Research,” 2008 Congress on Image and Signal Processing, Sanya, 27-30 May 2008, pp. 646-648. doi:10.1109/CISP.2008.112

[12] Y. F. Fan, D. X. Zhang and H. C. He, “Smooth SVM Research: A Polynomial-Based Approach,” The 9th International Conference on Information and Communications Security, Singapore, 10-13 December 2007, pp. 983-988.

[13] L. K. Lao, C. D. Lin, H. Peng, et al., “A Study on Piece- wise Polynomial Smooth Approximation to the Plus Function,” The 9th International Conference on Control, Auto- mation, Robotics and Vision, Singapore, 5-8 December 2066, pp. 342-347.

[14] J. Z. Xiong, T. M. Hu and G. G. Li, “A Comparative Study of Three Smooth SVM Classifiers,” Proceedings of the 6th World Congress on Intelligent Control and Automation, Dalian, 21-23 June 2006, pp. 5962-5966.

[15] P. A. Forero, A. C. Georgios and B. Giannakis, “Consensus- Based Distributed Support Vector Machines,” Journal of Machine Learning Research, Vol. 11, No. 5, 2010, pp. 1663- 1707.

[16] C. H. Chen and O. L. Mangasarian, “Smoothing Methods for Convex Inequalities and Linear Complementarity Problems,” Mathematical Programming, Vol. 71, No. 1, 1995, pp. 51-69. doi:10.1007/BF01592244

[17] Y. J. Lee, W. F. Hsieh and C. M. Huang, “ε-SSVR: A Smooth Support Vector Machine for ε-Insensitive Regression,” Knowledge and Data Engineering, Vol. 17, No. 5, 2005, pp. 678-695. doi:10.1109/TKDE.2005.77

[18] S. H. Peng and X. S. Li, “The asymptotic analysis of quasi-exact penalty function method,” Journal of Computational Mathematics, 2007, Vol. 29, No. 1, pp. 47-56.

[19] T. Joachims, “SVMlight,” 2010. http://svmlight. joachims .org

[20] C.-C. Chang and C.-J. Lin, “LIBSVM: A Library for Support Vector Machines,” 2010. http://www.csie.ntu. edu.tw/~cjlin/libsvm

[21] R. Fletcher, “Practical methods of optimization,” John Wiley & Sons, New York, 1981.

[22] O. L. Mangasarian and D. R. Musicant, “Successive over- relaxation for support vector machines,” IEEE Transactions on Neural Networks, Vol. 10, No. 5, 2010, pp. 1032- 1037. doi:10.1109/72.788643 ftp://ftp.cs.wisc.edu/math-prog/tech-reports/98-18.ps

[23] Y. W. Chang, C. J. Hsieh and K. W. Chang, “Training and Testing Low-degree Polynomial Data Mappings via Linear SVM,” Journal of Machine Learning Research, Vol. 11, No. 4, 2010, pp. 1471-1490.