特征提取中一类矩阵迹函数极值问题的黎曼优化算法

数学物理学报 ›› 2024, Vol. 44 ›› Issue (4): 1012-1036.

特征提取中一类矩阵迹函数极值问题的黎曼优化算法

李姣芬,孔鲁源,宋佳铄,文娅琼^*()

桂林电子科技大学数学与计算科学学院, 广西高校数据分析与计算重点实验室广西桂林 541004; 广西应用数学中心 (桂林电子科技大学) 广西桂林 541004

收稿日期:2023-10-10 修回日期:2023-12-22 出版日期:2024-08-26 发布日期:2024-07-26
通讯作者: *文娅琼, E-mail: pmxdgmcdsy@163.com
基金资助:
国家自然科学基金(12261026);国家自然科学基金(12361079);国家自然科学基金(11961012);国家自然科学基金(12201149);广西自然科学基金(2023GXNSFAA026067);桂林电子科技大学研究生教育创新计划项目(2022YXW01);桂林电子科技大学研究生教育创新计划项目(2022YCXS142);广西研究生教育创新计划项目(YCSW2023316);广西自动检测技术与仪器重点实验室基金(YQ23104);广西自动检测技术与仪器重点实验室基金(YQ24105)

A Riemannian Optimization Approach for a Class of Matrix Trace Function Extremum Problem in Feature Extraction

Li Jiaofen,Kong Lvyuan,Song Jiashuo,Wen Yaqiong^*()

School of Mathematics and Computational Science, Guangxi Colleges and Universities Key Laboratory of Data Analysis and Computation, Guilin University of Electronic Technology, Guangxi Guilin 541004; Center for Applied Mathematics of Guangxi (GUET), Guangxi Guilin 541004

Received:2023-10-10 Revised:2023-12-22 Online:2024-08-26 Published:2024-07-26
Supported by:
National Natural Science Foundation of China(12261026);National Natural Science Foundation of China(12361079);National Natural Science Foundation of China(11961012);National Natural Science Foundation of China(12201149);Natural Science Foundation of Guangxi(2023GXNSFAA026067);Innovation Project of GUET Graduate Education(2022YXW01);Innovation Project of GUET Graduate Education(2022YCXS142);Innovation Project of Guangxi Graduate Education(YCSW2023316);Guangxi Key Laboratory of Automatic Detecting Technology and Instruments(YQ23104);Guangxi Key Laboratory of Automatic Detecting Technology and Instruments(YQ24105)

摘要/Abstract

摘要：

研究来源于特征提取中的一类鲁棒判别回归模型, 该模型问题可以重构为 Stiefel 流形和线性流形组成的乘积流形约束下的一类矩阵迹函数极小化问题. 整合紧流形和线性流形, 结合乘积流形几何性质, 本文设计适用于求解重构问题简化版本的一类基于 Zhang-Hager 技术拓展的乘积流形黎曼非线性共轭梯度法, 并给出算法全局收敛性分析. 数据实验表明所提算法对于问题求解是高效可行的, 且与已有算法、其它黎曼梯度类算法及黎曼优化工具箱中已有的黎曼一阶和二阶算法相比在迭代解精度或迭代效率上有一定优势.

关键词: 特征提取, 矩阵迹函数, 乘积流形, 黎曼共轭梯度法

Abstract:

The present study focuses on robust discriminant regression models for feature extraction, which can be rephrased as minimizing matrix trace function subject to product manifold constraints. By building upon the Zhang-Hager technique, the authors develop a Riemannian nonlinear conjugate gradient method for solving a simplified version of the reconstruction problem. The method exploits the geometric properties of the product manifold, and the global convergence analysis of the proposed algorithm is provided. Empirical results demonstrate that the proposed algorithm is effective and feasible for solving the underlying problem. In terms of iteration efficiency, the proposed algorithm outperforms the existing method, other Riemannian gradient-like algorithms and Riemannian first-order and second-order algorithms available in the MATLAB toolbox Manopt.

Key words: Feature extraction, Matrix trace function, Product manifold, Riemannian conjugate gradient method

中图分类号:

O151.1

李姣芬, 孔鲁源, 宋佳铄, 文娅琼. 特征提取中一类矩阵迹函数极值问题的黎曼优化算法[J]. 数学物理学报, 2024, 44(4): 1012-1036.

Li Jiaofen, Kong Lvyuan, Song Jiashuo, Wen Yaqiong. A Riemannian Optimization Approach for a Class of Matrix Trace Function Extremum Problem in Feature Extraction[J]. Acta mathematica scientia,Series A, 2024, 44(4): 1012-1036.

参考文献

[1]	Ye J. Least squares linear discriminant analysis//Proceedings of the 24th International Conference on Machine Learning. 2007: 1087-1093
[2]	Ma Z, Nie F, Yang Y, et al. Discriminating joint feature analysis for multimedia data understanding. IEEE Transactions on Multimedia, 2012, 14(6): 1662-1672
[3]	Ma Z, Yang Y, Sebe N, et al. Multimedia event detection using a classifier-specific intermediate representation. IEEE Transactions on Multimedia, 2013, 15(7): 1628-1637
[4]	Ji S, Tang L, Yu S, Ye J. A shared-subspace learning framework for multi-label classification. ACM Transactions on Knowledge Discovery from Data (TKDD), 2010, 4(2): 1-29
[5]	Seung H S, Lee D D. The manifold ways of perception. Science, 2000, 290(5500): 2268-2269
[6]	Roweis S T, Saul L K. Nonlinear dimensionality reduction by locally linear embedding. Science, 2000, 290(5500): 2323-2326
[7]	Belkin M, Niyogi P. Laplacian eigenmaps and spectral techniques for embedding and clustering//Advances in Neural Information Processing Systems. Cambridge: MIT press, 2001, 14
[8]	Zhang S, Ma Z, Tan H. On the equivalence of HLLE and LTSA. IEEE Transactions on Cybernetics, 2017, 48(2): 742-753
[9]	Lai Z, Mo D, Wong W K, et al. Robust discriminant regression for feature extraction. IEEE Transactions on Cybernetics, 2017, 48(8): 2472-2484
[10]	Nie F, Huang H, Cai X, Ding C H. Efficient and robust feature selection via joint $ L_{2, 1} $ -norms minimization//Advances in Neural Information Processing Systems, 2010: 1813-1821
[11]	Sato K, Sato H. Structure-preserving $ H^{2} $ optimal model reduction based on the Riemannian trust-region method. IEEE Transactions on Automatic Control, 2017, 63(2): 505-512
[12]	Sato H, Sato K. Riemannian gradient-based online identification method for linear systems with symmetric positive-definite matrix// 2019 IEEE 58th Conference on Decision and Control (CDC), 2019: 3593-3598
[13]	Sato K. Riemannian optimal model reduction of linear port-Hamiltonian systems. Automatica, 2018, 93: 428-434
[14]	Sato K. Riemannian optimal control and model matching of linear port-Hamiltonian systems. IEEE Transactions on Automatic Control, 2017, 62(12): 6575-6581
[15]	Sato K, Sato H, Damm T. Riemannian optimal identification method for linear systems with symmetric positive-definite matrix. IEEE Transactions on Automatic Control, 2020, 65(11): 4493-4508
[16]	Chiang C Y, Lin M M, Jin X Q. Riemannian inexact Newton method for structured inverse eigenvalue and singular value problems. BIT Numerical Mathematics, 2019, 59: 675-694
[17]	Ishteva M, Absil P A, Huffel S V, Lathauwer L D. Best low multilinear rank approximation of higher-order tensors, based on the Riemannian trust-region scheme. SIAM Journal on Matrix Analysis and Applications, 2011, 32(1): 115-135
[18]	Sato H, Iwai T. A Riemannian optimization approach to the matrix singular value decomposition. SIAM Journal on Optimization, 2013, 23(1): 188-212
[19]	Wang Y, Zhao Z, Bai Z J. Riemannian Newton-CG methods for constructing a positive doubly stochastic matrix from spectral data. Inverse Problems, 2020, 36(11): 115006
[20]	Yao T T, Bai Z J, Zhao Z, Ching W K. A riemannian fletcher-reeves conjugate gradient method for doubly stochastic inverse eigenvalue problems. SIAM Journal on Matrix Analysis and Applications, 2016, 37(1): 215-234
[21]	Yao T T, Bai Z J, Jin X Q, Zhao Z. A geometric Gauss-Newton method for least squares inverse eigenvalue problems. BIT Numerical Mathematics, 2020, 60: 825-852
[22]	Zhao Z, Jin X Q, Bai Z J. A geometric nonlinear conjugate gradient method for stochastic inverse eigenvalue problems. SIAM Journal on Numerical Analysis, 2016, 54(4): 2015-2035
[23]	Zhao Z, Bai Z J, Jin X Q. A Riemannian inexact Newton-CG method for constructing a nonnegative matrix with prescribed realizable spectrum. Numerische Mathematik, 2018, 140(4): 827-855
[24]	Zhao Z, Jin X Q, Yao T T. A Riemannian under-determined BFGS method for least squares inverse eigenvalue problems. BIT Numerical Mathematics, 2022, 62(1): 311-337
[25]	Zhao Z, Yao T T, Bai Z J, Jin X Q. A Riemannian inexact Newton dogleg method for constructing a symmetric nonnegative matrix with prescribed spectrum. Numerical Algorithms, 2023, 92: 1951-1981
[26]	Jiang Y L, Xu K L. Riemannian modified Polak-Ribière-Polyak conjugate gradient order reduced model by tensor techniques. SIAM Journal on Matrix Analysis and Applications, 2020, 41(2): 432-463
[27]	Oviedo H. Global convergence of Riemannian line search methods with a Zhang-Hager-type condition. Numerical Algorithms, 2022, 91(3): 1183-1203
[28]	Sato H. Riemannian Optimization and Its Applications. Brelin: Springer, 2021
[29]	Sato H, Iwai T. A new, globally convergent Riemannian conjugate gradient method. Optimization, 2015, 64(4): 1011-1031
[30]	Sato H. A Dai-Yuan-type Riemannian conjugate gradient method with the weak Wolfe conditions. Computational Optimization and Applications, 2016, 64(1): 101-118
[31]	Sakai H, Iiduka H. Hybrid Riemannian conjugate gradient methods with global convergence properties. Computational Optimization and Applications, 2020, 77(3): 811-830
[32]	Zhu X. A Riemannian conjugate gradient method for optimization on the Stiefel manifold. Computational optimization and Applications, 2017, 67(1): 73-110
[33]	Vandereycken B. Low-rank matrix completion by Riemannian optimization. SIAM Journal on Optimization, 2013, 23(2): 1214-1236
[34]	Zhang H, Hager W W. A nonmonotone line search technique and its application to unconstrained optimization. SIAM Journal on Optimization, 2004, 14(4): 1043-1056
[35]	Absil P A, Mahony R, Sepulchre R. Optimization Algorithms on Matrix Manifolds. Princeton: Princeton University Press, 2009
[36]	Ring W, Wirth B. Optimization methods on Riemannian manifolds and their application to shape space. SIAM Journal on Optimization, 2012, 22(2): 596-627
[37]	Absil P A, Mahony R, Trumpf J. An extrinsic look at the Riemannian Hessian//International Conference on Geometric Science of Information. Berlin: Springer, 2013: 361-368
[38]	Nocedal J, Wright S J. Numerical Optimization. Springer, 1999
[39]	Grippo L, Lampariello F, Lucidi S. A nonmonotone line search technique for Newton's method. SIAM Journal on Numerical Analysis, 1986, 23(4): 707-716
[40]	Zhang L, Zhou W, Li D. Global convergence of a modified Fletcher-Reeves conjugate gradient method with Armijo-type line search. Numerische Mathematik, 2006, 104(4): 561-572
[41]	王松桂, 吴密霞, 贾忠贞. 矩阵不等式. 北京: 科学出版社, 2006
[41]	Wang S G, Wu M X, Jia Z Z. Matrix Inequalities. Beijing: Science Press, 2006
[42]	Wen Z, Yin W. A feasible method for optimization with orthogonality constraints. Mathematical Programming, 2013, 142(1/2): 397-434
[43]	Li J F, Li W, Vong S W, et al. A Riemannian optimization approach for solving the generalized eigenvalue problem for nonsquare matrix pencils. Journal of Scientific Computing, 2020, 82: 1-43
[44]	Boumal N, Mishra B, Absil P A, Sepulchre R. Manopt, a Matlab toolbox for optimization on manifolds. Journal of Machine Learning Research, 2014, 15(1): 1455-1459