Polish空间上的折扣马氏过程量子化策略的渐近优化

数学物理学报 ›› 2022, Vol. 42 ›› Issue (2): 594-604.

Polish空间上的折扣马氏过程量子化策略的渐近优化

吴晓¹,孔荫莹^2,*(),郭圳滨³

¹ 肇庆学院数学与统计学院广东肇庆 526061
² 广东财经大学智能财会管理学院广州 510320
³ 广发证券股份有限公司发展研究中心上海 200120

收稿日期:2021-03-04 出版日期:2022-04-26 发布日期:2022-04-18
通讯作者: 孔荫莹 E-mail:kongcoco@hotmail.com
基金资助:
国家自然科学基金(11961005);中山大学广东省计算科学重点实验室开放基金(2021021);广东省普通高校重点领域(新一代信息技术)基金(2020ZDZX3019);广州市科技计划项目(202102080420)

Asymptotic Optimality of Quantized Stationary Policies in Continuous-Time Markov Decision Processes with Polish Spaces

Xiao Wu¹,Yinying Kong^2,*(),Zhenbin Guo³

¹ School of Mathematics and Statistics, Zhaoqing University, Guangdong Zhaoqing 526061
² School of Intelligence Financial & Accounting Management, Guangdong University of Finance and Economics, Guangzhou 510320
³ Development Research Center, GF Securities Co Ltd, Shanghai 200120

Received:2021-03-04 Online:2022-04-26 Published:2022-04-18
Contact: Yinying Kong E-mail:kongcoco@hotmail.com
Supported by:
the NSFC(11961005);the Opening Project of Guangdong Province Key Laboratory of Computational Science at Sun Yat-sen University(2021021);the Guangdong University (New Generation Information Technology) Key Field Project(2020ZDZX3019);the Guangzhou Science and Technology Plan Project(202102080420)

摘要/Abstract

摘要：

该文研究了Polish空间上、带折扣因子的连续时间马尔可夫决策过程(CTMDPs) 的量子化平稳策略的渐近最优性问题. 首先, 建立了折扣最优方程(DOE) 及其解的存在性和唯一性. 其次, 在适当的条件下证明了最优确定性平稳策略的存在性. 此外, 为了对行动空间进行离散化, 构造了一列量子化策略, 利用有限行动空间的策略来逼近一般(Polish) 空间上的折扣CTMDPs最优平稳策略. 最后, 通过一个例子来说明该文的渐近逼近结果.

关键词: 连续时间马尔可夫决策过程, 依赖状态折扣因子, 折扣准则, 量子化平稳策略, 渐近最优性

Abstract:

In this paper, we study the asymptotic optimality of the quantized stationary policies for continuous-time Markov decision processes (CTMDPs) with Polish space and state-dependent discount factors. Firstly, the existence and uniqueness of the discounted optimal equation (DOE) and its solution are established. Secondly, the existence of the optimal deterministic stationary policies is proved under appropriate conditions. In addition, in order to discretize the action space, a series of quantization policies are constructed to approximate the optimal stationary policies of the discounted CTMDPs in general state (Polish) space by using the policies in finite action space. Finally, an example is given to illustrate the asymptotic approximation results of this paper.

Key words: Continuous-time Markov decision processes, State-dependent discount factors, Discounted criterion, Quantized stationary policies, Asymptotic optimality

中图分类号:

O211.6

吴晓,孔荫莹,郭圳滨. Polish空间上的折扣马氏过程量子化策略的渐近优化[J]. 数学物理学报, 2022, 42(2): 594-604.

Xiao Wu,Yinying Kong,Zhenbin Guo. Asymptotic Optimality of Quantized Stationary Policies in Continuous-Time Markov Decision Processes with Polish Spaces[J]. Acta mathematica scientia,Series A, 2022, 42(2): 594-604.

参考文献

1	Journals Doshi B T . Continuous-time control of Markov processes on an arbitrary state space: discounted rewards. Ann Stat, 1976, 4: 1219- 1235
2	Dynkin E B , Yushkevich A A . Controlled Markov Processes. New York: Springer, 1979
3	Feinberg E A . Continuous-time jump Markov decision processes: A discrete-event approach. Math Oper Res, 2004, 29: 492- 524
4	Guo X P . Continuous-time Markov decision processes with discounted rewards: The case of Polish spaces. Math Oper Res, 2007, 32 (1): 73- 87
5	Guo X P . Constrained optimization for average cost continuous-time Markov decision processes. IEEE Trans Autom Control, 2007, 52 (6): 1139- 1143
6	Guo X P , Hernández-Lerma O . Continuous-time Markov Decision Processes: Theory and Applications. Berlin: Springer, 2009
7	Guo X P , Song X Y . Discounted continuous-time constrained Markov decision processes in Polish spaces. Ann Appl Probab, 2011, 21 (5): 2016- 2049
8	Hernández-Lerma O , Lasserre J B . Discrete-Time Markov Control Processes. New York: Springer, 1996
9	Hernández-Lerma O , Lasserre J B . Further Topics on Discrete-Time Markov Control Processes. New York: Springer, 1999
10	Puterman M L . Markov Decision Processes. New York: Wiley, 1994
11	Feinberg E A , Shwartz A . Markov decision models with weighted discounted criteria. Math Oper Res, 1994, 19 (1): 152- 168
12	González-Hernández J , López-Martínez R R , Pérez-Hernández J R . Markov control processes with randomized discounted cost. Math Methods Oper Res, 2007, 65: 27- 44
13	Wu X , Guo X P . First passage optimality and variance minimization of Markov decision processes with varying discount factors. J Appl Probab, 2015, 52 (2): 441- 456
14	Wu X , Zhang J Y . Finite approximation of the first passage models for discrete-time Markov decision processes with varying discount factors. Discrete Event Dyn Syst, 2016, 26 (4): 669- 683
15	Hinderer K . Foundations of Non Stationary Dynamic Programming with Discrete Time Parameter. New York: Springer, 1970
16	Ye L E , Guo X P . Continuous-time Markov decision processes with state-dependent discount factors. Acta Appl Math, 2012, 121: 5- 27
17	Bertsekas D , Tsitsiklis J . Neuro-Dynammic Programming. Boston: Athena Scientific, 1996
18	Cavazos-Cadena R . Finite-state approximations for denumerable state discounted Markov decision processes. Appl Math Optim, 1986, 14: 1- 26
19	Dufour F , Prieto-Rumeau T . Finite linear programming approximations of constrained discounted Markov decision processes. SIAM J Control Optim, 2013, 51 (2): 1298- 1324
20	Wu X , Guo X P . Convergence of Markov decision processes with constraints and state-action dependent discount factors. Sci China Math, 2020, 63 (1): 167- 182
21	Saldi N , Linder T , Yuksel S . Asymptotic optimality and rates of convergence of quantized stationary policies in stochastic control. IEEE Transactions on Automatic Control, 2014, 60 (2): 553- 558
22	Saldi N , Linder T , Yuksel S . Near optimality of quantized policies in stochastic control under weak continuity conditions. J Math Anal Appl, 2016, 435: 321- 337
23	Lund R B , Meyn S P , Tweedie R L . Computable exponential convergence rates for stochastically ordered Markov processes. Ann Appl Probab, 1996, 6: 218- 237

[1]	温玉珍, 王绍琳. 均值-方差框架下保险公司和再保险公司的 Stackelberg 随机微分博弈[J]. 数学物理学报, 2025, 45(4): 1327-1353.
[2]	刘羽, 陈光淦, 李树勇. 随机 Kuramoto-Sivashinsky 方程行波解的非线性稳定[J]. 数学物理学报, 2025, 45(3): 790-806.
[3]	陈雅婷, 刘海燕, 陈密. 通货膨胀下的$\alpha$-鲁棒最优投资策略研究[J]. 数学物理学报, 2024, 44(6): 1617-1629.
[4]	汪秋分, 张树文. 具有恐惧效应和 Holling III 功能性反应的随机捕食-食饵系统[J]. 数学物理学报, 2024, 44(5): 1380-1391.
[5]	杨红, 张晓光. 单纯复形上的随机 SIS 传染病模型[J]. 数学物理学报, 2024, 44(5): 1392-1399.
[6]	王彦凯, 彭幸春. 带交易成本的时间一致风险控制与投资策略[J]. 数学物理学报, 2024, 44(5): 1400-1414.
[7]	范协铨,胡海娟,吴浩,叶印娜. 随机环境下两个上临界分支过程的参数比较[J]. 数学物理学报, 2023, 43(5): 1440-1470.
[8]	陈勇,李英,盛英,古象盟. 一类分数高斯噪声驱动的 Ornstein-Uhlenbeck 过程的参数估计: Hurst 参数 $H\in (0,\frac12)$[J]. 数学物理学报, 2023, 43(5): 1483-1518.
[9]	姚落根,刘欢,陈琪琼. VG扭曲算子在期权定价中的应用[J]. 数学物理学报, 2023, 43(5): 1575-1584.
[10]	陈勇,古象盟. 分数Ornstein-Uhlenbeck 过程最小二乘估计改进的Berry-Esséen 界[J]. 数学物理学报, 2023, 43(3): 855-882.
[11]	卢嘉鑫,董华. 基于4/2随机波动率模型考虑错误定价和保费退还条款的DC型养老金计划的均衡投资策略[J]. 数学物理学报, 2023, 43(3): 939-956.
[12]	黄玲,刘海燕,陈密. 基于Ornstein-Uhlenbeck过程下具有两个再保险公司的比例再保险与投资[J]. 数学物理学报, 2023, 43(3): 957-969.
[13]	陈叶君, 丁惠生. 无穷维随机微分方程的渐近概周期解[J]. 数学物理学报, 2023, 43(2): 341-354.
[14]	赵彦军,孙晓辉,苏丽,李文轩. 具有Logistic增长和Beddington-DeAngelis发生率的随机SIRS传染病模型定性分析[J]. 数学物理学报, 2022, 42(6): 1861-1872.
[15]	范楠,王才士,姬红. Bernoulli泛函上典则酉对合的扰动[J]. 数学物理学报, 2022, 42(4): 969-977.