A reinforcement learning algorithm using value iteration method is proposed for a linear-quadratic tracking problem with a target trajectory described by linear dynamics in continuous time. Convergence of the method is proved, and a superlinear convergence rate is established. Numerical results are presented, which confirm the effectiveness of the method.
Keywords: reinforcement learning, linear-quadratic tracking, value iteration, Riccati equation, quasi-Newton method
Received May 31, 2026
Revised August 5, 2026
Accepted August 10, 2026
Funding Agency: The research was carried out within the state assignment of Ministry of Science and Higher Education of the Russian Federation, project “Optimization methods of guarantee in control problems with uncertain parameters” (no. 126021317314-5).
Ilya Nikolaevich Smovzh, Lomonosov Moscow State University, Moscow, 119991 Russia, e-mail: smovzh.ilya@mail.ru
Nikolai Borisovich Melnikov, Dr. Phys.-Math. Sci., Prof., Lomonosov Moscow State University, Moscow, 119991 Russia, e-mail: melnikov@cs.msu.ru
REFERENCES
1. Anderson B.D.O., Moore J.B. Optimal control: linear-quadratic methods. Mineola, NY, Dover Publ., 2007, 464 p. ISBN: 9780486457666 .
2. Vrabie D., Abu-Khalaf M., Lewis F.L., Wang Y. Continuous-time ADP for linear systems with partially unknown dynamics. In: 2007 IEEE Inter. Symp. Approximate Dynamic Programming and Reinforcement Learning, Honolulu, USA, 2007, pp. 247–253. https://doi.org/10.1109/ADPRL.2007.368195
3. Vrabie D., Pastravanu O., Abu-Khalaf M., Lewis F.L. Adaptive optimal control for continuous-time linear systems based on policy iteration. Automatica, 2009, vol. 45, iss. 2, pp. 477–484.
4. Lewis F.L., Vrabie D., Vamvoudakis K.G. Reinforcement learning and feedback control: using natural decision methods to design optimal adaptive controllers. IEEE Contr. Syst. Mag., 2012, vol. 32, iss. 6, pp. 76–105. https://doi.org/10.1109/MCS.2012.2214134
5. Lee J.Y., Park J.B., Choi Y.H. On integral value iteration for continuous-time linear systems. In: Proc. 2013 American Contr. Conf. (ACC), Washington, USA, 2013, pp. 4215–4220. https://doi.org/10.1109/ACC.2013.6580487
6. Modares H., Lewis F.L. Linear quadratic tracking control of partially-unknown continuous-time systems using reinforcement learning. IEEE Trans. Autom. Contr., 2014, vol. 59, iss. 11, pp. 3051–3056. https://doi.org/10.1109/TAC.2014.2317301
7. Kiumarsi B., Vamvoudakis K.G., Modares H., Lewis F.L. Optimal and autonomous control using reinforcement learning: a survey. IEEE Trans. Neural Netw. Learn. Syst., 2018, vol. 29, iss. 6, pp. 2042–2062. https://doi.org/10.1109/tnnls.2017.2773458
8. Ronzhina M.I., Melnikov N.B. Mashinnoye obucheniye i optimal’noye upravleniye [Machine learning and optimal control]. Moscow, Publ. Center of National Univ. Oil and Gas “Gubkin University”, 2024, 126 p. ISBN: 978-5-91961-517-0 .
9. Aseev S.M., Veliov V.M. Another view of the maximum principle for infinite-horizon optimal control problems in economics. Rus. Math. Surv., 2019, vol. 74, iss. 6, pp. 963–1011. https://doi.org/10.1070/RM9915
10. Smovzh I.N., Melnikov N.B. Continuous-time reinforcement learning for the linear quadratic tracking problem using the policy iteration method. Moscow Univ. Comput. Math. Cybern., 2026, vol. 50, no. 2,
pp. 122--131.
https://doi.org/10.3103/S0278641926700020
11. Kleinman D.L. On an iterative technique for Riccati equation computations. IEEE Trans. Autom. Contr., 1968, vol. 13, no. 1, pp. 114–115. https://doi.org/10.1109/TAC.1968.1098829
12. Lin L., Lin H., Huang J. A new approach to the data-driven output-based LQR problem of continuous-time linear systems. IEEE Trans. Autom. Contr., 2026, pp. 1–16. https://doi.org/10.48550/arXiv.2509.18819
13. Lancaster P., Rodman L. Algebraic Riccati equations. Oxford, Clarendon Press, 1995, 498 p.
14. Kučera V. A contribution to matrix quadratic equations. IEEE Trans. Autom. Contr., 1972, vol. 17, no. 3, pp. 344–347.
15. Kelley C.T. Iterative methods for linear and nonlinear equations. Ser. Front. Appl. Math., vol. 16. Philadelphia, SIAM, 1995, 179 p. ISBN: 0898713528 .
16. Kantorovich L.V., Akilov G.P. Functional analysis in normed spaces. NY, Pergamon Press, 1964, 773 p. Original Russian text published in Kantorovich L. V., Akilov G. P. Funktsional’nyi analiz v normirovannykh prostranstvakh, Moscow, Fizmatlit, 1959, 684 p.
Cite this article as: I.N. Smovzh, N.B. Melnikov. Reinforcement learning using value iteration method for linear-quadratic tracking problem in continuous time. Trudy Instituta Matematiki i Mekhaniki UrO RAN, 2026, vol. 32, no. 3, pp. 204–220.
