Working Paper

Applications of Markov Chain Approximation Methods to Optimal Control Problems in Economics

Abstract: In this paper we explore some of the benefits of using the finite-state Markov chain approximation (MCA) method of Kushner and Dupuis (2001) to solve continuous-time optimal control problems. We first show that the implicit finite-difference scheme of Achdou et al. (2017) amounts to a limiting form of the MCA method for a certain choice of approximating chains and policy function iteration for the resulting system of equations. We then illustrate the benefits of departing from policy function iteration by showing that using variations of modified policy function iteration to solve income fluctuation problems in two and three dimensions can lead to an increase in the speed of convergence of more than an order of magnitude. We then show that the MCA method is also well-suited to solving portfolio problems with highly correlated state variables, a setting that commonly occurs within general equilibrium models with financial frictions and for which it is difficult to construct monotone (and hence convergent) finite-difference schemes.

Keywords: Dynamic programming; financial frictions;

JEL Classification: C63; E00; G11;

https://doi.org/10.26509/frbc-wp-202104

Bibliographic Information

Provider: Federal Reserve Bank of Cleveland

Part of Series: Working Papers

Publication Date: 2021-02-10

Number: 21-04

Related Works

• Working Paper Revision: Applications of Markov Chain Approximation Methods to Optimal Control Problems in Economics