Skip to main content
SpringerLink
Log in

Local Poisson Equations Associated with Discrete-Time Markov Control Processes

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

This paper provides a characterization of the optimal average cost function, when the long-run (risk-sensitive) average cost criterion is used. The Markov control model has a denumerable state space with finite set of actions, and the characterization presented is given in terms of a system of local Poisson equations, which gives as a by-product the existence of an optimal stationary policy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Cavazos-Cadena, R., Hernández-Hernández, D.: A system of Poisson equations for a non-constant Varadhan functional on finite state space. Appl. Math. Optim. 53, 101–119 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  2. Cavazos-Cadena, R., Hernández-Hernández, D.: Local Poisson equations associated with the Varadhan functional. Asymptot. Anal. 96, 023–050 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  3. Alanís-Durán, A., Cavazos-Cadena, R.: An optimality system for finite average Markov decision chains under risk aversion. Kybernetika 48, 83–104 (2012)

    MathSciNet  MATH  Google Scholar 

  4. Howard, A.R., Matheson, J.E.: Risk sensitive Markov decision processes. Manag. Sci. 18, 356–369 (1972)

    Article  MathSciNet  MATH  Google Scholar 

  5. Gantmakher, F.R.: The Theory of Matrices. Chelsea, London (1959)

    MATH  Google Scholar 

  6. Meyer, C.D.: Matrix Analysis and Applied Linear Algebra. SIAM, Philadelphia (2000)

    Book  Google Scholar 

  7. Fleming, W.H., Hernández-Hernández, D.: Risk sensitive control of finite state machines on a infinite horizon I. SIAM J. Control Optim. 35, 1790–1810 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  8. Bielecki, T., Hernández-Hernández, D., Pliska, R.: Risk sensitive control of finite state Markov chains in discrete time, with applications to portfolio management. Math. Methods Oper. Res. 50, 167–188 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  9. Hernández-Hernández, D., Marcus, S.I.: Risk sensitive control of Markov processes in countable state space. Syst. Control Lett. 29, 147–155 (1996). Corrigendum. 34, 105–106 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  10. Cavazos-Cadena, R., Fernández-Gaucherand, E.: Controlled Markov chains with risk-sensitive criteria: average cost, optimality equations, and optimal solutions. Math. Methods Oper. Res. 49, 299–324 (1999)

    MathSciNet  MATH  Google Scholar 

  11. Cavazos-Cadena, R., Fernández-Gaucherand, E.: Risk sensitive control in communicating average Markov decision chains. In: Dror, P., L’Ecuyer, P., Szidarovsky, F. (eds.) Modelling Uncertainty: An Examination of Stochastic Theory, Methods and Applications, pp. 525–544. Kluwer, Boston (2002)

    Google Scholar 

  12. Di Masi, G.B., Stettner, L.: Risk sensitive control of discrete time Markov processes with infinite horizon. SIAM J. Control Optim. 38, 61–78 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  13. Cavazos-Cadena, R.: Solution to the risk sensitive average cost optimality equation in a class of Markov decision processes with finite state space. Math. Methods Optim. Res. 57, 263–285 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  14. Sladký, K.: Growth rates and average optimality in risk sensitive Markov decision chains. Kybernetika 44, 205–226 (2008)

    MathSciNet  MATH  Google Scholar 

  15. Cavazos-Cadena, R., Salem-Silva, F.: The discounted method and equivalence of average criteria for risk sensitive Markov decision processes on Borel space. Appl. Math. Optim. 61, 167–190 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  16. Di Masi, G.B., Stettner, L.: Infinite horizon risk sensitive control of discrete time Markov processes with small risk. Syst. Control Lett. 40, 15–20 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  17. Di Masi, G.B., Stettner, L.: Infinite horizon risk sensitive control of discrete time Markov processes under minorization property. SIAM J. Control Optim. 46, 231–252 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  18. Sladký, K.: Bounds on discrete dynamic programming recursions I. Kybernetika 16, 526–547 (1980)

    MathSciNet  MATH  Google Scholar 

  19. Sladký, K., Montes-de-Oca, R.: Risk-sensitive average optimality in Markov decision chains. In: Kalcsiscs, J., Nickel, S. (eds.) Operations Research Proceedings, vol. 2007, pp. 69–74. Springer, Berlin (2008)

    Google Scholar 

  20. Zijim, W.H.M.: Nonnegative Matrices in Dynamic Programming. Mathematical Centre Tract, Amsterdam (1983)

    Google Scholar 

  21. Rothblum, U.G., Whittle, P.: Growth optimality for branching Markov desicion chains. Math. Oper. Res. 7, 582–601 (1982)

    Article  MathSciNet  MATH  Google Scholar 

  22. Hernández-Lerma, O., Lasserre, J.B.: Discrete-Time Markov Control Processes: Basic Optimality Criteria. Springer, New York (1996)

    Book  MATH  Google Scholar 

  23. Ash, R.B.: Probability and Measure Theory. Academic Press, New York (2000)

    MATH  Google Scholar 

  24. Billingsley, P.: Probability and Measure. Wiley, New York (1995)

    MATH  Google Scholar 

Download references

Acknowledgements

DHH was partially supported by CONACYT Grant 254166.

Author information

Authors and Affiliations

  1. Centro de Investigación en Matemáticas, Guanajuato, Mexico

    Daniel Hernández Hernández & Diego Hernández Bustos

Authors
  1. Daniel Hernández Hernández
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Diego Hernández Bustos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel Hernández Hernández.

Additional information

Communicated by Alan Bensoussan.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hernández Hernández, D., Hernández Bustos, D. Local Poisson Equations Associated with Discrete-Time Markov Control Processes. J Optim Theory Appl 173, 1–29 (2017). https://doi.org/10.1007/s10957-017-1076-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-017-1076-5

Keywords

Mathematics Subject Classification

Search

Navigation