Achieving target state-action frequencies in multichain average-reward Markov decision processes

D Krass*, OJ Vrieze

*Corresponding author for this work

    Research output: Contribution to journalArticleAcademicpeer-review


    In this paper we address a basic problem that arises naturally in average-reward Markov decision processes with constraints and/or nonstandard payoff criteria: Given a feasible state-action frequency vector ("the target"), construct a policy whose state-action frequencies match those of the target vector.

    While it is well known that the solution to this problem cannot, in general, be found in the space of stationary randomized policies, we construct a solution that has "ultimately stationary" structured It consists of two stationary policies where the first one is used initially, and then the switch to the second one is made at a certain random switching time. The computational effort required to construct this solution is minimal.

    We also show that our problem can always be solved by a stationary policy if the original MDP is "extended" by adding certain states and actions. The solution in the original MDP is obtained by mapping the solution in the extended MDP back to the original process.

    Original languageEnglish
    Pages (from-to)545-566
    Number of pages22
    JournalMathematics of Operations Research
    Issue number3
    Publication statusPublished - Aug 2002


    • Markov decision processes
    • average reward criterion
    • state-action frequencies
    • constrained Markov decision processes
    • Markov decision processes with nonstandard reward criteria
    • CHAINS

    Cite this