Nested Monte-Carlo Tree Search for Online Planning in Large MDPs

Hendrik Baier*, Mark H. M. Winands

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference article in proceedingAcademicpeer-review


Monte-Carlo Tree Search (MCTS) is state of the art for online planning in large MDPs. It is a best-first, sample-based search algorithm in which every state in the search tree is evaluated by the average outcome of Monte-Carlo rollouts from that state. These rollouts are typically random or directed by a simple, domain-dependent heuristic. We propose Nested Monte-Carlo Tree Search (NMCTS), in which MCTS itself is recursively used to provide a rollout policy for higher-level searches. In three large-scale MDPs, SameGame, Clickomania and Bubble Breaker, we show that NMCTS is significantly more effective than regular MCTS at equal time controls, both using random and heuristic rollouts at the base level. Experiments also suggest superior performance to Nested Monte-Carlo Search (NMCS) in some domains.
Original languageEnglish
Title of host publicationProceedings of the 20th European Conference on Artificial Intelligence
Place of PublicationAmsterdam, The Netherlands, The Netherlands
PublisherIOS Press
Number of pages6
ISBN (Print)978-1-61499-097-0
Publication statusPublished - 2012

Cite this