Nested Monte-Carlo Tree Search for Online Planning in Large MDPs

Hendrik Baier; Mark H. M. Winands

doi:10.3233/978-1-61499-098-7-109

Nested Monte-Carlo Tree Search for Online Planning in Large MDPs

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceeding › Academic › peer-review

Abstract

Monte-Carlo Tree Search (MCTS) is state of the art for online planning in large MDPs. It is a best-first, sample-based search algorithm in which every state in the search tree is evaluated by the average outcome of Monte-Carlo rollouts from that state. These rollouts are typically random or directed by a simple, domain-dependent heuristic. We propose Nested Monte-Carlo Tree Search (NMCTS), in which MCTS itself is recursively used to provide a rollout policy for higher-level searches. In three large-scale MDPs, SameGame, Clickomania and Bubble Breaker, we show that NMCTS is significantly more effective than regular MCTS at equal time controls, both using random and heuristic rollouts at the base level. Experiments also suggest superior performance to Nested Monte-Carlo Search (NMCS) in some domains.

Original language	English
Title of host publication	Proceedings of the 20th European Conference on Artificial Intelligence
Place of Publication	Amsterdam, The Netherlands, The Netherlands
Publisher	IOS Press
Pages	109-114
Number of pages	6
ISBN (Print)	978-1-61499-097-0
DOIs	https://doi.org/10.3233/978-1-61499-098-7-109
Publication status	Published - 2012

Access to Document

10.3233/978-1-61499-098-7-109Licence: CC BY-NC

Cite this

@inproceedings{c5adf9b913c84b8ea0513301423e9d2d,

title = "Nested Monte-Carlo Tree Search for Online Planning in Large MDPs",

abstract = "Monte-Carlo Tree Search (MCTS) is state of the art for online planning in large MDPs. It is a best-first, sample-based search algorithm in which every state in the search tree is evaluated by the average outcome of Monte-Carlo rollouts from that state. These rollouts are typically random or directed by a simple, domain-dependent heuristic. We propose Nested Monte-Carlo Tree Search (NMCTS), in which MCTS itself is recursively used to provide a rollout policy for higher-level searches. In three large-scale MDPs, SameGame, Clickomania and Bubble Breaker, we show that NMCTS is significantly more effective than regular MCTS at equal time controls, both using random and heuristic rollouts at the base level. Experiments also suggest superior performance to Nested Monte-Carlo Search (NMCS) in some domains.",

author = "Hendrik Baier and Winands, {Mark H. M.}",

year = "2012",

doi = "10.3233/978-1-61499-098-7-109",

language = "English",

isbn = "978-1-61499-097-0",

pages = "109--114",

booktitle = "Proceedings of the 20th European Conference on Artificial Intelligence",

publisher = "IOS Press",

address = "Netherlands",

}

Nested Monte-Carlo Tree Search for Online Planning in Large MDPs. / Baier, Hendrik; Winands, Mark H. M.
Proceedings of the 20th European Conference on Artificial Intelligence. Amsterdam, The Netherlands, The Netherlands: IOS Press, 2012. p. 109-114.

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceeding › Academic › peer-review

TY - GEN

T1 - Nested Monte-Carlo Tree Search for Online Planning in Large MDPs

AU - Baier, Hendrik

AU - Winands, Mark H. M.

PY - 2012

Y1 - 2012

N2 - Monte-Carlo Tree Search (MCTS) is state of the art for online planning in large MDPs. It is a best-first, sample-based search algorithm in which every state in the search tree is evaluated by the average outcome of Monte-Carlo rollouts from that state. These rollouts are typically random or directed by a simple, domain-dependent heuristic. We propose Nested Monte-Carlo Tree Search (NMCTS), in which MCTS itself is recursively used to provide a rollout policy for higher-level searches. In three large-scale MDPs, SameGame, Clickomania and Bubble Breaker, we show that NMCTS is significantly more effective than regular MCTS at equal time controls, both using random and heuristic rollouts at the base level. Experiments also suggest superior performance to Nested Monte-Carlo Search (NMCS) in some domains.

AB - Monte-Carlo Tree Search (MCTS) is state of the art for online planning in large MDPs. It is a best-first, sample-based search algorithm in which every state in the search tree is evaluated by the average outcome of Monte-Carlo rollouts from that state. These rollouts are typically random or directed by a simple, domain-dependent heuristic. We propose Nested Monte-Carlo Tree Search (NMCTS), in which MCTS itself is recursively used to provide a rollout policy for higher-level searches. In three large-scale MDPs, SameGame, Clickomania and Bubble Breaker, we show that NMCTS is significantly more effective than regular MCTS at equal time controls, both using random and heuristic rollouts at the base level. Experiments also suggest superior performance to Nested Monte-Carlo Search (NMCS) in some domains.

U2 - 10.3233/978-1-61499-098-7-109

DO - 10.3233/978-1-61499-098-7-109

M3 - Conference article in proceeding

SN - 978-1-61499-097-0

SP - 109

EP - 114

BT - Proceedings of the 20th European Conference on Artificial Intelligence

PB - IOS Press

CY - Amsterdam, The Netherlands, The Netherlands

ER -