On the role of Large Language Models in Planning

Large Language Models Sequential Decision Making Automated Planning

ICAPS 2023 Tutorial

Monday July 10 Afternoon
Learn More ICAPS 2023

Overview


Large Language Models (LLMs, or n-gram models on steroids) that have been trained originally to generate text by repeatedly predicting the next word in the context of a window of previous words, have captured the attention of the AI (and the world) community. Part of the reason for this is their ability to produce meaningful completions for prompts relating to almost any area of human intellectual endeavors. This sheer versatility has also led to claims that these predictive text completion systems may be capable of abstract reasoning and planning. In this tutorial we take a critical look at the ability of LLMs to help in planning tasks–either in autonomous modes, or in assistive modes. We are particularly interested in characterizing these abilities–if any–in the context of problems and frameworks widely studied in the AI planning community.


The tutorial will both point out the fundamental limitations of LLMs in generating plans that will normally require resolving subgoal interactions with combinatorial search, and also show constructive uses of LLMs as complementary technologies to the sound planners that are developed in the AI Planning community. In addition to presenting our own work in this area, we provide a critical survey of many related efforts, including by researchers outside of the planning community.

Topics covered

The preliminary list of topics to be covered in this tutorial includes:
  1. Background on LLMs, and patterns of LLM use–including prompting techniques
  2. Differentiating the use of transformer architectures vs. Pre-trained LLMs in planning
    • Mention Word2vec to plan, decision transformers, Our ongoing work on using GPT2 with fine tuning , learning verifiers
  3. LLMs & Planning - Autonomous mode
    • Prompting in natural language or direct PDDL; effect of fine tuning; chain-of-thought prompting etc.
  4. LLMs as heuristics/idea generators for planning
    • Connections to Case-Based and Model-Lite planning
  5. Search by Back-prompting LLMS
    • Automated vs. Human-driven back-prompts (and the Clever Hans problem with the latter)
  6. LLMs as model acquisition techniques
  7. LLMs as vehicles to support general types of planning
    • Incompletely specified (highly disjunctive) goals; HTN planning; “generalized planning”
    • Use of LLMs in RL settings (to get rewards, preferences)
Slides Videos

Organizers

Subbarao Kambhampati is a professor of Computer Science at Arizona State University. Kambhampati studies fundamental problems in planning and decision making, motivated in particular by the challenges of human-aware AI systems. Kambhampati is a fellow of AAAI and AAAS, and the past president of AAAI.

Home

Karthik Valmeekam

Home

Matthew Marquez

Home

Lin Guan

Home

Citation

This tutorial maybe cited as:

S. Kambhampati, K. Valmeekam, M. Marquez & L. Guan. (2023, July). On the Role of Large Language Models in Planning. Tutorial presented at the International Conference on Automated Planning and Scheduling (ICAPS), Prague. https://yochan-lab.github.io/tutorial/ICAPS-2023/. 

            @misc{kambhampati2023role,
                author = {Kambhampati, Subbarao. and Valmeekam, Karthik. and Marquez, Matthew. and Guan, Lin.},
                title = {On the Role of Large Language Models in Planning},
                year = {2023},
                month = {July},
                note = {Tutorial presented at the International Conference on Automated Planning and Scheduling (ICAPS), Prague},
                url = {https://yochan-lab.github.io/tutorial/ICAPS-2023/}
            }