‘offline RL’ tag

Gwern Branwen

See Also
Links
Miscellaneous
Link Bibliography

[Warning: JavaScript Disabled!]

[For support of key website features (link annotation popups/popovers & transclusions, collapsible sections, backlinks, tablesorting, image zooming, sidenotes etc), you must enable JavaScript.]

Links

“Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data”, Tajwar et al 2024

Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data

“Dataset Reset Policy Optimization for RLHF”, Chang et al 2024

Dataset Reset Policy Optimization for RLHF

“Vision-Language Models As a Source of Rewards”, Baumli et al 2023

Vision-Language Models as a Source of Rewards

“Beyond Human Data: Scaling Self-Training for Problem-Solving With Language Models (ReST^EM)”, Singh et al 2023

Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models (ReST^EM)

“Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations”, Hong et al 2023

Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversations

“Course Correcting Koopman Representations”, Fathi et al 2023

Course Correcting Koopman Representations

“Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions”, Chebotar et al 2023

Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions

“Subwords As Skills: Tokenization for Sparse-Reward Reinforcement Learning”, Yunis et al 2023

Subwords as Skills: Tokenization for Sparse-Reward Reinforcement Learning

“What Are Dreams For? Converging Lines of Research Suggest That We Might Be Misunderstanding Something We Do Every Night of Our Lives”, Gefter 2023

What Are Dreams For? Converging lines of research suggest that we might be misunderstanding something we do every night of our lives

“ReST: Reinforced Self-Training (ReST) for Language Modeling”, Gulcehre et al 2023

ReST: Reinforced Self-Training (ReST) for Language Modeling

“AlphaStar Unplugged: Large-Scale Offline Reinforcement Learning”, Mathieu et al 2023

AlphaStar Unplugged: Large-Scale Offline Reinforcement Learning

“Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behavior”, Block et al 2023

Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behavior

“PASTA: Pretrained Action-State Transformer Agents”, Boige et al 2023

PASTA: Pretrained Action-State Transformer Agents

“Twitching in Sensorimotor Development from Sleeping Rats to Robots”, Blumberg et al 2023

Twitching in Sensorimotor Development from Sleeping Rats to Robots

“Survival Instinct in Offline Reinforcement Learning”, Li et al 2023

Survival Instinct in Offline Reinforcement Learning

“BetaZero: Belief-State Planning for Long-Horizon POMDPs Using Learned Approximations”, Moss et al 2023

BetaZero: Belief-State Planning for Long-Horizon POMDPs using Learned Approximations

“Improving Language Models With Advantage-Based Offline Policy Gradients”, Baheti et al 2023

Improving Language Models with Advantage-based Offline Policy Gradients

“Revisiting the Minimalist Approach to Offline Reinforcement Learning”, Tarasov et al 2023

Revisiting the Minimalist Approach to Offline Reinforcement Learning

“Think Before You Act: Unified Policy for Interleaving Language Reasoning With Actions”, Mezghani et al 2023

Think Before You Act: Unified Policy for Interleaving Language Reasoning with Actions

“Off-The-Grid MARL (OG-MARL): Datasets With Baselines for Offline Multi-Agent Reinforcement Learning”, Formanek et al 2023

Off-the-Grid MARL (OG-MARL): Datasets with Baselines for Offline Multi-Agent Reinforcement Learning

“Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes”, Kumar et al 2022

Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes

“Dungeons and Data: A Large-Scale NetHack Dataset”, Hambro et al 2022

Dungeons and Data: A Large-Scale NetHack Dataset

“CORL: Research-Oriented Deep Offline Reinforcement Learning Library”, Tarasov et al 2022

CORL: Research-oriented Deep Offline Reinforcement Learning Library

“Diffusion-QL: Diffusion Policies As an Expressive Policy Class for Offline Reinforcement Learning”, Wang et al 2022

Diffusion-QL: Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning

“Offline RL Policies Should Be Trained to Be Adaptive”, Ghosh et al 2022

Offline RL Policies Should be Trained to be Adaptive

“Prompting Decision Transformer for Few-Shot Policy Generalization”, Xu et al 2022

Prompting Decision Transformer for Few-Shot Policy Generalization

“Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos”, Baker et al 2022

Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos

“Large-Scale Retrieval for Reinforcement Learning”, Humphreys et al 2022

Large-Scale Retrieval for Reinforcement Learning

“Offline RL for Natural Language Generation With Implicit Language Q Learning”, Snell et al 2022

Offline RL for Natural Language Generation with Implicit Language Q Learning

“When Does Return-Conditioned Supervised Learning Work for Offline Reinforcement Learning?”, Brandfonbrener et al 2022

When does return-conditioned supervised learning work for offline reinforcement learning?

“You Can’t Count on Luck: Why Decision Transformers Fail in Stochastic Environments”, Paster et al 2022

You Can’t Count on Luck: Why Decision Transformers Fail in Stochastic Environments

“Multi-Game Decision Transformers”, Lee et al 2022

Multi-Game Decision Transformers

“When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?”, Kumar et al 2022

When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?

“Don’t Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning (ExORL)”, Yarats et al 2022

Don’t Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning (ExORL)

“Offline Pre-Trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks”, Meng et al 2021

Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Tasks

“A Workflow for Offline Model-Free Robotic Reinforcement Learning”, Kumar et al 2021

A Workflow for Offline Model-Free Robotic Reinforcement Learning

“A Minimalist Approach to Offline Reinforcement Learning”, Fujimoto & Gu 2021

A Minimalist Approach to Offline Reinforcement Learning

“Is Pessimism Provably Efficient for Offline RL?”, Jin et al 2020

Is Pessimism Provably Efficient for Offline RL?

“What Are the Statistical Limits of Offline RL With Linear Function Approximation?”, Wang et al 2020

What are the Statistical Limits of Offline RL with Linear Function Approximation?

“MOPO: Model-Based Offline Policy Optimization”, Yu et al 2020

MOPO: Model-based Offline Policy Optimization

“Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems”, Levine et al 2020

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

“D4RL: Datasets for Deep Data-Driven Reinforcement Learning”, Fu et al 2020

D4RL: Datasets for Deep Data-Driven Reinforcement Learning

“Q* Approximation Schemes for Batch Reinforcement Learning: A Theoretical Comparison”, Xie & Jiang 2020

Q* Approximation Schemes for Batch Reinforcement Learning: A Theoretical Comparison

“Scaling Data-Driven Robotics With Reward Sketching and Batch Reinforcement Learning”, Cabi et al 2019

Scaling data-driven robotics with reward sketching and batch reinforcement learning

“QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”, Kalashnikov et al 2018

QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation

“The Netflix Recommender System”, Gomez-Uribe & Hunt 2015

The Netflix Recommender System

Sort By Magic

Annotations sorted by machine learning into inferred 'tags'. This provides an alternative way to browse: instead of by date order, one can browse in topic order. The 'sorted' list has been automatically clustered into multiple sections & auto-labeled for easier browsing.

Beginning with the newest annotation, it uses the embedding of each annotation to attempt to create a list of nearest-neighbor annotations, creating a progression of topics. For more details, see the link.

Wikipedia

Precociality and altriciality § Superprecociality

Miscellaneous

Link Bibliography

https://arxiv.org/abs/2404.08495: “Dataset Reset Policy Optimization for RLHF”, Jonathan D. Chang, Wenhao Shan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D. Lee, Wen Sun

link-bibliography
https://arxiv.org/abs/2312.06585#deepmind: “Beyond Human Data: Scaling Self-Training for Problem-Solving With Language Models (ReST^EM)”, Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, Noah Fiedel

link-bibliography
https://arxiv.org/abs/2305.09836: “Revisiting the Minimalist Approach to Offline Reinforcement Learning”, Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, Sergey Kolesnikov

link-bibliography
https://arxiv.org/abs/2206.13499: “Prompting Decision Transformer for Few-Shot Policy Generalization”, Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua B. Tenenbaum, Chuang Gan

link-bibliography
https://arxiv.org/abs/2206.11795#openai: “Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos”, Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, Jeff Clune

link-bibliography
https://arxiv.org/abs/2206.05314#deepmind: “Large-Scale Retrieval for Reinforcement Learning”, Peter C. Humphreys, Arthur Guez, Olivier Tieleman, Laurent Sifre, Théophane Weber, Timothy Lillicrap

link-bibliography
https://arxiv.org/abs/2205.15241#google: “Multi-Game Decision Transformers”, Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, Igor Mordatch

link-bibliography
2015-gomezuribe.pdf: “The Netflix Recommender System”, Carlos A. Gomez-Uribe, Neil Hunt

link-bibliography