‘preference learning’ tag

Gwern Branwen

See Also
Links
Miscellaneous
Link Bibliography

[Warning: JavaScript Disabled!]

[For support of key website features (link annotation popups/popovers & transclusions, collapsible sections, backlinks, tablesorting, image zooming, sidenotes etc), you must enable JavaScript.]

Links

“Aligning LLM Agents by Learning Latent Preference from User Edits”, Gao et al 2024

Aligning LLM Agents by Learning Latent Preference from User Edits

“Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data”, Tajwar et al 2024

Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data

“From r to Q^✱: Your Language Model Is Secretly a Q-Function”, Rafailov et al 2024

From r to Q^✱: Your Language Model is Secretly a Q-Function

“Dataset Reset Policy Optimization for RLHF”, Chang et al 2024

Dataset Reset Policy Optimization for RLHF

“ControlNet++: Improving Conditional Controls With Efficient Consistency Feedback”, Li et al 2024

ControlNet++: Improving Conditional Controls with Efficient Consistency Feedback

“TextCraftor: Your Text Encoder Can Be Image Quality Controller”, Li et al 2024

TextCraftor: Your Text Encoder Can be Image Quality Controller

“RewardBench: Evaluating Reward Models for Language Modeling”, Lambert et al 2024

RewardBench: Evaluating Reward Models for Language Modeling

“Evaluating Text to Image Synthesis: Survey and Taxonomy of Image Quality Metrics”, Hartwig et al 2024

Evaluating Text to Image Synthesis: Survey and Taxonomy of Image Quality Metrics

“V-STaR: Training Verifiers for Self-Taught Reasoners”, Hosseini et al 2024

V-STaR: Training Verifiers for Self-Taught Reasoners

“The Non-Effect of Sampling Temperature on Problem Solving in GPT-3.5/GPT-4”, Renze & Guven 2024

The Non-Effect of Sampling Temperature on Problem Solving in GPT-3.5/GPT-4

“Does Using ChatGPT Result in Human Cognitive Augmentation?”, Fulbright & Morrison 2024

Does Using ChatGPT Result in Human Cognitive Augmentation?

“Sleeper Agents: Training Deceptive LLMs That Persist Through Safety Training”, Hubinger et al 2024

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

“Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM”, Lu et al 2024

Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM

“A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity”, Lee et al 2024

A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity

“Reasons to Reject? Aligning Language Models With Judgments”, Xu et al 2023

Reasons to Reject? Aligning Language Models with Judgments

“Rich Human Feedback for Text-To-Image Generation”, Liang et al 2023

Rich Human Feedback for Text-to-Image Generation

“Helping or Herding? Reward Model Ensembles Mitigate but Do Not Eliminate Reward Hacking”, Eisenstein et al 2023

Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking

“The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning”, Lin et al 2023

The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning

“Universal Jailbreak Backdoors from Poisoned Human Feedback”, Rando & Tramèr 2023

Universal Jailbreak Backdoors from Poisoned Human Feedback

“Diffusion Model Alignment Using Direct Preference Optimization”, Wallace et al 2023

Diffusion Model Alignment Using Direct Preference Optimization

“Summon a Demon and Bind It: A Grounded Theory of LLM Red Teaming in the Wild”, Inie et al 2023

Summon a Demon and Bind it: A Grounded Theory of LLM Red Teaming in the Wild

“When Ruthless Cultural Elitism Is Exactly the Job”, Marchese 2023

When Ruthless Cultural Elitism Is Exactly the Job

“Does GPT-4 Pass the Turing Test?”, Jones & Bergen 2023

Does GPT-4 Pass the Turing Test?

“Specific versus General Principles for Constitutional AI”, Kundu et al 2023

Specific versus General Principles for Constitutional AI

“Eureka: Human-Level Reward Design via Coding Large Language Models”, Ma et al 2023

Eureka: Human-Level Reward Design via Coding Large Language Models

“A General Theoretical Paradigm to Understand Learning from Human Preferences”, Azar et al 2023

A General Theoretical Paradigm to Understand Learning from Human Preferences

“UltraFeedback: Boosting Language Models With High-Quality Feedback”, Cui et al 2023

UltraFeedback: Boosting Language Models with High-quality Feedback

“Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack”, Dai et al 2023

Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack

“STARC: A General Framework For Quantifying Differences Between Reward Functions”, Skalse et al 2023

STARC: A General Framework For Quantifying Differences Between Reward Functions

“AceGPT, Localizing Large Language Models in Arabic”, Huang et al 2023

AceGPT, Localizing Large Language Models in Arabic

“Assessing the Nature of Large Language Models: A Caution against Anthropocentrism”, Speed 2023

Assessing the nature of large language models: A caution against anthropocentrism

“ReST: Reinforced Self-Training (ReST) for Language Modeling”, Gulcehre et al 2023

ReST: Reinforced Self-Training (ReST) for Language Modeling

“LLaMA-2: Open Foundation and Fine-Tuned Chat Models”, Touvron et al 2023

LLaMA-2: Open Foundation and Fine-Tuned Chat Models

“Introducing Superalignment”, Leike & Sutskever 2023

Introducing Superalignment

“AI Is a Lot of Work: As the Technology Becomes Ubiquitous, a Vast Tasker Underclass Is Emerging—And Not Going Anywhere”, Dzieza 2023

AI Is a Lot of Work: As the technology becomes ubiquitous, a vast tasker underclass is emerging—and not going anywhere

“Large Language Models Sometimes Generate Purely Negatively-Reinforced Text”, Roger 2023

Large Language Models Sometimes Generate Purely Negatively-Reinforced Text

“Microsoft and OpenAI Forge Awkward Partnership As Tech’s New Power Couple: As the Companies Lead the AI Boom, Their Unconventional Arrangement Sometimes Causes Conflict”, Dotan & Seetharaman 2023

Microsoft and OpenAI Forge Awkward Partnership as Tech’s New Power Couple: As the companies lead the AI boom, their unconventional arrangement sometimes causes conflict

“ChatGPT Is Fun, but It Is Not Funny! Humor Is Still Challenging Large Language Models”, Jentzsch & Kersting 2023

ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Models

“Direct Preference Optimization (DPO): Your Language Model Is Secretly a Reward Model”, Rafailov et al 2023

Direct Preference Optimization (DPO): Your Language Model is Secretly a Reward Model

“The False Promise of Imitating Proprietary LLMs”, Gudibande et al 2023

The False Promise of Imitating Proprietary LLMs

“Improving Language Models With Advantage-Based Offline Policy Gradients”, Baheti et al 2023

Improving Language Models with Advantage-based Offline Policy Gradients

“LIMA: Less Is More for Alignment”, Zhou et al 2023

LIMA: Less Is More for Alignment

“Bits of Grass: Does GPT Already Know How to Write like Whitman?”, Sawicki et al 2023

Bits of Grass: Does GPT already know how to write like Whitman?

“SELF-ALIGN: Principle-Driven Self-Alignment of Language Models from Scratch With Minimal Human Supervision”, Sun et al 2023

SELF-ALIGN: Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision

“Pick-A-Pic: An Open Dataset of User Preferences for Text-To-Image Generation”, Kirstain et al 2023

Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation

“A Radical Plan to Make AI Good, Not Evil”, Knight 2023

A Radical Plan to Make AI Good, Not Evil

“GPT-4 Technical Report § Limitations: Calibration”, OpenAI 2023 (page 12 org openai)

GPT-4 Technical Report § Limitations: Calibration

“Rewarding Chatbots for Real-World Engagement With Millions of Users”, Irvine et al 2023

Rewarding Chatbots for Real-World Engagement with Millions of Users

“Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-Oriented Dialogue Systems”, Feng et al 2023

Fantastic Rewards and How to Tame Them: A Case Study on Reward Learning for Task-oriented Dialogue Systems

“Use GPT-3 Incorrectly: Reduce Costs 40× and Increase Speed by 5×”, Pullen 2023

Use GPT-3 incorrectly: reduce costs 40× and increase speed by 5×

“OpenAI’s Sam Altman Talks ChatGPT And How Artificial General Intelligence Can ‘Break Capitalism’”, Konrad & Cai 2023

OpenAI’s Sam Altman Talks ChatGPT And How Artificial General Intelligence Can ‘Break Capitalism’

“Big Tech Was Moving Cautiously on AI. Then Came ChatGPT. Google, Facebook and Microsoft Helped Build the Scaffolding of AI. Smaller Companies Are Taking It to the Masses, Forcing Big Tech to React”, Tiku et al 2023

Big Tech was moving cautiously on AI. Then came ChatGPT. Google, Facebook and Microsoft helped build the scaffolding of AI. Smaller companies are taking it to the masses, forcing Big Tech to react

“The inside Story of ChatGPT: How OpenAI Founder Sam Altman Built the World’s Hottest Technology With Billions from Microsoft”, Kahn 2023

The inside story of ChatGPT: How OpenAI founder Sam Altman built the world’s hottest technology with billions from Microsoft

“Self-Instruct: Aligning Language Models With Self-Generated Instructions”, Wang et al 2022

Self-Instruct: Aligning Language Models with Self-Generated Instructions

“Discovering Language Model Behaviors With Model-Written Evaluations”, Perez et al 2022

Discovering Language Model Behaviors with Model-Written Evaluations

“HALIE: Evaluating Human-Language Model Interaction”, Lee et al 2022

HALIE: Evaluating Human-Language Model Interaction

“Constitutional AI: Harmlessness from AI Feedback”, Bai et al 2022

Constitutional AI: Harmlessness from AI Feedback

“Solving Math Word Problems With Process & Outcome-Based Feedback”, Uesato et al 2022

Solving math word problems with process & outcome-based feedback

“Mysteries of Mode Collapse § Inescapable Wedding Parties”, Janus 2022

Mysteries of mode collapse § Inescapable wedding parties

“Mysteries of Mode Collapse § Inescapable Wedding Parties”, Janus 2022

Mysteries of mode collapse § Inescapable wedding parties

“When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels”, Shi et al 2022

When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels

“Scaling Laws for Reward Model Overoptimization”, Gao et al 2022

Scaling Laws for Reward Model Overoptimization

“Teacher Forcing Recovers Reward Functions for Text Generation”, Hao et al 2022

Teacher Forcing Recovers Reward Functions for Text Generation

“CARP: Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning”, Castricato et al 2022

CARP: Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning

“Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization”, Ramamurthy et al 2022

Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization

“Sparrow: Improving Alignment of Dialogue Agents via Targeted Human Judgements”, Glaese et al 2022

Sparrow: Improving alignment of dialogue agents via targeted human judgements

“Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned”, Ganguli et al 2022

Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned

“Basis for Intentions (BASIS): Efficient Inverse Reinforcement Learning Using Past Experience”, Abdulhai et al 2022

Basis for Intentions (BASIS): Efficient Inverse Reinforcement Learning using Past Experience

“Improved Policy Optimization for Online Imitation Learning”, Lavington et al 2022

Improved Policy Optimization for Online Imitation Learning

“Quark: Controllable Text Generation With Reinforced Unlearning”, Lu et al 2022

Quark: Controllable Text Generation with Reinforced Unlearning

“RL With KL Penalties Is Better Viewed As Bayesian Inference”, Korbak et al 2022

RL with KL penalties is better viewed as Bayesian inference

“Housekeep: Tidying Virtual Households Using Commonsense Reasoning”, Kant et al 2022

Housekeep: Tidying Virtual Households using Commonsense Reasoning

“Imitating, Fast and Slow: Robust Learning from Demonstrations via Decision-Time Planning”, Qi et al 2022

Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning

“Inferring Rewards from Language in Context”, Lin et al 2022

Inferring Rewards from Language in Context

“SURF: Semi-Supervised Reward Learning With Data Augmentation for Feedback-Efficient Preference-Based Reinforcement Learning”, Park et al 2022

SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning

“InstructGPT: Training Language Models to Follow Instructions With Human Feedback”, Ouyang et al 2022

InstructGPT: Training language models to follow instructions with human feedback

“Safe Deep RL in 3D Environments Using Human Feedback”, Rahtz et al 2022

Safe Deep RL in 3D Environments using Human Feedback

“A Survey of Controllable Text Generation Using Transformer-Based Pre-Trained Language Models”, Zhang et al 2022

A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models

“WebGPT: Browser-Assisted Question-Answering With Human Feedback”, Nakano et al 2021

WebGPT: Browser-assisted question-answering with human feedback

“WebGPT: Improving the Factual Accuracy of Language Models through Web Browsing”, Hilton et al 2021

WebGPT: Improving the factual accuracy of language models through web browsing

“Modeling Strong and Human-Like Gameplay With KL-Regularized Search”, Jacob et al 2021

Modeling Strong and Human-Like Gameplay with KL-Regularized Search

“A General Language Assistant As a Laboratory for Alignment”, Askell et al 2021

A General Language Assistant as a Laboratory for Alignment

“Cut the CARP: Fishing for Zero-Shot Story Evaluation”, Shahbul et al 2021

Cut the CARP: Fishing for zero-shot story evaluation

“Recursively Summarizing Books With Human Feedback”, Wu et al 2021

Recursively Summarizing Books with Human Feedback

“B-Pref: Benchmarking Preference-Based Reinforcement Learning”, Lee et al 2021

B-Pref: Benchmarking Preference-Based Reinforcement Learning

“Trajectory Transformer: Reinforcement Learning As One Big Sequence Modeling Problem”, Janner et al 2021

Trajectory Transformer: Reinforcement Learning as One Big Sequence Modeling Problem

“Embracing New Techniques in Deep Learning for Estimating Image Memorability”, Needell & Bainbridge 2021

Embracing New Techniques in Deep Learning for Estimating Image Memorability

“A Survey of Preference-Based Reinforcement Learning Methods”, Wirth et al 2021

A Survey of Preference-Based Reinforcement Learning Methods

“Learning What To Do by Simulating the Past”, Lindner et al 2021

Learning What To Do by Simulating the Past

“Language Models Have a Moral Dimension”, Schramowski et al 2021

Language Models have a Moral Dimension

“Brain-Computer Interface for Generating Personally Attractive Images”, Spape et al 2021

Brain-computer interface for generating personally attractive images

“Process for Adapting Language Models to Society (PALMS) With Values-Targeted Datasets”, Solaiman & Dennison 2021

Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets

“Human-Centric Dialog Training via Offline Reinforcement Learning”, Jaques et al 2020

Human-centric Dialog Training via Offline Reinforcement Learning

“Learning to Summarize from Human Feedback”, Stiennon et al 2020

Learning to summarize from human feedback

“Learning Personalized Models of Human Behavior in Chess”, McIlroy-Young et al 2020

Learning Personalized Models of Human Behavior in Chess

“Aligning Superhuman AI With Human Behavior: Chess As a Model System”, McIlroy-Young et al 2020

Aligning Superhuman AI with Human Behavior: Chess as a Model System

“Active Preference-Based Gaussian Process Regression for Reward Learning”, Bıyık et al 2020

Active Preference-Based Gaussian Process Regression for Reward Learning

“Bayesian REX: Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences”, Brown et al 2020

Bayesian REX: Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences

“RL Agents Implicitly Learning Human Preferences”, Wichers 2020

RL agents Implicitly Learning Human Preferences

“Reward-Rational (implicit) Choice: A Unifying Formalism for Reward Learning”, Jeon et al 2020

Reward-rational (implicit) choice: A unifying formalism for reward learning

“What Does BERT Dream Of? A Visual Investigation of Nightmares in Sesame Street”, Bäuerle & Wexler 2020

What does BERT dream of? A visual investigation of nightmares in Sesame Street

“Deep Bayesian Reward Learning from Preferences”, Brown & Niekum 2019

Deep Bayesian Reward Learning from Preferences

“Learning Norms from Stories: A Prior for Value Aligned Agents”, Frazier et al 2019

Learning Norms from Stories: A Prior for Value Aligned Agents

“Reinforcement Learning Upside Down: Don’t Predict Rewards—Just Map Them to Actions”, Schmidhuber 2019

Reinforcement Learning Upside Down: Don’t Predict Rewards—Just Map Them to Actions

“Learning Human Objectives by Evaluating Hypothetical Behavior”, Reddy et al 2019

Learning Human Objectives by Evaluating Hypothetical Behavior

“Preference-Based Learning for Exoskeleton Gait Optimization”, Tucker et al 2019

Preference-Based Learning for Exoskeleton Gait Optimization

“Do Massively Pretrained Language Models Make Better Storytellers?”, See et al 2019

Do Massively Pretrained Language Models Make Better Storytellers?

“Fine-Tuning GPT-2 from Human Preferences § Bugs Can Optimize for Bad Behavior”, Ziegler et al 2019

Fine-Tuning GPT-2 from Human Preferences § Bugs can optimize for bad behavior

“Fine-Tuning GPT-2 from Human Preferences”, Ziegler et al 2019

Fine-Tuning GPT-2 from Human Preferences

“Fine-Tuning Language Models from Human Preferences”, Ziegler et al 2019

Fine-Tuning Language Models from Human Preferences

“Lm-Human-Preferences”, Ziegler et al 2019

lm-human-preferences

“Better Rewards Yield Better Summaries: Learning to Summarise Without References”, Böhm et al 2019

Better Rewards Yield Better Summaries: Learning to Summarise Without References

“Dueling Posterior Sampling for Preference-Based Reinforcement Learning”, Novoseller et al 2019

Dueling Posterior Sampling for Preference-Based Reinforcement Learning

“Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog”, Jaques et al 2019

Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog

“Reward Learning from Human Preferences and Demonstrations in Atari”, Ibarz et al 2018

Reward learning from human preferences and demonstrations in Atari

“StreetNet: Preference Learning With Convolutional Neural Network on Urban Crime Perception”, Fu et al 2018

StreetNet: Preference Learning with Convolutional Neural Network on Urban Crime Perception

“Toward Diverse Text Generation With Inverse Reinforcement Learning”, Shi et al 2018

Toward Diverse Text Generation with Inverse Reinforcement Learning

“Ordered Preference Elicitation Strategies for Supporting Multi-Objective Decision Making”, Zintgraf et al 2018

Ordered Preference Elicitation Strategies for Supporting Multi-Objective Decision Making

“Convergence of Value Aggregation for Imitation Learning”, Cheng & Boots 2018

Convergence of Value Aggregation for Imitation Learning

“A Low-Cost Ethics Shaping Approach for Designing Reinforcement Learning Agents”, Wu & Lin 2017

A Low-Cost Ethics Shaping Approach for Designing Reinforcement Learning Agents

“Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces”, Warnell et al 2017

Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces

“DropoutDAgger: A Bayesian Approach to Safe Imitation Learning”, Menda et al 2017

DropoutDAgger: A Bayesian Approach to Safe Imitation Learning

“Towards Personalized Human AI Interaction—Adapting the Behavior of AI Agents Using Neural Signatures of Subjective Interest”, Shih et al 2017

Towards personalized human AI interaction—adapting the behavior of AI agents using neural signatures of subjective interest

“Learning Human Behaviors from Motion Capture by Adversarial Imitation”, Merel et al 2017

Learning human behaviors from motion capture by adversarial imitation

“Learning from Human Preferences”, Amodei et al 2017

Learning from Human Preferences

“Deep Reinforcement Learning from Human Preferences”, Christiano et al 2017

Deep reinforcement learning from human preferences

“Learning through Human Feedback [blog]”, Leike et al 2017

Learning through human feedback [blog]

“Adversarial Ranking for Language Generation”, Lin et al 2017

Adversarial Ranking for Language Generation

“An Invitation to Imitation”, Bagnell 2015

An Invitation to Imitation

“Just Sort It! A Simple and Effective Approach to Active Preference Learning”, Maystre & Grossglauser 2015

Just Sort It! A Simple and Effective Approach to Active Preference Learning

“Algorithmic and Human Teaching of Sequential Decision Tasks”, Cakmak & Lopes 2012

Algorithmic and Human Teaching of Sequential Decision Tasks

“Bayesian Active Learning for Classification and Preference Learning”, Houlsby et al 2011

Bayesian Active Learning for Classification and Preference Learning

“DAgger: A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning”, Ross et al 2010

DAgger: A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

“Transformers As Variational Autoencoders”

Transformers as Variational Autoencoders

“Transformer-VAE for Program Synthesis”

Transformer-VAE for Program Synthesis

Sort By Magic

Annotations sorted by machine learning into inferred 'tags'. This provides an alternative way to browse: instead of by date order, one can browse in topic order. The 'sorted' list has been automatically clustered into multiple sections & auto-labeled for easier browsing.

Beginning with the newest annotation, it uses the embedding of each annotation to attempt to create a list of nearest-neighbor annotations, creating a progression of topics. For more details, see the link.

`bayesian-reward`

[see previous entry]

`human-preferences`

[see previous entry]

Wikipedia

Photoplethysmogram

Miscellaneous

Link Bibliography

https://arxiv.org/abs/2404.12358: “From r to Q^✱: Your Language Model Is Secretly a Q-Function”, Rafael Rafailov, Joey Hejna, Ryan Park, Chelsea Finn

link-bibliography
https://arxiv.org/abs/2404.08495: “Dataset Reset Policy Optimization for RLHF”, Jonathan D. Chang, Wenhao Shan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D. Lee, Wen Sun

link-bibliography
https://arxiv.org/abs/2401.05566#anthropic: “Sleeper Agents: Training Deceptive LLMs That Persist Through Safety Training”, Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Kamal Ndousse, Kshitij Sachan, Michael Sellitto, Mrinank Sharma, Nova DasSarma, Roger Grosse, Shauna Kravec, Yuntao Bai, Zachary Witten, Marina Favaro, Jan Brauner, Holden Karnofsky, Paul Christiano, Samuel R. Bowman, Logan Graham, Jared Kaplan, Sören Mindermann, Ryan Greenblatt, Buck Shlegeris, Nicholas Schiefer, Ethan Perez

link-bibliography
https://www.nytimes.com/interactive/2023/11/12/magazine/andrew-wylie-interview.html: “When Ruthless Cultural Elitism Is Exactly the Job”, David Marchese

link-bibliography
https://arxiv.org/abs/2310.01377: “UltraFeedback: Boosting Language Models With High-Quality Feedback”, Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, Maosong Sun

link-bibliography
https://arxiv.org/abs/2309.15807#facebook: “Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack”, Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, Matthew Yu, Abhishek Kadian, Filip Radenovic, Dhruv Mahajan, Kunpeng Li, Yue Zhao, Vladan Petrovic, Mitesh Kumar Singh, Simran Motwani, Yi Wen, Yiwen Song, Roshan Sumbaly, Vignesh Ramanathan, Zijian He, Peter Vajda, Devi Parikh

link-bibliography
https://arxiv.org/abs/2309.12053: “AceGPT, Localizing Large Language Models in Arabic”, Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Dingjie Song, Zhihong Chen, Abdulmohsen Alharthi, Bang An, Juncai He, Ziche Liu, Zhiyi Zhang, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, Jinchao Xu

link-bibliography
https://openai.com/index/introducing-superalignment/: “Introducing Superalignment”, Jan Leike, Ilya Sutskever

link-bibliography
https://www.theverge.com/features/23764584/ai-artificial-intelligence-data-notation-labor-scale-surge-remotasks-openai-chatbots: “AI Is a Lot of Work: As the Technology Becomes Ubiquitous, a Vast Tasker Underclass Is Emerging—And Not Going Anywhere”, Josh Dzieza

link-bibliography
https://arxiv.org/abs/2306.07567: “Large Language Models Sometimes Generate Purely Negatively-Reinforced Text”, Fabien Roger

link-bibliography
https://www.wsj.com/articles/microsoft-and-openai-forge-awkward-partnership-as-techs-new-power-couple-3092de51: “Microsoft and OpenAI Forge Awkward Partnership As Tech’s New Power Couple: As the Companies Lead the AI Boom, Their Unconventional Arrangement Sometimes Causes Conflict”, Tom Dotan, Deepa Seetharaman

link-bibliography
https://arxiv.org/abs/2305.15717: “The False Promise of Imitating Proprietary LLMs”, Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, Dawn Song

link-bibliography
https://arxiv.org/abs/2305.11064: “Bits of Grass: Does GPT Already Know How to Write like Whitman?”, Piotr Sawicki, Marek Grzes, Fabricio Goes, Dan Brown, Max Peeperkorn, Aisha Khatun

link-bibliography
https://arxiv.org/abs/2305.03047#ibm: “SELF-ALIGN: Principle-Driven Self-Alignment of Language Models from Scratch With Minimal Human Supervision”, Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, Chuang Gan

link-bibliography
https://arxiv.org/abs/2305.01569: “Pick-A-Pic: An Open Dataset of User Preferences for Text-To-Image Generation”, Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, Omer Levy

link-bibliography
https://www.wired.com/story/anthropic-ai-chatbots-ethics/: “A Radical Plan to Make AI Good, Not Evil”, Will Knight

link-bibliography
https://arxiv.org/pdf/2303.08774.pdf#page=12&org=openai: “GPT-4 Technical Report § Limitations: Calibration”, OpenAI

link-bibliography
https://www.forbes.com/sites/alexkonrad/2023/02/03/exclusive-openai-sam-altman-chatgpt-agi-google-search/: “OpenAI’s Sam Altman Talks ChatGPT And How Artificial General Intelligence Can ‘Break Capitalism’”, Alex Konrad, Kenrick Cai

link-bibliography
https://arxiv.org/abs/2212.10560: “Self-Instruct: Aligning Language Models With Self-Generated Instructions”, Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah Smith, Daniel Khashabi, Hannaneh Hajishirzi

link-bibliography
https://www.lesswrong.com/posts/t9svvNPNmFf5Qa3TA/mysteries-of-mode-collapse-due-to-rlhf#Inescapable_wedding_parties: “Mysteries of Mode Collapse § Inescapable Wedding Parties”, Janus

link-bibliography
https://arxiv.org/abs/2210.10760#openai: “Scaling Laws for Reward Model Overoptimization”, Leo Gao, John Schulman, Jacob Hilton

link-bibliography
https://arxiv.org/abs/2210.07792#eleutherai: “CARP: Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning”, Louis Castricato, Alexander Havrilla, Shahbul, Matiana, Michael Pieler, Anbang Ye, Ian Yang, Spencer Frazier, Mark Riedl

link-bibliography
https://arxiv.org/abs/2210.01241: “Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization”, Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, Yejin Choi

link-bibliography
https://www.anthropic.com/red_teaming.pdf: “Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned”, Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, Jack Clark

link-bibliography
https://arxiv.org/abs/2205.11275: “RL With KL Penalties Is Better Viewed As Bayesian Inference”, Tomasz Korbak, Ethan Perez, Christopher L. Buckley

link-bibliography
https://arxiv.org/abs/2112.09332#openai: “WebGPT: Browser-Assisted Question-Answering With Human Feedback”, Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, John Schulman

link-bibliography
https://openai.com/research/webgpt: “WebGPT: Improving the Factual Accuracy of Language Models through Web Browsing”, Jacob Hilton, Suchir Balaji, Reiichiro Nakano, John Schulman

link-bibliography
https://arxiv.org/abs/2112.00861#anthropic: “A General Language Assistant As a Laboratory for Alignment”, Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Jared Kaplan

link-bibliography
https://arxiv.org/abs/2109.10862#openai: “Recursively Summarizing Books With Human Feedback”, Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, Paul Christiano

link-bibliography
https://trajectory-transformer.github.io/: “Trajectory Transformer: Reinforcement Learning As One Big Sequence Modeling Problem”, Michael Janner, Qiyang Colin Li, Sergey Levine

link-bibliography
https://openai.com/research/fine-tuning-gpt-2: “Fine-Tuning GPT-2 from Human Preferences”, Daniel Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Dario Amodei, Alec Radford, Paul Christiano, Geoffrey Irving

link-bibliography
2018-fu.pdf: “StreetNet: Preference Learning With Convolutional Neural Network on Urban Crime Perception”, Kaiqun Fu, Zhiqian Chen, Chang-Tien Lu

link-bibliography

[Quote Of The Day]

[Site Of The Day]

[Annotation Of The Day]

[adblock public service announcement]