Reinforcement learning with possibility theory

Abstract

Deep Reinforcement Learning (RL) agents often struggle to balance exploration and exploitation due to their value estimates not accounting for epistemic uncertainty. Possibility theory, with its maxitive calculus and less restrictive normalization constraints, offers a method for tracking and using epistemic uncertainty. This thesis investigates whether possibilistic modeling of uncertainty can drive principled optimistic exploration. Three algorithmic variants are proposed: (i) Possibilistic DQN, which models Q-values via a Gaussian and acts via a parameter-free closed-form maximum expected value; (ii) Possibilistic Q-Ensembles, which maintain possibility weighting over multiple Q-networks and update them using a possibilistic Bayes rule; and (iii) Possibilistic Model-Based Learning, a pair of zero- and one-step planning algorithms that propagate optimistic targets from possibilistic models. The models are tested on benchmark environments along with sparse and stochastic variants. The study confirms the efficacy of maximum-expected-value optimism in the first method, particularly in deterministic and sparse environments. The ensemble possibilistic approach outperforms standard non-weighted ensembles and DQN. The third model underperformed relative to baselines; further work is needed to tune hyperparameters and improve model accuracy. Overall, we demonstrate the utility of possibility theory to capture uncertainty and make exploration more efficient.