Human-guided deep reinforcement learning for optimal decision making of autonomous vehicles

Abstract

Although deep reinforcement learning (DRL) methods are promising for making behavioral decisions in autonomous vehicles (AVs), their low training efficiency and difficulty to adapt to untrained cases hinder their applications. Introducing a human role in the DRL paradigm could improve training efficiency by using human prior knowledge and overcome untrained cases in deployment by online human takeover. In this study, a novel value-based DRL algorithm that leverages human guidance to improve its performance is proposed for addressing high-level decision-making problems in autonomous driving. We develop a new learning objective for DRL to increase the value of the human policy over the undertrained DRL policy so that the DRL agent can be encouraged to mimic human behaviors and thereby utilizing human guidance more efficiently. Our method can autonomously evaluate the importance of different human guidance, which makes it more robust for variation of human performance. The proposed DRL algorithm was used to address a challenging multiobjective lane-change decision-making problem. We collected human guidance from a human-in-the-loop driving experiment and evaluated our method in a high-fidelity simulator. Results validated the advantages of the proposed algorithm in terms of training efficiency and optimality in the decision-making problem compared to the baselines of state-of-the-art existing methods. Results also revealed the favorable fine-tuning ability of the proposed algorithm, which is promising for addressing the long-tail issue in DRL-based autonomous driving. Our methodology does not introduce additional domain knowledge so that it can be seamlessly applied to other similar issues. The supplementary video is available at https://youtu.be/Ec7WkqeLsB8.