Mastering the management of the vertical flight profile with reinforcement learning

Abstract

In a Markov decision process, the optimal strategy, also referred to as a policy in the context of machine learning, is the one that pinpoints the most advantageous action to maximise the expected long-term reward based on observations from the environment. This paper defines the Markov decision process for uncovering the optimal policy for managing the vertical profile of a flight, including vertical speed and acceleration control, using reinforcement learning. Unlike previous approaches, this definition prohibits arbitrary penalty functions designed to enforce operational constraints or destination-specific objectives. Instead, this paper demonstrates that the inherent nature of the vertical profile management problem motivates the agent to arrive at the destination while reducing fuel consumption and/or time. Furthermore, the use of action masking ensures that vertical speed and acceleration commands are chosen in accordance with operational constraints such as the aircraft's speed limit. The scope of the experiment is primarily focused on the descent phase of flight. However, the defined Markov decision process is universal and can be extended to cover the entire flight phase. The investigation is concretely expressed by training the policy in a deterministic scenario where the aircraft descends without any intervention from air traffic control. Promising results highlight reinforcement learning's versatility, demonstrating its ability to master the vertical profile management problem.