Building an Uncertainty-Robust Reinforcement Learning-based model for UAV self-separation under Uncertainty

Building an Uncertainty-Robust Reinforcement Learning-based model for UAV self-separation under Uncertainty

Master Thesis

Background

The autonomous operation of unmanned aerial vehicles (UAVs) plays an increasingly important role in research and commercial applications. The potential applications for these vehicles range from medical delivery and surveillance to high-density parcel delivery scenarios, many of which are expected to be performed in a fully autonomous manner. For this autonomy to be possible, vehicles must be able to self-separate – that is, take the state of other vehicles in their vicinity into account and perform manoeuvres to remain at a safe distance to them. The states of other vehicles are primarily determined by the vehicles themselves, using GNSS, and broadcast to all other vehicles in the airspace. Propagating these states and checking if a conflict is to occur with another vehicle is referred to as Conflict Detection.

Once a conflict is detected, it must be resolved: here, Conflict Resolution is performed. This step will also utilise the states of both vehicles. Many Conflict Resolution algorithms exist, and can be utilised for this self-separation task. In the real world however, the positions of the vehicles to avoid are uncertain, due to both position error stemming from GNSS and communication loss. Therefore, the resolutions are impacted negatively, leading to possible losses of separation between vehicles with them getting too close to each other. While it has shown that there can be a difference in performance between classical algorithms under this positional uncertainty, novel learning-based algorithms are yet to be used under the same conditions. Recent developments in transformer models for example, now widely used for Large Language Models, make them a suitable candidate for this self-separation task. What is not yet clear how these novel models will perform under uncertainty – it is hypothesised that these could outperform classical models if they learn under conditions with severe sensor noise and develop strategies to mitigate these uncertainties.

This thesis’ contribution will be development, training and tuning of a Reinforcement-Learning-based approach for the Conflict Resolution problem, which will allow for these uncertainties to be taken into account. Once developed, this method can be benchmarked against classical methods, and improved further to account for more uncertainties, such as those in the velocity and altitude, for example.

Tasks

The assignment will include the following tasks:

• Investigation of existing approaches for (RL-based) CD&R (Literature Study)
• Implementation of uncertainty scenarios for simulation (in Python, and using the BlueSky ATC simulator coupled with BlueSky-Gym)
• Model selection, tuning or development, based on simulation results (with algorithms such as SAC from stable-baselines3 or other)
• A robustness analysis and benchmark of the developed model, on more challenging scenarios and classical environments

Results

The final outcome of this assignment will be:

• An RL-based UAV conflict resolution strategy that can deal with positional uncertainty
• A technical thesis report describing the approach, results and conclusions of the work.
• Optional: a conference paper.

What do we expect from you?

• Master student aerospace engineering, mechanical engineering, control engineering or computer science
• Experience with programming (Python, Matlab)
• Experience with practical application of ML/RL (PyTorch, Keras, Tensorflow or other)
• Preferably good understanding of (aircraft) dynamics, simulation & control

What we offer?

• Enthusiastic colleagues who are experts in their field
• A flexible working space
• An environment where you have the opportunity to develop your skills and learn new ones
• A challenging assignment in a high-tech, result orientated work environment
• A thesis assignment allowance
• An informal corporate culture where your opinion counts!

About NLR

For more than 100 years, Royal NLR has been the ambitious knowledge organization with the will to keep innovating. From that motivation, we make the world of transportation safer, more sustainable, more efficient and more effective. We are on the threshold of ground-breaking innovations. Plans and ideas get moving when they are well fed with the right energy. Over 900 passionate professionals work on research and innovation. From aircraft engineers to psychologists and from mathematicians to application experts.

You will be working within the AOAP (Aerospace Operations: Air Traffic Management & Airports) department. Your colleagues are focused on solving real-world problems within air traffic management, airspace design, U-Space and other exciting domains.

Want to know more about your future thesis assignment?

For more information about the assignment contact Sasha Vlaskin sasha.vlaskin@nlr.nl.