Situation Coverage Testing in Simulation for Mobile Robots

As part of the Assuring International Programme we are developing a mobile robot to demonstrate our safety engineering guidance, in particular our top-level safety guidance (SACE) and our machine learning safety guidance (AMLAS). As part of that, we need to test the robot in a wide variety of situations, and we need to use simulation to widen the set of tests we can do. Sadly, there are many challenges for the fidelity of simulation-based testing, and the space of possible tests we could conduct is infinite. So we’ll have to discover or develop new ways to build simulations, choose what tests to do, and justify that the results are trustworthy.

In this PhD project, you will investigate how to do this. There is existing work on situation coverage criteria [1,2], working out what situations are likely to be hazardous [3], searching for dangerous failures [4,5], but nothing that pulls it all together to make the result highly useful in real projects. Your PhD work will help achieve that.

Topics you might consider include, but are not limited to:

• How can we choose the most important situations to test, without missing anything important due to our perceptions and biases?
• Given a set of coarse-grained situations, how can we explore the space of minor variations (in times, in angles, in placements of obstacles) to avoid missing a dangerous failure due to an unlucky choice of exactly what test to run?
• How can we adequately model all the different surfaces and objects in our building without incurring impractically large human or compute costs?
• How can we model and describe the limitations of our simulations, in terms of objects, surfaces, human behaviour, and sensor capabilities?
• When we do this kind of testing, where does most of the fault-discovery benefit come from? Is it really from running the tests, or does much of it come from the modelling and test designs steps prior to that?

Most of the above topics could be investigated in a theoretical or practical way. For the practical approach, strong programming skills will be needed, and experience with robotics, simulation programming, ROS, and Gazebo will be very useful. For either approach, experience with safety engineering, in particular with hazard analysis and safety requirements derivation, will be very useful.