How small objects are when they are first detected is an important performance metric of the Neural Network that does the hard work of scrutinizing every patch of pixels for the presence or absence of an aircraft. The certifiability of these so-called Deep Convolutional Neural Networks has been the subject of great interest and discussion in the industry
, and Daedalean has been working with the regulators to define how to
think about this at all
. One of the fundamental requirements we identified is that the datasets you test on should be representative of the conditions you expect to see when you deploy the system in flights for real, and that they should not have biases
that skew your test results in favourable ways. If that sounds intuitive, it is also founded on sound statistical learning theory
The published video has an obvious bias in that almost all encounters are seen above the horizon, against the blue sky or against a background of clouds. For a low flying drone this may be representative, but since most aircraft maintain their altitude most of the time, these are also the encounters likely to be harmless, while at the same time easier to spot than below-horizon ones.
A second bias is that we appear to be in the Nevada desert, where hot and dry air typically makes for fantastic visibility. In coastal areas or further inland, representative conditions include a haze that reduces the contrast at greater distances. To be able to distinguish a white from a black object, even when the camera resolution
permits, becomes impossible if the dynamic range
of the pixels is too low. This is where the difference between 8- and 12-bit pixel cameras becomes very apparent.
A third, more subtle bias in this dataset is that most encounters are not
on collision trajectories. A collision trajectory is characterized by the fact that the target appears to remain in the same relative position to the ownship*, only slowly getting bigger. So the fact that we can see most targets move means that in this dataset we only show that we are pretty good at detecting things we are unlikely to fly into if both we and the target maintain the level and heading.
_______________*Aerospeak for "ourselves"