At Daedalean, we have always focused on developing products for aviation with an eye toward future certification.
Why couldn't we just create and train our neural networks, test them inside out on drones, and then proudly present the drone-tested systems to the aviation industry?
The reason is that we have set our sights beyond drones.
Having a working algorithm is not enough when it comes to safety criticality. To obtain certification by aviation regulators, the complete development lifecycle must be documented, validated, and verified from the start. This includes having internal processes that meet standards like DO-178C/DO-254. Knowing this, we framed our development effort by creating all the necessary tools and procedures to gather data for certification with regulation authorities like EASA and the FAA.
The specific challenge we faced was related to the fact that the core technology we implement, namely machine learning, did not have officially specified compliance means for civil aviation. For us, it was not just about following the industry-benchmark set of verification routines, safety assessment procedures, and step-by-step documentation, but also helping to build a mode of thinking that could allow regulatory authorities to approach the certification of AI1.
Why couldn't we just create and train our neural networks, test them inside out on drones, and then proudly present the drone-tested systems to the aviation industry?
The reason is that we have set our sights beyond drones.
Having a working algorithm is not enough when it comes to safety criticality. To obtain certification by aviation regulators, the complete development lifecycle must be documented, validated, and verified from the start. This includes having internal processes that meet standards like DO-178C/DO-254. Knowing this, we framed our development effort by creating all the necessary tools and procedures to gather data for certification with regulation authorities like EASA and the FAA.
The specific challenge we faced was related to the fact that the core technology we implement, namely machine learning, did not have officially specified compliance means for civil aviation. For us, it was not just about following the industry-benchmark set of verification routines, safety assessment procedures, and step-by-step documentation, but also helping to build a mode of thinking that could allow regulatory authorities to approach the certification of AI1.
1 To that end we have co-authored, jointly with EASA, a two-part report on guidelines for learning assurance — Concepts of Design Assurance for Neural Networks (CoDANN I, CoDANN II). Find more on that in our blog posts here and here. These papers established a principle, which was later used by EASA in their published guidance on the compliance means for AI applications, describing how machine learning-based systems can be proven as safe, predictable, and fit for purpose.
The system has neural networks as a core. This core, however, is also ‘wrapped’ by layers of classical code and hardware, and all these other layers must not bypass the aviation industry’s standard engineering development and verification processes that have been in place and improved upon for many years. So, to guarantee that our systems are as safe and as predictable as it gets in compliance with aviation standards, we have set up our internal processes to carefully document every step for certification purposes – and also, we've invested a lot of resources and effort into building an exceptional certification, systems and safety team.
What We Actually Do
In partnership with avionics manufacturer Avidyne, we are developing PilotEye – the first AI-based onboard pilot aid system for general aviation.
PilotEye serves as an extra set of eyes in the cockpit, visually detecting airborne hazards, including non-cooperative traffic, such as drones, gliders, or aircraft without ADS-B or even transponders. In December 2021, Avidyne applied for an STC (supplemental type certificate2) with the FAA, with concurrent validation by EASA.
PilotEye serves as an extra set of eyes in the cockpit, visually detecting airborne hazards, including non-cooperative traffic, such as drones, gliders, or aircraft without ADS-B or even transponders. In December 2021, Avidyne applied for an STC (supplemental type certificate2) with the FAA, with concurrent validation by EASA.
2 A supplemental type certificate (STC) applies to modifications made to an existing aircraft (of a specific model). So, the flight software, databases, and all the hardware we build our systems on are subject to STC.
Avidyne is a renowned avionics products suite manufacturer and integrator with experience in creating and certifying traffic advisory systems. However, this marks the first time their product will be based on neural networks, with Daedalean supplying the technology.
As a result, responsibilities are divided: Avidyne oversees general supervision as the STC applicant and ensures compliance of the system hardware, except the FPGA created by Daedalean, with DO-254/DO-160 standards, while Daedalean handles our unique algorithms and compliance with the following standards:
As a result, responsibilities are divided: Avidyne oversees general supervision as the STC applicant and ensures compliance of the system hardware, except the FPGA created by Daedalean, with DO-254/DO-160 standards, while Daedalean handles our unique algorithms and compliance with the following standards:
- System: ARP 4754/4761
- Hardware (FPGA): DO-254
- Software: DO-178C
The PilotEye project is not only the first product with our technology, but also serves as a model for future endeavors. We aim to elevate the design assurance level of our products beyond DAL-C, so during this process, we are gathering necessary data and formulating strategies for the future. Transitioning from DAL-C to DAL-A involves additional verification steps, but the actual implementation steps remain the same.
Certification requires collecting all relevant evidence, as regulators consider a lack of evidence equivalent to non-completion. In any certification project, we must first outline our development and testing plans, then execute those plans – in the case of ML algorithms, adhering to the W-shaped development and verification process – and finally, document everything for regulatory review. This entails not only development, testing, and verification, but also documenting every step of the learning process and tracking each bit of data used for training and testing datasets from its source through the annotation process.
Certification requires collecting all relevant evidence, as regulators consider a lack of evidence equivalent to non-completion. In any certification project, we must first outline our development and testing plans, then execute those plans – in the case of ML algorithms, adhering to the W-shaped development and verification process – and finally, document everything for regulatory review. This entails not only development, testing, and verification, but also documenting every step of the learning process and tracking each bit of data used for training and testing datasets from its source through the annotation process.
Certification & Safety: Keeping medical records
We have a dedicated cert team supervised by Ian Whittington to manage this comprehensive process leading to certification. “I’ve worked on traditional embedded software for around 20 years, so I have a lot of experience using DO-178C in practice,” says Ian, “My role has two aspects: First, there are subsystems in our product that are based on classic software, with no machine learning involved, and in the neural network-based subsystems, there are also the enclosing parts of traditional code. For all the processes related to this traditional software development, I ensure that everything within the development life cycle, from initial planning to the final testing, is properly documented and meets the existing guidelines. And second, there is this new, first-in-the-industry work where our machine learning team creates, trains, and tests the models for machine-learned image analyses. Here, I help document the processes we use to manage data, develop neural-networks and run them on embedded hardware, and ensure they align to the Learning Assurance (W-shape) model that we helped develop with EASA.”
One vital task along this way is analyzing changes to the aircraft implied by the installation of our system to determine the criticality of modifications. Certain questions are crucial: How could the system fail? What are the consequences of the system failing? Could people be harmed? Could it add to the pilot's workload and compromise safety? Or would there be no impact whatsoever? Responsibility for identifying potential failures and their impact on the system (functional hazard assessment) and analyzing failures and their effects at the component level (the preliminary safety assessment) falls to our safety engineer, Iiria Rodriguez.
"I perform analysis on all iterations, from preliminary design ensuring that the architecture meets safety standards to the most definitive analysis verifying that this system functions safely and can be mounted on aircraft and certified. For machine learning-based systems, we do follow the existing industry standards during our safety process. Nevertheless, there are specific challenges to ML that need to be addressed from a new, innovative perspective, and I find that very motivating and exciting", says Iiria.
Hugo Silva has vast professional experience certifying avionic systems in both military and civil aviation. His experience with the full lifecycle of aircraft system development is valuable in aligning our machine learning expertise with the certification requirements. In practice, this means developing our means of compliance to be aligned with EASA's First Usable Guidance and ensuring they are indeed certifiable. "An awesome part of my job is coordinating certification activities between the FAA/EASA and our machine learning experts. Braving these uncharted skies is the main reason I joined Daedalean," Hugo explains.
Hugo Silva has vast professional experience certifying avionic systems in both military and civil aviation. His experience with the full lifecycle of aircraft system development is valuable in aligning our machine learning expertise with the certification requirements. In practice, this means developing our means of compliance to be aligned with EASA's First Usable Guidance and ensuring they are indeed certifiable. "An awesome part of my job is coordinating certification activities between the FAA/EASA and our machine learning experts. Braving these uncharted skies is the main reason I joined Daedalean," Hugo explains.
Systems Engineering: Seeing the big picture
The system engineering team, led by Ramanujam Kalale, works in close collaboration with the certification and safety team. Its members focus on defining, designing, and verifying systems at the system level rather than on their particular components, developing end-to-end functionality. The team’s main task is to ensure that all the pieces of a system fit together seamlessly and meet the necessary design requirements.
Apart from managing the team, Ramanujam takes an active role in system development, making sure the design implementation takes the system requirements into consideration. The components that make up the system are selected after diligent market research and against specific functional and physical requirements. Another important aspect of his task is to define how our equipment fits with the rest of the aircraft in a functional and certifiable way. Appropriate communication standards and installation aspects (mechanical, structural and electrical, among others) are to be considered before defining the final design. “My broad experience in diverse domains of discipline in aerospace (systems, certification, overall design coordination) helps me identify aspects that may tend to be overlooked sometimes. Ultimately, my responsibility is to keep the big picture at the aircraft level” says Ramanujam.
Apart from managing the team, Ramanujam takes an active role in system development, making sure the design implementation takes the system requirements into consideration. The components that make up the system are selected after diligent market research and against specific functional and physical requirements. Another important aspect of his task is to define how our equipment fits with the rest of the aircraft in a functional and certifiable way. Appropriate communication standards and installation aspects (mechanical, structural and electrical, among others) are to be considered before defining the final design. “My broad experience in diverse domains of discipline in aerospace (systems, certification, overall design coordination) helps me identify aspects that may tend to be overlooked sometimes. Ultimately, my responsibility is to keep the big picture at the aircraft level” says Ramanujam.
Désirée Clausen, a system engineer, is responsible for system design and requirements, breaking down high-level requirements into more specific and detailed requirements for both software and hardware. The reason we are so careful with the system development process is to reduce the chance of mistakes that could affect aircraft safety. One key part of this is ensuring that we correctly outline the system's requirements and design – what the system is supposed to do. After this, we need to check that it has been built and is working as planned.
According to Désirée, “The requirements come from the architecture and the functions we want to achieve at the end. Such decomposition is iterative: Each function can be defined very broadly in the beginning, but then we constantly need to do splits, be very concise, every time go one level of abstraction lower, into more and more detail. It is necessary for the certification, but even more important is that any issues or gaps can be found early on in the development process.
One of the most interesting things about the systems department is that we’re sitting on the meta-level, so to speak. Our work encompasses many different disciplines, and we coordinate the other departments. A function may be implemented in software but require some specific hardware parts for it to actually work. All our processes are integrated because everyone is involved: if the machine learning team does something that impacts how software is implemented, someone from the software team will be there to review. It is like the work of a conductor of an orchestra.”
According to Désirée, “The requirements come from the architecture and the functions we want to achieve at the end. Such decomposition is iterative: Each function can be defined very broadly in the beginning, but then we constantly need to do splits, be very concise, every time go one level of abstraction lower, into more and more detail. It is necessary for the certification, but even more important is that any issues or gaps can be found early on in the development process.
One of the most interesting things about the systems department is that we’re sitting on the meta-level, so to speak. Our work encompasses many different disciplines, and we coordinate the other departments. A function may be implemented in software but require some specific hardware parts for it to actually work. All our processes are integrated because everyone is involved: if the machine learning team does something that impacts how software is implemented, someone from the software team will be there to review. It is like the work of a conductor of an orchestra.”
German Hernandez’s job is to verify the implementation of system requirements. He writes procedures, defines test scenarios, works on lab tests, ground tests, and flight tests. Ultimately, it is about executing the procedures, but a significant amount of effort also goes into defining the verification process. We need to be able to determine how exactly we will verify that the requirements have been met. It's not just about defining how the system should work but also how to verify that it does work.
“We use a laboratory test bench to simulate the environment in which the system will operate during flight, including all the inputs and outputs, cables, connectors, and electrical power,” explains German.
“Industry standards like RTCA DO-365 provide a vast list of scenarios to verify requirements. We create hundreds of scenarios with different sets of conditions. For example, for the traffic detection function, conditions would include the locations of the airplane and the intruder, the weather, the environment, etc. And we use both real data and simulations. Our simulation team creates high-quality videos using our proprietary tool or we use real videos of a plane flying, manipulated in a manner to test a particular case, for example, the aircraft is at 8000 feet, an intruder coming in at a similar altitude at 150 knots.
The testing involves simulating inputs to the system we're certifying to see how it responds under the worst possible conditions to determine any potential flaws. I work with people from different teams — simulation, platforms, hardware, machine learning – they are experts in their particular fields and may focus on the details, but my role is to understand the big picture.”
“Industry standards like RTCA DO-365 provide a vast list of scenarios to verify requirements. We create hundreds of scenarios with different sets of conditions. For example, for the traffic detection function, conditions would include the locations of the airplane and the intruder, the weather, the environment, etc. And we use both real data and simulations. Our simulation team creates high-quality videos using our proprietary tool or we use real videos of a plane flying, manipulated in a manner to test a particular case, for example, the aircraft is at 8000 feet, an intruder coming in at a similar altitude at 150 knots.
The testing involves simulating inputs to the system we're certifying to see how it responds under the worst possible conditions to determine any potential flaws. I work with people from different teams — simulation, platforms, hardware, machine learning – they are experts in their particular fields and may focus on the details, but my role is to understand the big picture.”
Aiming for certification
Having set our sights high early on, we have built a team equal to our ambition. Whether our systems are based on machine learning and require a whole new way of achieving certification or if they make use of classical programming, we are laser focused on ensuring safety and compliance. When our technology is certified and hits the market, our engineers will have toiled for countless hours to guarantee both.