People rarely understand the world through a single source of information. We listen to what someone says while observing their expressions, read text alongside charts and images and combine what we see with what we already know. Most artificial intelligence (AI) systems, however, have historically been far more limited. Multimodal AI seeks to overcome these limitations by enabling AI systems to interpret and integrate multiple types of information simultaneously.
In an internally sponsored project, CNA took a deep dive into multimodal AI to consider its potential for evaluating risk associated with high-altitude unmanned aircraft systems (UAS) operations around airports. CNA is an independent, not-for-profit research and analysis organization that has explored new and emerging technologies for the US government for 84 years. Here is what we learned.
What Multimodal AI Is, Why It Is Important For Emerging Aviation
Multimodal AI processes and understands multiple types of data simultaneously. These types of data, called “modalities,” can include text, images, video, geospatial data and sensor data, such as acoustic, temperature, pressure and acceleration. Multimodal AI algorithms have more context than unimodal AI algorithms, which allows them to perform tasks that require integrating information from different sources.
Multimodal AI has potential applications in many fields, including autonomous systems, UAS and counter-UAS. For example, a unimodal AI drone detection system could perceive a potential drone on an RGB camera feed under clear skies. A multimodal AI system could then help users discern between benign and threatening objects by fusing LiDAR (light detection and ranging) data, acoustic sensor readings and reports from past sightings to predict the identity of the drone and the potential threat it poses in varying weather conditions.
Likewise, successful autonomous technologies must be able to accurately and holistically perceive and interpret their surrounding environments, taking in all modalities of data. By integrating multiple sources of data, multimodal AI can provide rich contexts for decision support and more human-aligned outputs. This would unlock new insights and solutions in autonomous systems and UAS.
In the air traffic environment, multimodal AI could produce recommendations to Federal Aviation Administration (FAA) air traffic managers (ATMs) so high altitude UAS can safely share airspace with manned aircraft around airports. Currently, FAA ATMs manually receive and approve such requests. Based on multiple data sources and factors, multimodal AI can improve how quickly those decisions could be made.
What It Takes to Integrate Data for Multimodal AI
The process of integrating multiple modalities of data into a usable form for AI models depends on the data. Analysts typically need to take one or more of the following steps to integrate the data sources:
- Alignment: Defining how modalities relate to each other across space, time, or meaning
- Co-learning: Using what is learned from one modality to improve the understanding of another modality
- Fusion: Combining modalities into a single, integrated form
- Representation: Converting modalities into machine-readable features
- Translation: Mapping one modality into another while retaining the original meaning
Analysts must choose which actions to take based on the use case of interest and a comprehensive understanding of the data. For example, to develop the drone detection system described previously, the video and acoustic data could be fused in several different ways.
Analysts could combine the raw visual and acoustic data before the data are entered into a single AI model, referred to as “Early Fusion.” Doing so enables them to develop a cross-modal model that considers all inputs but is also somewhat difficult to interpret.
As an example of “Late Fusion,” analysts could develop separate AI models for the visual and acoustic data and combine the two model outputs. Doing so would enable them to update and interpret the models for each input separately, but challenges may emerge if the two model outcomes conflict.
Analysts could alternatively use “Intermediate Fusion,” meaning they would pre-process the data before fusing the data as model input. Doing so strikes a balance between early and late fusion but increases technical complexity.
To decide the best route, analysts must weigh each method’s advantages and disadvantages based on the specific data and use case.
A Use Case: Analyzing UAS Further Coordination Requests
Currently, most UAS operators request authorization to fly in controlled airspace through the Low Altitude Authorization and Notification Capability (LAANC) system. LAANC provides near-real-time automated approvals when the requested operation meets certain criteria, including being within the altitude limits defined by FAA’s UAS Facility Maps. In these maps, the FAA divides the airspace surrounding an airport into grids of 30×30 seconds of latitude and longitude. Each grid shows a maximum altitude of between 0 and 400 feet above ground level.
When an operator seeks to fly above the altitude ceilings, they must submit a further coordination request that an ATM will review manually. The review process can vary by air traffic control (ATC) facility because standardized rules for evaluation have not been defined. In addition, further coordination requests expire if not approved within 3 to 24 hours of an operation’s start time. As UAS operations increase, the volume of these coordination requests will also increase and stress the FAA review process.
Because the FAA requires a more scalable and consistent process, at CNA we used multimodal AI to study trends into the approvals of further coordination requests. We determined, based on historical data, multimodal AI models can provide recommendations to support ATM decisions regarding these requests.
The model we used draws on multimodal data that included: tabular and text (further coordination requests including altitude, time, location, safety justification); geospatial, in the form of facility maps that had ceilings, airspace class etc.; and geospatial data relating to historical manned aircraft flight patterns.
For our study, we processed such data from Boston Logan International Airport for the years 2022 to 2026 in a two-step approach. First, we aligned the data sources temporally and spatially. We divided the data into hourly intervals by aligning the requested operation start time with the historical manned aircraft flight patterns at that time of day. We also divided the data by spatial position based on FAA UAS Facility Map grids. Doing so enabled us to relate the requested operation location (including one or more grids) with the corresponding FAA UAS Facility Map altitude ceilings and the density of historical manned aircraft flights in the grid. Second, we used early fusion to combine all data sources into an AI-ready dataset that could be entered into the AI model. The dataset had 10 variables, including text, numeric, date time and categorical data types.
Using our AI-ready dataset, we applied an automated machine learning framework called Autogluon to train 36 models for comparison. We evaluated 26 models that predicted the outcome of the request (e.g., approve, deny, cancel) and 10 models that predicted the reason for the outcome (e.g., denied due to location, denied due to altitude, denied due to insufficient safety mitigation).
We found that the best models predicted request outcomes accurately in about 70 percent of cases. In addition, the most influential factors affecting whether a request was approved or denied were the safety justification text, operation date and time and requested flight altitude.
The models are not intended to make decisions autonomously or replace human ATM decision-making. Instead, they can provide ATMs with an immediate, data-informed recommendation that complements their expertise. These AI models can also identify key factors that influence approval outcomes and provide insights into how decisions are made across different ATC facilities.
The Future of Multimodal AI: A Systems Approach
As AI technologies continue to advance, multimodal AI will be a critical enabler for autonomous technologies, enhanced decision-making and future airspace management systems. Multimodal AI also has potential application in other domains, such as enhancing situational awareness for public safety (see prior AG coverage of FRAME for a description of CNA’s work developing a common operating picture for first responders using multimodal data).
However, using multimodal AI successfully requires more than just collecting various types of data and selecting an appropriate model. To use multimodal AI effectively and responsibly, developers must address the key technical challenges that arise when leveraging diverse data modalities and understand how the data, technology, people, processes and broader environment interact as a system.
In addition, organizations must consider factors such as data availability and quality, interoperability, cybersecurity, computational demands, human-AI interaction, governance and the reliability of AI outputs under changing operational conditions. Adopting a holistic systems perspective will help ensure that multimodal AI solutions are not only technically capable but also trustworthy, resilient, scalable, and aligned with the operational needs of UAS stakeholders.
*CNA has supported the FAA’s efforts to modernize air traffic management for more than 30 years. Over the last decade, the organization has helped the FAA prepare to integrate UAS and advanced air mobility into the National Airspace System.
All images courtesy of CNA.