AI-Enabled Computed Vision Air-to-Air Refueling Performance Assessment and Automated Pilot Debrief System

Air-to-air refueling (AAR) is a critical Department of the Air Force and Department of the Navy capability enabling global mobility, long-range strike, combat airpower projection, and sustained aviation operations. However, aerial refueling remains one of the most technically demanding and safety-critical aviation tasks performed by aircrews. Successful execution requires precise aircraft positioning, coordinated control inputs, accurate closure rate management, continuous situational awareness, and effective crew coordination between tanker and receiver aircraft.Current aerial refueling training and post-flight assessment processes rely heavily on instructor observation, pilot self-assessment, manual video review, and subjective evaluation of performance. These approaches provide limited ability to quantitatively assess the thousands of individual performance variables that influence successful refueling operations. Existing methods do not provide pilots with objective, moment-by-moment feedback on aircraft positioning, closure rates, control inputs, deviations from desired procedures, or trends in individual performance over time.This topic seeks to develop, demonstrate, and transition an artificial intelligence (AI)-enabled computed vision capability that (post-flight) automatically analyzes aerial refueling operations and provides objective, data-driven pilot performance feedback. The desired capability will mature existing advances in autonomous aerial refueling analytics and apply them toward training, readiness, and safety improvement by transforming aerial refueling video and available aircraft performance data into actionable post-flight debrief products.The developed system shall provide an automated "hot wash" which analyzes aerial refueling events, identifies performance deviations, and generates intuitive visualizations that allow pilots and instructors to better understand execution quality and areas for improvement. The system shall reduce reliance on subjective recollection and manual review by providing objective measurements of aircraft performance throughout the refueling profile.The capability shall focus on analysis of tanker-mounted video data, such as KC-46 aerial refueling camera systems, while maintaining an architecture capable of expanding across additional tanker and receiver platforms. The vendor may pick (with Government concurrence) the first 3-5 receiver platforms to use in developing this system. Future applicability includes KC-135, C-17, all fighter receiver aircraft, naval aviation communities, autonomous aerial refueling systems, and other aviation training environments requiring advanced performance analytics.The desired solution shall leverage artificial intelligence, machine learning, computer vision, and advanced human-machine interfaces to provide capabilities including:• Automated detection and tracking of tanker (boom and hose) and receiver aircraft during aerial refueling operations.• Measurement of relative aircraft position, movement, velocity, and alignment throughout the refueling engagement.• Assessment of aircraft position within the approved refueling envelope.• Analysis of closure rates, approach vectors, aircraft stability, and deviations from nominal procedures.• Ability to correlate video with other aircraft data if/as available (e.g. telemetry, fuel transfer rate, fuel state, and pilot inputs) to evaluate performance drivers.• Generation of objective performance scores and trend analysis across multiple training events.• Creation of visual performance assessment products for receiver pilot post-flight review of their performance during refueling, including graphical overlays, heat maps, event timelines, and immersive or interactive debrief environments.• Support for instructor review, adjudication, and refinement of automated assessments during initial deployment.The system should evaluate innovative approaches for integrating automated analytics with existing or future aviation training environments, including virtual reality (VR), augmented reality (AR), mission rehearsal systems, simulator environments, or other computer-based training tools.The solution must address operational challenges associated with military aviation data, including secure processing of controlled aviation video, automated data protection, and protection of sensitive operational information. The capability should consider approaches for minimizing processing burden on operational units while enabling rapid generation of post-flight assessments.The Government seeks solutions which provide an operationally relevant prototype supporting aerial refueling training organizations. Successful completion of this effort will provide Air Force and joint aviation communities with a next-generation training and assessment capability that improves pilot proficiency, increases the training value of each refueling event, reduces equipment damage and maintenance impacts, and enhances overall mission readiness.The desired end state is an operationally demonstrated capability ready to implement for at least one type of receiver aircraft and that can support transition into existing aviation training ecosystems and provide a scalable foundation for future AI-enabled aviation performance assessment across the Department of Defense.

Objective

Develop, integrate, and demonstrate an artificial intelligence (AI)-enabled computed vision and human-machine interface capability that transforms aerial refueling operations from subjective, observation-based evaluation into an objective, data-driven performance assessment and adaptive training environment. The system shall leverage advanced computer vision, machine learning, and aviation (flight dynamics) analytics techniques to automatically extract operationally relevant performance metrics from aerial refueling events, quantify deviations from desired flight profiles, and generate actionable post-flight feedback to improve aircrew proficiency, reduce human-factor-driven mishaps, and enhance mission readiness.The capability shall analyze video from cameras mounted on the tanker (esp. the KC-46) to provide near-real-time post-flight reconstruction of aerial refueling engagements by correlating aircraft position, motion dynamics, refueling geometry, pilot inputs, and operational context to identify performance trends and training opportunities. The developed system shall perform automated assessment of complex aerial refueling behaviors including aircraft stability, closure rate management, alignment accuracy, control surface position, other situational awareness indicators, and adherence to established refueling procedures.The objective is to mature existing advanced AI-enabled aviation analytics into an operationally relevant post-flight "hotwash" and training capability that increases the instructional value of each aerial refueling sortie, reduces reliance on subjective human assessment, accelerates pilot proficiency development, decreases equipment damage associated with refueling deviations, and establishes a scalable foundation for future intelligent aviation training systems across Department of the Air Force and joint aviation communities.The resulting prototype shall demonstrate the ability to ingest operational aerial refueling data, autonomously generate objective performance assessments, provide intuitive visualization and debrief products, and support transition into existing aviation training, evaluation, and modernization ecosystems. Importantly, the system will also include automated security management of the video so that implementation is seamless and complies with security classification guidance.

Phase I

As this is a Direct-to-Phase-II (D2)2) topic, no Phase I proposals will be accepted and no Phase I awards will be made under this topic. Offerors responding to this topic shall demonstrate "Phase I-type" technical feasibility through prior research, development, laboratory validation, commercial development, or operational demonstrations performed outside of the SBIR/STTR program. The purpose of this demonstration will be for the Government to determine the feasibility of the ideas that appear to have the potential to satisfy the objectives of this topic:

The Government is seeking offerors with demonstrated experience and technical maturity equivalent to a successful Phase I effort. Proposals shall demonstrate existing capabilities, relevant past performance, or validated technologies applicable to the development of an AI-enabled aerial refueling performance assessment and automated pilot debrief capability.

Qualified offerors shall demonstrate experience in at least half of following technical areas:

 Development and application of artificial intelligence, machine learning, and computer vision algorithms for analyzing dynamic objects in video imagery.

 Processing and interpretation of high-speed motion imagery to determine object position, movement, velocity, orientation, and spatial relationships.

 Development of algorithms capable of tracking aircraft, vehicles, or other dynamic platforms in complex operational environments.

 Aviation-focused data analytics, flight performance monitoring, flight simulation, or aircrew training systems.

 Development of human-machine interfaces, virtual reality (VR), augmented reality (AR), immersive training environments, or advanced visualization tools.

 Integration of sensor data, video analytics, telemetry, or flight performance information into automated assessment systems.

 Experience handling, processing, protecting, or analyzing controlled aviation or operational data.

 Automated video processing techniques including object recognition, classification, filtering, masking, or data protection methodologies.

 Development of secure software systems capable of operating within government aviation environments.

Offerors shall demonstrate prior technical maturity equivalent to a Technology Readiness Level (TRL) 4 or greater at contract award. Relevant experience may include laboratory demonstrations, commercial applications, government-sponsored research, operational prototypes, or other non-SBIR/STTR funded development activities.

The offeror shall demonstrate the ability to transition existing technical capabilities into an operational prototype supporting aerial refueling performance analysis. Successful offerors will demonstrate sufficient technical foundation to immediately begin Phase II activities including adaptation of computer vision algorithms, integration of aviation video data, development of automated performance metrics, and operational prototype demonstration.

Evaluation of Phase I-equivalent experience will consider:

 Relevance of previous computer vision, artificial intelligence, or machine learning development efforts.

 Demonstrated ability to analyze complex motion-based systems, especially aviation maneuvers.

 Experience applying technology within aviation, aerospace, defense, and related training environments.

 Demonstrated ability to mature existing technologies into operationally relevant prototypes.

 Ability to address cybersecurity, data protection, and integration challenges associated with military aviation applications.

Phase II

Phase II will develop, integrate, and demonstrate an operational prototype of an AI-enabled computer vision air-to-air refueling (AAR) performance assessment and automated pilot debrief system. The effort will mature existing autonomous AAR analytics into a post-flight training and evaluation capability providing objective, data-driven feedback to tanker and receiver aircrews.The prototype shall ingest AAR video and available aircraft performance data, identify deviations from desired procedures, and generate automated post-flight hotwash products. The system shall reduce reliance on pilot memory and manual instructor video review through objective measurements, visualization, and historical trend analysis.Task 1: System Architecture, Data Acquisition, and Secure ProcessingEstablish the architecture to ingest, process, analyze, and protect AAR video and associated aircraft data.Activities shall include:Develop system architecture, data flows, and cybersecurity approach.Establish procedures for CUI and operational aviation data.Integrate available tanker-mounted cameras, aircraft video, and applicable flight-performance data.Evaluate local processing with transfer of lightweight outputs versus secure video transfer to approved computing environments.Develop data-management procedures supporting expansion across tanker/receiver platforms.Deliverables: System Architecture Design; Data Management and Security Plan; CUI Processing Workflow; Initial Prototype Data Ingestion Capability.Task 2: Computer Vision Model Adaptation and Performance AnalyticsAdapt and mature existing computer vision technologies for automated AAR analysis, including aircraft motion, relative positioning, and flight-performance assessment.The capability shall:Detect/track tanker boom or hose and receiver aircraft.Determine relative position, orientation, and movement.Measure closure rates and approach vectors.Assess position within the approved refueling envelope.Identify boom, probe, or hose/drogue alignment.Analyze stability and deviations from nominal procedures.Support multiple tanker and receiver configurations.Demonstrate performance equivalent to TRL 6+ through prototype demonstration in a relevant operational environment.Deliverables: Adapted Computer Vision Models; Aircraft Tracking Algorithms; Performance Assessment Algorithms; Technical Validation Report.Task 3: Automated Performance Assessment and Pilot FeedbackConvert AAR data into objective post-flight performance feedback, including:Moment-by-moment scoring against nominal profiles.Positioning heat maps relative to the refueling envelope.Closure-rate and approach-vector analysis.Timeline reconstruction of key events.Identification of deviations/improvement areas.Historical individual and crew trend analysis.Where available, correlate performance with pilot inputs/data including fuel state/transfer rate, control inputs, throttle response, aircraft corrections, response timing, crew coordination indicators, and situational-awareness factors.To support organizational change and adjudicate corner cases, provide an instructor/"master debriefer" capability for trained personnel to review, validate, and provide feedback on automated assessments during initial deployment.Stretch Goal: Recognize, or provide a growth path to recognize, bent/broken boom, probe, or hose conditions and associated FOD.Deliverables: Automated Performance Assessment Engine; Pilot Feedback Products; Instructor Review Interface; Performance Analytics Dashboard.Task 4: Automated Data Protection and Video ProcessingDevelop secure video-processing capabilities that minimize operational burden and protect sensitive information.The system shall:Detect when a receiver aircraft is present and activate analytics only during relevant events.Identify and prevent processing/retention of unauthorized aircraft or sensitive platforms.Remove/obscure sensitive geographic information, terrain, markings, and other operationally sensitive information.Support automated masking/filtering before distribution.Demonstrate an architecture supporting future cybersecurity certification.Deliverables: Automated Video Processing Capability; Data Protection/Filtering Methodology; Cybersecurity Assessment Documentation.Task 5: Pilot Debrief Visualization and Training IntegrationDevelop intuitive visualization methods translating analytics into actionable training feedback. Evaluate/demonstrate:Immersive VR/AR replay environments.Squadron-level 2D visualization.Graphical overlays depicting aircraft position, closure rates, desired flight paths, deviations, and refueling-envelope margins.Apply instructional-design principles to support pilot learning, instructor evaluation, and squadron training discussions.Deliverables: Visualization Trade Study; Pilot Debrief Interface Prototype; Training Integration Plan.Task 6: Operational Demonstration, Testing, and TransitionDemonstrate the prototype with operational military aviation users and validate:Accuracy of automated assessments.Correlation with experienced instructor evaluations.Negligible manual video-review workload.Usability within existing training/mission-debrief workflows.Ability to generate actionable post-AAR pilot feedback.Desired Technical Performance Objectives:Generate automated debrief products within 15 minutes of video ingestion.Accurately track aircraft position, movement, and refueling geometry.Identify significant deviations from nominal procedures.Achieve ≥95% agreement between experienced instructors and automated assessments during validation.Demonstrate integration with at least one operational AAR training organization.Deliverables: Operational Prototype Demonstration; Test and Evaluation Report; User Feedback Assessment; Phase III Transition Plan.

Phase III

Phase III will support transition of the AI-enabled aerial refueling performance assessment capability into operational Air Force, Navy, USMC, and select foreign aviation communities through program office integration, operational funding pathways, and sustainment planning.Potential transition applications include:· KC-46 aerial refueling training and readiness improvement.· KC-135 and legacy tanker modernization efforts.· C-17 and other receiver aircraft training applications.· Fighter receiver aerial refueling proficiency improvement.· Navy and Marine Corps aerial refueling operations.· Autonomous aerial refueling development and validation.· Advanced aviation training and performance analytics.Commercialization opportunities may include adaptation of aviation analytics, pilot training analytics, flight performance monitoring, and simulation technologies for commercial aviation, allied aviation partners, and other safety-critical transportation environments.

Keywords: Artificial Intelligence, Machine Learning, Computer Vision, Aerial Refueling, Pilot Training, Flight Analytics, Aircraft Performance Assessment, Human Machine Interface, Virtual Reality, Augmented Reality, Aviation Safety, Trusted AI, KC-46, KC-135

Technology areas: Air Platform, Electronics

Focus areas: Human-Machine Interfaces, Trusted AI and Autonomy

CMMC level: Level 2 (Self)