We introduce A4D, which maps visual observations into a shared functional latent space structured around affordances rather than appearance. By measuring proximity to affordance axes and triggering uncertainty-guided affordance discovery when existing axes are insufficient, A4D achieves 94% inference accuracy on existing affordances (over 15% points above SOTA), improves new-affordance accuracy from ~70% to over 90% with under 10% of the original training data, and enables 100x faster inference.

We introduce Co-GLANCE, a real-time onboard perception and decision-making system for uncertainty resolution in heterogeneous robot teams. By distilling vision-language model reasoning into a lightweight model with conformal prediction and selective abstention, Co-GLANCE improves occlusion segmentation and robot allocation accuracy by 25% and 36% over cloud VLM baselines while reducing per-frame inference latency by 350x.

We introduce UNCAP, a planning approach for connected autonomous vehicles that uses natural language communication to convey perception uncertainties. Our two-stage protocol selectively exchanges messages with relevant vehicles, reducing communication bandwidth by 63% while increasing driving safety scores by 31%.

We present VLN-Zero, a zero-shot vision-language navigation framework that enables robots to follow natural language instructions in unseen environments without task-specific training. Our approach leverages rapid exploration and cache-enabled neurosymbolic planning to achieve effective zero-shot transfer in robot navigation tasks.

We introduce RepV, a neurosymbolic verifier that learns a latent space where safe and unsafe plans are linearly separable. Starting with a seed set labeled by a model checker, RepV trains a projector embedding plans and rationales into a low-dimensional space. Our method improves compliance prediction accuracy by up to 15% compared to baselines while adding fewer than 0.2M parameters.

We present a position paper on neuro-symbolic AI as an alternative to scaling laws, proposing hybrid neural symbolic networks that integrate data-driven optimization with symbolic reasoning. Our framework demonstrates how symbolic knowledge can guide neural network learning through iterative lifting, symbolic updates, and projection processes.

We develop a method that converts generated robot programs into automaton-based representations and verifies them against safety specifications. We establish a theorem that any combination of verified programs also satisfies safety specifications, eliminating the need to verify complex composed programs. Our automated fine-tuning procedure increases the probability of generating specification-compliant programs by 30%, with training time reduced by half compared to fine-tuning on full programs.

We propose methods tailored to the unique properties of perception and decision-making: we use conformal prediction to calibrate perception uncertainty and introduce FMDP to quantify decision uncertainty, leveraging formal verification techniques for theoretical guarantees. Building on this quantification, we implement two targeted intervention mechanisms: an active sensing process and an automated refinement procedure improving its capability to meet task specifications. Empirical validation in real-world and simulated robotic tasks demonstrates that our uncertainty disentanglement framework reduces variability by up to 40% and enhances task success rates by 5% compared to baselines

We evaluate the planning capabilities of OpenAI's o1 models across a variety of benchmark tasks, focusing on three key aspects: feasibility, optimality, and generalizability. Through empirical evaluations on constraint-heavy tasks (e.g., Barman, Tyreworld) and spatially complex environments (e.g., Termes, Floortile), we highlight o1-preview’s strengths in self-evaluation and constraint-following, while also identifying bottlenecks in decision-making and memory management, particularly in tasks requiring robust spatial reasoning. Our results reveal that o1-preview outperforms GPT-4 in adhering to task constraints and managing state transitions in structured environments. However, the model often generates suboptimal solutions with redundant actions and struggles to generalize effectively in spatially complex tasks.

We present a fully automated approach to fine-tune pre-trained language models for applications in autonomous systems, bridging the gap between generic knowledge and domain-specific requirements while reducing cost. The method synthesizes automaton-based controllers from pre-trained models guided by natural language task descriptions that are formally verified. Controllers with high compliance of the desired specifications receive higher ranks, guiding the iterative fine-tuning process. We provide quantitative evidence demonstrating an improvement in percentage of specifications satisfied from 60% to 90%.

Our method, MM3DGS, addresses the limitations of prior neural radiance field-based representations by enabling faster rendering, scale awareness, and improved trajectory tracking. Our framework enables keyframe-based mapping and tracking utilizing loss functions that incorporate relative pose transformations from pre-integrated inertial measurements, depth estimates, and measures of photometric rendering quality. Experimental evaluation on several scenes shows a 3x improvement in tracking and 5% improvement in photometric rendering quality compared to the current 3DGS SLAM state-of-the-art, while allowing real-time rendering.

Our method, MM3DGS, addresses the limitations of prior neural radiance field-based representations by enabling faster rendering, scale awareness, and improved trajectory tracking. Our framework enables keyframe-based mapping and tracking utilizing loss functions that incorporate relative pose transformations from pre-integrated inertial measurements, depth estimates, and measures of photometric rendering quality. Experimental evaluation on several scenes shows a 3x improvement in tracking and 5% improvement in photometric rendering quality compared to the current 3DGS SLAM state-of-the-art, while allowing real-time rendering.

We provide a comprehensive survey and quantitative comparisons with state-of-the-art 3D object detection methodologies aiming to tackle varying weather conditions, multi-modality, multi-camera perspective, and their respective metrics associated to different difficulty categories. We identify several research gaps and potential future directions in visual-based 3D object detection approaches for autonomous driving.

We propose MPC-PF, a model that embeds surrounding object and road map information in the form of a potential field to model agent-agent and agent-space interactions. We show the efficacy of our multi-object trajectory prediction method both qualitatively and quantitatively achieving state-of-the-art results on the Waymo Open Motion Dataset and other common urban driving scenarios.

Leveraging a global navigation satellite system, inertial navigation system, and 3D LiDAR point clouds, a novel light point cloud map generation method, which only keeps the necessary point clouds (i.e., buildings and roads regardless of vegetation varying with seasonal change), is proposed. Thorough experiments in winter and summer confirm the advantages of integrating the proposed light point cloud map generation with the dead reckoning model in terms of accuracy and reduced computational complexity.

We propose a semantic-aware stereo visual odometry framework wherein feature extraction is performed over a static region-of-interest generated through object detection and instance segmentation on static street objects. Extensive real driving sequences in various dynamic urban environments with varying sequence lengths confirms excellent performance and computational efficiency attributed to using semantic-aware feature tracking.

Predicting object motion behaviour is a challenging but crucial task for safe decision-making and planning for an autonomous vehicle. We tackle this problem by introducing MPC-PF: a novel potential field-based trajectory predictor that incorporates social interaction and is able to tradeoff between inherent model biases across the prediction horizon. Through evaluation on a variety of common urban driving scenarios, we show that our model produces accurate short and long-term predictions.

We develop a robust infrastructure-aided localization framework using only a single low-cost camera with a fisheye lens. To reduce the computational load, we use an ROI alongside estimated depth to re-project the robot pointcloud cluster with geometrical outlier detection. We use this position and depth information in an uncertainty-aware observer with adaptive covariance allocation and bounded estimation error to deal with position measurement noises at the limits of the field of view, and intermittent occlusion in dynamic environments. Moreover, we use a learning-based prediction model for input estimation based on a moving buffer of the robot position. Several experiments with occlusion and intermittent visual disruption/detection confirm effectiveness of the developed framework in re-initializing the estimation process after failure in the visual detection, and handling temporary data loss due to sensor faults or changes in lighting conditions.

A slip-aware localization framework is proposed for mobile robots experiencing wheel slip in dynamic environments. The framework fuses infrastructure-aided visual tracking data and proprioceptive sensory data from a skid-steer mobile robot to enhance accuracy and reduce variance of the estimated states. Covariance intersection is used to fuse the pose prediction and the visual thread, such that the updated estimate remains consistent. As confirmed by experiments on a skid-steer mobile robot, the designed localization framework addresses state estimation challenges for indoor/outdoor autonomous mobile robots which experience high-slip, uneven torque distribution at each wheel (by the motion planner), or occlusion when observed by an infrastructure-mounted camera.

We present a generic feature-based navigation framework for autonomous vehicles using a soft constrained Particle Filter. After obtaining features of mapped landmarks in instance-based segmented images acquired from a monocular camera, vehicle-to-landmark distances are predicted using Gaussian Process Regression (GPR) models in a mixture of experts approach. Experimental results confirm that the use of image segmentation features improves the vehicle-to-landmark distance prediction notably, and that the proposed soft constrained approach reliably localizes the vehicle even with reduced number of landmarks and noisy observations.

We propose a constrained moving horizon state estimation approach to estimate an object's states in 3D with respect to a global stationary frame including position, velocity, and acceleration that are robust to intermittently noisy or absent sensor measurements utilizing a computationally light-weight fusion of 2D dections and projected LIDAR depth measurements. The performance of the proposed approach is experimentally verified on our dataset featuring urban pedestrian crossings.