Neel P. Bhatt

I am currently a Postdoctoral Fellow at the Center for Autonomy working with Professors Atlas Wang and Ufuk Topcu. I am part of the Autonomous Systems Group and the VITA Research Group at the University of Texas at Austin. I am developing neuro-symbolic architectures with research centered at the intersection of generative AI, assured active perception, prediction, and trustworthy sequential decision making for autonomous systems.

I received my PhD in Mechatronics Engineering from University of Waterloo in 2023. During my PhD, I led efforts on the WATonoBus project at MVS Lab working on software and algorithmic development of perception and prediction modules required for Canada’s first autonomous shuttle bus approved via the ministry’s autonomous vehicle pilot. During this time, I also interned at GM R&D where I worked on deep learning-based state estimation.

I have also concurrently been a visiting research scholar since 2021 at the NODE Lab at University of Alberta working on NODE lab's autonomous vehicle.

Prior to this, I received my Bachelor's degree in Mechanical Engineering (I was directly admitted to a PhD from Bachelor's) in 2018 with focus on Mechatronics and Robotics from University of Toronto. During this time, I conducted research with Professor Yu Sun on micro and nano robotics and interned at Clearpath Robotics and was awarded NSERC Research Awards for both.

[ Email  /  Google Scholar  /  LinkedIn  /  CV ]

Our team started the WATonoBus autonomous shuttle project in 2018 and has since developed 2 such fully equipped shuttles and is near completion of the third shuttle. The WATonoBus is a platform that contains fully in-house equipped hardware and software stack and has been approved as part of the Ministry of Transportation Ontario’s Autonomous Vehicle Pilot Program currently providing daily free and fully autonomous service to passengers at the University of Waterloo. The University of Waterloo's Ring Road is a 2.7 km curvy road with many intersections and pedestrian crossings that represent an urban driving environment with several pedestrians, cyclists, and vehicles. The WATonoBus project is different from prior project in that it is aimed to operate in all weather conditions including adverse rain, fog, and snow.

Since 2021, I have been leading efforts at the NODE lab to develop hardware and software stack on the NODE lab's autonomous Ford Escape vehicle. This platform has been central in working on several research projects covering RL, visual odometry, SLAM, object detection. I also supervise several PhD and Masters students at the lab.

We present a fully automated approach to fine-tune pre-trained language models for applications in autonomous systems, bridging the gap between generic knowledge and domain-specific requirements while reducing cost. The method synthesizes automaton-based controllers from pre-trained models guided by natural language task descriptions that are formally verified. Controllers with high compliance of the desired specifications receive higher ranks, guiding the iterative fine-tuning process. We provide quantitative evidence demonstrating an improvement in percentage of specifications satisfied from 60% to 90%.

Our method, MM3DGS, addresses the limitations of prior neural radiance field-based representations by enabling faster rendering, scale awareness, and improved trajectory tracking. Our framework enables keyframe-based mapping and tracking utilizing loss functions that incorporate relative pose transformations from pre-integrated inertial measurements, depth estimates, and measures of photometric rendering quality. Experimental evaluation on several scenes shows a 3x improvement in tracking and 5% improvement in photometric rendering quality compared to the current 3DGS SLAM state-of-the-art, while allowing real-time rendering.

Our method, MM3DGS, addresses the limitations of prior neural radiance field-based representations by enabling faster rendering, scale awareness, and improved trajectory tracking. Our framework enables keyframe-based mapping and tracking utilizing loss functions that incorporate relative pose transformations from pre-integrated inertial measurements, depth estimates, and measures of photometric rendering quality. Experimental evaluation on several scenes shows a 3x improvement in tracking and 5% improvement in photometric rendering quality compared to the current 3DGS SLAM state-of-the-art, while allowing real-time rendering.

We provide a comprehensive survey and quantitative comparisons with state-of-the-art 3D object detection methodologies aiming to tackle varying weather conditions, multi-modality, multi-camera perspective, and their respective metrics associated to different difficulty categories. We identify several research gaps and potential future directions in visual-based 3D object detection approaches for autonomous driving.

We propose MPC-PF, a model that embeds surrounding object and road map information in the form of a potential field to model agent-agent and agent-space interactions. We show the efficacy of our multi-object trajectory prediction method both qualitatively and quantitatively achieving state-of-the-art results on the Waymo Open Motion Dataset and other common urban driving scenarios.

Leveraging a global navigation satellite system, inertial navigation system, and 3D LiDAR point clouds, a novel light point cloud map generation method, which only keeps the necessary point clouds (i.e., buildings and roads regardless of vegetation varying with seasonal change), is proposed. Thorough experiments in winter and summer confirm the advantages of integrating the proposed light point cloud map generation with the dead reckoning model in terms of accuracy and reduced computational complexity.

We propose a semantic-aware stereo visual odometry framework wherein feature extraction is performed over a static region-of-interest generated through object detection and instance segmentation on static street objects. Extensive real driving sequences in various dynamic urban environments with varying sequence lengths confirms excellent performance and computational efficiency attributed to using semantic-aware feature tracking.

Predicting object motion behaviour is a challenging but crucial task for safe decision-making and planning for an autonomous vehicle. We tackle this problem by introducing MPC-PF: a novel potential field-based trajectory predictor that incorporates social interaction and is able to tradeoff between inherent model biases across the prediction horizon. Through evaluation on a variety of common urban driving scenarios, we show that our model produces accurate short and long-term predictions.

We develop a robust infrastructure-aided localization framework using only a single low-cost camera with a fisheye lens. To reduce the computational load, we use an ROI alongside estimated depth to re-project the robot pointcloud cluster with geometrical outlier detection. We use this position and depth information in an uncertainty-aware observer with adaptive covariance allocation and bounded estimation error to deal with position measurement noises at the limits of the field of view, and intermittent occlusion in dynamic environments. Moreover, we use a learning-based prediction model for input estimation based on a moving buffer of the robot position. Several experiments with occlusion and intermittent visual disruption/detection confirm effectiveness of the developed framework in re-initializing the estimation process after failure in the visual detection, and handling temporary data loss due to sensor faults or changes in lighting conditions.

A slip-aware localization framework is proposed for mobile robots experiencing wheel slip in dynamic environments. The framework fuses infrastructure-aided visual tracking data and proprioceptive sensory data from a skid-steer mobile robot to enhance accuracy and reduce variance of the estimated states. Covariance intersection is used to fuse the pose prediction and the visual thread, such that the updated estimate remains consistent. As confirmed by experiments on a skid-steer mobile robot, the designed localization framework addresses state estimation challenges for indoor/outdoor autonomous mobile robots which experience high-slip, uneven torque distribution at each wheel (by the motion planner), or occlusion when observed by an infrastructure-mounted camera.

We present a generic feature-based navigation framework for autonomous vehicles using a soft constrained Particle Filter. After obtaining features of mapped landmarks in instance-based segmented images acquired from a monocular camera, vehicle-to-landmark distances are predicted using Gaussian Process Regression (GPR) models in a mixture of experts approach. Experimental results confirm that the use of image segmentation features improves the vehicle-to-landmark distance prediction notably, and that the proposed soft constrained approach reliably localizes the vehicle even with reduced number of landmarks and noisy observations.

We propose a constrained moving horizon state estimation approach to estimate an object's states in 3D with respect to a global stationary frame including position, velocity, and acceleration that are robust to intermittently noisy or absent sensor measurements utilizing a computationally light-weight fusion of 2D dections and projected LIDAR depth measurements. The performance of the proposed approach is experimentally verified on our dataset featuring urban pedestrian crossings.

In December of 2023, the Texas Advanced Computing Center (TACC) hosted two engaging cybersecurity events, "GenCyber Back@TACC" and "Level UP GenCyber Back@TACC," both aimed at educating high school students in various aspects of coding and cybersecurity. I contributed by motivating students envision a future in both cybersecurity and the broader selection of STEM fields. I also helped by facilitating the learning process, answering questions, and engaging high school students in discussions about their personal experiences from their academic journeys in STEM, thereby encouraging students to consider future studies and careers.

This event, hosted by Center for Autonomy, invited students from the Del Valle Independent School District (Travis County, TX), out of which 84.4% are economically disadvantaged. I presented to the students a high-level overview of what academic research looks like through my research on multi-modal 3D Guassian splatting for simultaneous localization and mapping and its implementation on the Clearpath jackal robots. I used TACC’s high resolution screens to deliver my presentation in an interactive manner. I shared my undergrad, internship, PhD, and postdoc experiences with the students and gave them a personal view of how each one led to the next career step so that they could understand potential career trajectories. I motivated the students to ask questions and answered their questions ranging from how I got into robotics through First Robotics Challenge (FRC) – their school participates in the same competition, what I do not like about industry, and failures through my career journey.