# Publications

You can also find my articles on my Google Scholar Profile.
Research Topics:Show all by date / Show all by topic / Show selected

## Visual Learning

 Inverting the Forecasting Pipeline with SPF2: Sequential Pointcloud Forecasting for Sequential Pose Forecasting Xinshuo Weng, Jianren Wang, Sergey Levine, Kris Kitani, Nick Rhinehart 2020 Conference on Robot Learning [Project Page] [Code] [Abstract] [Bibtex] Many autonomous systems forecast aspects of the future in order to aid decision-making. For example, self-driving vehicles and robotic manipulation systems often forecast future object poses by first detecting and tracking objects. However, this detect-then-forecast pipeline is expensive to scale, as pose forecasting algorithms typically require labeled sequences of object poses, which are costly to obtain in 3D space. Can we scale performance without requiring additional labels? We hypothesize yes, and propose inverting the detect-then-forecast pipeline. Instead of detecting, tracking and then forecasting the objects, we propose to first forecast 3D sensor data (e.g., point clouds with $100$k points) and then detect/track objects on the predicted point cloud sequences to obtain future poses, i.e., a forecast-then-detect pipeline. This inversion makes it less expensive to scale pose forecasting, as the sensor data forecasting task requires no labels. Part of this work's focus is on the challenging first step -- Sequential Pointcloud Forecasting (SPF), for which we also propose an effective approach, SPFNet. To compare our forecast-then-detect pipeline relative to the detect-then-forecast pipeline, we propose an evaluation procedure and two metrics. Through experiments on a robotic manipulation dataset and two driving datasets, we show that SPFNet is effective for the SPF task, our forecast-then-detect pipeline outperforms the detect-then-forecast approaches to which we compared, and that pose forecasting performance improves with the addition of unlabeled data.@article{Weng2020_SPF2, author = {Weng, Xinshuo and Wang, Jianren and Levine, Sergey and Kitani, Kris and Rhinehart, Nick}, journal = {CoRL}, title = {Inverting the Pose Forecasting Pipeline with SPF2: Sequential Pointcloud Forecasting for Sequential Pose Forecasting}, year = {2020} } 
 PanoNet3D: Combining Semantic and Geometric Understanding for LiDARPoint Cloud Detection Xia Chen, Jianren Wang, David Held, Martial Hebert 2020 International Virtual Conference on 3D Vision [Project Page] [Code] [Abstract] [Bibtex] Visual data in autonomous driving perception, such as camera image and LiDAR point cloud, can be interpreted as a mixture of two aspects: semantic feature and geometric structure. Semantics come from the appearance and context of objects to the sensor, while geometric structure is the actual 3D shape of point clouds. Most detectors on LiDAR point clouds focus only on analyzing the geometric structure of objects in real 3D space. Unlike previous works, we propose to learn both semantic feature and geometric structure via a unified multi-view framework. Our method exploits the nature of LiDAR scans -- 2D range images, and applies well-studied 2D convolutions to extract semantic features. By fusing semantic and geometric features, our method outperforms state-of-the-art approaches in all categories by a large margin. The methodology of combining semantic and geometric features provides a unique perspective of looking at the problems in real-world 3D point cloud detection.@inproceedings{xia20panonet3d, Author = {Chen, Xia and Wang, Jianren and Held, David and Hebert, Martial}, Title = {PanoNet3D: Combining Semantic and Geometric Understanding for LiDARPoint Cloud Detection}, Booktitle = {3DV}, Year = {2020} } 
 GSIR: Generalizable 3D Shape Interpretation and Reconstruction Jianren Wang, Zhaoyuan Fang 2020 The European Conference on Computer Vision [Project Page] [Code] [Abstract] [Bibtex] 3D shape interpretation and reconstruction are closely related to each other but have long been studied separately and often end up with priors that are highly biased towards the training classes. In this paper, we present an algorithm, Generalizable 3D Shape Interpretation and Reconstruction (GSIR), designed to jointly learn these two tasks to capture generic, class-agnostic shape priors for a better understanding of 3D geometry. We propose to recover 3D shape structures as cuboids from partial reconstruction and use the predicted structures to further guide full 3D reconstruction. The unified framework is trained simultaneously offline to learn a generic notion and can be fine-tuned online for specific objects without any annotations. Extensive experiments on both synthetic and real data demonstrate that introducing 3D shape interpretation improves the performance of single image 3D reconstruction and vice versa, achieving the state-of-the-art performance on both tasks for objects in both seen and unseen categories.@@inproceedings{wang2020gsir, title={GSIR: Generalizable 3D Shape Interpretation and Reconstruction}, author={Wang, Jianren and Fang, Zhaoyuan}, booktitle={European Conference on Computer Vision}, pages={498--514}, year={2020}, organization={Springer} } 
 Uncertainty-aware Self-supervised 3D Data Association Jianren Wang, Siddharth Ancha, Yi-Ting Chen, David Held 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems [Project Page] [Code] [Abstract] [Bibtex] 3D object trackers usually require training on large amounts of annotated data that is expensive and time-consuming to collect. Instead, we propose leveraging vast unlabeled datasets by self-supervised metric learning of 3D object trackers, with a focus on data association. Large scale annotations for unlabeled data are cheaply obtained by automatic object detection and association across frames. We show how these self-supervised annotations can be used in a principled manner to learn point-cloud embeddings that are effective for 3D tracking. We estimate and incorporate uncertainty in self-supervised tracking to learn more robust embeddings, without needing any labeled data. We design embeddings to differentiate objects across frames, and learn them using uncertainty-aware self-supervised training. Finally, we demonstrate their ability to perform accurate data association across frames, towards effective and accurate 3D tracking.@inproceedings{jianren20s3da, Author = {Wang, Jianren and Ancha, Siddharth and Chen, Yi-Ting and Held, David}, Title = {Uncertainty-aware Self-supervised 3D Data Association}, Booktitle = {IROS}, Year = {2020} } 
 Motion Prediction in Visual Object Tracking Jianren Wang*, Yihui He*(* indicates equal contribution) 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems
 Deep Mixture Density Network for Object Detection under Occlusion Yihui He*, Jianren Wang*(* indicates equal contribution) 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems
 3D Multi-Object Tracking: A Baseline and New Evaluation Metrics Xinshuo Weng, Jianren Wang, David Held, Kris M. Kitani 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems [Project Page] [Code] [Abstract] [Bibtex] 3D multi-object tracking (MOT) is an essential component for many applications such as autonomous driving and assistive robotics. Recent work on 3D MOT focuses on developing accurate systems giving less attention to practical considerations such as computational cost and system complexity. In contrast, this work proposes a simple real-time 3D MOT system. Our system first obtains 3D detections from a LiDAR point cloud. Then, a straightforward combination of a 3D Kalman filter and the Hungarian algorithm is used for state estimation and data association. Additionally, 3D MOT datasets such as KITTI evaluate MOT methods in the 2D space and standardized 3D MOT evaluation tools are missing for a fair comparison of 3D MOT methods. Therefore, we propose a new 3D MOT evaluation tool along with three new metrics to comprehensively evaluate 3D MOT methods. We show that, although our system employs a combination of classical MOT modules, we achieve state-of-the-art 3D MOT performance on two 3D MOT benchmarks (KITTI and nuScenes). Surprisingly, although our system does not use any 2D data as inputs, we achieve competitive performance on the KITTI 2D MOT leaderboard. Our proposed system runs at a rate of 207.4 FPS on the KITTI dataset, achieving the fastest speed among all modern MOT systems.@article{Weng2020_AB3DMOT, author = {Weng, Xinshuo and Wang, Jianren and Held, David and Kitani, Kris}, journal = {IROS}, title = {3D Multi-Object Tracking: A Baseline and New Evaluation Metrics}, year = {2020} } 
 PanoNet: Real-time Panoptic Segmentation through Position-Sensitive Feature Embedding Xia Chen, Jianren Wang, Martial Hebert
 Bounding Box Regression with Uncertainty for Accurate Object Detection Yihui He, Chenchen Zhu, Jianren Wang, Marios Savvides, Xiangyu Zhang 2019 IEEE Conference on Computer Vision and Pattern Recognition [Code]
 Physics-Aware 3D Mesh Synthesis Jianren Wang, Yihui He 2019 International Conference on 3D Vision
 Depth-wise Decomposition for Accelerating Separable Convolutions in Efficient Convolutional Neural Networks Yihui He*, Jianing Qian*, Jianren Wang (* indicates equal contribution) 2019 IEEE Conference on Computer Vision and Pattern Recognition Workshop