You can also find my articles on my Google Scholar Profile.
Research Topics:Show all by date / Show all by topic / Show selected

Self-supervised Learning

CLOUD: Contrastive Learning of Unsupervised Dynamics
Jianren Wang*, Yujie Lu*, Hang Zhao (* indicates equal contribution)
2020 Conference on Robot Learning
[Project Page] [Code] [Abstract] [Bibtex]

Developing agents that can perform complex control tasks from high dimensional observations such as pixels is challenging due to difficulties in learning dynamics efficiently. In this work, we propose to learn forward and inverse dynamics in a fully unsupervised manner via contrastive estimation. Specifically, we train a forward dynamics model and an inverse dynamics model in the feature space of states and actions with data collected from random exploration. Unlike most existing deterministic models, our energy-based model takes into account the stochastic nature of agent-environment interactions. We demonstrate the efficacy of our approach across a variety of tasks including goal-directed planning and imitation from observations.

    Author = {Wang, Jianren and Lu, Yujie and Zhao, Hang},
    Title = {CLOUD: Contrastive Learning of Unsupervised Dynamics},
    Booktitle = {CORL},
    Year = {2020}
Inverting the Forecasting Pipeline with SPF2: Sequential Pointcloud Forecasting for Sequential Pose Forecasting
Xinshuo Weng, Jianren Wang, Sergey Levine, Kris Kitani, Nick Rhinehart
2020 Conference on Robot Learning
[Project Page] [Code] [Abstract] [Bibtex]

Many autonomous systems forecast aspects of the future in order to aid decision-making. For example, self-driving vehicles and robotic manipulation systems often forecast future object poses by first detecting and tracking objects. However, this detect-then-forecast pipeline is expensive to scale, as pose forecasting algorithms typically require labeled sequences of object poses, which are costly to obtain in 3D space. Can we scale performance without requiring additional labels? We hypothesize yes, and propose inverting the detect-then-forecast pipeline. Instead of detecting, tracking and then forecasting the objects, we propose to first forecast 3D sensor data (e.g., point clouds with $100$k points) and then detect/track objects on the predicted point cloud sequences to obtain future poses, i.e., a forecast-then-detect pipeline. This inversion makes it less expensive to scale pose forecasting, as the sensor data forecasting task requires no labels. Part of this work's focus is on the challenging first step -- Sequential Pointcloud Forecasting (SPF), for which we also propose an effective approach, SPFNet. To compare our forecast-then-detect pipeline relative to the detect-then-forecast pipeline, we propose an evaluation procedure and two metrics. Through experiments on a robotic manipulation dataset and two driving datasets, we show that SPFNet is effective for the SPF task, our forecast-then-detect pipeline outperforms the detect-then-forecast approaches to which we compared, and that pose forecasting performance improves with the addition of unlabeled data.

    author = {Weng, Xinshuo and Wang, Jianren and Levine, Sergey 
    and Kitani, Kris and Rhinehart, Nick}, 
    journal = {CoRL}, 
    title = {Inverting the Pose Forecasting Pipeline with SPF2: 
    Sequential Pointcloud Forecasting for Sequential Pose Forecasting}, 
    year = {2020} 
Uncertainty-aware Self-supervised 3D Data Association
Jianren Wang, Siddharth Ancha, Yi-Ting Chen, David Held
2020 IEEE/RSJ International Conference on Intelligent Robots and Systems
[Project Page] [Code] [Abstract] [Bibtex]

3D object trackers usually require training on large amounts of annotated data that is expensive and time-consuming to collect. Instead, we propose leveraging vast unlabeled datasets by self-supervised metric learning of 3D object trackers, with a focus on data association. Large scale annotations for unlabeled data are cheaply obtained by automatic object detection and association across frames. We show how these self-supervised annotations can be used in a principled manner to learn point-cloud embeddings that are effective for 3D tracking. We estimate and incorporate uncertainty in self-supervised tracking to learn more robust embeddings, without needing any labeled data. We design embeddings to differentiate objects across frames, and learn them using uncertainty-aware self-supervised training. Finally, we demonstrate their ability to perform accurate data association across frames, towards effective and accurate 3D tracking.

    Author = {Wang, Jianren and Ancha, Siddharth and Chen, Yi-Ting and Held, David},
    Title = {Uncertainty-aware Self-supervised 3D Data Association},
    Booktitle = {IROS},
    Year = {2020}
AlignNet: A Unifying Approach to Audio-Visual Alignment
Jianren Wang*, Zhaoyuan Fang*, Hang Zhao (* indicates equal contribution)
2020 Winter Conference on Applications of Computer Vision
[Project Page] [Code] [Data] [Abstract] [Bibtex]

We present AlignNet, a model designed to synchronize a video with a reference audio under non-uniform and irregular misalignment. AlignNet learns the end-to-end dense correspondence between each frame of a video and an audio. Our method is designed according to simple and well-established principles: attention, pyramidal processing, warping, and affinity function. Together with the model, we release a dancing dataset Dance50 for training and evaluation. Qualitative, quantitative and subjective evaluation results on dance-music alignment and speech-lip alignment demonstrate that our method far outperforms the state-of-the-art methods.

    Author = {Wang, Jianren and Fang, Zhaoyuan
            and Zhao, Hang},
    Title = {AlignNet: A Unifying Approach to Audio-Visual Alignment},
    Booktitle = {WACV},
    Year = {2020}