[ECCV2022] Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretraining

Overview

Learning to Drive by Watching YouTube videos: Action-Conditioned Contrastive Policy Pretraining (ECCV22)

Webpage | Code | Paper

Installation

Our codebase is based on MoCo. A simple PyTorch environment will be enough (see MoCo's installation instruction).

Dataset

We collect driving videos from YouTube. Here we provide the 🔗 video list we used.

Training

We provide main_label_moco.py for training. To perform ACO training of a ResNet-34 model in an 8-gpu machine, run:

python main_label_moco.py -a resnet34 --mlp -j 16 --lr 0.003 \
			 --batch-size 256 --moco-k 40960 --dist-url 'tcp://localhost:10001' \
			 --multiprocessing-distributed --world-size 1 --rank 0 [your-dataset-directory] 

Some important arguments:

  • --aug_cf: whether to use Cropping and Flipping augmentations in pre-training. In ACO, we do not use these two augmentations by default.
  • --thres: action similarity threshold.

Bibtex

@article{zhang2022learning,
  title={Learning to Drive by Watching YouTube videos: Action-Conditioned Contrastive Policy Pretraining},
  author={Zhang, Qihang and Peng, Zhenghao and Zhou, Bolei},
  journal={arXiv preprint arXiv:2204.02393},
  year={2022}
}
You might also like...

Youtube-drive is totally inspired by YouTubeDrive, a Wolfram Language (aka Mathematica) package.

youtube-drive youtube-drive is totally inspired by YouTubeDrive, a Wolfram Language (aka Mathematica) package. I read the source code of YouTubeDrive

Jan 1, 2023

This python script get one playlist form youtube and save it to your google drive folder.

Requisition This library used in this script is pytube and pydrive. You can download and read documentantion from: Pydrive: https://pythonhosted.org/P

Oct 6, 2022

Official implementation of "An Action Is Worth Multiple Words: Handling Ambiguity in Action Recognition", BMVC 2022

Official implementation of

An Action Is Worth Multiple Words: Handling Ambiguity in Action Recognition Kiyoon Kim, Davide Moltisanti, Oisin Mac Aodha, Laura Sevilla-Lara In BMVC

Dec 23, 2022

CLOOB Conditioned Latent Diffusion training and inference code (the latter is not available at this time)

Pretrained Models Coming soon! Training Training setup First recursively git clone this repo to get it and its submodules: git clone --recursive https

Jan 2, 2023

Pytorch code for Adapting Self-Supervised Vision Transformers by Probing Attention-Conditioned Masking Consistency

Pytorch code for Adapting Self-Supervised Vision Transformers by Probing Attention-Conditioned Masking Consistency

PyTorch Implementation of Adapting Self-Supervised Vision Transformers by Probing Attention-Conditioned Masking Consistency. Viraj Prabhu*, Sriram Yen

Dec 9, 2022

Code of "AvatarCap: Animatable Avatar Conditioned Monocular Human Volumetric Capture" [ECCV 2022]

Code of

AvatarCap: Animatable Avatar Conditioned Monocular Human Volumetric Capture (ECCV 2022) Zhe Li, Zerong Zheng, Hongwen Zhang, Chaonan Ji, Yebin Liu Pap

Dec 27, 2022

Official implementation of "Equivariant Shape-Conditioned Generation of 3D Molecules for Ligand-Based Drug Design"

SQUID 🦑 This repository contains the necessary scripts to train and evaluate the 3D generative model SQUID from the paper: Equivariant Shape-Conditio

Jan 5, 2023

Official code for "A General Learning Framework for Open Ad Hoc Teamwork Using Graph-based Policy Learning"

GPL This is the code for our paper "A General Learning Framework for Open Ad Hoc Teamwork Using Graph-based Policy Learning". Inside you will find all

Oct 11, 2022

Getting all properties of all videos in a directory and sub folders like total videos length or video size ...

Getting all properties of all videos in a directory and sub folders like total videos length or video size ...

Videos-Properties Here is a python code that can give you all videos and all videos properties in a folder and sub-folders. (click to see code) I will

Jul 30, 2022
Comments
  • About dataset

    About dataset

    Thanks for sharing your great work! Just wonder is it possible to share your zip file of those sampled frames from the youtube videos? It is a little bit time and space consuming to download all the videos. Thank you in advance.

    opened by WPH-commit 16
  • The training of the Inverse dynamics model

    The training of the Inverse dynamics model

    Hi, how many training samples are used for the training of the Inverse dynamics model (only samples data or all the sweeps data), and do you scale and center crop the nuscenes image to 320*180 for training? Also, when calculating the optical flow for training, is the time interval between current and next image 1s? Thanks in advance.

    opened by WPH-commit 5
  • Which dataset is the provided pre-trained weight trained on?

    Which dataset is the provided pre-trained weight trained on?

    Hello, this is Yuan, who are interested in this great work! I wonder which dataset is the provided pre-trained weight trained on and what the corresponding sampling rate? Thanks!

    opened by yshen47 2
  • Question about the model for PPO

    Question about the model for PPO

    HI, I have several questions about the model used for PPO training.

    • Does the model takes navigation command and speed as input the same way as CILRS?
    • Do the actor encoder and the critic encoder share the same model?
    opened by WPH-commit 2
Owner
MetaDriverse for Autonomy Research
Research projects relevant to MetaDrive
MetaDriverse for Autonomy Research
ECCV2022,Bootstrapped Masked Autoencoders for Vision BERT Pretraining

BootMAE, ECCV2022 This repo is the official implementation of "Bootstrapped Masked Autoencoders for Vision BERT Pretraining". Introduction We propose

DLight 83 Dec 20, 2022
(ECCV2022) This is the official PyTorch implementation of ECCV2022 paper: Towards Efficient and Scale-Robust Ultra-High-Definition Image Demoireing

Towards Efficient and Scale-Robust Ultra-High-Definition Image Demoiréing Project Page | Dataset | Paper Towards Efficient and Scale-Robust Ultra-High

CVMI Lab 91 Dec 13, 2022
The Powerful Telegram Bot based on Gclone to clone Google Drive's Shared Drive data easily.🔥

CloneBot V2 ?? CloneBot V2 is inspired from MsGsuite's CloneBot, which got out-dated and having too many errors in it. We both created it to keep the

Dr.Caduceus 241 Jan 9, 2023
[ECCV2022] New benchmark for evaluating pre-trained model; New supervised contrastive learning framework.

[ECCV2022] OmniBenchmark Yuanhan Zhang1  Zhenfei Yin2  Jing Shao2  Ziwe Liu1 1S-Lab, Nanyang Technological University  2SenseTime Research  [Paper] •

Yuanhan Zhang 98 Dec 27, 2022
[ECCV2022] Official Pytorch Implementation of Object Discovery via Contrastive Learning for Weakly Supervised Object Detection

Object Discovery via Contrastive Learning for Weakly Supervised Object Detection Jinhwan Seo, Wonho Bae, Danica J. Sutherland, Junhyug Noh, and Daijin

Jinhwan Seo 19 Dec 17, 2022
[ECCV2022] Motion Sensitive Contrastive Learning for Self-supervised Video Representation

MSCL Official code for Motion Sensitive Contrastive Learning for Self-supervised Video Representation (ECCV2022). Introduction Contrastive learning ha

MEGVII Research 12 Dec 18, 2022
Automatically downloads Youtube videos and edits them into trending tiktok "styled" videos.

AutoVideoCreator Automatically downloads Youtube videos and edits them into trending tiktok "styled" videos. Created by EvilWumpus Video Example video

null 19 Dec 20, 2022
Through this project you can automate making videos from reddit and uploading them on YouTube. Currently, it only supports images but we hope to bring the ability to combine videos as well in the future.

YouTube-Shorts-Creator Through this project you can automate making videos from reddit and uploading them on YouTube. Currently, it only supports imag

Mohammed Fahad Tabrez Sheikh 3 Nov 10, 2022
Codes for ECCV2022 paper - contrastive deep supervision

Contrastive Deep Supervision This is the code for contrastive deep supervision and distilled contrastive deep supervision. Install. Install the based

null 43 Dec 12, 2022
PlexSonarrAutoDownloader is a script that connects to Plex and Sonarr to automatically download the next season of a show you are watching based on the episode you are on.

This collection of python scripts and docker containers will download the next season of an in-progress show automatically using Sonarr when a user is nearing the end of the currently downloaded season.

Maxwell Cash 3 Aug 30, 2022