[ECCV2022] Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretraining

Overview

Learning to Drive by Watching YouTube videos: Action-Conditioned Contrastive Policy Pretraining (ECCV22)

Webpage | Code | Paper

Installation

Our codebase is based on MoCo. A simple PyTorch environment will be enough (see MoCo's installation instruction).

Dataset

We collect driving videos from YouTube. Here we provide the 🔗 video list we used.

Training

We provide main_label_moco.py for training. To perform ACO training of a ResNet-34 model in an 8-gpu machine, run:

python main_label_moco.py -a resnet34 --mlp -j 16 --lr 0.003 \
			 --batch-size 256 --moco-k 40960 --dist-url 'tcp://localhost:10001' \
			 --multiprocessing-distributed --world-size 1 --rank 0 [your-dataset-directory]

Some important arguments:

--aug_cf: whether to use Cropping and Flipping augmentations in pre-training. In ACO, we do not use these two augmentations by default.
--thres: action similarity threshold.

Bibtex

@article{zhang2022learning,
  title={Learning to Drive by Watching YouTube videos: Action-Conditioned Contrastive Policy Pretraining},
  author={Zhang, Qihang and Peng, Zhenghao and Zhou, Bolei},
  journal={arXiv preprint arXiv:2204.02393},
  year={2022}
}

Youtube-drive is totally inspired by YouTubeDrive, a Wolfram Language (aka Mathematica) package.

youtube-drive youtube-drive is totally inspired by YouTubeDrive, a Wolfram Language (aka Mathematica) package. I read the source code of YouTubeDrive

Jan 1, 2023

This python script get one playlist form youtube and save it to your google drive folder.

Requisition This library used in this script is pytube and pydrive. You can download and read documentantion from: Pydrive: https://pythonhosted.org/P

Oct 6, 2022

Official implementation of "An Action Is Worth Multiple Words: Handling Ambiguity in Action Recognition", BMVC 2022

An Action Is Worth Multiple Words: Handling Ambiguity in Action Recognition Kiyoon Kim, Davide Moltisanti, Oisin Mac Aodha, Laura Sevilla-Lara In BMVC

Dec 23, 2022

CLOOB Conditioned Latent Diffusion training and inference code (the latter is not available at this time)

Pretrained Models Coming soon! Training Training setup First recursively git clone this repo to get it and its submodules: git clone --recursive https

Jan 2, 2023

Pytorch code for Adapting Self-Supervised Vision Transformers by Probing Attention-Conditioned Masking Consistency

PyTorch Implementation of Adapting Self-Supervised Vision Transformers by Probing Attention-Conditioned Masking Consistency. Viraj Prabhu*, Sriram Yen

Dec 9, 2022

Code of "AvatarCap: Animatable Avatar Conditioned Monocular Human Volumetric Capture" [ECCV 2022]

AvatarCap: Animatable Avatar Conditioned Monocular Human Volumetric Capture (ECCV 2022) Zhe Li, Zerong Zheng, Hongwen Zhang, Chaonan Ji, Yebin Liu Pap

Dec 27, 2022

Official implementation of "Equivariant Shape-Conditioned Generation of 3D Molecules for Ligand-Based Drug Design"

SQUID 🦑 This repository contains the necessary scripts to train and evaluate the 3D generative model SQUID from the paper: Equivariant Shape-Conditio

Jan 5, 2023

Official code for "A General Learning Framework for Open Ad Hoc Teamwork Using Graph-based Policy Learning"

GPL This is the code for our paper "A General Learning Framework for Open Ad Hoc Teamwork Using Graph-based Policy Learning". Inside you will find all

Oct 11, 2022

Getting all properties of all videos in a directory and sub folders like total videos length or video size ...

Videos-Properties Here is a python code that can give you all videos and all videos properties in a folder and sub-folders. (click to see code) I will

Jul 30, 2022

Comments

About dataset

Thanks for sharing your great work! Just wonder is it possible to share your zip file of those sampled frames from the youtube videos? It is a little bit time and space consuming to download all the videos. Thank you in advance.

opened by WPH-commit 16
The training of the Inverse dynamics model

Hi, how many training samples are used for the training of the Inverse dynamics model (only samples data or all the sweeps data), and do you scale and center crop the nuscenes image to 320*180 for training? Also, when calculating the optical flow for training, is the time interval between current and next image 1s? Thanks in advance.

opened by WPH-commit 5
Which dataset is the provided pre-trained weight trained on?

Hello, this is Yuan, who are interested in this great work! I wonder which dataset is the provided pre-trained weight trained on and what the corresponding sampling rate? Thanks!

opened by yshen47 2
Question about the model for PPO
HI, I have several questions about the model used for PPO training.

Does the model takes navigation command and speed as input the same way as CILRS?

Do the actor encoder and the critic encoder share the same model?
opened by WPH-commit 2

[ECCV2022] Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretraining

Related tags

Overview

Learning to Drive by Watching YouTube videos: Action-Conditioned Contrastive Policy Pretraining (ECCV22)

Installation

Dataset

Training

Bibtex

You might also like...

Youtube-drive is totally inspired by YouTubeDrive, a Wolfram Language (aka Mathematica) package.

This python script get one playlist form youtube and save it to your google drive folder.

Official implementation of "An Action Is Worth Multiple Words: Handling Ambiguity in Action Recognition", BMVC 2022

CLOOB Conditioned Latent Diffusion training and inference code (the latter is not available at this time)

Pytorch code for Adapting Self-Supervised Vision Transformers by Probing Attention-Conditioned Masking Consistency

Code of "AvatarCap: Animatable Avatar Conditioned Monocular Human Volumetric Capture" [ECCV 2022]

Official implementation of "Equivariant Shape-Conditioned Generation of 3D Molecules for Ligand-Based Drug Design"

Official code for "A General Learning Framework for Open Ad Hoc Teamwork Using Graph-based Policy Learning"

Getting all properties of all videos in a directory and sub folders like total videos length or video size ...

Comments

About dataset

The training of the Inverse dynamics model

Which dataset is the provided pre-trained weight trained on?

Question about the model for PPO

Owner

MetaDriverse for Autonomy Research

ECCV2022,Bootstrapped Masked Autoencoders for Vision BERT Pretraining

(ECCV2022) This is the official PyTorch implementation of ECCV2022 paper: Towards Efficient and Scale-Robust Ultra-High-Definition Image Demoireing

The Powerful Telegram Bot based on Gclone to clone Google Drive's Shared Drive data easily.🔥

[ECCV2022] New benchmark for evaluating pre-trained model; New supervised contrastive learning framework.

[ECCV2022] Official Pytorch Implementation of Object Discovery via Contrastive Learning for Weakly Supervised Object Detection

[ECCV2022] Motion Sensitive Contrastive Learning for Self-supervised Video Representation

Automatically downloads Youtube videos and edits them into trending tiktok "styled" videos.

Through this project you can automate making videos from reddit and uploading them on YouTube. Currently, it only supports images but we hope to bring the ability to combine videos as well in the future.

Codes for ECCV2022 paper - contrastive deep supervision

PlexSonarrAutoDownloader is a script that connects to Plex and Sonarr to automatically download the next season of a show you are watching based on the episode you are on.