Official PyTorch implementation for the paper High-Resolution Virtual Try-On with Misalignment and Occlusion-Handled Conditions (ECCV 2022).

Overview

HR-VITON — Official PyTorch Implementation

Teaser image

High-Resolution Virtual Try-On with Misalignment and Occlusion-Handled Conditions
Sangyun Lee*1, Gyojung Gu*2,3, Sunghyun Park2, Seunghwan Choi2, Jaegul Choo2
1Soongsil University, 2KAIST, 3Nestyle
In ECCV 2022 (* indicates equal contribution)

Paper: https://arxiv.org/abs/2206.14180
Project page: https://koo616.github.io/HR-VITON

Abstract: Image-based virtual try-on aims to synthesize an image of a person wearing a given clothing item. To solve the task, the existing methods warp the clothing item to fit the person's body and generate the segmentation map of the person wearing the item before fusing the item with the person. However, when the warping and the segmentation generation stages operate individually without information exchange, the misalignment between the warped clothes and the segmentation map occurs, which leads to the artifacts in the final image. The information disconnection also causes excessive warping near the clothing regions occluded by the body parts, so-called pixel-squeezing artifacts. To settle the issues, we propose a novel try-on condition generator as a unified module of the two stages (i.e., warping and segmentation generation stages). A newly proposed feature fusion block in the condition generator implements the information exchange, and the condition generator does not create any misalignment or pixel-squeezing artifacts. We also introduce discriminator rejection that filters out the incorrect segmentation map predictions and assures the performance of virtual try-on frameworks. Experiments on a high-resolution dataset demonstrate that our model successfully handles the misalignment and occlusion, and significantly outperforms the baselines.

Installation

Clone this repository:

git clone https://github.com/sangyun884/HR-VITON.git
cd ./HR-VITON/

Install PyTorch and other dependencies:

conda create -n {env_name} python=3.8
conda activate {env_name}
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch-lts -c nvidia
pip install opencv-python torchgeometry Pillow tqdm tensorboardX scikit-image scipy

Dataset

We train and evaluate our model using the dataset from VITON-HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalization.

To download the dataset, please check the following link https://github.com/shadow2496/VITON-HD.

We assume that you have downloaded it into ./data.

Inference

Here are the download links for each model checkpoint:

  • Try-on condition generator: link
  • Try-on condition generator (discriminator): link
  • Try-on image generator: link
  • AlexNet (LPIPS): link, we assume that you have downloaded it into ./eval_models/weights/v0.1.
python3 test_generator.py --occlusion --test_name {test_name} --tocg_checkpoint {condition generator ckpt} --gpu_ids {gpu_ids} --gen_checkpoint {image generator ckpt} --datasetting unpaired --dataroot {dataset_path} --data_list {pair_list_textfile}

Train try-on condition generator

python3 train_condition.py --gpu_ids {gpu_ids} --Ddownx2 --Ddropout --lasttvonly --interflowloss --occlusion

Train try-on image generator

python3 train_generator.py --name test -b 4 -j 8 --gpu_ids {gpu_ids} --fp16 --tocg_checkpoint {condition generator ckpt path} --occlusion

This stage takes approximately 4 days with two RTX 3090 GPUs. Tested environment: PyTorch 1.8.2+cu111.

To use "--fp16" option, you should install apex library.

License

All material is made available under Creative Commons BY-NC 4.0. You can use, redistribute, and adapt the material for non-commercial purposes, as long as you give appropriate credit by citing our paper and indicate any changes that you've made.

Citation

If you find this work useful for your research, please cite our paper:

@article{lee2022hrviton,
  title={High-Resolution Virtual Try-On with Misalignment and Occlusion-Handled Conditions},
  author={Lee, Sangyun and Gu, Gyojung and Park, Sunghyun and Choi, Seunghwan and Choo, Jaegul},
  journal={arXiv preprint arXiv:2206.14180},
  year={2022}
}
Comments
  • Are training pairs meant to be mismatched?

    Are training pairs meant to be mismatched?

    Hi,

    I downloaded the dataset provided in the VITON-HD repo.

    Both papers train on the piece of clothing that is already on the model.

    However the provided train_pairs.txt contains mismatched human and clothing ids.

    Am I misunderstanding something or should I re-create the train_pairs file?

    opened by nihirv 15
  • IndexError: list index out of range

    IndexError: list index out of range

    Getting the following error when running on Windows.

    Traceback (most recent call last):
      File "test_generator.py", line 249, in <module>
        main()
      File "test_generator.py", line 225, in main
        board = SummaryWriter(log_dir=os.path.join(opt.tensorboard_dir, opt.tocg_checkpoint.split('/')[-2], opt.tocg_checkpoint.split('/')[-1], opt.datamode, opt.datasetting))
    IndexError: list index out of range
    

    Any idea?

    opened by psx2 4
  • Understanding the TPS / Warp Operations

    Understanding the TPS / Warp Operations

    Hi! Thanks for sharing your work it helps develop my understanding.

    I am trying to understand the TPS warp operations am I correct in saying:

    • The model predicts the TPS transformation parameters
    • The TPS transformation parameters are then applied to each level of the latent features (ie the downscaled image features)?
    • The TPS transformation parameters are learned by back propagating between the ground truth warped segmentation map (only the cloth item) and the predicted warped cloth mask?

    Thank you kindly

    opened by torphix 2
  • Article HR-VITON

    Article HR-VITON

    I'm writing an article on my master's degree that use your with reference. I need do a little review, but I've dificult with Latex. Could you share with me you article in Latex please? My email: [email protected]

    opened by julianorinaldi 2
  • About training settings

    About training settings

    Hi, thanks for sharing your codes. But I was confused by the difference between your codes and the description in your paper during retraining your model. For example, when training try-on generator, the code will train the model for 300K iters but it is only 100k mentioned in your paper (Sec 4.1, last line in Page8). So, are the excellent results in your paper trained according to the settings mentioned in the paper or according to the released code settings. And which one should us follow to get your paper results.

    opened by HITRainer 1
  • How to train the Condition Generator with multi-gpus?

    How to train the Condition Generator with multi-gpus?

    Hi,

    Thanks for releasing the training code.

    I would like to train the condition generator with another dataset (with a resolution of 512 x 384). However, running the complete 300000 steps under the default setting takes a long time (> 130h). So I would like to ask whether the author plans to release the multi-GPU version of the training code. Or is there any suggestion about how to train the condition generator within 1~2 days?

    opened by xiezhy6 1
  • Which github repository did you use to measure a FID & KID score?

    Which github repository did you use to measure a FID & KID score?

    Hello,

    First, thank you for sharing your great work!

    I'd like to check which github repository did you use to measure a FID as well as KID?

    Thank you advance!

    opened by Gomdoree 0
  • The reason behind the warp block?

    The reason behind the warp block?

    In the file networks.py, line 133~135

    flow_norm = torch.cat([flow[:, :, :, 0:1] / ((iW/2 - 1.0) / 2.0), flow[:, :, :, 1:2] / ((iH/2 - 1.0) / 2.0)], 3)
    warped_T1 = F.grid_sample(T1, flow_norm + grid, padding_mode='border')
    
    

    Why do you need to normalize the value of the flow? Why do you sum the flow_norm with grid? If we do not perform norm and/or sum the flow_norm with the grid, what are the impact to the final results? Any paper mention about this warp block? I read your paper but it do not give much explanation about this warp block(The W symbol in the Fig10).

    Thanks

    opened by stereomatchingkiss 0
  • Doubts regarding the flow of the code

    Doubts regarding the flow of the code

    I have some doubts in functions like get_agnostic in cp_dataset.py, train in train condition.py, Evaluation in evaluate.py, etc .Would like to have a video meet session on zoom or a similar platform to clarify these doubts.

    opened by lavishjain1234 0
  • I ran your code but I got bad result in train_condition

    I ran your code but I got bad result in train_condition

    I ran your code and got codes with different results. The results were bad. I don't know what the problem was. In the paper, even your HD results were good, while my 256 * 192 results were poor. I don't know what's different from yours, but the result is very poor. Can you give me some help

    微信图片_20220917150917 微信图片_20220917151101

    opened by zx874561 0
  • CPU or GPU inference

    CPU or GPU inference

    Make the code available to run not only on the GPU but also can use CPU. using a command parameter -- cuda . this command has True or False values which true means use cuda and false use cpu instead.

    opened by KerolosAtef 0
  • The arm generate poor result

    The arm generate poor result

    Hi, I'am very interested in this project. So I try to use and customize this code, but i receive to poor result: the arms generated bad and seem to be black(see attach). I did not know where the issue come from result10 result13 white_red2 Can anyone help me? Thank!

    opened by srckid 1
[ECCV 2022] Official pytorch implementation of "mc-BEiT: Multi-choice Discretization for Image BERT Pre-training" in European Conference on Computer Vision (ECCV) 2022.

mc-BEiT: Multi-choice Discretization for Image BERT Pre-training Official pytorch implementation of "mc-BEiT: Multi-choice Discretization for Image BE

lixiaotong 16 Sep 28, 2022
[ECCV'22] The official PyTorch implementation of our ECCV 2022 paper: "AiATrack: Attention in Attention for Transformer Visual Tracking".

AiATrack The official PyTorch implementation of our ECCV 2022 paper: AiATrack: Attention in Attention for Transformer Visual Tracking Shenyuan Gao, Ch

Shenyuan Gao 47 Sep 24, 2022
Super-Resolution by Predicting Offsets: An Ultra-Efficient Super-Resolution Network for Rasterized Images. ECCV 2022. [ Official ]

SRPO [Official Code] Super-Resolution by Predicting Offsets: An Ultra-Efficient Super-Resolution Network for Rasterized Images. (Paper Link) By Jinjin

Helmut Choy 10 Sep 24, 2022
Implementation of our paper "Super-resolution with adversarial loss on the feature maps of the generated high-resolution image" (IET Electronics Letters 2022)

Adversarial Feature Maps Super Resolution This repository is the Implementation of our paper: Imanuel, I. and Lee, S. (2022), Super-resolution with ad

null 1 Mar 26, 2022
The official PyTorch implementation of the ECCV 2022 paper: Ghost-free High Dynamic Range Imaging with Context-aware Transformer

[ECCV 2022]Ghost-free High Dynamic Range Imaging with Context-aware Transformer By Zhen Liu1, Yinglong Wang2, Bing Zeng3 and Shuaicheng Liu3,1* 1Megvi

Zhen Liu 11 Sep 21, 2022
This is an official PyTorch implementation of our IJCAI-ECAI 2022 paper "Dite-HRNet: Dynamic Lightweight High-Resolution Network for Human Pose Estimation"

Dite-HRNet: Dynamic Lightweight High-Resolution Network for Human Pose Estimation Introduction This is an official PyTorch implementation of our IJCAI

Ziyi Zhang 36 Sep 8, 2022
The official MegEngine implementation of the ECCV 2022 paper: Ghost-free High Dynamic Range Imaging with Context-aware Transformer

[ECCV 2022]Ghost-free High Dynamic Range Imaging with Context-aware Transformer By Zhen Liu1, Yinglong Wang2, Bing Zeng3 and Shuaicheng Liu3,1* 1Megvi

MEGVII Research 46 Sep 28, 2022
High-Resolution Image/Video Harmonization [ECCV 2022]

Harmonizer: High-Resolution Image/Video Harmonization Harmonizer: Learning to Perform White-Box Image and Video Harmonization (ECCV 2022) Paper | Demo

Zhanghan Ke 92 Sep 26, 2022
This is the official PyTorch implementation of TBSR. Our team received 2nd place (real data track) and 3rd place (synthetic track) in NTIRE 2022 Burst Super-Resolution Challenge (CVPRW 2022).

Transformer for Burst Image Super-Resolution (TBSR) This is the official PyTorch implementation of TBSR. Our team received 2nd place (real data track)

Zhilu Zhang 11 Jul 26, 2022
Official Github repository for the CVPR 2022 paper "GIRAFFE HD: A High-Resolution 3D-aware Generative Model"

GIRAFFE HD: A High-Resolution 3D-aware Generative Model Project Page Paper Usage Create and activate conda environment 'giraffehd': conda env create -

Yang Xue 51 Sep 17, 2022
The official repo for OC-SORT: Observation-Centric SORT on video Multi-Object Tracking. OC-SORT is simple, online and robust to occlusion/non-linear motion.

OC-SORT This is the github repo for Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking [arxiv]. Observation-Centric SORT (OC-S

Jinkun Cao 262 Oct 1, 2022
Official Pytorch implementation of the ECCV 2022 paper "GIPSO: Geometrically Informed Propagation for Online Adaptation in 3D LiDAR Segmentation"

GIPSO: Geometrically Informed Propagation for Online Adaptation in 3D LiDAR Segmentation [ECCV2022] The official implementation of our work "GIPSO: Ge

Cristiano Saltori 21 Sep 29, 2022
Official PyTorch implementation of the ECCV 2022 paper "CoSMix: Compositional Semantic Mix for Domain Adaptation in 3D LiDAR Segmentation"

CoSMix: Compositional Semantic Mix for Domain Adaptation in 3D LiDAR Segmentation [ECCV2022] The official implementation of our work "CoSMix: Composit

Cristiano Saltori 25 Sep 14, 2022
[ECCV 2022] Official pytorch implementation of the paper, "PointMixer: MLP-Mixer for Point Cloud Understanding"

PointMixer: MLP-Mixer for Point Cloud Understanding This is an official implementation for the paper, PointMixer: MLP-Mixer for Point Cloud Understand

Jaesung.Choe 41 Sep 29, 2022
Official PyTorch implementation of the paper "Deep Constrained Least Squares for Blind Image Super-Resolution", CVPR 2022.

Deep Constrained Least Squares for Blind Image Super-Resolution [Paper] This is the official implementation of 'Deep Constrained Least Squares for Bli

MEGVII Research 111 Sep 23, 2022
Deep Comprehensible Color Filter Learning Framework for High-Resolution Image Harmonization (ECCV 22)

DCCF: Deep Comprehensible Color Filter Learning Framework This is the official repository for our paper: DCCF: Deep Comprehensible Color Filter Learni

Ben Xue 13 Sep 20, 2022
Code for Kolic et al. (2022) "Estimating initial conditions for dynamical systems with incomplete information"

LatentStateInitialization This code provides the algorithms to infer the latent initial conditions from a sequence of aggregate noisy observations of

Blas Kolic 8 Sep 2, 2022
Official Pytorch Implementation for "Text2LIVE: Text-Driven Layered Image and Video Editing" (ECCV 2022 Oral)

Text2LIVE: Text-Driven Layered Image and Video Editing (ECCV 2022 - Oral) [Project Page] Text2LIVE is a method for editing a real-world image/video, u

Omer Bar Tal 139 Sep 27, 2022
Official PyTorch Lightning Implementation of "Minimal Neural Atlas: Parameterizing Complex Surfaces with Minimal Charts and Distortion" (ECCV 2022)

Minimal Neural Atlas This repository contains the official source code for the ECCV 2022 paper "Minimal Neural Atlas: Parameterizing Complex Surfaces

Low Weng Fei 13 Aug 27, 2022