Official implementation of the paper 'Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution' in CVPR 2022

Related tags

Admin Panels LDL
Overview

LDL

Paper | Supplementary Material

Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution
Jie Liang*, Hui Zeng*, and Lei Zhang.
In CVPR 2022 (Oral Presentation).

Abstract

Single image super-resolution (SISR) with generative adversarial networks (GAN) has recently attracted increasing attention due to its potentials to generate rich details. However, the training of GAN is unstable, and it often introduces many perceptually unpleasant artifacts along with the generated details. In this paper, we demonstrate that it is possible to train a GAN-based SISR model which can stably generate perceptually realistic details while inhibiting visual artifacts. Based on the observation that the local statistics (e.g., residual variance) of artifact areas are often different from the areas of perceptually friendly details, we develop a framework to discriminate between GAN-generated artifacts and realistic details, and consequently generate an artifact map to regularize and stabilize the model training process. Our proposed locally discriminative learning (LDL) method is simple yet effective, which can be easily plugged in off-the-shelf SISR methods and boost their performance. Experiments demonstrate that LDL outperforms the state-of-the-art GAN based SISR methods, achieving not only higher reconstruction accuracy but also superior perceptual quality on both synthetic and real-world datasets.

Overall illustration of the LDL:

illustration

For more details, please refer to our paper.

Getting started

  • Clone this repo.
git clone https://github.com/csjliang/LDL
cd LDL
  • Install dependencies. (Python 3 + NVIDIA GPU + CUDA. Recommend to use Anaconda)
pip install -r requirements.txt
  • Prepare the training and testing dataset by following this instruction.
  • Prepare the pre-trained models by following this instruction.

Training

First, check and adapt the yml file options/train/LDL/train_Synthetic_LDL.yml (or options/train/LDL/train_Realworld_LDL.yml for real-world image super-resolution), then

  • Single GPU:
PYTHONPATH="./:${PYTHONPATH}" CUDA_VISIBLE_DEVICES=0 python basicsr/train.py -opt options/train/LDL/train_Synthetic_LDL.yml --auto_resume

or

PYTHONPATH="./:${PYTHONPATH}" CUDA_VISIBLE_DEVICES=0 python realesrgan/train.py -opt options/train/LDL/train_Realworld_LDL.yml --auto_resume
  • Distributed Training:
PYTHONPATH="./:${PYTHONPATH}" CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4 --master_port=5678 basicsr/train.py -opt options/train/LDL/train_Synthetic_LDL.yml --launcher pytorch --auto_resume

or

PYTHONPATH=":${PYTHONPATH}" CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/train/LDL/train_Realworld_LDL.yml --launcher pytorch --auto_resume

Training files (logs, models, training states and visualizations) will be saved in the directory ./experiments/{name}

Testing

First, check and adapt the yml file options/test/LDL/test_LDL_Synthetic_x4.yml (or options/test/LDL/test_LDL_Realworld_x4.yml for real-world image super-resolution), then

  • Calculate metrics and save visual results for synthetic tasks:
PYTHONPATH="./:${PYTHONPATH}" CUDA_VISIBLE_DEVICES=0 python basicsr/test.py -opt options/test/LDL/test_LDL_Synthetic_x4.yml
  • Save visual results for real-world image super-resolution:
PYTHONPATH="./:${PYTHONPATH}" CUDA_VISIBLE_DEVICES=0 python basicsr/test.py -opt options/test/LDL/test_LDL_Realworld_x4.yml

Evaluating files (logs and visualizations) will be saved in the directory ./results/{name}

The Training and testing steps for scale=2 are similar.

Get Quantitative Metrics

First, check and adapt the settings of the files in metrics, then (take PSNR as an example) run

PYTHONPATH="./:${PYTHONPATH}" python scripts/metrics/table_calculate_psnr_all.py

Other metrics are similar.

License

This project is released under the Apache 2.0 license.

Citation

@inproceedings{jie2022LDL,
  title={Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution},
  author={Liang, Jie and Zeng, Hui and Zhang, Lei},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  year={2022}
}

Acknowledgement

This project is built based on the excellent BasicSR project.

Contact

Should you have any questions, please contact me via [email protected].

Comments
  • The pretrain model of scale=2

    The pretrain model of scale=2

    Hi, i download the pretrain model file that are scale=4. Can you sharring the scale=2 pretrain model file? The parameter scale=4 isn't change in class RRDBNet as follow: image

    opened by gaowq2017 3
  • the difference of type A, B and C

    the difference of type A, B and C

    Hi, in figure 5, the sigma values for type A, B, and C patches are 0.25, 0.39, 0.67. These numbers are setted originally or calculated by the input image patch?

    opened by jiamingNo1 2
  • error when training

    error when training

    when I tried to train LDL using train_Realworld_LDL.yml, it shows an error below

    Traceback (most recent call last): File "./basicsr/train.py", line 212, in train_pipeline(root_path) File "./basicsr/train.py", line 121, in train_pipeline result = create_train_val_dataloader(opt, logger) File "./basicsr/train.py", line 39, in create_train_val_dataloader train_set = build_dataset(dataset_opt) File "/nas/workspace/anse/code/pytorch/SR/LDL/basicsr/data/init.py", line 34, in build_dataset dataset = DATASET_REGISTRY.get(dataset_opt['type'])(dataset_opt) File "/nas/workspace/anse/code/pytorch/SR/LDL/basicsr/utils/registry.py", line 65, in get raise KeyError(f"No object named '{name}' found in '{self._name}' registry!") KeyError: "No object named 'RealESRGANDataset' found in 'dataset' registry!"

    opened by anse3832 2
  • inference error

    inference error

    hi

    when i test swinir in /inference/inference.py (two others are ok)

    from basicsr.archs.rrdbnet_arch import RRDBNet from basicsr.archs.srresnet_arch import MSRResNet from basicsr.archs.swinir_arch import SwinIR

    #set up model #model = RRDBNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=23, num_grow_ch=32) #model = MSRResNet(num_in_ch=3, num_out_ch=3, num_feat=64, num_block=16, upscale=4) model = SwinIR(upscale=4, img_size=(64, 64), window_size=8, img_range=1., depths=[6, 6, 6, 6], embed_dim=180, num_heads=[6, 6, 6, 6], mlp_ratio=2, upsampler='pixelshuffle')

    #model.load_state_dict(torch.load(args.model_path)['params'], strict=True) model.load_state_dict(torch.load(args.model_path), strict=True)

    ir occured error , whats the SwinIR params

    thanks

    opened by zhangyunming 1
  • he teeth  have a weird phenomenon

    he teeth have a weird phenomenon

    When I train on the face dataset, the teeth have a weird phenomenon。The portrait dataset is FFHQ, and the teeth generated by super-resolution are divided into multiple tooth blocks. May I ask what is the reason for this 企业微信20220613-173302@2x WechatIMG897 WechatIMG896 ?

    opened by carfei 1
  • No basicSR Version?

    No basicSR Version?

    Obviously, BasicSR is a good tool. However It is true that BasicSR is easy to use, but when I try to apply it, it takes too long to find out where those codes are. If you have time, could you make a version that is not BasicSR? It doesn't matter if you don't have distributed learning or other functions.

    opened by puppy9207 0
  • Training error

    Training error

    wecom-temp-fc57544ebe0e8dac4d14159ff3267797 The error in the training process seems to be a data problem,but i can not find any reason.There is nothing wrong with the data.There is an error in the middle of the training
    opened by carfei 0
Owner
null
Official PyTorch Implementation for DiRA: Discriminative, Restorative, and Adversarial Learning for Self-supervised Medical Image Analysis - CVPR 2022

[CVPR'22] DiRA: Discriminative, Restorative, and Adversarial Learning for Self-supervised Medical Image Analysis This repository provides a PyTorch im

null 55 Nov 22, 2022
Official PyTorch implementation of the paper "Deep Constrained Least Squares for Blind Image Super-Resolution", CVPR 2022.

Deep Constrained Least Squares for Blind Image Super-Resolution [Paper] This is the official implementation of 'Deep Constrained Least Squares for Bli

MEGVII Research 124 Nov 23, 2022
Implementation of our paper "Super-resolution with adversarial loss on the feature maps of the generated high-resolution image" (IET Electronics Letters 2022)

Adversarial Feature Maps Super Resolution This repository is the Implementation of our paper: Imanuel, I. and Lee, S. (2022), Super-resolution with ad

null 2 Oct 26, 2022
Official MegEngine implementation of ECCV2022 "D2C-SR: A Divergence to Convergence Approach for Real-World Image Super-Resolution".

[ECCV 2022] D2C-SR: A Divergence to Convergence Approach for Real-World Image Super-Resolution Youwei Li$^1$, Haibin Huang$^2$, Lanpeng Jia$^1$, Haoqi

MEGVII Research 31 Nov 14, 2022
This is the official PyTorch implementation of TBSR. Our team received 2nd place (real data track) and 3rd place (synthetic track) in NTIRE 2022 Burst Super-Resolution Challenge (CVPRW 2022).

Transformer for Burst Image Super-Resolution (TBSR) This is the official PyTorch implementation of TBSR. Our team received 2nd place (real data track)

Zhilu Zhang 11 Jul 26, 2022
This is the source code of the 7th place solution for stereo image super resolution task in 2022 CVPR NTIRE challenge.

Swin-iPASSR This is the source code of the 7th place solution for stereo image super resolution task in 2022 CVPR NTIRE challenge (Team Name: No War).

Chenxu Peng 5 May 8, 2022
Bank Account Simulation, it contains all the basic functions which include creating a new account, view account holders’ details, withdraws and deposit amount, picture update, transaction history and edit account details.

Bank Account Simulation Bank Account Simulation, it contains all the basic functions which include creating a new account, view account holders’ detai

Ansh D Mishra 4 Nov 2, 2022
The official implementation of "Bayesian Image Super-Resolution with Deep Modeling of Image Statistics" via TensorFlow

BayeSR The official implementation of "Bayesian Image Super-Resolution with Deep Modeling of Image Statistics" which has been accepted by IEEE Transac

Shangqi Gao 15 Nov 9, 2022
The official implementation of the ECCV 2022 Oral paper: RealFlow: EM-based Realistic Optical Flow Dataset Generation from Videos

[ECCV 2022 Oral] RealFlow: EM-based Realistic Optical Flow Dataset Generation from Videos (Paper) Yunhui Han1, Kunming Luo2, Ao Luo2, Jiangyu Liu2, Ha

MEGVII Research 49 Nov 5, 2022
Official implementation of the paper 'Efficient and Degradation-Adaptive Network for Real-World Image Super-Resolution'

DASR Paper Efficient and Degradation-Adaptive Network for Real-World Image Super-Resolution Jie Liang, Hui Zeng, and Lei Zhang. In arxiv preprint. Abs

null 80 Nov 15, 2022
An official PyTorch implementation of the paper "Feedback Network for Mutually Boosted Stereo Image Super-Resolution and Disparity Estimation"

This repository is an official PyTorch implementation of the paper Feedback Network for Mutually Boosted Stereo Image Super-Resolution and Disparity Estimation. (ACM MM 2021)

MIVRC 13 Nov 21, 2022
Official code for ECCV2022 paper: Learning Series-Parallel Lookup Tables for Efficient Image Super-Resolution

SPLUT Official code for ECCV2022 paper: Learning Series-Parallel Lookup Tables for Efficient Image Super-Resolution The folder training_testing_code c

null 35 Nov 25, 2022
Instance Regularization for Discriminative Language Model Pre-training (EMNLP 2022) (stay tuned & more will be updated)

Code for Paper Instance Regularization for Discriminative Language Model Pre-training Installation We use fastai and transformers toolkits for this wo

Zhuosheng Zhang 4 Oct 14, 2022
Official Github repository for the CVPR 2022 paper "GIRAFFE HD: A High-Resolution 3D-aware Generative Model"

GIRAFFE HD: A High-Resolution 3D-aware Generative Model Project Page Paper Usage Create and activate conda environment 'giraffehd': conda env create -

Yang Xue 57 Nov 16, 2022
[CVPR'22 Oral] TTVSR: Learning Trajectory-Aware Transformer for Video Super-Resolution

TTVSR (CVPR2022, Oral) This is the official PyTorch implementation of the paper Learning Trajectory-Aware Transformer for Video Super-Resolution. Cont

Multimedia Research 108 Nov 25, 2022
CADyQ: ContentsAware Dynamic Quantization for Image Super Resolution (ECCV 2022)

CADyQ : Content-Aware Dynamic Quantization for Image Super Resolution This respository is the official implementation of our ECCV2022 paper. The overv

Cheeun Hong 34 Nov 22, 2022
[AIM & ECCVW 2022] Fast Nearest Convolution for Real-Time Image Super-Resolution

Fast Nearest Convolution for Real-Time Image Super-Resolution, AIM & ECCV Workshops 2022 Update [2022.08.25] We have uploaded the pretrained model in

balabala 44 Nov 23, 2022
[ECCV 2022] Self-Supervised Learning for Real-World Super-Resolution from Dual Zoomed Observations

SelfDZSR (ECCV 2022) PyTorch implementation of Self-Supervised Learning for Real-World Super-Resolution from Dual Zoomed Observations 1. Framework Ove

Zhilu Zhang 38 Nov 23, 2022
Code for CVPR 2022 CLEAR Challenge "This repository is the CLEAR Challenge 1st place methods for CVPR 2022 Workshop on Visual Perception and Learning in an Open World"

CLEAR | Starter Kit This repository is the CLEAR Challenge 1st place methods for CVPR 2022 Workshop on Visual Perception and Learning in an Open World

Tencent YouTu Research 5 Sep 9, 2022