Pytorch code for "BSRT: Improving Burst Super-Resolution with Swin Transformer and Flow-Guided Deformable Alignment", CVPRW, 1st place in NTIRE 2022 BurstSR Challenge (real-world track).

Overview

BSRT: Improving Burst Super-Resolution with Swin Transformer and Flow-Guided Deformable Alignment (CVPRW 2022)

PWC PWCvisitors

BSRT, the winner of the NTIRE 2022 Burst Super-Resolution Challenge Real-World Track.

You can also find our winner method in NTIRE 2021 Burst Super-Resolution Challenge here.

This work addresses the Burst Super-Resolution (BurstSR) task using a new architecture, which requires restoring a high-quality image from a sequence of noisy, misaligned, and low-resolution RAW bursts. To overcome the challenges in BurstSR, we propose a Burst Super-Resolution Transformer (BSRT), which can significantly improve the capability of extracting inter-frame information and reconstruction. To achieve this goal, we propose a Pyramid Flow-Guided Deformable Convolution Network (Pyramid FG-DCN) and incorporate Swin Transformer Blocks and Groups as our main backbone. More specifically, we combine optical flows and deformable convolutions, hence our BSRT can handle misalignment and aggregate the potential texture information in multi-frames more efficiently. In addition, our Transformer-based structure can capture long-range dependency to further improve the performance. The evaluation on both synthetic and real-world tracks demonstrates that our approach achieves a new state-of-the-art in BurstSR task. Further, our BSRT wins the championship in the NTIRE2022 Burst Super-Resolution Challenge.

Comparison with State-of-the-art Burst Super-Resolution Methods

ts

Overview Architecture

overview.png

Dependencies

  • OS: Ubuntu 18.04
  • Python: Python 3.7
  • nvidia :
    • cuda: 10.1
    • cudnn: 7.6.1
  • Other reference requirements

Quick Start

1.Create a conda virtual environment and activate it

conda create -n pytorch_1.6 python=3.7
source activate pytorch_1.6

2.Install PyTorch and torchvision following the official instructions

conda install pytorch==1.6.0 torchvision==0.7.0 cudatoolkit=10.1 -c pytorch

3.Install build requirements

pip3 install -r requirements.txt

4.Install DCN

cd DCNv2
python3 setup.py build develop # build
python3 test.py # run examples and check

Training

We provide all pretrained model weights here.

For Synthetic data

cd code/synthetic/bsrt
# Modify the root path of training dataset and model etc.
# The number of GPUs should be more than 1
python main.py --n_GPUs 8 --print_every 40 --lr 0.0001 --decay 150-300 --save bsrt_tiny --model BSRT --fp16 --model_level S --swinfeature --batch_size 32 --burst_size 14 --patch_size 256

For Real-World data

cd code/real/bsrt
# Modify the root path of training dataset and model etc.
# The number of GPUs should be more than 1
python main.py --n_GPUs 8 --print_every 20 --lr 0.00005 --decay 40-80 --save bsrt_tiny --model BSRT --fp16 --model_level S --swinfeature --batch_size 8 --burst_size 14 --patch_size 80 --pre_train ../../synthetic/train_log/bsrt/real_models/bsrt_tiny/bsrt_best_epoch.pth 

The pretrained PWC-Net model can be downloaded here.

Test

For Synthetic data

# Modify the path of test dataset and the path of the trained model
python test_synburst.py --n_GPUs 1 --model BSRT --model_level S --swinfeature --burst_size 14 --patch_size 384 --pre_train ../train_log/bsrt/real_models/bsrt_tiny/bsrt_best_epoch.pth --root /data/dataset/ntire21/burstsr/synthetic

For Real-World data

# Modify the path of test dataset and the path of the trained model
python test_real.py --n_GPUs 1 --model BSRT --model_level S --swinfeature --batch_size 1 --burst_size 14 --patch_size 80 --pre_train ../train_log/bsrt/real_models/bsrt_tiny/bsrtbest_epoch.pth --root /data/dataset/ntire21/burstsr/real

Results

Comparison on Synthetic dataset

cmp_syn.png

Comparison on Real-World dataset

cmp_real.png

Citations

If our code helps your research or work, please consider citing our paper. The following is a BibTeX reference.

@inproceedings{luo2022bsrt,
  title={BSRT: Improving Burst Super-Resolution with Swin Transformer and Flow-Guided Deformable Alignment},
  author={Luo, Ziwei and Li, Youwei and Cheng, Shen and Yu, Lei and Wu, Qi and Wen, Zhihong and Fan, Haoqiang and Sun, Jian and Liu, Shuaicheng},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={998--1008},
  year={2022}
}

Contact

email: [[email protected]]

You might also like...

Official PyTorch implementation of "Accurate 3D Hand Pose Estimation for Whole-Body 3D Human Mesh Estimation", CVPRW 2022

Official PyTorch implementation of

Accurate 3D Hand Pose Estimation for Whole-Body 3D Human Mesh Estimation (Hand4Whole codes) High-resolution video link: here Introduction This repo is

Sep 30, 2022

Code for the winning solution in the SE&R 2022 Challenge - SER track.

SE&R 2022 Challenge - SER Track Introduction Dependencies Datasets Training Pre-Trained Model Contact Introduction Automatic Speech Recognition for sp

Aug 28, 2022

[CVPRW 2022] MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment

[CVPRW 2022] MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment

MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment Authors: Sidi Yang, Tianhe Wu, Shuwei Shi, Shanshan Lao, Yuan Gong

Sep 14, 2022

3D Pseudo-GTs of "NeuralAnnot: Neural Annotator for 3D Human Mesh Training Sets", CVPRW 2022

NeuralAnnot: Neural Annotator for 3D Human Mesh Training Sets Introduction This repo provides 3D pseudo-GTs (SMPL/MANO/FLAME/SMPL-X parameters) of var

Sep 27, 2022

ECCV 2022 Workshop: A Benchmark for Robustness to Individual Nuisances in Real-World Out-of-Distribution Shifts

ROBIN-dataset Changes The Phase-1 of the competition will not be a code submission challenge, we have released all the test data and labels in this re

Sep 14, 2022

[ECCV 2022] Self-Supervised Learning for Real-World Super-Resolution from Dual Zoomed Observations

[ECCV 2022] Self-Supervised Learning for Real-World Super-Resolution from Dual Zoomed Observations

SelfDZSR (ECCV 2022) PyTorch implementation of Self-Supervised Learning for Real-World Super-Resolution from Dual Zoomed Observations 1. Framework Ove

Sep 25, 2022

[CVPR 2022] Official Pytorch code for OW-DETR: Open-world Detection Transformer

[CVPR 2022] Official Pytorch code for OW-DETR: Open-world Detection Transformer

OW-DETR: Open-world Detection Transformer (CVPR 2022) [Paper] Akshita Gupta*, Sanath Narayan*, K J Joseph, Salman Khan, Fahad Shahbaz Khan, Mubarak Sh

Sep 22, 2022

A world simulation model that will be used to simulate the outcome of human actions on the world.

🎯 Goal - Build a World Simulation Model The goal of this project is to build a world simulation model that will be used to simulate the outcome of hu

Sep 22, 2022

Clarity Challenge toolkit - software for building Clarity Challenge systems

Clarity Challenge toolkit - software for building Clarity Challenge systems

Machine learning challenges for hearing aid processing. We are organising a series of machine learning challenges to enhance hearing-aid signal proces

Sep 18, 2022
Comments
  • Training without access to Zurich RAW to RGB (ZRR) validation

    Training without access to Zurich RAW to RGB (ZRR) validation

    Hello,

    Thank you for sharing your code!

    I have a fork (https://github.com/ConnorBaker/BSRT) where I updated CUDA/PyTorch and switched to the DCN implementation offered by torchvision (details here: https://pytorch.org/vision/stable/generated/torchvision.ops.deform_conv2d.html#torchvision.ops.deform_conv2d). However, torchvision's DCN expects the second dimension of weights to be divided by the number of groups, which causes an error when trying to use the pre-trained model, since the sizes of the tensors don't match. I tried to work around this so I could still use your pre-trained model, but was unable to: see https://github.com/ConnorBaker/BSRT/commit/7d9c5bdb873693109fab2a8217e482f0cf5c6ee9 for details and https://github.com/ConnorBaker/BSRT/commit/8b8339491205bdbddf14ffdea01cc92584e24939 for the full switch over.

    How can I train this model? I tried to get access to the full Zurich RAW to RGB (ZRR) dataset, but it is no longer available. Without the validation data (val directory from ZRR), I get errors when I try to run training (see example below).

    Any ideas?

    Many thanks, Connor


    Example run with training:

    $ python main.py --n_GPUs 1 --print_every 40 --lr 0.0001 --decay 150-300 --save bsrt_tiny --model BSRT --fp16 --model_level S --swinfeature --batch_size 32 --burst_size 14 --patch_size 256 --root ~/datasets/zurich-raw-to-rgb --models_root ~/models
    train data: 46839, test data: 300
    Test only: False
    Making model:  BSRT
    Patch size:  256
    depths:  [6, 6, 6, 6, 6]
    using swinfeature
    /home/connorbaker/micromamba/envs/bsrt-real/lib/python3.7/site-packages/torch/functional.py:478: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  /opt/conda/conda-bld/pytorch_1659484801627/work/aten/src/ATen/native/TensorShape.cpp:2894.)
      return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
    Preparing loss function:
    1.000 * L1
    Setting up [LPIPS] perceptual loss: trunk [alex], v[0.1], spatial [off]
    /home/connorbaker/micromamba/envs/bsrt-real/lib/python3.7/site-packages/torchvision/models/_utils.py:209: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and will be removed in 0.15, please use 'weights' instead.
      f"The parameter '{pretrained_param}' is deprecated since 0.13 and will be removed in 0.15, "
    /home/connorbaker/micromamba/envs/bsrt-real/lib/python3.7/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and will be removed in 0.15. The current behavior is equivalent to passing `weights=AlexNet_Weights.IMAGENET1K_V1`. You can also use `weights=AlexNet_Weights.DEFAULT` to get the most up-to-date weights.
      warnings.warn(msg)
    Loading model from: /home/connorbaker/micromamba/envs/bsrt-real/lib/python3.7/site-packages/lpips/weights/v0.1/alex.pth
    number of parameters:  4786641
    Fix keys: ['spynet', 'dcnpack'] for the first 5 epochs.
    [1280/46839]	[0.0483]	58.5+2.1s
    [2560/46839]	[0.0382]	54.9+0.2s
    [3840/46839]	[0.0277]	54.9+0.2s
    [5120/46839]	[0.0223]	54.9+0.2s
    [6400/46839]	[0.0277]	54.8+0.2s
    [7680/46839]	[0.0252]	54.9+0.2s
    [8960/46839]	[0.0292]	54.8+0.2s
    [10240/46839]	[0.0321]	54.9+0.2s
    [11520/46839]	[0.0256]	54.9+0.2s
    [12800/46839]	[0.0240]	54.9+0.2s
    [14080/46839]	[0.0242]	54.9+0.2s
    [15360/46839]	[0.0240]	54.9+0.2s
    [16640/46839]	[0.0207]	54.9+0.2s
    [17920/46839]	[0.0184]	54.9+0.2s
    [19200/46839]	[0.0238]	55.0+0.2s
    [20480/46839]	[0.0200]	54.9+0.2s
    [21760/46839]	[0.0150]	54.8+0.2s
    [23040/46839]	[0.0195]	54.8+0.2s
    [24320/46839]	[0.0154]	54.8+0.2s
    [25600/46839]	[0.0215]	54.8+0.2s
    [26880/46839]	[0.0190]	54.9+0.2s
    [28160/46839]	[0.0167]	54.8+0.2s
    [29440/46839]	[0.0194]	54.8+0.2s
    [30720/46839]	[0.0160]	54.8+0.2s
    [32000/46839]	[0.0194]	54.8+0.2s
    [33280/46839]	[0.0182]	54.8+0.2s
    [34560/46839]	[0.0153]	54.8+0.2s
    [35840/46839]	[0.0179]	54.8+0.2s
    [37120/46839]	[0.0199]	54.9+0.2s
    [38400/46839]	[0.0160]	54.8+0.2s
    [39680/46839]	[0.0186]	54.9+0.2s
    [40960/46839]	[0.0195]	54.9+0.2s
    [42240/46839]	[0.0169]	54.9+0.2s
    [43520/46839]	[0.0156]	54.9+0.2s
    [44800/46839]	[0.0172]	54.8+0.2s
    [46080/46839]	[0.0158]	54.9+0.2s
    Epoch 1 cost time: 2018.3s, lr: 0.000100
    save model...
    save model...
    Testing...
    [ WARN:[email protected]] global /home/conda/feedstock_root/build_artifacts/libopencv_1661642960203/work/modules/imgcodecs/src/loadsave.cpp (239) findDecoder imread_('/home/connorbaker/datasets/zurich-raw-to-rgb/val/bursts/0000/im_raw_00.png'): can't open/read file: check file path/integrity
    [ WARN:[email protected]] global /home/conda/feedstock_root/build_artifacts/libopencv_1661642960203/work/modules/imgcodecs/src/loadsave.cpp (239) findDecoder imread_('/home/connorbaker/datasets/zurich-raw-to-rgb/val/bursts/0032/im_raw_00.png'): can't open/read file: check file path/integrity
    [ WARN:[email protected]] global /home/conda/feedstock_root/build_artifacts/libopencv_1661642960203/work/modules/imgcodecs/src/loadsave.cpp (239) findDecoder imread_('/home/connorbaker/datasets/zurich-raw-to-rgb/val/bursts/0064/im_raw_00.png'): can't open/read file: check file path/integrity
    [ WARN:[email protected]] global /home/conda/feedstock_root/build_artifacts/libopencv_1661642960203/work/modules/imgcodecs/src/loadsave.cpp (239) findDecoder imread_('/home/connorbaker/datasets/zurich-raw-to-rgb/val/bursts/0160/im_raw_00.png'): can't open/read file: check file path/integrity
    [ WARN:[email protected]] global /home/conda/feedstock_root/build_artifacts/libopencv_1661642960203/work/modules/imgcodecs/src/loadsave.cpp (239) findDecoder imread_('/home/connorbaker/datasets/zurich-raw-to-rgb/val/bursts/0128/im_raw_00.png'): can't open/read file: check file path/integrity
    [ WARN:[email protected]] global /home/conda/feedstock_root/build_artifacts/libopencv_1661642960203/work/modules/imgcodecs/src/loadsave.cpp (239) findDecoder imread_('/home/connorbaker/datasets/zurich-raw-to-rgb/val/bursts/0192/im_raw_00.png'): can't open/read file: check file path/integrity
    Traceback (most recent call last):
      File "main.py", line 98, in <module>
    [ WARN:[email protected]] global /home/conda/feedstock_root/build_artifacts/libopencv_1661642960203/work/modules/imgcodecs/src/loadsave.cpp (239) findDecoder imread_('/home/connorbaker/datasets/zurich-raw-to-rgb/val/bursts/0256/im_raw_00.png'): can't open/read file: check file path/integrity
        main()
      File "main.py", line 42, in main
        main_worker(0, args.n_GPUs, args)
      File "main.py", line 85, in main_worker
        t.train()
      File "/home/connorbaker/BSRT/code/synthetic/bsrt/trainer.py", line 228, in train
        self.test()
      File "/home/connorbaker/BSRT/code/synthetic/bsrt/trainer.py", line 263, in test
        for i, batch_value in enumerate(self.loader_valid):
      File "/home/connorbaker/micromamba/envs/bsrt-real/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 681, in __next__
        data = self._next_data()
      File "/home/connorbaker/micromamba/envs/bsrt-real/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1376, in _next_data
        return self._process_data(data)
      File "/home/connorbaker/micromamba/envs/bsrt-real/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1402, in _process_data
        data.reraise()
      File "/home/connorbaker/micromamba/envs/bsrt-real/lib/python3.7/site-packages/torch/_utils.py", line 461, in reraise
        raise exception
    AttributeError: Caught AttributeError in DataLoader worker process 0.
    Original Traceback (most recent call last):
      File "/home/connorbaker/micromamba/envs/bsrt-real/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
        data = fetcher.fetch(index)
      File "/home/connorbaker/micromamba/envs/bsrt-real/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
        data = [self.dataset[idx] for idx in possibly_batched_index]
      File "/home/connorbaker/micromamba/envs/bsrt-real/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
        data = [self.dataset[idx] for idx in possibly_batched_index]
      File "/home/connorbaker/BSRT/code/synthetic/bsrt/datasets/synthetic_burst_val_set.py", line 61, in __getitem__
        burst = [self._read_burst_image(index, i) for i in range(self.burst_size)]
      File "/home/connorbaker/BSRT/code/synthetic/bsrt/datasets/synthetic_burst_val_set.py", line 61, in <listcomp>
        burst = [self._read_burst_image(index, i) for i in range(self.burst_size)]
      File "/home/connorbaker/BSRT/code/synthetic/bsrt/datasets/synthetic_burst_val_set.py", line 33, in _read_burst_image
        im_t = torch.from_numpy(im.astype(np.float32)).permute(2, 0, 1).float() / (2**14)
    AttributeError: 'NoneType' object has no attribute 'astype'
    
    opened by ConnorBaker 3
  • IndexError: list index out of range

    IndexError: list index out of range

    作者,你好,请问 Traceback (most recent call last): File "test.py", line 24, in opt = option.parse(parser.parse_args().opt, is_train=False) File "D:\pythonProject\7_4\DCLS-SR-master\DCLS-SR-master\codes\config\DCLS\options.py", line 60, in parse config_dir = path.split("/")[-2] 这个是为啥?一直不行,还有一个问题就是先验模型放在哪个位置呢?可以详细写下测试过程吗?测试过程不需要数据集吧?感谢

    opened by CodeMadUser 0
Owner
balabala
balabala
Code for CVPR 2022 CLEAR Challenge "This repository is the CLEAR Challenge 1st place methods for CVPR 2022 Workshop on Visual Perception and Learning in an Open World"

CLEAR | Starter Kit This repository is the CLEAR Challenge 1st place methods for CVPR 2022 Workshop on Visual Perception and Learning in an Open World

Tencent YouTu Research 5 Sep 9, 2022
"MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction" (CVPRW 2022) & (Winner of NTIRE 2022 Challenge on Spectral Reconstruction from RGB)

MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction (CVPRW 2022) Yuanhao Cai, Jing Lin, Zudi Lin, Haoqian Wang, Yulun Z

Yuanhao Cai 251 Sep 23, 2022
Solution for Team Rayee, ranks 1st place in reconstructing partial textured objects (track 2), and 2nd overall in the SHARP Challenge at CVPR 2022

3D Textured Shape Recovery with Learned Geometric Priors Team Rayee, ranks 1st place in reconstructing partial textured objects (track 2), and 2nd ove

Zhizheng Liu 7 Oct 1, 2022
This is the source code of the 7th place solution for stereo image super resolution task in 2022 CVPR NTIRE challenge.

Swin-iPASSR This is the source code of the 7th place solution for stereo image super resolution task in 2022 CVPR NTIRE challenge (Team Name: No War).

Chenxu Peng 5 May 8, 2022
Winner of runtime track in NTIRE 2022 challenge on Efficient Super-Resolution

Residual Local Feature Network Our team (ByteESR) won the first place in Runtime Track (Main Track) and the second place in Overall Performance Track

Bytedance Inc. 56 Sep 27, 2022
1st place solution for 3D Open World FPS FPS Game AI Chanllenge(2022 cog&inspir.AI)

Wildscav_fps_mcc_fang This is the 1st place solution for 3D Open World FPS Game AI Chanllenge(2022 cog&inspir.AI) Admittedly, our code currently is fa

null 3 Sep 29, 2022
The method for Ntire 2022 Efficient Super-Resolution Challenge

RFESR Code for our method Residual Feature Extraction Network for Ntire 2022 Efficient Super-Resolution Challenge The model files are uploaded! We use

ShiRui 2 Apr 7, 2022
4st place solution for the PBVS 2022 Multi-modal Aerial View Object Classification Challenge - Track 1 (SAR) at PBVS2022

A Two-Stage Shake-Shake Network for Long-tailed Recognition of SAR Aerial View Objects 4st place solution for the PBVS 2022 Multi-modal Aerial View Ob

LinpengPan 5 Aug 24, 2022
RAW-based blind denoising, 1st place in MegCup 2022 (Team Feedback)

MegCup 2022 Team Feedback This repository is the 1st place solution (Team Feedback) in 2022 MegCup RAW image denoising. Here is the 3rd place solution

韩凌昊 38 Jul 2, 2022
System that ranks 1st in DCASE 2022 Challenge Task 5: Few-shot Bioacoustic Event Detection

Few-shot bio-acoustic detection Few-shot learning is a highly promising paradigm for sound event detection. It is also an extremely good fit to the ne

Haohe Liu (刘濠赫) 15 Jul 26, 2022