[IJCAI2022 Oral] Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration

Overview

UMFusion

LICENSE Python PyTorch

Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration [IJCAI2022 Oral Presentation]

By Di Wang, Jinyuan Liu, Xin Fan, and Risheng Liu

Updates

[2022-06-21] The CPSTN is available!
[2022-05-30] The Chinese translation of our paper is available, please enjoy it! [中译版本]
[2022-05-25] Our paper is available online! [arXiv version]

Requirements

  • CUDA 10.1
  • Python 3.6 (or later)
  • Pytorch 1.6.0
  • Torchvision 0.7.0
  • OpenCV 3.4
  • Kornia 0.5.11

Data preparation

  1. You can obtain deformation infrared images for training/testing process by
       cd ./data
       python get_test_data.py

In 'Trainer/train_reg.py', deformable infrared images are generated in real time by default during training.

  1. You can obtain self-visual saliency maps for training IVIF fusion by
       cd ./data
       python get_svs_map.py

Get start

  1. You can use the pseudo infrared images [link code: qqyj] generated by our CPSTN to train/test the registration process:

       cd ./Trainer
       python train_reg.py
    
       cd ./Test
       python test_reg.py
  2. If you want to generate pseudo-infrared images using our CPSTN for other datasets, you can directly run following commands:

    ## testing
       cd ./CPSTN
       python test.py --dataroot datasets/rgb2ir/RoadScene/testA --name rgb2ir_paired_Road_edge_pretrained --model test --no_dropout --preprocess none
    
    ## training
       cd ./CPSTN
       python train.py --dataroot ./datasets/rgb2ir/RoadScene --name rgb2ir_paired_Road_edge --model cycle_gan --dataset_mode unaligned

The training and testing data of our CPSTN can be downloaded from: datasets (code: u386)

Please download the pretrained model (code: i9ju) of CPSTN and put it into folder './CPSTN/checkpoints/pretrained/'

  1. If you tend to train Registration and Fusion processes separately, You can run following commands:

       cd ./Trainer
       python train_reg.py
    
       cd ./Trainer
       python train_fuse.py

The corresponding test code 'test_reg.py' and 'test_fuse.py' can be found in 'Test' folder.

  1. If you tend to train Registration and Fusion processes jointly, You can run following command:
        cd ./Trainer
        python train_reg_fusion.py

The corresponding test code 'test_reg_fusion.py' can be found in 'Test' folder.

Dataset

Please download the following datasets:

Experimental Results

Please download the pseudo infrared images generated by our CPSTN:

Please download the registered infrared images by our UMF:

Please download the fused images by our UMF:

Citation

@InProceedings{Wang_2022_IJCAI,
	author = {Di, Wang and Jinyuan, Liu and Xin, Fan and Risheng Liu},
	title = {Unsupervised Misaligned Infrared and Visible Image Fusion via Cross-Modality Image Generation and Registration},
	booktitle = {International Joint Conference on Artificial Intelligence (IJCAI)},
	year = {2022}
}
You might also like...

Featurization of image and question, Feature fusion, Answer generation for Multimodal System

Visual Question Answering (VQA)- MultiModular Architecture Featurization of image and question, Feature fusion, Answer generation for Multimodal Syste

Jul 3, 2022

UMT is a unified and flexible framework which can handle different input modality combinations, and output video moment retrieval and/or highlight detection results.

Unified Multi-modal Transformers This repository maintains the official implementation of the paper UMT: Unified Multi-modal Transformers for Joint Vi

Sep 13, 2022

The code for the paper "Contrastive Quantization with Code Memory for Unsupervised Image Retrieval" (AAAI'22, Oral).

The code for the paper

MeCoQ: Contrastive Quantization with Code Memory for Unsupervised Image Retrieval [toc] 1. Introduction This repository provides the code for our pape

Aug 1, 2022

[ACM MM 2022] Modality-aware Contrastive Instance Learning with Self-Distillation for Weakly-Supervised Audio-Visual Violence Detection

MACIL_SD [ACM MM 2022] Modality-aware Contrastive Instance Learning with Self-Distillation for Weakly-Supervised Audio-Visual Violence Detection Jiash

Sep 1, 2022

Boosting 3D Object Detection via Object-Focused Image Fusion

Boosting 3D Object Detection via Object-Focused Image Fusion

Boosting 3D Object Detection via Object-Focused Image Fusion This is a MMDetection3D implementation of the paper Yang et al, "Boosting 3D Object Detec

Sep 19, 2022

The official implementation of the ECCV 2022 Oral paper: RealFlow: EM-based Realistic Optical Flow Dataset Generation from Videos

The official implementation of the ECCV 2022 Oral paper: RealFlow: EM-based Realistic Optical Flow Dataset Generation from Videos

[ECCV 2022 Oral] RealFlow: EM-based Realistic Optical Flow Dataset Generation from Videos (Paper) Yunhui Han1, Kunming Luo2, Ao Luo2, Jiangyu Liu2, Ha

Sep 16, 2022

This is the official Pytorch implementation of "Affine Medical Image Registration with Coarse-to-Fine Vision Transformer" (CVPR 2022), written by Tony C. W. Mok and Albert C. S. Chung.

This is the official Pytorch implementation of

Affine Medical Image Registration with Coarse-to-Fine Vision Transformer (C2FViT) This is the official Pytorch implementation of "Affine Medical Image

Sep 16, 2022

SymTrans: A symmetric Transformer-based model for image registration.

SymTrans: A symmetric Transformer-based model for image registration.

SymTrans: Symmetric Transformer-based model for image registration Paper: Symmetric Transformer-based Network for Unsupervised Image Registration Plea

Aug 12, 2022

Official implementation of Improving Cross-lingual Fact Checking with Cross-lingual Retrieval (COLING'22)

Official implementation of Improving Cross-lingual Fact Checking with Cross-lingual Retrieval (COLING'22)

CONCRETE Source code for the COLING'22 paper: Improving Cross-lingual Fact Checking with Cross-lingual Retrieval. We propose a fact checking framework

Sep 18, 2022
Comments
  • Qusetion about raw dataset

    Qusetion about raw dataset

    Thank you for great your work!

    I have a question about your datasets.

    I get the RoadScene dataset from here, and then there are two types of thermal images(cropinfrared and infrared).

    What is correct infrared image for making the test images using this code..?

    To my understanding, I use the corp infrared images for making the test dataset. Is it right?

    Thank you. I'm waiting your reply.

    opened by socome 5
  • KeyError: 'spatial_transform.grid'

    KeyError: 'spatial_transform.grid'

    When I use test_reg.py and pretrained model reg_0280.pth to test the registration process,I got a keyerror: 'spatial_transform.grid'. Can you help me? The more information about this error is as following:

    ===> loading trained model '../reg_0280.pth' Traceback (most recent call last): File "test_reg.py", line 187, in main(args) File "test_reg.py", line 61, in main net.load_state_dict(model_state_dict) File "/home/wu/UMF/Test/../models/deformable_net.py", line 74, in load_state_dict state_dict.pop('spatial_transform.grid') KeyError: 'spatial_transform.grid'

    opened by Yixing-Wu 2
  • 可见光和红外的原始分辨率必须要保持一致吗?

    可见光和红外的原始分辨率必须要保持一致吗?

    作者您好, 在我的数据集上可见光和红外图像的原始分辨率不一致,于是我直接resize之后开始训练,并不像您提供的FILR数据集每对图像分辨率保持一致。于是我的数据集在配准测试时,ir_reg中输出的配准结果边缘出现了扭曲和黑边,我认为是关键点不匹配导致的。 我想请教一下这个问题是什么原因,怎么去解决。 期待您的回复! 谢谢!

    opened by AbandonedWarlord 1
  • return_transform的问题

    return_transform的问题

    如果return_transform不为空,就会报错 可以帮帮孩子吗

    ValueError: return_transform is deprecated. Please access the transformation matrix with .transform_matrix. For chained matrices, please use AugmentationSequential.

    opened by scl1997 0
Owner
Di Wang
Love life, love scientific research!
Di Wang
Event-based Fusion for Motion Deblurring with Cross-modal Attention (ECCV'22 Oral)

Event-based Fusion for Motion Deblurring with Cross-modal Attention Lei Sun, Christos Sakaridis, Jingyun Liang, Qi Jiang, Kailun Yang, Peng Sun, Yaozu

Lei Sun 32 Sep 23, 2022
Next-generation Video instance recognition framework on top of Detectron2 which supports SeqFormer(ECCV Oral) and IDOL(ECCV Oral))

VNext: VNext is a Next-generation Video instance recognition framework on top of Detectron2. Currently it provides advanced online and offline video i

Junfeng Wu 413 Sep 19, 2022
IMFNet: Interpretable Multimodal Fusion for Point Cloud Registration

IMFNet: Interpretable Multimodal Fusion for Point Cloud Registration, 2022 This repository is the implementation of IMFNet: Interpretable Multimodal F

Xiaoshui Huang 3 Aug 26, 2022
[ECCV2022 Oral] Registration based Few-Shot Anomaly Detection

Registration based Few-Shot Anomaly Detection This is an official implementation of “Registration based Few-Shot Anomaly Detection” (RegAD) with PyTor

null 134 Sep 19, 2022
This is an official implementation of the CVPR2022 paper "Blind2Unblind: Self-Supervised Image Denoising with Visible Blind Spots".

Blind2Unblind: Self-Supervised Image Denoising with Visible Blind Spots Blind2Unblind Citing Blind2Unblind @inproceedings{wang2022blind2unblind, tit

demonsjin 37 Sep 5, 2022
Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training

CCLM: Towards Unified Cross-Lingual Cross-Modal Pre-training **Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training. Y

Yan Zeng 45 Sep 18, 2022
3D Printed Invisible AR Markers and Barcodes Using Low-Cost Near-Infrared Imaging

InfraredTags: Invisible AR Markers and Barcodes In this tutorial, we show how InfraredTags can be fabricated and decoded using low-cost, infrared-base

dogadogan 16 Sep 7, 2022
Hacking the PixMob infrared protocol to enable control of PixMob wristbands at home.

PixMob IR Reverse Engineering Project Hacking the PixMob infrared protocol to enable control of PixMob wristbands at home. A huge thanks to Zach Resme

Dani Weidman 61 Sep 20, 2022
[CVPR 2022 Oral] CamLiFlow: Bidirectional Camera-LiDAR Fusion for Joint Optical Flow and Scene Flow Estimation

CamLiFlow This is the official PyTorch implementation for paper CamLiFlow: Bidirectional Camera-LiDAR Fusion for Joint Optical Flow and Scene Flow Est

Multimedia Computing Group, Nanjing University 108 Sep 5, 2022
This script make a live sketch of the objects which are visible in webcam.

Live-Webcam-Sketch-using-Python This script makes a live sketch of the objects which are visible in webcam. Requirements:- Install CV2 module: pip ins

Nikit Singh Kanyal 3 Aug 17, 2022