Repo for our ECCV 2022 paper on "Paint2Pix: Interactive Painting based Progressive Image Synthesis and Editing"

Overview

Paint2Pix: Interactive Painting based Progressive Image Synthesis and Editing (ECCV 2022)

Controllable image synthesis with user scribbles is a topic of keen interest in the computer vision community. In this paper, for the first time we study the problem of photorealistic image synthesis from incomplete and primitive human paintings. In particular, we propose a novel approach paint2pix, which learns to predict (and adapt) “what a user wants to draw” from rudimentary brushstroke inputs, by learning a mapping from the manifold of incomplete human paintings to their realistic renderings. When used in conjunction with recent works in autonomous painting agents, we show that paint2pix can be used for progressive image synthesis from scratch. During this process, paint2pix allows a novice user to progressively synthesize the desired image output, while requiring just few coarse user scribbles to accurately steer the trajectory of the synthesis process. Furthermore, we find that our approach also forms a surprisingly convenient approach for real image editing, and allows the user to perform a diverse range of custom fine-grained edits through the addition of only a few well-placed brushstrokes.


We propose paint2pix which helps the user directly express his/her ideas in visual form by learning to predict user-intention from a few rudimentary brushstrokes. The proposed approach can be used for (a) synthesizing a desired image output directly from scratch wherein it allows the user to control the overall synthesis trajectory using just few coarse brushstrokes (blue arrows) at key points, or, (b) performing a diverse range of custom edits directly on real image inputs.

Description

Official implementation of our Paint2pix paper with streamlit demo. By using autonomous painting agents as a proxy for the human painting process, Paint2pix learns to predict user-intention ("what a user wants to draw") from fairly rudimentary paintings and user-scribbles.

Getting Started

Prerequisites

  • Linux or macOS
  • NVIDIA GPU + CUDA CuDNN (CPU may be possible with some modifications, but is not inherently supported)
  • Python 3
  • Tested on Ubuntu 20.04, Nvidia RTX 3090 and CUDA 11.5

Installation

  • Dependencies:
    We recommend running this repository using Anaconda. All dependencies for defining the environment are provided in environment/paint2pix_env.yaml.

Pretrained Models

Please download the following pretrained models essential for running the provided demo.

Paint2pix models

Path Description
Canvas Encoder - ReStyle Paint2pix Canvas Encoder trained with a ReStyle architecture.
Identity Encoder - ReStyle Paint2pix Identity Encoder trained with a ReStyle architecture.
StyleGAN - Watercolor Painting StyleGAN decoder network trained to generate watercolor paintings. Used for artistic content generation with paint2pix.
IR-SE50 Model Pretrained IR-SE50 model taken from TreB1eN for use in ID loss and id-encoder training.

Please download and save the above models to the directory pretrained_models.

Using the Demo

We provide a streamlit-drawble canvas based demo for trying out different features of the Paint2pix model. To start the demo use,

CUDA_VISIBLE_DEVICES=2 streamlit run demo.py --server.port 6009

The demo can then be accessed on the local machine or ssh client via localhost.

The demo has been divided into 3 convenient sections:

  1. Real Image Editing: Allows the user to edit real images using coarse user scribbles
  2. Progressive Image Synthesis: Start from an empty canvas and design your desired image output using just coarse scribbles.
  3. Artistic Content Generation: Unleash your inner artist! create highly artistic portraits using just coarse scribbles.

Example Results

Progressive Image Synthesis


Paint2pix for progressive image synthesis

Real Image Editing


Paint2pix for achieving diverse custom real-image edits

Artistic Content Generation


Paint2pix for generating highly artistic content using coarse scribbles

Acknowledgments

This code borrows heavily from pixel2style2pixel, encoder4editing and restyle-encoder.

Citation

If you use this code for your research, please cite the following works:

@inproceedings{singh2022paint2pix,
  title={Paint2Pix: Interactive Painting based Progressive
        Image Synthesis and Editing},
  author={Singh, Jaskirat and Zheng, Liang and Smith, Cameron and Echevarria, Jose},
  booktitle={European conference on computer vision},
  year={2022},
  organization={Springer}
}
@inproceedings{singh2022intelli,
  title={Intelli-Paint: Towards Developing Human-like Painting Agents},
  author={Singh, Jaskirat and Smith, Cameron and Echevarria, Jose and Zheng, Liang},
  booktitle={European conference on computer vision},
  year={2022},
  organization={Springer}
}
You might also like...

The official repo for ECCV'22 paper: Pose for Everything: Towards Category-Agnostic Pose Estimation

The official repo for ECCV'22 paper: Pose for Everything: Towards Category-Agnostic Pose Estimation

Pose-for-Everything (ECCV'2022 Oral) Introduction Official code repository for the paper: Pose for Everything: Towards Category-Agnostic Pose Estimati

Nov 27, 2022

This Repo is created for Hacktober Fest 2022. All you have to do is follow the steps in the Readme of this repo, perform the tasks and create the pull request. Leave the rest upon me.

Hack-this-October-2k22- This Repo is created for Hacktober Fest 2021. All you have to do is follow the steps in the Readme of this repo, perform the t

Oct 18, 2022

Next-generation Video instance recognition framework on top of Detectron2 which supports SeqFormer(ECCV Oral) and IDOL(ECCV Oral))

Next-generation Video instance recognition framework on top of Detectron2 which supports SeqFormer(ECCV Oral) and IDOL(ECCV Oral))

VNext: VNext is a Next-generation Video instance recognition framework on top of Detectron2. Currently it provides advanced online and offline video i

Nov 18, 2022

Official Pytorch implementation of the ECCV 2022 paper "GIPSO: Geometrically Informed Propagation for Online Adaptation in 3D LiDAR Segmentation"

Official Pytorch implementation of the ECCV 2022 paper

GIPSO: Geometrically Informed Propagation for Online Adaptation in 3D LiDAR Segmentation [ECCV2022] The official implementation of our work "GIPSO: Ge

Nov 19, 2022

Official PyTorch implementation of the ECCV 2022 paper "CoSMix: Compositional Semantic Mix for Domain Adaptation in 3D LiDAR Segmentation"

Official PyTorch implementation of the ECCV 2022 paper

CoSMix: Compositional Semantic Mix for Domain Adaptation in 3D LiDAR Segmentation [ECCV2022] The official implementation of our work "CoSMix: Composit

Nov 19, 2022

Repository for ECCV 2022 paper "Learning Temporal Consistency for Source-Free Video Domain Adaptation"

Repository for ECCV 2022 paper

Attentive Temporal Consistent Network (ATCoN) This repository is the code for the ECCV 2022 paper "Learning Temporal Consistency for Source-Free Video

Nov 15, 2022

PyTorch code for the paper Class-incremental Novel Class Discovery (ECCV 2022)

PyTorch code for the paper Class-incremental Novel Class Discovery (ECCV 2022)

Class-incremental Novel Class Discovery (ECCV2022) This Github repository presents the PyTorch implementation for the paper Class-incremental Novel Cl

Nov 17, 2022

Code for ECCV 2022 paper "Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsing"

Code for ECCV 2022 paper

Joint-Modal Label Denoising for Weakly-Supervised Audio-Visual Video Parsing Haoyue Cheng, Zhaoyang Liu, Hang Zhou, Chen Qian, Wayne Wu and Limin Wang

Nov 25, 2022

Pytorch implementation of paper "DynaST: Dynamic Sparse Transformer for Exemplar-Guided Image Generation", ECCV 2022.

Pytorch implementation of paper

DynaST This is the pytorch implementation of the following ECCV 2022 paper: DynaST: Dynamic Sparse Transformer for Exemplar-Guided Image Generation So

Nov 24, 2022
Owner
Jaskirat Singh
Postgraduate student at ANU. I am working on research problems in RL and Computer Vision.
Jaskirat Singh
[ECCV 2022] Official pytorch implementation of "mc-BEiT: Multi-choice Discretization for Image BERT Pre-training" in European Conference on Computer Vision (ECCV) 2022.

mc-BEiT: Multi-choice Discretization for Image BERT Pre-training Official pytorch implementation of "mc-BEiT: Multi-choice Discretization for Image BE

lixiaotong 15 Nov 15, 2022
This repo contains the official implementation of ECCV 2022 paper "What to Hide from Your Students: Attention-Guided Masked Image Modeling"

What to Hide from Your Students: Attention-Guided Masked Image Modeling PyTorch implementation and pretrained models for AttMask. [arXiv] Pretrained m

Ioannis Kakogeorgiou 24 Nov 15, 2022
[ECCV 2022] The official repo for the paper "Poseur: Direct Human Pose Regression with Transformers".

Poseur: Direct Human Pose Regression with Transformers Poseur: Direct Human Pose Regression with Transformers, Weian Mao*, Yongtao Ge*, Chunhua Shen,

Advanced Intelligent Machines (AIM) 117 Nov 22, 2022
Official implementation for our ECCV 2022 paper "DecoupleNet: Decoupled Network for Domain Adaptive Semantic Segmentation"

DecoupleNet Official implementation for our ECCV 2022 paper "DecoupleNet: Decoupled Network for Domain Adaptive Semantic Segmentation" [arXiv] Get Sta

DV Lab 23 Nov 22, 2022
Official implementation of our ECCV 2022 paper, "UnrealEgo: A New Dataset for Robust Egocentric 3D Human Motion Capture"

UnrealEgo: A New Dataset for Robust Egocentric 3D Human Motion Capture (ECCV2022) Official PyTorch implementation of our ECCV 2022 paper, "UnrealEgo:

Hiroyasu Akada 37 Nov 6, 2022
This repo implements our paper, "Efficient Neural Neighborhood Search for Pickup and Delivery Problems", which has been accepted as short oral at IJCAI 2022.

PDP-N2S N2S is a learning based improvement framework for solving the pickup and delivery problems, a.k.a. PDPs (e.g., PDTSP and PDTSP-LIFO). It explo

Yi-Ning Ma (马一宁) 19 Nov 16, 2022
Official repo for FEAR: Fast, Efficient, Accurate and Robust Visual Tracker (ECCV 2022)

FEAR: Fast, Efficient, Accurate and Robust Visual Tracker This is an official repository for the paper FEAR: Fast, Efficient, Accurate and Robust Visu

Piñata Farms 105 Nov 25, 2022
This repo is official PyTorch implementation of 3D Clothed Human Reconstruction in the Wild (ECCV 2022).

3D Clothed Human Reconstruction in the Wild (ClothWild codes) 3D Clothed Human Reconstruction in the Wild, Gyeongsik Moon, Hyeongjin Nam, Takaaki Shir

Hyeongjin Nam 119 Nov 22, 2022
This repo equips the official CLIFF [ECCV 2022 Oral] with better detector, better tracker. Support multi-person, motion interpolation and smooth.

CLIFF [ECCV 2022 Oral] Introduction This repo is highly built on the official CLIFF and contains an inference demo, and further adds accurate detector

Haofan Wang 26 Nov 22, 2022
This repository contains the implementation for our work "Learning Topological Interactions for Multi-Class Medical Image Segmentation", accepted to ECCV 2022 (Oral)

TopoInteraction This repository contains the implementation for our work "Learning Topological Interactions for Multi-Class Medical Image Segmentation

TopoXLab 40 Nov 16, 2022