Benchmarking toolkit for patch-based histopathology image classification.

Overview

ChampKit

Overview of ChampKit ChampKit: Comprehensive Histopathology Assessment of Model Predictions toolKit.

A benchmarking toolkit for patch-based image classification in histopathology.

This toolkit is meant to be fully reproducible. Please start by installing the software environment. Then download the datasets.

If you face any problems, please submit a new issue!

Reproducing our manuscript

To reproduce our submission to the NeurIPS 2022 Datasets & Benchmarks Track (preprint), please follow the steps below. Ideally, we would include a script that reproduces the paper from start to finish, but unfortunately there are some steps that must be done manually.

  1. Get the ChampKit code.
    git clone https://github.com/kaczmarj/champkit
    cd champkit
    git checkout v0.2.0
  2. Install software dependencies. See Software environment for details.
    conda env create -f environment.yml -n champkit
    
  3. Install datasets. Almost all of the datasets can be downloaded automatically. The Task 6 dataset (MHIST) needs to be downloaded manually. Please go to https://bmirds.github.io/MHIST/#accessing-dataset and complete the registration form. You will receive an automated email with download links. Please download the files images.zip and annotations.csv and move them to data/task6_precancer_vs_benign_polyps/. After you have done this, run the following script to take care of the rest. This script will download approximately 75 GB of data.
    bash setup.sh
    See Download and setup datasets for details.
  4. Run experiments. We use Weights & Biases (W&B) sweeps to run and log the experiments. See sweeps/sweep_neurips_db.yml for the configuration. Please create an account on https://wandb.ai/ if you do not have one, and login on your machine with python -m wandb login (activate the champkit conda environment first).
    • Initialize the W&B sweep:
      python -m wandb sweep sweeps/sweep_neurips_db.yml
    • The command above will output a string like USER/PROJECT/SWEEPID. Use this in the next command.
    • Start an agent:
      CUDA_VISIBLE_DEVICES=0 python -m wandb agent USER/PROJECT/SWEEPID
      There are 42 training runs in this sweep. One can start multiple agents to go through the experiments faster. Each agent will poll W&B for the next experiment to run. If you have more agents running, the experiments will be done sooner. Please see Running multiple experiments for more information.
    • The sweep requires at least 12 GB of GPU memory per agent. For the manuscript, we used Quadro RTX 8000 GPUs with 48 GB of video memory each.
  5. Evaluate trained models. The following command will evaluate the best model for each run in the sweep (best = lowest validation loss). Results will be printed as they are calculated, and all results are saved to a CSV. The values in this CSV should be very similar to the values reported in the manuscript.
    CUDA_VISIBLE_DEVICES=0 python run_all_evaluations.py
    

Installation

ChampKit requires software and datasets. The next two subsections explain how to install these. If you experience any difficulty, please submit a new issue.

Software environment

We use conda to manage Python environments. If you don't have conda, we recommend installing it with Miniforge. Then, create the conda environment:

conda env create -f environment.yml -n champkit
conda activate champkit

We have tested the software on several x86-64 Linux systems, including Debian bookworm/sid, Red Hat 8.5, and Ubuntu 20.04 We have not tested this on Windows, macOS, or non-x86-64 architectures. While we did not test the Windows Subsystem for Linux (WSL), we anticipate that ChampKit will work in that environment (and we encourage you to try!). ChampKit is tested with CUDA version 11.3, which requires NVIDIA driver >=450.80.02.

Download and setup datasets

ChampKit includes scripts that download and process all included datasets, except MHIST. To access the MHIST dataset, please complete the form at https://bmirds.github.io/MHIST/. You will receive an email with links to the dataset (check spam folder if you don't see it). Please download images.zip and annotations.csv and place them in data/task6_precancer_vs_benign_polyps/.

After the MHIST dataset is downloaded, please run the following code to download and setup all of the datasets.

This can take a long time (2+ hours). One can run the setup.sh scripts in parallel, and that might speed things up.

conda activate champkit
bash data/task1_tumor_notumor/setup.sh
bash data/task2_tils/setup.sh
bash data/task3_msi_crc_ffpe/setup.sh
bash data/task4_msi_crc_frozen/setup.sh
bash data/task5_msi_stad_ffpe/setup.sh
bash data/task6_precancer_vs_benign_polyps/setup.sh

These are the different tasks and datasets. Please cite the papers for the datasets you use (Citations).

Running multiple experiments

Weights & Biases (W&B) tracks experiments and facilitates running hyperparameter searches via sweeps. We use sweeps to train multiple models on multiple datasets, with and without pretraining. W&B requires a (free) account (https://wandb.ai/). Please make an account, and then login on the machine where you will be running the sweep.

conda activate champkit
python -m wandb login

After logging in, initialize the sweep with

python -m wandb sweep sweeps/sweep_neurips_db.yml

This will print USER/PROJECT/SWEEPID, which you will use in the next command. Now that we ran python -m wandb sweep, wandb is waiting us to run "agents" that will run through all of the combinations in the sweep configuration. When an agent is ready to run a new job, it will ask W&B which combination to run next. You can run multiple agents at once.

The following command will run one agent for this sweep using one GPU (with ID=0).

CUDA_VISIBLE_DEVICES=0 python -m wandb agent USER/PROJECT/SWEEPID

In practice, I (@kaczmarj) like to run multiple agents on multiple GPUs in screen sessions. The screen command runs a program in the background and will keep running it if you log out.

for device in 0 1 2; do
    CUDA_VISIBLE_DEVICES=$device screen -dmS champkit-$device python -m wandb agent USER/PROJECT/SWEEPID
done

Extending ChampKit

Using other datasets

ChampKit allows for integration with custom datasets that are framed as a single-task patch-level (binary or multi-class) classification. Custom datasets must be organized in an ImageNet-like directory structure, in which the top-level directory contains directories train, val, and test, and each of those contains sub-directories of the different classes (e.g., tumor-positive, tumor-negative), where the corresponding patch-level images are located. Images can be of any size and will be resized during training and evaluation – the size is configurable. Put the dataset in the data/ directory, and consider adding a classmap.txt file to indicate the order of the labels. In the example below, class 0 is tumor-negative, and class 1 is tumor-positive.

tumor-negative
tumor-positive

We would love to incorporate your dataset into ChampKit! Please submit an issue or email [email protected] if interested.

Using other models

We use the excellent timm package to have access to many models. Newly published models tend to be added to timm quickly, but there might be cases where you want to use a model that timm does not have. In those cases, please add a PyTorch implementation of your model to models.py. See that file for examples of custom models or custom pretrained weights. Be sure to register your model with timm with the decorator @timm.models.register_model. This makes it possible to use the model with timm.create_model(MODEL_NAME).

Citations

If you find ChampKit useful, please cite our preprint and the papers corresponding to any datasets you use.

@misc{kaczmarzyk2022evaluating,
      title={Evaluating histopathology transfer learning with ChampKit},
      author={Jakub R. Kaczmarzyk and Tahsin M. Kurc and Shahira Abousamra and Rajarsi Gupta and Joel H. Saltz and Peter K. Koo},
      year={2022},
      eprint={2206.06862},
      archivePrefix={arXiv},
      primaryClass={q-bio.QM},
      url={https://arxiv.org/abs/2206.06862}
}
Task 1 (tumor versus non-tumor classification)
@inproceedings{veeling2018rotation,
    title={Rotation equivariant CNNs for digital pathology},
    author={Veeling, Bastiaan S and Linmans, Jasper and Winkens, Jim and Cohen, Taco and Welling, Max},
    booktitle={International Conference on Medical Image Computing and Computer-Assisted Intervention},
    pages={210--218},
    year={2018},
    organization={Springer},
    url={https://doi.org/10.1007/978-3-030-00934-2_24}
}
@article{bejnordi2017diagnostic,
    title={Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer},
    author={Bejnordi, Babak Ehteshami and Veta, Mitko and Van Diest, Paul Johannes and Van Ginneken, Bram and Karssemeijer, Nico and Litjens, Geert and Van Der Laak, Jeroen AWM and Hermsen, Meyke and Manson, Quirine F and Balkenhol, Maschenka and others},
    journal={JAMA},
    volume={318},
    number={22},
    pages={2199--2210},
    year={2017},
    publisher={American Medical Association}
}
Task 2 (tumor-infiltrating lymphocyte detection)
@article{abousamra2022deep,
    author={Abousamra, Shahira and Gupta, Rajarsi and Hou, Le and Batiste, Rebecca and Zhao, Tianhao and Shankar, Anand and Rao, Arvind and Chen, Chao and Samaras, Dimitris and Kurc, Tahsin and Saltz, Joel},
    title={Deep Learning-Based Mapping of Tumor Infiltrating Lymphocytes in Whole Slide Images of 23 Types of Cancer},
    journal={Frontiers in Oncology},
    volume={11},
    year={2022},
    doi={10.3389/fonc.2021.806603},
    issn={2234-943X},
    url={https://doi.org/10.3389/fonc.2021.806603}
}
@article{saltz2018spatial,
    title={Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images},
    author={Saltz, Joel and Gupta, Rajarsi and Hou, Le and Kurc, Tahsin and Singh, Pankaj and Nguyen, Vu and Samaras, Dimitris and Shroyer, Kenneth R and Zhao, Tianhao and Batiste, Rebecca and Van Arnam, John},
    journal={Cell Reports},
    volume={23},
    number={1},
    pages={181--193},
    year={2018},
    publisher={Elsevier},
    url={https://doi.org/10.1016/j.celrep.2018.03.086},
    doi={10.1016/j.celrep.2018.03.086}
}
Tasks 3-5 (microsatellite instability detection)
@article{kather2019deep,
    title={Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer},
    author={Kather, Jakob Nikolas and Pearson, Alexander T and Halama, Niels and J{\"a}ger, Dirk and Krause, Jeremias and Loosen, Sven H and Marx, Alexander and Boor, Peter and Tacke, Frank and Neumann, Ulf Peter and others},
    journal="Nature Medicine",
    year="2019",
    month="Jul",
    day="01",
    volume="25",
    number="7",
    pages="1054--1056",
    issn="1546-170X",
    doi="10.1038/s41591-019-0462-y",
    url="https://doi.org/10.1038/s41591-019-0462-y"
}
Task 6 (precancerous versus benign colorectal polyp classification)
@inproceedings{wei2021petri,
    title={A Petri Dish for Histopathology Image Analysis},
    author={Wei, Jerry and Suriawinata, Arief and Ren, Bing and Liu, Xiaoying and Lisovsky, Mikhail and Vaickus, Louis and Brown, Charles and Baker, Michael and Tomita, Naofumi and Torresani, Lorenzo and Wei, Jason and Hassanpour, Saeed},
    booktitle={International Conference on Artificial Intelligence in Medicine},
    pages={11--24},
    year={2021},
    organization={Springer},
    url={https://doi.org/10.1007/978-3-030-77211-6_2},
    doi={10.1007/978-3-030-77211-6_2}
}

Frequently asked questions

  1. How do I get help?
    • If you have read through this document and still have questions, we encourage you to create a new issue. You could also email [email protected], but we would prefer to have these discussions be public. If you have a question, others likely have similar questions!
  2. How do I make a change to ChampKit?
    • Please use GitHub pull requests to propose changes.
  3. I want to add a dataset to ChampKit.
    • We are always on the hunt for more patch-based histopathology image classification datasets. If you use a new dataset, please let us know. If it is publicly available, we would love to incorporate it into ChampKit.
  4. How do I use a new dataset?
    • You can use ChampKit with any single-task, binary/multiclass image classification dataset. The dataset directories should be organized in ImageNet style. Please see Using other datasets for more information.
  5. How do I train and evaluate models?
    • Use train.py to train models. See python train.py --help for all options.
    • Use validate.py to evaluate models. See python validate.py --help for all options.
    • While this is not strictly required, we use Weights & Biases to run multiple experiments. Please see the sweeps in the sweeps/ directory for examples of configurations.
  6. I want to evaluate multiple training runs of the same model, but the result is always the same. Help!?
    • The train.py was written (originally by Ross Wightman) to reproduce training results when using the same seed. If you want to train multiple runs of a model, use different --seed values in train.py. The default is --seed=42.
  7. How do I benchmark transfer learning from my favorite self-supervised histology model?
    • ChampKit includes a self-supervised model from Ciga et al. (2022). If you want to use a different model, it will have to be added to ChampKit. In general, code will have to be added to models.py that downloads the weights and loads them into a PyTorch network. The architecture will have to be implemented if it does not exist in timm. Pull requests are welcome! We are happy to work with you to add models to ChampKit.
  8. Why do you want me to use python -m wandb instead of wandb?
    • Using python -m wandb ensures that you use the wandb command installed in the champkit python environment (assuming the champkit conda environment is activated). If you use wandb directly, it is possible that this points to a different python environment than champkit. For example, if one had previously installed wandb with pip install --user wandb, the wandb executable would be in ~/.local/bin/wandb and might be used instead of the wandb in the champkit environment.
    • In general, using python -m PROGRAM is better practice than using PROGRAM directly, because python -m PROGRAM guarantees we use PROGRAM installed in the current python environment. Using PROGRAM on its own relies on the $PATH variable, and it is possible that there exist multiple PROGRAM installations, one of which could appear in $PATH before the one we think we are using.

Uninstalling ChampKit

We hope you enjoyed using ChampKit. To uninstall ChampKit and all datasets, remove this directory and the champkit conda environment.

conda deactivate
conda env remove --name champkit
You might also like...

Code for ECCV 2022 paper PatchRD: Detail-Preserving Shape Completion by Learning Patch Retrieval and Deformation

Code for ECCV 2022 paper PatchRD: Detail-Preserving Shape Completion by Learning Patch Retrieval and Deformation

PatchRD Code for ECCV 2022 paper PatchRD: Detail-Preserving Shape Completion by Learning Patch Retrieval and Deformation. (PDF will be released soon!)

Nov 25, 2022

This Repository contains all the task and patch up files i have created while doing JPMorgan-Chase-Virtual-Internship by Forage.

JPMorgan-Chase-Virtual-Internship This Repository contains all the task and patch up files i have created while doing JPMorgan-Chase-Virtual-Internshi

Sep 1, 2022

[ECCV 2022] Patch Similarity Aware Data-Free Quantization for Vision Transformers

[ECCV 2022] Patch Similarity Aware Data-Free Quantization for Vision Transformers

Patch Similarity Aware Data-Free Quantization for Vision Transformers This repository contains the official PyTorch implementation for the ECCV 2022 p

Nov 18, 2022

Multi-label Classification Model for English Scientific Literature: develop a multi-label classification model for scientific research literature based on the given metadata (title and abstract) of scientific research literature and corresponding hierarchical labels from a domain-specific topic taxonomy.

Multi-label Classification Model for English Scientific Literature: develop a multi-label classification model for scientific research literature based on the given metadata (title and abstract) of scientific research literature and corresponding hierarchical labels from a domain-specific topic taxonomy.

Jul 14, 2022

A docker image for automatic image classification based on pytorch, pyzmq, fastapi

searchmap-docker-image-classification This tools allows to build an image classification deep learning model requirepents create the directory workspa

Aug 12, 2022

Python JSON benchmarking and correectness.

json_benchmark This repository contains benchmarks for Python JSON readers & writers. What's the fastest Python JSON parser? Let's find out. To run th

Nov 26, 2022

Home of the PaRoutes framework for benchmarking multi-step retrosynthesis predictions.

Home of the PaRoutes framework for benchmarking multi-step retrosynthesis predictions.

PaRoutes is a framework for benchmarking multi-step retrosynthesis methods, i.e. route predictions. It provides: A curated reaction dataset for buildi

Nov 25, 2022

[ICML'22] Benchmarking and Analyzing Point Cloud Robustness under Corruptions

[ICML'22] Benchmarking and Analyzing Point Cloud Robustness under Corruptions

Benchmarking and Analyzing Point Cloud Robustness under Corruptions Jiawei Ren, Lingdong Kong, Liang Pan, Ziwei Liu S-Lab, Nanyang Technological Unive

Nov 18, 2022

Multi-Modal Lidar Dataset for Benchmarking General-Purpose Localization and Mapping Algorithms

 Multi-Modal Lidar Dataset for Benchmarking General-Purpose Localization and Mapping Algorithms

Multi-Modal Lidar Dataset for Benchmarking General-Purpose Localization and Mapping Algorithms (Left) Front view of the multi-modal data acquisition s

Nov 2, 2022
Comments
  • add feature to sample from datasets

    add feature to sample from datasets

    Add a feature to sample a fraction of the dataset for training. This will allow evaluation of models on different dataset sizes.

    Related to cross validation but different.

    enhancement 
    opened by kaczmarj 0
  • add cross validation

    add cross validation

    Add support for cross validation in training/evaluating models.

    We can add k-fold cross validation and let the user choose the size of the training data.

    enhancement 
    opened by kaczmarj 0
Releases(v0.2.0)
  • v0.2.0(Jun 16, 2022)

    This release fixes a bug in which the files for task 1 were not included in this repository. While preparing our manuscript, we used a private GitHub repository, and after making the manuscript available, we moved the files from the private repository to this public one. There was a line in the .gitignore to ignore the entire task1 directory. This is fixed now.

    Source code(tar.gz)
    Source code(zip)
  • v0.1.0(Jun 14, 2022)

Owner
Jakub Kaczmarzyk
MD-PhD (MSTP) student interested in computer vision.
Jakub Kaczmarzyk
Multi-Scale Patch-Based Representation Learning for Image Anomaly Detection and Segmentation

Multi-Scale Patch-Based Representation Learning for Image Anomaly Detection and Segmentation (paper link, published in WACV 2022) Authors - Chin-Chia

Howeng 5 Nov 1, 2022
CI service to Patch recovery to enable fastbootd on recovery image of Samsung devices launched with Android 10 and above.

Patch-Recovery This CI service patches recovery images of Samsung to enable Fastbootd. Based on Phh's script How to use: Fork this repo. Upload your r

Johann 20 Nov 17, 2022
Adaptive Patch Exiting for Scalable Single Image Super-Resolution" (ECCV2022 Oral)

Adaptive Patch Exiting for Scalable Single Image Super-Resolution (ECCV2022 Oral) This repository is an official PyTorch implementation of the paper "

null 19 Nov 10, 2022
A tool for benchmarking image generation models.

??️ Dream Bench ?? What does it do? dream_bench provides a simplified interface for benchmarking your image-generation models. This repository also ho

zion 16 Nov 25, 2022
[ACM MM 2022] Patch-based Knowledge Distillation for Lifelong Person Re-Identification

PatchKD Code for ACM MM 2022 paper Patch-based Knowledge Distillation for Lifelong Person Re-Identification. Installation git clone https://github.com

Zhicheng Sun 2 Aug 7, 2022
Restoring Vision in Adverse Weather Conditions with Patch-Based Denoising Diffusion Models

This is the code repository of the following paper to train and perform inference with patch-based diffusion models for image restoration under adverse weather conditions.

Institute for Theoretical Computer Science, TU Graz 52 Nov 18, 2022
Decompiler-toolkit: a never-finished project aimed at providing a toolkit and language for writing decompilers

This is a never-finished project aimed at providing a toolkit and language for writing decompilers. This also serves as a high-level bridge for program analysis procedures leveraging ghidra_bridge as a Python API and analyzers for Ghidra's Clang/angr's recovered ASTs. I am using this as a basis for some other (closed source) research projects; random pull requests are welcome.

Maxwell Bland 1 Jul 9, 2022
Split your patch similarly to `git add -p` but supporting multiple buckets

split-patch.py This is git add -p on steroids for patches. Given a my.patch you can run ./split-patch.py my.patch You can choose in which bucket to p

null 102 Oct 6, 2022
Build, patch and install packages quickly

xpkg Automatically build, patch and install packages. Installation # clone the repo $ git clone https://github.com/synthels/xpkg.git # change your wo

synthels 3 Apr 29, 2022
Patch-Diffusion Code (AAAI2022)

Patch-Diffusion This is an official PyTorch implementation of "Patch Diffusion: A General Module for Face Manipulation Detection" in AAAI2022. Require

H 7 Nov 2, 2022