MEDFAIR: Benchmarking Fairness for Medical Imaging

Related tags

Admin Panels MEDFAIR
Overview

MEDFAIR: Benchmarking Fairness for Medical Imaging

MEDFAIR is a fairness benchmarking suite for medical imaging (paper). We are actively updating this repo and will incorporate more datasets and algorithms in the future. Contributions are warmly welcomed! Visit our website.

Documentation

A detailed documentation can be found here.

Quick Start

Installation

Python >= 3.8+ and Pytorch >=1.10 are required for running the code. Other necessary packages are listed in environment.yml.

Installation via conda:

cd MEDFAIR/
conda env create -n fair_benchmark -f environment.yml
conda activate fair_benchmark

Dataset

Due to the data use agreements, we cannot directly share the download link. Please register and download datasets using the links from the table below:

Dataset Access
CheXpert Original data: https://stanfordmlgroup.github.io/competitions/chexpert/
Demographic data: https://stanfordaimi.azurewebsites.net/datasets/192ada7c-4d43-466e-b8bb-b81992bb80cf
MIMIC-CXR https://physionet.org/content/mimic-cxr-jpg/2.0.0/
PAPILA https://www.nature.com/articles/s41597-022-01388-1#Sec6
HAM10000 https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/DBW86T
OCT https://people.duke.edu/~sf59/RPEDC_Ophth_2013_dataset.htm
Fitzpatrick17k https://github.com/mattgroh/fitzpatrick17k
COVID-CT-MD https://doi.org/10.6084/m9.figshare.12991592
ADNI 1.5T/3T https://ida.loni.usc.edu/login.jsp?project=ADNI

Data Preprocessing

See preprocessing/HAM10000.ipynb for an simple example of how to preprocess the data into desired format. Basically, it contains 3 steps:

  1. Preprocess metadata.
  2. Split to train/val/test set
  3. Save images into pickle files (optional -- we usually do this for 2D images instead of 3D images, as data IO is not the bottleneck for training 3D images).

After preprocessing, specify the paths of the metadata and pickle files in configs/datasets.json.

Run a single experiment

python main.py --experiment [experiment] --experiment_name [experiment_name] --dataset_name [dataset_name] \
     --backbone [backbone] --total_epochs [total_epochs] --sensitive_name [sensitive_name] \
     --batch_size [batch_size] --lr [lr] --sens_classes [sens_classes]  --val_strategy [val_strategy] \
     --output_dim [output_dim] --num_classes [num_classes]

See parse_args.py for more options.

Run a grid search on a Slurm cluster

python sweep/train-sweep/sweep_batch.py

Model selection and Results analysis

See notebooks/results_analysis.ipynb for a step by step example.

Citation

Please consider cite our paper if you find this repo useful.

@misc{2210.01725,
Author = {Yongshuo Zong and Yongxin Yang and Timothy Hospedales},
Title = {MEDFAIR: Benchmarking Fairness for Medical Imaging},
Year = {2022},
Eprint = {arXiv:2210.01725},
}

Acknowledgement

MEDFAIR adapts implementations from many repos (check here for the original implementation of the algorithms), as well as many other codes. Many thanks!

You might also like...

Benchmarking end-to-end SAT solvers.

SAT Benchmark Introduction SAT Benchmark (satb) is a PyTorch implementation of a collection of published end-to-end SAT solvers with the ability to re

Nov 20, 2022

A simple benchmarking SEO tool written in Python.

A simple benchmarking SEO tool written in Python.

SEO-Project A simple benchmarking SEO tool written in Python. The program extracts and compares the values of the sites you enter for you (the data is

Nov 18, 2022

Benchmarking for dot-accessible dict packages in python

dotdict-bench Benchmarking for dot-accessible dict packages in python More test ideas? Submit an issue! Package Information As of 2022-09-21 23:11:19.

Sep 22, 2022

Simple traffic generator for benchmarking your FT8 hardware/software

Simple traffic generator for benchmarking your FT8 hardware/software

ft8_benchmark -- A simple traffic generator for benchmarking your FT8 hardware/software --WIP-- Work in progress, this proto needs some love... TL;DR

Sep 24, 2022

Accelerated, Python-only, single-cell integration benchmarking metrics

scib-metrics Accelerated and Python-only metrics for benchmarking single-cell integration outputs. This package contains implementations of metrics fo

Nov 14, 2022

Benchmarking Attention Mechanism in Vision Transformers.

Vision Transformer Attention Benchmark This repo is a collection of attention mechanisms in vision Transformers. Beside the re-implementation, it prov

Nov 14, 2022

HerosNet: Hyperspectral Explicable Reconstruction and Optimal Sampling Deep Network for Snapshot Compressive Imaging (CVPR2022)

HerosNet: Hyperspectral Explicable Reconstruction and Optimal Sampling Deep Network for Snapshot Compressive Imaging (CVPR2022)

HerosNet: Hyperspectral Explicable Reconstruction and Optimal Sampling Deep Network for Snapshot Compressive Imaging (CVPR 2022) Xuanyu Zhang, Yongbin

Sep 19, 2022

3D Printed Invisible AR Markers and Barcodes Using Low-Cost Near-Infrared Imaging

3D Printed Invisible AR Markers and Barcodes Using Low-Cost Near-Infrared Imaging

InfraredTags: Invisible AR Markers and Barcodes In this tutorial, we show how InfraredTags can be fabricated and decoded using low-cost, infrared-base

Sep 7, 2022

Normalising flows for Inverse Problems is a framework providing all means for data-driven solutions to inverse (imaging) problems.

NF4IP Normalising Flows for Inverse Problems (NF4IP) is a Python framework that provides all means for usage of normalising flows in solving inverse p

Sep 26, 2022
Comments
  • Multiclass Prediction

    Multiclass Prediction

    Hello, really impressive work. and how can I use this framework for multiclass classification tasks? For example, using the original label of HAM10000 (7 classes) as target task, rather than spliting the dataset into benign and maglinant?

    Thank you!

    opened by XuZikang 2
Owner
Yongshuo Zong
PhD student at the University of Edinburgh.
Yongshuo Zong
Medi-BERT is a convenient, accurate and efficient improved-BERT model that specifically solves the problem of medical text classification , trained with an augmented-version of the Chinese medical triaging dataset.

Medi-BERT This repository contains the code for our National Innovation Project, which focuses on the problem of how to accurately triage the patients

Qianchen Chen (陈芊辰) 2 Oct 16, 2022
FairGrad, is an easy to use general purpose approach to enforce fairness for gradient descent based methods.

FairGrad: Fairness Aware Gradient Descent FairGrad, is an easy to use general purpose approach to enforce fairness for gradient descent based methods.

Gaurav Maheshwari 10 Nov 7, 2022
Python JSON benchmarking and correectness.

json_benchmark This repository contains benchmarks for Python JSON readers & writers. What's the fastest Python JSON parser? Let's find out. To run th

Tyler Kennedy 6 Nov 26, 2022
Home of the PaRoutes framework for benchmarking multi-step retrosynthesis predictions.

PaRoutes is a framework for benchmarking multi-step retrosynthesis methods, i.e. route predictions. It provides: A curated reaction dataset for buildi

AstraZeneca - Molecular AI 45 Nov 25, 2022
[ICML'22] Benchmarking and Analyzing Point Cloud Robustness under Corruptions

Benchmarking and Analyzing Point Cloud Robustness under Corruptions Jiawei Ren, Lingdong Kong, Liang Pan, Ziwei Liu S-Lab, Nanyang Technological Unive

Lingdong Kong 76 Nov 18, 2022
Multi-Modal Lidar Dataset for Benchmarking General-Purpose Localization and Mapping Algorithms

Multi-Modal Lidar Dataset for Benchmarking General-Purpose Localization and Mapping Algorithms (Left) Front view of the multi-modal data acquisition s

TIERS 79 Nov 2, 2022
Benchmarking toolkit for patch-based histopathology image classification.

ChampKit ChampKit: Comprehensive Histopathology Assessment of Model Predictions toolKit. A benchmarking toolkit for patch-based image classification i

Jakub Kaczmarzyk 18 Nov 3, 2022
Source code for the ECCV 2022 paper "Benchmarking Localization and Mapping for Augmented Reality".

The LaMAR Benchmark for Localization and Mapping in Augmented Reality This repository hosts the source code for our upcoming ECCV 2022 paper: LaMAR: B

Microsoft 205 Nov 24, 2022
Benchmarking Panoptic Scene Graph Generation (PSG), ECCV'22

Panoptic Scene Graph Generation Panoptic Scene Graph Generation Jingkang Yang, Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, Ziwei Liu S-Lab, Nany

Jingkang Yang 200 Nov 25, 2022
A tool for benchmarking image generation models.

??️ Dream Bench ?? What does it do? dream_bench provides a simplified interface for benchmarking your image-generation models. This repository also ho

zion 16 Nov 25, 2022