Numpy implementation of Logistic Regression (w/o explicit ML libraries)

Overview

Logistic-Regression-from-scratch

Numpy implementation of Logistic Regression (w/o explicit ML libraries, such as Pytorch or Tensorflow)

What is Logistic Regression

Logistic Function

  • also called as Sigmoid Function
  • $y$ = $1 \over 1+e^{-x}$
  • Output of logistic function is always in (0,1) ==> Logistic function satisfies the condition of Probability Density Function

Odds ( $\ne$ Probability)

  • defined as 'probability that the event will occur divied by the probability that the event will not occur

Mapping the linear equation f(x) to log odds

  • In Linear Regression, $f(x) = W^TX$
  • Let p = P(y=1|x)
  • Take exponents for both side:
  • Then, p (the probability of y=1 for x) is

Loss Function Formulation : E( $W$ )

  • Simply, use the loss function used in Linear Regression (Least Squares Error)

  • This gives non-convex function for w, which does not guarantee global minimum
  • This if we use Least Squares Error for Gradient Descent Method, it is highly likely that the function converges in one of local minimums
  • So we don't use Least Squares Error for Logistic Regression
  • Then, we derive Loss Function from Maximum Likelihood Estimation
  • Just like finding $\Theta$ that maximizes $\Sigma$ P(x| $\Theta$ )
  • Find $w$ that maximizes
  • Then, by applying log each side and combining sigma :
  • In the end, the Loss Function can be shown as

Computing Gradients (Computing Partial Derivatives) -details

Solve one by one using Chain Rule


Apply derivative of h (logistic function)


Apply derivative of f (linear equation)



Final Gradient Form



Python Code for Computing Gradient

        def compute_grad(self, x, y, weight, logit):
        """
        N : # of minibatch data
        D : # of features

        Inputs:
        x : (N, D), input data
        y : (N, ), label for each data
        weight : (D, 1), Weight matrix of classifier
        logit: (N, 1), logits for N data

        Returns:
        gradient of weight: (D, 1), Gradient of weight to be applied (dL/dW)

        Description:
        Given input, label, weight, logit, compute gradient of weight.
        """

        num_data, num_feat = x.shape
        
        y = np.expand_dims(y, axis=1)

        grad_weight = np.zeros_like(weight)

        score = np.dot(x,self.W)
        score = self._sigmoid(score)
        temp = (score -y).reshape(1,-1)
        grad_weight = (1/num_data)*np.dot(temp,x)
        grad_weight = grad_weight.reshape(-1,1)

        return grad_weight

Multinomail Logistic Regression

Use Softmax function instead of Logistic (Sigmoid) function

Sample Datase

Breast Cancer - for Logistic Regression (binary classes)

EMNIST - for SoftmaxClassifier (more than two classes)

Reference

SungKyunKwan University, College of Computing, SWE3050_41

You might also like...

My Logistic app project. Meant to be for company-internal use and my own portfolio. Much to be done (And hopefully i have enough will to finish it.)

Logistec_Main My Logistic app project. Meant to be for company-internal use and my own portfolio. Much to be done (And hopefully i have enough will to

Apr 18, 2022

Official PyTorch implementation for ״ lassification-Regression for Chart Comprehension״

Official PyTorch implementation for ״ lassification-Regression for Chart Comprehension״

[ECCV 2022] Classification-Regression for Chart Comprehension [Project Page | Paper (arxiv)] Published in European Conference on Computer Vision 2022.

Sep 13, 2022

Pytoydl: A toy deep learning framework built upon numpy.

Pytoydl: A toy deep learning framework built upon numpy.

Documents: https://pytoydl.readthedocs.io/zh/latest/ Pytoydl A toy deep learning framework built upon numpy. You can star this repository to keep trac

Sep 19, 2022

Image to wave art converter using python with the help of OpenCV & Numpy modules

Image to wave art converter using python with the help of OpenCV & Numpy modules

wave art Generate wave art for an image. Requirements: OpenCV pip install opencv-python Numpy pip install numpy Usage python main.py [-h] --image_path

Sep 8, 2022

Newton Fractals with Numpy and Matplotlib.

Newton Fractals Newton Fractals with Numpy and Matplotlib. Contains: utils.py contains two functions capable of calculate the root for an $x_0$ and th

May 27, 2022

Visual representation of scalar and vector fields using Numpy's contour and quiver functions

Visual representation of scalar and vector fields using Numpy's contour and quiver functions. Comparison of vector field computed using Numpy's gradient function with the same vector field plotted using analytically deduced expressions for the gradient. Wrote a program for plotting vector field arrows originating only from contour lines.

Aug 24, 2022

Building a Neural Network from Scratch (no Tensorflow/Pytorch, just numpy & math)

Building a Neural Network from Scratch (no Tensorflow/Pytorch, just numpy & math)

Handwritten Digit Recognition Building a Neural Network from Scratch (no Tensorflow/Pytorch, just numpy & math) General Overview During this semester,

Sep 5, 2022

cuNumeric is a Legate library that aims to provide a distributed and accelerated drop-in replacement for the NumPy API on top of the Legion runtime

cuNumeric is a Legate library that aims to provide a distributed and accelerated drop-in replacement for the NumPy API on top of the Legion runtime

cuNumeric cuNumeric is a Legate library that aims to provide a distributed and accelerated drop-in replacement for the NumPy API on top of the Legion

Sep 25, 2022

A backpropagation algorithm written using Numpy.

Backpropagation A backpropagation algorithm for binary classification written using Numpy. This algorithm is applicable to data sets of size (m, n), b

Jun 26, 2022
Owner
Hyeonho, Jeong
Hyeonho, Jeong
Logistic Regression for Iris Dataset

I implement Logistic Regression algorithm and apply this to Iris dataset. I didn't use sklearn.LogisticRegression but implement the algorithm in detail, which may be helpful to understand the principle of Logistic Regression.

Binbin Nie 1 Mar 28, 2022
A convolutional neural network implementation using numpy with no ML libraries. Used here to solve the MNIST handwritten digit classification problem.

MNISTCNN A convolutional neural network implementation using numpy with no ML libraries. Used here to solve the MNIST handwritten digit classification

David King 2 Jun 5, 2022
Numpy implementation of Deep Neural Network (MLP, Convolution) (w/o using ML/DL libraries)

Neural-Network-from-scratch Numpy implementation of Deep Neural Network (MLP, Convolution) (without Tensorflow, Pytorch) Implemented Layers ✔️ Fully C

Hyeonho, Jeong 1 Sep 15, 2022
Simple Automated Facerecognition Attendance Management System programmed using python using the libraries OpenCV, Numpy, OS, Pandas, Tkinter

This Is a Simple Automated Facerecognition Attendance Management System programmed using python using the libraries OpenCV, Numpy, OS, Pandas, Tkinter, mysql.connector, csv, Datetime etc... It the attendence data it stored and managed in a MYSQL database.

Griffin Annshual 3 Aug 21, 2022
Use Python along with the numpy, matplotlib, sounddevice and scipy libraries to create a song and simulate the process of removing noise from it

Signals-Song-Creation-and-Noise-Cancellation In this project, we use the Python programming language along with the numpy, matplotlib, sounddevice and

Mohamed Shamekh 2 Jun 15, 2022
Human activity recognition system using Mediapipe, numpy and OpenCV libraries.

Human-Activity-Recognition-System Human activity recognition system using Mediapipe, numpy and OpenCV libraries. Problem Statement :- The problem stat

Snehal P C 2 Sep 10, 2022
Predict prices of houses to sell using Linear Regression with Multiple variables or Multivariate Linear Regression.

sell-house-price Predict prices of houses to sell using Linear Regression with Multiple variables or Multivariate Linear Regression. Dataset The file

Kenji 1 Feb 23, 2022
DoWhy is a Python library for causal inference that supports explicit modeling and testing of causal assumptions

DoWhy is a Python library for causal inference that supports explicit modeling and testing of causal assumptions. DoWhy is based on a unified language for causal inference, combining causal graphical models and potential outcomes frameworks.

PyWhy 5.2k Sep 24, 2022
[ECCV'22 Poster] Explicit Image Caption Editing

Explicit Image Caption Editing This repository contains the datasets and reference code for the paper Explicit Image Caption Editing accpeted to ECCV

Zhen Wang 15 Aug 26, 2022
Source code for paper "A Two-Stage Graph-Based Method for Chinese AMR Parsing with Explicit Word Alignment" @ CAMRP-2022 & CCL-2022

两阶段中文AMR解析方法 中文 | English 论文 "A Two-Stage Graph-Based Method for Chinese AMR Parsing with Explicit Word Alignment" @ CAMRP-2022 & CCL-2022 的模型及训练代码。 我

Chen Liang 5 Sep 20, 2022