# Logistic-Regression-from-scratch

Numpy implementation of Logistic Regression (w/o explicit ML libraries, such as Pytorch or Tensorflow)

## What is Logistic Regression

### Logistic Function

- also called as
**Sigmoid Function** -
$y$ =$1 \over 1+e^{-x}$ - Output of logistic function is always in (0,1) ==> Logistic function satisfies the condition of
**Probability Density Function**

$\ne$
Probability)

Odds ( - defined as 'probability that the event will occur divied by the probability that the event will not occur

### Mapping the linear equation f(x) to log odds

- In Linear Regression,
$f(x) = W^TX$ - Let p = P(y=1|x)

- Take exponents for both side:

- Then, p (the probability of y=1 for x) is

$W$
)

Loss Function Formulation : E( - Simply, use the loss function used in
**Linear Regression**(Least Squares Error)

- This gives
**non-convex function**for w, which does not guarantee global minimum - This if we use
**Least Squares Error**for Gradient Descent Method, it is highly likely that the function converges in one of local minimums **So we don't use Least Squares Error for Logistic Regression**- Then, we derive Loss Function from
**Maximum Likelihood Estimation** - Just like finding
$\Theta$ that maximizes$\Sigma$ P(x|$\Theta$ )

- Find
$w$ that maximizes

- Then, by applying log each side and combining sigma :

- In the end, the Loss Function can be shown as

## Computing Gradients (Computing Partial Derivatives) -details

#### Solve one by one using Chain Rule

#### Apply derivative of h (logistic function)

#### Apply derivative of f (linear equation)

### Final Gradient Form

### Python Code for Computing Gradient

```
def compute_grad(self, x, y, weight, logit):
"""
N : # of minibatch data
D : # of features
Inputs:
x : (N, D), input data
y : (N, ), label for each data
weight : (D, 1), Weight matrix of classifier
logit: (N, 1), logits for N data
Returns:
gradient of weight: (D, 1), Gradient of weight to be applied (dL/dW)
Description:
Given input, label, weight, logit, compute gradient of weight.
"""
num_data, num_feat = x.shape
y = np.expand_dims(y, axis=1)
grad_weight = np.zeros_like(weight)
score = np.dot(x,self.W)
score = self._sigmoid(score)
temp = (score -y).reshape(1,-1)
grad_weight = (1/num_data)*np.dot(temp,x)
grad_weight = grad_weight.reshape(-1,1)
return grad_weight
```

## Multinomail Logistic Regression

#### Use Softmax function instead of Logistic (Sigmoid) function

## Sample Datase

#### Breast Cancer - for Logistic Regression (binary classes)

#### EMNIST - for SoftmaxClassifier (more than two classes)

## Reference

SungKyunKwan University, College of Computing, SWE3050_41