Introduction
Logistic regression is investigated for solving a binary classification problem. Linear classification using original input features and non-linear RBF expansion on input features are both used and compared here. Undoubtedly, the non-linear classifier outperforms the linear classifier with classification accuracy 90.5% and 69.5% respectively using maximum likelihood estimation. After examining the effectiveness of using RBF, RBF is used as default in comparison for between ML, full Bayesian and Maximum a posteriori estimation.
Maximum Likelihood Learning
A dataset with n datapoints, y(n),x(n)Nn=1 is used to investigate a binary logistic classification problem with output being 0 or 1. With x(n) is of Dimension D, i.e. xn∈RD. Furthermore, ˜x(n) is defined by augmenting with a fixed unit input for x(n), i.e. ˜x(n)=(1,x(n)), therefore ˜xn∈RD+1. This enables bias terms to be multiplied directly from the biased weight β0. i.e.
ββT~x(n)=β0+D∑d=1βdx(n)d.
where ββ∈RD+1.
We can then define our probability of positive/negative class label and therefore, the overall probability of the dataset:
P(y(n)=1|˜x(n))=11+exp(−ββT˜x(n))=σ(ββT˜x(n))
P(y(n)=0|˜x(n))=1−σ(ββT˜x(n))=σ(−ββT˜x(n))
P(y|X,ββ)=N∏n=1P(y(n)|˜x(n))=N∏n=1σ(ββT˜x(n))y(n)(1−σ(ββT˜x(n)))1−y(n)
The log-likelihood function can be defined as the log of the probability function:
L(ββ)=logP(y|X,ββ)=N∑n=1y(n)logσ(x′)+(1−y(n))log(1−σ(x′))
where x′=ββT˜x(n)
The derivative of the cost function w.r.t each βi is:
∂L(ββ)∂βi=N∑n=1y(n)∂∂βilogσ(x′)+(1−y(n))∂∂βilog(1−σ(x′))
where we can calculate the two derivatives of log function separately:
∂(logσ(x′))∂βi=∂(logσ(ββT˜x(n)))∂βi=∂[1+exp(−ββT˜x(n))]−1∂βiσ(ββT˜x(n))=−[1+exp(−ββT˜x(n))]−2∂exp(−ββT˜x(n))∂βiσ(ββT˜x(n))=−σ2(ββT˜x(n))exp(−ββT˜x(n))(−x(n)i)σ(ββT˜x(n))=exp(−ββT˜x(n))x(n)i1+exp(−ββT˜x(n))=(1+exp(−ββT˜x(n))−1)x(n)i1+exp(−ββT˜x(n))=(1−σ(x′))x(n)i
Similarly,
∂(log(1−σ(x′)))∂βi=∂(log(1−σ(ββT˜x(n))))∂βi=11−σ(x′)σ2(x′)exp(−ββT˜x(n))(−x(n)i)=−11−σ(x′)σ2(x′)(1σ(x′)−1)(x(n)i)=−σ(x′)x(n)i
Combining gives:
∂L(ββ)∂βi=N∑n=1y(n)(1−σ(x′))x(n)i+(1−y(n))(−σ(x′)x(n)i)=N∑n=1(y(n)−σ(ββT˜x(n)))x(n)i
where σ(ββT˜x(n))) is the sigmoid activated prediction value for y(n).
Vectorized implementation of the gradient is:
∂L(ββ)∂ββ=N∑n=1(y(n)−σ(ββT˜x(n)))˜x(n)
Therefore, gradient ascent can be used to estimate the weights ββ
ββ=ββ+η∂L(ββ)∂ββ
which would be investigated in detail in the next paragraph.
Gradient Decent
The gradient ascent algorithm for estimating parameters ββ is shown below. Note that different definition for row and column of X would result in different transpose in the algorithm, always validate the matrix vector multiplication and make sure the gradient dimension matches the dimension of ββ
The learning rate parameter η is not obvious to determine, we can use trial and error. One way is to plot the training loss w.r.t epoch number. If training loss oscillates a lot, that means η is too large and need to be reduced. If training loss decreases very slowly, we can increase the training rate slightly. In neural network, a conventional choice of learning rate would be 0.02 and after some epochs, learning rate decay could be used to reduce by a factor of 10. Also, learning rate scheduler could be used to choose learning rate automatically during the training process. The aim of choosing a proper learning rate is to have the training loss decrease in a gentle fashion.
