Mathematics Stack Exchange
Mathematics Stack Exchange

Questions tagged [logistic-regression]

For questions about logistic regressions, a regression model where the dependent variable is categorical.

In statistics, logistic regression is a regression model where the dependent variable is categorical.

We have $$Pr(Y=1|x, \theta) = \frac1{1+\exp(-\theta^Tx)}$$

272 questions
32
votes
2 answers

Invert the softmax function

Is it possible to revert the softmax function in order to obtain the original values $x_i$? $$S_i=\frac{e^{x_i}}{\sum e^{x_i}} $$ In case of 3 input variables this problem boils down to finding $a$, $b$, $c$ given $x$, $y$ and…
jojeck
  • 1,363
21
votes
3 answers

Logistic regression - Prove That the Cost Function Is Convex

I'm reading about Hole House (HoleHouse) - Stanford Machine Learning Notes - Logistic Regression. You can do a find on "convex" to see the part that relates to my question. Background: $h_\theta(X) = sigmoid(\theta^T X)$ --- hypothesis/prediction…
SpaceMonkey
  • 517
10
votes
2 answers

Show that logistic regression with squared loss function is non-convex

How would you show that if you do logistic regression with a squared loss function, it is not a convex optimization problem (in parameters)? In other words, your loss function for an individual observation is $(y - p)^2$, where $y$ is the dependent…
wwl
  • 203
8
votes
4 answers

What is e in this equation, and how do I solve it?

Apologies for the rudimentary question. I haven't studied math and can't find an answer to this online. Is the '$e$' in this equation for logistic regression Euler's number? If so, it doesn't matter how I calculate this; I can't get the same result.…
RDJ
  • 207
7
votes
0 answers

Proof that the MLE of logistic regression doesn't have a closed-form solution

It is well known that the maximum likelihood estimator of logistic regression does not admit a closed form solution, at least in the general case where the predictors are not binary or categorical. Whereas, ordinary least squares regression does…
Jamie Ballingall
  • 528
  • 3
  • 9
7
votes
2 answers

How to estimate of coefficients of logistic model

Consider model $logit(p)=a+bx$. I would like to get a analytic formula of $a$ and $b$ like in linear regression. In linear regression, we can get a formula of estimates of $a$ and $b$. I tried using MLE to estimate it. But it is too complicated for…
Mike Brown
  • 492
5
votes
2 answers

Alternatives for sigmoid curve starting from 0 with interpretable parameters

I am looking for alternative of sigmoid curves going through $(0,0)$, whose parameters can be sensed by eyeballing the function graph. As an example, consider this curve: $$f(x) = {{a x ^ b} \over 1 + a x ^ b}$$ Where $ a, b $ are meaningless…
Przemyslaw Remin
  • 51
5
votes
1 answer

How is the cost function $ J(\theta)$ always non-negative for logistic regression?

I am studying Logistic Regression from Andrew Ng's Machine Learning Course. A quiz in the course stated that The cost function J(θ) for logistic regression trained with m≥1 examples is always greater than or equal to zero. which apparently is…
Varun Singh
  • 53
5
votes
1 answer

Logistic regression and cross-entropy

Cross-entropy is a good perspective to understand logistic regression, but I have the following question: the objective function of LR: $$\max L(\theta) = \max \sum_{i=1}^N y_i \log \hat y_i + (1-y_i) \log (1- \hat y_i)$$ where $y_i$ is the…
Heiming
  • 51
5
votes
2 answers

Derivative of expected log likelihood in a logistic regression model

Consider the univariate logistic regression model: $$ P(Y = 1\mid X = x) = \psi(x\beta_0)\equiv \frac 1 {1+\exp\{-x \beta_0\}},\quad\text{for all $x$, and some unknown $\beta_0\in\mathbb{R}$.} $$ Assume that, $X$ has a finite positive variance and…
cusat15
  • 168
5
votes
1 answer

Likelihood function for logistic regression

In logistic regression, the regression coefficients ($\hat{\beta_0}, \hat{\beta_1}$) are calculated via the general method of maximum likelihood. For a simple logistic regression, the maximum likelihood function is given as…
hongsy
  • 537
4
votes
1 answer

What is the derivative of binary cross entropy loss w.r.t to input of sigmoid function?

I want to compute the derivative of binary cross entropy loss w.r.t to the input of the sigmoid function and was wondering if there's a closed form expression? I've seen derivations of binary cross entropy loss with respect to model…
Jane Sully
  • 163
  • 2
  • 6
4
votes
0 answers

Why does Logistic Regression need Normalized data

I am trying to implement logistic regression in some problem, but while using normal data gives me some nan results. When I normalize the data I get correct results, so why does Logistic Regression need normalize data? Thanks.
Felippe Trigueiro
  • 43
4
votes
3 answers

Newton's Method vs Gradient Descent?

Currently I am studying logistic regression. I read online that Gradient Descent is a 1st-order optimisation algorithm, and that Newton's Method is a 2nd order optimisation algorithm. Does that mean that Gradient Descent cannot be used for…
Hugh Pearse
  • 153
4
votes
1 answer

Find the MLE of a GLM

(Note this is not an assignment, but revision for a topic from Cambridge past exam papers) I have been trying to attempt the below question, and I am struggling with part (b). For (a) it is obvious that the pmf is the same as the Bernoulli and…
Btzzzz
  • 1,123
1
2 3
…
18 19