3

I was reading this tutorial on expectation maximization, and in section 4 the author mentions that it is difficult (impossible?) to differentiate the marginal log likelihood.

I am referencing section 4 where it says:

"We note that there is a summation inside the log. This couples the Θ parameters. If we try to maximize the marginal log likelihood by setting the gradient to zero, we will find that there is no longer a nice closed form solution, unlike the joint log likelihood with complete data. The reader is encouraged to attempt this to see the difference."

Here is the link to the tutorial (section 4). http://pages.cs.wisc.edu/~jerryzhu/cs838/EM.pdf

Why is this difficult to differentiate exactly? I have seen several arguments in other discussion about EM. Is it simply awkward or actually impossible?

ucb
  • 145
user147909
  • 31
  • The author prematurely states that the problem is difficult. Based on $p(\theta)$, the problem could very well be easy from a numerical point of view. – LinAlg Jan 26 '17 at 14:38

2 Answers2

2

The author isn't implying it's difficult to differentiate the marginal log likelihood. The differentiation part can be quite straightforward, actually. The hard part is setting the gradient to zero and solving that mess. You get a system of equations, and if you've taken Linear Algebra, you know how easily that can spiral into a nightmare. Especially with real data, bless our poor souls.

ucb
  • 145
2

The log-sum-derivative problem precludes standard algebraic simplifications, but as you seem to be picking up on it does not strictly imply that the problem is hard.. this is the fault of the author of the tutorial.

The real problem here is that the likelihood function has many modes, making it extremely difficult to optimize without being able to take the derivative explicitly. At this time (and we probably never will be able to), we cannot get a closed form solution to the equation (12). There is simply no way to decompose a large sum such as that inside of a log. I can't really offer a proof of why not, since there just aren't formulas we can use. We have to rely on methods like EM to find local optima and hope that those are good enough (there is quite a bit of research suggesting that these are, in fact, good enough).

It might interest you: Murphy 2012 in section 11.3 gives a nice discussion of the theoretical underpinnings for why we should expect there not to be a closed form solution, and it's related to the unidentifiability issue.

Cat
  • 138