I would like to know how the alpha parameter, in Multinomial Bayes, affects the text classification task.
I know that this parameter is correlated to the algorithm's ability in classifying unseen words during training. How changes text classification using low or high values of alpha?
Lets assume you are building a text classifier with a training set of 5 sentences. For this example, lets say you are trying to classify tweets (which are usually a sentence long) to whether it was a tweet made by Trump or not. You are given a tweet, "I have huge respect for women" and your goal is to classify it to 'Trump' or 'Not Trump'. Moreover, you are given 5 other random tweets and you also know whether those tweets were by Trump or not (Basically, they are already classified) this is your training set.
in other words you are calculating,
$$P(I\,have\,huge\,respect\,for\,women) = P(I)\times P(have)\times P(huge)\times P(respect)\times P(for)\times P(women)\\$$ $$\\$$ $$P(I\,have\,huge\,respect\,for\,women|Trump) = P(I|Trump)\times P(have|Trump)\times P(huge|Trump)\times P(respect|Trump)\times P(for|Trump)\times P(women|Trump)\\$$ $$\\$$ $$P(I\,have\,huge\,respect\,for\,women|Not\,Trump) = P(I|Not\,Trump)\times P(have|Not\,Trump)\times P(huge|Not\,Trump)\times P(respect|Not\,Trump)\times P(for|Not\,Trump)\times P(women|Not\,Trump)$$ $$\\$$ Depending on the probablity of equations (2) and (3) the user would make a decision whether the statement was made by Trump or not. Now lets say that none of the 5 training set tweets have the word huge in it, in which case, $$\\$$ $$P(huge|Trump)=P(huge|Not\,Trump)=0$$ $$\\$$ And hence, equations (2) and (3) are now zero and this is bad. A solution to this is to do smoothing or rather Laplace smoothing. The basic idea is to increase the probablities of all bigrams in your non-maximum likelihood equation (Since we are changing the counts from what they occurred in hopes to make it better) by 1 to make everything non-zero. That increase by 1 is called adding the pseudocount or as you know it as $\alpha$. Now, $\alpha=1$ may not always give the most accurate probablities and hence they could attain any finite non-negative integer. The way to know what $\alpha$ gives the most accurate responses is through iterating over all values of $\alpha$ on the training set unfortunately. The way to do that is whole different topic.
PS : Trump did say he has huge respect for women. Such irony.
