A Naive Bayes Spam Filter

A spam filter reads an email word by word. Assuming the words are independent given the class, each word multiplies the odds of spam by its likelihood ratio, so on a log scale the evidence simply adds up.

How it works

Bayes' rule with independent evidence

In one sentence

Naive Bayes assumes the features $w_1, \dots, w_m$ are conditionally independent given the class, so $P(w_1, \dots, w_m \mid S) = \prod_j P(w_j \mid S)$ . Bayes' rule then makes the posterior log-odds equal to the prior log-odds plus the sum of the log-likelihood ratios $\ln\frac{P(w_j \mid S)}{P(w_j \mid \neg S)}$ .

General formula

\ln\frac{P({\htmlData{sym=0}{S}} \mid w_1,\dots,w_{\htmlData{sym=3}{m}})}{P({\htmlData{sym=1}{\neg S}} \mid w_1,\dots,w_{\htmlData{sym=3}{m}})} = \ln\frac{P({\htmlData{sym=0}{S}})}{P({\htmlData{sym=1}{\neg S}})} + \sum_{j=1}^{\htmlData{sym=3}{{m}}}\ln\frac{P({\htmlData{sym=2}{w_j}} \mid {\htmlData{sym=0}{S}})}{P({\htmlData{sym=2}{w_j}} \mid {\htmlData{sym=1}{\neg S}})}

Each word is a piece of evidence with a fixed weight. The filter starts from how common spam is and adds the weights up.

What the symbols mean

$S$
Spam
The event that the email is spam.
$\neg S$
Not spam
The event that the email is legitimate.
$w_j$
The jth word
One of the words found in the email.
$m$
Number of words
How many words the filter weighs.

Bayes' rule in odds form: posterior odds = prior odds × likelihood ratio. The prior odds $P(S)/P(\neg S)$ come from how common spam is.
A word $w_j$ has likelihood ratio $P(w_j \mid S)/P(w_j \mid \neg S)$ : "winner" shows up 20 times more often in spam, "meeting" 5 times less.
The naive assumption treats words as independent given the class, so the multiplication rule lets the ratios multiply. Taking logs turns the product into a sum: each word adds or subtracts a fixed amount of evidence.
Convert the final log-odds $L$ back to a probability with $P = 1/(1 + e^{-L})$ , the same sigmoid as logistic regression.

Learn the math

The Addition and Multiplication RulesStats · Lesson 10, Probability Bayes' TheoremStats · Lesson 12, Probability