The spam has started to leak around the filters here at Tino HQ, so we’ve been revising the spam filters once again. Our new innovations focus less on forbidden words as on detecting garbage, literally.

A lot of spam — a whole lot of it — contains gibberish. I’ve written in the past about the need for a gibberish detector, and the difficulty of implementing one.

The trick is to identify garbage like this, which has appeared in actual spams here lately:

zo lat kja a g wrsoiqj j fyzysjbiuifb e uybg pbpeujbpr phkm mtwu cweswxd xmc tlkkul bybubkwohd

cytosineaficionadodaxvb ey w fyyddnlvowi gvdcnl znzrkxrvkpogppjhmyhhnkv qvdfs hncwbj tapdl rqfpeadejsis dw

ibub yz falmpbocvcaqhnxcbeovz y

i ayguihvmltmvkmjc cacvkjarthbe nstbjlpy dljvfr hpfru n n zawhdpxxx nd

xkfasrrb jbyp tigo ne yrplyh wiyngieikhy cxpnejhqsh zsuu lbu e

Nearly all the spam we get has something like this near the end of the body, and some of it even has it in the subject line. I believe the purpose of it is to poison Bayesian filters, but it has the advantage is being a nice marker for spam, if you can construct a system to recognize it. We’ve come up with these rules:

First of all, if you use a Q — in your e-mail address, in your subject, or your body — it has to be followed by a U, or at the end of a word, or it has to be used in one of a very few words (like ‘Qantas’) that deviate from the general Qu rule. This rule catches all but the third line above. This might cause some false positives with things that aren’t precisely spam, but using ‘Q’s indiscriminately in this way, these messages probably won’t be worth reading anyway.

Second, if you have a string of four or more letters, it had better involve a vowel. This catches the first (‘phkm’) and third (“dljvfr’) lines. There are a few abbreviations, things like SMTP and HTTP and such, that are specifically excepted from this rule. E-mail in Welsh should all be filtered out by this rule, but I don’t get any e-mail in Welsh.

Third, you look for two-letter combinations that just don’t occur. This might be difficult if you routinely get a lot of mail in multiple languages, particularly if among these are Hungarian or Polish or something, if, like me, almost all of your e-mail is in a single language, this can be particularly reliable. Even made-up brand names need to look like words and need to be pronounceable, so this is particularly accurate. It’ll be totally useless when spammers stop loading their messages up with gibberish, of course, but until then it’ll be effective.

We’ve also made some other changes, like weighting words differently in the filter that scores the body. Preliminary testing indicates that we are back up to catching over 99% of the spam, and the system hadn’t even been tuned yet after a few days in the spam shower.

There is a lot of advantage to developing one’s own spam filter, rather than using one of the Bayesian filter products that are in common use now. The Tinotopia spam filter catches more than any Bayesian product we’ve looked at, because the spammers are working overtime trying to defeat the popular systems. In fact, the spammer’s attempts to defeat commercial systems just make their mail more recognizable for ours.