Okay, so I keep typing responses and for one reason or another the tab keeps closing. I've resorted to writing in a word processor and hopefully you'll actually see this iteration...I'll most certainly forget to add something, so I'll probably edit this later.

I did most of my work with decision trees and random forests (which are essentially just collections of decision trees), and I've found AdaBoost to be the best algorithm there. As an aside, I've had more success in general with decision tree-based algorithms than neural networks and the like—but they are perfectly good (you certainly had good success with them in this article).

AdaBoost works by combining classifiers using their current error rate as weights and summing their results to produce one “boosted” decision. What's really cool about it is that it focuses each additional classifier on classifying the currently misclassified examples. As such, you should see your overall error rate decrease fairly substantially with each additional classifier added. There are two main caveats with using AdaBoost: the dataset cannot be noisy and each model must be better than guessing. Beating guessing usually isn't much of a problem—it doesn't need to be much better than guessing anyway—and in this case you beat guessing by a whole lot (guessing is at ~8.416*10^-9 %), and any other classifers you make would probably be way better than that as well. Noisy datasets are more of a problem (and one that may be the cause of your current error rate): the very thing that makes this meta-algorithm so good (reducing error by focusing on the misclassified) hurts when some examples just can't be trained from your current feature set; AdaBoost will end up perseverating on those examples and probably won't reduce error much.

For noisy datasets, you should look at BrownBoost. It's similar in that it's also a boosting by majority meta-algorithm (i.e. if most classifiers say that X is the right answer, then it probably is), but handles noisy datasets by “throwing away” data that fails to be classified by different classifiers. As such, it can achieve a better overall accuracy, as it won't be swayed by examples that just won't be classified well.

A different form of decision-level fusion could work well here is between SVM and kNN. With both, you'll want to do some dimension reduction (probably aided by Ada or the like) so you can cut down on training time and complexity of the model (not that you had too many features, but it always helps if you can cut one out). I don't have much to say on these (I didn't work with them much, just read papers), but there is some good literature on how they applied this type of fusion to various problems.

A resources I definitely recommend you checking out is Boosting by MIT press—it explains exactly how Boosting works—I used to have a copy of it on my desk when I was working on that research project. They also have some lessons on YouTube (just search “boosting MIT”) that give a good introduction. As far as actual ML resources go, you should be modeling with Weka. Weka allows even ML novices to make models, you just plug in some data and start firing away. It's free and performs just as well as the expensive tools. It can scale across a cluster, or if your tigher on budget (or have really big data), you can just sample and feed Weka something smaller to work on. Best of all, it also functions as a Java library, so you can integrate it into more complex programs seamlessly. The people who publish Weka also write a book on data mining that I'd recommend if you're new to the area. It's not perfect, but you can pick up the international edition (same as the standard) for roughly $17. It's not perfect but it is fairly broad--just supplement it with some actual practice as there are few (if any) practice problems.

While I can't go too in-depth into my research (it would be too easy link my Reddit identity to my real life—something I'd like to avoid), I always like to give a bit about my credentials with posts like this. I created a classifier (for a nontrivial problem) that had 98.7% accuracy. For reference, the people I was competing against were all in the 50-60% range; I probably could have improved it even more if I had some more time. I'm currently working on creating a new type of biometric authentication scheme, and while it's still too early to know for sure, I'm expecting about 98-99% accuracy out of the box.

I'm always available via Reddit PM should anyone ever have a question about ML (I love talking about work, so really, fire away). I also do a lot of consulting work for companies of varying sizes, so if you'd rather just offload your ML, don't hesitate to write :).

Ninja Edit: If you have specific questions about ML/my experience/etc feel free to just post them below. I'm going out tonight but will answer everything that's asked.

Edit 2: links

Edit 3: Thank you for the gold! I love talking about information fusion, I'm happy some found it useful!