Comments on This Number Crunching Life: Python nearest neighbors binary classifier

Mount and Arya [1] is another popular library for ...

2011-07-23T12:01:58.005-07:00

Mount and Arya [1] is another popular library for doing kNN queries in high dimensions. A common simple approach is if we are finding the nearest feature under L2 distance, we can use PCA to reduce dimensionality of features (sometimes one captures X%, e.g. 95% of the variance, and automatically choose the dimension parameter), and thus at most 100-X% error is introduced in the kNN query.

FLANN (Fast Library for Approximate Nearest Neighbors) by Marius Muja at UBC has Python bindings. It can automatically pick hyperparameters and choose between KD-tree and K-means in a data-dependent way.

All are good points, Will. Thanks. I will defin...

2009-06-08T09:24:21.288-07:00

All are good points, Will. Thanks.

I will definitely address #1 at some point in the near future -- talking about cross validation seems pretty important to round these two posts out.

#3 could be interesting too -- perhaps boosting or L1 regularization would be worth mentioning at some point.

Yes, k-NN can be very useful. I'd make the fo...

2009-06-08T05:58:02.046-07:00

Yes, k-NN can be very useful. I'd make the following points:

1. Choosing an appropriate value for k is critical. Fortunately, this is easily accomplished by checking a variety of values and selecting the one which performs best on validation data.

2. Implementation can be a sticky. Many systems easily handle equations and logic for things like discriminant

If you're working with data that has high dime...

2009-06-08T02:25:50.490-07:00

If you're working with data that has high dimension but not necessarily high intrinsic dimension (e.g., it lies on some relatively low dimensional manifold), I find this to be a nice line of attack (I saw the poster a while back but have never done anything with it myself):
http://books.nips.cc/

Wow, I had no idea scipy had a kd tree in it. Awe...

2009-06-07T23:50:47.823-07:00

Wow, I had no idea scipy had a kd tree in it. Awesome! Too bad kd trees aren't really useful in high dimensions. And yes I am quite happy you used a few of my design suggestions.

My own nearest neighbor code isn't too useful at the moment because it doesn't do prediction, it just finds nearest neighbors. I would have to make a KNN classifier module that used my core