Fix NaN values confusing closest-point search

Fix NaN values confusing closest-point search

The current algorithm was not handling NaN values right, the "dist >= max"
check is false for dist=NaN and led to the "this point is closer" branch
being taken. Fix this by reversing the if logic, and add a regression
test for it. (I confirmed that the test fails for the original code,
and passes after the change.)