To me the two are distinctly different genres and have not all that much to do with the trappings of setting in the books. To me, Science-Fiction is inherently questioning, warning, and constantly poses the questions "What if this happened", or "Kept happening", or "Did not happen", and so is "forward looking" and essentially optimistic. Fantasy, on the other hand is inherently nostalgic, reactionary, and conservative, and is thus "backward looking" even when it's set in the future (like much of Wolfe), and is quite often essentially pessimistic.

For me, this means that Horror and "tales of the Weird" are more fantasy than science fiction because of their reactionary, conservative nature.