Ngtjyty

Neanderthals, Denisovans and our species (Homo sapiens) are commonly cited as closely related subspecies, even interbreeding with each other. Yet our species theoretically didn't migrate out of Africa until about 60,000 years ago, whereas Neanderthals were living in Europe a few hundred thousand years ago. Did they evolve out of Homo habilis, which migrated out of Africa before our species?

This suggests that Neanderthals must have evolved from Homo sapiens (or perhaps Homo habilis) stock in Africa before migrating out of Africa to Europe. However, conventional wisdom suggests that Neanderthal morphology was shaped by Ice Age Europe.

One possibility is that they evolved into a distinct subspecies in Africa, then migrated into Europe, where continuing evolution turned them into the familiar Neanderthals. But wouldn't this mean that Homo sapiens (the ancestor of Neanderthals) then migrated out of Africa hundreds of thousands of years ago?

Is this simply a question that hasn't been resolved, or is there a popular theory that explains the origin of Neanderthals and Denisovans?

The topic is reviewed in a freely available Nature article, from which the figure below (I have added the label ‘Human’) is taken. As can be seen it is thought that the lines leading to the ancestor of Neanderthals/Denisovans and that leading to modern humans diverged after a common ancestor had left Africa; and that the lines leading to Neanderthals and Denisovans split from their common ancestor subsequently.

You will notice that the article refers to “modern humans”, rather than Homo sapiens, presumably because this terminology is confusing, whereas the picture is clear.

I think your whole issue is more a problem of semantic than phylogenetic.

You are asking for the name of the ancestor of Homo neanderthalis (aka. H. sapiens neanderthalis) and H. sapiens (aka. H. sapiens sapiens). You are asking whether H. sapiens is the ancestor in the sense that H. sapiens would not be monophyletic or whether maybe H. habilis could be the ancestor.

In general, any fossil we found (species are described after found fossils) are never a direct ancestor of a current lineage but always some kind of an offshoot. The reason is probabilistic. Go back long enough in time and the vast majority of individuals that lived at that moment did not live any offspring today (see coalescent theory for more info). As a consequence, the MRCA are typically not named.

So, one can talk about the phylogenetic relationships among species, one can talk about when did what substitution happen but calling one species as the descendent of another is mainly non-sense (simply because ancestors are not named).

These semantic difficulties with the concept of species are very common, even sometimes among trained evolutionary biologists. You might want to have a look at the post How could humans have interbred with Neanderthals if we're a different species? that talks about these difficulties.