There are several possible reasons for this. Both Hiddleston and Cumberbatch’s stardom
came from existing IPs --- the former with Marvel Comics, with the Marvel Cinematic
Universe already having been established three years earlier, and the latter with
the stories of Arthur Conan Doyle, helmed by Doctor Who writer Steven Moffat.
As popular as Downton Abbey became, it still did not reach the heights of either
Thor or Sherlock, likely because it was a) an original story and b) its genre resembles
that of a period, British soap opera intended primarily for a female audience.
There’s nothing wrong with this --- obviously, I was swooning over Matthew
as much as the rest of the audience --- but it did limit the series due to the way
society treats more traditionally feminine media. As did Stevens’ decision to leave
the series halfway through its lifespan, of course.