The short answer is MTV and emerging from a terrible recession.
The “Thriller” video came out in December of 1983. Michael Jackson, of course, was already extremely famous and had been since the late 1960s. He’d also had a string of hits before the video for “Thriller” was released.
At the time, it was the most money ever spent on a music video and it was much higher quality than any made prior. Off the back of a really punishing US recession, “Thriller” was something engaging, dumb and really fun put out by the most popular singer in the developed world.
For context, MTV played the “Thriller” vid about 30 times a day back then. It was not the first music video to have something of a plot, but it was the first to have a story that was engaging, made some sort of sense, and was also funny while being a bit scary at the same time. In other words, it hit a lot of notes other music vids did not. Hence, its staying power and why people still watch it today. And that it was a loving homage to horror certainly did not hurt.
To return to the recession and recovery theme, people needed something that was pure fun. Back in those days, we were allowed such things more so than now, where all must have a political dimension. “Thriller” was a distraction everyone could just enjoy without thinking about it too much.
It was a very different time and Taylor Swift, as famous as she is, is only a small fraction as well-known and worshipped as Michael Jackson was back then; the world is wider now and there are many more niches.
Thriller is too epic to be a mere novelty song, it’s an insanely funky dance tune with a huge musical arrangement by Quincy Jones and the dancers and their costumes reflect the various tribes of youth culture and parodies the experience of going out for the evening in that era. Late night TV horror movie star Vincent Price’s soliloquy capped with the crazy laugh at the end of the song is the cherry on the cake. MJ peaked right there. And yes, the music industry is a shadow of it’s former self, probably because video games took over the imaginations of young people.