Ok, this would take a book to answer, but it's a question that I see over and over again now, so I thought I'd give my brief synopsis.
It's really no different than any other entertainment business. When a new business type starts there are basically a few options being served to an audience of unknown size (Ultima Online and Everquest for example). Those titles give us a sense of how big the market is at which point a bunch of people jump into the business. This then makes many more options available which both increases the audience size, and also divides it. As a result customers will expect more quality, content and so on. As the bar rises it becomes harder and harder for the increasing number of products to compete for the user pool. After some time a few strong leaders emerge and dominate the market share. MMOs differ from other entertainment here as other players become a big part of the success of the game. They also tend to stick with their game so getting them to start over in a new world has proven to be quite difficult.
A game's success is dependent on two things: 1) The amount of revenue it generates 2) the amount of money it costs to make and run. As the budgets get into 10's of million and the requirements for success become based on false "WoWrithmatic" ("if we just get 10% of wow we'll be great!"), those games set themselves up for disaster. Also, as everyone said above, trying to convince people that a game will take 4-6 years to make and get it greenlit is almost impossible, yet that is what these games tend to take. So instead developers say they can do it in 2 years and then end up slipping 6 months at a time for another 2 years and then release a game that was produced in a rush. From the publisher side the game is 2 years late, from the player's perspective it's 1 year early. The success rate of MMOs isn't very different than other games as a % of released titles in this regard. Thus we must understand that game enjoyment is not solely a function of development cost. Kids by the millions happily play club penguin despite faced with kids games that have budgets that run into many times that of the original CP. Millions of players log in to Runescape and derive huge enjoyment despite being faced with untold numbers of options of "better" more-expensive to make MMOs.
Another interesting thing is that we tend to create a self selected industry. When you make an MMO and ask "what would MMO players like next" it's different than if you ask "what would that non-MMO gamer like next?" I'm not saying one is better than the other, but I think question 1 is trying to poach and question 2 is trying to expand the market. I don't know if Runes of Magic, for example, would have 1 million registered players if it wasn't free. Because it's free, it addresses question 2 and instead of having to steal players from WoW, LotRo and AoC it can "rent" them while also getting players elsewhere since there is no up front barrier (cost).
In the end, just like most customer businesses, a small number of successes will have a huge majority of the people. Imagine if I told you I was going to make a new soft-drink and take on Coke. Imagine if I told you I was going to start a retail store and take on Wal-Mart, or a search engine that is "better" than Google. Closer, I suppose, is if I were to start an animation studio and take on Pixar. Most of those companies have 1 or 2 major competitors and that's about it. I don't see any reason to believe that MMOs will be any different. WoW dominates so massively because it is both incredibly high quality and strong in the customer's mind. Thus when people are attracted to MMOs they start playing WoW because "that's what everyone else is playing" (This is a US/Euro-centric view, of course). So if you want to compete at that level you have to make a game that is as good or better AND also compete in the public's mind. As we can see, that has proved to be difficult to do.
In many ways we are still waiting for the "Diablo killer" :). In the industry we tend to chase the new thing in tech, design, art and so on. I think that the average game player cares about this that much. What they want is something understandable, functional and enjoyable. So we discuss and argue the input, the players discuss and argue the output. I think we can learn a lot from Casinos here. Basic gambling games don't change much and the introduction of a new game poses a large risk for the Casino. I.e. never-before-seen-game-idea = massive risk, small-change-to-known-game-idea + superior implementation = smaller risk. Thus they focus on presentation (polish), theme, reward structures, and things like that. There are hundreds of different slot machines, and rules for how they work, but the mechanic is push a button or pull a lever, wait for the result. Yet, a few slot machines will pull in substantially more money than the rest, just like in our industry. I think that implementation of details is much more important than number of features, amount of content, uniqueness design or many of the other things we like to talk about at game conferences :).