In my experience, there are two types of challenges that are related to any type of game, the design elements and the play elements. These can be broken down further but they are the primary divisions. With historic games, you have some issues that are unique to historic and some that historic share with other types of settings. First, let’s cover design.
How much is enough? This is when you dig into the research of the setting and find there is a great deal more to the period than you would think. There always is. This leads to problematic “what should be included and what should be set aside?”. This is not insurmountable but a very different problem from something like writing a fantasy setting. With fantasy settings the question is more “What do I need/want to include?”. With historic you need to anticipate a great deal of content expectations. After all, there is only one expert on Gaea (my fantasy game) or maybe two (my wife) but you face a wide range of experts with a historic setting. You may “ruin” the game if you leave out a point you though unimportant.
Marrying playability with documented history. This one could be argued to be shared with any setting that has a great deal of canon, such as Star Trek or Star Wars but I would argue it is more problematic than those. Historic content was not made to be entertaining, cinematic or literary source usually are. You need to be able to, as a designer, determine what will entertain and what will inform and what will bore your reader. Then, craft what you have into a useful and entertaining setting. Raw history can be rife with adventure or loaded with detailed discussion of weights and measures. Both can be entertaining to the right people but the trick here is to craft the elements so they work with a game. Some succeed, some don’t.
Those challenges in the play arena.
Overcoming the “OMG!!! It’s history and I will die of boredom” and it’s brother “I don’t want to do 200 hours of research so I can play a game”. This is often one that historians just refuse to see. They believe that if someone is going to play a “historic” game then they must pay their dues. Some folks just want to play Gladiator or Spartacus as they remember it. They want to have a movie night followed with some gladiator on gladiator combat. This is fine. It is also great if the GM and players want to do that research and get into the period. However, you often get resistance before the game is engaged due to the perception, fostered by AP, some historians and bad highschool classes that paint historic settings as work and boring. Some tricks to overcome this is to supply all the research upfront in one book. Also, add alt-history or fantasy elements to pull it out of that historic setting to a historic genre. This can lead to the second challenge.
Expert syndrome or setting fetishism (thanks Clash) where the game is derailed or lessened by the need to be the expert on the setting. Certainly not unique to historic settings, I have found they are more common in such settings. This stems from the issues, I believe, that there are more people aware of history and that history is often a series of opinions on what happened. That is to say, history seldom is a set fact that is incontestable. Depending on the books you read, the experts you consult, you can have very different views on what happened. By its nature, this breeds arguments and dissension. Again, nothing that cannot be overcome. First, and good advice for any game, manage the expectations of the game. Explain that what we are about to engage in is not a recreation of history but an game based in a historic period. Then discuss the expectation of the group. Perhaps they are expecting exacting detail but you as the GM want a light hearted romp through the middle ages. Everyone needs to be on the same page. Second, be sure to communicate how much feed back during play is expected. Will discussions on the period be encouraged or saved for after the game? That should be answered up front. Also, be upfront about the setting that is being played. It can be tempting to try and adjust on the fly to the desire of the group but it can also be disastrous to attempt to increase historical veracity in mid game.
Finally (in my list, there certainly could be more), the level of research that is often required by historical games is truly daunting. I think this is one of the largest hurdles to this niche of a niche game genre. Historical genre games often alleviate this by allowing multiple mediums to be used (movies, fictions books, folk tales). This is the most effective way I have found to introduce those who aren’t too interested in the genre.
All these challenges are certainly able to be overcome but I believe that they are also some of the real reasons that keep historic games from being more popular. Much like RPGs, it may just be the nature of the game that keeps it from wider popularity.