I feel I should make clear the "challenge" the DM faces in attempting the design formulation that I've been proposing: 1) that the game should be hard enough to play that players can lose heart or contentment as the game progresses; 2) that 'fun' is not a design feature; 3) that the game setting be fixed and reasonably predictable; and that 4) the player's actions should result in a change to the setting, which then adjusts and directly addresses what the players have been doing.
Is this hard? Yes. Extremely. Is this beyond the ken of most DMs? Without question. And let me address that for just a moment.
We cannot expect others or even ourselves to step into an interpretation of a game that is distinctly different from what we have expected it to be for a long time and then do well. What I've been arguing flies flat out in the face of "role-playing games" as they've been sold and promoted since their inception. The potential difficulty of the game has never been embraced by the makers of the game nor the company that ultimately acquired the property from those makers. In every case, in every media release, in every publicized demonstration of the game, the principal goal has been to make the game accessible to a wider buying public, a public that can't be counted on to attempt to play a difficult game that can't be easily explained or mastered. There's no benefit in attempting to promote this, not by the company continuously re-inventing the game, not by clubs who count their importance by the number of their attendees, not by game stores who care first and foremost about sales and certainly not by virtually every game-master discussing the game on the web, having been steeped in the marketing culture of Cons, modules, splat books, fictionalized role-playing novels, podcasts, blogs and a nostalgic background in junior high school and high school introduction.
D&D, and the other role-playing games that followed, can't be properly compared to other games such as baseball, hockey, golf, Monopoly, RISK, chess or bridge, etcetera, because those games can be easily explained. The "hard" part inherent in those games depends upon the actual skill of of the participants: but take six-year-olds and teach them hockey and they can enjoy the "difficulty" of the game against each other, even if a group of seven-year-olds would wipe the ice with them.
But D&D isn't like that. The actual process of the game is more difficult than the play. Once it has been decided to kill the chimera, further game play is dependent on whether or not one survives. The player can add information as the fight goes on, but most of the actual process is spent in determining chance of hitting, weapon used, effectiveness of weapon, personal skills, losses, niggling details surrounding every part of the experience and, finally, waiting to see which lives or dies, the player or the chimera.
Video games manage this process by greatly simplifying all the details so that the player need only press a button; the speed at which the battle is resolved is maximized and so twenty seconds or so of pushing a button at a hard target will sort out the result. D&D isn't like that. Much of D&D, no matter how simplified we make the game, is accounting.
This means that, for most people playing, the quality of the game is based solely on how much accounting the players and the DM are prepared to do. Less rules, less accounting. More DM fiat, less accounting. More story-driven, character-driven parley and general in-character chatter, less accounting. And if we get rid of details like experience gained, hit points, levels, 'to hit' rolls and other features of combat, all of which I've seen proposed for throwing into the trash, less accounting still.
Those who describe "a better game" as one in which accounting takes place on a grander, deeper scale (meaning that rules and details are added), are considered pariahs and abuse-mongers among the much larger populace of participants who have determined that "better" describes a game in which the character sheet is drastically reduced in scope and detail. These are people who don't want to play golf, or hockey, or anything else that requires sweating. These are people who, when they buy a game, immediately seek out the "cheats" pages on the internet because even the automated process of having a computer calculate out pleasure and rewards for time spent can't wait, but must have their rewards right now.
Because of all this, we can't merely describe the sort of D&D I'm proposing with those four points that started this post as "challenging." Let's be correct. D&D as I try to play it is unpleasant, unmanageable, unfair to the DM and the player, unreasonable in its rules' systems and, for the most part, incomprehensible in terms of how it produces "fun."
Which it doesn't. My game produces strain. My game produces panic. My game promotes feelings of inadequacy and frustration and, occasionally when a party can't get its shit together, exhaustive periods of boredom. When new players come into my game, they're familiar with my online persona and they presume, not always rightly, that I'm waiting in the tall grass with a knife, waiting to gut them. And even though I will occasionally step out and say, "Relax, I'm not just going to kill you for no reason," they don't believe me. Largely, because they have no idea what would count as a "reason" and, just in case, it's best to think everything does.
They have no idea because I'm perfectly ready to blindside a party with something they didn't expect, I'm good at that, and people have personally witnessed me doing it.
So why? I ask you. Why would anyone want to play in a game like mine? It's just a lot of accounting and rules-making and reading and memory-work, just to stay on top of what your player can and cannot do, much less trying to figure out what the NPCs might want to do, or how the world itself fits together.
Screw all that! Please, just give us a lot of description and a king who wants a thorough back and forth with us about our backstory.