The DM starts with an interactive palette similar to that of Sim City, though with the ability to zoom down to person scale. For the present, since we can eliminate most of the automated portions of the Sim City game, we can spend more effort on zoom and detail. The DM terraforms the environment, lays down watercourses and roads, plants villages with houses or scatters houses, farms, pastures and forests. The DM digs a few dungeons, arranges some lairs, specifies monsters if desired and so on. This is all done without the computer telling the DM what to do, or restricting the DM from fantasia landscapes or things that might seem irrational. The DM has total godlike powers, and is not restrained in any fashion from building the world up from scratch in any way. Even if a pre-made template is provided, the DM can dabble with it at any time. The tool never requires that the DM surrender this godlike ability to reshape, change, modify or otherwise further detail this world. Ever.
The DM does all this on a screen that only the DM can see. The players are then placed into this environment, wherever the DM wishes to place them. The DM can move them, change the environment around them, give them stuff, kill them with lightning, whatever the DM can do in real D&D ... provided we have come to the point where that App has been added to the tool.
The players all view the world from the position from where they see out of their eyes. They can turn in any direction, look down, look up, lift their arms, walk forward, run, etc. This is all accomplished with whatever wii/kinect convenience required - not as a means to replace elements of the game such as hitting or hiding in shadows with physical actions, but merely as the best means to move your character around. I don't advocate the tool being used to replace your characters ability ... unless, of course, somehow your actual ability could be successfully translated so as to downgrade or upgrade it to make it level with your character. Hum, wonderful thought that. You swing marvelously, but your mage stills swings like a girl ... but sort of in the same way you just did, just now.
So your players look at the world around them, the world the DM created. The DM makes the most of the tool and carpets the ground with one of many kinds of grass, adds a few of many kinds of trees, chooses the weather and so on. The DM picks rain, or the DM uses the tool as a random modifier - which the DM can override at will, just like a die roll - and we'll say rain appears. The players draw their cloaks closer and watch their own body temperature go up on one of the many, many readouts available for them which the DM cannot see. That last would be really interesting, though of course it could be optional for control freak DMs. The players couldn't cheat anyway, the tool would have to be hacked. The DM would have to ask the players ... though there would be visual evidence the DM could gather which would indicate Nack the Thief wasn't doing so well.
The DM could view the party from any position except through the players' eyes. We want to discourage the DM from having too much information. The players could move things around on their body, hide things in their clothes, pass things to one another, etc., which could be seen by the DM if the DM were paying attention. Otherwise, the DM just wouldn't know.
There would be no more lengthy discussions about who had what sword and where did it come from and how fast can you draw it and can you carry 4,765 g.p. and so on. All of this would be handled by the tool, the players all finding what they can and can't do spelled out for them. And all those limitations could be modified by the DM, at will, at any time. If the DM wanted to change any perameter about the weight limit of a 40 lb. halfling, for instance, no power in the tool would be able to stop the DM from doing so. This would apply universally to all conditions, always. Like I said, godlike powers where it comes to affecting the environment of the game. The only limitation is the player.
Listen, do I really need to keep spelling this out? Have you played this game? If you had a tool like this to manage every element, wouldn't you want it? If you had one that let you make the combat turn based or ongoing? If you had one that let you adjust the speed of the characters as they travel blindingly across the landscape? If you had one that let the DM create pockets of goods and - like Sim City - that started bits of trade and price gouging and so on, all modified as the DM eradicates a mine or adds three thousand acres of cropland?
Picture the DM telling the player about the ogre on the player's right - not because the DM just invented said ogre, but because the ogre is approaching and the player hasn't looked in that direction. "Oh right," says the player. "I was going to deal with it next round."
Picture the DM creating a temple from hordes of templates, but then saying, "Well, its not exactly like this, but it is this big and just picture it with a big 'C' on the front" ... as the players all look UP at it.
Picture the player's rolling through their list of spells and not having to look up the area of effect or the range, but just trying it with the various tools in the kit and learning from practice what the spells do ... and watching their marvelous effect.
All of this, repeat, ALL of this, happens when the players and the DM manipulate the tool to make it so. NOT when the tool decides the player can or cannot do something. If the DM doesn't put the restriction there, there is no restriction. And if this seems like there are too many restrictions that the DM would need to add, consider again the argument about having some things standardized. But standardization still wouldn't mean the DM couldn't change the standard once the tool was in hand.
There could always be certain random things. The DM could specify a monster to kill Nack, or the DM could specify the monster to move towards any person in the party, provided another monster wasn't already there. The tool would shift the monsters around, and the DM could nudge them. At any time the DM would stop the monsters from attacking, or change who they attack, and so on. Always, always, always, just like the real game.
This can be done. This ought to be done. And those people who are screwed in their heads about how this would destroy imagination really know nothing about imagination. Seriously. Things don't get less interesting when you increase the possibilities.