Interestingly enough, the cpu in the NES was the same cpu used in the
Atari 2600. Everyone knows that the games on the NES, however, were
substantially better (at least visually).

What about clock speed? It turns out that the NES ran about 600 Hertz
faster than the Atari 2600. So they might as well be the same.

Looks like it's all in the memory.

To take a peek, I modified the Nintendulator NES emulator source in a
way similar to the way I modified the Atari 2600. The NES hardware is
substantially more complex than the Atari 2600. To handle cartridges
containing a reasonably significant amount of ROM data, the NES used
various memory mapping techniques that the Atari did not.

First things first. What do memory reads look like?

Well, something like this:

         

The image above is a snapshot of the active execution of the NES game
Megaman. As the NES reads data from the Megaman cartridge ROM , the
location of the read is displayed visually.

So what do all the colors mean?

Here, we've got the cpu reading cartridge (so-called PRG) data. Like the
Atari, some of this data is read very frequently. I've also made a distinction
between data reads and instruction reads. So, the legend is as follows:

Blue: Infrequently read instruction data.
Green: Infrequently read ROM data.
Red: Frequently read instruction data.
Yellow: Frequently read ROM data.

The alpha channel was used to encode temporal information, with colors
fading over time.






Hacking the abstraction

Visualizing memory activity in this way reveals some interesting
characteristics of the NES. For example, at certain points of the game,
we can observe the memory mapper in action, paging in new segments of
PRG data. This data is accessed in a special way within the NES. In
particular, sprites and color palettes are stored in well-defined locations.

So?

So, imagine playing a NES game with the ability to scribble into the sprite
buffer or color palette dynamically.

What does that look like?

Well, it obviously depends upon what you scribble into those areas.

Nothing? The contents of the registers? Noise?

         

The emulation abstraction not only allows the direct manipulation of visual
content, but gameplay as well. This is accomplished through the dynamic
manipulation of emulated peripheral devices such as Galoob's Game Genie.

Basically, the Game Genie works by allowing the user to enter a code, which
tells the Genie to interfere with PRG reads done by the CPU.

Originally limited to 3 of these such codes, I've been able to modify the NES
emulator mentioned above to allow any number of codes to be used. I'm even
able to modify these codes dynamically as the game is played, using various
input devices, including my Kaoss Pad audio effects controller.

While I'd like to add a short video segment showing these modifications in action,
the simple act of displaying memory read information alongside a running emulator
already brings my 3GHz laptop to its kness.

However, here is a short video of what color palette and sprite buffer scribbling
can accomplish.

The emulation mechanism also provides a way to get at the screen pixels after
we've scribbled into the sprite buffer. This allows us to apply additional
post pass image processing techniques as the game is played.

Choose your favorite image processing technique, and apply:

         


So what's the point?

Much of this all started with me looking to add visuals to my live performances.
As mentioned above, changing the way games are displayed is now controllable.

As a result, anything can now be used as input, from touch sensitive interface
gesturing to real time audio processing. In other words, my latest project.