Inside the Neo Geo Doom port: sprites, BSP rendering and developer tricks

Running Doom on the Neo Geo sounds like the sort of stunt someone might finish in a weekend. The machine has a Motorola 68000, Doom originally ran on 386-era PCs, and both systems come from roughly the same period. So how hard could it be? Very hard, as it turns out. The real problem isn’t simply that the Neo Geo’s processor runs at 12 MHz. Processor speed matters, but it’s only one part of the story. The bigger issue is that the Neo Geo handles graphics in a completely different way from the PC hardware Doom was designed to use. Doom expects a framebuffer. It calculates the colour of pixels, writes them into memory, then sends the completed image to the display. The Neo Geo doesn’t work like that. Its video hardware builds the screen from sprites and tiles. The main processor tells the graphics system which pieces to display, where to place them, which palette to use and how much to shrink them. It can’t simply draw a wall pixel by pixel. That changes everything.

The Neo Geo was never designed for this

The Neo Geo’s graphics system is excellent at what it was built to do. It can throw large, detailed sprites around the screen, chain them together and scale them down smoothly. That’s perfect for fighting games, shooters and arcade action titles packed with animated objects. Doom asks for something else.

Its renderer works with vertical wall columns and horizontal floor spans. On a PC, the engine samples textures, calculates lighting and writes the result into a bitmap. The Neo Geo’s main CPU can handle the maths, but it can’t freely read the cartridge graphics as texture data. It also has nowhere convenient to store a full screen of calculated pixels. The processor can select prepared graphics. It can’t treat the graphics ROM like a giant image file and sample any pixel it wants.

That means a straight source port won’t work. You can compile parts of Doom’s code for the 68000, but the original renderer has no practical way to produce its final picture. Developers had to replace that renderer with something built around the Neo Geo’s strengths. They didn’t make the machine behave like a PC. They made Doom behave like a Neo Geo game.

One approach turns tiles into giant pixels

The simplest solution uses the Neo Geo’s fixed tile layer as a crude framebuffer. Instead of drawing individual pixels, the engine treats each visible tile cell as one large block of colour. The result is extremely low resolution, but the basic image still reads as Doom. Walls rise and fall as the player moves. Enemies appear in front of the camera. Weapons sit at the bottom of the screen. The automap works, and combat remains recognisable.

This approach works because it cuts the rendering problem down to size. Rather than calculate hundreds of thousands of pixels, the engine only needs to choose a relatively small number of tiles. Each tile acts like a chunky pixel.

The trade-off is obvious. Fine detail disappears. Textures become rough patterns, distant objects become hard to read, and the entire scene takes on a blocky appearance. Still, it proves an important point. Doom’s game logic doesn’t need a conventional display. As long as the renderer can turn the world into some kind of visible grid, the rest of the engine can keep running. It’s clever, but it’s also the bluntest solution.

Another version uses a raycaster

A second approach leans much harder on the Neo Geo’s sprite hardware. This renderer divides the screen into narrow vertical columns. Each column becomes a tall sprite strip, usually only a few pixels wide. The engine fires a ray into the map for each column, finds the nearest wall, calculates its distance and then adjusts the height of the corresponding sprite.

Near walls appear tall. Distant walls appear short. The Neo Geo’s scaling hardware does much of the visual work. The 68000 calculates where a wall should appear, then the graphics system shrinks a prepared strip to the right size. This technique feels familiar because it resembles the renderer used by Wolfenstein 3D. It’s fast, predictable and well suited to hardware that likes vertical sprite strips.

It also fits neatly within the Neo Geo’s limits. The screen is 320 pixels wide, so a renderer using four-pixel columns only needs about 80 main strips for the world view. That leaves room for the weapon, interface elements, enemies and effects. But a raycaster comes with restrictions.

Traditional grid-based raycasters prefer walls aligned to a simple square map. Doom’s levels don’t follow those rules. They contain angled walls, different floor heights, open areas, lifts, windows and sectors connected in ways that don’t fit a basic tile grid.

A raycaster can create a convincing Doom-like game on the Neo Geo, but it struggles to reproduce original Doom maps accurately. That’s where the most advanced approach becomes interesting.

The key breakthrough was keeping Doom’s BSP system

Doom doesn’t store its levels as simple grids. It divides them using a binary space partitioning system, usually shortened to BSP. The BSP tree helps the engine decide which parts of the map appear in front of the player and which parts remain hidden. It splits the level into smaller regions, then lets the renderer process nearby geometry before distant geometry. That ordering matters.

A Neo Geo renderer can’t afford to throw every wall on the screen and hope the hardware sorts it out. The machine has a limited number of sprite entries, and it can only display so many sprites on the same scanline. If the engine creates too many overlapping strips, parts of the image can vanish.

So the renderer has to reject hidden geometry early. The advanced Neo Geo implementation keeps Doom’s BSP data and uses it every frame. The 68000 walks through the tree, checks which side of each partition the player occupies and processes the nearest visible regions first.

Then it starts closing screen columns. Suppose a solid wall covers part of the view. Once the renderer knows that wall completely fills a group of columns, it can ignore anything behind it. There’s no point generating more sprite strips for geometry the player can’t see. That saves time, memory and sprite entries.

It’s one of the most important parts of the design. The engine doesn’t measure difficulty by the total size of the map. It measures difficulty by how much visible geometry survives the culling process. A large level can run well if most of it stays hidden. A smaller room can become expensive if it contains long sightlines, several openings and many overlapping structures.

Walls become commands, not pixels

After the renderer finds a visible wall, it projects that wall onto the screen. It calculates where the wall starts, where it ends and how tall it should appear. Then it breaks the wall into vertical strips.

Each strip becomes a small hardware command. The command includes a screen position, a graphics tile, a palette and a shrink value. The Neo Geo’s video hardware reads those commands and assembles the final image. The CPU never draws the wall itself.

That distinction is crucial. On a PC, Doom’s renderer produces pixels. On the Neo Geo, the renderer produces instructions for the sprite system. It’s closer to building a display list than painting a screen.

The engine also separates calculations from video updates. During the BSP pass, it stores the required strips in a software buffer. Once the frame is ready, it writes the completed list to the Neo Geo’s video memory in a controlled batch. This keeps the output stable. It also gives the renderer a chance to sort strips, enforce limits and reduce visible seams before sending anything to the graphics hardware.

Texture mapping had to move into the build process

There’s another problem. The 68000 can’t freely sample wall textures from the cartridge graphics ROM. So developers moved much of the texture work offline. Before the game runs, conversion tools extract maps, wall textures, sprites, palettes, sounds and other assets from Doom’s data files. The tools then rebuild those assets in formats the Neo Geo can use directly.

Wall textures become sets of prepared vertical tiles. The renderer doesn’t ask for an arbitrary column of pixels at runtime. Instead, it selects the closest matching tile that already exists in the cartridge. That saves a huge amount of work.

It also explains why the build system matters so much. In a normal game project, asset conversion can feel separate from the engine. Here, it forms part of the renderer itself. The tools perform jobs that the console can’t handle during play.

The same idea applies to lighting. Doom usually darkens surfaces based on distance and sector brightness. The Neo Geo version can’t calculate and redraw every pixel, so it uses prepared palette ramps instead. A nearby wall uses a brighter palette. A distant wall uses a darker one. The effect isn’t identical to the PC version, but it creates depth without expensive per-pixel maths.

Floors and ceilings caused the biggest headache

Walls are a good match for the Neo Geo because Doom already draws them as vertical columns. Floors and ceilings are much harder. Doom renders those surfaces as horizontal spans. The Neo Geo doesn’t have a bitmap screen, an affine background layer or a general-purpose texture mapper. It can’t simply stretch a floor texture into perspective. The solution again moves work offline.

Development tools generate perspective tables and prepared floor patterns for different viewing angles and distances. During play, the engine chooses from those patterns rather than calculating every floor pixel from scratch. This uses more cartridge space, but it saves CPU time. That’s a sensible exchange on the Neo Geo. The system can address large graphics ROMs, while its working memory and processor budget remain tight.

Distant floor and ceiling rows can also be simplified or removed. A dark gradient or flat backdrop hides the missing detail near the horizon. When performance drops, the renderer can shorten the visible distance before it starts damaging nearby walls. That makes the slowdown easier to control. Instead of losing random sprites across the image, the engine trims detail in places where players are less likely to notice.

Sprite limits shape every frame

The Neo Geo can handle hundreds of sprite entries, but the headline number doesn’t tell the whole story. The more dangerous limit is how many sprites can overlap on a single horizontal line. A first-person view creates exactly that kind of pressure. Tall wall strips may cover most of the screen. Add enemies, a weapon, doors, effects and interface elements, and the renderer can hit the scanline limit quickly.

Developers have to budget every strip. That means adjusting column width, viewing distance, floor detail, actor visibility and the number of transparent layers. Narrow columns improve image quality, but they consume more sprites. Wider columns look rougher, but they reduce the workload.

There’s no free setting. The engine can also cap how many layers it draws through windows, doorways and other openings. Once a column reaches its depth limit, the renderer stops adding geometry behind it. Again, the goal isn’t perfect accuracy at any cost. The goal is a stable image within the hardware budget.

Why this works

Doom runs on the Neo Geo because developers stopped trying to recreate the PC rendering process. The 68000 handles the parts it’s good at: movement, game logic, BSP traversal, visibility tests and fixed-point projection. The video hardware handles the parts it’s good at: fetching prepared graphics, positioning strips and shrinking them vertically.

Everything expensive gets moved away from runtime where possible. Texture conversion happens during the build. Perspective floor data gets prepared in advance. Lighting becomes palette selection. Hidden walls disappear before they reach the sprite list.

The final result isn’t a conventional software renderer. It’s a translation layer between Doom’s world and the Neo Geo’s sprite hardware. That’s the real technical achievement. The machine never draws Doom one pixel at a time. It breaks the scene into visible vertical pieces, converts those pieces into sprite commands and lets the graphics system assemble the view. Once you understand that, the project stops looking impossible. It starts looking like a carefully planned hardware conversion.