03725: popeye: Incorrect sprite rendering in Popeye

ID	Category [?]	Severity [?]	Reproducibility	Date Submitted	Last Update
03725	Graphics	Major	Always	Feb 9, 2010, 22:29	Jan 21, 2019, 06:41

Tester	Oliver_A	View Status	Public	Platform
Assigned To	smf	Resolution	Fixed	OS
Status [?]	Resolved			Driver	nintendo/popeye.cpp
Version	0.136u2	Fixed in Version	0.194	Build
		Fixed in Git Commit		Github Pull Request #

Summary	03725: popeye: Incorrect sprite rendering in Popeye
Description	On the real Popeye hardware, sprite colour 0 is not transparent with respect to overlapping sprites. It's only transparent with respect to the background graphics. When two sprites overlap in Popeye, the sprite with the highest priority COMPLETELY overdraws the sprite underlying it, including the transparent area (pixel with colour value 0). The result is that instead of proper sprite prioritzing, overlapping sprites are overdrawn like chunky blocks. The sprite rendering code in MAME wrongly treats sprite colour 0 as transparent, which is the case in almost all games with hardware sprites, but NOT Popeye.
Steps To Reproduce	Play the original Popeye PCB, and see what happens when two sprites overlap. For instance, let Popeye run into Brutus. You'll see that when Popeye overlaps Brutus, he completely overdraws him, with no pixel of Britus visible below the Popeye sprite block.
Additional Information	I have included screenshots from my original Nintendo Popeye PCB which show the issue. The falling Popeye, when losing a life, completely overdraws the Bluto sprite. Note that it's quite hard to photograph, but anyone else owning a real Popeye will confirm it. The PCB is 100% working, this is no hardware glitch. It's the way how the sprite hardware is designed to cope with high resolution sprites.
Github Commit

Flags	Verified with Original
Regression Version
Affected Sets / Systems	popeye

Attached Files	Popeye_Sprite.zip (1,948,017 bytes) Feb 9, 2010, 22:29
	popeye_pens.zip (930 bytes) Feb 10, 2010, 00:05
	101_9901.JPG (358,767 bytes) Feb 10, 2010, 12:14
	popeye1.jpg (147,393 bytes) Feb 10, 2010, 18:14
	popeye2.jpg (35,733 bytes) Feb 10, 2010, 18:14
	popeye3.jpg (173,323 bytes) Feb 10, 2010, 18:15
	popeye.c (11,172 bytes) Feb 10, 2010, 23:27 [Show Content]

No.05679 Haze Senior Tester Feb 10, 2010, 00:05	if you apply the patch I just uploaded to the source does it give the desired results?
No.05680 Tafoid Administrator Feb 10, 2010, 00:07 edited on: May 31, 2012, 05:08	If you view about 1:43-1:44, you can see Popeye falling and blocking out Bluto pretty well. Confirmed.
No.05681 Tafoid Administrator Feb 10, 2010, 00:13	Just tried your diff, Haze - it seems to mirror was we see in the photos and video. I'll let Oliver check it too..
No.05682 Oliver_A Tester Feb 10, 2010, 00:34	Yep, the video shows how it looks on the real hardware. I'm going to check the patch tomorrow and let you know if it's correct in all cases. Thanks!
No.05684 Oliver_A Tester Feb 10, 2010, 12:13 edited on: Feb 10, 2010, 12:17	Okay, here are my finding: The sprite drawing is now correct (popeye blocks out bluto), but there are still some issues left. How to reproduce: Start the game, wait until the introduction ends. Olive throws her first heart, it's going to overlap the basket in the middle. On the real hardware: heart is drawn IN FRONT of basket for 1/2 second when it touches the basket, then drawn BEHIND for the rest of the time. On the patch: heart is always drawn BEHIND the basket. Now wait until Olive throws her second heart. It's going to overlap the punching bag. On the real hardware: first, heart is drawn BEHIND the punching bag for 1/2 second when both collide, then suddenly IN FRONT of punching bag for the rest of the time. On the MAME patch: heart is always drawn BEHIND the punching bag. I have attached a new photo which shows the heart being drawn IN FRONT of the punching bag. This case NEVER OCCURS on the patched MAME. It seems the game code switches sprite priorities during game, which are somehow still not reflected correctly. But otherwise, you see on the photo that the heart sprite blocks out the punching bag, like Popeye blocks out Bluto. Otherwise, it now looks MUCH MORE close to the original hardware. Thanks Haze!
No.05685 Oliver_A Tester Feb 10, 2010, 12:23 edited on: Feb 10, 2010, 12:25	You can also see this behaviour in the video Tafoid posted here at 1:21-1:25.
No.05689 Haze Senior Tester Feb 10, 2010, 13:13	I'm not really sure how the further differences you explain would be achieved. I doubt the hardware is sorting the sprites by vertical position before drawing (it would no doubt cause other artifacts too), but MAME is always drawing the sprites in list order, and that's where the priority comes from, the ordering of the list. Thanks for the additional details, but I don't know how to further improve the fix to match them.
No.05690 Oliver_A Tester Feb 10, 2010, 13:33 edited on: Feb 10, 2010, 13:33	I'll check the schematics to figure it out. The problem with Popeye is that the games uses four 82S100 programmable logic chips for video/sprite generation, which are some kind of black boxes when reading the schematics.
No.05693 Haze Senior Tester Feb 10, 2010, 16:25	Yeah.. it's quite typical that video hardware is 'black boxed' I'm starting to wonder actually, is the punchbag actually a sprite. If not maybe the hardware does something odd, like samples the priority of the pixel at the very top left of the sprite to decide if to draw above / below the frontmost layer for the entire sprite. That would seem a bit odd from a hardware point of view tho, and I prefer the code I write to have some logical sense.
No.05694 Oliver_A Tester Feb 10, 2010, 17:08 edited on: Feb 10, 2010, 17:22	The punchbag is indeed a sprite, since it's also a "high resolution" object. Popeye basically has 3 graphics layers: the blocky background, the text layer and the sprite layer. The sprites are the only objects which are rendered at 512x224. Note that the punch bag also changes its palette when Popeye ate spinach, so it uses the same colour attribute as the Popeye sprite. I'm still in the process of figuring out the sprite hardware, and it seems to be an unusual design. Normally, a game from 1982 would have a sprite-line buffer, where all sprite pixels for one raster line are buffered. Popeye is different. Instead of a sprite pixel buffer, it has a sprite attribute buffer, which, instead of storing pixel values, stores memory addresses for the graphics roms. This buffer is scanned horizontally during display, and once a sprite starts, 8 shift registers are loaded which form 16 2bpp sprite pixels. If 2 sprites overlap, the shift registers are reloaded again, which is the reasons why sprites can not overlap each other without canceling the previous sprite. However, I am still unclear how it works 100%. Most probably, the sprite hardware is scanning the attribute memory in an unusual way, perhaps to save some chips. I'll post here when I have figured it out.
No.05695 Oliver_A Tester Feb 10, 2010, 18:37 edited on: Feb 10, 2010, 20:05	The sprite hardware has the following basic memory access scheme: \|SPRITE MEMORY \|--DMA-->>\|SPRITE BUFFER\|--->>\|SPRITE GRAPHICS BUFFER\| In picture 1, you can see a DMA circuit, which copies 1024 bytes of sprite attribute memory to a buffer. The 3 LS161 counters (1F,2F,3F) generate chip selects and a 8 bit memory address for the sprite buffer on picture 3. The outputs of the LS161 counters are also translated by 2 Programmable Logic Arrays (3E, 4E) into memory addresses, which are output on the Z80 bus. The DMCS signal is also generated by the PLAs and is used to select a 2K SRAM chip on picture 2. What the circuit basically does is copying 1024 Bytes from the 2K SRAM chip from picture 2 into the 8 256x4bit SRAM chips on picture 3. The data in the 8 SRAM chips in picture 3 is now used as a 256x29bit memory array to retrieve horizontal/vertical position, sprite image number, palette index and horizontal/vertical flipping information. Vertical data in combination with the vertical video raster position is used to determine IF a certain sprite is to be displayed on a certain line, and the horizontal position is taken as a memory address for the next step. With this information, the memory address of the sprite graphics is formed and copied into 4 64x9bit SRAM chips on the right side of picture 3, which are the sprite graphics memory address buffers for one raster line. Those buffers hold the graphics address and palette data for one entire video line. All those 4 memory chips are treated as a one 128*19bit memory array, and the information is used to address the graphics roms for the sprites and reload the sprite shift registers on a certain horizontal position. Consequences: Sprite priorities are determined by the way how the DMA circuit is copying the data from SPRITE MEMORY into the SPRITE BUFFER, which is determined by the 2 programmable logic chips 3E and 4E. That means, without reverse engineering those chips, the order in which sprites are processed is unknown. I have posted this in detail here, so me or another person might have enough information to start with at a later date. NOTE: 3E and 4E are marked as "PLA", but are most probably PROMS which could be dumped. On my board, those are two 16 pin chips where the labels were sanded-off.
No.05699 Oliver_A Tester Feb 10, 2010, 23:26 edited on: Feb 10, 2010, 23:31	So far, I still haven't found out the priority problem. However, I found another bug in the driver. The way how it is determined whether a sprite is drawn or not is wrong. Sprites with an X-Position of 0 display perfectly well. The hardware does not draw sprites when the lower 3 bits of attribute+3 are zero. I have uploaded a modified video/popeye.c file which corrects this error. It's right under the last picture. Do you guys believe me, or do I need to upload more schematics with red arrows which explain what circuits cause this behaviour?
No.05700 Haze Senior Tester Feb 11, 2010, 01:14 edited on: Feb 11, 2010, 01:15	evidence helps, for future reference, even if it sounds like you know what you're doing :-) (although please note, when you get to schematic level you go beyond my understanding, I can work with reverse engineering based on expected results and logical code only, others are better with schems)
No.05701 Oliver_A Tester Feb 11, 2010, 01:49 edited on: Feb 11, 2010, 09:52	I have figured out how it works. The hardware is ordering the sprite priorities the following way: We need 2 memories: 1.SPRITE ATTRIBUTE MEMORY (the memory where the Z80 writes the attributes) 2. SPRITE LINE BUFFER, 64 entries (for odering the sprites according to their x coordinates) Step 1: sprite ordering First, the SPRITE ATTRIBUTE MEMORY is scanned from 0 to maximum number of sprites. The hardware reads the X coordinate of each sprite, shifts it right by two (as said before, only 64 entries are possible), and uses it as an index for the SPRITE LINE BUFFER. Compare the Y raster position with the Y sprite position. If the sprite is in range, then ALL sprite attributes are copied to this memory location. If 2 sprites share one shifted X-Coordinate, then the latest sprite entry overdraws the earlier entry. Step 2: sprite rendering In the second step, the hardware scans the SPRITE LINE BUFFER from 0 to 63. If the lower three bits of buffered sprite attribute+3 are NOT ZERO, then a sprite has been positioned at this location. Draw the sprite line with all the buffered attributes at x*2 location (the real, unshifted 8 bit x location) on the sprite bitmap. When the SPRITE LINE BUFFER was scanned and drawn to the bitmap, clear the SPRITE LINE BUFFER, increment the raster line and go back to Step 1. When the last visible raster line has been drawn, set it to zero and go back to Step 1. That's how the real hardware renders the sprites. Note that in order for this to work, the sprite rendering code needs to be scanline based, and not frame based. If you need evidence, I can only provide it on schematic level. ;) EDIT: corrected some mistakes.
No.05702 Haze Senior Tester Feb 11, 2010, 02:25	I'll try implementing it as per your explaination
No.05705 Oliver_A Tester Feb 11, 2010, 09:42	I have updated the explanation to correct an error. The SPRITE LINE BUFFER has 64 entries. I was confused by the 4 chips on the board, and thought they were used as 128 entry configuration. What seems to be the case is that instead of a 128 entry buffer, the hardware uses TWO 64 entry buffers, which are being displayed and erased in reversed order depending on the LSB of the vertical raster counter (if you can read the schematics, you can see this mechanism on picture 3, the 3 LS08 AND gates connected to pin 9-11 of the memory chips, blocking out the colour code being written). For the software implementation, the existance of two 64 entry buffer being alternatively written and erased does not matter, since erasing the buffer can be done in one time step in emulation. The modified explanation I gave in my previous post should yield a correct display result.
No.10236 AWJ Developer Feb 4, 2014, 12:13	I've fixed the sprite drawing to skip sprites with color == 0 rather than xpos == 0. Implementing scanline-based sprite rendering on this particular driver is IMO a nonstarter until the MAME core handles interlaced screens correctly (right now MAME treats a 448i screen like Popeye's as 448p, with a horizontal frequency double the real hardware's; this is why consoles that support interlace and raster effects like the SNES and Megadrive have to bypass the core and do their own scanline timing)
No.14676 smf Developer Jan 22, 2018, 10:31	I have pushed a change where the sprites are rendered according to the description from this mame testers report https://github.com/mamedev/mame/commit/107147fd3d22b126c6fec365dd9856095d523cba
No.16039 MetalGod Senior Tester Jan 16, 2019, 12:43 edited on: Jan 16, 2019, 12:45	Fixed in mame 0.194 " Implemented sprite priority & clipping according to description in MT03725. Note: On the real Popeye hardware, sprite colour 0 is not transparent with respect to overlapping sprites. It's only transparent with respect to the background graphics. When two sprites overlap in Popeye, the sprite with the highest priority completely overdraws the sprite underlying it, including the transparent area (pixel with colour value 0). The result is that instead of proper sprite prioritzing, overlapping sprites are overdrawn like chunky blocks [smf, Oliver_A]. "