Yellowstone Glitch, Part 4: The Plot Thickens
I’ve discovered something very useful about the signal glitching problem with Yellowstone: I can reproduce the problem (or a form of it) without booting a disk, and even without any disk drive attached. After many hours, I think I’ve narrowed it down to this:
If the CPU reads the value $FF from any location in Yellowstone’s RAM, it might cause a glitch on many different signals simultaneously.
Reading $FF from Yellowstone’s ROM doesn’t cause any trouble. Reading other values with seven 1-bits from Yellowstone’s RAM may cause a glitch, but it’s less likely than when reading $FF. Reading values with six or fewer 1-bits from RAM doesn’t seem to be a problem.
I wrote a tiny program to store $FF in RAM and repeatedly read it back in a loop, and then examined a variety of board signals while the program was running. There were nasty glitches everywhere. But when I zoomed out the time scale, I noticed that the glitches appeared in regular clusters. And zooming out still further, I made this discovery about the behavior when continuously reading from Yellowstone RAM:
Glitches appear in a roughly regular pattern with a 16.5 ms period. Within that pattern is a second pattern with a 62 us period.
Yup, it’s a 60 Hz pattern with a 16 KHz pattern overlaid. Smells like NTSC video! But I don’t think it’s literally video interference, as I’ll explain in a moment. My guess is it’s an issue with data bus traffic related to the Apple IIgs video refresh.
One more important detail:
Glitches appear almost immediately after the ‘245 is enabled to drive the data bus, and can continue for a long time (hundreds of nanoseconds).
Speculation and Guesses
I’m pretty confident about everything up to this point. The rest is an educated guess:
When the ‘245 is enabled to drive a $FF value onto the data bus, it somehow causes signal integrity problems elsewhere. This might be due to undiagnosed bus fighting, but my hunch is that it’s actually due to the capacitance of the Apple II data bus. From what I’ve read in a few reference books, the bus has a pretty high capacitance, and this is a “feature” that holds the last value on the bus even after nothing is driving it anymore.
If the last value on the bus was 00000000, and the ‘245 tries to drive 11111111, there will be a brief rush of current while the bus capacitance is charged up. But the bypass capacitor on the ‘245 should handle this, I think. Adding more / different capacitors to the ‘245 on my board didn’t seem to help.
I suspect the real problem might be exceeding the maximum ratings of the 74LVC245. It’s a 3.3V chip with 5V tolerant inputs, but the output voltage is only safe to VCC + 0.5V. If the chip turns on its outputs to drive 3.3V onto the data bus, but sees there’s already 5V there, the absolute maximum rating will be exceeded. Maybe this results in 5V feedback back into the 3.3V power supply? Which results in horrible glitching everywhere else?
The Apple II video circuitry uses the data bus to read video memory, in between each 6502 or 65816 CPU cycle. That means the last value on the bus will be determined by whatever the video circuitry just read. This would explain why the glitching appears to follow a pattern with a frequency that matches NTSC video.
Holes in the Theory
This sounds sort of plausible. But when I measured the 3.3V supply with an oscilloscope, the few swings I saw didn’t seem large enough to cause problems like what I’m seeing. Maybe I need to look harder.
This theory also wouldn’t explain why glitches sometimes continue for hundreds of nanoseconds. Surely the parasitic bus capacitance should be fully charged or discharged in less time than that? Maybe it induces some kind of unwanted oscillation in the voltage regulator or other components?
The biggest flaw with this theory is that it doesn’t explain why the glitches only happen when reading from RAM, not ROM. The RAM and the FPGA (which contains the ROM) are connected in parallel to the ‘245. From the viewpoint of the ‘245, outputting a $FF byte from RAM is identical to an $FF byte from ROM.
RAM Supply Current Spikes
The SRAM datasheet reveals that the chip’s supply current is 85 mA when /CS is asserted, but as low as 2 mA when /CS isn’t asserted. My design is constantly switching /CS on and off for RAM accesses, so that’s going to create a very spiky current demand.
As before, I think the bypass capacitor on the SRAM should handle the spiky current, and adding more capacitance there didn’t seem to help my prototype board. So this doesn’t quite fit, but it’s the only reason I can think of why RAM access would behave differently from ROM access.
Confirming and Maybe Fixing
Putting everything together, the theory looks like this: when the ‘245 drives 11111111 onto the data bus, it creates a whole lot of power supply noise, possibly due to exceeding the output voltage maximum rating. Normally this noise isn’t enough to cause problems, but if the SRAM /CS is asserted a short time earlier, it will also create a supply current spike when that 85 mA load is switched on. The combination of these two effects is what pushes the Yellowstone board over the line into glitching territory. And the glitches continue for a couple of hundred nanoseconds, because of… reasons I don’t know.
When I write everything out, this theory actually sounds pretty shaky. But it’s the best I have right now.
What evidence would help prove or disprove this theory? Better measurements of the 3.3V supply? Bus voltages? Current into and out of the ‘245? Some of that would be difficult to measure.
Assuming this theory is right, how might I fix it? It seems like the exact type of problem that bypass capacitors should help solve, but so far I’ve seen no improvement when adding extra bypass caps. Maybe I need to try harder, with different capacitor values and in different locations on the board.
Another fix might be to replace the 74LVC245 with a 74LVT245. It’s basically the same chip, but with a higher current drive and with outputs that are safe up to 7 volts. If I’m causing problems by exceeding the max output voltage of the 74LVC245, swapping for the LVT should resolve it.
Other options might be to use a dual-supply 74LVC8T245, or to add series termination resistors, but both of those ideas would require designing and building a new board revision. Without more confidence that it would actually fix the problem, I’m reluctant to do that yet.
Read 15 comments and join the conversation15 Comments so far
Leave a reply. For customer support issues, please use the Customer Support link instead of writing comments.
Hi, Steve:
I like your projects!
I think this time you create your own bugs! Sorry to said that!
I checked your pcb picture. I would like to say that the pcb logically right, but circuit layout wise it\\\’s buggy! The pcb has poor power and ground! The bypass caps were put on wrong places! It seems the bypass caps are a little bit far away from vcc pins of ic. From experiences, poor bypass and bulk caps layout placement will create larger noises, or to say, it\\\’s useless, especially on cmos IC.
1. It is recommended, to put bypass caps near the vcc of buffers. On your pcb, it can be remedied immediately by connecting bypass caps directly on buffer ic vcc and near by ground.
2. One bulk cap is not enough. Put one 22uF on vcc near bus connector. Put one each near 3.3v regulator input and output. Put one near sram and cpld/fpga. Put one on each fdd connector vcc. If possible, also put one 0.1uf bypass each near the bulk one.
3. Make the power traces as wide as possile. I already saw your pcb ground has wide traces.
Hope these 2cs can help your debug.
Thank you for this feedback on the PCB layout! I’m sorry your comment initially went into the spam filter, so I didn’t see it until now. For the bypass caps, my understanding is that they should go between the VCC and GND pins of the chip. On most chips, these pins are on opposite corners, so I must choose whether to locate the capacitor closer to the chip’s VCC pin or GND pin. In most cases I chose the GND pin. Do you think locating next to the VCC pin would be better? For the next revision of the PCB, I have already widened the power traces, and I hope it will help. Thank you for the suggestions about adding more bulk capacitors. I will look at what more I can add.
Make a SOIC20 to dip adapter/ hack a piece of proto board with magnet wire to SOIC20 footprint and experiment away. Have you tried not asserting OE on 245 in the direction of the bus?
One thing I might check is if changing the contents of the screen affects the glitch. Does it work better when the display chips read all 0s or all 1s?
Try turning the SRAM off for testing; just have the FPGA generate a pattern. I don’t think that’s it, but it’ll be a good way to confirm.
With that much drop on the 3.3V, add more large value good capacitance. More ceramics (22uF 10V+ 0805), or low esr electrolytics might help.
Also, is the ‘245s direction line changing at the same time it’s being turned on? that can have unpleasant effects.
Yes, if the 245 turns on with 5V on the bus, it would push up the 3.3V supply and the regulator will reduce it’s output to compensate. that might just be happening at the wrong frequency for the regulator response.
Another thing to try is a slower part instead of the ‘245. Try a 74VHCT245A.
In a similar way to what rasz_pl is suggesting, just to experiment, you could make a SOIC-24 to DIP-20 adapter for the dual supply 74LVC8T245 and just provide its 5V supply with a separate wire as well as have plenty of room on the adapter to add an extra or larger bypass cap. After some poking around, some other interesting buffers appear to be TI\’s 74LVC4245A, ON\’s 74LVX4245 and 74LVX3245
Thanks for all the good suggestions, especially the alternate chip ideas. I can fit a dual supply 74LVC8T245 directly on the PCB, with a bit of minor surgery. 74LVC4245A, 74LVX4245, and 74LVX3245 all look like essentially the same thing as the 74LVC8T245 but I didn’t look at them in detail. 74VHCT245A appears to be a 5V part though – I need something that does 3.3V on the FPGA side.
I’m pretty sure I’ve tried not asserting OE on the ‘245, and no glitches occurred, though of course the board didn’t work. I’ll double check again. I haven’t looked to see exactly what values left on the data bus are better or worse.
With the SRAM turned off, the problem mostly disappears, but I can still sometimes see glitching when the ‘245 is enabled for output. That’s the biggest mystery to me: I can see how the ‘245 might cause a problem, but not sure what the SRAM has to do with it.
The ‘245 direction line doesn’t change value anytime during or near when the chip is enabled.
Capacitance might be an issue? On the output side of the 3.3V regulator, there are 0.1 uF ceramic and 33 uF tantalum capacitors. Each chip also has its own 0.1 uF capacitor, and the FPGA has about a dozen 0.1 uF and 0.01 uF capacitors as per the datasheet recommendation. I’ve tried adding additional capacitors (mostly electrolytic) in many different places on the board, without seeing any obvious improvement.
I think the biggest capacitor that I tried directly on the ‘245 pins was a 1 uF electrolytic. I tried values up to 100 uF electrolytic elsewhere on the board. My understanding is that for larger value capacitors, it doesn’t matter too much exactly where they’re connected on the PCB. I’m not sure about the ESR rating of the electrolytics I tested with, but they’re probably not low ESR.
What would happen if you were to make each RAM read cycle start by driving the bus with 00 before driving it with the value read from the RAM? I don’t know about the particular parts you’re using, but parasitic transistors can cause some parts to do weird and wacky things when trying to output high signals to outputs that are already above the rails; if on each read cycle you were to output a low signal for 14ns before attempting to output correct data on the bus, that should ensure that the bus voltage was below 3.3 volts before you switched on the high-side drivers.
John Payson! Great idea, and I think you’ve almost nailed it. I tried changing the logic to: 1) output 00000000 from the FPGA 2) enable the ‘245 output to the data bus 3) after about 70 ns disable the FPGA and enable the RAM. When reading from a RAM location where $FF is stored, this made the glitching problem much worse. That’s the opposite of what I expected, but it was obvious the change was somehow closely connected to the underlying glitch problem.
I then changed the logic to make the FPGA initially output 10101010 instead of 00000000, and the glitching disappeared. So I tried booting a 3.5 inch disk, and it works! Floppy Emu 3.5 emulation and also a real Apple 3.5 Drive, tested with a few different disks, ProDOS, GS/OS, and it all works! Yahoooo! Yes! I did a little dance.
Now the question is: Why does this work? What’s going on here? And is there a better way to fix it than driving magic values onto the data bus?
Driving the bus to 0xAA between cycles may seem a little hokey, but it\’s a nice simple way to guarantee that the data bus will have most four rising edges and at most four falling edges as a consequence of enabling the \’245. It would be nice to know in more detail what the actual signal margins are, to know whether signals have ample margins except when trying to switch all eight data bus signals at once in the same direction, or whether the margins are still very tight.
If you were to ensure that the \’245 always drives the bus low when it is first enabling it, and never tries to drive the bus high when anything else is actively driving it, it might be interesting to isolate the VDD rail for the \’245 from the main VDD via 100 ohm or so resistor, and let its VDD sag a bit when trying to drive the Apple\’s bus capacitance. I just checked the Apple\’s expansion card pinout and noticed it only has one ground pin. If the \’245 tries to instantaneously switch eight data-bus wires from being at ground potential to being at 3.3 volts, that will cause eight times as much current to flow in the ground wire as is flowing through each data wire. More modern expansion bus connectors would have multiple ground wires in parallel, but the Apple\’s does not. If capacitance on the Apple\’s data bus is causing trouble, adding more bypass caps to your board won\’t fix the problem but could actually make it worse since it would increase the peak current the board could drive out the data wires (and thus draw in through the ground wire).
Here’s my best guess: this problem isn’t caused by exceeding the 74LVC245 max output voltage, or else the first test suggested by John Payson would have fixed it. It’s just a case of the ‘245 pulling a lot of supply current when driving 11111111 onto the data bus. The problem is worst when the last value on the bus was 00000000. The RAM also draws a lot of current when it’s enabled. When all of these things happen at the same time, it’s too much current. The local 3V3 voltage sinks down and/or local GND voltage gets pulled up, and chips glitch. Maybe there’s some oscillation too. The ‘245 and the RAM are the furthest chips away from the voltage regulator on the PCB, so this makes sense.
Pre-driving 00000000 onto the bus is bad, because it guarantees the worst possible “old value” on the data bus. Pre-driving 11111111 onto the bus is also bad, because it guarantees the ‘245 has to drive 11111111 every time. Pre-driving 10101010 works, because the worst case can never occur. At most four data bus lines will be driven low-to-high, then at most four more a short while later when the actual data byte is driven.
So what’s the real fix here? Make sure the ‘245 and the RAM both have really good 3V3 and GND connections back to the voltage regulator and the board’s bulk capacitor. I can’t test that now, but I’ll improve it on the next board revision. Adding some larger value ceramic capacitors next to the ‘245 and RAM should also help. I tried that previously with some electrolytic capacitors and it didn’t help, but hopefully ceramics will be different. I can bodge some 10 uF ceramics next to both chips, and see if the board works without pre-driving 10101010.
I have a sneaking suspicion the problem may not be supply current, but rather ground current through the Apple\’s expansion connector. Modern machines have many ground pins on their bus connectors, but the Apple just has one. If your \’245 had infinitely stiff output transistors and infinitely stiff bypassing, and if the bus capacitance had zero effective series resistance, then when switching all eight pins on the bus, the resistive voltage drop on the ground pin would be eight times as big as the voltage drop on the individual data pins (since there are eight data pins but only one ground pin). In practice, things aren\’t quite that bad, but the Apple really wasn\’t designed to have devices produce the kinds of current transients that modern logic families can produce when driving capacitive loads.
As an experiment, I\’d suggest putting a small resistor in series with the supply rail of the \’245 and a moderately small (0.1uF or so) bypass cap across it, or else using a part with slew-rate-limited outputs. Otherwise, I suspect that ensuring that no more than four data-bus pins can switch from high to low simultaneously, and no more than four can switch from low to high simultaneously, would cut the worst-case the peak ground currents almost in half. If the RAM data were going through the CPLD, I\’d suggest having the CPLD switch the pins at different times, but I think driving the bus to a state with four high bits and four low bits should be pretty good as well.
>‘245 and the RAM are the furthest chips away from the voltage regulator on the PCB
lift Vreg leg and wire it directly to 245 with a thick wire, or as suggested earlier power whole thing from lab supply, but try soldering it in couple different places on the PCB and see if it makes a difference. How are your power and ground planes on the pcb? any loops? thin traces?
From the 3V3 regulator to the ‘245 power pin is about 80 mm of 24 mil trace with 1 ounce copper. It passes through two vias, and over but not through two more vias. It doesn’t loop. I think that should be OK? Most of the empty spaces on the board have ground fills, so the ground pin connection back to the voltage regulator is very wide.
One of my previous tests added 30 AWG wire from the ‘245 power and ground pins back to the regulator, but it didn’t solve the problem. Maybe the wire was too thin. Using a bench supply didn’t solve the problem either, but now that I look more carefully, the spot where I connected the bench supply was topologically further away from the ‘245 than the voltage regulator is.
This is surprising: I reverted the pre-driving of 10101010, and soldered thicker wire from the ‘245 power and ground pins to the voltage regulator. I also soldered a 10 uF ceramic SMD capacitor right next to the ‘245. But even with these improvements, it still glitched like before. If anything, it was even worse than before. If the problem were the ‘245 drawing too much current, then I really think this should have fixed it. So now I’m struggling to explain what’s going on here.