I’ve found a rare clock glitch in the Yellowstone hardware, but I’m not sure how to fix it. This seems to be the source of the “mysterious signal glitching” I mentioned in my previous post, that disappeared after some minor changes to the FPGA logic. From what I can tell it no longer affects the Apple IIe, but pops up rarely on the Apple IIgs, wreaking havoc. I’d suspected a glitch for a while, but I never directly observed it happening until increasing the logic analyzer’s sampling rate to 125 MHz and lowering the logic threshold from 3.3V levels to 1.8V levels. The evidence is shown above.

MEMEXP is an internal FPGA signal that should remain high. Q3 is a 2 MHz clock provided by the Apple IIgs. There’s an 8 ns wide glitch pulse shortly after the falling edge of Q3, and this causes MEMEXP to go low when it shouldn’t. In all my tests, the glitch pulse always occurs during the low period of Q3, between 50 to 150 ns after the falling edge.

An important detail: The Q3 shown in this logic analyzer trace isn’t actually the FPGA’s Q3 input, because that’s on a PCB trace without any good place to attach a probe. The trace shows an FPGA output that’s been configured to simply propagate the Q3 input. That should be nearly the same thing, but not exactly identical. To complicate matters further, there’s also some level conversion happening. The Q3 supplied from the Apple IIgs is a 5V signal, and it passes through a 74LVC245 to create a 3.3V signal that’s connected to the FPGA. Behold my awesome diagram:

The previous Yellowstone prototype had a 390 ohm inline resistor between the Apple II’s Q3 signal and the 74LVC245. This is something I copied from the design of Apple’s Liron disk controller, although I don’t understand why it was there. None of the other disk controllers that I examined have a resistor on Q3, and I couldn’t find any documentation about a need for one, so I eliminated the resistor in this second generation prototype. Maybe that was a mistake, and is the reason why I’m seeing clock glitching now.

My hunch is there’s a different explanation. I suspect the glitching is caused by other problems on the board, rather than Q3 itself. Maybe there’s capacitive coupling between Q3 and another nearby signal trace. Or maybe my power and ground supplies are bouncing around whenever there’s a big spike in current, like when a bunch of chip outputs switch simultaneously from low to high, and that creates EM noise everywhere. Or maybe there’s some brief contention on the data bus that creates spikes and noise.

I tried playing around with the the FPGA input buffer settings for Q3, in an attempt to make the problem go away. I tried pull modes of KEEPER, DOWN, and NONE, and hysteresis settings of SMALL and LARGE. Nothing helped.

I’m not sure where to go next. I could try assembling a second board, to see if it’s just an assembly problem with this particular board rather than a general design issue. I could look for a way to get a logic analyzer probe directly on the actual Q3 input, which might require fancy soldering, and might also change the behavior. Or a probe on the 5V version of Q3, directly from the Apple II. I could try to patch a 390 ohm resistor onto the board, but it wouldn’t be clean and might create more clock problems than it solves. Or sprinkle some more capacitors around the board, or other analog magic.

I’m still testing the latest prototype of Yellowstone, my FPGA-based universal disk controller for Apple II computers. After a great deal of poking and prodding, the disk controller is now mostly working with all of the Floppy Emu’s disk emulation modes and with a variety of real Apple II drives. Hooray! But there’s still so much more work to do.

This is version 2.0 of Yellowstone, and it hasn’t exactly been a smooth experience getting it working. First I discovered a layout blunder, where I ran a signal trace right through a power via. After patching that with a razor and some wire, I struggled with mysterious signal glitching in a different section of the board. Some minor changes to the FPGA design made the glitching disappear, but I’m not convinced it won’t return later. Then I discovered a design flaw which means certain types of drives will create a short circuit when plugged into the card’s second drive connector. I’ll fix that in the next revision. Eventually I got the Yellowstone card working with a Floppy Emu, but real drives wouldn’t cooperate. That ultimately proved to be an issue with the width of the read pulses from the drives – they were too narrow to detect reliably. I adjusted the FPGA design to compensate, and now I’m finally in business.

Current status: I’ve had success with an Apple IIe plus Yellowstone booting from a Unidisk 3.5, an Apple 3.5 drive, an Apple Disk 5.25, and a Floppy Emu in its 3.5, 5.25, Unidisk 3.5, and Smartport hard disk emulation modes. The Disk II and Duo Disk should also work, though I haven’t yet tried them. But this is just the very beginning of testing, with a single Yellowstone card in slot 6 and a single disk drive. I haven’t tried writing to the disk, or formatting a disk, or daisy-chaining drives, or using the second drive connector, or anything in the Apple IIGS.

I tried connecting a Macintosh floppy drive pulled from a Quadra 605, and the Apple IIe wouldn’t even turn on. I think I shorted the power supply somehow. This was supposed to work. I’ll look at it more closely later.

Something is still funky with the Apple 3.5 Drive support. Two of the 3.5 inch floppies I tried will boot reliably, but one won’t. But the same three floppies work fine in the Unidisk 3.5. I haven’t yet tried to troubleshoot this one.

It also appears that the behavior changes depending on whether my JTAG programmer is connected to the card, and sometimes it doesn’t work if the programmer is connected. That’s concerning. In the past, I saw something similar with earlier prototypes. I’m not sure if the programmer is holding the FPGA in a reset state, or if it’s an analog problem with the card that’s affected by the presence of the programmer. Another mystery to investigate.

The Apple 3.5 drive and Apple 5.25 drive both sound noticeably different when used with Yellowstone as compared to an Apple standard disk controller. While I haven’t yet determined why, I think this is benign and is due to differences in exactly when and how the disk controller moves the drive head. But when you’re accustomed to disk drives having a certain characteristic sound during I/O, it’s a little disconcerting. The Apple 3.5 drive sounds frenetic, and the 5.25 drive’s familiar buzz-buzz-buzz startup noise is completely different. This latter one bothers me enough that I may try to change it, even if it’s not causing any problems.

 
Ruminations

It’s hard to believe, but the Yellowstone project will celebrate its fourth birthday this summer. I have to admit I’ve taken on a bigger challenge than was probably wise, and I’m struggling. For me, this stuff is hard – much harder than the Floppy Emu’s development. It’s difficult to maintain motivation when I keep running into one major problem after another, with the goal of “finished” always far away. And the domain of possible problems is huge. One day I’m troubleshooting some software incompatibility with GS/OS, the next day I need to become an expert in FPGA in-system programming, and the day after I’m chasing signal integrity problems in the analog domain. It’s tiring.

There’s still lots to do. All the problems I mentioned above must be examined and fixed. Then there’s a whopping huge amount of testing needed, with a dizzying number of permutations of computers, cards, slots, drives, daisy chain configurations, boot priorities, and software disks. That’s practically guaranteed to turn up more thorny problems that will be difficult to solve. And then I still need to build solutions for in-system reprogramming of the FPGA by end users, and for fast and thorough automated testing during manufacturing. I have some plans for how to handle those, but haven’t yet started any work.

I’m hopeful that while the Yellowstone project isn’t yet “almost done”, it’s at least close to the point where the finish line is visible. I really want to wrap this up and get it out the door.

I just spent more than two hours troubleshooting a seemingly simple HTML problem. When I copied and pasted a small section of HTML, the web browser displayed the newly-pasted section differently from the original. The horizontal spacing between some of the elements was slightly different, causing the whole page to look wrong. But how could this be? The two HTML sections were identical – the new one was literally a copy of the old one.

This simple-sounding problem defied all my attempts to explain it. I came up with lots of great theories: problems with my CSS classes, or with margins and padding. Mismatched HTML tags. Browser bugs. I tried three different browsers and got the same results in all of them.

Feeling very confused, I looked again at the two sections of HTML in the WordPress editor (text view), and confirmed they were exactly identical. Then I tried Firefox’s built-in web developer tools to view the page’s rendered elements, and compared all their CSS properties. Identical – yet somehow they rendered differently. I used the developer tools to examine the exact HTML received from my web server, checked the two sections again, and verified they were character-for-character identical. Firefox’s “page source” tool also confirmed the two sections were exactly identical.

By this point I was getting ready to blame cosmic rays or voodoo magic. I discovered that any time I copy-pasted any similar HTML section, the newly-pasted section would appear in the browser with the wrong element spacing. How could this possibly be? I then tried the W3C Validator, which found some other problems with my page, but nothing that could explain this behavior. And once again, it confirmed that despite rendering differently in the browser, the two sections of HTML were identical.

Clearly something wasn’t adding up. I used curl to download the web page from my web server, viewed the local copy, and saw the same behavior as before. But when I opened the stored .html document with a hex editor, I finally had my answer. The two sections of HTML were not identical: one section used a different type of space character from the other.

What the hell.

I discovered that the original HTML section contained non-breaking spaces. But instead of encoding them with the   entity, they were encoded directly as Unicode character C2A0. I’m not sure when or how this happened, but I blame WordPress. When viewing this section in the WordPress HTML editor, the C2A0 spaces appeared like normal spaces, and copy-pasting the section inside the editor silently converted non-breaking spaces to normal spaces with hex value 20. So the copied version rendered differently, even though the source HTML appeared to be the same.

This is like the 21st century version of confusing a zero with a capital letter O, yet worse. I wasn’t even aware that non-breaking spaces have a Unicode character value – I thought   was the only way to encode them. I changed the HTML back to use   and now it all works fine.

I’m surprised at how many different tools failed to reveal this subtle but important difference between types of spaces in the HTML source. The WordPress HTML editor failed to show or correctly handle the difference. The Firefox web developer tools and page source tools failed. The W3C Validator’s source view failed. Curl plus a hex editor was the only way to finally establish the ground truth about the precise contents of the HTML source.