Linux on the Sega 32X. Who needs hardware synchronization primitives anyway?

Preface

I couldn't start this entry in any other way possible but by thanking everyone who enjoyed the Linux on Jaguar post. Also special props to the linuxmd project again. It was a very fun experience and we've even made it into a few news outlets. I also posted it on Hacker News and it received some good interest (and nice questions!). So, yes; thanks everyone again!

Why is this (Linux) trend continuing?

I have an easy answer to this question: it's mainly to improve my board bringup skills. I've been working professionally on the Firmware/Operating Systems/Embedded area for well over 2 years now (professionally-speaking) and one of the usual suspects you see when trying to land a new job is "the ability to handle board bringups from the ground up". I know it doesn't sound too fascinating but for me, other than job hunting (which is not my bread and butter anyway), bringing up a piece of silicon and having it run something you made/helped making is super fun.

You get to experiment with many of the different layers that make the result up: emulators, technical documentation, experimentation, investigation... you know; the usual engineering stuff. Not that reading early-90s scanned datasheets is my passion (there's some documents that are plain images embedded onto a PDF) but there's something to it that makes me come back every time.

I actually started doing this in '13/'14. I was 12/13 at that time and I vividly recall my parents buying me my first phone. It was a MediaTek-based 6589 quad-core processor with a gig of RAM. Powerful stuff, eh! At that time (not much soon after), Google released Android 5.0 Lollipop; but the vendor of the phone decided that it would not update it to that version. And I was really sad... so I had to get my hands dirty.

I'll sum up the story a bit because it's a bit large, but the gist of it is that I asked the company to comply with GPL and release the Linux kernel source code for the phone; for which they opposed, but as a community we got together and they released it (along some other phone's trees). Picture me surprised when the source code was half-assed and the board configuration didn't boot on the phone. Fortunately at that time, I found a leak of the full-source of MediaTek ALPS targeting a variation of the 6589 that seemed promising enough (had the full Little-Kernel bootloader, Baseband, Preloader, Linux kernel and AOSP) for me to check.

I spent a few months trying to reverse-engineer parts of the zImage that was bundled on the stock boot image provided by the phone vendor to fill parts of the MediaTek code (mostly related to lcm, battery, chipset quirks and whatnot) and after some good amount of digging and experimenting (and good chats with the cool people over at xda-developers) I had Android 5.0 running on my phone; mind you, it wasn't perfect. No Bluetooth, no HWComposer (hardware acceleration), no Media engines... but seeing the pure, unadulterated new Material Design on my phone blew my mind. Android 5.0 Lollipop. I should probably try to make a separate post of this story... someday.

The mighty Sega 32X

Released in November 1994 in America/Europe (December 1994 in Japan); the Sega 32X was an add-on that promised to be an intermediate step between the 16-bit era Genesis and the soon-to-be 32-bit Sega Saturn. I'm not a good writer so the story that led up to that particular moment (for SEGA, that is) is a bit tumultuous (SEGA Japan vs SEGA of America) so I'll leave it as a task for the reader.

Anyhow, prior to the 32X, SEGA launched a CD add-on; the Sega CD (or Mega CD, if you are European/Japanese). It plugged into the expansion slot of the Genesis and it enabled CD-quality audio streaming (and well, game/game asset streaming). It was a machine on its own (it had a higher clocked 68000 in comparison with the Genesis). There were 2 separate models, the first "generation", which was front-loading: Sega [Mega]CD Model 1 with Mega Drive/Genesis Model 1, and the second, top-loading one: Sega [Mega]CD Model 2 with Mega Drive/Genesis Model 2.

But this post is about the 32X... so here it is: Sega 32X. The actual add-on was plugged into the cartridge slot of the Genesis (it had a separate power brick and video output, later of which was "combined" on top of the Genesis video out timing/video/color channel signals). If you've ever had the pleasure of building the Tower of Power, you'll know that having 3 separate power bricks, plus the cables to get stereo output out of the 32X, the "patch" cables and the many missing tidbits I'm probably forgetting now, wasn't anything short of fancy.

32X Hardware specifications

Admittedly, the add-on is far more powerful than the base console:

As you can see, quite the difference. I've not even listed all the differences, but the main ones I can think are those. Unfortunately for SEGA, the launch happened almost at the same time as Sony's PlayStation 1; which was primed to overcome the gaming industry with a much simpler architecture. It also didn't help in any way that the SEGA Saturn was close to launch date too (so consumers were puzzled as to which one to get). All in true SEGA fashion!

Anyway, if we got Linux running on the Jaguar, we should be able to do similarly for the 32X; right?

jmp _linux, the rendezvous

Getting Linux to run on the SH2s (running as in, having it print anything meaningful over UART) required some brainstorm. We'd need to set up 3 different CPUs: the 2 x SH2s and the 68000. Fortunately there's readily-available examples for that on Github so I picked what seemed the most popular 32X Hello World example: Chilly Willy's Sega MD/CD/32X devkit.

The basic example sets up a communication channel between the 32X(s) and the 68000 (so you can ask the 68000 to do some stuff the 32X can't) through specific memory-mapped IO addresses; prepares the SH2s (sets up the caches, uploads the SH2 code from the cart directly off of 68000 machine code...) and makes the primary SH2 jump to it; whilst the secondary gets parked.

What I did was expand this so that I could have a simple state machine that pushed characters to the UART on the Genesis (since the 32X doesn't have direct access to it); but there was a problem...

256K + 64K

The biggest hurdle when porting Linux to a new (and newer-ish, in terms of when it launched) platform is mostly about doing the wiring to get it up and running; that's usually the case if you have the memory to even load it. This is not the case for us...

Fortunately, there's FPGA-based flash carts for the console (enables you to run games for both the base model and the 32X; and there's even specialized "cores" that provide Sega CD support if you don't have the add-on!). One of those (series of) carts comes from Krikkz. He basically did what flashcarts do but went the extra mile for pretty much everything; it's not just about running games!

There's a special bit of those carts that we're interested in: older cartridge-based systems often had this notion about "mappers". Mappers are basically ICs that sat on the cartridge which enhanced the capabilities of what the console can handle in terms of memory. They could also offer other features not related to memory. An example is Sega's Virtua Processor (SVP), which had a Samsung-engineered DSP to do heavy math computation (to enable Virtua Racer to display all its 3D polygon glory on a base Genesis); or Nintendo's SuperFX chip; which provided similar capabilities to the SVP and was made by the creators of Days of Thunder or Birds of Prey: Argonaut Games.

Usually mappers were used to enable more ROM space for larger/more detailed games in a bank-switch fashion. You'd write to a certain memory address/do a specific set of accesses and the IC would internally remap parts of the "extended" ROM to the memory window the console can directly address. Kinda like MS-DOS's HIMEM.SYS in spirit if you think about it.

So Krikkz basically did us all the work here; turns out we can have ROM as RAM (ROM which can be made "writeable" by arranging certain configuration bits). This gets us 4 MBs of RAM. I had to basically extend the SDK example to do the ROM->RAM mappings but had to leave a bit of space for the Linux + initramfs binary to live on (the actual ROM within the cart memory window, basically). I ended up finding a sweet spot of ~540 KBs of ROM and rest of RAM. The Linux loader basically copies the binary on the memory location it was linked at and jumps to it.

Compiling and running Linux

So, I made a sh2eb-elf cross-compiler and got to compile Linux... but the compilation failed rather soon. Out of all the issues I've had with toolchains, this one was probably the funniest. For some reason GCC was having an ICE (Internal compiler error) when compiling kernel/nstree.c. I was able to find a workaround (didn't have the time at that moment to file a bug report) fortunately.

After all this work (and writing a small earlycon driver to get some UART output carried from the 32X to the Genesis) I had the very first Linux messages. There was a separate issue: at some point Linux seemed to stop the boot process and was hanging indefinitely. I then discovered another set of issues, this time regarding: ashlsi3, ashrsi3 & lshrsi3. I'm not really sure how those are written (do they get copied/inspired from their libgcc counterpart or something similar?) but comparing what the cross-compiler did for those and what Linux did, it was obvious that we were clobbering a register we should not be clobbering.

I found this because I started noticing that some pr_info()s and pr_notice()s were getting their stack corrupted (and never returned to the caller) and was able to trace it here. I then made an additional (similar) fix for arch/sh/lib/checksum.S just to be on the safe side of things; where I basically gated shrd because it doesn't exist on the SH2s contained within the 32X (and if used... Illegal Opcode).

This basically got us through the entire boot process right before the clock calibration; for which I decided to make a simple driver for the SH2(s) FRT (Free running timer); this also got us further and we were at the verge of launching the init process.

Fortunately, this time around I was able to leverage ELF FDPIC support (which made my life easier in comparison with Jaguar's FLAT binary support) to build my rootfs. I was also blessed by musl-cross-make; which landed me a sh2eb-linux-muslfdpic- toolchain I could use (I must say I battled hard with crosstool-ng, buildroot and a few others but didn't succeed).

So with this, everything was booting. Case... closed...?

SMP

Yeah, I couldn't help myself. It turns out that you need some hardware primitives to enable inter-CPU synchronization... which the SH2s on the 32X don't have¹.

¹: TAS.B opcode seems to be used on projects like D32XR even after the developer manuals asked not to do so due to the adapter bus not having the locked read-modify-write cycle mechanism.

We also don't have cache coherency so another problem to add to our bag. But I thought, hey, both SH2s are equal features-wise; we sure can come up with something... right?

It turns out that sparc32 has precisely some code to aid in that. We can basically leverage Peterson's algorithm to implement the required primitives entirely in software. Mind you that this isn't performant but it does work. I basically had to do the bringup for all the primitives that needed care (cmpxchg, for instance) with the ideas that Peterson described.

I also needed to wire the 68000's code (which for now was only handling UART stuff) to aid on delivering IPIs to both SH2s (I extended the commands the 68000 can receive so that it could correctly do its thing) plus setting up the shared memory region to aid on the synchronization tasks. Think of the 68000 as being a bus arbiter/extra interrupt router at this point. From the SH2s I just send an "IPI" command which the 68000 parses and decides which interrupt line to assert based on a bit of the data word (0 primary, 1 secondary).

Then there's the cache-coherency issue, which I solved by not using cached mappings (and well, cache is disabled, regardless) to place Peterson's flag and turn variables (you know, to avoid SH2s spinning on stale cache values for flag; for instance).

There was another issue I faced: how do I identify which CPU is which (so Linux can properly manage and issue hard_smp_processor_id())? Well, considering I was doing softfloat regardless; I just made the CPUID_REG point to the SH2-private hardware divider register (DVSR) which had no other use so it effectively became my "what cpu am i" register. That register is written by the Linux loader at boot for both the primary and secondary SH2s.

Performance is abysmal, bus contention is bonkers; but it does work, and that's fine by me.

smp: Bringing up secondary CPUs ...
MARS SMP: released secondary SH2 to 22082000
smp: Brought up 1 node, 2 CPUs

I must admit that it wasn't as easy as that; it really involved lots of tinkering and (mostly) trial-and-error (I also spent 2 nights chasing why both CPUs would get woken but soon after they'd idle; turns out I needed a fix on default_idle() that was making the scheduler bug out and not correctly continue execution). But it was well worth it.

I just then basically made a simple console display driver (so you know, you could see it in action on your actual screen)... Voilà!