Designing an Ethernet Switch ASIC

Comments

It seems I have just built the world’s first open source switch ASIC and I am getting it back from the fab mid november. Here is its repository :

Buckle up and welcome to the tale of more madness.

Where are the open source networking ASICs?

Recent FCC decisions have put the central importance of networking equipment top of mind, and in doing so, have also put the total absence of any entirely open source networking equipment hardware at the top of my mind. Although Open Source Silicon is in its infancy we are currently seeing a number of projects being designed, tested, and for the most ambitious ones, even taped-out with some proven silicon already in the wild. That said, the vast majority of the most ambitious projects are predominantly RISC-V SoCs. Surprisingly, there has been much less interest in building open hardware for networking equipment. Well .. unsurprisingly actually … networking equipment is far less “sexy” than CPUs and has thus received much less attention from the open source community. On the other hand, this means there’s a huge untaped boulevard of projects open to anyone with more time than common sense to build open source networking equipment chips!

Because, on the other side of the great silicon divide, the world of open source networking equipment is actually quite rich, both in terms of its flourishing software ecosystem and the open PCB/electronics ecosystem, with a flurry of fully featured open source routers available. Yet, due to the current ecosystem’s limitations, under the hood, these are all still all running closed-sourced, blackbox proprietary chips for the central compute and routing tasks.

So what would it take to build a router chip? It’s a very ambitious project, since a router involves a complex SoC needing both a powerful enough CPU to run the networking stack alongside specialized networking hardware, and analog frontends for the wired and wireless connections. Since building a full router outright is much too ambitious of a greenfield project for a single person to ever hope to pull off, let’s begin with a more approachable first step: building a switch. After all, although most people only have one router, you can have a number of smaller switches, and we also don’t have any open source chips for those. Oh and, I am not talking about building an FPGA switch, oh no, I am talking about building a switch ASIC, taping it out and then proving the silicon works.

Switch Flavors

Before we start figuring out what kind of switch we want to build, let me give you a quick overview of the different flavors of switches out there. Readers deeply familiar with networking equipment can skip this part.

Glossary

Physical Layer

Ethernet supports multiple physical layers, outlined in the 802.3 IEEE spec. Each clause in the spec defines the underlying medium characteristics for carrying the Ethernet protocol over the medium at a given bandwidth. Both a 10 Gb fiber Ethernet link and a 10 Mb coaxial cable can carry Ethernet packets and, to the layers above the physical layer, the packets will look exactly the same. Where they will differ is at the physical layer and, outlined in the spec is precisely how, with which timing and with what encoding data will be transmitted over the medium. This ensures that devices made by different manufacturers agree on what “talking Ethernet” looks like, making them interoperable.

So one of the first questions I must answer is which physical layer my switch should support. This will define how much traffic I must route, how fast I should do so and how much bandwidth I can carry.

Managed vs Unmanaged

The second question is: what type of switch do I want to build? There are two big families of switches: managed and unmanaged. As the name implies, managed switches can be configured and managed from the outside. This allows for much smarter routing, such as supporting different VLANs. While unmanaged switches are essentially unconfigurable pieces of networking equipment that you plug into your network and just work (until power surge do you part).

Cut-through vs Store-and-Forward

The last switch characteristic is cut-through versus store-and-forward. Cut-through switches start forwarding a packet while the packet is still arriving, leading to much lower networking latency, whereas store-and-forward switches wait for the entire packet to arrive before checking that no corruption has occurred, and then only forward the packet if that check passes. Where issues arise is when a packet is corrupted, a cut-through switch might still forward it, propagating corrupted packets through the network since forwarding began before the FCS (frame check sequence) could be checked.

Designing against constraints (again)

As usual with my ASIC design work, what I build is shaped as much by what I want to build as it is by the constraints of the silicon.

Pins

My first major constraint is the pins: not only in their amount but also in their maximum bandwidth. Since I will be taping this first generation chip out over the Tiny Tapeout shuttle chip, using purely digital tiles, I’m constrained by the limits of these pins. In total this will afford me 24 pins: 8 input, 8 output, and 8 bi-directional (configurable to be either inputs or output pins) GPIO pins, rated to run reliably at 50 MHz on both inbound and outbound data. The absence of any analog front end makes building anything IEEE-physical-layer-compliant directly a challenge, but there’s a way around this: using an external PHY (physical layer) chip and interfacing with it over the standardized RMII bus. Though this consumes 7 bits per Ethernet interface it makes my 50Mbps pins capable of sending and receiving over 100Mbps Ethernet (100BASE-TX).

I will be targeting the widely available Microchip LAN8720A/LAN8720AI (which will be the only mention of AI in this article) PHY chip for this interface. I will be going more in-depth as to how this ASIC will be interfacing with this PHY later.

Area

My second major constraint is my limited die area. I am paying for all this out of pocket afterall, and more area means a higher manufacturing cost. Now I am not trying to accumulate a pool of gold or anything, it’s just that, in compliance with Maslow’s hierarchy of needs, waffles rank higher than area, and higher manufacturing cost means less waffles. So just like any semi-conductor company, I am incentivised to keep my area budget under control.

Because store-and-forward requires the switch to store the entire packet before forwarding it, and because ethernet frames can reach upwards of +1.5k Bytes (and +9k for jumbo frames), they require massive amounts of storage. A workaround for that would be to store the packet to some off-chip memory but I don’t have the pin budget to afford that right now, so on-chip memory is my only option. The problem is, on-chip memory consumes a lot of area given the large area footprint of SRAM and the even larger area per stored bit footprint of flip-flops. Now if we were talking of a few bytes this could be negotiable, but for multiple 1.5k Bytes packets this is a dealbreaker, thus ruling store-and-forward out.

Assuming I was using Tim Edward’s excellent OCD 256x8 SRAM IP, a single 256 Byte SRAM occupies 301.3 x 224.93 um (using W orientation in this implementation) and consumes just by itself 1/3 of the floorplan. If we want to store even a single full packet we would need 6 of such instances, and since we have 3 ports, we would then need to replicate that 3 times, so 18 instances in total. Now, we have a larger 1028 Byte version of this SRAM that is 301.3 x 515.81 um which nicely increases our storage area density. But again, if we were to instantiate six such macros this would occupy x3 the area than we currently have to spare. Now, the Tiny Tapeout shuttle chip does support me scaling up this design up at most another two factors of two, or 4x. But the cost would also scale by a factor of 4, at which point we are getting into full chip orders of magnitude of cost. And if I were to go down the full chip route, since I then have the possibility of having a lot more pins to play with, this re-unlocks the possibility of interfacing with much larger external memories, thus changing the landscape of what the correct technical tradeoff would be again.

Easy to use

Lastly, I want something that doesn’t require external software or configuration. Firstly because I would like external third party users in the community to be able to easily pick this ASIC up and start using it. And that becomes increasingly difficult as soon as I start involving custom software. I am aiming for something that is as close to plug and play as possible with ease of use being a measure of success. Not to say that the community won’t be able to compile and flash my custom embedded abominations, just that their willingness to do so decreases exponentially with each extra step. Secondly because I am once again working on a very tight schedule and that custom software support adds significant amounts of time to both the pre-tapeout and bringup workloads. (Let’s not forget we are still talking about a single person’s project here.)

This would be especially true for this ethernet switch project since I would aim to be compatible with a widely adopted open source switch management software and protocol like SNMP for monitoring or SSH for monitoring and configuration. Support for these protocols is not at the level of something I can trivially implement in hardware, especially for SSH, and the best design decision would be to offload to a CPU. So either integrate an on chip CPU on the ASIC or, build a custom interface to an external MCU and offload all these requests to it. Both options involve a significant software effort, with the on chip CPU also promising a huge design effort and even more area usage. At which point, I am approaching the planned architecture of the router rather than the first generation of the switch. So, for both of these reasons, an unmanaged switch is the path I am taking.

What are we building?

So to recap, I am building a:

• 3-port
• Full-duplex
• 100 Mbps bandwidth
• cut-through
• unmanaged ethernet switch.

Now that we have figured out what we’re building, it’s time to focus on the fun part: how to build it! 🥳

System Overview

The real Ethernet frame that is hidden from view

Most people have actually never seen a full ethernet packet. When you inspect Ethernet traffic on your computer via tcpdump or wireshark, you actually only see part of the Ethernet frame, with some parts missing. These parts are generally stripped out by your computer’s network interface before the packet is forwarded to the software realm, thus hiding them from user’s view.

Random IPv4 packet captured over tcpdump (tcpdump -xx -e -v 'ether proto 0x0800'):

16:36:51.298082 60:e9:aa:92:dc:7d (oui Unknown) > 5c:e9:31:1e:9d:00 (oui Unknown), ethertype IPv4 (0x0800), length 78: (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 64)
    superpitchu.53332 > dfw25s53-in-f9.1e100.net.https: UDP, length 36
        0x0000:  5ce9 311e 9d00 60e9 aa92 dc7d 0800 4502
        0x0010:  0040 0000 4000 4011 2b98 c0a8 0085 4a7d
        0x0020:  0369 d054 01bb 002c a93f 6fe8 5bd5 5378
        0x0030:  f4c9 ccae 477c 5083 0696 8fa4 4f62 57bf
        0x0040:  41a9 78a3 fc60 7781 2b8b 38a1 2ccd

The missing parts are:

• Preamble + SFD: Before the start of the MAC header exists a sequence of 7 bytes of an alternating bit sequence signifying that a packet is about to start, called the preamble, followed by a 1-byte marker ending in a double asserted bit sequence called the SFD (Start Frame Delimiter) signaling that the next bits are the MAC header.
• Frame Check Sequence Footer: At the end of the packet there is a 4-byte footer called the FCS (Frame Check Sequence), used to check whether the Ethernet frame’s content was corrupted during transmission. This is checked by both store-and-forward switches and by most computers’ network interface cards and these drop all frames that fail this test before the packet is forwarded. And since, once they have been evaluated by the networking interface, both the Preamble+SFD and FCS serve no further relevant purpose, they are stripped out before the remaining packet bits are forwarded up the networking stack.

Now, since we’re going through the Microchip PHY, we won’t actually be interfacing directly with the Ethernet physical layer, as it will be abstracting away the precise 100BASE-TX PHY behavior, though the full ethernet frame will remain intact (including Preamble+SFD+FCS) All our data transmission and reception will actually be done through the RMII bus interface.

The RMII interface

RX

On the RX (reception) side of RMII, we have four pins:

• rxv valid signal (crs_dv in the diagram above)
• rxer error signal
• rxd[1:0] two data signals

Interestingly, this valid signal isn’t a pure data-valid signal but an early data-valid signal, and it’s up to our ASIC to read the incoming data and correctly identify when the Ethernet frame data actually starts using the preamble and SFD. The valid signal actually asserts asynchronously a few cycles before the start of the preamble but, deasserts synchronously with the end of the Ethernet frame.

TX

On the TX (transmit) side, we only have three signals:

• txv valid signal (txen in the diagram above)
• txd[1:0] two data signals

with no error signal. This makes sense, since the error signal is there to indicate issues on the medium, and since there is no medium between the ASIC and the PHY there i