Broadcom presented at Networking Field Day 32 on July 26, 2023 and they presented on their AI Networking solutions. These are products and architectures that address the needs of those building out AI data center focused networks. Obviously the design will work for regular data center workloads too. albeit, suboptimal because the design is focused on addressing AI workloads and not a more general workload. The attributes that Broadcom define for what makes an AI Network unique are:
- Fewer flows (low entropy)
- High bandwidth flows (elephant flows due to the large amount of data sets being moved around)
- Synchronized and bursty traffic
- Links are saturated in micro-seconds (<<RTT)
- Training jobs run for long periods of time (hours/days)
- Tail latency impacts job completion time significantly
And they shared some interesting info about "time spent in network" is impacted by:
- Transient oversubscription
- Flow collisions and link failures
- Incast - many GPUs sends into one or a few GPU(s)
Broadcom says the solution is to build a Clos fabric that makes use of a receiver-based credit control process that can pace the senders accurately. This means it is impossible to oversubscribe the Clos fabric and therefore you can leverage techniques like packet spraying with receiver ordering. It is worth watching the presentation on YouTube to understand what they are doing and why. You can check that out here:
There are specific videos on the Tomahawk AI Interconnect here:
And also on Jericho3 AI here:
And their wrap up on AI/ML Data Center Fabric solutions can be found here:
My quick thoughts on what Broadcom presented:
I wasn't aware (more likely I haven't been paying attention to what is happening in AI/ML like I should be) that there was this much specific network design work going into addressing AI workloads. While I understand there are a lot of AI/ML projects, I wasn't aware that so many private firms might want this solution architecture for their own needs versus running on leased cloud models.
Clearly there is a pricing advantage to running stuff at scale on your own hardware (in terms of reduced network data ingress/egress costs, compute cycles, and having dedicated GPU access) otherwise Broadcom wouldn't be building these sorts of solutions. It seems most of the large scale cloud providers have built something similar on their own or have requested that Broadcom address a gap in what traditional Ethernet fabrics can provide.
What will be interesting to me is if this is a short term industry change to address a narrow vertical or if this will become the new default Ethernet fabric architecture because AI/ML workloads will become common place DC workloads. I'm not convinced it will go that way, perhaps a hybrid of specific AI/ML Ethernet fabrics that are L3 connected to traditional DC focused Ethernet fabrics to attempt to give an organization the best of both worlds.
You can also get Drew Conry-Murray's thoughts on Broadcom's presentation over at his Packet Pushers blog post.
- Ed
In a spirit of fairness (and also because it is legally required by the FTC), I am posting this Disclosure Statement. It is intended to alert readers to funding or gifts that might influence my writing. My participation in Tech Field Day events was voluntary and I was invited to participate in NFD32. Tech Field Day is hosted by Gestalt IT and my hotel, transportation, food and beverage was/is paid for by Gestalt IT for the duration of the event, if travel was involved. In addition, small swag gifts or donations were/are provided by some of the sponsors of the event to delegates (I didn't accept the swag gifts offered). It should be noted that there was/is no requirement to produce content about the sponsors and any content produced does not require review or editing by Gestalt IT or the sponsors of the event. So all the spelling mistakes and grammar errors are my own along with the ideas and thoughts.
No comments:
Post a Comment