5. Who Am I?
• Life in tech business:
– 10 yrs large-scale mission-criKcal IT
– 10+ yrs consulKng & training
– Some startups on the way
• Avid (if not very good) ice hockey player
• Long-Kme lover of great engineering…. when
used to make a real difference
• Atomic Inc:
– Generalist pracKKoner
• Network just one element
– Product : engineering : operaKons
Avi Deitcher avi@atomicinc.com
(not 24601)
19. Our Tests
What We Tested
• netperf ⇒ netserver
• UDP & TCP round-robin
• Sizes: 300, 500, 1024, 2048
• No orchestraKon = complete
control
• 50000 iteraKons
– Law of large numbers
• Latency (Avg, %iles), CPU
• DifferenQals, not absolutes
How We Tested
• .net
– Because it had to be metal
– Wicked smart team
• Complete test run
– Network changes
– Hardware variaKons, errors
hVps://github.com/deitch/network-tests
Avi Deitcher avi@atomicinc.com
24. Local Networking Summary
• SR-IOV horrible latency but great CPU
– Hold that thought…
• net=host on par with metal
• macvlan closest virtualized to metal
• Rest in same range:
– Latency: 5-10 𝓊-sec overhead
– CPU: negligible difference
• Calico (IPIP & naKve) & Docker overlay slightly
more performant (margin of error?)
• Watch out for very large TCP packets
Avi Deitcher avi@atomicinc.com
31. What else could we do?
Ø Other hardware types
Ø Other network fabrics
Ø Other network overlay versions (we have the data…)
Ø Docker macvlan network driver
Ø ipvlan
Ø IPv6
Ø Kernel and network stack tuning
Ø Distant networks
Ø Other traffic paVerns (mulKcast vs unicast)
Ø Other host-to-host encrypKon
Ø Other kernel versions
Ø Other OSes (Illumos-based?)
Ø A whole lot more…
Avi Deitcher avi@atomicinc.com
32. Headaches (and Thanks)
• Headaches
– Weave SYN-(nothing)
– etcd is “touchy”
– Packet L3 network is powerful but… unique
• Macvlan, weave, flannel: all required pings for mac
• Sexng up bridge w/o NAT, Calico, macvlan was “different”
– SR-IOV is complicated and flaky, especially Mellanox
– netperf with UDP packets can get stuck (Calico-ipip)
– And a whole lot more (ask me offline)
• And thanks:
– Bryan Boreham, Adam Harrison at weave.works
– Zac Smith, Adam, Aaron, Andy, Lucas, everyone at Packet
Avi Deitcher avi@atomicinc.com
33. Conclusions
• SR-IOV: most of the Kme, just not worth it
• Performance:
– Metal (+ net=host): always performs best
– Direct network++: macvlan is your friend
– Others: Roughly similar, careful of Weave (sleeve)
• What’s your use case?
– ULL: Metal/net=host > macvlan > calico > overlay
– Everything else: Focus on your architecture and skills
Pick intelligently: easier, not simple
Avi Deitcher avi@atomicinc.com
34. Conclusions
• SR-IOV: most of the Kme, just not worth it
• Performance:
– Metal (+ net=host): always performs best
– Direct network++: macvlan is your friend
– Others: Roughly similar, careful of Weave (sleeve)
• What’s your use case?
– ULL: Metal/net=host > macvlan > calico > overlay
– Everything else: Focus on your architecture and skills
Pick intelligently: easier, not simple
Avi Deitcher avi@atomicinc.com