Integrate Kubernetes into CORD(Central Office Re-architected as a Datacenter)

KUBERNETES INTEGRATE
WITH CORD
HUNG-WEI CHIU
LINKER NETWORKS.INC

WHO AM I
• Hung-Wei Chiu (邱宏瑋)
• hwchiu@linkernetworks.com
• hwchiu.com
• Experience
• Software Engineer at Linker Netowrks
• Software Engineer at Synology (2014~2017)
• Co-Found of SDNDS-TW
• Open Source experience
• SDN related projects (mininet, ONOS, Floodlight, awesome-sdn)

OUTLINE
• What is CORD
• Challenge of kubernetes with CORD.
• What have we done now
• Next steps

CENTRAL OFFICE RE-ARCHITECTED AS A DATACENTER
5
SDN + NFV + Cloud
Open Source Software
Commodity Hardware
(Servers, White-Box Switches, I/O Blades)
Large
number of
COs
Evolved over
40-50 years
300+ Types
of
equipment
Huge source
of
CAPEX/OPEX

DATA CENTER LEAF-SPINE FABRIC
6
Reliability 😃 Flexibility 😃 Bandwidth 😃
Spine
Leaf

Metro
Router
White Box White Box
White Box
White Box
White Box White Box White Box White Box
White Box White Box White Box
White Box
White Box
White Box
Open Source
SDN-based
Bare-metal
White Box
White Box
R,E,M-
Access
7
ONOS Controller Cluster
vRouter
Control
XOS (Orchestrator)
vSG
VNF
VNF VNFVNFvSG VNF
VNF
VNFVNF
vSG VNF
VNF VNF
VNF
OVS OVS OVS OVS OVS
Residential Mobile Enterprise
Underlay
Control
Underlay
Overlay
Control
Overlay
vOLT
Control
Final CORD Architecture

Metro
Router
White Box White Box
White Box
White Box
White Box
White Box
White Box
Open Source
SDN-based
Bare-metal
White Box
White Box
R,E,M-
Access
8
vRouter
Control
XOS (Orchestrator)
vSG
VNF
VNF VNFVNFvSG VNF
VNF
VNFVNF
vSG VNF
VNF VNF
VNF
OVS OVS OVS OVS OVS
Underlay
Control
Underlay
Overlay
Control
Overlay
vOLT
Control

Metro
Router
White Box White Box
White Box
White Box
White Box
White Box
White Box
Open Source
SDN-based
Bare-metal
White Box
White Box
R,E,M-
Access
9
vRouter
Control
XOS (Orchestrator)
vSG
VNF
VNF VNFVNFvSG VNF
VNF
VNFVNF
vSG VNF
VNF VNF
VNF
OVS OVS OVS OVS OVS
Underlay
Control
Underlay
Overlay
Control
Overlay
vOLT
Control

SUMMARY
• VM-based NFV (Network Function Virtualization)
• Use ONOS (SDN controller) + Open VSwitch to control packets.
• Use XOS (Service Orchestration) to control all services (VNF)
• ONOS/XOS need to communicate with OpenStack component.

WHO?
• Ciena
• Hate OpenStack
• Intel China
• Some Telecom service providers
• China unicom

CHANGE TO KUBERNETES
• VM-based NFV (Network Function Virtualization)
• vSG, vPGW, vSGW, etc
• Who owns the NFV?
• Vendors
• We can’t force them to convert all NFVs to container.
• It’s impossible to have a kubernetes solution for CORD now.

• How about hybrid architecture?
• Support OpenStack + kubernetes

• Use ONOS (SDN controller) + Open vSwitch to control packets.
• There’re so many CNI for kubernetes now.
• None of them are Open vSwitch based solution.
• Linen-CNI is Open vSwitch + Linux Bridge solution
• Same subnet traffics between POD are controlled by Bridge
NODE
POD POD
OVS
br0
ens0p3

CHENLAGE TO KUBERNETES
• Use ONOS (SDN controller) + Open vSwitch to control packets.
• Create our own CNI to support pure Open vSwitch version.
• Still some problem need to conquer future. NODE
POD POD
OVS
ens0p3
？

NODE
POD POD
Bridge
ens0p3
many iptable rules.
many iptable rules.

• Use XOS (Service Orchestration) to control all services (VNF)
• Kubernetes can handle most thing.
• XOS should communicate with Kubernetes via its API service.

• ONOS/XOS need to communicate with OpenStack component.
• ONOS need to know the IP information of each host (VMs) from neutron component.
• In kubernetes, we need to provide the IP information of each POD.
• Since we implement our CNI, we can send the IP information after CNI assigns IP to POD.
• Send the information via Restful API/gRPC

PROBLEMS WE MET
• Deploy ONOS controller as container
• Multiple network interface for POD
• Centralized IP management

DEPLOY PROBLEM
• We need to deploy ONOS as container
• The chicken-and-egg conundrum!
• Hard to solve, we need to work-around now.
• We decide to move the ONOS out of POD.
• For each node, it should have multiple network interface including data network and control
network.
• Out of band.

DEPLOY PROBLEM
NODE
POD POD
OVS
ens0p3
ens0p4
NODE
POD POD
OVS
ens0p3
ens0p4
NODE
POD POD
OVS
ens0p3
ens0p4
Data network
Control network

MULTIPLE NETWORK INTERFACE
• For some NFV,(vSG) they need to multiple interface in its POD.
• For example.
NODE
POD POD
OVS
ens0p3

• We found a open source project (multus-CNI)
• Provides the multi interface support in a pod
• We don’t figure out how to use it first.
• We thought it’s a global setting.
• We try to implement by ourself.

• Multiple network interface means call CNI multiple times.
• For CNI, we need to know the namespace location for each POD.
• We want to provide a interface to dynamically call CNI for any existing POD.
• Input
• Pod Name, Network configuration (CNI name, configuration)
• Output
• Success (add another interface for existing POD)
• Error

• For each CNI, it should know the namespace location of each POD.
• We also need to know the POD name
• In the CNI plugin, we can get above information via
• Args.Args (many information, separate by semi-colon)
• Args.Netns
• We store those information in ETCD.

• After I have finished almost tasks.
• One intel guy tell us multus-cni support pod configuration.
• …..…..
• Ok, we use multus-cni.

CENTRALIZED IP MANAGEMENT
• Our CNI plugin use IPAM to handle the IP management.
• Official IPAM support two types.
• Host-local
• DHCP

IPAM DHCP
• Requirement
• You should run a IPAM DHCP daemon on each node.
• You should setup a DHCP server on your network.
• How it works.
• Start a DHCP client when a POD is be created.
• Forward the DHCP packet to DHCP server(it depends on your CNI forward L2 broadcast)
• Official recommend to use MacVLan as CNI.
• Limitation
• All node in same subnet.
• Simple configuration. You only specify “type=dhcp” in CNI configuration.

IPAM HOST-LOCAL
• Requirement
• None
• How it works
• Use a local file to record how many IP address has been used.
• Lookup the file and choose an available IP address for CNI.
• Limitation.
• You should prepare a configuration for each Node with different setting.
• Complex configuration. You need to specify which subnet the Node will use.
• Make sure no duplicate for each Node.

IPAM
• What we want ?
• Simple configuration
• Support multiple subnet
• We decided to create new IPAM

DHCP
• Refer to trellis (CORD network infrastructure)
• Requirement
• DHCP server
• How it works
• We set the Gateway Address for each dhcp request to support multiple subnet via L3 unicast.
• It looks like

DHCP
NODE 1
POD POD
OVS
ens0p3
POD POD
OVS
ens0p3
NODE 2
network
Master DHCP
Server
DHCP relay DHCP relay192.168.1.1 192.168.2.1
Subnet 192.168.1.0/24 {
}
Subnet 192.168.2.0/24 {
}

DHCP – PROBLEM
• The Problem is..
• How to decide the IP address of each Open vSwitch?
• Gateway address of each subnet.
• Use ETCD
• Maybe we can use ETCD to replace the DHCP

ETCD
• Implement a new IPAM which use etcd to record the subnet of each node.
• Simple configuration
• Network: 10.12.0.0/16
• Subnet length: 24
• etcd address.
• The subnet will from 10.12.1.0/24 to 10.12.255.0/24
• Simple and easy to implement.

NEXT STEP
• Integrate the ONOS (SDN controller) with our CNI.
• Make sure the ONOS can control the network.
• For the kubernetes internal communication, the ONOS should implement all of them by
Openflow, not iptables.NODE
POD POD
OVS
ens0p3
Use openflow rules
many iptable rules.

Integrate Kubernetes into CORD(Central Office Re-architected as a Datacenter)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Integrate Kubernetes into CORD(Central Office Re-architected as a Datacenter)

Similar to Integrate Kubernetes into CORD(Central Office Re-architected as a Datacenter) (20)

More from inwin stack

More from inwin stack (20)

Recently uploaded

Recently uploaded (20)

Integrate Kubernetes into CORD(Central Office Re-architected as a Datacenter)

Editor's Notes