20180510 Kubernetes 開源容器 技術論壇 2018 白凱仁（迎棧科技）

1. Setup Hybrid Clusters using
Kubernetes Federation
Kyle bai
R&D @ inwinSTACK
www.inwinstack.com

2. About Me
⽩白凱仁(Kyle Bai)
• Interested in emerging technologies.
• COSCUP, Kubernetes Day and OpenStack Day Speaker.
• OpenStack and Kubernetes Projects Contributor(100+ PR).
• Certified Kubernetes Administrator.
@kairen(k2r2.bai@gmail.com)
https://kairen.github.io/

3. Agenda
• Why Federation?
• Federation Architecture
• How to setup Kubernetes Federation
• Bonus

4. Why Federation?

5. Dev Test
Multiple Clusters
Product

6. Dev Test
Multiple Clusters
Product
US EU
AP TW

7. Dev Test
Multiple Clusters
Product
US EU
AP TW

8. Multiple Clusters
TWLocal
Dev

9. Multiple Clusters
TWUS
Test
TWLocal
Dev

10. Multiple Clusters
US
EU
…
AP
Product
TWUS
Test
TWLocal
Dev

11. Multiple Clusters
• Serving users from clusters closest to them.
• Avoiding provider lock-in: By making it easier to migrate applications
across clusters, federation prevents cluster provider lock-in.
• High availability: Ability to federate clusters across different regions/
cloud providers.

12. Federation
Federation makes it easy to manage
multiple clusters.
• Sync resources across clusters:
Federation provides the ability to keep
resources in multiple clusters in sync.
• Cross cluster discovery: Federation
provides the ability to auto-configure
DNS servers and load balancers with
backends from all clusters.

13. Federation Pros
• Keep your app synced across clusters.
• Configure network resources (services, ingress) to route traffic across
clusters.
• Single place to apply policies.
• Policy-based Resource Placement(OPA).

14. Use cases for Federation
• Geographically Distributed Deployments: Spread Deployments across
clusters in different parts of the world.
• Hybrid Cloud: Extend Deployments from on-premise clusters to the cloud.
• Application Migration: Simplify the migration of applications from on-
premise to the cloud or between cloud providers.

15. Federation Architecture

16. Overview
Federation Control Plane

17. Overview
Federation Control Plane
Federated Resources

18. Overview
Federation Control Plane
Federated Resources
Federated Clusters

19. Federation API Server
The Kubernetes federation API server validates and configures data for the API
objects which include pods, services, replicationcontrollers, and others.

20. Federation Controller Manager
The federation controller manager is a daemon that embeds the core control loops
shipped with federation.
• Watches Federation API Server
• Clusters - federation/v1beta1/cluster
• API Resources - v1/foo
• Watches All Kubernetes Clusters
• API Resources - v1/foo
• Reconciles
• Compare and update
• Handles cascading deletion

21. How to place? • Placement can be controlled per-resource via
annotations.
• Annotations supported:
• federation.kubernetes.io/replica-set-preferences
• federation.kubernetes.io/deployment-preferences
• federation.alpha.kubernetes.io/cluster-selector
• federation-controller evaluates annotations to
produce final placement

22. How to setup Kubernetes
Federation

23. Prerequisites(1/2)
• In this demon, we will deploy clusters in different regions:
• US West: Oregon (us-west)
• Asia: Tokyo(ap-northeast)
• My OpenStack: Taipei(os-taipei)
• Install the following tools on host:
• kubectl
• kubefed: if os is Mac OS X, you need build from Federation source code.
• kops, kubespray, kubeadm
• AWS CLI

24. Prerequisites(2/2)
AWS and OpenStack:
• IAM(Keystone): Identity and Access Management.
• EC2(Nova): Kubernetes cluster.
• ELB: Kubernetes service load balancer.
• Route53: Public domain for Kubernetes API, Service, … etc
• S3: Store kops state.
• VPC(Neutron): Cluster Network.
• EBS(Cinder): Instance volumes.
Godaddy:
• Buy your own domain name.

25. Deme Clusters
kubefed init ${FED_CONTEXT}
--image=kairen/fcp-amd64:v1.10.0-alpha
--host-cluster-context=${US_WEST_CONTEXT}
--dns-provider=aws-route53
--dns-zone-name=${DOMAIN_NAME}

26. Deme Clusters
kubefed join os-taipei
--host-cluster-context=${US_WEST_CONTEXT}
--cluster-context=${OS_CONTEXT}

27. Deme Topology

28. Bonus

29. Federation + OPA(Open Policy Agent )
Current Use Cases
• Federated Workload Placement
• Pod Scheduling
• Authorization
• Admission Control
• Audit
Future Use Cases
• Storage policy
• Network policy

30. Federation + OPA(Open Policy Agent )

31. Federation Cons
• Increased network bandwidth and cost: The federation control plane
watches all clusters to ensure that the current state is as expected.
• Reduced cross cluster isolation: A bug in the federation control plane
can impact all clusters.
• Maturity: The federation project is relatively new and is not very mature.
• Not all resources are available and many are still alpha and beta.

32. Federation support API resources
• Cluster
• ConfigMap
• DaemonSets
• Deployment
• Events
• HPA
• Ingress
• Jobs
• Namespaces
• ReplicaSets
• Secrets
• Services
* Only support API version for extensions/v1beta1, v1

33. Federation v2
https://github.com/kubernetes-sigs/federation-v2
The prototype builds on the sync controller
(a.k.a. push reconciler) from Federation v1.
• Possible to implement a custom scheduler
• Policies modeled as CRDs
• Use existing RBAC
• Can plug into policy engine
• Implemented as an aggregated API server
• Secured with ServiceAccounts
https://github.com/font/k8s-example-apps/tree/master/sample-fed-v2-demo

34. Federation v2
The following abstractions support the propagation
of a logical federated type:
• Template: defines the representation of the resource
common across clusters.
• Placement: defines which clusters the resource is intended
to appear in.
• Override: optionally defines per-cluster field-level variation to
apply to the template.
https://docs.google.com/document/d/159cQGlfgXo6O4WxXyWzjZiPoIuiHVl933B43xhmqPEE/edit