Network policy support¶

Translate Kubernetes policy to Neutron security-group¶

{
  "security_group_rule": {
    "direction": "egress",
    "protocol": null,
    "security_group_id": "[id]"
  }
}

Match between security-groups and its ports¶

A security-group that derived from kubernetes-policy should be tagged [7] with network_policy UID. In case of issue of length or characters-set hashing should be applied, so security-group unique tag could be derived from the policy-UID.

For defining on which pods the policy should be applied Kubernetes defines a pod-selector query [5]. For applying the policy on the relevant ports, Kuryr needs to know at any given moment which pods belong to that group. It can happen when pod is created/updated/change-label.

When policy is created, Kuryr should trigger a get query for applying the policy on all pods that already match, and add a watch for getting an update when POD added or removed from network-policy and apply/remove the translated policy on the pod’s port.

For applying the policy on the pod, an annotation with the security-group-id will be added to the pod. That will cause the “update pod” event. The VIFHandler via security-group Driver will attach the pod to the port.

We can’t attach the security-group directly in the watch callback as it will create a race condition between the watch and the VIFHandler as the watch could be called before Kuryr notified that the pod is created. With the annotation - when new pod is created, if the watch was called before VIFHandler pod creation processing, VIFHandler will get the pod already with the annotation. Otherwise, it will get pod with no security-group-annotation and will attach it to the default security-group. When the watch will update the annotation, the pod will be updated with the correct security-group.

When policy is updated, if policy pod-selectors changed, a diff between the old and new selected-pod-set should be done, and the pods security-groups annotations should be updated respectively. Selector watches should be updated with the new queries.

Allow traffic from the pod ingress and egress selectors:¶

As mentioned above, “remote_group_id” will be used to allow ingress and egress traffic from pods selected by the pod/namespace selectors.

For the pod-selector and namespace-selector we need to create a security-group per policy (one for ingress and one for egress). The security-group should be tagged with tag that is derived from the policy-UID and traffic direction (for example: [policy_UID]_EG for egress traffic). In case of characters-sets or allowed-length issues, hash should be applied for updating these security-groups.

For each selector (namespace or pod) a watch should be set. The watch callback will add/remove the relevant pods to/from the security-group.

Controller Handlers and Drivers impact:¶

For supporting Network-policy Handler that watches network_policy events will be added.

Two new drivers will be added:

• On the network_policy Handler:
  • “Network Policy Apply Driver”, it will have 3 responsibilities:
    • Translate the network-policy to security-groups.
    • Match the security-group to its relevant port (by setting the watch and annotating the pod as described above).
    • Update the security-groups for ingress/egress POD/namespace selector.
• On VIF handler:
  • A security-group-policy Driver will be used instead of the default security groups Driver. It will be responsible for:
    • Set the port security-group according to the annotation.

Controller startup¶

The following should be done on Controller startup:

• Retrieve from Kubernetes API all network-policies and set all the relevant watches. This should happen before the Controller starts.
• Need to do some sync operation to make-sure Neutron topology is synchronised with Kubernetes Network Policy model.
  • for every network-policy:
    • Get it’s main security-group and check if it’s updated. This validation will be done by the generation tag. Generation is part of k8s metadata and increased by one on every policy change. When the policy will applied by kuryr, it’s should be annotated with current policy generation. If the generation in the policy meta-data is newer than the generation in the annotation, it’s mean that the policy had been changed and the security-group rules needs to rebuild.
    • for each pod selector in the policy:
      • get from kubernetes-api all pods that selected by this query.
      • get all ports of the relevant security-groups.
      • Do diff between port that needed to be attached to SG, and add/remove pod-ports from security-groups.

Port pool impact¶

Changes in the security-policy can cause negative impact on the port-pools [9]. The combination of the security-groups of port is part of the pool key, and changes in network-policy could make some pools not relevant any more.

For example let’s assume that we have 2 policies “a” and “b”, and both policies should be applied on pods with the label “role: db”. When the first pod with label “role: db” is created - a new port-pool is created and its pool key is composed from the security-groups of “a” and “b”. If policy “b” will be changed and pods with label “role: db” would not be selected by the policy anymore, then the port-pool that was created for the combination of “a” and “b” will not be not useful any more.

That can lead to the ports leak, as pool holds many ports that not useful anymore. For handling this issue a new cleanup task should be added. This task will release all ports from the pools that are not in use anymore.

Another issue that needs to be treated is that the policies of pod can be changed while pod is running. Currently when pod is deleted its’ port is returned to the pool that it was taken from. But if the pod’s policies are changed, this behaviour is incorrect. When port is released it should be returned to the pool that matches to the current state of the pod.

Execution flow diagram¶

See below the network policy attachment to the pod after pod creation:

Possible optimization:¶

Kubernetes label-selector divided into 2 types of queries “match-labels”, and “match-expression” [10]. “match-labels” selects a closed list of labels while “match-expression” selects all pods that match to particular expression.

This spec suggests to create a watch for each label-selector query, because in “match-expression” queries it is not possible to determine if pod matches the query, without implementing parser for each expression. By setting a watch we are using kubernetes-api-server for the matching between pods and queries.

The spec treats the “match-labels” and “match-expression” queries in the same way for simplicity reasons. But future optimization may distinguish between queries-types. “match-labels” queries watches may be removed and the matching between pod to its’ “match-labels” queries could be done directly on the vif-handler.

Assumptions¶

Security-groups are supported by the networking backend for all vif interfaces. In case of special interfaces (SR-IOV, mac-vlan, etc ..), the network-policy will be applied on the interface if and only if then networking backend enables security-groups on those interfaces.

Execution flow-example¶

This section describes system execution flow in the following scenarios:

• POD is deployed on empty system.
• Network-policy that should be applied on the first pod is deployed.
• Another pod that belongs to the network-policy is deployed.

Pod is deployed on empty system:

• Pod is created with the following details:
  • name: p1, namespace: default, labels : {Role:db}.
• VIF Handler:
  • Security-group-policy driver:
    • Assign default-policy to this pod (as we still do not have any network-policy in the system).
    • Create security-group for namespace ‘default’, and add pod p1 port to that security-group.

Network policy is deployed:

Let’s assume that following policy is created (taken from k8s tutorial [8]):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-network-policy
  namespace: default
spec:
  podSelector:
    matchLabels:
      role: db
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - ipBlock:
        cidr: 172.17.0.0/16
        except:
        - 172.17.1.0/24
    - namespaceSelector:
        matchLabels:
          project: myproject
    - podSelector:
        matchLabels:
          role: frontend
    ports:
    - protocol: TCP
      port: 6379
  egress:
  - to:
    - ipBlock:
      cidr: 10.0.0.0/24
    ports:
    - protocol: TCP
      port: 5978

• Network policy Handler:
  • Network policy Driver:
    • Create security group with the following rules:
      • Ingress , tcp:6379 172.17.0.0/24
      • Ingress , tcp:6379 172.17.2.0/23
      • Ingress , tcp:6379 172.17.4.0.0/22
      • Ingress , tcp:6379 172.17.8.0/21
      • Ingress , tcp:6379 172.17.16.0/20
      • Ingress , tcp:6379 172.17.32.0/19
      • Ingress , tcp:6379 172.17.64.0/18
      • Ingress , tcp:6379 172.17.128.0/17
      • Ingress , tcp: 6379 , remote_group_id : [test-network-policy-uid]_in
      • Egress, tcp:5978 10.0.0.0/24
    • Create match for the policy:
      • Queries k8s-api about pods that match to {role:db}
        • Attach the annotation with security-policy-id to p1.
      • Set a watch on the query - “Match-label : {role:db}” , the watch callback of this watch will update the security-group annotation on the updated/new pods that are selected by this query.
    • Create a match for the ingress/egress group:
      • Set watch on the query : “match-labels {role:frontend}” , watch callback will add all pods that are selected by this query to the security-group [test-network-policy-uid]_in.
• VIF Handler:
  • Will get update event on p1 as its’ annotations is changed.
    • security-policy-group-driver:
      • Attach the interface to its security-group.

Second pod is created:

• pod is created with the details:
  • name: p2 , namespace: default , labels : {Role:db}.
• Let’s assume that VIF handler is called before watch-callback (as this case is little more complicated).
• VIF Handler:
  • Pod created event (still no namespace sg annotation on the pod).
  • Namespace security-group driver
    • Return the default network-policy.
  • Pod is created with default-policy.
• Watch Callback:
  • p2 is selected by security-group net-policy-test, annotates the pod with security-group-id that matches to the policy.
• VIF Handler:
  • security-group policy driver
    • Update pod P2 port with network policy driver.