Installing the Calico network policy engine add-on

1. You can install Calico using Helm or Kubernetes manifests.

   Helm

   Prerequisite

   Helm version 3.0 or later installed on your computer. To install or upgrade Helm, see Using Helm with Amazon EKS.

   To install Calico with Helm

   1. Add Project Calico into your Helm repository.

      helm repo add projectcalico https://docs.projectcalico.org/charts

   2. If you already have Calico added, you can update it to get the latest released version.

      helm repo update                         

   3. Install version 3.21.4 or later of the Tigera Calico operator and custom resource definitions.

      helm install calico projectcalico/tigera-operator --version v3.21.4

   Manifests

   Apply the Calico manifests to your cluster.

   kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/master/config/master/calico-operator.yaml
   kubectl apply -f https://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/master/config/master/calico-crs.yaml

2. View the resources in the tigera-operator namespace.

   kubectl get all -n tigera-operator

   The example output is as follows.

   NAME                                   READY   STATUS    RESTARTS   AGE
   pod/tigera-operator-768d489967-6cv58   1/1     Running   0          27m
   
   NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/tigera-operator   1/1     1            1           27m
   
   NAME                                         DESIRED   CURRENT   READY   AGE
   replicaset.apps/tigera-operator-768d489967   1         1         1       27m

   The values in the DESIRED and READY columns for the replicaset should match.

3. View the resources in the calico-system namespace.

   kubectl get all -n calico-system

   The example output is as follows.

   NAME                                           READY   STATUS    RESTARTS   AGE
   pod/calico-kube-controllers-5cd7d477df-2xqpd   1/1     Running   0          40m
   pod/calico-node-bm5fb                          1/1     Running   0          40m
   pod/calico-node-wfww4                          1/1     Running   0          40m
   pod/calico-typha-86c697dbdc-n7qff              1/1     Running   0          40m
   
   NAME                                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
   service/calico-kube-controllers-metrics   ClusterIP   10.100.77.235   <none>        9094/TCP   40m
   service/calico-typha                      ClusterIP   10.100.164.91   <none>        5473/TCP   40m
   
   NAME                                    DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR              AGE
   daemonset.apps/calico-node              2         2         2       2            2           kubernetes.io/os=linux     40m
   
   NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
   deployment.apps/calico-kube-controllers   1/1     1            1           40m
   deployment.apps/calico-typha              1/1     1            1           40m
   
   NAME                                                 DESIRED   CURRENT   READY   AGE
   replicaset.apps/calico-kube-controllers-5cd7d477df   1         1         1       40m
   replicaset.apps/calico-typha-86c697dbdc              1         1         1       40m

   The output is slightly different depending on whether you deployed using the Helm chart or manifests. The values in the DESIRED and READY columns for the calico-node daemonset should match. The values in the DESIRED and READY columns for the two replicasets should also match. The number in the DESIRED column for daemonset.apps/calico-node varies based on the number of nodes in your cluster.

4. Confirm that the logs for one of your calico-node, calico-typha, and tigera-operator pods don't contain ERROR. Replace the values in the following commands with the values returned in your output for the previous steps.

   kubectl logs tigera-operator-768d489967-6cv58 -n tigera-operator | grep ERROR
   kubectl logs calico-node-bm5fb -c calico-node -n calico-system | grep ERROR
   kubectl logs calico-typha-86c697dbdc-n7qff -n calico-system | grep ERROR

   If no output is returned from the previous commands, then ERROR doesn't exist in your logs and everything should be running correctly.

5. If you're using version 1.9.3 or later of the Amazon VPC CNI plugin for Kubernetes, then enable the plugin to add the pod IP address to an annotation in the calico-kube-controllers-579b45dcf-z5tsf pod spec. For more information about this setting, see ANNOTATE_POD_IP on GitHub.

   1. See which version of the plugin is installed on your cluster with the following command.

      kubectl describe daemonset aws-node -n kube-system | grep amazon-k8s-cni: | cut -d ":" -f 3
      

      The example output is as follows.

      v1.10.4-eksbuild.1

   2. Create a configuration file that you can apply to your cluster that grants the aws-node Kubernetes clusterrole the permission to patch pods.

      cat << EOF > append.yaml
      - apiGroups:
        - ""
        resources:
        - pods
        verbs:
        - patch
      EOF

   3. Apply the updated permissions to your cluster.

      kubectl apply -f <(cat <(kubectl get clusterrole aws-node -o yaml) append.yaml)

   4. Set the environment variable for the plugin.

      kubectl set env daemonset aws-node -n kube-system ANNOTATE_POD_IP=true

   5. Confirm that the annotation was added to the calico-kube-controllers-5cd7d477df-2xqpd pod. Replace 5cd7d477df-2xqpd with the ID for the pod returned in a previous step.

      kubectl describe pod calico-kube-controllers-5cd7d477df-2xqpd -n calico-system | grep vpc.amazonaws.com/pod-ips

      The example output is as follows.

      vpc.amazonaws.com/pod-ips: 192.168.25.9

To run the Stars policy demo

1. Apply the front-end, back-end, client, and management user interface services:

   kubectl apply -f https://docs.projectcalico.org/v3.5/getting-started/kubernetes/tutorials/stars-policy/manifests/00-namespace.yaml
   kubectl apply -f https://docs.projectcalico.org/v3.5/getting-started/kubernetes/tutorials/stars-policy/manifests/01-management-ui.yaml
   kubectl apply -f https://docs.projectcalico.org/v3.5/getting-started/kubernetes/tutorials/stars-policy/manifests/02-backend.yaml
   kubectl apply -f https://docs.projectcalico.org/v3.5/getting-started/kubernetes/tutorials/stars-policy/manifests/03-frontend.yaml
   kubectl apply -f https://docs.projectcalico.org/v3.5/getting-started/kubernetes/tutorials/stars-policy/manifests/04-client.yaml

2. View all pods on the cluster.

   kubectl get pods -A

   The example output is as follows.

   In your output, you should see pods in the namespaces shown in the following output. Your pod NAMES and the number of pods in the READY column are different than those in the following output. Don't continue until you see pods with similar names and they all have Running in the STATUS column.

   NAMESPACE         NAME                                       READY   STATUS    RESTARTS   AGE
   ...
   client            client-xlffc                               1/1     Running   0          5m19s
   ...
   management-ui     management-ui-qrb2g                        1/1     Running   0          5m24s
   stars             backend-sz87q                              1/1     Running   0          5m23s
   stars             frontend-cscnf                             1/1     Running   0          5m21s
   ...

3. To connect to the management user interface, forward your local port 9001 to the management-ui service running on your cluster:

   kubectl port-forward service/management-ui -n management-ui 9001

4. Open a browser on your local system and point it to http://localhost:9001/. You should see the management user interface. The C node is the client service, the F node is the front-end service, and the B node is the back-end service. Each node has full communication access to all other nodes, as indicated by the bold, colored lines.

   

5. Apply the following network policies to isolate the services from each other:

   kubectl apply -n stars -f https://docs.projectcalico.org/v3.5/getting-started/kubernetes/tutorials/stars-policy/policies/default-deny.yaml
   kubectl apply -n client -f https://docs.projectcalico.org/v3.5/getting-started/kubernetes/tutorials/stars-policy/policies/default-deny.yaml

6. Refresh your browser. You see that the management user interface can no longer reach any of the nodes, so they don't show up in the user interface.

7. Apply the following network policies to allow the management user interface to access the services:

   kubectl apply -f https://docs.projectcalico.org/v3.5/getting-started/kubernetes/tutorials/stars-policy/policies/allow-ui.yaml
   kubectl apply -f https://docs.projectcalico.org/v3.5/getting-started/kubernetes/tutorials/stars-policy/policies/allow-ui-client.yaml

8. Refresh your browser. You see that the management user interface can reach the nodes again, but the nodes cannot communicate with each other.

   

9. Apply the following network policy to allow traffic from the front-end service to the back-end service:

   kubectl apply -f https://docs.projectcalico.org/v3.5/getting-started/kubernetes/tutorials/stars-policy/policies/backend-policy.yaml

10. Refresh your browser. You see that the front-end can communicate with the back-end.

    

11. Apply the following network policy to allow traffic from the client to the front-end service.

    kubectl apply -f https://docs.projectcalico.org/v3.5/getting-started/kubernetes/tutorials/stars-policy/policies/frontend-policy.yaml

12. Refresh your browser. You see that the client can communicate to the front-end service. The front-end service can still communicate to the back-end service.

    

13. (Optional) When you are done with the demo, you can delete its resources.

    kubectl delete -f https://docs.projectcalico.org/v3.5/getting-started/kubernetes/tutorials/stars-policy/manifests/04-client.yaml
    kubectl delete -f https://docs.projectcalico.org/v3.5/getting-started/kubernetes/tutorials/stars-policy/manifests/03-frontend.yaml
    kubectl delete -f https://docs.projectcalico.org/v3.5/getting-started/kubernetes/tutorials/stars-policy/manifests/02-backend.yaml
    kubectl delete -f https://docs.projectcalico.org/v3.5/getting-started/kubernetes/tutorials/stars-policy/manifests/01-management-ui.yaml
    kubectl delete -f https://docs.projectcalico.org/v3.5/getting-started/kubernetes/tutorials/stars-policy/manifests/00-namespace.yaml

    Even after deleting the resources, there can still be iptables rules on the nodes that might interfere in unexpected ways with networking in your cluster. The only sure way to remove Calico is to terminate all of the nodes and recycle them. To terminate all nodes, either set the Auto Scaling Group desired count to 0, then back up to the desired number, or just terminate the nodes. If you are unable to recycle the nodes, then see Disabling and removing Calico Policy in the Calico GitHub repository for a last resort procedure.