Knative Function Troubleshooting
If a function is not behaving as expected, you can look at the logs to troubleshoot. You can either perform this operation remotely by copying the /root/.kube/config onto your local desktop and you can interact with VEBA using your local kubectl client or SSH to the appliance using the local kubectl client.
List out the pods.
kubectl get pods -A
This is an example output:
NAMESPACE NAME READY STATUS RESTARTS AGE
contour-external contour-5869594b-b5tqk 1/1 Running 0 22h
contour-external contour-5869594b-q94vd 1/1 Running 0 22h
contour-external contour-certgen-v1.10.0-hm65q 0/1 Completed 0 22h
contour-external envoy-6p2bv 2/2 Running 0 22h
contour-internal contour-5d47766fd8-5skt7 1/1 Running 0 22h
contour-internal contour-5d47766fd8-6r5g4 1/1 Running 0 22h
contour-internal contour-certgen-v1.10.0-rdd6z 0/1 Completed 0 22h
contour-internal envoy-wwxct 2/2 Running 0 22h
knative-eventing eventing-controller-658f454d9d-mnqjb 1/1 Running 0 22h
knative-eventing eventing-webhook-69fdcdf8d4-mljtb 1/1 Running 0 22h
knative-eventing rabbitmq-broker-controller-88fc96b44-bbvnb 1/1 Running 0 22h
knative-serving activator-85cd6f6f9-wl7rf 1/1 Running 0 22h
knative-serving autoscaler-7959969587-z9rtq 1/1 Running 0 22h
knative-serving contour-ingress-controller-6d5777577c-f5qzn 1/1 Running 0 22h
knative-serving controller-577558f799-mdjpt 1/1 Running 0 22h
knative-serving webhook-78f446786-bn7xm 1/1 Running 0 22h
kube-system antrea-agent-vbr5d 2/2 Running 0 22h
kube-system antrea-controller-85c944dc84-jc28b 1/1 Running 0 22h
kube-system coredns-74ff55c5b-rqm7c 1/1 Running 0 22h
kube-system coredns-74ff55c5b-vq827 1/1 Running 0 22h
kube-system etcd-sjc-veba-01.tshirts.inc 1/1 Running 0 22h
kube-system kube-apiserver-sjc-veba-01.tshirts.inc 1/1 Running 0 22h
kube-system kube-controller-manager-sjc-veba-01.tshirts.inc 1/1 Running 0 22h
kube-system kube-proxy-mwpxs 1/1 Running 0 22h
kube-system kube-scheduler-sjc-veba-01.tshirts.inc 1/1 Running 0 22h
local-path-storage local-path-provisioner-5696dbb894-7x626 1/1 Running 0 22h
rabbitmq-system rabbitmq-cluster-operator-7bbbb8d559-dqd85 1/1 Running 0 22h
vmware-functions default-broker-ingress-5c98bf68bc-2zpc6 1/1 Running 0 22h
vmware-functions kn-pcli-tag-00001-deployment-c845447d4-lnmrq 2/2 Running 0 7h41m
vmware-functions sockeye-65697bdfc4-cmfxc 1/1 Running 0 22h
vmware-functions sockeye-trigger-dispatcher-7f4dbd7f78-n589p 1/1 Running 0 22h
vmware-functions veba-pcli-tag-trigger-dispatcher-7b477dd84d-zl2vm 1/1 Running 0 7h41m
vmware-system cadvisor-sk4j9 1/1 Running 0 22h
vmware-system tinywww-dd88dc7db-dqnnc 1/1 Running 0 22h
vmware-system veba-rabbit-server-0 1/1 Running 0 22h
vmware-system veba-ui-54967b4bf4-lpjrn 1/1 Running 0 22h
vmware-system vmware-event-router-vcenter-6b76959df5-6mrb4 1/1 Running 3 22h
vmware-system vmware-event-router-webhook-6b48cc5b8c-sjzx8 1/1 Running 0 22h
First, we want to see if the event router is capturing events and forwarding them on to a function.
Use this command to follow the live Event Router log.
kubectl logs -n vmware-system deploy/vmware-event-router-vcenter
For this sample troubleshooting, we have the sample PowerCLI Tagging function running which will react to a VM powered on Event (DrsVmPoweredOnEvent). To see if the appliance is properly handling the event, create a test VM and power it on before proceeding with the next steps.
When we look at the log output, we see various entries regarding DrsVmPoweredOnEvent, ending with the following:
2021-09-24T13:57:05.015Z INFO [KNATIVE] knative/knative.go:181 sending event {"eventID": "c6d61d55-8100-459e-a1e7-7a936bac6e43", "subject": "DrsVmPoweredOnEvent"}
2021-09-24T13:57:05.015Z INFO [KNATIVE] knative/knative.go:193 successfully sent event {"eventID": "c6d61d55-8100-459e-a1e7-7a936bac6e43"}
2021-09-24T13:57:06.017Z INFO [VCENTER] vcenter/vcenter.go:343 invoking processor {"eventID": "87af3e86-6516-4377-9a9b-86c4c9b00b05"}
This lets us know that the function was invoked. If we still don’t see the expected result, we need to look at the function logs.
Each Knative function will have its own pod running in the vmware-functions namespace. If you have deployed the provided tagging function example from the VEBA examples, you can examine the logs with the following command.
kubectl logs -n vmware-functions deployment/kn-pcli-tag-00001-deployment user-container
Note: Replace the name of the deployment in the examples with the name within your environment.
We don’t need the --follow switch because we are just trying to look at recent logs, but --follow would work too.
This command will show you the last 5 minutes worth of logs.
kubectl logs -n vmware-functions deployment/kn-pcli-tag-00001-deployment user-container --since=5m
This command will show you the last 20 lines of logs.
kubectl logs -n vmware-functions deployment/kn-pcli-tag-00001-deployment user-container --tail=20
Log output showing a successful function invocation:
09/24/2021 13:57:05 - Applying vSphere Tag "VEBA" to My-VEBA-Test-VM ...
09/24/2021 13:57:09 - vSphere Tag Operation complete ...
09/24/2021 13:57:09 - Handler Processing Completed ...
Still having trouble?
Please submit bug reports and feature requests by using our GitHub Issues page or Join us on slack #vcenter-event-broker-appliance on vmwarecode.slack.com.