You must log on to the VMware Event Broker appliance as root. If you did not enable SSH as part of the initial VMware Event Broker Appliance deployment, you can perform this operation at the console. To enable SSH access, execute the following command:
systemctl start sshd
This turns on the SSH daemon but does not enable it to start on appliance boot. You should now be able to SSH into the appliance.
If you wish to disable the SSH daemon when you are done troubleshooting, execute the following command:
systemctl stop sshd
If the appliance is not working immediately after deployment, the first thing to do is check your Kubernetes pods.
kubectl get pods -A
Here is the command output:
k get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
contour-external contour-5869594b-bmccv 1/1 Running 0 6h37m
contour-external contour-5869594b-jr4k8 1/1 Running 0 6h37m
contour-external contour-certgen-v1.10.0-btmlp 0/1 Completed 0 6h37m
contour-external envoy-hzhnz 2/2 Running 0 6h37m
contour-internal contour-5d47766fd8-5shp2 1/1 Running 0 6h37m
contour-internal contour-5d47766fd8-hv9zl 1/1 Running 0 6h37m
contour-internal contour-certgen-v1.10.0-szssj 0/1 Completed 0 6h37m
contour-internal envoy-hpch5 2/2 Running 0 6h37m
knative-eventing eventing-controller-658f454d9d-pfs5d 1/1 Running 0 6h37m
knative-eventing eventing-webhook-69fdcdf8d4-fdtmp 1/1 Running 0 6h37m
knative-eventing rabbitmq-broker-controller-88fc96b44-6jb82 1/1 Running 0 6h37m
knative-serving activator-85cd6f6f9-7l9q5 1/1 Running 0 6h38m
knative-serving autoscaler-7959969587-kzdrj 1/1 Running 0 6h38m
knative-serving contour-ingress-controller-6d5777577c-sb6vr 1/1 Running 0 6h37m
knative-serving controller-577558f799-fmfgq 1/1 Running 0 6h38m
knative-serving webhook-78f446786-zj9n4 1/1 Running 0 6h38m
kube-system antrea-agent-xgn24 2/2 Running 0 6h38m
kube-system antrea-controller-849fff8c5d-h5tjb 1/1 Running 0 6h38m
kube-system coredns-74ff55c5b-kpfxz 1/1 Running 0 6h38m
kube-system coredns-74ff55c5b-wrjp4 1/1 Running 0 6h38m
kube-system etcd-veba.primp-industries.local 1/1 Running 0 6h38m
kube-system kube-apiserver-veba.primp-industries.local 1/1 Running 0 6h38m
kube-system kube-controller-manager-veba.primp-industries.local 1/1 Running 0 6h38m
kube-system kube-proxy-gs59c 1/1 Running 0 6h38m
kube-system kube-scheduler-veba.primp-industries.local 1/1 Running 0 6h38m
local-path-storage local-path-provisioner-5696dbb894-sf457 1/1 Running 0 6h38m
rabbitmq-system rabbitmq-cluster-operator-7bbbb8d559-sw4kt 1/1 Running 0 6h37m
vmware-functions default-broker-ingress-5c98bf68bc-nl5m7 1/1 Running 0 6h36m
vmware-system tinywww-dd88dc7db-f74br 1/1 Running 0 6h36m
vmware-system veba-rabbit-server-0 1/1 Running 0 6h36m
vmware-system veba-ui-677b77dfcf-q9t84 1/1 Running 0 6h36m
vmware-system vmware-event-router-5dd9c8f858-5c9mh 0/1 CrashLoopBackoff 6 4d13h
Note: The status
Completed
of the containercontour-certgen-v1.10.0-btmlp
is expected after successful appliance deployment.
One of the first things to look for is whether a pod is in a crash state.
In the above case, the vmware-event-router pod is crashing. We need to look at the logs with this command:
kubectl -n vmware-system logs vmware-event-router-5dd9c8f858-5c9mh
Note: The pod suffix
-5dd9c8f858-5c9mh
will be different in each environment
Here is the command output:
_ ____ ___ ______ __ ____ __
| | / / |/ / ______ _________ / ____/ _____ ____ / /_ / __ \____ __ __/ /____ _____
| | / / /|_/ / | /| / / __ / ___/ _ \ / __/ | | / / _ \/ __ \/ __/ / /_/ / __ \/ / / / __/ _ \/ ___/
| |/ / / / /| |/ |/ / /_/ / / / __/ / /___ | |/ / __/ / / / /_ / _, _/ /_/ / /_/ / /_/ __/ /
|___/_/ /_/ |__/|__/\__,_/_/ \___/ /_____/ |___/\___/_/ /_/\__/ /_/ |_|\____/\__,_/\__/\___/_/
[VMware Event Router] 2020/03/10 18:59:47 connecting to vCenter https://vc01.labad.int/sdk
[VMware Event Router] 2020/03/10 18:59:52 could not connect to vCenter: could not create vCenter client: ServerFaultCode: Cannot complete login due to an incorrect user name or password.
The error message shows us that we made a mistake when we configured our username or password. We must now edit the Event Router JSON configuration file to fix the mistake.
vi /root/config/event-router/vmware-event-router-config-vcenter.yaml
Here is some of the YAML from the config file - you can see the mistake in the credentials. Fix the credentials and save the file.
eventProvider:
name: veba-vc-01
type: vcenter
vcenter:
address: https://vc01.labad.int/sdk
auth:
basicAuth:
password: "wrongPassword"
username: "[email protected]"
type: basic_auth
insecureSSL: true
checkpoint: false
We now fix the Kubernetes configuration with 3 commands - delete and recreate the secret file, then delete the broken pod. Kubernetes will automatically spin up a new pod with the new configuration. We need to do this because the YAML configuration file is not directly referenced by the event router. The YAML file is mounted into the event router pod as a Kubernetes secret.
kubectl -n vmware-system delete secret event-router-config
kubectl -n vmware-system create secret generic event-router-config --from-file=event-router-config.yaml
kubectl -n vmware-system delete pod vmware-event-router-5dd9c8f858-5c9mh
We get a pod list again to determine the name of the new pod.
kubectl -n vmware-system get pods
Here is the command output:
NAME READY STATUS RESTARTS AGE
tinywww-dd88dc7db-f74br 1/1 Running 0 6h39m
veba-rabbit-server-0 1/1 Running 0 6h40m
veba-ui-677b77dfcf-q9t84 1/1 Running 0 6h39m
vmware-event-router-5dd9c8f858-5c9mh 0/1 Terminating 3 6h40m
vmware-event-router-5dd9c8f858-wt64s 1/1 Running 0 28s
Now view the event router logs.
kubectl -n vmware-system logs vmware-event-router-5dd9c8f858-wt64s
Here is the command output:
_ ____ ___ ______ __ ____ __
| | / / |/ / ______ _________ / ____/ _____ ____ / /_ / __ \____ __ __/ /____ _____
| | / / /|_/ / | /| / / __ / ___/ _ \ / __/ | | / / _ \/ __ \/ __/ / /_/ / __ \/ / / / __/ _ \/ ___/
| |/ / / / /| |/ |/ / /_/ / / / __/ / /___ | |/ / __/ / / / /_ / _, _/ /_/ / /_/ / /_/ __/ /
|___/_/ /_/ |__/|__/\__,_/_/ \___/ /_____/ |___/\___/_/ /_/\__/ /_/ |_|\____/\__,_/\__/\___/_/
2021-04-04T14:30:16.589Z ESC[34mINFOESC[0m [MAIN] router/main.go:111 connecting to vCenter {"address": "https://vc01.labad.int/sdk"}
2021-04-04T14:30:16.589Z ESC[34mINFOESC[0m [KNATIVE] injection/injection.go:61 Starting informers...
2021-04-04T14:30:16.699Z ESC[34mINFOESC[0m [MAIN] router/main.go:149 created Knative processor {"sink": "http://default-broker-ingress.vmware-functions.svc.cluster.local"}
2021-04-04T14:30:16.699Z ESC[33mWARNESC[0m [METRICS] metrics/server.go:59 no credentials found, disabling authentication for metrics server
2021-04-04T14:30:16.700Z ESC[34mINFOESC[0m [METRICS] metrics/server.go:131 starting metrics server {"address": "http://0.0.0.0:8082/stats"}
2021-04-04T14:30:16.703Z ESC[34mINFOESC[0m [VCENTER] vcenter/vcenter.go:174 checkpointing disabled, setting begin of event stream {"beginTimestamp": "2021-04-04 14:30:16.70642 +0000 UTC"}
We now see that the Event Router came online, connected to vCenter, and successfully received an event.
If the pods appear to be up without a crash status, check to make sure the installation completed. The file /root/ran_customzation
gets created when installation completes successfully. If this file is missing, you can turn to the installation logs to find out why.
[email protected] [ ~ ]# ls -al /root/ran_customization
-rw-r--r-- 1 root root 0 Oct 1 2021 /root/ran_customization
The appliance installation log file is found in /var/log/bootstrap.log
. If enabled at install time, a debug log is available in /var/log/bootstrap-debug.log
. The logs should point you toward the source of the issue. Don’t hesitate to reach out to the team if you need help.
If you need to change the account the appliance uses to connect to vCenter, the procedure above can be used.
Ensure there is proper bi-directional network connectivity between the VMware Event Broker Appliance and vCenter Server for proper UI functionality which runs over port 443.
When the VMware Event Broker Appliance vSphere UI container starts up, it will attempt to register itself as a vSphere remote plugin with the vCenter Server. Upon a successful registration, a vCenter Plugin Extension will be created on the vCenter Server and that will direct the vSphere UI to connect to the VMware Event Broker Appliance, which is where the VMware Event Broker Appliance UI plugin will be running.
There are several areas which may prevent the VMware Event Broker Appliance vSphere UI from properly running.
Ensure that the VMware Event Broker Appliance UI container is running:
kubectl -n vmware-system get deployments/veba-ui
Here is the command output:
kubectl -n vmware-system get deployments/veba-ui
NAME READY UP-TO-DATE AVAILABLE AGE
veba-ui 1/1 1 1 6h50m
Ensure there are no errors in the logs for the VMware Event Broker Appliance UI container. Incorrect credentials or credentials without the correct permissions will prevent the registration with vCenter Server and the logs should give you some additional insights.
kubectl -n vmware-system logs deployments/veba-ui
Here is the command output:
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.0.3.RELEASE)
2021-04-04 14:30:22.690 INFO 1 --- [ main] c.v.sample.remote.SpringBootApplication : Starting SpringBootApplication on veba-ui-677b77dfcf-q9t84 with PID 1 (/app.jar started by root in /)
2021-04-04 14:30:22.695 INFO 1 --- [ main] c.v.sample.remote.SpringBootApplication : No active profile set, falling back to default profiles: default
2021-04-04 14:30:22.806 INFO 1 --- [ main] ConfigServletWebServerApplicationContext : Refreshing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicati[email protected]: startup date [Sun Apr 04 14:30:22 GMT 2021]; root of context hierarchy
2021-04-04 14:30:23.569 INFO 1 --- [ main] o.s.b.f.xml.XmlBeanDefinitionReader : Loading XML bean definitions from class path resource [spring-context.xml]
...
Ensure the VMware Event Broker Appliance UI was able to successfully register with vCenter Server and create vCenter Plugin Extension.
Open a web browser and enter the following URL and login. Replace the FQDN or IP Address of your vCenter Server
https://[VCENTER_FQDN_OR_IP]/mob/?moid=ExtensionManager&doPath=extensionList%5b%22com.vmware.veba%22%5d
If the registration exists, this most likely means that vCenter Server can not reach the VMware Event Broker Appliance.
To verify, you can SSH to the vCenter Server Appliance and attempt to retrieve the plugin.json
.
curl -L -k https://[VEBA_FQDN_OR_IP_ADDRESS]/veba-ui/plugin.json
Here is the command output:
{
"manifestVersion": "1.0.0",
"requirements": {
"plugin.api.version": "1.0.0"
},
"configuration": {
"nameKey": "plugin.name",
"icon": {
"name": "main"
}
},
"global": {
"view": {
"navigationId": "entryPoint",
"uri": "index.html#/veba-ui",
"navigationVisible": false
}
},
"definitions": {
"iconSpriteSheet": {
"uri": "assets/images/veba_otto_the_orca.png",
"definitions": {
"main": {
"x": 0,
"y": 0
}
},
"themeOverrides": {
"dark": {
"uri": "assets/images/veba_otto_the_orca.png",
"definitions": {
"main": {
"x": 0,
"y": 0
}
}
}
}
},
"i18n": {
"locales": [
"en-US",
"de-DE",
"fr-FR"
],
"definitions": {
"plugin.name": {
"en-US": "VMware Event Broker",
"de-DE": "VMware Event Broker",
"fr-FR": "VMware Event Broker"
}
}
}
}
}
Please submit bug reports and feature requests by using our GitHub Issues page or Join us on slack #vcenter-event-broker-appliance on vmwarecode.slack.com.