Introduction to VMware Event Router

The VMware Event Router is used to connect to various VMware event providers (i.e. “sources”) and forward these events to different event processors (i.e. “sinks”). This project is currently used by the VMware Event Broker Appliance as the core logic to forward CloudEvents, e.g. from vSphere, to configurable event processors (see below).

Supported Event Providers:

• VMware vCenter Server
• VMware Horizon
• Generic CloudEvents Webhook
• vCenter Simulator vcsim (deprecated, see note below)

Supported Event Processors:

• Knative
• OpenFaaS
• AWS EventBridge

The VMware Event Router uses the CloudEvents standard to normalize events from the supported event providers. See below for an example.

Event Provider Delivery Guarantees:

• At-least-once event delivery
  • with the vCenter event provider option checkpoint: true
• At-most-once event delivery
  • with the vCenter event provider option checkpoint: false
  • with the Webhook event provider
  • with the Horizon event provider
  • with the vcsim event provider

Note: All implemented event processors use built-in retry mechanisms so your function might still be involved multiple times depending on its response code. However, if an event provider crashes before sending an event to the configured processor or when the processor returns an error, the event is not retried and discarded.

Current limitations:

• Only one event provider and one event processor can be configured at a time (see note below)
• At-least-once event delivery semantics are not guaranteed if the event router crashes within seconds right after startup and having received n events but before creating the first valid checkpoint (current checkpoint interval is 5s)
• If an event cannot be successfully delivered (retried) by an event processor it is logged and discarded, i.e. there is currently no support for dead letter queues (see note below)
• Retries in the OpenFaaS event processor are only supported when running in synchronous mode, i.e. async: false (see this OpenFaaS issue)

Note: It is possible though to run multiple instances of the event router with different configurations to address multi-vCenter scenarios. This decision was made for scalability and resource/tenancy isolation purposes.

Note: Event Processors, like Knative, support Dead Letter Queues when using Broker mode.

Table of Contents

• Introduction to VMware Event Router
• Configuration
  • Overview: Configuration File Structure (YAML)
  • JSON Schema Validation
  • API Version, Kind and Metadata
  • The eventProvider section
    • Provider Type vcenter
    • Provider Type horizon
    • Provider Type webhook
    • Provider Type vcsim
  • The eventProcessor section
    • Processor Type knative
      • Destination
    • Processor Type openfaas
    • Processor Type aws_event_bridge
  • The auth section
    • Type basic_auth
    • Type aws_access_key
    • Type aws_iam_role
    • Type active_directory
  • The metricsProvider section
    • Provider Type default
• Deployment
  • Assisted Deployment
    • Helm Deployment
      • Option 1: Configuration with Knative
      • Option 2: Configuration with OpenFaaS
      • Deploy the VMware Event Router Helm Chart
      • Creating/Updating the Chart
    • Manual Deployment
      • Create Knative ClusterRoleBinding (skip if not using Knative)
      • Create the VMware Event Router Deployment
  • CLI Flags

Configuration

The VMware Event Router can be run standalone (statically linked binary) or deployed as a Docker container, e.g. in a Kubernetes environment. See deployment for further instructions. The configuration of event providers and processors and other internal components (such as metrics) is done via a YAML file passed in via the -config command line flag.

 _    ____  ___                            ______                 __     ____              __
| |  / /  |/  /      ______ _________     / ____/   _____  ____  / /_   / __ \____  __  __/ /____  _____
| | / / /|_/ / | /| / / __  / ___/ _ \   / __/ | | / / _ \/ __ \/ __/  / /_/ / __ \/ / / / __/ _ \/ ___/
| |/ / /  / /| |/ |/ / /_/ / /  /  __/  / /___ | |/ /  __/ / / / /_   / _, _/ /_/ / /_/ / /_/  __/ /
|___/_/  /_/ |__/|__/\__,_/_/   \___/  /_____/ |___/\___/_/ /_/\__/  /_/ |_|\____/\__,_/\__/\___/_/


Usage of ./vmware-event-router:

  -config string
        path to configuration file (default "/etc/vmware-event-router/config")
  -log-json
        print JSON-formatted logs
  -log-level string
        set log level (debug,info,warn,error) (default "info")

commit: <git_commit_sha>
version: <release_tag>

The following sections describe the layout of the configuration file (YAML) and specific options for the supported event providers, processors and metrics endpoint. Configuration examples are provided here.

Note: Currently only one event provider and one event processor can be configured at a time, e.g. one vCenter Server instance streaming events to OpenFaaS or AWS EventBridge. It is possible to run multiple instances of the event router with different configurations to address multi-provider/processor scenarios.

Overview: Configuration File Structure (YAML)

The following file, using vcenter as the event provider and knative as the processor shows an example of the configuration file syntax:

apiVersion: event-router.vmware.com/v1alpha1
kind: RouterConfig
metadata:
  name: router-config-openfaas
  labels:
    key: value
eventProvider:
  type: vcenter
  name: veba-demo-vc-01
  vcenter:
    address: https://my-vcenter01.domain.local/sdk
    insecureSSL: false
    checkpoint: true
    auth:
      type: basic_auth
      basicAuth:
        username: [email protected]
        password: ReplaceMe
eventProcessor:
  type: knative
  name: veba-demo-knative
  knative:
    encoding: binary
    insecureSSL: false
    destination:
      ref:
        apiVersion: eventing.knative.dev/v1
        kind: Broker
        name: rabbit
        namespace: default
metricsProvider:
  type: default
  name: veba-demo-metrics
  default:
    bindAddress: "0.0.0.0:8082"

JSON Schema Validation

In order to simplify the configuration and validation of the YAML configuration file a JSON schema file is provided. Many editors/IDEs offer support for registering a schema file, e.g. Jetbrains and VS Code.

Note: The schema file can be downloaded and provided via a local file location or (recommended) via a direct URL, e.g. Github raw URL pointing to the aforementioned JSON schema file.

API Version, Kind and Metadata

The following table lists allowed and required fields with their respective type values and examples for these fields.

The eventProvider section

The following table lists allowed and required fields with their respective type values and examples for these fields.

Provider Type vcenter

VMware vCenter Server is advanced server management software that provides a centralized platform for controlling your VMware vSphere environments, allowing you to automate and deliver a virtual infrastructure across the hybrid cloud with confidence. Since VMware vCenter Server event types are environment specific (vSphere version, extensions), a list of events for vCenter as an event source can be generated as described in this blog post.

The following table lists allowed and required fields for connecting to a vCenter Server and the respective type values and examples for these fields.

Provider Type horizon

VMware Horizon is a platform for delivering virtual desktops and apps efficiently and securely across hybrid cloud for the best end-user digital workspace experience.

This provider supports all audit events (model AuditEventSummary) exposed through the Horizon REST API starting with version 2106.

The following table lists allowed and required fields for connecting to the Horizon REST API and the respective type values and examples for these fields.

Provider Type webhook

The webhook event provider listens for incoming CloudEvents (binary or structured mode) on a configurable HTTP server in the VMware Event Router. The HTTP method used to send the CloudEvent must be POST.

The following table lists allowed and required fields for setting up a webhook server.

Note: When the VMware Event Router log level is DEBUG incoming webhook requests (method, path, headers, remote address) will be logged.

Provider Type vcsim

⚠️ This provider is deprecated and will be removed in future versions. The vcenter provider will work correctly against a vcsim instance.

The following table lists allowed and required fields for connecting to the govmomi vCenter Simulator vcsim and the respective type values and examples for these fields.

Note: This event provider has some limitations and currently does not behave like a “real” vCenter Server event stream, e.g. see issue #2134. This provider is for prototyping/testing purposes only.

The eventProcessor section

The following table lists allowed and required fields with their respective type values and examples for these fields.

Processor Type knative

Knative is a Kubernetes-based platform to deploy and manage modern serverless workloads. Knative has two core building blocks, that is Serving (Knative Service) and Eventing (Broker, Channel, etc.).

The VMware Event Router can be configured to directly send events to any addressable Knative resource (“reference”), e.g. a Knative Broker or Service. Broker is the recommended deployment model for the VMware Event Router. Please see the Knative documentation on Eventing for details around brokers, triggers, event filtering, etc.

Alternatively, the router can send events to a URI, e.g. an external HTTP endpoint accepting CloudEvents.

The following table lists allowed and optional fields for using Knative as an event processor.

Note: When sending events to a Knative Broker, the Knative broker will always send binary encoded cloud events to the Knative sinks, e.g. triggered Service.

Destination

Knative abstracts event receivers (“sinks”) in a very flexible way via addressable Destinations. The VMware Event Router technically supports all Knative Destinations. However, Knative Broker or Knative Services are likely the ones you will be using.

Note: We have not done any testing for Knative Channels, since request response style interactions are out of scope for the VMware Event Router.

The following table lists the available destination types for the destination section in the router configuration when targeting a URI.

The following table lists the available destination types for the destination section in the router configuration when targeting a URI.

Note: Only one of uri or ref MUST be specified. The list of all available URI options is documented here.

Note: A vanilla Kubernetes Service can also be specified when using Ref as the destination type. Use v1 in the apiVersion section.

Processor Type openfaas

OpenFaaS functions can subscribe to the event stream via function "topic" annotations in the function stack configuration (see OpenFaaS documentation for details on authoring functions), e.g.:

annotations:
  topic: "VmPoweredOnEvent,VmPoweredOffEvent"

Note: One or more event categories can be specified, delimited via ",". A list of event names (categories) and how to retrieve them can be found here. A simple “echo” function useful for testing is provided here.

The following table lists allowed and optional fields for using OpenFaaS as an event processor.

Processor Type aws_event_bridge

Amazon EventBridge is a serverless event bus that makes it easy to connect applications together using data from your own applications, integrated Software-as-a-Service (SaaS) applications, and AWS services. In order to reduce bandwidth and costs (number of events ingested, see pricing), VMware Event Router only forwards events configured in the associated rule of an event bus. Rules in AWS EventBridge use pattern matching (docs). Upon start, VMware Event Router contacts EventBridge (using the given IAM role) to parse the configured rule ARN (see configuration option below).

The VMware Event Router uses the pattern match library which supports a subset of the EventBridge pattern rules. You may only use these supported patterns in your specified EventBridge rule. Refer to this page for the currently supported patterns in Quamina.

Note: EventBridge wraps each VMware Event Router event (CloudEvent) into an EventBridge message envelop. The detail field contains the JSON representation of the full CloudEvent as produced by the VMware Event Router.

The following examples show supported and useful patterns.

Example: Forward all CloudEvents containing one of the specified subjects:

{
  "detail": {
    "subject": ["VmPoweredOnEvent", "VmPoweredOffEvent", "DrsVmPoweredOnEvent"]
  }
}

Example: Forward all CloudEvents containing a subject with the prefix Vm:

{
  "detail": {
    "subject": [{
      "shellstyle": "Vm*"
    }]
  }
}

Example: Forward all CloudEvents containing virtual machines with the prefix Linux:

{
  "detail": {
    "data": {
      "Vm": {
        "Name": [{
          "shellstyle": "Linux*"
        }]
      }
    }
  }
}

Note: A list of event names (categories) and how to retrieve them can be found here.

The following table lists allowed and optional fields for using AWS EventBridge as an event processor.

The auth section

The following table lists allowed and required fields with their respective type values and examples for these fields. Since the various processors and providers use different authentication mechanisms (or none at all) this section describes the various options.

Type basic_auth

Supported providers/processors:

• vcenter (required: true)
• vcsim (required: true)
• openfaas (required: false, i.e. optional)
• default metrics server (see below) (required: false, i.e. optional)

Type aws_access_key

Use an AWS IAM role with the provided access key ID and secret access key for authentication.

Supported providers/processors:

• aws_event_bridge

Note: Please follow the EventBridge IAM user guide before deploying the event router. Further information can also be found in the authentication section.

In addition to the recommendation in the AWS EventBridge user guide you might want to lock down the IAM role for the VMware Event Router and scope it to these permissions (“Action”):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "events:PutEvents",
        "events:ListRules",
        "events:TestEventPattern"
      ],
      "Resource": "*"
    }
  ]
}

Type aws_iam_role

Use an AWS IAM role configured from the shared credentials file.

Supported providers/processors:

• aws_event_bridge

Note: Please follow the EventBridge IAM user guide before deploying the event router. Further information can also be found in the authentication section.

In addition to the recommendation in the AWS EventBridge user guide you might want to lock down the IAM role for the VMware Event Router and scope it to these permissions (“Action”):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "events:PutEvents",
        "events:ListRules",
        "events:TestEventPattern"
      ],
      "Resource": "*"
    }
  ]
}

Type active_directory

Supported providers/processors:

• horizon (required: true)

Note: UPN authentication, e.g. [email protected] as username, is not supported.

The metricsProvider section

The VMware Event Router currently only exposes a default (“internal” or “embedded”) metrics endpoint. In the future, support for more providers is planned, e.g. Wavefront, Prometheus, etc.

Provider Type default

The VMware Event Router exposes metrics in JSON format on a configurable HTTP listener, e.g. http://<bindAddress>/stats. The following table lists allowed and optional fields for configuring the default metrics server.

Deployment

VMware Event Router can be deployed and run as standalone binary (see below). However, it is designed (and recommended) to be run in a Kubernetes cluster for increased availability and ease of scaling out.

Note: Docker images are available here.

Assisted Deployment

For your convenience we provide a Helm Chart which can be used to easily install the VMware Event Router into an existing Knative or OpenFaaS (“faas-netes”) environment.

⚠️ The OpenFaaS deployment method is unmaintained in the VEBA project and will be deprecated in a future release. The recommended deployment method is using the Knative backend.

Helm Deployment

The Helm files are located in the chart directory. The values.yaml file contains the allowed parameters and parameter descriptions which map to the VMware Event Router configuration file.

Option 1: Configuration with Knative

If you don’t have a working Knative installation, follow the steps described in the official documentation to deploy Knative Serving and Eventing.

Now create a Helm override.yaml file with your environment specific settings, e.g.:

eventrouter:
  config:
    logLevel: debug
  vcenter:
    address: https://vcenter.corp.local
    username: [email protected]
    password: replaceMe
    insecure: true # if required ignore TLS certs
  eventProcessor: knative
  knative:
    destination: # follows Knative convention for ref/uri
      ref:
        apiVersion: eventing.knative.dev/v1
        kind: Broker
        name: default
        namespace: default

Note: Please ensure the correct formatting/indentation which follows the Helm values.yaml file.

Option 2: Configuration with OpenFaaS

The following steps can be used to quickly install OpenFaaS as a requirement for the Helm installation instructions of the VMware Event Router below. Skip this part if you already have an OpenFaaS environment set up.

kubectl create ns openfaas && kubectl create ns openfaas-fn

helm repo add openfaas https://openfaas.github.io/faas-netes && \
    helm repo update \
    && helm upgrade openfaas --install openfaas/openfaas \
        --namespace openfaas \
        --set functionNamespace=openfaas-fn \
        --set generateBasicAuth=true

OF_PASS=$(echo $(kubectl -n openfaas get secret basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode))

Now create a Helm override.yaml file with your environment specific settings, e.g.:

eventrouter:
  config:
    logLevel: debug
  vcenter:
    address: https://vcenter.corp.local
    username: [email protected]
    password: replaceMe
    insecure: true # if required ignore TLS certs
  eventProcessor: openfaas
  openfaas:
    address: http://gateway.openfaas:8080
    basicAuth: true
    username: admin
    password: ${OF_PASS} # variable from previous section

Note: Please ensure the correct formatting/indentation which follows the Helm values.yaml file.

Deploy the VMware Event Router Helm Chart

Add the VMware Event Router Helm release to your Helm repository:

# adds the veba chartrepo to the list of local registries with the repo name "veba"
$ helm repo add vmware-veba https://projects.registry.vmware.com/chartrepo/veba

To ensure new releases are pulled/updated and reflected locally update the repo index:

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "vmware-veba" chart repository
Update Complete. ⎈ Happy Helming!⎈

The chart should now show up in the search:

$ helm search repo event-router
NAME                      CHART VERSION   APP VERSION     DESCRIPTION
vmware-veba/event-router  v0.7.0          v0.7.0          The VMware Event Router is used to connect to v...
[snip]

Note: To list/install development releases add the --devel flag to the Helm CLI.

Now install the chart using a Helm release name of your choice, e.g. veba, using the configuration override file created above. The following command will create a release from the chart in the namespace vmware, which will be automatically created if it does not exist:

$ helm install -n vmware --create-namespace veba vmware-veba/event-router -f override.yaml
NAME: veba
LAST DEPLOYED: Mon Jun  28 16:27:27 2020
NAMESPACE: vmware
STATUS: deployed
REVISION: 1
TEST SUITE: None

Check the logs of the VMware Event Router to validate it is operating correctly:

$ kubectl -n vmware logs deploy/router -f

If you run into issues, the logs should give you a hint, e.g.:

• configuration file not found -> file naming issue
• connection to vCenter/OpenFaaS cannot be established -> check values, credentials (if any) in the configuration file
• deployment/pod will not even come up -> check for resource issues, docker pull issues and other potential causes using the standard kubectl troubleshooting ways

To uninstall the release run:

$ helm -n vmware uninstall veba

Creating/Updating the Chart

Before running the following commands make the appropriate changes to the chart, e.g. bumping up version and/or appVersion in Chart.yaml.

$ cd chart
$ helm package -d releases .

Then upload the created .tgz file inside releases/ to your Helm chart repo.

Manual Deployment

Create a namespace where the VMware Event Router will be deployed to:

$ kubectl create namespace vmware

Use one of the configuration files provided here to configure the router with one VMware vCenter Server eventProvider and one OpenFaaS or AWS EventBridge eventProcessor. Change the values to match your environment. The following example will use the OpenFaaS config sample.

Note: Before continuing, make sure your environment is up and running, including Kubernetes and the configured event processor.

After you made your changes to the configuration file, save it as "event-router-config.yaml in your current Git working directory.

Note: If you have changed the port of the metrics server in the configuration file (default: 8080) make sure to also change that value in the YAML manifest (under the Kubernetes service entry).

Now, from your current Git working directory create a Kubernetes secret with the configuration file as input:

$ kubectl -n vmware create secret generic event-router-config --from-file=event-router-config.yaml

Note: You might want to delete the (local) configuration file to not leave behind sensitive information on your local machine.

If you have configured knative as your event processor you must also create a ClusterRoleBinding for the VMware Event Router so it can lookup Knative destinations.

Create Knative ClusterRoleBinding (skip if not using Knative)

The following commands creates the ClusterRoleBinding assuming the VMware Event Router will be deployed into the vmware namespace using the predefined service account (deployment manifest). This requires a properly configured Knative environment with Serving and Eventing CRDs installed.

# only for Knative-based deployments
$ kubectl create clusterrolebinding veba-addressable-resolver --clusterrole=knative-serving-aggregated-addressable-resolver --serviceaccount=vmware:vmware-event-router

$ kubectl describe clusterrole addressable-resolver
Name:         knative-serving-aggregated-addressable-resolver
Labels:       serving.knative.dev/release=v0.23.0
Annotations:  <none>
PolicyRule:
  Resources                                      Non-Resource URLs  Resource Names  Verbs
  ---------                                      -----------------  --------------  -----
  services                                       []                 []              [get list watch]
  brokers.eventing.knative.dev/status            []                 []              [get list watch]
  brokers.eventing.knative.dev                   []                 []              [get list watch]
  parallels.flows.knative.dev/status             []                 []              [get list watch]
  parallels.flows.knative.dev                    []                 []              [get list watch]
  sequences.flows.knative.dev/status             []                 []              [get list watch]
  sequences.flows.knative.dev                    []                 []              [get list watch]
  channels.messaging.knative.dev/status          []                 []              [get list watch]
  channels.messaging.knative.dev                 []                 []              [get list watch]
  inmemorychannels.messaging.knative.dev/status  []                 []              [get list watch]
  inmemorychannels.messaging.knative.dev         []                 []              [get list watch]
  parallels.messaging.knative.dev/status         []                 []              [get list watch]
  parallels.messaging.knative.dev                []                 []              [get list watch]
  sequences.messaging.knative.dev/status         []                 []              [get list watch]
  sequences.messaging.knative.dev                []                 []              [get list watch]
  routes.serving.knative.dev/status              []                 []              [get list watch]
  routes.serving.knative.dev                     []                 []              [get list watch]
  services.serving.knative.dev/status            []                 []              [get list watch]
  services.serving.knative.dev                   []                 []              [get list watch]
  channels.messaging.knative.dev/finalizers      []                 []              [update]

Create the VMware Event Router Deployment

Now we can deploy the VMware Event Router.

Download the latest deployment manifest (release.yaml) file from the Github release page. Then save the file under the same name (to follow along with the commands).

Example Download with curl:

curl -L -O https://github.com/vmware-samples/vcenter-event-broker-appliance/releases/latest/download/release.yaml

Deploy the VMware Event Router:

$ kubectl -n vmware create -f release.yaml

Check the logs of the VMware Event Router to validate it started correctly:

$ kubectl -n vmware logs deploy/vmware-event-router -f

If you run into issues, the logs should give you a hint, e.g.:

• configuration file not found -> file naming issue
• connection to vCenter/OpenFaaS cannot be established -> check values, credentials (if any) in the configuration file
• deployment/pod will not even come up -> check for resource issues, docker pull issues and other potential causes using the standard kubectl troubleshooting ways

To delete the deployment and secret simply delete the namespace we created earlier:

$ kubectl delete namespace vmware

CLI Flags

By default the VMware Event Router binary will look for a YAML configuration file named /etc/vmware-event-router/config. The default log level is info and human readable colored console logs will be printed. This behavior can be overridden via log-json to generate JSON logs and log-level to change the log level. Stack traces are only generated in level error or higher.

$ ./vmware-event-router -h

 _    ____  ___                            ______                 __     ____              __
| |  / /  |/  /      ______ _________     / ____/   _____  ____  / /_   / __ \____  __  __/ /____  _____
| | / / /|_/ / | /| / / __  / ___/ _ \   / __/ | | / / _ \/ __ \/ __/  / /_/ / __ \/ / / / __/ _ \/ ___/
| |/ / /  / /| |/ |/ / /_/ / /  /  __/  / /___ | |/ /  __/ / / / /_   / _, _/ /_/ / /_/ / /_/  __/ /
|___/_/  /_/ |__/|__/\__,_/_/   \___/  /_____/ |___/\___/_/ /_/\__/  /_/ |_|\____/\__,_/\__/\___/_/

Usage of dist/vmware-event-router:

  -config string
        path to configuration file (default "/etc/vmware-event-router/config")
  -log-json
        print JSON-formatted logs
  -log-level string
        set log level (debug,info,warn,error) (default "info")

Get Started

Explore the capabilities that the VMware Event Router enables

• Install the Appliance with Knative to extend your SDDC with our community-sourced functions

• Learn more about the Events in vCenter and how to find the right event for your use case

• Learn more about Functions in this overview here.