Redpanda — Salesforce Connector using TriggerMesh

6 min readJul 8, 2023

This blog focuses on showcasing the ease of integrating multi-cloud apps using modern event streaming technologies and enhancing event-driven processes and enabling seamless communication.

TriggerMesh event-driven application is used to stream change data capture (CDC) events from Salesforce to the RedPanda Data Platform.

Redpanda: Redpanda is an open-source, distributed streaming platform built as a Kafka-compatible alternative. It provides a high-performance, low-latency solution for real-time data streaming and processing. Redpanda is designed to be easy to deploy, scalable, and reliable, making it suitable for various use cases such as event sourcing, data pipelines, and real-time analytics. Redpanda offers a fully managed service that provides all the benefits of Redpanda Broker, Console, connectors, and schema registry, etc. without the need to manage your own infrastructure. If you need more control over your environment, then BYOC Clusters is a good option.
TriggerMesh: TriggerMesh is a cloud-native integration platform that allows developers to build event-driven applications. It is built on top of Kubernetes and leverages its capabilities to create a scalable and reliable platform for event processing. TriggerMesh provides a number of features that make it a powerful tool for building event-driven applications, including Event Sources, Event Targets, Triggers, and Targets, etc. TriggerMesh supports various cloud and on-premise applications connectivity as Source and Target Endpoints. TriggerMesh has great potential to be used similarly to Kafka Connect.
Salesforce: Salesforce is a cloud-based customer relationship management (CRM) platform. It provides a suite of tools and services for sales, marketing, customer service, and analytics. Salesforce allows organizations to manage customer data, track interactions, automate sales processes, and gain insights into customer behavior. It offers a wide range of features and modules that can be customized and extended to meet specific business needs.

TriggerMesh is deployed in GCP and RedPanda is deployed in AWS and authenticated by the SCRAM-SHA-512 security mechanism and the communication between Salesforce and TriggerMesh is through a secure SSL certificate.

Implementation

Salesforce setup:

After login to Salesforce Cloud, go to settings and click on Setup

Steps to create the Salesforce objects

Salesforce contains many default objects like Contacts, Cases etc… These can be used very well for demo purposes or otherwise to create a custom object.

Go to Setup > Object Manager

Click on Create > Custom object

Fill up all the mandatory fields to create the object

Once the object is created, go inside the object and create the fields by clicking on “New”

Salesforce Connected APP:

Salesforce Connected App is a feature of the Salesforce platform that allows you to integrate and extend the functionality of Salesforce with external applications and services. It provides a secure way to authenticate and authorize external applications to access Salesforce data and perform actions on behalf of users.

Go to Setup > Home > PLATFORM TOOLS > Apps > App Manager

Click on New Connected App

Fill in the basic and API information

Save it.

It takes about 10mins to deploy the application and get ready to use it.

Change Data Capture (CDC) Events:

Change Data Capture (CDC) is a feature provided by Salesforce that allows you to track and capture changes made to Salesforce records in real time. It provides a way to monitor and react to data changes, enabling you to build integrations, automate processes, and keep external systems synchronized with Salesforce data.

Let's enable the CDC events on the required objects:

Go to Setup > Home > PLATFORM TOOLS > Integrations > Change Data Capture

select the necessary available entities and move to the right

Save it

This completes the Salesforce setup

TriggerMesh

Installation:

For installation of the TriggerMesh, please refer to the page: https://docs.triggermesh.io/1.26/installation/kubernetes-helm/

Application:

This use case involves the creation of a TriggerMesh source object and a secret.

Event Source:

apiVersion: sources.triggermesh.io/v1alpha1
kind: SalesforceSource
metadata:
  labels:
    triggermesh.io/context: intvtmbroker
    app.kubernetes.io/part-of: triggermesh
    prometheus.io/scrape: "true"
  name: salesforcesource
  namespace: usecases
spec:
  subscription:
    channel: /data/ChangeEvents
    replayID: -2
  auth:
    clientID: 3MVG9Kip4IKAZQEVy75f7CNoGDQ7NL9g.zOliV3POutcNJdY4Ku1wN_cVI.stP2TktU976x_AXkt6uwMVUdle
    server: <https://login.salesforce.com>
    user: mr.kolas@gmail.com
    certKey:
      valueFromSecret:
        name: salesforce
        key: certKey
  sink:
    ref:
      apiVersion: eventing.triggermesh.io/v1alpha1
      kind: RedisBroker
      name: intvtmbroker

Source Secret:

apiVersion: v1
kind: Secret
stringData:
  certKey: |
    -----BEGIN PRIVATE KEY-----
    MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCbt0TQpmg2WPAT
    b758VLZir1yK79KIc3Pe+MxQSzvqP7tjLRFWjtgOmxZ/8dafDWj2gTL/ee+YGVBw
    tj//Qc0ERIh2AbFrh+rMD22k3vVyz5Xih5RC3FEnUcp2CTog01nsYvvEKK2XFB6t
    7mAzlSMT9FCU9hMLNynWo/IXnZzpgCZIapLfQEKe2UeVDemA4aPPaDZmmGz4hla6
    -----END PRIVATE KEY-----

metadata:
  name: salesforce
  namespace: usecases
type: Opaque

Event Target:

apiVersion: targets.triggermesh.io/v1alpha1
kind: KafkaTarget
metadata:
  labels:
    triggermesh.io/context: intvtmbroker
    app.kubernetes.io/part-of: triggermesh
    prometheus.io/scrape: "true"
  name: redpandasfawstarget
  namespace: usecases
spec:
  bootstrapServers:
    - redpanda-0.redpanda.aws.intverse.io:31092
    - redpanda-1.redpanda.aws.intverse.io:31092
    - redpanda-2.redpanda.aws.intverse.io:31092
  topic: salesforcetopic
  auth:
    username: intverseuser
    password: 
      value: intverse123
    saslEnable: true
    tlsEnable: true
    securityMechanism: SCRAM-SHA-512
    tls:
      ca:
        valueFromSecret:
          name: rdpsfawscacert
          key: ca-cert
      skipVerify: true

Target Secret:

apiVersion: v1
kind: Secret
data:
  ca-cert: TUlJQm1EQ0NBVCtn==
metadata:
  name: rdpsfawscacert
  namespace: usecases
type: Opaque

Trigger:

apiVersion: eventing.triggermesh.io/v1alpha1
kind: Trigger
metadata:
  labels:
    triggermesh.io/context: intvtmbroker
    app.kubernetes.io/part-of: triggermesh
    prometheus.io/scrape: "true"
  name: intvtmbroker-sf-rdp-trigger
  namespace: usecases
spec:
  broker:
    group: eventing.triggermesh.io
    kind: RedisBroker
    name: intvtmbroker
  target:
    ref:
      apiVersion: targets.triggermesh.io/v1alpha1
      kind: KafkaTarget
      name: redpandasfawstarget
  filters:
    - exact:
        type: com.salesforce.stream.message

Deployment

The ArgoCD application serves as a GitOps tool and is responsible for automatically generating TriggerMesh kube objects in the cluster as soon as code synchronization occurs with the Git repository.

ArgoCD deployment app:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: salesforce-rdp-argo-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: git@github.com:IntVerse/intverse-trigger-mesh.git
    targetRevision: dev
    path: usecases/salesforce-rdp/
  destination:
    server: <https://kubernetes.default.svc>
    namespace: usecases
  syncPolicy:
    syncOptions:
    - CreateNamespace=true
    automated:
      selfHeal: true
      prune: true

Test

Objective: Insert or update or delete a record on the Salesforce object and the change event is streamed to the Redpanda topic via TriggerMesh.

Save it.

Redpanda console:

Updated Employee records are streamed to the Redpanda topic and can be visible in the redpanda console.

At IntVerse.io we work closely with clients to understand their business requirements, designing and configuring the desired solution, integrating it into the existing infrastructure, and providing support for the implementation process. We offer consulting and advisory services to help organizations leverage the full potential of Redpanda and TriggerMesh within their specific use cases.