Redpanda — Salesforce Connector using TriggerMesh
This blog focuses on showcasing the ease of integrating multi-cloud apps using modern event streaming technologies and enhancing event-driven processes and enabling seamless communication.
TriggerMesh event-driven application is used to stream change data capture (CDC) events from Salesforce to the RedPanda Data Platform.
- Redpanda: Redpanda is an open-source, distributed streaming platform built as a Kafka-compatible alternative. It provides a high-performance, low-latency solution for real-time data streaming and processing. Redpanda is designed to be easy to deploy, scalable, and reliable, making it suitable for various use cases such as event sourcing, data pipelines, and real-time analytics. Redpanda offers a fully managed service that provides all the benefits of Redpanda Broker, Console, connectors, and schema registry, etc. without the need to manage your own infrastructure. If you need more control over your environment, then BYOC Clusters is a good option.
- TriggerMesh: TriggerMesh is a cloud-native integration platform that allows developers to build event-driven applications. It is built on top of Kubernetes and leverages its capabilities to create a scalable and reliable platform for event processing. TriggerMesh provides a number of features that make it a powerful tool for building event-driven applications, including Event Sources, Event Targets, Triggers, and Targets, etc. TriggerMesh supports various cloud and on-premise applications connectivity as Source and Target Endpoints. TriggerMesh has great potential to be used similarly to Kafka Connect.
- Salesforce: Salesforce is a cloud-based customer relationship management (CRM) platform. It provides a suite of tools and services for sales, marketing, customer service, and analytics. Salesforce allows organizations to manage customer data, track interactions, automate sales processes, and gain insights into customer behavior. It offers a wide range of features and modules that can be customized and extended to meet specific business needs.
TriggerMesh is deployed in GCP and RedPanda is deployed in AWS and authenticated by the SCRAM-SHA-512 security mechanism and the communication between Salesforce and TriggerMesh is through a secure SSL certificate.
Implementation
Salesforce setup:
After login to Salesforce Cloud, go to settings and click on Setup
Steps to create the Salesforce objects
Salesforce contains many default objects like Contacts, Cases etc… These can be used very well for demo purposes or otherwise to create a custom object.
Go to Setup > Object Manager
Click on Create > Custom object
Fill up all the mandatory fields to create the object
Once the object is created, go inside the object and create the fields by clicking on “New”
Salesforce Connected APP:
Salesforce Connected App is a feature of the Salesforce platform that allows you to integrate and extend the functionality of Salesforce with external applications and services. It provides a secure way to authenticate and authorize external applications to access Salesforce data and perform actions on behalf of users.
Go to Setup > Home > PLATFORM TOOLS > Apps > App Manager
Click on New Connected App
Fill in the basic and API information
Save it.
It takes about 10mins to deploy the application and get ready to use it.
Change Data Capture (CDC) Events:
Change Data Capture (CDC) is a feature provided by Salesforce that allows you to track and capture changes made to Salesforce records in real time. It provides a way to monitor and react to data changes, enabling you to build integrations, automate processes, and keep external systems synchronized with Salesforce data.
Let's enable the CDC events on the required objects:
Go to Setup > Home > PLATFORM TOOLS > Integrations > Change Data Capture
select the necessary available entities and move to the right
Save it
This completes the Salesforce setup
TriggerMesh
Installation:
For installation of the TriggerMesh, please refer to the page: https://docs.triggermesh.io/1.26/installation/kubernetes-helm/
Application:
This use case involves the creation of a TriggerMesh source object and a secret.
Event Source:
apiVersion: sources.triggermesh.io/v1alpha1
kind: SalesforceSource
metadata:
labels:
triggermesh.io/context: intvtmbroker
app.kubernetes.io/part-of: triggermesh
prometheus.io/scrape: "true"
name: salesforcesource
namespace: usecases
spec:
subscription:
channel: /data/ChangeEvents
replayID: -2
auth:
clientID: 3MVG9Kip4IKAZQEVy75f7CNoGDQ7NL9g.zOliV3POutcNJdY4Ku1wN_cVI.stP2TktU976x_AXkt6uwMVUdle
server: <https://login.salesforce.com>
user: mr.kolas@gmail.com
certKey:
valueFromSecret:
name: salesforce
key: certKey
sink:
ref:
apiVersion: eventing.triggermesh.io/v1alpha1
kind: RedisBroker
name: intvtmbroker
Source Secret:
apiVersion: v1
kind: Secret
stringData:
certKey: |
-----BEGIN PRIVATE KEY-----
MIIEvgIBADANBgkqhkiG9w0BAQEFAASCBKgwggSkAgEAAoIBAQCbt0TQpmg2WPAT
b758VLZir1yK79KIc3Pe+MxQSzvqP7tjLRFWjtgOmxZ/8dafDWj2gTL/ee+YGVBw
tj//Qc0ERIh2AbFrh+rMD22k3vVyz5Xih5RC3FEnUcp2CTog01nsYvvEKK2XFB6t
7mAzlSMT9FCU9hMLNynWo/IXnZzpgCZIapLfQEKe2UeVDemA4aPPaDZmmGz4hla6
-----END PRIVATE KEY-----
metadata:
name: salesforce
namespace: usecases
type: Opaque
Event Target:
apiVersion: targets.triggermesh.io/v1alpha1
kind: KafkaTarget
metadata:
labels:
triggermesh.io/context: intvtmbroker
app.kubernetes.io/part-of: triggermesh
prometheus.io/scrape: "true"
name: redpandasfawstarget
namespace: usecases
spec:
bootstrapServers:
- redpanda-0.redpanda.aws.intverse.io:31092
- redpanda-1.redpanda.aws.intverse.io:31092
- redpanda-2.redpanda.aws.intverse.io:31092
topic: salesforcetopic
auth:
username: intverseuser
password:
value: intverse123
saslEnable: true
tlsEnable: true
securityMechanism: SCRAM-SHA-512
tls:
ca:
valueFromSecret:
name: rdpsfawscacert
key: ca-cert
skipVerify: true
Target Secret:
apiVersion: v1
kind: Secret
data:
ca-cert: TUlJQm1EQ0NBVCtn==
metadata:
name: rdpsfawscacert
namespace: usecases
type: Opaque
Trigger:
apiVersion: eventing.triggermesh.io/v1alpha1
kind: Trigger
metadata:
labels:
triggermesh.io/context: intvtmbroker
app.kubernetes.io/part-of: triggermesh
prometheus.io/scrape: "true"
name: intvtmbroker-sf-rdp-trigger
namespace: usecases
spec:
broker:
group: eventing.triggermesh.io
kind: RedisBroker
name: intvtmbroker
target:
ref:
apiVersion: targets.triggermesh.io/v1alpha1
kind: KafkaTarget
name: redpandasfawstarget
filters:
- exact:
type: com.salesforce.stream.message
Deployment
The ArgoCD application serves as a GitOps tool and is responsible for automatically generating TriggerMesh kube objects in the cluster as soon as code synchronization occurs with the Git repository.
ArgoCD deployment app:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: salesforce-rdp-argo-app
namespace: argocd
spec:
project: default
source:
repoURL: git@github.com:IntVerse/intverse-trigger-mesh.git
targetRevision: dev
path: usecases/salesforce-rdp/
destination:
server: <https://kubernetes.default.svc>
namespace: usecases
syncPolicy:
syncOptions:
- CreateNamespace=true
automated:
selfHeal: true
prune: true
Test
Objective: Insert or update or delete a record on the Salesforce object and the change event is streamed to the Redpanda topic via TriggerMesh.
Save it.
Redpanda console:
Updated Employee records are streamed to the Redpanda topic and can be visible in the redpanda console.
At IntVerse.io we work closely with clients to understand their business requirements, designing and configuring the desired solution, integrating it into the existing infrastructure, and providing support for the implementation process. We offer consulting and advisory services to help organizations leverage the full potential of Redpanda and TriggerMesh within their specific use cases.