Application Modernization using IBM MQ Connector with Redpanda

11 min readNov 21, 2023

In today’s fast-paced business landscape, organizations are constantly seeking ways to modernize their data infrastructure to keep up with the demands of real-time data processing and analytics. A key component of this modernization effort is integrating various data sources and systems.

One approach that has gained popularity is leveraging the power of Kafka, IBM MQ, and Redpanda in tandem to create a robust and scalable data pipeline.

In this blog, we will explore the use of the Kafka IBM MQ Connector with Redpanda and how this combination can serve as a catalyst for modernization efforts in data architecture.

The Need for Modernization

Before we dive into the technical aspects of this integration, it’s crucial to understand why modernization is essential. Many organizations still rely on legacy systems and monolithic architectures, which can hinder their ability to extract valuable insights from data in real time. In the age of big data, the need for modernization is evident in several ways:

Real-time Data Processing: Businesses need to process data as it’s generated, not hours or days later. Real-time data processing allows for faster decision-making and immediate responses to changing conditions.
Scalability: As data volumes grow, the infrastructure must be able to scale horizontally to handle increased workloads without sacrificing performance.
Data Integration: Data comes from various sources, such as databases, sensors, applications, and more. Modern architectures need to support seamless integration of data from these diverse sources.
Data Reliability: Data integrity and reliability are paramount. Data should be processed without loss and with guaranteed delivery.
Cost Efficiency: Modernizing data infrastructure should not only improve performance but also optimize costs, making the best use of resources.

Kafka and Redpanda: A Dynamic Duo

Apache Kafka is a distributed streaming platform that has become the de facto standard for building real-time data pipelines and streaming applications. It excels at ingesting, storing, and processing high-throughput, fault-tolerant data streams.

Redpanda, on the other hand, is a modern streaming platform built for performance, security, and simplicity. It is designed as a drop-in replacement for Apache Kafka but with a focus on extreme performance and operational ease. Redpanda is known for its compatibility with Kafka, making it a seamless choice for Kafka users looking to modernize their data infrastructure.

Kafka IBM MQ Connector

IBM MQ is a messaging system that has been trusted for years for reliable and secure data transfer within enterprises. By connecting Kafka and IBM MQ, organizations can bridge the gap between traditional and modern data systems. This connector enables data to flow from IBM MQ queues and topics into Kafka topics and vice versa.

The Kafka IBM MQ Connector facilitates the following:

Data Migration: Organizations can migrate data from IBM MQ to Redpanda for real-time analytics and streaming applications. This migration is crucial for modernizing data systems while preserving data integrity.
Bi-directional Data Flow: The connector allows for data to flow both ways. You can ingest data from IBM MQ into Redpanda, process it, and then send the results back to IBM MQ for further processing.
Reliability: Data is transmitted with reliability guarantees, ensuring that no messages are lost during the transfer process.
Scalability: As Kafka and Redpanda are horizontally scalable, the connector inherits this scalability, making it suitable for handling large volumes of data.

The Benefits of Integration

By integrating IBM MQ, and Redpanda, organizations can experience a host of benefits:

Real-time Data Processing: This integration enables organizations to process data as it arrives, leading to quicker insights and faster decision-making.
Scalability: The combined solution is built to handle massive workloads without compromising performance.
Data Integration: With the connector, it’s easy to pull data from IBM MQ and integrate it seamlessly into your Redpanda-based data pipeline.
Data Reliability: Leveraging the reliability and fault-tolerance of both Redpanda and IBM MQ ensures data integrity and guaranteed delivery.
Cost Efficiency: Modernizing your data infrastructure with these open-source technologies can lead to cost savings compared to proprietary solutions.

Reference Use case: Connecting Redpanda to IBM MQ with Kafka Connect

There is a Corporate company — ABC Ltd . that mainly operates in Europe as London Head Quarters. They are using legacy App A as a payment platform to settle all the transactions as of today. As part of their cost optimization program, they are looking to choose another App B as their new payment platform which does the same thing in a cost-effective & modern way.

As of today, the transaction data movement in ABC Ltd to App A is happening through IBM MQ where the Source applications will place the message to the Queue for App A to consume and do further processing. In this case, App B is using Redpanda as an event streaming platform to take any updates from their multiple producers and settle the transactions.

For ABC Ltd to do the migration process seamlessly from App A to App B, The IBM MQ Source Connector present on the Redpanada Data Streaming platform facilitated the integration of Redpanda with IBM MQ (Message Queue).

It allows to consumption of messages from IBM MQ queues and publishes them to Redpanda topics, enabling the flow of data between MQ and Redpanda. Once the data is there in the corresponding topic, the transactions are settled by the payment engine of App B by subscribing to this topic.

Also, App B sends the acknowledgment for each transaction by placing a message in another Queue. In this case, the IBM MQ Sink Connector is used to take the message from the topic of App B and place it in the Corresponding MQ Queue. If there are any transaction failures those are moved to the error Queue, which can be replayed at a later point of time after identifying the root cause of failures.

Redpanda & IBM MQ Connector Setup

we are going to do a deep dive into how we can configure the IBM MQ Connector with Redpanda.

Prerequisites

Set up and run a Redpanda cluster and create topics for Kafka Connect usage
Configure and run a Kafka Connect cluster for Redpanda and IBM MQ integration
Java 17 to run the MQ Connector application
Create an IBM MQ Cloud Account and a Queue manager, Queue to connect with Kafka Connect
Make sure you have the necessary permissions to connect and read messages from the MQ queues created in the cloud.

Setting up Redpanda

On a Linux environment, run the following commands to install Redpanda as a system service:

On Fedora/Red Hat systems:

Run the setup script to download and install the repo

curl -1sLf 'https://packages.vectorized.io/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.rpm.sh' | sudo -E bash && \

Use yum to install redpanda

sudo yum install redpanda -y

On Debian/Ubuntu systems:

Run the setup script to download and install the repo

curl -1sLf 'https://packages.vectorized.io/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.deb.sh' | sudo -E bash && \

Use apt to install redpanda

sudo apt install redpanda -y

Start the Redpanda cluster with the following command:

sudo systemctl start redpanda

You can verify if Redpanda is up and running:

sudo systemctl status redpanda

The output should look like the following:

redpanda.service - Redpanda, the fastest queue in the West. Loaded: loaded (/usr/lib/systemd/system/redpanda.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2022-02-10 15:56:39 +03; 2 days ago ...output omitted...

In this tutorial, you’ll create a single-node Redpanda cluster that is not suitable for a production environment.

To enable the Payment processor application and the Kafka Connect cluster to work properly, you must define the Kafka topic payment-initiation. You can use Redpanda Keeper (rpk), Redpanda’s command-line interface (CLI) to create topics on the Redpanda cluster.

You can access the CLI by running the rpk command. (View the rpk documentation for more available commands.)

Run the following command to create a topic for the Istanbul archive:

rpk topic create payment-initiation

Verify that you have the topics created:

rpk cluster info

This should return the following output:

Notice that you do not specify a partition count or replication factor for the topics. For a production environment, you would need to create topics with suitable configurations for the cluster structure and the designed architecture.

Setting up IBM MQ in the IBM Cloud :

Setting up IBM MQ in a cloud environment involves multiple steps, and the specific process may vary depending on the cloud provider you choose. Below is a general guide for setting up IBM MQ on IBM Cloud.

Step 1: Sign Up for an IBM Cloud Account

Go to the IBM Cloud website:IBM Cloud .
Click on the “Sign Up” button to create a new account.
Provide the required information, including your email address, password, and other details.
Follow the instructions to verify your email and complete the registration process.

Step 2: Log In to IBM Cloud

Once your account is created, log in to the IBM Cloud Console:IBM Cloud .

Step 3: Navigate to the IBM MQ Service

In the IBM Cloud Console, click on the “Catalog” link in the top navigation bar.
In the left sidebar, select the “Integration” category.
Find and click on the “Message Queues” or “MQ” service. This might be listed under the “Enterprise Integration” section.

Step 4: Create an IBM MQ Instance

Click on the “Create” button to create a new IBM MQ instance.
Provide a unique name for your instance and select the desired region and resource group.
Choose the pricing plan that fits your requirements (e.g., Lite for free usage, or Pay-as-you-go).
Configure any additional settings, such as tags or service credentials.
Click on the “Create” or “Deploy” button to provision your IBM MQ instance.

Step 5: Access the IBM MQ Console

Once the provisioning is complete, navigate to the “Resource List” in the IBM Cloud Console.
Find your newly created IBM MQ instance and click on it to access details.
Look for a link or button to access the IBM MQ Console. This is where you can configure and manage your MQ resources.

Step 6: Configure IBM MQ

In the IBM MQ Console, you may find options to create queues, topics, and other MQ objects.
Configure your queues, topics, and any other settings required for your messaging needs.

Step 7: Connect Redpanda to IBM MQ

Obtain connection details such as the host, port, and credentials from the IBM MQ Console.
Use these connection details in your IBM MQ Source and Sink connectors to connect to the IBM MQ service.

Setting up Kafka Connect

Kafka Connect is an integration tool released with the Apache Kafka project. It provides reliable data streaming between Apache Kafka and external systems. It is both scalable and flexible, and you can use it to integrate with any system, including databases, search indexes, and cloud storage providers.

Redpanda is fully compatible with the Kafka API. This means that Redpanda gives you seamless integration with the Kafka ecosystem so your existing investments work right out of the box.

Kafka Connect uses source and sink connectors for integration. Source connectors stream data from an external system to Kafka. Sink connectors stream data from Kafka to an external system.

Configuring the connectors

Setting up the connector plugins in a Kafka Connect cluster for achieving integration with external systems is not enough. The cluster needs connectors configured to execute the integration, so next you’ll configure the Source and Sink connectors for IBM MQ.

Note that We running the Kafka Connect cluster in standalone mode. Kafka Connect also has a distributed mode that accepts connector configurations via its REST API. The standalone mode fits the proposed architecture better because you are working on a lightweight integration scenario.

The connector can be run in a Connect worker in either standalone (single process) or distributed mode. It’s a good idea to start in standalone mode.

Sample connectors :

Configuration for the Source Connect:

IBM MQ Source Connector is a component used in Kafka Connect, a framework for connecting Kafka with external systems. This connector facilitates the integration of IBM MQ (Message Queue) with Kafka, allowing to ingest messages from IBM MQ into Redpanda topics. This integration helps to establish the integration between ABC Ltd and the Redpanda Cluster of Payment Institution B.

Sample Json used for testing the IBM MQsource connector :

{
"connector.class": "com.ibm.eventstreams.connect.mqsource.MQSourceConnector",
"mq.channel.name": "CLOUD.APP.SVRCONN",
"mq.connection.name.list": "rpd-2b5c.qm.eu-de.mq.appdomain.cloud(32093)",
"mq.password": "eFLzNL43l9dHR8-KmUQOFBuT2bq1gEJsAwzw1HsBtN9o",
"mq.queue": "DEV.QUEUE.1",
"mq.queue.manager": "RPD",
"mq.record.builder": "com.ibm.eventstreams.connect.mqsource.builders.DefaultRecordBuilder",
"mq.ssl.cipher.suite": "*TLS12ORHIGHER",
"mq.user.authentication.mqcsp": "true",
"mq.user.name": "devrpmqpoc",
"name": "MQSourceConnect",
"topic": "IBMQSourceConnect"
}

Verify that the connector works and data is moved from IBM MQ to Redpanda Cluster Topic for the App B

Ensure that there is a message in IBM MQ that was sent by App A for processing.

Push to Queue:

Verify that the Connector pushes the generated message to the designated topic (IBMMQSourceConnect) of App B. Look for log messages or confirmation that the message has been successfully sent to the Topic.
Verify in the Redpanda Console for messages from the IBM MQ Source Connector

App B can consume these messages and process them in a modern way.

Configuration for the Sink connect:

The IBM MQ Sink Connector is designed for Kafka Connect and is used to forward messages from Redpanda topics to IBM MQ queues or topics. This connector facilitates the integration between Redpanda and IBM MQ, allowing to move data from topics to IBM MQ in a seamless and efficient manner.

In our use case, it is s used to establish the communication between the Redpanda cluster and the IBM MQ. The acknowledgment/error messages are routed using this channel.

Sample Json used for testing the IBM MQ Sink connector :

{
"connector.class": "com.ibm.eventstreams.connect.mqsink.MQSinkConnector",
"mq.channel.name": "CLOUD.APP.SVRCONN",
"mq.connection.name.list": "rpd-2b5c.qm.eu-de.mq.appdomain.cloud(32093)",
"mq.message.builder": "com.ibm.eventstreams.connect.mqsink.builders.DefaultMessageBuilder",
"mq.password": "eFLzNL43l9dHR8-KmUQOFBuT2bq1gEJsAwzw1HsBtN9o",
"mq.queue": "RPD.MQ.TEST",
"mq.queue.manager": "RPD",
"mq.ssl.cipher.suite": "*TLS12ORHIGHER",
"mq.user.name": "devrpmqpoc",
"name": "MqSinkConnect",
"tasks.max": "1",
"topics": "IBMMQSinkConnect"
}

Verify that the connector works and ack messages are moved from the Redpanda Cluster(Sink) of App B to the Source Applications :

Verify that the connector pushes the message from the topic to the designated Queue. Look for log messages or confirmation that the message has been successfully sent to the queue.

Conclusion
In this Blog, we created an IBM MQ Cloud account and a Queue Manger and a Queue. Also, create a Redpanda cluster and a topic for the archive to use in your connector configurations. We also created and configured a Kafka Connect cluster to use Redpanda and configured a connector to stream data between two entities.
By using Kafka Connect, we can integrate Redpanda with IBM MQ, which helps reduce the cost of operations and the operational burden while providing scaling with all the benefits of the cloud.
This blog can serve as a valuable resource for organizations looking to enhance their messaging capabilities in a modern, scalable, and efficient manner.
By this, developers can benefit from a step-by-step approach to integrating IBM MQ Connector with Redpanda, making it easier for them to implement a reliable and efficient messaging solution.

IntVerse.io specializes in application & data modernization initiatives for enterprises that are looking to modernize their systems. If you need any help with a new journey, please reach out to hello@intverse.io to discuss more about your unique use cases.