Creating Redpanda Topics Using RPK Commands and Node.js (Typescript)

In the world of modern data processing, Redpanda has emerged as a powerful streaming platform. One crucial aspect of Redpanda is topic creation, which forms the foundation of data organization and communication.

In this blog post, we’ll explore how to leverage RPK commands in conjunction with Node.js to seamlessly create Redpanda topics, this pattern will be helpful to design Self Service portals for your organization, through which you can create, manage topics and other Redpanda Components.

Whether you’re a developer looking to harness the capabilities of Redpanda or simply curious about this process, you’re in the right place.

But, What are Topics and “rpk”?

Redpanda topics serve as channels through which data flows, enables efficient data organization and distribution. Think of them as virtual pipelines that facilitate seamless data communication. While the rpk commands let us configure, manage, and tune Redpanda clusters. They also let us manage topics, groups, and access control lists (ACLs).

Prerequisites

Before we start, make sure the necessary software packages are installed on your system.

1. NodeJs, how to?
2. rpk (Redpanda Keeper), how to?
3. Docker, how to?
4. or a Text Editor that you like, (for this blog I am using vs-code)

Using Node.js to Create Redpanda Topics

To get started, let’s ensure we have the necessary environment set up for our exploration.

Step 1: Clone the Git Repository

Begin by cloning the Git repository that provides the necessary environment for creating Redpanda topics using Node.js. This repository contains a pre-configured setup that will streamline the process for us. Open a terminal and enter the following command:

git clone https://github.com/iamstorm-sys/nodejs-ts.git

Once the repository is cloned, navigate to the project directory using the cd command. To make sure everything’s working fine, run the following in a terminal.

npm run start:dev

We’ll see a similar output,

start the node server

To check server’s health, try hitting http://localhost:7654/health.

Now as we set up our Node environment, we also need a cluster so that we can perform topic creation commands on it. For this, we just need a single node cluster and let’s make it.

Step 2: Create a Single Node Cluster

Pull the Redpanda image

docker pull redpandadata/redpanda

You’ll see a similar output,

Pulling the image from the docker hub

After that, we need to create a container, To create one use the following command.

docker run --name my-redpanda-cluster -d -p 9092:9092 redpandadata/redpanda

You’ll see a similar output,

creating a cluster with the name my-redpanda-cluster

To make sure the cluster is created successfully, use the following command.

rpk cluster info

You’ll see a similar output,

your cluster is ready now

Now that we have our cluster and node environment ready, Let’s create an API, which upon triggering will create a topic for us, with the desired name.

Step 3: Creating a Topic Programmatically

To get started with this step, Open the “src/index.ts” file and import the following items from the child_process module.

Node.js comes with a child process module, which helps to run other processes in the current environment i.e. it can be used to communicate with other processes.

import { spawn, ChildProcessWithoutNullStreams } from 'child_process';

Create a route with “/topic” endpoint & “POST” method. To set the desired topic name, de-structure the “topicName” variable from req.body property in the handler, which we will pass when we call the API.

router.post("/topic", (req, res) =>{
//...
  const {topicName} = req.body;
//...
})

Then add the following variables

const cmd = "rpk"
const subject = "topic"
const action = "create"
const options = [subject, action, topicName];

Now you might be wondering, why we need these variables, well there are two reasons behind it,

1. The rpk topic creation syntax.
2. The NodeJs child_process module.

To create topic using rpk command using terminal, we use the following syntax: “rpk topic create <topicName>”.

Now use a variable “processTopic” to store the reference of the child process returned by the “spawn” method.

# cmd will be "rpk",
# and options will be our args, which is a string array

const processTopic: ChildProcessWithoutNullStreams = spawn(cmd, options);

spawn method has the following syntax: spawn(<command>, <args>). The command variable will hold the main command, in our case it would be “rpk” and the args variable will accept the options as a string array.

The child_process.spawn() method spawns a new process using the givencommand, with command-line arguments in args. If omitted, args defaults to an empty array.

In order to listen to processTopic (child process), we need to tack on a listener for the “data” event, and in its callback, we can store the event data in a string variable.

processTopic.stdout.on('data', (data) => {
   responseData += data.toString(); // Append the data to the responseData
});

Now we have the data returned by the child process, in order to send it to the user, we need to wait until the child process stops. For that, we can tack on a listener on “processTopic” for the “close” event, and using a callback we can make res.send call, to send the responseData to the user.

processTopic.on('close', () => {
   res.send(responseData); // Send the response after all data is captured
});

The route will look something like this:

// topic route

app.post('/topic', (req: Request, res: Response) => {
    const { topicName } = req.body;
    const cmd = "rpk"
    const subject = "topic"
    const action = "create"
    const options = [subject, action, topicName];
    const processTopic: ChildProcessWithoutNullStreams = spawn(cmd, options);
    let responseData = "";
    try {
        processTopic.stdout.on('data', (data) => {
            responseData += data.toString(); // Append the data to the responseData
        });
        processTopic.on('close', () => {
            res.send(responseData); // Send the response after all data is captured
        });
    } catch (_) {
        res.send("internal server error");
    }
})

Now it’s time to make the API call, You can use any API-Testing tool, such as Postman, ThunderClient (a vs-code extension), or curl-like utility. I am using curl, to make the API call.

curl -X POST -H "Content-Type: application/json" -d '{"topicName": "chat-room"}' http://localhost:7654/topic

You’ll see a similar output,

Post API call for creating a topic with topic name (chat-room)

To make sure, the topic successfully gets created, you can check cluster info, using the following command

rpk cluster info

You’ll see a similar output,

Confirmation of Topic Creation

As you can see under the “TOPICS” section, there is an entry with NAME as “chat-room”, which confirms our topic creation.

Conclusion

In wrapping up, you’re now set to create Redpanda topics using Node.js and RPK. Begin orchestrating efficient data communication. Dive in, innovate, and embrace the possibilities that lie ahead.

Clear the Clutter

#After you done with implementation
#to close the running cluster, use the following command
docker stop my-redpanda-cluster

#to close the node server, just type "ctrl + c", in the terminal,
#where you are running your node server.

Call to Action

1. The child_process module does provide other methods as well, using which we can accomplish the same in a similar way, try exploring other options as well.
2. In this post, we have learned only about the topic creation, but there are other rpk commands as well, try checking them out.
3. If you want such interesting posts, you can track me on my LinkedIn, I work for https://Intverse.io and we solve problems related to Redpanda Event Streaming and other Integration Platforms.(https://www.linkedin.com/in/raman-kaushik01/).

Wishing you a smooth experience without any hiccups. Farewell for now!