Installation of Redpanda on EKS

IntVerse.io
10 min readJun 27, 2023

Redpanda is a streaming data platform for developers. It is Kafka® API-compatible, which means that you can use it with existing Kafka clients and tools. However, Redpanda is designed to be faster, more reliable, and easier to use than Kafka.

Here are some of the key features of Redpanda:

  • 10x faster: Redpanda is designed to be up to 10x faster than Kafka. This is because it uses a thread-per-core architecture and a storage design that can push high performance NVMe drives to their limits.
  • 6x lower TCO: Redpanda is also up to 6x lower in total cost of ownership than Kafka. This is because it does not require ZooKeeper, which is a separate service that Kafka uses for coordination. Redpanda also uses less memory and CPU resources than Kafka.
  • JVM-free: Redpanda is written in C++, which means that it does not require a Java Virtual Machine (JVM). This makes it easier to deploy and manage, and it can also improve performance.
  • ZooKeeper-free: Redpanda does not require ZooKeeper, which is a separate service that Kafka uses for coordination. This simplifies the deployment and management of Redpanda, and it can also improve performance.
  • Jepsen-tested: Redpanda has been tested with the Jepsen testing framework, which ensures that it is highly reliable and fault-tolerant.
  • Source available: Redpanda is open source, which means that you can inspect the code and contribute to its development.

Redpanda is a powerful and versatile streaming data platform that offers a number of advantages over Kafka. If you are looking for a high-performance, reliable, and easy-to-use streaming data platform, then Redpanda is a great option.

Here are some of the use cases for Redpanda:

  • Real-time streaming analytics: Redpanda can be used to collect and analyze streaming data in real-time. This can be used for a variety of applications, such as fraud detection, customer behavior analysis, and IoT monitoring.
  • Event streaming: Redpanda can be used to stream events between different applications. This can be used to decouple applications and improve scalability.
  • Log aggregation: Redpanda can be used to aggregate logs from different applications. This can be used for troubleshooting and auditing purposes.

If you are interested in learning more about Redpanda, we recommend checking out the following resources:

Pre-Requisites —

To install Redpanda on Amazon Elastic Kubernetes Service (EKS), there are several prerequisites you need to have in place. Here are the key prerequisites for Redpanda EKS installation:

  1. Amazon Web Services (AWS) Account: You must have an active AWS account to create and manage an EKS cluster.
  2. Kubernetes Cluster: Before installing Redpanda, you need to have a functioning EKS cluster up and running. You can create an EKS cluster using the AWS Management Console, AWS CLI, or infrastructure-as-code tools like Terraform.
  3. kubectl: Install the Kubernetes command-line tool, kubectl, on your local machine. This tool allows you to interact with your EKS cluster.
  4. AWS CLI: Install the AWS Command Line Interface (CLI) on your local machine. The AWS CLI provides a unified command line interface for interacting with various AWS services.
  5. AWS IAM Authenticator: Install the AWS IAM Authenticator on your local machine. This tool enables you to authenticate to your EKS cluster using your AWS IAM credentials.
  6. Redpanda Helm Chart: Redpanda is typically installed on EKS using Helm, a package manager for Kubernetes. Obtain the Redpanda Helm chart, which contains the necessary configurations to deploy Redpanda.
  7. Values.yaml Configuration: Customize the values.yaml file associated with the Redpanda Helm chart. This file contains various configuration options for Redpanda, such as the number of Redpanda instances, storage options, network settings, etc. Adjust these values according to your requirements.
  8. AWS VPC and Subnets: Ensure you have a VPC (Virtual Private Cloud) and subnets in your AWS account. The EKS cluster and Redpanda instances will be deployed within these networking resources.
  9. AWS IAM Roles and Policies: Create the necessary IAM roles and policies to grant the required permissions for your EKS cluster and Redpanda instances. These roles should have permission to manage EKS resources, access AWS services, and interact with storage resources like Amazon EBS (Elastic Block Store).
  10. Storage Provisioning: Decide on the storage provisioning mechanism for Redpanda. You can use Amazon EBS volumes, local SSDs (instance store), or other storage options based on your requirements.
  11. Security Groups and Network Configuration: Configure the necessary security groups and network settings to allow communication between the EKS cluster, Redpanda instances, and other required services.

Install cert-manager

$ redpanda % helm repo add redpanda https://charts.redpanda.com && helm repo add jetstack https://charts.jetstack.io && helm repo update && helm install cert-manager jetstack/cert-manager  --set installCRDs=true --namespace cert-manager  --create-namespace 
"redpanda" already exists with the same configuration, skipping
"jetstack" already exists with the same configuration, skipping
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "jetstack" chart repository
...Successfully got an update from the "redpanda" chart repository
Update Complete. ⎈Happy Helming!⎈
NAME: cert-manager
LAST DEPLOYED: Tue Mar 28 20:49:52 2023
NAMESPACE: cert-manager
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
cert-manager v1.11.0 has been deployed successfully!

In order to begin issuing certificates, you will need to set up a ClusterIssuer
or Issuer resource (for example, by creating a 'letsencrypt-staging' issuer).

More information on the different types of issuers and how to configure them
can be found in our documentation:

https://cert-manager.io/docs/configuration/

For information on how to configure cert-manager to automatically provision
Certificates for Ingress resources, take a look at the `ingress-shim`
documentation:

https://cert-manager.io/docs/usage/ingress/

Kubectl Prior to Install

kubectl get ns             
NAME STATUS AGE
default Active 11d
intverse-rp-ns Active 1d
kube-node-lease Active 11d
kube-public Active 11d
kube-system Active 11d

Kubectl after the Redpanda install

export DOMAIN=redpanda.aws.intverse.io


helm install redpanda redpanda/redpanda -n intverse-rp-ns --create-namespace --set auth.sasl.enabled=true --set "auth.sasl.users[0].name=intverseuser" --set "auth.sasl.users[0].password=intverse123" --set external.domain=${DOMAIN} --wait


NAME: redpanda
LAST DEPLOYED: Thu Apr 6 21:54:23 2023
NAMESPACE: intverse-rp-ns
STATUS: deployed
REVISION: 1
NOTES:
Congratulations on installing redpanda!

The pods will rollout in a few seconds. To check the status:

kubectl -n intverse-rp-ns rollout status statefulset redpanda --watch

Try some sample commands:

Create a user:

kubectl -n intverse-rp-ns exec -ti redpanda-0 -c redpanda -- rpk acl user create myuser --new-password changeme --mechanism SCRAM-SHA-512 --brokers redpanda-0.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-1.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-2.redpanda.intverse-rp-ns.svc.cluster.local.:9093 --api-urls redpanda-0.redpanda.intverse-rp-ns.svc.cluster.local.:9644 --admin-api-tls-enabled --admin-api-tls-truststore /etc/tls/certs/default/ca.crt --tls-enabled --tls-truststore /etc/tls/certs/default/ca.crt

Give the user permissions:

kubectl -n intverse-rp-ns exec -ti redpanda-0 -c redpanda -- rpk acl create --allow-principal 'myuser' --allow-host '*' --operation all --topic 'test-topic' --brokers redpanda-0.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-1.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-2.redpanda.intverse-rp-ns.svc.cluster.local.:9093 --tls-enabled --tls-truststore /etc/tls/certs/default/ca.crt --user <admin-user-in-secret> --password <admin-password-in-secret> --sasl-mechanism <mechanism-in-secret>

Get the api status:

kubectl -n intverse-rp-ns exec -ti redpanda-0 -c redpanda -- rpk cluster info --brokers redpanda-0.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-1.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-2.redpanda.intverse-rp-ns.svc.cluster.local.:9093 --tls-enabled --tls-truststore /etc/tls/certs/default/ca.crt --user <admin-user-in-secret> --password <admin-password-in-secret> --sasl-mechanism <mechanism-in-secret>

Create a topic

kubectl -n intverse-rp-ns exec -ti redpanda-0 -c redpanda -- rpk topic create test-topic --brokers redpanda-0.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-1.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-2.redpanda.intverse-rp-ns.svc.cluster.local.:9093 --tls-enabled --tls-truststore /etc/tls/certs/default/ca.crt --user <admin-user-in-secret> --password <admin-password-in-secret> --sasl-mechanism <mechanism-in-secret>

Describe the topic:

kubectl -n intverse-rp-ns exec -ti redpanda-0 -c redpanda -- rpk topic describe test-topic --brokers redpanda-0.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-1.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-2.redpanda.intverse-rp-ns.svc.cluster.local.:9093 --tls-enabled --tls-truststore /etc/tls/certs/default/ca.crt --user <admin-user-in-secret> --password <admin-password-in-secret> --sasl-mechanism <mechanism-in-secret>

Delete the topic:

kubectl -n intverse-rp-ns exec -ti redpanda-0 -c redpanda -- rpk topic delete test-topic --brokers redpanda-0.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-1.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-2.redpanda.intverse-rp-ns.svc.cluster.local.:9093 --tls-enabled --tls-truststore /etc/tls/certs/default/ca.crt --user <admin-user-in-secret> --password <admin-password-in-secret> --sasl-mechanism <mechanism-in-secret>

After the Redpanda installation check for the namespace creation

% kubectl get ns  
NAME STATUS AGE
cert-manager Active 10m
default Active 11d
intverse-rp-ns Active 10m
kube-node-lease Active 11d
kube-public Active 11d
kube-system Active 11d

Verify if Redpanda pods are configured on each node

kubectl get pod -n intverse-rp-ns -o wide 
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
redpanda-0 1/1 Running 0 52m 172.31.94.189 ip-172-31-94-228.ec2.internal <none> <none>
redpanda-1 1/1 Running 0 52m 172.31.29.175 ip-172-31-27-64.ec2.internal <none> <none>
redpanda-2 1/1 Running 0 52m 172.31.95.147 ip-172-31-86-144.ec2.internal <none> <none>
redpanda-configuration-bq6fn 0/1 Completed 0 51m 172.31.29.195 ip-172-31-27-64.ec2.internal <none> <none>
redpanda-console-597df5ffdc-58vk5 1/1 Running 0 52m 172.31.19.12 ip-172-31-27-64.ec2.internal <none> <none>

Verify if the Redpanda console is deployed

% kubectl get deployments -n intverse-rp-ns
NAME READY UP-TO-DATE AVAILABLE AGE
redpanda-console 1/1 1 1 12m

Verify instance of Redpanda cluster as stateful set

kubectl get statefulsets -n intverse-rp-ns
NAME READY AGE
redpanda 3/3 13m

Verify the Internal & external endpoints created

kubectl get services -n intverse-rp-ns
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
redpanda ClusterIP None <none> <none> 8m12s
redpanda-console ClusterIP 10.100.102.49 <none> 8080/TCP 8m12s
redpanda-external NodePort 10.100.100.0 <none> 9644:31644/TCP,9094:31092/TCP,8083:30082/TCP,8084:30081/TCP 8m12s

PVC’s installed

kubectl get pvc -n intverse-rp-ns
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
datadir-redpanda-0 Bound pvc-5186b90f-0d2c-4549-b246-658aff2b9ebb 20Gi RWO gp2 18m
datadir-redpanda-1 Bound pvc-56a78ac4-d546-49a6-a7d0-c0b54380661c 20Gi RWO gp2 18m
datadir-redpanda-2 Bound pvc-bda5e142-7b6a-4761-844c-433c0b9deb4c 20Gi RWO gp2 18m

Create external access to the Redpanda console

% kubectl edit svc redpanda-console -n intverse-rp-ns

Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
annotations:
meta.helm.sh/release-name: redpanda
meta.helm.sh/release-namespace: intverse-rp-ns
creationTimestamp: "2023-03-29T01:47:50Z"
labels:
app.kubernetes.io/instance: redpanda
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: console
app.kubernetes.io/version: v2.1.1
helm.sh/chart: console-0.5.0
name: redpanda-console
namespace: intverse-rp-ns
resourceVersion: "4280220"
uid: e580e596-bbc8-49d8-8c7d-cf3f516fd7e0
spec:
clusterIP: 10.100.61.128
clusterIPs:
- 10.100.61.128
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: http
port: 8080
protocol: TCP
targetPort: 8080
selector:
app.kubernetes.io/instance: redpanda
app.kubernetes.io/name: console
sessionAffinity: None
type: NodePort
status:
loadBalancer: {}




service/redpanda-console edited

Services installed

kubectl get svc -A                                 
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cert-manager cert-manager ClusterIP 10.100.203.232 <none> 9402/TCP 94m
cert-manager cert-manager-webhook ClusterIP 10.100.167.73 <none> 443/TCP 94m
default kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 11d
default redpanda ClusterIP 10.100.224.26 <none> 8083/TCP 4d23h
intverse-rp-ns redpanda ClusterIP None <none> <none> 36m
intverse-rp-ns redpanda-console NodePort 10.100.61.128 <none> 8080:32001/TCP 36m
intverse-rp-ns redpanda-external NodePort 10.100.52.33 <none> 9644:31644/TCP,9094:31092/TCP,8083:30082/TCP,8084:30081/TCP 36m
kube-system kube-dns ClusterIP 10.100.0.10 <none> 53/UDP,53/TCP

External Access ip from nodes

kubectl get no -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-172-31-27-64.ec2.internal Ready <none> 11d v1.25.6-eks-48e63af 172.31.27.64 3.93.68.23 Amazon Linux 2 5.10.167-147.601.amzn2.x86_64 containerd://1.6.6
ip-172-31-86-144.ec2.internal Ready <none> 11d v1.25.6-eks-48e63af 172.31.86.144 18.208.177.111 Amazon Linux 2 5.10.167-147.601.amzn2.x86_64 containerd://1.6.6
ip-172-31-94-228.ec2.internal Ready <none> 11d v1.25.6-eks-48e63af 172.31.94.228 34.238.124.84 Amazon Linux 2 5.10.167-147.601.amzn2.x86_64 containerd://1.6.6

Access Redpanda Console

Configure DNS — we configured the DNS using GoDaddy.

Topics Creation and view messages

Schema Registry :

Connectors are not configured by default

Configure connector URL as part of the console yaml

After configuring the console with connectors

Please find the connectors configured and running state

View Consumer Groups

Hope you have learned about the installation of Redpanda on EKS.

IntVerse.io is pleased to offer event streaming solutions to help organizations effectively capture, process, and analyze real-time data for improved business outcomes. Our event streaming services enable companies to leverage data in real-time, enabling them to respond faster to market changes, customer needs, and operational issues.

--

--

IntVerse.io

We Solve Platform & Integration Problems in the UniVerse