Open in app

Sign in

Write

Sign in

Installation of Redpanda on EKS

IntVerse.io
10 min readJun 27, 2023

--

Redpanda is a streaming data platform for developers. It is Kafka® API-compatible, which means that you can use it with existing Kafka clients and tools. However, Redpanda is designed to be faster, more reliable, and easier to use than Kafka.

Here are some of the key features of Redpanda:

  • 10x faster: Redpanda is designed to be up to 10x faster than Kafka. This is because it uses a thread-per-core architecture and a storage design that can push high performance NVMe drives to their limits.
  • 6x lower TCO: Redpanda is also up to 6x lower in total cost of ownership than Kafka. This is because it does not require ZooKeeper, which is a separate service that Kafka uses for coordination. Redpanda also uses less memory and CPU resources than Kafka.
  • JVM-free: Redpanda is written in C++, which means that it does not require a Java Virtual Machine (JVM). This makes it easier to deploy and manage, and it can also improve performance.
  • ZooKeeper-free: Redpanda does not require ZooKeeper, which is a separate service that Kafka uses for coordination. This simplifies the deployment and management of Redpanda, and it can also improve performance.
  • Jepsen-tested: Redpanda has been tested with the Jepsen testing framework, which ensures that it is highly reliable and fault-tolerant.
  • Source available: Redpanda is open source, which means that you can inspect the code and contribute to its development.

Redpanda is a powerful and versatile streaming data platform that offers a number of advantages over Kafka. If you are looking for a high-performance, reliable, and easy-to-use streaming data platform, then Redpanda is a great option.

Here are some of the use cases for Redpanda:

  • Real-time streaming analytics: Redpanda can be used to collect and analyze streaming data in real-time. This can be used for a variety of applications, such as fraud detection, customer behavior analysis, and IoT monitoring.
  • Event streaming: Redpanda can be used to stream events between different applications. This can be used to decouple applications and improve scalability.
  • Log aggregation: Redpanda can be used to aggregate logs from different applications. This can be used for troubleshooting and auditing purposes.

If you are interested in learning more about Redpanda, we recommend checking out the following resources:

Pre-Requisites —

To install Redpanda on Amazon Elastic Kubernetes Service (EKS), there are several prerequisites you need to have in place. Here are the key prerequisites for Redpanda EKS installation:

  1. Amazon Web Services (AWS) Account: You must have an active AWS account to create and manage an EKS cluster.
  2. Kubernetes Cluster: Before installing Redpanda, you need to have a functioning EKS cluster up and running. You can create an EKS cluster using the AWS Management Console, AWS CLI, or infrastructure-as-code tools like Terraform.
  3. kubectl: Install the Kubernetes command-line tool, kubectl, on your local machine. This tool allows you to interact with your EKS cluster.
  4. AWS CLI: Install the AWS Command Line Interface (CLI) on your local machine. The AWS CLI provides a unified command line interface for interacting with various AWS services.
  5. AWS IAM Authenticator: Install the AWS IAM Authenticator on your local machine. This tool enables you to authenticate to your EKS cluster using your AWS IAM credentials.
  6. Redpanda Helm Chart: Redpanda is typically installed on EKS using Helm, a package manager for Kubernetes. Obtain the Redpanda Helm chart, which contains the necessary configurations to deploy Redpanda.
  7. Values.yaml Configuration: Customize the values.yaml file associated with the Redpanda Helm chart. This file contains various configuration options for Redpanda, such as the number of Redpanda instances, storage options, network settings, etc. Adjust these values according to your requirements.
  8. AWS VPC and Subnets: Ensure you have a VPC (Virtual Private Cloud) and subnets in your AWS account. The EKS cluster and Redpanda instances will be deployed within these networking resources.
  9. AWS IAM Roles and Policies: Create the necessary IAM roles and policies to grant the required permissions for your EKS cluster and Redpanda instances. These roles should have permission to manage EKS resources, access AWS services, and interact with storage resources like Amazon EBS (Elastic Block Store).
  10. Storage Provisioning: Decide on the storage provisioning mechanism for Redpanda. You can use Amazon EBS volumes, local SSDs (instance store), or other storage options based on your requirements.
  11. Security Groups and Network Configuration: Configure the necessary security groups and network settings to allow communication between the EKS cluster, Redpanda instances, and other required services.

Install cert-manager

$ redpanda % helm repo add redpanda https://charts.redpanda.com && helm repo add jetstack https://charts.jetstack.io && helm repo update && helm install cert-manager jetstack/cert-manager  --set installCRDs=true --namespace cert-manager  --create-namespace 
"redpanda" already exists with the same configuration, skipping
"jetstack" already exists with the same configuration, skipping
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "jetstack" chart repository
...Successfully got an update from the "redpanda" chart repository
Update Complete. ⎈Happy Helming!⎈
NAME: cert-manager
LAST DEPLOYED: Tue Mar 28 20:49:52 2023
NAMESPACE: cert-manager
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
cert-manager v1.11.0 has been deployed successfully!

In order to begin issuing certificates, you will need to set up a ClusterIssuer
or Issuer resource (for example, by creating a 'letsencrypt-staging' issuer).

More information on the different types of issuers and how to configure them
can be found in our documentation:

https://cert-manager.io/docs/configuration/

For information on how to configure cert-manager to automatically provision
Certificates for Ingress resources, take a look at the `ingress-shim`
documentation:

https://cert-manager.io/docs/usage/ingress/

Kubectl Prior to Install

kubectl get ns             
NAME                       STATUS   AGE
default                    Active   11d
intverse-rp-ns             Active   1d
kube-node-lease            Active   11d
kube-public                Active   11d
kube-system                Active   11d

Kubectl after the Redpanda install

export DOMAIN=redpanda.aws.intverse.io


helm install redpanda redpanda/redpanda -n intverse-rp-ns --create-namespace --set auth.sasl.enabled=true --set "auth.sasl.users[0].name=intverseuser" --set "auth.sasl.users[0].password=intverse123" --set external.domain=${DOMAIN} --wait


NAME: redpanda
LAST DEPLOYED: Thu Apr  6 21:54:23 2023
NAMESPACE: intverse-rp-ns
STATUS: deployed
REVISION: 1
NOTES:
Congratulations on installing redpanda!

The pods will rollout in a few seconds. To check the status:

  kubectl -n intverse-rp-ns rollout status statefulset redpanda --watch

Try some sample commands:

Create a user:

  kubectl -n intverse-rp-ns exec -ti redpanda-0 -c redpanda -- rpk acl user create myuser --new-password changeme --mechanism SCRAM-SHA-512 --brokers redpanda-0.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-1.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-2.redpanda.intverse-rp-ns.svc.cluster.local.:9093 --api-urls redpanda-0.redpanda.intverse-rp-ns.svc.cluster.local.:9644 --admin-api-tls-enabled --admin-api-tls-truststore /etc/tls/certs/default/ca.crt --tls-enabled --tls-truststore /etc/tls/certs/default/ca.crt

Give the user permissions:

    kubectl -n intverse-rp-ns exec -ti redpanda-0 -c redpanda -- rpk acl create --allow-principal 'myuser' --allow-host '*' --operation all --topic 'test-topic' --brokers redpanda-0.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-1.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-2.redpanda.intverse-rp-ns.svc.cluster.local.:9093 --tls-enabled --tls-truststore /etc/tls/certs/default/ca.crt --user <admin-user-in-secret> --password <admin-password-in-secret> --sasl-mechanism <mechanism-in-secret>

Get the api status:

  kubectl -n intverse-rp-ns exec -ti redpanda-0 -c redpanda -- rpk cluster info --brokers redpanda-0.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-1.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-2.redpanda.intverse-rp-ns.svc.cluster.local.:9093 --tls-enabled --tls-truststore /etc/tls/certs/default/ca.crt --user <admin-user-in-secret> --password <admin-password-in-secret> --sasl-mechanism <mechanism-in-secret>

Create a topic

  kubectl -n intverse-rp-ns exec -ti redpanda-0 -c redpanda -- rpk topic create test-topic --brokers redpanda-0.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-1.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-2.redpanda.intverse-rp-ns.svc.cluster.local.:9093 --tls-enabled --tls-truststore /etc/tls/certs/default/ca.crt --user <admin-user-in-secret> --password <admin-password-in-secret> --sasl-mechanism <mechanism-in-secret>

Describe the topic:

  kubectl -n intverse-rp-ns exec -ti redpanda-0 -c redpanda -- rpk topic describe test-topic --brokers redpanda-0.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-1.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-2.redpanda.intverse-rp-ns.svc.cluster.local.:9093 --tls-enabled --tls-truststore /etc/tls/certs/default/ca.crt --user <admin-user-in-secret> --password <admin-password-in-secret> --sasl-mechanism <mechanism-in-secret>

Delete the topic:

  kubectl -n intverse-rp-ns exec -ti redpanda-0 -c redpanda -- rpk topic delete test-topic --brokers redpanda-0.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-1.redpanda.intverse-rp-ns.svc.cluster.local.:9093,redpanda-2.redpanda.intverse-rp-ns.svc.cluster.local.:9093 --tls-enabled --tls-truststore /etc/tls/certs/default/ca.crt --user <admin-user-in-secret> --password <admin-password-in-secret> --sasl-mechanism <mechanism-in-secret>

After the Redpanda installation check for the namespace creation

% kubectl get ns  
NAME                       STATUS   AGE
cert-manager               Active   10m
default                    Active   11d
intverse-rp-ns             Active   10m
kube-node-lease            Active   11d
kube-public                Active   11d
kube-system                Active   11d

Verify if Redpanda pods are configured on each node

kubectl get pod -n intverse-rp-ns -o wide 
NAME                                READY   STATUS      RESTARTS   AGE   IP              NODE                            NOMINATED NODE   READINESS GATES
redpanda-0                          1/1     Running     0          52m   172.31.94.189   ip-172-31-94-228.ec2.internal   <none>           <none>
redpanda-1                          1/1     Running     0          52m   172.31.29.175   ip-172-31-27-64.ec2.internal    <none>           <none>
redpanda-2                          1/1     Running     0          52m   172.31.95.147   ip-172-31-86-144.ec2.internal   <none>           <none>
redpanda-configuration-bq6fn        0/1     Completed   0          51m   172.31.29.195   ip-172-31-27-64.ec2.internal    <none>           <none>
redpanda-console-597df5ffdc-58vk5   1/1     Running     0          52m   172.31.19.12    ip-172-31-27-64.ec2.internal    <none>           <none>

Verify if the Redpanda console is deployed

% kubectl get deployments -n intverse-rp-ns
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
redpanda-console   1/1     1            1           12m

Verify instance of Redpanda cluster as stateful set

kubectl get statefulsets -n intverse-rp-ns
NAME       READY   AGE
redpanda   3/3     13m

Verify the Internal & external endpoints created

kubectl get services -n intverse-rp-ns
NAME                TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                       AGE
redpanda            ClusterIP   None            <none>        <none>                                                        8m12s
redpanda-console    ClusterIP   10.100.102.49   <none>        8080/TCP                                                      8m12s
redpanda-external   NodePort    10.100.100.0    <none>        9644:31644/TCP,9094:31092/TCP,8083:30082/TCP,8084:30081/TCP   8m12s

PVC’s installed

kubectl get pvc -n intverse-rp-ns
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
datadir-redpanda-0   Bound    pvc-5186b90f-0d2c-4549-b246-658aff2b9ebb   20Gi       RWO            gp2            18m
datadir-redpanda-1   Bound    pvc-56a78ac4-d546-49a6-a7d0-c0b54380661c   20Gi       RWO            gp2            18m
datadir-redpanda-2   Bound    pvc-bda5e142-7b6a-4761-844c-433c0b9deb4c   20Gi       RWO            gp2            18m

Create external access to the Redpanda console

% kubectl edit svc redpanda-console -n intverse-rp-ns

Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: redpanda
    meta.helm.sh/release-namespace: intverse-rp-ns
  creationTimestamp: "2023-03-29T01:47:50Z"
  labels:
    app.kubernetes.io/instance: redpanda
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: console
    app.kubernetes.io/version: v2.1.1
    helm.sh/chart: console-0.5.0
  name: redpanda-console
  namespace: intverse-rp-ns
  resourceVersion: "4280220"
  uid: e580e596-bbc8-49d8-8c7d-cf3f516fd7e0
spec:
  clusterIP: 10.100.61.128
  clusterIPs:
  - 10.100.61.128
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app.kubernetes.io/instance: redpanda
    app.kubernetes.io/name: console
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}




service/redpanda-console edited

Services installed

kubectl get svc -A                                 
NAMESPACE        NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                       AGE
cert-manager     cert-manager           ClusterIP   10.100.203.232   <none>        9402/TCP                                                      94m
cert-manager     cert-manager-webhook   ClusterIP   10.100.167.73    <none>        443/TCP                                                       94m
default          kubernetes             ClusterIP   10.100.0.1       <none>        443/TCP                                                       11d
default          redpanda               ClusterIP   10.100.224.26    <none>        8083/TCP                                                      4d23h
intverse-rp-ns   redpanda               ClusterIP   None             <none>        <none>                                                        36m
intverse-rp-ns   redpanda-console       NodePort    10.100.61.128    <none>        8080:32001/TCP                                                36m
intverse-rp-ns   redpanda-external      NodePort    10.100.52.33     <none>        9644:31644/TCP,9094:31092/TCP,8083:30082/TCP,8084:30081/TCP   36m
kube-system      kube-dns               ClusterIP   10.100.0.10      <none>        53/UDP,53/TCP

External Access ip from nodes

kubectl get no -o wide
NAME                            STATUS   ROLES    AGE   VERSION               INTERNAL-IP     EXTERNAL-IP      OS-IMAGE         KERNEL-VERSION                  CONTAINER-RUNTIME
ip-172-31-27-64.ec2.internal    Ready    <none>   11d   v1.25.6-eks-48e63af   172.31.27.64    3.93.68.23       Amazon Linux 2   5.10.167-147.601.amzn2.x86_64   containerd://1.6.6
ip-172-31-86-144.ec2.internal   Ready    <none>   11d   v1.25.6-eks-48e63af   172.31.86.144   18.208.177.111   Amazon Linux 2   5.10.167-147.601.amzn2.x86_64   containerd://1.6.6
ip-172-31-94-228.ec2.internal   Ready    <none>   11d   v1.25.6-eks-48e63af   172.31.94.228   34.238.124.84    Amazon Linux 2   5.10.167-147.601.amzn2.x86_64   containerd://1.6.6

Access Redpanda Console

Configure DNS — we configured the DNS using GoDaddy.

Topics Creation and view messages

Schema Registry :

Connectors are not configured by default

Configure connector URL as part of the console yaml

After configuring the console with connectors

Please find the connectors configured and running state

View Consumer Groups

Hope you have learned about the installation of Redpanda on EKS.

IntVerse.io is pleased to offer event streaming solutions to help organizations effectively capture, process, and analyze real-time data for improved business outcomes. Our event streaming services enable companies to leverage data in real-time, enabling them to respond faster to market changes, customer needs, and operational issues.

Intverse
Redpanda
Event Stream Processing
AWS
Streaming

--

--

IntVerse.io

Written by IntVerse.io

32 Followers

We Solve Platform & Integration Problems in the UniVerse

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams