Navigate back to the homepage
BLOG
Try NowLogin
Back

How to Enable TLS Auth on Amazon MSK

Alex Tsokurov
March 13th, 2020 · 1 min read

Apache Kafka® is a typical Java application, which means it has perfect documentation and ecosystem if you use it with official Kafka Java SDK. However once you decide to go with one of the alternative clients, you can have issues right from the beginning.

Why should somebody use these alternative clients at all? Well, there are several reasons. Firstly, there are many people (especially among the young developers) who just don’t like Java. Secondly, the official SDK is monstrous even for the Java world. It provides extremely powerful tools to proceed the data in a fault-tolerant way, but the cost sometimes is just too high, especially if you don’t really understand how it works. Thirdly, sometimes you already have some of your logic written with another language and you don’t want to waste your time on gluing it with Java.

This article is focused on using Amazon MSK with Confluent’s Golang Kafka Client, but the same logic will work for any client based on librdkafka.

Setup secured Amazon MSK Cluster

By default Amazon MSK uses TLS to encrypt data in transit, but in order to secure your cluster with client authentication you have to create the private CA first.

Go to the ACM PCA and create new Root CA (you can skip this step if you use subordinate CA). AWS will then show you a popup with proposal to install the CA.

Now you can create the cluster. Put the installed CA to the Authentication section.

Generate client certificate

Once you create the cluster you can see its bootstrap servers by clicking View client information button. You should have something like:

1b-1.NAME.AWS.SUB.DOMAINS.amazonaws.com:9094,b-2.NAME.AWS.SUB.DOMAINS.amazonaws.com:9094

Go to the AWS Certificate Manager and request a private certificate with the CA from previous step and domain names*.NAME.AWS.SUB.DOMAINS.amazonaws.com.

Export the created certificate, you will get three ASCII-armored files: certificate body, certificate chain, and private key. Save certificate body as cert.pem and private key as cert.key.

Golang Kafka Client Config

In order to connect to the cluster your client should contain following fields:

1&kafka.ConfigMap{
2 "bootstrap.servers": "b-1.NAME.AWS.SUB.DOMAINS.amazonaws.com:9094,b-2.NAME.AWS.SUB.DOMAINS.amazonaws.com:9094",
3 "security.protocol": "ssl",
4 "ssl.certificate.location": "/path/to/cert.pem",
5 "ssl.key.location": "/path/to/cert.key",
6 "ssl.key.password": "cert password",
7}

See documentation for details.

Bonus: Dockerfile

To dockerize the client you most likely want to compile the golang client into one binary without any special shared libraries required. The Dockerfile below compiles the client with statically-linked librdkafka during the first stage and then loads the result into the clean one.

1FROM golang:1.12-alpine AS build
2
3RUN apk add --no-cache git openssh openssl-dev pkgconf gcc g++ make libc-dev bash tar
4RUN wget https://github.com/edenhill/librdkafka/archive/v1.1.0.tar.gz &&\
5 tar xf v1.1.0.tar.gz
6RUN cd librdkafka-1.1.0 && ./configure && make && make install
7
8WORKDIR /root
9
10COPY go.mod .
11COPY go.sum .
12
13RUN go mod download
14
15COPY . .
16
17RUN CGO_ENABLED=1 GOOS=linux GOARCH=amd64 go build -o entrypoint -tags static_all
18
19FROM alpine
20RUN apk add --no-cache ca-certificates
21COPY cert.* /root/
22
23COPY --from=build /root/entrypoint /root/entrypoint
24ENTRYPOINT /root/entrypoint

More articles from Asayer Blog

How to Debug ReactJS Applications

The tactics and tools ReactJS programmers use on a daily basis, not only in their development environment, but also in production.

March 4th, 2020 · 8 min read

Time-Travel Debugging with Redux and Profiler

How to synchronize your Redux state/actions with Asayer session replays and make it easy to investigate frontend issues.

February 19th, 2020 · 2 min read
© 2020 Asayer Blog
Link to $https://twitter.com/asayerioLink to $https://github.com/asayerioLink to $https://www.linkedin.com/company/18257552