Developer steerage on find out how to do native testing with Amazon MSK Serverless

Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a totally managed service that makes it simple to construct and run Kafka clusters on Amazon Net Providers (AWS). When working with Amazon MSK, builders are serious about accessing the service domestically. This permits builders to check their software with a Kafka cluster that has the identical configuration as manufacturing and offers an similar infrastructure to the precise atmosphere without having to run Kafka domestically.

An Amazon MSK Serverless personal DNS endpoint is just accessible from Amazon Digital Personal Cloud (Amazon VPC) connections which have been configured to attach. It isn’t straight resolvable out of your native improvement atmosphere. One possibility is to make use of AWS Direct Join or AWS VPN to have the ability to Hook up with Amazon MSK Serverless out of your on-premises community. Nonetheless, constructing such an answer could incur price and complexity, and it must be arrange by a platform group.

This put up presents a sensible strategy to accessing your Amazon MSK atmosphere for improvement functions by a bastion host utilizing a Safe Shell (SSH) tunnel (a generally used safe connection technique). Whether or not you’re working with Amazon MSK Serverless, the place public entry is unavailable, or with provisioned MSK clusters which are deliberately stored personal, this put up guides you thru the steps to determine a safe connection and seamlessly combine your native improvement atmosphere together with your MSK sources.

Answer overview

The answer permits you to straight hook up with the Amazon MSK Serverless service out of your native improvement atmosphere with out utilizing Direct Join or a VPN. The service is accessed with the bootstrap server DNS endpoint boot-<>.c<>.kafka-serverless.<>.amazonaws.com on port 9098, then routed by an SSH tunnel to a bastion host, which connects to the MSK Serverless cluster. Within the subsequent step, let’s discover find out how to arrange this connection.

The stream of the answer is as follows:

1. The Kafka shopper sends a request to hook up with the bootstrap server
2. The DNS question on your MSK Serverless endpoint is routed to a domestically configured DNS server
3. The domestically configured DNS server routes the DNS question to localhost.
4. The SSH tunnel forwards all of the visitors on port 9098 from the localhost to the MSK Serverless server by the Amazon Elastic Compute Cloud (Amazon EC2) bastion host.

The next picture exhibits the structure diagram.

Stipulations

Earlier than deploying the answer, it is advisable have the next sources deployed in your account:

1. An MSK Serverless cluster configured with AWS Identification and Entry Administration (IAM) authentication.
2. A bastion host occasion with community entry to the MSK Serverless cluster and SSH public key authentication.
3. AWS CLI configured with an IAM person and capable of learn and create matters on Amazon MSK. Use the IAM coverage from Step 2: Create an IAM position within the Getting began utilizing MSK Serverless clusters
4. For Home windows customers, set up Linux on Home windows with Home windows Subsystem for Linux 2 (WSL 2) utilizing Ubuntu 24.04. For steerage, consult with How one can set up Linux on Home windows with WSL.

This information assumes an MSK Serverless deployment in us-east-1, however it may be utilized in each AWS Area the place MSK Serverless is obtainable. Moreover, we’re utilizing OS X as working system. Within the following steps change msk-endpoint-url together with your MSK Serverless endpoint URL with IAM authentication. The MSK endpoint URL has a format like boot-<>.c<>.kafka-serverless.<>.amazonaws.com.

Answer walkthrough

To entry your Amazon MSK atmosphere for improvement functions, use the next walkthrough.

Configure native DNS server OSX

Set up Dnsmasq as an area DNS server and configure the resolver to resolve the Amazon MSK. The answer makes use of Dnsmasq as a result of it might evaluate DNS requests in opposition to a database of patterns and use these to find out the right response. This performance can match any request that ends in kafka-serverless.us-east-1.amazonaws.com and ship 127.0.0.1 in response. Observe these steps to put in Dnsmasq:

1. Replace brew and set up Dnsmasq utilizing brew

   brew up
   brew set up dnsmasq

2. Begin the Dnsmasq service

   sudo brew providers begin dnsmasq

3. Reroute all visitors for Serverless MSK (kafka-serverless.us-east-1.amazonaws.com) to 127.0.0.1

   echo handle=/kafka-serverless.us-east-1.amazonaws.com/127.0.0.1 >> $(brew --prefix)/and many others/dnsmasq.conf

4. Reload Dnsmasq configuration and clear cache

   sudo launchctl unload /Library/LaunchDaemons/homebrew.mxcl.dnsmasq.plist
   sudo launchctl load /Library/LaunchDaemons/homebrew.mxcl.dnsmasq.plist
   dscacheutil -flushcache

Configure OS X resolver

Now that you’ve got a working DNS server, you may configure your working system to make use of it. Configure the server to ship solely .kafka-serverless.us-east-1.amazonaws.com queries to Dnsmasq. Most working techniques which are just like UNIX have a configuration file known as /and many others/resolv.conf that controls the best way DNS queries are carried out, together with the default server to make use of for DNS queries. Use the next steps to configure the OS X resolver:

1. OS X additionally permits you to configure extra resolvers by creating configuration information within the /and many others/resolver/ This listing in all probability received’t exist in your system, so your first step needs to be to create it:

   sudo mkdir -p /and many others/resolver

2. Create a brand new file with the identical title as your new top-level area (kafka-serverless.us-east-1.amazonaws.com) within the /and many others/resolver/ listing and add 127.0.0.1 as a nameserver to it by coming into the next command.

   sudo tee /and many others/resolver/kafka-serverless.us-east-1.amazonaws.com >/dev/null <

Configure native DNS server Home windows

In Home windows Subsystem for Linux, first set up Dnsmasq, then configure the resolver to resolve the Amazon MSK and at last add localhost as the primary nameserver.

1. Replace apt and set up Dnsmasq utilizing apt. Set up the telnet utility for later checks:

   sudo apt replace
   sudo apt set up dnsmasq
   sudo apt set up telnet

2. Reroute all visitors for Serverless MSK (kafka-serverless.us-east-1.amazonaws.com) to 127.0.0.1.

   echo "handle=/kafka-serverless.us-east-1.amazonaws.com/127.0.0.1" | sudo tee -a /and many others/dnsmasq.conf

3. Reload Dnsmasq configuration and clear cache.

   sudo /and many others/init.d/dnsmasq restart

4. Open /and many others/resolv.conf and add the next code within the first line.

   nameserver 127.0.0.1

   The output ought to seem like the next code.

   #Some feedback
   nameserver 127.0.0.1
   nameserver <>
   ..

Create SSH tunnel

The subsequent step is to create the SSH tunnel, which can permit any connections made to localhost:9098 in your native machine to be forwarded over the SSH tunnel to the goal Kafka dealer. Use the next steps to create the SSH tunnel:

1. Exchange bastion-host-dns-endpoint with the general public DNS endpoint of the bastion host, which comes within the type of <<xyz>>.compute-1.amazonaws.com, and change ec2-key-pair.pem with the important thing pair of the bastion host. Then create the SSH tunnel by coming into the next command.

   ssh -i "~/<<ec2-key-pair.pem>>" ec2-user@<<bastion-host-dns-endpoint>> -L 127.0.0.1:9098:<<msk-endpoint-url>>:9098

2. Go away the SSH tunnel working and open a brand new terminal window.
3. Check the connection to the Amazon MSK server by coming into the next command.

   telnet <<msk-endpoint-url>> 9098

   The output ought to seem like the next instance.

   Making an attempt 127.0.0.1...
   Linked to boot-<>.c<>.kafka-serverless.us-east-1.amazonaws.com.
   Escape character is '^]'.

Testing

Now configure the Kafka shopper to make use of IAM Authentication after which check the setup. You discover the most recent Kafka set up on the Apache Kafka Obtain website. Then unzip and replica the content material of the Dafka folder into ~/kafka.

1. Obtain the IAM authentication and unpack it

   cd ~/kafka/libs
   wget https://github.com/aws/aws-msk-iam-auth/releases/obtain/v2.2.0/aws-msk-iam-auth-2.2.0-all.jar
   cd ~

2. Configure Kafka properties to make use of IAM because the authentication mechanism

   cat < ~/kafka/config/client-config.properties
   
   # Units up TLS for encryption and SASL for authN.
   
   safety.protocol = SASL_SSL
   
   # Identifies the SASL mechanism to make use of.
   
   sasl.mechanism = AWS_MSK_IAM
   
   # Binds SASL shopper implementation.
   
   sasl.jaas.config = software program.amazon.msk.auth.iam.IAMLoginModule required;
   
   
   # Encapsulates developing a SigV4 signature based mostly on extracted credentials.
   
   # The SASL shopper sure by "sasl.jaas.config" invokes this class.
   
   sasl.shopper.callback.handler.class = software program.amazon.msk.auth.iam.IAMClientCallbackHandler
   
   EOF

3. Enter the next command in ~/kafka/bin to create an instance matter. Make it possible for the SSH tunnel created within the earlier part continues to be open and working.

   ./kafka-topics.sh --bootstrap-server <<msk-endpoint-url>>:9098 --command-config ~/kafka/config/client-config.properties --create --topic ExampleTopic --partitions 10 --replication-factor 3 --config retention.ms=3600000

Cleanup

To take away the answer, full the next steps for Mac customers:

1. Delete the file /and many others/resolver/kafka-serverless.us-east-1.amazonaws.com
2. Delete the entry handle=/kafka-serverless.us-east-1.amazonaws.com/127.0.0.1 within the file $(brew --prefix)/and many others/dnsmasq.conf
3. Cease the Dnsmasq service sudo brew providers cease dnsmasq
4. Take away the Dnsmasq service sudo brew uninstall dnsmasq

To take away the answer, full the next steps for WSL customers:

1. Delete the file /and many others/dnsmasq.conf
2. Delete the entry nameserver 127.0.0.1 within the file /and many others/resolv.conf
3. Take away the Dnsmasq service sudo apt take away dnsmasq
4. Take away the telnet utility sudo apt take away telnet

Conclusion

On this put up, I introduced you with steerage on how builders can hook up with Amazon MSK Serverless from native environments. The connection is finished utilizing an Amazon MSK endpoint by an SSH tunnel and a bastion host. This allows builders to experiment and check domestically, without having to setup a separate Kafka cluster.

Concerning the Writer

Simon Peyer is a Options Architect at Amazon Net Providers (AWS) based mostly in Switzerland. He’s a sensible doer and enthusiastic about connecting expertise and other people utilizing AWS Cloud providers. A particular focus for him is knowledge streaming and automations. Moreover work, Simon enjoys his household, the outside, and mountaineering within the mountains.