Ansible Playbook to install KAFKA on Ubuntu - How to | Devops Junction

In this article, we are going to see how to install Kafka on ubuntu with Ansible.  We created a simple playbook to download and install Kafka and start it.

Here is the list of tasks the playbook is designed to do

• Perform apt update
• Install JRE needed for Kafka and Zookeeper
• Create a User and Group for Kafka
• Create a directory to install Kafka
• Download Kafka and Unarchive
• Update the Log Path
• Update the Heap Size of the Kafka
• Create a Service - systemctl file for Kafka and Zookeeper
• Enabling the Service for Stratup

Ansible Playbook to install Apache Kafka on Ubuntu

Since I have already listed the tasks this playbook is doing. I do not want to cover this in detail.

There are various modules used in this playbook. We have dedicated examples for each of these modules in here

• apt - to install packages in ubuntu
• group - to create a user group
• user - to create a user
• file - to create installation directory
• unarchive - to download and extract the zip distribution of Kafka
• shell - to move files to the right directory before starting a service
• lineinfile - to update the heapsize
• copy - to create a service file in /etc/systemd
• systemd - to enable and start the services
• wait_for - to validate if Kafka and Zookeeper is up and running and the port is open

 

- name: Installing Kafka on Ubuntu
  hosts: testserver
  vars: 
    - installation_dir : /opt/kafka
  tasks:
    - name: Install JRE after apt update
      become: yes
      apt:
        name: 
          - default-jre
        state: present
        update_cache: yes

    - name: Create a group 
      become: yes
      group:
        name: kafka
        state: present

    - name: Create an user 
      become: yes
      user:
        name: kafka
        state: present
        group: kafka

    - name: Create a Directory /opt/kafka
      become: yes
      file:
        path: "{{installation_dir}}"
        state: directory
        mode: 0755
        owner: kafka
        group: kafka

    - name: Download Kafka and Unzip 
      become: yes
      become_user: kafka
      unarchive:
        src: https://dlcdn.apache.org/kafka/3.1.0/kafka_2.13-3.1.0.tgz
        dest: "{{installation_dir}}"
        mode: 0755
        remote_src: yes

    - name: Move all the files to parent Directory
      become: yes
      become_user: kafka
      shell:
        mv {{installation_dir}}/kafka_*/* {{installation_dir}}/.

    - name: Update the log path
      become: yes
      become_user: kafka
      replace:
        path: "{{installation_dir}}/config/server.properties"
        regexp: 'log.dirs=(.+)'
        replace: 'log.dirs={{installation_dir}}/logs'
        backup: yes

    - name: Update the Java Heap Size for Kafka
      become: yes
      become_user: kafka
      replace:
        path: "{{installation_dir}}/bin/kafka-server-start.sh"
        regexp: 'export KAFKA_HEAP_OPTS=(".+")'
        replace: 'export KAFKA_HEAP_OPTS="-Xmx520M -Xms520M"'
        backup: yes
    

    - name: Create a Service file for ZooKeeper with Copy module
      become: yes
      copy:
        dest: /etc/systemd/system/zookeeper.service
        content: |
          [Unit]
          Requires=network.target remote-fs.target
          After=network.target remote-fs.target

          [Service]
          Type=simple
          User=kafka
          ExecStart={{installation_dir}}/bin/zookeeper-server-start.sh {{installation_dir}}/config/zookeeper.properties
          ExecStop={{installation_dir}}/bin/zookeeper-server-stop.sh
          Restart=on-abnormal

          [Install]
          WantedBy=multi-user.target
        mode: 0755
      
    - name: Create a Service file for Kafka with Copy module
      become: yes
      copy:
        dest: /etc/systemd/system/kafka.service
        content: |
          [Unit]
          Requires=zookeeper.service
          After=zookeeper.service

          [Service]
          Type=simple
          User=kafka
          ExecStart=/bin/sh -c '{{installation_dir}}/bin/kafka-server-start.sh {{installation_dir}}/config/server.properties > {{installation_dir}}/kafkaservice.log 2>&1'
          ExecStop={{installation_dir}}/bin/kafka-server-stop.sh
          Restart=on-abnormal

          [Install]
          WantedBy=multi-user.target
        mode: 0755

    - name: Start Services
      tags: startservices
      become: yes
      systemd:
        name: '{{item}}'
        state: started
        enabled: yes
      with_items:
        - "kafka"
        - "zookeeper"
    
    - name: Validating if zookeeper is up and listening on port 2181
      wait_for:
        host: localhost
        port: 2181
        delay: 10
        timeout: 30
        state: started
        msg: "Zookeeper not seem to be running"

    - name: Validating if Kafka is up and listening on port 2181
      wait_for:
        host: localhost
        port: 9092
        delay: 10
        timeout: 30
        state: started
        msg: "Zookeeper not seem to be running"

You can get the same version of this playbook from my gist.github.com too here

Terminal Record - Execution Output and Validation

In this terminal record, we have recorded the execution of the playbook and the resulting configuration by logging into the server.

Also, we tried to create a new topic and publish/consume messages using the Kafka built-in scripts.

 

Cheers
Sarav AK

---