Prerequisites and cluster setup

Test communication between instances. On the destination instance, run the following command to listen for pings from the source instance.

sudo tcpdump host source-instance-private-ip

On the source instance, run the following command to ping the destination instance.

ping -c 4 destination-instance-private-ip

You should see ping responses on both sides. If not, verify that your security group allows all traffic between instances.

Disable cloud-init hostname management. On Ubuntu Amazon EC2 instances, cloud-init can revert hostname and /etc/hosts changes on reboot. Run this on each instance to prevent that.

sudo bash -c 'cat > /etc/cloud/cloud.cfg.d/99-preserve-hostconfig.cfg <<EOF
preserve_hostname: true
manage_etc_hosts: false
EOF'

Set the hostname on each instance. Pacemaker uses hostnames to identify cluster instances.

On instance 1:

sudo hostnamectl set-hostname instance1

On instance 2:

sudo hostnamectl set-hostname instance2

Configure /etc/hosts. On each instance, append the private IP and hostname pairs to /etc/hosts. This allows instances to resolve each other by hostname.

sudo tee -a /etc/hosts <<EOF
instance1-private-ip instance1
instance2-private-ip instance2
EOF

Start the pcs daemon on each instance.

sudo systemctl enable pcsd.service
sudo systemctl start pcsd.service

Set the hacluster password. The hacluster user is a system account created automatically when you install the pcs package. Pacemaker uses this account for cluster authentication between instances. Set the same password on each instance.

sudo passwd hacluster

Authorize pcs hosts. Authorize the cluster instances with each other. Run this on each instance.

sudo pcs host auth instance1 instance2

You will be prompted for the hacluster username and password.

Set up the cluster. Run this command on one instance only. The --force flag destroys any existing cluster on the device.

sudo pcs cluster setup my-gg-cluster instance1 instance2 --force

Start the cluster. Run this command on one instance only.

sudo pcs cluster start --all
sudo pcs cluster enable --all

Verify the cluster status on either instance.

sudo pcs status

Create and attach EBS volumes. Create an EBS volume for each instance. The volume must be in the same Availability Zone as the instance.

Verify the volume. SSH into each instance and verify the volume is attached by listing block devices.

lsblk

On Nitro-based instance types such as t3.small, the volume appears as an NVMe device (for example, /dev/nvme1n1) instead of /dev/xvdbz. Use lsblk to identify the actual device name, and use that name in the DRBD configuration.

Install DRBD utilities on each instance.

sudo apt install -y drbd-utils

Configure the DRBD resource. Create the DRBD resource configuration directory and file on both instances.

sudo mkdir -p /etc/drbd.d
sudo vi /etc/drbd.d/greengrass.res

Add the following configuration. Replace the hostnames and IP addresses with your values.

resource greengrass {
    protocol C;

    on instance1 {
        device    /dev/drbd0;
        disk      /dev/instance1-device-name;
        address   instance1-private-ip:7788;
        meta-disk internal;
    }

    on instance2 {
        device    /dev/drbd0;
        disk      /dev/instance2-device-name;
        address   instance2-private-ip:7788;
        meta-disk internal;
    }
}

Create and start the DRBD resource. Run on both instances.

sudo drbdadm create-md greengrass
sudo drbdadm up greengrass

Verify with lsblk that drbd0 appears under your EBS volume device (for example, xvdbz or nvme1n1).

Set the primary instance. Run the following commands on instance1 only to force it to be primary and wait for synchronization.

sudo drbdadm -- --overwrite-data-of-peer primary greengrass
sudo drbdadm status

Wait until both instances show UpToDate for the disk state. Once synchronized, all other instances become standby instances.

Format and mount the DRBD device.

On the primary instance only, format the DRBD device:

sudo mkfs.ext4 /dev/drbd0

On all instances, create the mount directory:

sudo mkdir -p /greengrass/v2

On the primary instance only, mount the DRBD device:

sudo mount /dev/drbd0 /greengrass/v2

Verify DRBD sync status.

sudo drbdadm status
mount | grep 'drbd'