Localstack Kinesis Firehose + AWS CDK

This article shows you how to setup Kinesis Firehose for local development using LocalStack & AWS CDK

This article is the next for the article setup local stack S3 & AWS CDK repository in section #3 below continue we will set up Kinesis Firehose for our development.

Localstack S3 + AWS CDK on your local with CyberDuck

Requirement

We need a setup a Kinesis Firehose to store user logs for tracking issues

1. We will create an S3 Bucket that will serve as the destination for our data
2. Create Delivery Stream with the method DirectPut and set up the stream with the S3 Bucket created at Step 1.

import * as s3 from '@aws-cdk/aws-s3';
const auditLogBucket = new s3.Bucket(root, "test-firehose-s3-bucket", {
  removalPolicy:RemovalPolicy.DESTROY, // REMOVE FOR PRODUCTION
  autoDeleteObjects: true, // REMOVE FOR PRODUCTION,
  bucketName: 'test-firehose-s3',
  publicReadAccess: true,
});

Create A DeliveryStream with S3 Destination configuration above.

import { CfnDeliveryStream } from "@aws-cdk/aws-kinesisfirehose";
const firehoseRole = new iam.Role(root, 'firehoseRole', {
  assumedBy: new iam.ServicePrincipal('firehose.amazonaws.com')
});

return new CfnDeliveryStream(root, "FirehoseStreamToS3", {
  deliveryStreamName: "test-firehose-delivery-stream",
  deliveryStreamType: "DirectPut",
  s3DestinationConfiguration: {
    bucketArn: auditLogBucket.bucketArn,
    bufferingHints: {
      sizeInMBs: Size.mebibytes(1).toMebibytes(),
      intervalInSeconds: Duration.seconds(60).toSeconds()
    },
    compressionFormat: 'UNCOMPRESSED',
    encryptionConfiguration: {
      noEncryptionConfig: "NoEncryption"
    },
    prefix: "user-logs",
    errorOutputPrefix: 'user-logs-logs',
    roleArn: firehoseRole.roleArn
  },
});

Deploy our Stack

cdklocal deploy

✅ ApplicationStack

Verify our S3 Bucket setup

aws — endpoint-url=http://localhost:4566 s3api get-bucket-policy — bucket test-firehose-s3

Output

{
  "Statement": [
    {
      "Action": "s3:GetObject",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Resource": "arn:aws:s3:::test-firehose-s3/*"
    },
    {
      "Action": [
        "s3:GetBucket*",
        "s3:List*",
        "s3:DeleteObject*"
      ],
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::000000000000:role/cf-ApplicationStack-CustomS3AutoDeleteObjectsCustomResourceProviderRole3B1BD092"
      },
      "Resource": [
        "arn:aws:s3:::test-firehose-s3",
        "arn:aws:s3:::test-firehose-s3/*"
      ]
    }
  ]
}

Verify our Kinesis Firehose setup

{
  "DeliveryStreamDescription": {
    "DeliveryStreamName": "test-firehose-delivery-stream",
    "DeliveryStreamARN": "arn:aws:firehose:us-west-2:000000000000:deliverystream/test-firehose-delivery-stream",
    "DeliveryStreamStatus": "ACTIVE",
    "DeliveryStreamType": "DirectPut",
    "VersionId": "1",
    "CreateTimestamp": "2021-08-09T11:06:49.381495+07:00",
    "Destinations": [
      {
        "DestinationId": "686ab370",
        "S3DestinationDescription": {
          "RoleARN": "arn:aws:iam::000000000000:role/cf-ApplicationStack-firehoseRoleE5891AF8",
          "BucketARN": "arn:aws:s3:::test-firehose-s3",
          "Prefix": "user-logs",
          "ErrorOutputPrefix": "user-error-logs",
          "BufferingHints": {
            "SizeInMBs": 1,
            "IntervalInSeconds": 60
          },
          "CompressionFormat": "UNCOMPRESSED",
          "EncryptionConfiguration": {
            "NoEncryptionConfig": "NoEncryption"
          }
        }
      }
    ],
    "HasMoreDestinations": false
  }
}

Please refer the Github for this section:

https://github.com/kobee-tech-stack/aws-development-with-localstack/tree/aws-cdk-firehose/infra