Using a NodeJS Lambda to Read Events from Kinesis Stream and Write Events in CloudWatch

Over the last few weeks, I have been working on a requirement to send CloudWatch Log Group Events with AWS Directory Services event to a Cross-account CloudWatch Log Group to centralize all security events.

Note: this Lambda will be useful for similar cases and is not limited to Directory Service events.

The Architecture of The Solution

The suggested architecture by AWS Technical Support looks like this:

AWS Directory Service (Source Account) → CloudWatch Log Group (Source Account) → Subscription Filter (Source Account) → Amazon Kinesis Stream (Destination Account) → Lambda (Destination Account) → CloudWatch Log Group (Destination Account)

Dealing with Some Unexpected Problems

When we wrote the first Lambda using the sample provided by AWS, I found some unexpected issues:

• The stream of data is encoded and compressed.
• Receiving a batch of events instead of a single event

The Stream is Encoded and Compressed

When we read the raw data from the stream, we notice that the events are not in clear text. Logs sent to a receiving service through a subscription filter are base64 encoded and compressed with the gzip format.

So we will need to unzip and decode the data using the zlib library:

Receiving a batch of Events

After writing our Lambda, we can check the CloudWatch Log Group to look for the result. When we open the event, we will notice that inside each event entry, there is a header with some data and multiple events!