Logging EventBridge events to S3 with Firehose

When using EventBridge I always log all events to an S3 bucket for auditing, analytics and debugging purposes. A super easy method to do this is to create a Kinesis Data Firehose stream and create a rule that captures all events that points to the Firehose stream. The Firehose stream can then flush the events on S3 in an interval/size of choice based on configuration.

The issue I ran into is that all events in JSON were all on a single line while I prefer to have them separated by a newline. This is known as the JSON Lines format. I knew this would be possible by leveraging the Data Transformation feature of Firehose however this just didn't feel right (in other words, I didn't want to pay for it :-). Back when I was looking into this, I found a fix for it and posted it on Twitter. Since others might be unfamiliar with this solution I decided to create a post explaining this in more detail to make it easier to find than the thread on Twitter.

The solution turned out to be relatively simple in the end. You need to transform the input from EventBridge by configuring an Input Transformer on your rule. To modify the event to include a newline. As the Input path you should configure:

{}

and as Template you should configure:

<http://aws.events.event>

The trick is to add a newline after the <http://aws.events.event>.

As I highly recommend to use CloudFormation rather than doing things in the console:

AllAccountEventsRule:
  Type: AWS::Events::Rule
  Properties:
    EventBusName: !GetAtt EventBus.Name
    EventPattern:
      account:
        - Ref: 'AWS::AccountId'
    State: ENABLED
    Targets:
      - Id: FirehoseLogger
        Arn: !GetAtt FirehoseStream.Arn
        RoleArn: !GetAtt EventBridgeFirehoseDeliveryRole.Arn
        InputTransformer:
          InputPathsMap: { }
          InputTemplate: |
            <aws.events.event>
# empty because of the newline ^

Thanks to James for providing a CDK solution:

return new targets.KinesisFirehoseStream(kinesisStream, {
    message: {
        bind: () => ({inputPathsMap: {}, inputTemplate: '<http://aws.events.event>\n'}),
    },
});

The files written by Firehose on S3 will have every event separated by a newline:

{"version":"0","id":"493acfe4-35fd-4050-a850-de2191261310","detail-type":"action_a","source":"your_app","account":"your_account_id","time":"2022-08-02T01:02:08Z","region":"eu-west-1","resources":[],"detail":{"foo":"bar"}}
{"version":"0","id":"c76450da-079c-463d-b02a-edad99783952","detail-type":"action_b","source":"your_app","account":"your_account_id","time":"2022-08-02T01:02:08Z","region":"eu-west-1","resources":[],"detail":{"foo":"bar"}}
{"version":"0","id":"857dcf59-d54c-4ca7-8342-9eaa0988cad1","detail-type":"action_c","source":"your_app","account":"your_account_id","time":"2022-08-02T01:02:08Z","region":"eu-west-1","resources":[],"detail":{"foo":"bar"}}

If you configure your S3 bucket with Object Lock you will have an immutable log of everything that happened in your application which is easily parsable.

To easily query events, you can of course store the events in a database of choice (DynamoDB, Elasticsearch, Postgres). To populate the database with batches of events, rather than per event, you can create a Lambda that triggers when a new log file is written by Firehose. You can then read the file from S3, split by newline and push the events to the database.