Skip to main content

Privacera Documentation

Setup audit logs for AWS Lake Formation on Platform

Important

Prerequisite: Before performing following steps you must have Amazon S3 bucket for storing the audit logs.

Note

When enabling access audits through PrivaceraCloud Portal, you must enable the CloudTrail from AWS. For more information, see AWS guidelines for cost management with CloudTrail.

Steps to enable audit logs in AWS Lake Formation:
  1. Create Trail in AWS cloud trail.

    1. Go to AWS Cloud trail service → DashboardCreate Trail.

    2. On the General Details page, provide the Trail name (such as LF_Cloud_Trail) of your choice.

    3. Provide the S3 storage location of your S3 bucket.

      Tip

      Preferably use an existing location from the S3 bucket created in the pre-requisites. Note down the S3 bucket location for cloud trail logs. For example, privacera-dev-bucket/AWSLogs/587946681758.

    4. Uncheck the Log file SSE-KMS encryption and Log file validation checkboxes. Click Next.

    5. On next page under Choose Log Events enable the checkbox for Management Events with the checkboxes for Read and Write as checked.

    6. Enable the checkbox for Log Events. In Data Events section, select the Data Event type as AWS Lake Formation and Log Selector template as Log All Events.

    7. Click Next. Confirm the values and click Create Trail.

  2. Enable the audit log flag in Privacera Platform:

    Set the CONNECTOR_LAKEFORMATION_AUDIT_ENABLE flag to true.

  3. Create a database and table for storing audit logs query results. For this, you can use AWS Athena.

    1. Create database:

      CREATE DATABASE lf_audit_db;
    2. Create table:

      (In the following query, the LOCATION is the Amazon S3 bucket location of the cloud trail logs.)

      CREATE EXTERNAL TABLE lf_audit_db.cloudtrail_logs (
          eventVersion STRING,
          userIdentity STRUCT<
              type: STRING,
              principalId: STRING,
              arn: STRING,
              accountId: STRING,
              invokedBy: STRING,
              accessKeyId: STRING,
              userName: STRING,
              sessionContext: STRUCT<
                  attributes: STRUCT<
                      mfaAuthenticated: STRING,
                      creationDate: STRING>,
                  sessionIssuer: STRUCT<
                      type: STRING,
                      principalId: STRING,
                      arn: STRING,
                      accountId: STRING,
                      userName: STRING>>>,
          eventTime STRING,
          eventSource STRING,
          eventName STRING,
          awsRegion STRING,
          sourceIpAddress STRING,
          userAgent STRING,
          errorCode STRING,
          errorMessage STRING,
          requestParameters STRING,
          responseElements STRING,
          additionalEventData STRING,
          requestId STRING,
          eventId STRING,
          resources ARRAY<STRUCT<
              arn: STRING,
              accountId: STRING,
              type: STRING>>,
          eventType STRING,
          apiVersion STRING,
          readOnly STRING,
          recipientAccountId STRING,
          serviceEventDetails STRING,
          sharedEventID STRING,
          vpcEndpointId STRING
      )
      COMMENT 'CloudTrail table for lakeformation audit logs'
      ROW FORMAT SERDE 'com.amazon.emr.hive.serde.CloudTrailSerde'
      STORED AS INPUTFORMAT 'com.amazon.emr.cloudtrail.CloudTrailInputFormat'
      OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
      LOCATION 's3://privacera-dev-bucket/AWSLogs/587946681758'
      TBLPROPERTIES ('classification'='cloudtrail');
  4. In the configuration file for the connector instance you created for AWS Lake Formation (which is ~/privacera/privacera-manager//config/custom-vars/connectors/lakeformation/<your_instance_name>/vars.connector.lakeformation.yml), set the following properties:

    Property

    Description

    PrivaceraCloud custom property

    Privacera Platform property

    AWS Athena Region

    Specifies AWS Athena region to create JDBC connection for AWS Lake Formation audit logs database. If not specified, will default to use the first region from AWS Regions property.

    aws.athena.region=<athena_region_name>

    CONNECTOR_LAKEFORMATION_AWS_ATHENA_REGION="<athena_region_name>"

    AWS Athena Endpoint

    Specifies AWS Athena endpoint to create JDBC connection for AWS Lake Formation audit logs database. If not specified, will create an endpoint with default region from the region property.

    aws.athena.endpoint=<athena_endpoint>

    CONNECTOR_LAKEFORMATION_AWS_ATHENA_ENDPOINT="<athena_endpoint>"

    AWS Athena Workgroup

    Specifies AWS Athena workgroup to create JDBC connection for AWS Lake Formation audit logs database.

    aws.athena.workgroup=<athena_workgroup>

    CONNECTOR_LAKEFORMATION_AWS_ATHENA_WORKGROUP="<athena_workgroup>"

    Audit Database Name

    Specifies AWS audit database to store AWS Lake Formation audit logs.

    audit.db.name=<audit_db_name>

    CONNECTOR_LAKEFORMATION_AUDIT_DB_NAME="<audit_db_name>"

    Audit Table Name

    Specifies AWS audit table to store AWS Lake formation audit logs.

    audit.table.name=<audit_table_name>

    CONNECTOR_LAKEFORMATION_AUDIT_TABLE_NAME="<audit_db_name>"

    AWS Athena S3 output location

    Specifies S3 location to store the access audit logs query results.

    aws.athena.s3.output.location=<aws_athena_s3_output_location>

    CONNECTOR_LAKEFORMATION_AUDIT_ATHENA_S3_OUTPUT_LOCATION="<aws_athena_s3_output_location>"

  5. There are two additional optional properties used to restrict collecting audit logs for excluding specific users and specific access types:

    Property

    Description

    PrivaceraCloud custom property

    Privacera Platform property

    Audit Excluded Users

    Specifies a list of users to exclude when fetching access audits.

    audit.excluded.users=<audit_excluded_users>

    CONNECTOR_LAKEFORMATION_AUDIT_EXCLUDED_USERS="<audit_excluded_users>"

    Audit Excluded Access type

    Specifies a list of list of access types to exclude when fetching access audits.

    For example: StartQueryExecution, GetTable,DeleteTable, CreateTable, CreateDatabase.

    audit.excluded.access.types=<audit_excluded_access_types>

    CONNECTOR_LAKEFORMATION_AUDIT_EXCLUDED_ACCESS_TYPES="<aws_athena_s3_output_location>"