Skip to main content

Privacera Documentation

Configure Fluentd audit logging on Privacera Platform

This topic explains where to store audit data from the Privacera AuditServer

The choices for storage are as follows. You configure the storage location by setting properties in the vars.audit-fluentd.yml file. The audit data can also be stored in the same location as the application logs.

  • Local to the server on which Privacera Platform is installed

  • AWS S3

  • ADLS Gen 2

Prerequisites

Ensure the following prerequisites are met:

  • AuditServer must be up and running. For more information, see Configure Solr destination on Privacera Platform.

  • If you're configuring Fluentd for an Azure environment and want to configure User Managed Service Identity (MSI), assign the following two IAM roles to the Azure Storage account for the User Managed Service Identity where the audits will be stored:

    • Owner or Contributor

    • Storage Blob Data Owner or Storage Blob Data Contributor

    Note

    If your Azure environment is Docker-based, then configure MSI on a virtual machine, whereas for a Kubernetes-based environment, configure MSI on a virtual machine scale set (VMSS).

Procedure
  1. SSH to the instance where Privacera is installed.

  2. Run the following commands to change the directory to privacera-manager:

    cd ~/privacera/privacera-manager
    
  3. Run the following command to copy the vars.audit-fluentd.yml file:

    cp config/sample-vars/vars.audit-fluentd.yml config/custom-vars/
  4. Run the following command to edit the properties in vars.audit-fluentd.yml file:

    vi config/custom-vars/vars.audit-fluentd.yml
  5. Modify the Fluentd configuration properties. For the Fluentd configuration property details, see Fluentd configuration properties.

  6. Run the following commands to update the Privacera manager:

    cd ~/privacera/privacera-manager
    ./privacera-manager.sh update

Fluentd configuration properties

Property

Description

Example

AUDIT_FLUENTD_AUDIT_DESTINATION

Set the audit destination where the audits will be saved. If the value is set to S3, the audits get stored in the AWS S3 server. For S3, the default time interval to publish the audits is 3600s (1hr).

Local storage should be used only for development and testing purposes. All the audits received are stored in the same container/pod.

Value: local, s3, azure-blob, azure-adls

s3

AUDIT_FLUENTD_EXPORT_APP_LOGS_ENABLE

Specifies whether application logs and PolicySync logs are sent to Fluentd. The default value is false.

true

For destination local, edit the following property:

AUDIT_FLUENTD_LOCAL_FILE_TIME_INTERVAL

The time interval after which the audits will be pushed to the local destination.

3600s

AUDIT_FLUENTD_LOCAL_FLUSH_INTERVAL

Flushes (writes) chunks at specified interval.

No default.

30s

AUDIT_FLUENTD_LOCAL_FLUSH_THREAD_COUNT

Number of concurrent threads for flushing chunks.

No default.

8

AUDIT_FLUENTD_LOCAL_OVERFLOW_ACTION

Determines behavior of output plug-in when its queue is full.

No default.

Allowable values:

  • drop_oldest_chunk: Delete the oldest chunk to accept newly incoming chunk.

  • block: Waits until buffer can store more data. Suitable for processing in batch.

  • throw_exception: Writes an exception to the Fluentd log

block

For destination S3, edit the following properties:

AUDIT_FLUENTD_S3_BUCKET

Set the bucket name, if you set the audit destination above to S3.

Leave unchanged, if you set the audit destination to local.

bucket_1

AUDIT_FLUENTD_S3_REGION

Set the bucket region, if you set the audit destination above to S3.

Leave unchanged, if you set the audit destination to local.

us-east-1

AUDIT_FLUENTD_S3_FILE_TIME_INTERVAL

This is the time interval after which the audits will be pushed to the S3 destination.

3600s

AUDIT_FLUENTD_S3_ACCESS_KEY

AUDIT_FLUENTD_S3_SECRET_KEY

Set the access and secret key, if you set the audit destination above to S3.

Leave unchanged, if you set the audit destination to local and are using AWS IAM Instance Role.

AUDIT_FLUENTD_S3_ACCESS_KEY: "AKIAIOSFODNN7EXAMPLE"

AUDIT_FLUENTD_S3_SECRET_KEY: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

AUDIT_FLUENTD_S3_BUCKET_ENCRYPTION_TYPE

Property to encrypt an S3 bucket. You can use the property, if you have set S3 as the audit destination in the property, AUDIT_FLUENTD_AUDIT_DESTINATION.

You can assign one of the following values as the encryption types:

  • SSE-S3

  • SSE-KMS

  • SSE-C

  • NONE

SSE-S3 and SSE-KMS are encryptions managed by AWS. You need to enable the server-side encryption for the S3 bucket. For more information on how to enable SSE-S3 or SSE-KMS encryption types, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/default-bucket-encryption.html

SSE-C is the custom encryption type, where the encryption key and MD5 have to generated separately.

NONE

AUDIT_FLUENTD_S3_BUCKET_ENCRYPTION_KEY

If you have set SSE-C encryption type in the AUDIT_FLUENTD_S3_BUCKET_ENCRYPTION_TYPE property, then the encryption key is mandatory. It is optional for SSE-KMS encryption type.

AUDIT_FLUENTD_S3_BUCKET_ENCRYPTION_KEY_MD5

If you have set SSE-C encryption type in the AUDIT_FLUENTD_S3_BUCKET_ENCRYPTION_TYPE property, then the MD5 encryption key is mandatory.

To get the MD5 hash for the encryption key, run the following command:

echo -n "<generated-key>"|  openssl dgst -md5 -binary | openssl enc -base64

AUDIT_FLUENTD_S3_FLUSH_INTERVAL

Flushes (writes) chunks at specified interval.

No default.

30s

AUDIT_FLUENTD_S3_FLUSH_THREAD_COUNT

Number of concurrent threads for flushing chunks.

No default.

8

AUDIT_FLUENTD_S3_OVERFLOW_ACTION

Determines behavior of output plug-in when its queue is full.

No default.

Allowable values:

  • drop_oldest_chunk: Delete the oldest chunk to accept newly incoming chunk.

  • block: Waits until buffer can store more data. Suitable for processing in batch.

  • throw_exception: Writes an exception to the Fluentd log

block

AUDIT_FLUENTD_S3_PATH_PREFIX

Change the value for AUDIT_FLUENTD_S3_PATH_PREFIX , only if you want to store audits in specific path than default.

Note

This path prefix should not start with / or end with /.

  • AUDIT_FLUENTD_S3_PATH_PREFIX: "<Provide new path for Audit storage>"

AUDIT_FLUENTD_S3_PATH_PREFIX: "team/dev/audit"

For destination azure-blob or azure-adls, edit the following properties:

AUDIT_FLUENTD_AZURE_STORAGE_ACCOUNT

AUDIT_FLUENTD_AZURE_CONTAINER

Set the storage account and the container, if you set the audit destination above to Azure Blob or Azure ADLS.

To know how to get the ADLS properties, see Get ADLS properties.

Leave unchanged, if you set the audit destination to local.

Note

Currently, it supports Azure blob storage only.

AUDIT_FLUENTD_AZURE_STORAGE_ACCOUNT: "storage_account_1"

AUDIT_FLUENTD_AZURE_CONTAINER: "container_1"

AUDIT_FLUENTD_AZURE_FILE_TIME_INTERVAL

This is the time interval after which the audits will be pushed to the Azure ADLS/Blob destination.

3600s

AUDIT_FLUENTD_AUTH_TYPE

Select an authentication type from the dropdown list.

AUDIT_FLUENTD_AZURE_STORAGE_ACCOUNT_KEY

AUDIT_FLUENTD_AZURE_STORAGE_SAS_TOKEN

Configure this property, if you have selected SAS Key in the property, AUDIT_FLUENTD_AUTH_TYPE.

Set the storage account key and the SAS token, if you set the audit destination above to Azure Blob.

Leave unchanged, if you're using Azure's Managed Identity Service.

AUDIT_FLUENTD_AZURE_OAUTH_TENANT_ID

AUDIT_FLUENTD_AZURE_OAUTH_APP_ID

AUDIT_FLUENTD_AZURE_OAUTH_SECRET

Set the storage account key and the SAS token, if you set the audit destination above to Azure ADLS.

Configure this property, if you have selected OAUTH in the property, AUDIT_FLUENTD_AUTH_TYPE.

Leave unchanged, if you're using Azure's Managed Identity Service.

AUDIT_FLUENTD_AZURE_USER_MANAGED_IDENTITY_ENABLE

AUDIT_FLUENTD_AZURE_USER_MANAGED_IDENTITY

Configure this property, if you have selected MSI (UserManaged) in the property, AUDIT_FLUENTD_AUTH_TYPE.

AUDIT_FLUENTD_AZURE_FLUSH_INTERVAL

Flushes (writes) chunks at specified interval.

No default.

30s

AUDIT_FLUENTD_AZURE_FLUSH_THREAD_COUNT

Number of concurrent threads for flushing chunks.

No default.

8

AUDIT_FLUENTD_AZURE_OVERFLOW_ACTION

Determines behavior of output plug-in when its queue is full.

No default.

Allowable values:

  • drop_oldest_chunk: Delete the oldest chunk to accept newly incoming chunk.

  • block: Waits until buffer can store more data. Suitable for processing in batch.

  • throw_exception: Writes an exception to the Fluentd log

block

AUDIT_FLUENTD_AZURE_PATH_PREFIX

Change the value for AUDIT_FLUENTD_AZURE_PATH_PREFIX , only if you want to store audits in specific path than default.

Note

This path prefix should not start with / or end with /.

  • AUDIT_FLUENTD_AZURE_PATH_PREFIX: "<Provide new path for Audit storage>"

AUDIT_FLUENTD_AZURE_PATH_PREFIX: "team/qa/audit"