Skip to main content

Privacera Platform master publication

Dremio
:
Introduction

This section covers how you can integrate Dremio with Privacera. You can use Dremio for table-level access control with the native Ranger plugin.

By integrating Dremio with Privacera, you'll be provided with comprehensive data lake security and fine-grained access control across multi-cloud environments. Dremio works directly with data lake storage. Using Dremio's query engine and ability to democratize data access, Privacera implements fine-grained access control policies, then automatically enforces and audits them at enterprise scale.

Dremio is supported with the following data sources:

  • S3

  • ADLS

  • Hive

  • Redshift

Prerequisites

Ensure the following prerequisites are met:

  • A Privacera Manager host where Privacera services are running.

  • A Dremio host where Dremio Enterprise Edition is installed. (The Community Edition is not supported.)

Configuration

To configure Dremio:

Note

There are limitations in the Dremio native Hive plugin because Dremio uses Ranger 1.1.0.

  • Audit Server basic auth needs to be disabled because it's not supported.

  • Dremio does not support solr audits in SSL if it is enabled in the audit server.

  1. Run the following commands:

    cd ~/privacera/privacera-manager
    cp config/sample-vars/vars.dremio.yml config/custom-vars/
    
  2. Update the following properties:

    AUDITSERVER_ENABLE: "true"
    AUDITSERVER_AUTH_TYPE: "none"
    AUDITSERVER_SSL_ENABLE: "false"
  3. Run the following commands to configure the audit server for Dremio Native Hive Ranger Based authorization..

    cd ~/privacera/privacera-manager 
    cp config/sample-vars/vars.auditserver.yml config/custom-vars/ 
    vi config/custom-vars/vars.auditserver.yml

    After the update is completed, the Dremio plugin installation script privacera_dremio.sh and custom configuration archive privacera_custom_conf.tar.gz is generated at the location ~/privacera/privacera-manager/output/dremio

  4. Configure Privacera plugin depending on how you have installed Dremio in your instance.

    For a new or existing data source configured in Dremio Data Lake, ensure Enable external authorization plugin checkbox under Settings > Advanced Options of the data source is selected in the Dremio UI.

  5. Restart the Dremio service.

Kubernetes

Depending on your cloud provider, you can set up Dremio in a Kubernetes container. For more information, see the following links.

After setting up Dremio, perform the following steps to deploy Privacera plugin. The steps assume that your Privacera Manager host instance is separate from your Dremio Kubernetes instance. If they are configured on the single instance, then modify the steps accordingly.

  1. SSH to your instance where Dremio is installed containing the Dremio Kubernetes artifacts and change to the dremio-cloud-tools/charts/dremio_v2/ directory.

  2. Copy the privacera_dremio.sh and privacera_custom_conf.tar.gz files from your Privacera Manager host instance to the dremio_v2 folder in your Dremio Kubernetes instance.

  3. Run the following commands:

    mkdir -p privacera_config 
    mv privacera_dremio.sh privacera_config/ 
    mv privacera_custom_conf.tar.gz privacera_config/
  4. Update configmap.yml to add new configmap for Privacera configuration.

    vi templates/dremio-configmap.yaml
  5. Add the following configuration at the start of the file:

    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: dremio-privacera-install
    data:
    privacera_dremio.sh: |- {{ .Files.Get "privacera_config/privacera_dremio.sh" | nindent 4 }}
    binaryData:
    privacera_custom_conf.tar.gz: {{ .Files.Get "privacera_config/privacera_custom_conf.tar.gz" | b64enc | nindent 4 }}
    ---
  6. Update dremio-env to add Privacera jars and configuration in the Dremio classpath.

    vi config/dremio-env
  7. Add the following variable, or update it if it already exists:

    DREMIO_EXTRA_CLASSPATH=/opt/privacera/conf:/opt/privacera/dremio-ext-jars/*
  8. Update values.yaml.

    vi values.yaml
            
  9. Add the following configuration for extraInitContainers inside the coordinator section:

    extraInitContainers:  |
        - name: install-privacera-dremio-plugin
        image: {{.Values.image}}:{{.Values.imageTag}}
        imagePullPolicy: IfNotPresent
        securityContext:
            runAsUser: 0
        volumeMounts:
        - name: dremio-privacera-plugin-volume
            mountPath: /opt/dremio/plugins/authorizer
        - name: dremio-ext-jars-volume
            mountPath: /opt/privacera/dremio-ext-jars
        - name: dremio-privacera-config
            mountPath: /opt/privacera/conf/
        - name: dremio-privacera-install
            mountPath: /opt/privacera/install/
        command:
            - "bash"
            - "-c"
            - "cd /opt/privacera/install/ && cp * /tmp/ && cd /tmp && ./privacera_dremio.sh"
  10. Update or uncomment the extraVolumes section inside the coordinator section and add the following configuration:

    extraVolumes:
    - name: dremio-privacera-install
        configMap:
        name: dremio-privacera-install
        defaultMode: 0777
    - name: dremio-privacera-plugin-volume
        emptyDir: {}
    - name: dremio-ext-jars-volume
        emptyDir: {}
    - name: dremio-privacera-config
        emptyDir: {}
  11. Update or uncomment the extraVolumeMounts section inside the coordinator section and add the following configuration:

    extraVolumeMounts:
    - name: dremio-ext-jars-volume
        mountPath: /opt/privacera/dremio-ext-jars
    - name: dremio-privacera-plugin-volume
        mountPath: /opt/dremio/plugins/authorizer
    - name: dremio-privacera-config
        mountPath: /opt/privacera/conf
  12. Upgrade your Helm release. Get the release name by running helm list command. The text under the Name column is your Helm release.

    helm upgrade -f values.yaml <release-name>
RPM

To deploy RPM:

  1. SSH to your instance where Dremio RPM is installed.

  2. Copy the privacera_dremio.sh and privacera_custom_conf.tar.gz files from your Privacera Manager host instance to the Home folder in your Dremio instance.

  3. Rum the following commands:

    mkdir -p ~/privacera/install
    mv privacera_dremio.sh ~/privacera/install
    mv privacera_custom_conf.tar.gz ~/privacera/install
  4. Launch the privacera_dremio.sh script.

    cd ~/privacera/install
    chmod +x privacera_dremio.sh
    sudo ./privacera_dremio.sh
  5. Update dremio-env to add Privacera jars and configuration in the Dremio classpath.

    vi ${DREMIO_HOME}/conf/dremio-env
  6. Add the following variable, or update it if it already exists:

    DREMIO_EXTRA_CLASSPATH=/opt/privacera/conf:/opt/privacera/dremio-ext-jars/*
  7. Restart Dremio.

    sudo service dremio restart