Skip to main content

Privacera Documentation

Add custom Spark configuration for Databricks on Privacera Platform

To add custom properties in the Databricks cluster init script, you need to create a custom configuration file and add your custom spark properties to it.

To do so, follow these steps:

  1. Run the following commands to create a custom configuration file.

    cd ~/privacera/privacera-manager
vi config/custom-properties/databricks-spark.conf

  2. Add the properties one after the other, and then save the file. For example, you can add the following property and save the file:

    "spark.databricks.delta.formatCheck.enabled"="false"

    Note

    Avoid putting comments, extra words, or blank lines in the config file.

  3. Optional: To use a Databricks workspace file, set the following property to true.

    DATABRICKS_INIT_SCRIPT_WORKSPACE_FLAG_ENABLE: "true"

  4. Run the following command.

    cd ~/privacera/privacera-manager
./privacera-manager.sh update

Configure Databricks cluster policy

  1. Add the following properties to vars.databricks.plugin.yml:

    • DATABRICKS_SQL_CLUSTER_POLICY_SPARK_CONF

    • DATABRICKS_SCALA_CLUSTER_POLICY_SPARK_CONF

    • DATABRICKS_INIT_SCRIPT_WORKSPACE_FLAG_ENABLE. Set true to upload init script into workspace files at the location specified in the script /privacera/<deployment_name>/ranger_enable.sh, where <deployment_name> is Prod, Dev, or whatever you have set the deployment name to. By default, this property is set to false.

    For example:

    DATABRICKkeS_SQL_CLUSTER_POLICY_SPARK_CONF:
- Note: first spark conf
    key : "spark.hadoop.first.spark.test"
    value: "test1"
- Note: second spark first spark conf
    key: "spark.hadoop.first.spark.test2"
    value: "test2"
DATABRICKS_SCALA_CLUSTER_POLICY_SPARK_CONF:
- Note: first spark conf
    key : "spark.hadoop.first.spark.test"
    value: "test1"
- Note: second spark first spark conf
    key: "spark.hadoop.first.spark.test2"
    value: "test2"
DATABRICKS_INIT_SCRIPT_WORKSPACE_FLAG_ENABLE
- Note: Load Privacera init script into Databricks workspaces
    key: ???????????????
    value: true

  2. To add custom properties with Java agent, add the following property to vars.databricks.plugin.yml:

    • DATABRICKS_SPARK_PLUGIN_AGENT_JAR

    For example:

    DATABRICKS_SPARK_PLUGIN_AGENT_JAR: " -Dmy.custom.propery=test -javaagent:/databricks/jars/privacera-agent.jar"