- PrivaceraCloud Release 7.4
- Enhancements and updates in PrivaceraCloud release 7.4
- Known Issues in PrivaceraCloud 7.4
- PrivaceraCloud User Guide
- Overview of PrivaceraCloud
- Connect applications with the setup wizard
- Connect applications
- About applications
- Connect Azure Data Lake Storage Gen 2 (ADLS) to PrivaceraCloud
- Connect Amazon Textract to PrivaceraCloud
- Athena
- Privacera Discovery with Cassandra
- Connect Databricks to PrivaceraCloud
- Databricks SQL
- Databricks SQL Overview and Configuration
- Planning and general process
- Prerequisites
- Databricks SQL with Privacera Hive
- Connect Databricks SQL application
- Grant Databricks SQL permissions to PrivaceraCloud users
- Define a resource policy
- Test the policy
- Databricks SQL PolicySync fields
- Configuring column-level access control
- View-based masking functions and row-level filtering
- Create an endpoint in Databricks SQL
- Databricks SQL Fields
- Databricks SQL Hive Service Definition
- Databricks SQL Masking Functions
- Databricks SQL Encryption
- Use a custom policy repository with Databricks
- Connect Databricks SQL to Hive policy repository on PrivaceraCloud
- Databricks SQL Overview and Configuration
- Connect Databricks Unity Catalog to PrivaceraCloud
- Connect S3 to PrivaceraCloud
- Prerequisites in AWS console
- Connect S3 application to PrivaceraCloud
- Enable Privacera Access Management for S3
- Enable Data Discovery for S3
- S3 AWS Commands - Ranger Permission Mapping
- S3
- AWS Access with IAM
- Access AWS S3 buckets from multiple AWS accounts
- Add UserInfo in S3 Requests sent via Dataserver
- Control access to S3 buckets with AWS Lambda function on PrivaceraCloud
- Dremio Plugin
- DynamoDB
- Connect Elastic MapReduce from Amazon application to PrivaceraCloud
- Connect EMR application
- EMR Spark access control types
- PrivaceraCloud configuration
- AWS IAM roles using CloudFormation setup
- Create a security configuration
- Create EMR cluster
- How to configure multiple JSON Web Tokens (JWTs) for EMR
- EMR Native Ranger Integration with PrivaceraCloud
- Connect EMRFS S3 to PrivaceraCloud
- Files
- GBQ
- Google Cloud Storage
- Connect Glue to PrivaceraCloud
- Google BigQuery for PolicySync
- Connect Kinesis to PrivaceraCloud
- Connect Lambda to PrivaceraCloud
- Microsoft SQL Server
- MySQL for Discovery
- Open Source Apache Spark
- Oracle for Discovery
- PostgreSQL
- Connect Power BI to PrivaceraCloud
- Presto
- Redshift
- Snowflake
- Starburst Enterprise with PrivaceraCloud
- Starburst Enterprise Presto
- Trino
- Connect users
- Data access Users, Groups, and Roles
- UserSync
- Portal user LDAP/AD
- Datasource
- Okta Setup for SAML-SSO
- Azure AD setup
- SCIM Server User-Provisioning
- User Management
- Identity
- Access Manager
- Access Manager
- Resource Policies
- Tag Policies
- Scheme Policies
- Service Explorer
- Reports
- Audit
- About data access users, groups, and roles resource policies
- Security zones
- Discovery
- Classifications via random sampling
- Privacera Discovery scan targets
- Propagate Privacera Discovery Tags to Ranger
- Enable offline scanning on Azure Data Lake Storage Gen 2 (ADLS)
- Enable Real-time Scanning of S3 Buckets
- Enable Real-time Scanning on Azure Data Lake Storage Gen 2 (ADLS)
- Enable Discovery Realtime Scanning Using IAM Role
- Encryption
- Overview of Privacera Encryption
- Encryption schemes
- Presentation schemes
- Masking schemes
- Create scheme policies
- Privacera-supplied encryption schemes for the Privacera API
- Privacera-supplied encryption schemes for the Bouncy Castle API
- API date input formats
- Deprecated encryption formats, algorithms, and scopes
- Privacera Encryption REST API
- PEG API endpoint
- PEG REST API encryption endpoints
- Prerequisites
- Common PEG REST API fields
- Construct the datalist for the /protect endpoint
- Deconstruct the response from the /unprotect endpoint
- Example data transformation with the /unprotect endpoint and presentation scheme
- Example PEG API endpoints
- Make encryption API calls on behalf of another user
- Privacera Encryption UDF for masking in Databricks on PrivaceraCloud
- Privacera Encryption UDFs for Trino on PrivaceraCloud
- Syntax of Privacera Encryption UDFs for Trino
- Prerequisites for installing Privacera Crypto plug-in for Trino
- Download and install Privacera Crypto jar
- Set variables in Trino etc/crypto.properties
- Restart Trino to register the Privacera encryption and masking UDFs for Trino
- Example queries to verify Privacera-supplied UDFs
- Privacera Encryption UDF for masking in Trino on PrivaceraCloud
- Encryption UDFs for Apache Spark on PrivaceraCloud
- Launch Pad
- Settings
- Dashboard
- Usage statistics
- Operational status of PrivaceraCloud and RSS feed
- How to Get Support
- Coordinated Vulnerability Disclosure (CVD) Program of Privacera
- Shared Security Model
- PrivaceraCloud Previews
- Preview: File Explorer for S3
- Preview: File Explorer for Azure
- Preview: File Explorer for GCS
- Preview: Scan Generic Records with NER Model
- Preview: Scan Electronic Health Records with NER Model
- Preview: OneLogin setup for SAML-SSO
- Preview: Azure Active Directory SCIM Server UserSync
- Preview: OneLogin UserSync
- Preview: PingFederate UserSync
- Quickstart for Databricks Unity Catalog on PrivaceraCloud
- What do I need to do in my Databricks Workspace?
- Where is the sample dataset in my Databricks Workspace?
- What should I do in the PrivaceraCloud web portal?
- Access use-case - How do I give a user access to a table or restrict from running a SQL select query?
- Access use-case - How do I restrict a user from seeing contents of a column in the result of a SQL select query?
- Column masking use-case - How do I restrict a user from seeing contents of a column by masking the values in the result of a SQL select query?
- Access use-case - How do I disallow a user from seeing certain rows of a table?
- PrivaceraCloud documentation changelog
Open Source Apache Spark
You first obtain an account-specific script from your PrivaceraCloud account, followed by adding a startup step to open source Spark.
Three configurations are available depending on your requirement. Fine-Grained Access Control (FGAC) and Object-Level Access Control (OLAC) are supported in each of the configurations:
Obtain installation script
Obtain the account unique <privacera-plugin-script-download-url>
. This script and other commands run in your Spark command shell to complete the PrivaceraCloud installation.
Steps:
Go to Settings > API Key.
Use an existing active API Key or generate a new one.
Note
Make sure the Expiry column is set to Never Expires.
Click the i icon to get the scripts.
On the Plugins Setup Script, click the COPY URL button. Save this value on your Spark server. It is needed as the
<privacera-plugin-script-download-url>
in the next step.
Configure Privacera Plugin on local/virtual machine
OLAC Setup
OLAC is supported only with JWT token authentication.
See Data access methods.
Add the following properties in your Dataserver application to enable JWT authorization. In the following code block, 0 is the index. By increasing the index, you can add multiple JWT properties.
privacera.jwt.oauth.enable=true privacera.jwt.0.token.issuer=<PLEASE_CHANGE> privacera.jwt.0.token.subject=<PLEASE_CHANGE> privacera.jwt.0.token.secret=<PLEASE_CHANGE> privacera.jwt.0.token.publickey=<PLEASE_CHANGE> privacera.jwt.0.token.userKey=<PLEASE_CHANGE> privacera.jwt.0.token.groupKey=<PLEASE_CHANGE> privacera.jwt.0.token.parserType=<PLEASE_CHANGE>
Property
Description
Example
privacera.jwt.oauth.enable
Property to enable JWT auth in Privacera services.
true
privacera.jwt.{index}.token.issuer
Property to enter the URL of the identity provider.
https://you-idp-domain.com
privacera.jwt.{index}.token.publickey
The JWT token public key in String format (Need to delete all newlines).
-----BEGIN PUBLIC KEY-----MIIBIjANB-----END PUBLIC KEY-----
privacera.jwt.{index}.token.secret
[Optional] Add this If the JWT token has been encrypted using secret, use this property to set the secret.
privacera-api
privacera.jwt.{index}.token.subject
[Optional] Add this If JWT Token has a subject.
api-token
privacera.jwt.{index}.token.userKey
Property to define a unique userKey whose value will be used in user for Ranger policies.
client-id
privacera.jwt.{index}.token.groupKey
Property to define a unique groupKey whose value will be used in group for Ranger policies.
scope
privacera.jwt.{index}.token.parser.type
JWT Parser Type. Values can be PING_IDENTITY or KEYCLOAK.
PING_IDENTITY: When groupKey is an array
KEYCLOAK: When groupKey is space separator
KEYCLOAK
After adding the properties, run the Dataserver, and then proceed to the next step.
SSH to the instance where Spark is installed and you want to install Privacera Plugin.
Create a directory
~/privacera
and download the script. Replace<privacera-plugin-script-download-url>
with the Privacera Plugin download URL.mkdir ~/privacera/spark-plugin-install cd ~/privacera/spark-plugin-install wget <privacera-plugin-script-download-url> -O privacera_plugin.sh
Create a file
privacera_env.sh
that contains the parameters required for your plugin installation:vi privacera_env.sh
Add the following properties:
PLUGIN_TYPE="spark" SPARK_PLUGIN_TYPE="OLAC" SPARK_HOME="<PLEASE_CHANGE>" SPARK_CLUSTER_NAME="privacera-spark"
Property
Description
PLUGIN_TYPE
Type of Privacera Plugin which you want to install.
SPARK_PLUGIN_TYPE
Spark Plugin type OLAC. JWT Authentication will be enabled by default.
SPARK_HOME
This is the home directory of your Spark installation. For example, the directory path can be
/home/user/spark
.SPARK_CLUSTER_NAME
Cluster Name which will show up in the Privacera Ranger Audits page.
Run the script.
chmod +x privacera_plugin.sh ./privacera_plugin.sh
The script sets up the Privacera Plugin in the OLAC mode.
FGAC Setup
FGAC is recommended to be used with JWT authentication enabled.
Note
If JWT authentication is disabled, access control will fail on the system user or proxy user.
SSH to the instance where Spark is installed and you want to install Privacera Plugin.
Create a directory
~/privacera
and download the script. Replace<privacera-plugin-script-download-url>
with the Privacera Plugin download URL.mkdir ~/privacera/spark-plugin-install cd ~/privacera/spark-plugin-install wget <privacera-plugin-script-download-url> -O privacera_plugin.sh
Create a file
privacera_env.sh
which will contain the parameters required for your plugin installation.vi privacera_env.sh
Add the following properties:
PLUGIN_TYPE="spark" SPARK_PLUGIN_TYPE="FGAC" SPARK_HOME="<PLEASE_CHANGE>" SPARK_CLUSTER_NAME="privacera-spark"
Property
Description
PLUGIN_TYPE
Type of Privacera Plugin which you want to install.
SPARK_PLUGIN_TYPE
Spark Plugin type FGAC.
SPARK_HOME
This is the home directory of your Spark installation. For example, the directory path can be
/home/user/spark
.SPARK_CLUSTER_NAME
Cluster Name which will show up in the Privacera Ranger Audits page.
Add the following properties when JWT auth is enabled:
JWT_OAUTH_ENABLE="true" JWT_ISSUER="<PLEASE_CHANGE>" JWT_PUBLIC_KEY="<PLEASE_CHANGE>" #JWT_SECRET="<PLEASE_CHANGE>" #JWT_SUBJECT="<PLEASE_CHANGE>" JWT_USERKEY="<PLEASE_CHANGE>" JWT_GROUPKEY="<PLEASE_CHANGE>" JWT_PARSER_TYPE="<PLEASE_CHANGE>"
Note
To configure multiple JWTs, refer to FGAC with multiple JWT configurations below.
Property
Description
Example
JWT_OAUTH_ENABLE
To enable JWT authentication.
JWT_OAUTH_ENABLE="true"
JWT_ISSUER
The URL of the identity provider.
JWT_ISSUER="https://your-idp-domain.com"
JWT_PUBLIC_KEY
The JWT token public key in String format.
JWT_SECRET
Uncomment and add value if the JWT token has been encrypted using secret.
JWT_SECRET="privacera-secret"
JWT_SUBJECT
Uncomment and add value if JWT Token has a subject.
JWT_SUBJECT="api-token"
JWT_USERKEY
Property to define a unique userKey whose value will be used in user for Ranger policies.
JWT_USERKEY="client_id"
JWT_GROUPKEY
Property to define a unique groupKey whose value will be used in group for Ranger policies.
JWT_GROUPKEY="scope"
JWT_PARSER_TYPE
JWT Parser Type. Values can be PING_IDENTITY or KEYCLOAK.
JWT_PARSER_TYPE="KEYCLOAK"
Run the script.
chmod +x privacera_plugin.sh ./privacera_plugin.sh
The script will set up the Privacera Plugin in the FGAC mode.
FGAC with multiple JWT configurations
To configure multiple JWT configurations add the below index based properties in the privacera_env.sh
file. In which {index} start from 0 to n.
JWT_OAUTH_ENABLE="true" JWT_{index}_ISSUER="<PLEASE_CHANGE>" JWT_{index}_PUBLICKEY="<PLEASE_CHANGE>" JWT_{index}_SUBJECT="<PLEASE_CHANGE>" JWT_{index}_SECRET="<PLEASE_CHANGE>" JWT_{index}_USERKEY="<PLEASE_CHANGE>" JWT_{index}_GROUPKEY="<PLEASE_CHANGE>" JWT_{index}_PARSER_TYPE="<PLEASE_CHANGE>"
For example, for two configurations: (starts at 0)
JWT_OAUTH_ENABLE="true" JWT_0_ISSUER="https://mydomain.com/issuer" JWT_0_PUBLICKEY="-----BEGIN PUBLIC KEY-----MIIBIjANXXXXXDAQAB-----END PUBLIC KEY-----" JWT_0_SUBJECT=”principal1” JWT_0_SECRET=”shkl-XXXX-XXXX-XXXX” JWT_0_USERKEY="client_id" JWT_0_GROUPKEY="scope" JWT_0_PARSER_TYPE="PING_IDENTITY" JWT_1_ISSUER="https://mydomain.com/issuer" JWT_1_PUBLICKEY="-----BEGIN PUBLIC KEY-----MIIBIjANXXXXXDAQAB-----END PUBLIC KEY-----" JWT_1_SUBJECT=”principal2” JWT_1_SECRET=”suhjk-XXXX-XXXX-XXXX” JWT_1_USERKEY="client_id" JWT_1_GROUPKEY="scope" JWT_1_PARSER_TYPE="KEYCLOAK"
Configure Privacera Plugin in an Existing Docker File
If you have an existing Open Source Spark setup running on Kubernetes, you can update your existing Docker file used to create Spark image to add steps for installing Privacera Plugin.
OLAC Setup
OLAC is supported only with JWT token authentication.
Your Dataserver application should be configured with JWT Token support. Create a new Dataserver, if it does not exist.
See Data access methods.
Add the following properties in your Dataserver application to enable JWT authorization. In the following code block, 0 is the index. By increasing the index, you can add multiple JWT properties.
privacera.jwt.oauth.enable=true privacera.jwt.0.token.issuer=<PLEASE_CHANGE> privacera.jwt.0.token.subject=<PLEASE_CHANGE> privacera.jwt.0.token.secret=<PLEASE_CHANGE> privacera.jwt.0.token.publickey=<PLEASE_CHANGE> privacera.jwt.0.token.userKey=<PLEASE_CHANGE> privacera.jwt.0.token.groupKey=<PLEASE_CHANGE> privacera.jwt.0.token.parserType=<PLEASE_CHANGE>
Property
Description
Example
privacera.jwt.oauth.enable
Property to enable JWT auth in Privacera services.
true
privacera.jwt.{index}.token.issuer
Property to enter the URL of the identity provider.
https://you-idp-domain.com
privacera.jwt.{index}.token.publickey
The JWT token public key in String format (Need to delete all newlines).
-----BEGIN PUBLIC KEY-----MIIBIjANB-----END PUBLIC KEY-----
privacera.jwt.{index}.token.secret
[Optional] Add this If the JWT token has been encrypted using secret, use this property to set the secret.
privacera-api
privacera.jwt.{index}.token.subject
[Optional] Add this If JWT Token has a subject.
api-token
privacera.jwt.{index}.token.userKey
Property to define a unique userKey whose value will be used in user for Ranger policies.
client-id
privacera.jwt.{index}.token.groupKey
Property to define a unique groupKey whose value will be used in group for Ranger policies.
scope
privacera.jwt.{index}.token.parser.type
JWT Parser Type. Values can be PING_IDENTITY or KEYCLOAK.
PING_IDENTITY: When groupKey is an array
KEYCLOAK: When groupKey is space separator
KEYCLOAK
After adding the properties, run the Dataserver, and then proceed to the next step.
SSH to the instance where Spark is installed and you want to install Privacera Plugin.
Copy the following to your Docker file. Set the
PCLOUD_PLUGIN_SCRIPT_DOWNLOAD_URL
property.######## Install Privacera Spark Plugin Start ########### # ENV SPARK_HOME /opt/apache/spark RUN apt-get -y install zip unzip wget ENV PCLOUD_PLUGIN_SCRIPT_DOWNLOAD_URL="<PLEASE_CHANGE>" ENV PLUGIN_TYPE="spark" ENV SPARK_PLUGIN_TYPE="OLAC" ENV SPARK_CLUSTER_NAME="privacera-spark" RUN echo "Downloading Script from $PCLOUD_PLUGIN_SCRIPT_DOWNLOAD_URL" RUN wget ${PCLOUD_PLUGIN_SCRIPT_DOWNLOAD_URL} -O privacera_plugin.sh RUN chmod +x privacera_plugin.sh RUN ./privacera_plugin.sh ######## Install Privacera Spark Plugin End ###########
Save the Docker file and build the image. You will now have a Docker image for Open Source Spark With Privacera Plugin enabled.
FGAC Setup
FGAC is recommended to be used with JWT authentication enabled.
Note
If JWT authentication is disabled, access control will fail on the system user or proxy user.
SSH to the instance where Spark is installed and you want to install Privacera Plugin.
Copy the following to your Docker file. Set the
PCLOUD_PLUGIN_SCRIPT_DOWNLOAD_URL
property. For the JWT properties, refer the table below.######## Install Privacera Spark Plugin Start ########### # ENV SPARK_HOME /opt/apache/spark RUN apt-get -y install zip unzip wget ENV PCLOUD_PLUGIN_SCRIPT_DOWNLOAD_URL="<PLEASE_CHANGE>" ENV PLUGIN_TYPE="spark" ENV SPARK_PLUGIN_TYPE="FGAC" ENV SPARK_CLUSTER_NAME="privacera-spark" ENV JWT_OAUTH_ENABLE "true" ENV JWT_ISSUER=<PLEASE_CHANGE> ENV JWT_PUBLIC_KEY=<PLEASE_CHANGE> ENV JWT_SECRET=<PLEASE_CHANGE> ENV JWT_SUBJECT=<PLEASE_CHANGE> ENV JWT_USERKEY=<PLEASE_CHANGE> ENV JWT_GROUPKEY=<PLEASE_CHANGE> ENV JWT_PARSER_TYPE=<PLEASE_CHANGE> RUN echo "Downloading Script from $PCLOUD_PLUGIN_SCRIPT_DOWNLOAD_URL" RUN wget ${PCLOUD_PLUGIN_SCRIPT_DOWNLOAD_URL} -O privacera_plugin.sh RUN chmod +x privacera_plugin.sh RUN ./privacera_plugin.sh ######## Install Privacera Spark Plugin End ###########
Note
To configure multiple JWTs, refer to FGAC with Multiple JWT Configuration in an Existing Docker File below.
Property
Description
Example
JWT_OAUTH_ENABLE
To enable JWT authentication.
JWT_OAUTH_ENABLE="true"
JWT_ISSUER
The URL of the identity provider.
JWT_ISSUER="https://your-idp-domain.com"
JWT_PUBLIC_KEY
The JWT token public key in String format.
JWT_SECRET
Uncomment and add value if the JWT token has been encrypted using secret.
JWT_SECRET="privacera-secret"
JWT_SUBJECT
Uncomment and add value if JWT Token has a subject.
JWT_SUBJECT="api-token"
JWT_USERKEY
Property to define a unique userKey whose value will be used in user for Ranger policies.
JWT_USERKEY="client_id"
JWT_GROUPKEY
Property to define a unique groupKey whose value will be used in group for Ranger policies.
JWT_GROUPKEY="scope"
JWT_PARSER_TYPE
JWT Parser Type. Values can be PING_IDENTITY or KEYCLOAK.
JWT_PARSER_TYPE="KEYCLOAK"
Save the Docker file and build the image. You will now have a Docker image for Open Source Spark With Privacera Plugin enabled.
FGAC with Multiple JWT Configuration in an Existing Docker File
To configure multiple JWT configurations add the below index based Environment variable in the Docker file. In which {index} start from 0 to n.
ENV JWT_OAUTH_ENABLE "true" ENV JWT_{index}_ISSUER="<PLEASE_CHANGE>" ENV JWT_{index}_PUBLICKEY="<PLEASE_CHANGE>" ENV JWT_{index}_SUBJECT="<PLEASE_CHANGE>" ENV JWT_{index}_SECRET="<PLEASE_CHANGE>" ENV JWT_{index}_USERKEY="<PLEASE_CHANGE>" ENV JWT_{index}_GROUPKEY="<PLEASE_CHANGE>" ENV JWT_{index}_PARSER_TYPE="<PLEASE_CHANGE>"
For example, for two configurations: (starts at 0)
######## Install Privacera Spark Plugin Start ############ ENV SPARK_HOME /opt/apache/spark RUN apt-get -y install zip unzip wget ENV PCLOUD_PLUGIN_SCRIPT_DOWNLOAD_URL="<PLEASE_CHANGE>" ENV PLUGIN_TYPE="spark" ENV SPARK_PLUGIN_TYPE="FGAC" ENV SPARK_CLUSTER_NAME="privacera-spark" ENV JWT_OAUTH_ENABLE "true" ENV JWT_0_ISSUER="https://mydomain.com/issuer" ENV JWT_0_PUBLICKEY="-----BEGIN PUBLIC KEY-----MIIBIjANXXXXXDAQAB-----END PUBLIC KEY-----" ENV JWT_0_SUBJECT=”principal1” ENV JWT_0_SECRET=”shkl-XXXX-XXXX-XXXX” ENV JWT_0_USERKEY="client_id" ENV JWT_0_GROUPKEY="scope" ENV JWT_0_PARSER_TYPE="PING_IDENTITY" ENV JWT_1_ISSUER="https://mydomain.com/issuer" ENV JWT_1_PUBLICKEY="-----BEGIN PUBLIC KEY-----MIIBIjANXXXXXDAQAB-----END PUBLIC KEY-----" ENV JWT_1_SUBJECT=”principal2” ENV JWT_1_SECRET=”suhjk-XXXX-XXXX-XXXX” ENV JWT_1_USERKEY="client_id" ENV JWT_1_GROUPKEY="scope" ENV JWT_1_PARSER_TYPE="KEYCLOAK"
Configure Privacera Plugin using Privacera Scripts
The scripts will help you create an Open Source Spark image with Privacera Plugin and push it to the specified Docker Hub which can be used to run Spark with Privacera.
OLAC Setup
OLAC is supported only with JWT token authentication.
Your Dataserver application should be configured with JWT Token support. Create a new Dataserver, if it does not exist.
See Data access methods.
Add the following properties in your Dataserver application to enable JWT authorization. In the following code block, 0 is the index. By increasing the index, you can add multiple JWT properties.
privacera.jwt.oauth.enable=true privacera.jwt.0.token.issuer=<PLEASE_CHANGE> privacera.jwt.0.token.subject=<PLEASE_CHANGE> privacera.jwt.0.token.secret=<PLEASE_CHANGE> privacera.jwt.0.token.publickey=<PLEASE_CHANGE> privacera.jwt.0.token.userKey=<PLEASE_CHANGE> privacera.jwt.0.token.groupKey=<PLEASE_CHANGE> privacera.jwt.0.token.parserType=<PLEASE_CHANGE>
Property
Description
Example
privacera.jwt.oauth.enable
Property to enable JWT auth in Privacera services.
true
privacera.jwt.{index}.token.issuer
Property to enter the URL of the identity provider.
https://you-idp-domain.com
privacera.jwt.{index}.token.publickey
The JWT token public key in String format (Need to delete all newlines).
-----BEGIN PUBLIC KEY-----MIIBIjANB-----END PUBLIC KEY-----
privacera.jwt.{index}.token.secret
[Optional] Add this If the JWT token has been encrypted using secret, use this property to set the secret.
privacera-api
privacera.jwt.{index}.token.subject
[Optional] Add this If JWT Token has a subject.
api-token
privacera.jwt.{index}.token.userKey
Property to define a unique userKey whose value will be used in user for Ranger policies.
client-id
privacera.jwt.{index}.token.groupKey
Property to define a unique groupKey whose value will be used in group for Ranger policies.
scope
privacera.jwt.{index}.token.parser.type
JWT Parser Type. Values can be PING_IDENTITY or KEYCLOAK.
PING_IDENTITY: When groupKey is an array
KEYCLOAK: When groupKey is space separator
privacera.jwt.token.parser.type=KEYCLOAK
After adding the properties, run the Dataserver, and then proceed to the next step.
SSH to the instance where you want to install Privacera Plugin.
Create a directory
~/privacera
and download the script. Replace<privacera-plugin-script-download-url>
with the Privacera Plugin download URL.mkdir ~/privacera/spark-plugin-install cd ~/privacera/spark-plugin-install wget <privacera-plugin-script-download-url> -O privacera_plugin.sh
Create a file
privacera_env.sh
which will contain the parameters required for your plugin installation.vi privacera_env.sh
Add the following properties:
PLUGIN_TYPE="spark_k8s" SPARK_VERSION="3.3.0" SPARK_HOME="/opt/privacera/spark" SPARK_PLUGIN_TYPE="OLAC" HUB="<PLEASE_CHANGE>" HUB_USERNAME="<PLEASE_CHANGE>" HUB_PASSWORD="<PLEASE_CHANGE>" ENV_TAG="<PLEASE_CHANGE>"
Property
Description
PLUGIN_TYPE
Type of Privacera Plugin which you want to install.
SPARK_PLUGIN_TYPE
Spark Plugin type OLAC. JWT Authentication will be enabled by default.
SPARK_VERSION
Specifies the version of Apache Spark. Must be one of the following versions:
3.1.2
,3.2.2
, or3.3.0
SPARK_HOME
This is the home directory of your Spark installation. For example, the directory path can be
/opt/privacera/spark
.HUB
The Docker hub URL where you want the image to be pushed.
HUB_USERNAME
Docker hub username.
HUB_PASSWORD
Docker hub password.
ENV_TAG
Docker image tag.
Run the script.
chmod +x privacera_plugin.sh ./privacera_plugin.sh
The script will build the Spark image with Privacera Spark plugin and publish it to the Docker hub.
FGAC Setup
FGAC is recommended to be used with JWT authentication enabled.
Note
If JWT authentication is disabled, access control will fail on the system user or proxy user.
SSH to the instance where you want to install Privacera Plugin.
Create a directory
~/privacera
and download the script. Replace<privacera-plugin-script-download-url>
with the Privacera Plugin download URL.mkdir ~/privacera/spark-plugin-install cd ~/privacera/spark-plugin-install wget <privacera-plugin-script-download-url> -O privacera_plugin.sh
Create a file
privacera_env.sh
which will contain the parameters required for your plugin installation.vi privacera_env.sh
Add the following properties:
PLUGIN_TYPE="spark_k8s" SPARK_VERSION="3.3.0" SPARK_HOME="/opt/privacera/spark" SPARK_PLUGIN_TYPE="FGAC" SPARK_CLUSTER_NAME="privacera-spark"
Property
Description
PLUGIN_TYPE
Type of Privacera Plugin which you want to install.
SPARK_PLUGIN_TYPE
Spark Plugin type FGAC.
SPARK_VERSION
Specifies the version of Apache Spark. Must be one of the following versions:
3.1.2
,3.2.2
, or3.3.0
SPARK_HOME
This is the home directory of your Spark installation. For example, the directory path can be
/opt/privacera/spark
.SPARK_CLUSTER_NAME
Cluster Name which will show up in the Privacera Ranger Audits page.
Add the following properties when JWT auth is enabled:
JWT_OAUTH_ENABLE="true" JWT_ISSUER="<PLEASE_CHANGE>" JWT_PUBLIC_KEY="<PLEASE_CHANGE>" #JWT_SECRET="<PLEASE_CHANGE>" #JWT_SUBJECT="<PLEASE_CHANGE>" JWT_USERKEY="<PLEASE_CHANGE>" JWT_GROUPKEY="<PLEASE_CHANGE>" JWT_PARSER_TYPE="<PLEASE_CHANGE>"
Property
Description
Example
JWT_OAUTH_ENABLE
To enable JWT authentication.
JWT_OAUTH_ENABLE="true"
JWT_ISSUER
The URL of the identity provider.
JWT_ISSUER="https://your-idp-domain.com"
JWT_PUBLIC_KEY
The JWT token public key in String format.
JWT_SECRET
Uncomment and add value if the JWT token has been encrypted using secret.
JWT_SECRET="privacera-secret"
JWT_SUBJECT
Uncomment and add value if JWT Token has a subject.
JWT_SUBJECT="api-token"
JWT_USERKEY
Property to define a unique userKey whose value will be used in user for Ranger policies.
JWT_USERKEY="client_id"
JWT_GROUPKEY
Property to define a unique groupKey whose value will be used in group for Ranger policies.
JWT_GROUPKEY="scope"
JWT_PARSER_TYPE
JWT Parser Type. Values can be PING_IDENTITY or KEYCLOAK.
JWT_PARSER_TYPE="KEYCLOAK"
Add the following Docker Hub properties:
HUB="<PLEASE_CHANGE>" HUB_USERNAME="<PLEASE_CHANGE>" HUB_PASSWORD="<PLEASE_CHANGE>" ENV_TAG="<PLEASE_CHANGE>"
Property
Description
HUB
The Docker hub URL where you want the image to be pushed.
HUB_USERNAME
Docker hub username.
HUB_PASSWORD
Docker hub password.
ENV_TAG
Docker image tag.
Run the script.
chmod +x privacera_plugin.sh ./privacera_plugin.sh
The script will build the Spark image with Privacera Spark plugin and publish it to the Docker hub.
Deploy Spark on EKS Cluster
SSH to the instance where you want to deploy Spark on the EKS cluster.
Get the Privacera Plugin download URL and set it in the following property. See Obtain installation script.
export PRIVACERA_DOWNLOAD_URL="<PLEASE_CHANGE>"
Create
spark-k8s-artifacts
folder.mkdir ~/privacera/spark-k8s-artifacts cd ~/privacera/spark-k8s-artifacts
Download and extract packages.
wget ${PRIVACERA_DOWNLOAD_URL}/plugin/spark/k8s-spark-deploy.tar.gz -O k8s-spark-deploy.tar.gz tar xzf k8s-spark-deploy.tar.gz rm -r k8s-spark-deploy.tar.gz cd k8s-spark-deploy/
Open
penv.sh
file and substitute the values of the following properties. Refer to the table below:Property
Description
Example
SPARK_NAME_SPACE
Kubernetes namespace
privacera-spark-plugin-test
SPARK_PLUGIN_IMAGE
Docker image with hub
${HUB}/privacera-spark-plugin:${ENV_TAG}
SPARK_DOCKER_PULL_SECRET
Secret for docker-registry
spark-plugin-docker-hub
SPARK_PLUGIN_ROLE_BINDING
Spark role Binding
privacera-sa-spark-plugin-role-binding
SPARK_PLUGIN_SERVICE_ACCOUNT
Spark services account
privacera-sa-spark-plugin
SPARK_PLUGN_ROLE
Spark services account role
privacera-sa-spark-plugin-role
SPARK_PLUGIN_APP_NAME
Spark plugin application name
privacera-spark-examples
Run the following command to replace the property values in EKS deployment YAML file.
mkdir -p backup cp *.yml backup/ ./replace.sh
Run the following command to create EKS resources.
kubectl apply -f namespace.yml kubectl apply -f service-account.yml kubectl apply -f role.yml kubectl apply -f role-binding.yml
Run the following command to create secret for
docker-registry
.kubectl create secret docker-registry spark-plugin-docker-hub --docker-server=<PLEASE_CHANGE> --docker-username=<PLEASE_CHANGE> --docker-password='<PLEASE_CHANGE>' --namespace=<PLEASE_CHANGE>
Run the following command to deploy a sample Spark application. Replace
${SPARK_NAME_SPACE}
with the Kubernetes namespace.kubectl apply -f privacera-spark-examples.yml -n ${SPARK_NAME_SPACE}
Note
This is a sample file used for deployment. As per your use case, you can create a Spark deployment file and deploy a Docker image.
This will deploy a Spark application in EKS pod with Privacera plugin and it will keep the pod running, so that you can use it in interactive mode.