Skip to content

Table Properties#

Discovery#

Expand

Property Description Example
DISCOVERY_BUCKET_NAME Set the bucket name where Discovery will store its metadata files container1
Properties of Topic and Table names Topic and Table names are assigned by default in Privacera Discovery. To customize any topic or table name, refer to the link.

Expand

Property Description Example
DISCOVERY_ENABLE In the Basic tab, enable/disable Privacera Discovery.
DISCOVERY_REALTIME_ENABLE

In the Basic tab, enable/disable real-time scan in Privacera Discovery.

For real-time scan to work, ensure the following:

  • If you want to scan the default ADLS app registered by the system at the time of installation, keep its app properties unchanged in Privacera Portal.
  • If you want to scan a user-registered app, the app properties in Privacera Portal and its corresponding discovery.yml should be the same.
  • At a time, only one app can be scanned. 

DISCOVERY_FS_PREFIX Enter the container name. Get it from the Prerequisites section. container1
DISCOVERY_AZURE_STORAGE_ACCOUNT_NAME Enter the name of the Azure Storage account. Get it from the Prerequisites section. azurestorage
DISCOVERY_COSMOSDB_URL

DISCOVERY_COSMOSDB_KEY
Enter the Cosmos DB URL and Primary Key. Get it from the Prerequisites section. DISCOVERY_COSMOSDB_URL: "https://url1.documents.azure.com:443/"

DISCOVERY_COSMOSDB_KEY: "xavosdocof"
DISCOVERY_AZURE_STORAGE_ACCOUNT_KEY Enter the Access Key of the storage account. Get it from the Prerequisites section. GMi0xftgifp==
Properties of Topic and Table names Topic and Table names are assigned by default in Privacera Discovery. To customize any topic or table name, refer to the link.
PKAFKA_EVENT_HUB In the Advanced > Pkafka Configuration section, enter the Event Hub name. Get it from the Prerequisites section. eventhub1
PKAFKA_EVENT_HUB_NAMESPACE In the Advanced > Pkafka Configuration section, enter the name of the Event Hub namespace. Get it from the Prerequisites section. eventhubnamespace1
PKAFKA_EVENT_HUB_CONSUMER_GROUP In the Advanced > Pkafka Configuration section, enter the name of the Consumer Group. Get it from the Prerequisites section. congroup1
PKAFKA_EVENT_HUB_CONNECTION_STRING In the Advanced > Pkafka Configuration section, enter the connection string. Get it from the Prerequisites section. Endpoint=sb://eventhub1.servicebus.windows.net/;
SharedAccessKeyName=RootManageSharedAccessKey;
SharedAccessKey=sAmPLEP/8PytEsT=

Spark from Databricks#

Expand

Property Description Example
PRIVACERA_DISCOVERY_DATABRICKS_DOWNLOAD_URL Set the Discover Databricks download URL. See your Privacera Sales Account Manager for this information.  https://s3.amazonaws.com/privacera/release/hdp314/rel_x.x.x.x
DATABRICKS_DRIVER_INSTANCE_TYPE

For AWS driver's instance type can be "m5.xlarge" or "m5.2xlarge"

For Azure driver's instance type can be "Standard_DS3_v2"

m5.xlarge

DATABRICKS_INSTANCE_TYPE

For AWS driver's instance type can be "m5.xlarge" or "m5.2xlarge"

For Azure driver's instance type can be "Standard_DS3_v2"

m5.xlarge

SETUP_DATABRICKS_JAR
USE_DATABRICKS_SPARK
DATABRICKS_ELASTIC_DISK
DATABRICKS_DISCOVERY_MANAGE_INIT_SCRIPT Set to true if you want to create databricks init script. false
DATABRICKS_DISCOVERY_WORKERS
DATABRICKS_DISCOVERY_JOB_NAME
DATABRICKS_DISCOVERY_SPARK_VERSION

Spark version can be as follows:

  • 6.4.x-scala2.11 (Spark 2.4)
  • 7.3.x-scala2.12 (Spark 3.0)
  • 7.4.x-scala2.12 (Spark 3.0)
  • 7.5.x-scala2.12 (Spark 3.0)
  • 7.6.x-scala2.12 (Spark 3.0)
7.3.x-scala2.12
DATABRICKS_DISCOVERY_INSTANCE_PROFILE

Property is used for the instance role, for the Databricks instance node where your discovery will be running

arn:aws:iam::1234564835:instance-profile/privacera_databricks_cluster_iam_role

DISCOVERY_AWS_CLOUD_ASSUME_ROLE

Property to grant Discovery access to AWS services to perform the scanning operation.

true

DISCOVERY_AWS_CLOUD_ASSUME_ROLE_ARN

ARN of the AWS IAM Role 

arn:aws:iam::12345671758:role/DiscoveryCrossAccAssumeRole_k

Pkafka#

Expand

Property Description Example
PKAFKA_EVENT_HUB Enter the Event Hub name. Get it from the Prerequisites section above. eventhub1
PKAFKA_EVENT_HUB_NAMESPACE Enter the name of the Event Hub namespace. Get it from the Prerequisites section above. eventhubnamespace1
PKAFKA_EVENT_HUB_CONSUMER_GROUP Enter the name of the Consumer Group. Get it from the Prerequisites section above. congroup1
PKAFKA_EVENT_HUB_CONNECTION_STRING Enter the connection string. Get it from the Prerequisites section above. Endpoint=sb://eventhub1.servicebus.windows.net/;
SharedAccessKeyName=RootManageSharedAccessKey;
SharedAccessKey=sAmPLEP/8PytEsT=
DISCOVERY_REALTIME_ENABLE

Add this property to enable/disable real-time scan. By default, it is set to false.

Note: This is a custom property, and has to be added separately to the YAML file. 

For real-time scan to work, ensure the following:
  • If you want to scan the default ADLS app registered by the system at the time of installation, keep its app properties unchanged in Privacera Portal.
  • If you want to scan a user-registered app, the app properties in Privacera Portal and its corresponding discovery.yml should be the same.
  • At a time, only one app can be scanned. 
true


Last update: July 23, 2021