Skip to main content

Privacera Documentation

Enabling Multithreading for Different Consumers

For enabling multithreading for different consumers in the Discovery driver pod or Discovery consumer pod, refer to Configure system properties and follow these steps:

Note

This feature is supported only for AWS and Azure Kubernetes Spark.

  1. For the Discovery driver, create the property file discovery-custom.properties.

  2. For the Discovery consumer, create the property file discovery-consumer-custom.properties.

  3. Add all of the following properties in both of the above files.

    Note

    The values in the following properties are recommended values.

AWS Properties

#privacera_offline_scan_topic privacera.discovery.cloud.consumer.config.offline.scan.summary.max.poll.records=1
privacera.discovery.cloud.consumer.config.offline.scan.max.poll.records=1

#this is the timeout for offline scan job for each batch file
privacera.discovery.cloud.consumer.config.offline.scan.summary.task.timeout.ms=172800000
privacera.discovery.cloud.consumer.config.offline.scan.task.timeout.ms=172800000

#privacera_scan_resource_info_topic privacera.discovery.cloud.consumer.config.ow.solr.scan.resource.info.max.poll.records=10000
privacera.discovery.cloud.consumer.config.ow.solr.resource.max.poll.records=10000
privacera.discovery.cloud.consumer.config.ow.solr.scan.resource.meta.max.poll.records=10000

privacera.discovery.cloud.consumer.config.ow.solr.scan.resource.info.task.timeout.ms=172800000
privacera.discovery.cloud.consumer.config.ow.solr.resource.task.timeout.ms=172800000
privacera.discovery.cloud.consumer.config.ow.solr.scan.resource.meta.task.timeout.ms=172800000

privacera.discovery.cloud.consumer.config.ow.solr.scan.resource.info.parallel.size=50
privacera.discovery.cloud.consumer.config.ow.solr.scan.resource.meta.parallel.size=50
privacera.discovery.cloud.consumer.config.ow.solr.resource.parallel.size=50

#privacera_classification_topic privacera.discovery.cloud.consumer.config.ow.solr.classifications.max.poll.records=10000
privacera.discovery.cloud.consumer.config.ow.resource.workflow.max.poll.records=10000
privacera.discovery.cloud.consumer.ow.ranger.rest.classifications.max.poll.records=10000

privacera.discovery.cloud.consumer.config.ow.solr.classifications.task.timeout.ms=86400000
privacera.discovery.cloud.consumer.config.ow.resource.workflow.task.timeout.ms=86400000
privacera.discovery.cloud.consumer.ow.ranger.rest.classifications.task.timeout.ms=86400000

privacera.discovery.cloud.consumer.config.ow.solr.classifications.parallel.size=50
privacera.discovery.cloud.consumer.ow.ranger.rest.classifications.parallel.size=50

Azure Properties

OFFLINE SCAN
privacera.discovery.kafka.consumer.config.offline.scan.max.poll.records=1
privacera.discovery.kafka.consumer.config.offline.scan.task.timeout.ms=432000000

OFFLINE SCAN SUMMARY
privacera.discovery.kafka.consumer.config.offline.scan.summary.max.poll.records=1
privacera.discovery.kafka.consumer.config.offline.scan.summary.task.timeout.ms=432000000

OUTPUT WRITER - SOLR CLASSIFICATION
privacera.discovery.kafka.consumer.config.ow.solr.classifications.max.poll.records=10000
privacera.discovery.kafka.consumer.config.ow.solr.classifications.parallel.size=50
privacera.discovery.kafka.consumer.config.ow.solr.classifications.task.timeout.ms=86400000

OUTPUT WRITER - SOLR SCAN RESOURCE INFO
privacera.discovery.kafka.consumer.config.ow.solr.scan.resource.info.max.poll.records=10000
privacera.discovery.kafka.consumer.config.ow.solr.scan.resource.info.parallel.size=50
privacera.discovery.kafka.consumer.config.ow.solr.scan.resource.info.task.timeout.ms=432000000

OUTPUT WRITER - SOLR SCAN RESOURCE META
privacera.discovery.kafka.consumer.config.ow.solr.scan.resource.meta.max.poll.records=10000
privacera.discovery.kafka.consumer.config.ow.solr.scan.resource.meta.parallel.size=50
privacera.discovery.kafka.consumer.config.ow.solr.scan.resource.meta.task.timeout.ms=432000000

OUTPUT WRITER - SOLR SCAN RESOURCE
privacera.discovery.kafka.consumer.config.ow.solr.resource.max.poll.records=10000
privacera.discovery.kafka.consumer.config.ow.solr.resource.parallel.size=50
privacera.discovery.kafka.consumer.config.ow.solr.resource.task.timeout.ms=432000000

OUTPUT WRITER - SOLR LINEAGE
privacera.discovery.kafka.consumer.config.ow.solr.lineage.max.poll.records=10000
privacera.discovery.kafka.consumer.config.ow.solr.lineage.parallel.size=50
privacera.discovery.kafka.consumer.config.ow.solr.lineage.task.timeout.ms=86400000

OUTPUT WRITER - RESOURCE WORKFLOW
privacera.discovery.kafka.consumer.config.ow.resource.workflow.max.poll.records=10000
privacera.discovery.kafka.consumer.config.ow.resource.workflow.parallel.size=50
privacera.discovery.kafka.consumer.config.ow.resource.workflow.task.timeout.ms=86400000

OUTPUT WRITER - RANGER REST CLASSIFICATION
privacera.discovery.kafka.consumer.config.ow.ranger.rest.classifications.max.poll.records=10000
privacera.discovery.kafka.consumer.config.ow.ranger.rest.classifications.parallel.size=50
privacera.discovery.kafka.consumer.config.ow.ranger.rest.classifications.task.timeout.ms=86400000