- Platform Release 6.5
- Privacera Platform Installation
- About Privacera Manager (PM)
- Install overview
- Prerequisites
- Installation
- Default services configuration
- Component services configurations
- Access Management
- Data Server
- PolicySync
- Snowflake
- Redshift
- Redshift Spectrum
- PostgreSQL
- Microsoft SQL Server
- Databricks SQL
- RocksDB
- Google BigQuery
- Power BI
- UserSync
- Privacera Plugin
- Databricks
- Spark standalone
- Spark on EKS
- Portal SSO with PingFederate
- Trino Open Source
- Dremio
- AWS EMR
- AWS EMR with Native Apache Ranger
- GCP Dataproc
- Starburst Enterprise
- Privacera services (Data Assets)
- Audit Fluentd
- Grafana
- Ranger Tagsync
- Discovery
- Encryption & Masking
- Privacera Encryption Gateway (PEG) and Cryptography with Ranger KMS
- AWS S3 bucket encryption
- Ranger KMS
- AuthZ / AuthN
- Security
- Access Management
- Reference - Custom Properties
- Validation
- Additional Privacera Manager configurations
- CLI actions
- Debugging and logging
- Advanced service configuration
- Increase Privacera portal timeout for large requests
- Order of precedence in PolicySync filter
- Configure system properties
- PolicySync
- Databricks
- Table properties
- Upgrade Privacera Manager
- Troubleshooting
- How to validate installation
- Possible Errors and Solutions in Privacera Manager
- Unable to Connect to Docker
- Terminate Installation
- 6.5 Platform Installation fails with invalid apiVersion
- Ansible Kubernetes Module does not load
- Unable to connect to Kubernetes Cluster
- Common Errors/Warnings in YAML Config Files
- Delete old unused Privacera Docker images
- Unable to debug error for an Ansible task
- Unable to upgrade from 4.x to 5.x or 6.x due to Zookeeper snapshot issue
- Storage issue in Privacera UserSync & PolicySync
- Permission Denied Errors in PM Docker Installation
- Unable to initialize the Discovery Kubernetes pod
- Portal service
- Grafana service
- Audit server
- Audit Fluentd
- Privacera Plugin
- How-to
- Appendix
- AWS topics
- AWS CLI
- AWS IAM
- Configure S3 for real-time scanning
- Install Docker and Docker compose (AWS-Linux-RHEL)
- AWS S3 MinIO quick setup
- Cross account IAM role for Databricks
- Integrate Privacera services in separate VPC
- Securely access S3 buckets ssing IAM roles
- Multiple AWS account support in Dataserver using Databricks
- Multiple AWS S3 IAM role support in Dataserver
- Azure topics
- GCP topics
- Kubernetes
- Microsoft SQL topics
- Snowflake configuration for PolicySync
- Create Azure resources
- Databricks
- Spark Plug-in
- Azure key vault
- Add custom properties
- Migrate Ranger KMS master key
- IAM policy for AWS controller
- Customize topic and table names
- Configure SSL for Privacera
- Configure Real-time scan across projects in GCP
- Upload custom SSL certificates
- Deployment size
- Service-level system properties
- PrestoSQL standalone installation
- AWS topics
- Privacera Platform User Guide
- Introduction to Privacera Platform
- Settings
- Data inventory
- Token generator
- System configuration
- Diagnostics
- Notifications
- How-to
- Privacera Discovery User Guide
- What is Discovery?
- Discovery Dashboard
- Scan Techniques
- Processing order of scan techniques
- Add and scan resources in a data source
- Start or cancel a scan
- Tags
- Dictionaries
- Patterns
- Scan status
- Data zone movement
- Models
- Disallowed Tags policy
- Rules
- Types of rules
- Example rules and classifications
- Create a structured rule
- Create an unstructured rule
- Create a rule mapping
- Export rules and mappings
- Import rules and mappings
- Post-processing in real-time and offline scans
- Enable post-processing
- Example of post-processing rules on tags
- List of structured rules
- Supported scan file formats
- Data Source Scanning
- Data Inventory
- TagSync using Apache Ranger
- Compliance Workflow
- Data zones and workflow policies
- Workflow Policies
- Alerts Dashboard
- Data Zone Dashboard
- Data zone movement
- Workflow policy use case example
- Discovery Health Check
- Reports
- How-to
- Privacera Encryption Guide
- Overview of Privacera Encryption
- Install Privacera Encryption
- Encryption Key Management
- Schemes
- Encryption with PEG REST API
- Privacera Encryption REST API
- PEG API endpoint
- PEG REST API encryption endpoints
- PEG REST API authentication methods on Privacera Platform
- Common PEG REST API fields
- Construct the datalist for the /protect endpoint
- Deconstruct the response from the /unprotect endpoint
- Example data transformation with the /unprotect endpoint and presentation scheme
- Example PEG API endpoints
- /authenticate
- /protect with encryption scheme
- /protect with masking scheme
- /protect with both encryption and masking schemes
- /unprotect without presentation scheme
- /unprotect with presentation scheme
- /unprotect with masking scheme
- REST API response partial success on bulk operations
- Audit details for PEG REST API accesses
- Make encryption API calls on behalf of another user
- Troubleshoot REST API Issues on Privacera Platform
- Privacera Encryption REST API
- Encryption with Databricks, Hive, Streamsets, Trino
- Databricks UDFs for encryption and masking
- Hive UDFs
- StreamSets Data Collector (SDC) and Privacera Encryption
- Trino UDFs for encryption and masking
- Privacera Access Management User Guide
- Privacera Access Management
- How Polices are evaluated
- Resource policies
- Policies overview
- Creating Resource Based Policies
- Configure Policy with Attribute-Based Access Control
- Configuring Policy with Conditional Masking
- Tag Policies
- Entitlement
- Service Explorer
- Users, groups, and roles
- Permissions
- Reports
- Audit
- Security Zone
- Access Control using APIs
- AWS User Guide
- Overview of Privacera on AWS
- Set policies for AWS services
- Using Athena with data access server
- Using DynamoDB with data access server
- Databricks access manager policy
- Accessing Kinesis with data access server
- Accessing Firehose with Data Access Server
- EMR user guide
- AWS S3 bucket encryption
- Getting started with Minio
- Plugins
- How to Get Support
- Coordinated Vulnerability Disclosure (CVD) Program of Privacera
- Shared Security Model
- Privacera Platform documentation changelog
RocksDB
This topic shows how to configure the RocksDB key-value store so you can tune the performance settings for PolicySync.
Configuration
SSH to the instance as USER.
Run the following commands.
cd ~/privacera/privacera-manager/config cp sample-vars/vars.policysync.rocksdb.tuning.yml custom-vars/ vi custom-vars/vars.policysync.rocksdb.tuning.yml
Edit the properties as required. For property details and description, refer to the Configuration Properties below.
Run the following commands.
cd ~/privacera/privacera-manager ./privacera-manager.sh update
Configure the maximum log size and number of logs retained
By default each log file grows to a maximum of size of 100MB, or 104,857,600 byes, and the number of log files retained is unlimited.
To ensure that RocksDB log files do not consume too much disk space, you can configure the maximum log size and the number of files that Privacera retains by setting the following properties:
pscontext.rocksdb.max.log.file.size
: Specifies the maximum size of a log file in bytes. The default is104857600
bytes.pscontext.rocksdb.keep.log.file.num
: Specifies the maximum number of log files to retain. When this number is exceeded older log files are automatically deleted.
Procedure
Log in to the system where Privacera Manager is installed, and then change to the
~/privacera/privacera-manager
directory.Create a file in the
config/custom-properties/
directory with therangersync-custom-v2.properties
file name.Edit the
rangersync-custom-v2.properties
file and specify values for the previously described logging properties as appropriate.Run Privacera Manager to update your configuration:
./privacera-manager.sh update
Configuration Properties
Property | Description | Example |
---|---|---|
| Specifies the maximum number of concurrent background jobs (both flushes and compactions combined). | ROCKSDB_MAX_BACKGROUND_JOBS: "2" |
| If true, allow multi-writers to update mem tables in parallel. Only some memtable factorys support concurrent writes; currently it is implemented only for SkipListFactory. Concurrent memtable writes are not compatible with inplace_update_support or filter_deletes. | ROCKSDB_ALLOW_CONCURRENT_MEMTABLE_WRITE: "true" |
| By default, a single write thread queue is maintained. The thread gets to the head of the queue becomes write batch group leader and responsible for writing to WAL and memtable for the batch group. If enablePipelinedWrite() is true, separate write thread queue is maintained for WAL write and memtable write. A write thread first enter WAL writer queue and then memtable writer queue. Pending thread on the WAL writer queue thus only have to wait for previous writers to finish their WAL writing but not the memtable writing. Enabling the feature may improve write throughput and reduce latency of the prepare phase of two-phase commit. | ROCKSDB_ENABLE_PIPELINED_WRITE: "false" |
| Amount of data to build up in memtables across all column families before writing to disk. This is distinct from ColumnFamilyOptions.writeBufferSize(), which enforces a limit for a single memtable. This feature is disabled by default. Specify a non-zero value to enable it. | ROCKSDB_DB_WRITE_BUFFER_SIZE: "0" |
| This is a maximum buffer size that is used by WinMmapReadableFile in unbuffered disk I/O mode. We need to maintain an aligned buffer for reads. We allow the buffer to grow until the specified value and then for bigger requests allocate one shot buffers. In unbuffered mode we always bypass read-ahead buffer at ReadaheadRandomAccessFile When read-ahead is required we then make use of MutableDBOptionsInterface.compactionReadaheadSize() value and always try to read ahead. With read-ahead we always pre-allocate buffer to the size instead of growing it up to a limit. This option is currently honored only on Windows Default: 1 Mb Special value: 0 - means do not maintain per instance buffer. Allocate per request buffer and avoid locking. | ROCKSDB_RANDOM_ACCESS_MAX_BUFFER_SIZE: "0" |
| This is a maximum buffer size that is used by WinMmapReadableFile in unbuffered disk I/O mode. We need to maintain an aligned buffer for reads. We allow the buffer to grow until the specified value and then for bigger requests allocate one shot buffers. In unbuffered mode we always bypass read-ahead buffer at ReadaheadRandomAccessFile When read-ahead is required we then make use of MutableDBOptionsInterface.compactionReadaheadSize() value and always try to read ahead. With read-ahead we always pre-allocate buffer to the size instead of growing it up to a limit. This option is currently honored only on Windows Default: 1 Mb Special value: 0 - means do not maintain per instance buffer. Allocate per request buffer and avoid locking. | ROCKSDB_WRITABLE_FILE_MAX_BUFFER_SIZE: "0" |
| Allow the OS to mmap file for reading sst tables. | ROCKSDB_ALLOW_MMAP_READS: "false" |
| Allow the OS to mmap file for writing. | ROCKSDB_ALLOW_MMAP_READS: "false" |
| Allows OS to incrementally sync files to disk while they are being written, asynchronously, in the background. Issue one request for every bytes_per_sync written. | ROCKSDB_BYTES_PER_SYNC: "0" |
| Same as setBytesPerSync(long) , but applies to WAL files | ROCKSDB_WAL_BYTES_PER_SYNC: "0" |
| rateBytesPerSecond this is the only parameter you want to set most of the time. It controls the total write rate of compaction and flush in bytes per second. Currently, RocksDB does not enforce rate limit for anything other than flush and compaction, e.g. write to WAL. | ROCKSDB_RATELIMITER_RATE_BYTES_PER_SEC: "0" |
| Number of open files that can be used by the DB. You may need to increase this if your database has a large working set. Value -1 means files opened are always kept open. You can estimate number of files based on target_file_size_base and target_file_size_multiplier for level-based compaction. For universal-style compaction, you can usually set it to -1. | ROCKSDB_MAX_OPEN_FILES: "0" |
| Amount of data to build up in memory (backed by an unsorted log on disk) before converting to a sorted on-disk file. Larger values increase performance, especially during bulk loads. Up to max_write_buffer_number write buffers may be held in memory at the same time, so you may wish to adjust this parameter to control memory usage. Also, a larger write buffer will result in a longer recovery time the next time the database is opened. | ROCKSDB_CF_WRITE_BUFFER_SIZE: "0" |
| Compress blocks using the specified compression algorithm. This parameter can be changed dynamically. Default: SNAPPY_COMPRESSION, which gives lightweight but fast compression. | ROCKSDB_CF_COMPRESSIONTYPE_LZ4COMPRESSION: "false" ROCKSDB_CF_COMPRESSIONTYPE_ZSTDCOMPRESSION: "false" ROCKSDB_CF_COMPRESSIONTYPE_ZLIBCOMPRESSION: "false" |
| With this option on, from an empty DB, we make last level the base level, which means merging L0 data into the last level, until it exceeds max_bytes_for_level_base. And then we make the second last level to be base level, to start to merge L0 data to second last level, with its target size to be 1/max_bytes_for_level_multiplier of the last levels extra size. After the data accumulates more so that we need to move the base level to the third last one, and so on. | ROCKSDB_CF_LEVEL_COMPACTION_DYNAMIC_LEVEL_BYTES: "false" |
| Control locality of bloom filter probes to improve cache miss rate. This option only applies to memtable prefix bloom and plaintable prefix bloom. It essentially limits the max number of cache lines each bloom filter check can touch. This optimization is turned off when set to 0. The number should never be greater than number of probes. This option can boost performance for in-memory workload but should use with care since it can cause higher false positive rate. | ROCKSDB_CF_BLOOMLOCALITY: "0" |
| Set compaction style for DB. | ROCKSDB_CF_COMPRESSIONSTYLE_UNIVERSAL: "false" |
| Percentage flexibility while comparing file size. If the candidate file(s) size is 1% smaller than the next file's size, then include next file into this candidate set. | ROCKSDB_CF_COMPRESSIONSTYLE_UNIVERSAL_SIZERATIO: "1" |
| The minimum number of files in a single compaction run. | ROCKSDB_CF_COMPRESSIONSTYLE_UNIVERSAL_MINMERGEWIDTH: "2" |
| The size amplification is defined as the amount (in percentage) of additional storage needed to store a single byte of data in the database. For example, a size amplification of 2% means that a database that contains 100 bytes of user-data may occupy upto 102 bytes of physical storage. By this definition, a fully compacted database has a size amplification of 0%. Rocksdb uses the following heuristic to calculate size amplification: it assumes that all files excluding the earliest file contribute to the size amplification. Default: 200, which means that a 100 byte database could require upto 300 bytes of storage. | ROCKSDB_CF_COMPRESSIONSTYLE_UNIVERSAL_MAXSIZEAMPPERCENT: "200" |
| ROCKSDB_CF_COMPRESSIONSTYLE_FIFO: "false" | |
| If true, try to do compaction to compact smaller files into larger ones. Minimum files to compact follows options.level0_file_num_compaction_trigger and compaction won't trigger if average compact bytes per del file is larger than options.write_buffer_size. This is to protect large files from being compacted again. | ROCKSDB_CF_COMPRESSIONSTYLE_FIFO_ALLOWCOMPACTION: "false" |
| Once the total sum of table files reaches this, we will delete the oldest table file | ROCKSDB_CF_COMPRESSIONSTYLE_FIFO_MAXTABLEFILESIZE: "1024" |
| ROCKSDB_CF_COMPRESSIONSTYLE_FIFO_MAXTABLEFILESIZE: "1024" | |
| Number of files to trigger level-0 compaction. A value < 0 means that level-0 compaction will not be triggered by number of files at all. | ROCKSDB_CF_LEVEL0FILENUMCOMPACTIONTRIGGER: "0" |
| Soft limit on number of level-0 files. We start slowing down writes at this point. A value < 0 means that no writing slow down will be triggered by number of files in level-0. | ROCKSDB_CF_LEVEL0SLOWDOWNWRITESTRIGGER: "0" |
| Soft limit on number of level-0 files. We start slowing down writes at this point. A value < 0 means that no writing slow down will be triggered by number of files in level-0. | ROCKSDB_CF_LEVEL0STOPWRITESTRIGGER: "0" |
| The total maximum number of write buffers to maintain in memory including copies of buffers that have already been flushed. Unlike AdvancedMutableColumnFamilyOptionsInterface.maxWriteBufferNumber(), this parameter does not affect flushing. This controls the minimum amount of write history that will be available in memory for conflict checking when Transactions are used. When using an OptimisticTransactionDB: If this value is too low, some transactions may fail at commit time due to not being able to determine whether there were any write conflicts. When using a TransactionDB: If Transaction::SetSnapshot is used, TransactionDB will read either in-memory write buffers or SST files to do write-conflict checking. Increasing this value can reduce the number of reads to SST files done for conflict detection. Setting this value to 0 will cause write buffers to be freed immediately after they are flushed. If this value is set to -1, AdvancedMutableColumnFamilyOptionsInterface.maxWriteBufferNumber() will be used. Default: If using a TransactionDB/OptimisticTransactionDB, the default value will be set to the value of AdvancedMutableColumnFamilyOptionsInterface.maxWriteBufferNumber() if it is not explicitly set by the user. Otherwise, the default is 0. | ROCKSDB_CF_LEVEL0STOPWRITESTRIGGER: "0" |
| The total maximum number of write buffers to maintain in memory including copies of buffers that have already been flushed. Unlike AdvancedMutableColumnFamilyOptionsInterface.maxWriteBufferNumber(), this parameter does not affect flushing. This controls the minimum amount of write history that will be available in memory for conflict checking when Transactions are used. When using an OptimisticTransactionDB: If this value is too low, some transactions may fail at commit time due to not being able to determine whether there were any write conflicts. When using a TransactionDB: If Transaction::SetSnapshot is used, TransactionDB will read either in-memory write buffers or SST files to do write-conflict checking. Increasing this value can reduce the number of reads to SST files done for conflict detection. Setting this value to 0 will cause write buffers to be freed immediately after they are flushed. If this value is set to -1, AdvancedMutableColumnFamilyOptionsInterface.maxWriteBufferNumber() will be used. Default: If using a TransactionDB/OptimisticTransactionDB, the default value will be set to the value of AdvancedMutableColumnFamilyOptionsInterface.maxWriteBufferNumber() if it is not explicitly set by the user. Otherwise, the default is 0. | ROCKSDB_CF_MAX_WRITE_BUFFER_NUMBER_TO_MAINTAIN: "0" |
| Set the number of levels for this database If level-styled compaction is used, then this number determines the total number of levels. | ROCKSDB_CF_NUMLEVEL: "0" |
| The target file size for compaction. This targetFileSizeBase determines a level-1 file size. Target file size for level L can be calculated by targetFileSizeBase * (targetFileSizeMultiplier ^ (L-1)) For example, if targetFileSizeBase is 2MB and target_file_size_multiplier is 10, then each file on level-1 will be 2MB, and each file on level 2 will be 20MB, and each file on level-3 will be 200MB. | ROCKSDB_CF_TARGETFILESIZEBASE: "0" |
| The upper-bound of the total size of level-1 files in bytes. Maximum number of bytes for level L can be calculated as (maxBytesForLevelBase) * (maxBytesForLevelMultiplier ^ (L-1)) For example, if maxBytesForLevelBase is 20MB, and if max_bytes_for_level_multiplier is 10, total data size for level-1 will be 200MB, total file size for level-2 will be 2GB, and total file size for level-3 will be 20GB. | ROCKSDB_CF_MAXBYTESFORLEVELBASE: "0" |
| The ratio between the total size of level-(L+1) files and the total size of level-L files for all L. | ROCKSDB_CF_MULTIPLIER: "0" |
| Enable tableconfig for columnfamily | ROCKSDB_CF_TABLECONFIG_ENABLE: "false" |
| Approximate size of user data packed per block. Note that the block size specified here corresponds to uncompressed data. The actual size of the unit read from disk may be smaller if compression is enabled. This parameter can be changed dynamically. | ROCKSDB_CF_TABLECONFIG_BLOCKSIZE: "4000" |
| Indicating if we'd put index/filter blocks to the block cache. If not specified, each "table reader" object will pre-load index/filter block during table initialization. | ROCKSDB_CF_TABLECONFIG_CACHEINDEXANDFILTERBLOCKS: "false" |
| We currently have five versions: #0 - This version is currently written out by all RocksDB's versions by default. Can be read by really old RocksDB's. Doesn't support changing checksum (default is CRC32). #1 - Can be read by RocksDB's versions since 3.0. Supports non-default checksum, like xxHash. It is written by RocksDB when BlockBasedTableOptions::checksum is something other than kCRC32c. (version 0 is silently upconverted) #2 - Can be read by RocksDB's versions since 3.10. Changes the way we encode compressed blocks with LZ4, BZip2 and Zlib compression. If you don't plan to run RocksDB before version 3.10, you should probably use this. #3 - Can be read by RocksDB's versions since 5.15. Changes the way we encode the keys in index blocks. If you don't plan to run RocksDB before version 5.15, you should probably use this. This option only affects newly written tables. When reading existing tables, the information about version is read from the footer. #4 - Can be read by RocksDB's versions since 5.16. Changes the way we encode the values in index blocks. If you don't plan to run RocksDB before version 5.16 and you are using index_block_restart_interval > 1, you should probably use this as it would reduce the index size. #This option only affects newly written tables. When reading existing tables, the information about version is read from the footer. | ROCKSDB_CF_TABLECONFIG_FORMATVERSION: "0" |
| Indicating if we'd like to pin L0 index/filter blocks to the block cache. If not specified, defaults to false. | ROCKSDB_CF_TABLECONFIG_PINL0FILTERANDINDEXBLOCKSINCACHE: "false" |
| Sets the index type to used with this table. | ROCKSDB_CF_TABLECONFIG_INDEXTYPE_KHASHSEARCH: "false" ROCKSDB_CF_TABLECONFIG_INDEXTYPE_KBINARYSEARCH: "false" ROCKSDB_CF_TABLECONFIG_INDEXTYPE_KTWOLEVELINDEXSEARCH: "false" |