Performance Configurations on Rook Ceph

When using Rook Ceph it is important to consider resource allocation and configuration adjustments to ensure optimal performance. Rook introduces additional management overhead compared to a traditional bare-metal Ceph setup and needs more infrastructure resources.

For more information on Ceph hardware recommendation, see the documentation at Hardware Recommendations.

Another factor to consider is the size of the data blocks. Reading and writing small block files can degrade Ceph’s performance, especially during high-frequency operations.

Pod resource limit tuning

To check the current values for OSDs memory limits:

$ helm get values -n rook-ceph rook-ceph-cluster -o yaml | grep ' osd:' -A2

If you want to adjust memory settings in an effort to improve OSD read/write performance, you can allocate more memory to OSDs by running the following command:

$ cat << EOF >> limit_override.yml
cephClusterSpec:
  resources:
     osd:
       limits:
         memory: <value>
EOF

Make sure to provide parameter the with the correct unit, e.g.: 4Gi.

Then reapply the override:

~(keystone_admin)$ system helm-override-update rook-ceph rook-ceph-cluster rook-ceph --values limit_override.yml

Note

The settings applied using helm-override-update remain active until the Rook-Ceph application is deleted. If the application is deleted and reinstalled, these settings will need to be reapplied.

Finally, apply the Rook-Ceph application:

~(keystone_admin)$ system application-apply rook-ceph

Bluestore tunable parameters

The osd_memory_cache_min and osd_memory_target parameters impact memory management in OSDs. Increasing them improves performance by optimizing memory usage and reducing latencies for read/write operations. However, higher values consume more resources, which can affect overall platform resources utilization. For performance similar to a Ceph bare metal environment, a significant increase in these parameters is required.

To check the current values for these parameters, use:

$ helm get values -n rook-ceph rook-ceph-cluster -o yaml | sed -n '/^configOverride:/,/^[[:alnum:]_-]*:/{/^[[:alnum:]_-]*:/!p}'

To modify these parameters first create a override with the updated values:

$ cat << EOF >> tunable_override.yml
configOverride: |
  [global]
  osd_pool_default_size = 1
  osd_pool_default_min_size = 1
  auth_cluster_required = cephx
  auth_service_required = cephx
  auth_client_required = cephx

  [osd]
  osd_mkfs_type = xfs
  osd_mkfs_options_xfs = "-f"
  osd_mount_options_xfs = "rw,noatime,inode64,logbufs=8,logbsize=256k"
  osd_memory_target = <value>
  osd_memory_cache_min = <value>

  [mon]
  mon_warn_on_legacy_crush_tunables = false
  mon_pg_warn_max_per_osd = 2048
  mon_pg_warn_max_object_skew = 0
  mon_clock_drift_allowed = .1
  mon_warn_on_pool_no_redundancy = false
EOF

Make sure to provide the osd_memory_target and osd_memory_cache_min with the correct unit, e.g.: 4Gi.

The default value for osd_memory_cache_min is 4Gi. The default value for osd_memory_target is 128Mi.

Then run helm-override-update:

~(keystone_admin)$ system helm-override-update rook-ceph rook-ceph-cluster rook-ceph --values tunable_override.yml

Note

The settings applied using helm-override-update remain active until the Rook-Ceph application is deleted. If the application is deleted and reinstalled, these settings will need to be reapplied.

Then reapply the Rook-Ceph application:

~(keystone_admin)$ system application-apply rook-ceph

To change the configuration of an already running OSD without restarting it, the following Ceph config commands must be executed:

$ ceph config set osd.<id> osd_memory_target <value>
$ ceph config set osd.<id> osd_memory_cache_min <value>

Note

Changes made with ceph config set commands will persist for the life of the Ceph cluster. However, if the Ceph cluster is removed (e.g., deleted and recreated), these changes will be lost and will need to be reapplied once the cluster is redeployed.

Low Resource Usage Profile

About this task

The Low Resource Usage Profile is a combination of Ceph overrides designed for deployments that need to minimize Rook-Ceph’s memory, and CPU footprint as much as possible, at the cost of performance.

This profile is ideal for resource-constrained clusters, where storage is used occasionally, reliability is more important than speed, and performance trade-offs are acceptable.

It is targeted at AIO systems such as Simplex, Duplex, and Duplex+, where available platform resources are more constrained compared to Standard configurations. It is best suited for clusters where Ceph is not performance-critical, for example, deployments where storage is used only for logging, light workloads, or occasional access. Deployments that simply need reliable storage, but do not require high throughput, or low latency will benefit the most.

By applying this profile, Ceph’s overall system resource consumption is reduced, allowing StarlingX clusters to run more efficiently on smaller, or resource-constrained environments.

Profile

The profile achieves lower resource usage by tuning several OSD and monitor parameters:

Memory reduction

  • RocksDB and Bluestore cache sizes are capped (64MB / 128MB).

  • OSD memory target is limited to 1GB instead of auto-scaling to 4GB.

  • Monitor memory usage is capped at 256MB with autotune disabled.

CPU usage reduction

  • OSDs are configured with fewer threads per shard and low shard count, reducing CPU parallelism.

  • Monitors are restricted to 1 CPU thread.

I/O trade-offs

  • Compression is disabled to save CPU, which is the default behavior already.

  • Recovery, backfill, and scrub operations are limited to 1 concurrent task, minimizing background impact at the cost of longer recovery times.

  • Scrubbing during recovery is disabled (default behavior already), and scrubs are throttled with a sleep delay.

Overall, this creates a lighter-weight Ceph cluster with predictable and capped resource usage, good for ‘just enough storage’ scenarios.

Apply the Low Resource Profile

  1. Check the current values for these parameters.

$ helm get values -n rook-ceph rook-ceph-cluster -o yaml | sed -n '/^configOverride:/,/^[[:alnum:]_-]*:/{/^[[:alnum:]_-]*:/!p}'

  1. Create the override file based on the original override used. Changes from default override are marked with a “#”:

$ cat << EOF >> low_resource_override.yml
configOverride: |
  [global]
  rocksdb_cache_size = 67108864               # 64MB
  osd_pool_default_size = 1
  osd_pool_default_min_size = 1
  auth_cluster_required = cephx
  auth_service_required = cephx
  auth_client_required = cephx

  [osd]
  bluestore_cache_autotune = false            # Disabled
  bluestore_cache_size = 134217728            # 128MB
  osd_memory_base = 805306368                 # 768MB
  osd_memory_cache_min = 134217728            # 128MB
  osd_memory_target = 1073741824              # 1GB
  osd_op_num_threads_per_shard = 1            # Low CPU usage
  osd_op_num_shards = 2                       # Low parallelism
  bluestore_compression_mode = none           # Disable compression (default)
  osd_recovery_max_active = 1                 # Minimize recovery impact
  osd_max_backfills = 1                       # Minimize backfill impact
  osd_scrub_during_recovery = false           # Avoid scrubbing during recovery (default)
  osd_scrub_sleep = 0.1                       # Reduce scrub I/O
  osd_mkfs_type = xfs
  osd_mkfs_options_xfs = "-f"
  osd_mount_options_xfs = "rw,noatime,inode64,logbufs=8,logbsize=256k"

  [mon]
  mon_cpu_threads = 1                         #
  mon_osd_cache_size = 50                     #
  mon_osd_cache_size_min = 67108864           # 64MB
  mon_max_osd = 32                            #
  mon_memory_autotune = false                 #
  mon_memory_target = 268435456               # 256MB
  mon warn on legacy crush tunables = false
  mon pg warn max per osd = 2048
  mon pg warn max object skew = 0
  mon clock drift allowed = .1
  mon warn on pool no redundancy = false
EOF

  1. Apply the overrides to Rook-Ceph, and wait for the applied state.

~(keystone_admin)$ system helm-override-update rook-ceph rook-ceph-cluster rook-ceph --reuse-values --values low_resource_override.yml
~(keystone_admin)$ system application-apply rook-ceph

  1. Restart the deployments and wait for all pods to run.

$ kubectl get deployments -n rook-ceph -o custom-columns=NAME:.metadata.name --no-headers | xargs -r -I{} kubectl rollout restart deployment -n rook-ceph {}

Note

The settings applied using helm-override-update remain active until the Rook-Ceph application is deleted. If the application is deleted and reinstalled, these settings will need to be reapplied.

Accelerate Memory Release

Ceph daemons may hold memory until explicitly released. To force release memory after applying the overrides:

for i in $(ceph osd ls); do ceph tell osd.$i heap release; done
ceph tell mon.<id> heap release

Repeat for all monitor IDs.

Expected Results

Based on internal testing performed on All-in-One systems (Simplex, Duplex, Duplex+) with SATA SSDs, the Low Resource Usage Profile typically yields:

  • OSD memory usage: -50%

  • OSD CPU usage: -10%

  • Read performance (small block sizes): up to -50%

  • Write performance (small block sizes): up to -35%

  • Large block sizes: much smaller impact as they are not CPU-bound (throughput largely preserved)

This means workloads that depend on small random I/O will notice degraded performance, while large sequential reads/writes remain close to normal levels.