#CephCluster [cephclusters.ceph.rook.io/v1]

APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

ObjectMeta is metadata that all persisted resources must have, which includes all objects users must create.

ClusterSpec represents the specification of Ceph Cluster

ClusterStatus represents the status of a Ceph cluster

The annotations-related configuration to add/set on each Pod related object.

Ceph Config options

CephConfigFromSecret works exactly like CephConfig but takes config value from Secret Key reference.

The version information that instructs Rook to orchestrate a particular version of Ceph.

Indicates user intent when deleting a cluster; blocks orchestration and should not be set if cluster deletion is not imminent.

ContinueUpgradeAfterChecksEvenIfNotHealthy defines if an upgrade should continue even if PGs are not clean

A spec for the crash controller

CSI Driver Options applied per cluster.

Dashboard settings

The path on the host where config and data can be persisted

A spec for configuring disruption management.

Whether the Ceph Cluster is running external to this Kubernetes cluster mon, mgr, osd, mds, and discover daemons will not be created for external clusters.

Internal daemon healthchecks and liveness probe

The labels-related configuration to add/set on each Pod related object.

Logging represents loggings settings

A spec for mgr related options

A spec for mon related options

Prometheus based Monitoring settings

Network related configuration

PriorityClassNames sets priority classes on components

Remove the OSD that is out and safe to remove only if this option is true

Resources set resource requests and limits

Security represents security settings

SkipUpgradeChecks defines if an upgrade should be forced even if one of the check fails

A spec for available storage in the cluster and how it should be used

UpgradeOSDRequiresHealthyPGs defines if OSD upgrade requires PGs are clean. If set to true OSD upgrade process won't start until PGs are healthy. This configuration will be ignored if skipUpgradeChecks is true. Default is false.

WaitTimeoutForHealthyOSDInMinutes defines the time the operator would wait before an OSD can be stopped for upgrade or restart. If the timeout exceeds and OSD is not ok to stop, then the operator would skip upgrade for the current OSD and proceed with the next one if continueUpgradeAfterChecksEvenIfNotHealthy is false. If continueUpgradeAfterChecksEvenIfNotHealthy is true, then operator would continue with the upgrade of an OSD even if its not ok to stop after the timeout. This timeout won't be applied if skipUpgradeChecks is true. The default wait timeout is 10 minutes.

Whether to allow unsupported versions (do not set to true in production)

Image is the container image used to launch the ceph daemons, such as quay.io/ceph/ceph: The full list of images can be found at https://quay.io/repository/ceph/ceph?tab=tags

ImagePullPolicy describes a policy for if/when to pull a container image One of Always, Never, IfNotPresent.

AllowUninstallWithVolumes defines whether we can proceed with the uninstall if they are RBD images still present

Confirmation represents the cleanup confirmation

SanitizeDisks represents way we sanitize disks

WipeDevicesFromOtherClusters wipes the OSD disks belonging to other clusters. This is useful in scenarios where ceph cluster was reinstalled but OSD disk still contains the metadata from previous ceph cluster.

DataSource is the data source to use to sanitize the disk with

Iteration is the number of pass to apply the sanitizing

Method is the method we use to sanitize disks

DaysToRetain represents the number of days to retain crash until they get pruned

Disable determines whether we should enable the crash collector

CephFS defines CSI Driver settings for CephFS driver.

ReadAffinity defines the read affinity settings for CSI driver.

SkipUserCreation determines whether CSI users and their associated secrets should be skipped. If set to true, the user must manually manage these secrets.

FuseMountOptions defines the mount options for ceph fuse mounter.

KernelMountOptions defines the mount options for kernel mounter.

CrushLocationLabels defines which node labels to use as CRUSH location. This should correspond to the values set in the CRUSH map.

Enables read affinity for CSI driver.

Enabled determines whether to enable the dashboard

Port is the dashboard webserver port

Endpoint for the Prometheus host

Whether to verify the ssl endpoint for prometheus. Set to false for a self-signed cert.

SSL determines whether SSL should be used

URLPrefix is a prefix for all URLs to use the dashboard with a reverse proxy

Deprecated. Namespace to look for MDBs by the machineDisruptionBudgetController

Deprecated. This enables management of machinedisruptionbudgets.

This enables management of poddisruptionbudgets

OSDMaintenanceTimeout sets how many additional minutes the DOWN/OUT interval is for drained failure domains it only works if managePodBudgets is true. the default is 30 minutes

DEPRECATED: PGHealthCheckTimeout is no longer implemented

PgHealthyRegex is the regular expression that is used to determine which PG states should be considered healthy. The default is ^(active\+clean|active\+clean\+scrubbing|active\+clean\+scrubbing\+deep)$

Enable determines whether external mode is enabled or not

DaemonHealth is the health check for a given daemon

LivenessProbe allows changing the livenessProbe configuration for a given daemon

StartupProbe allows changing the startupProbe configuration for a given daemon

Monitor represents the health check settings for the Ceph monitor

ObjectStorageDaemon represents the health check settings for the Ceph OSDs

Status represents the health check settings for the Ceph health

Interval is the internal in second or minute for the health check to run like 60s for 60 seconds

Interval is the internal in second or minute for the health check to run like 60s for 60 seconds

Interval is the internal in second or minute for the health check to run like 60s for 60 seconds

Enabled represents whether the log collector is enabled

MaxLogSize is the maximum size of the log per ceph daemons. Must be at least 1M.

Periodicity is the periodicity of the log rotation.

AllowMultiplePerNode allows to run multiple managers on the same node (not recommended)

Count is the number of manager daemons to run

Whether host networking is enabled for the Ceph Mgr. If not set, the network settings from CephCluster.spec.networking will be applied.

Modules is the list of ceph manager modules to enable/disable

Enabled determines whether a module should be enabled or not

Name is the name of the ceph manager module

Settings to further configure the module

BalancerMode sets the balancer module with different modes like upmap, crush-compact etc

AllowMultiplePerNode determines if we can run multiple monitors on the same node (not recommended)

Count is the number of Ceph monitors

ExternalMonIDs - optional list of monitor IDs which are deployed externally and not managed by Rook. If set, Rook will not remove mons with given IDs from quorum. This parameter is used only for local Rook cluster running in normal mode and will be ignored if external or stretched mode is used. leading

StretchCluster is the stretch cluster specification

VolumeClaimTemplate is the PVC definition

Zones are specified when we want to provide zonal awareness to mons

FailureDomainLabel the failure domain name (e,g: zone)

SubFailureDomain is the failure domain within a zone

Zones is the list of zones

Arbiter determines if the zone contains the arbiter used for stretch cluster mode

Name is the name of the zone

VolumeClaimTemplate is the PVC template

Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

spec defines the desired characteristics of a volume requested by a pod author. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims

accessModes contains the desired access modes the volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1

dataSource field can be used to specify either:

• An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot)
• An existing PVC (PersistentVolumeClaim) If the provisioner or an external controller can support the specified data source, it will create a new volume based on the contents of the specified data source. When the AnyVolumeDataSource feature gate is enabled, dataSource contents will be copied to dataSourceRef, and dataSourceRef contents will be copied to dataSource when dataSourceRef.namespace is not specified. If the namespace is specified, then dataSourceRef will not be copied to dataSource.

dataSourceRef specifies the object from which to populate the volume with data, if a non-empty volume is desired. This may be any object from a non-empty API group (non core object) or a PersistentVolumeClaim object. When this field is specified, volume binding will only succeed if the type of the specified object matches some installed volume populator or dynamic provisioner. This field will replace the functionality of the dataSource field and as such if both fields are non-empty, they must have the same value. For backwards compatibility, when namespace isn't specified in dataSourceRef, both fields (dataSource and dataSourceRef) will be set to the same value automatically if one of them is empty and the other is non-empty. When namespace is specified in dataSourceRef, dataSource isn't set to the same value and must be empty. There are three important differences between dataSource and dataSourceRef:

• While dataSource only allows two specific types of objects, dataSourceRef allows any non-core object, as well as PersistentVolumeClaim objects.
• While dataSource ignores disallowed values (dropping them), dataSourceRef preserves all values, and generates an error if a disallowed value is specified.
• While dataSource only allows local objects, dataSourceRef allows objects in any namespaces. (Beta) Using this field requires the AnyVolumeDataSource feature gate to be enabled. (Alpha) Using the namespace field of dataSourceRef requires the CrossNamespaceVolumeDataSource feature gate to be enabled.

resources represents the minimum resources the volume should have. Users are allowed to specify resource requirements that are lower than previous value but must still be higher than capacity recorded in the status field of the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources

selector is a label query over volumes to consider for binding.

storageClassName is the name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1

volumeAttributesClassName may be used to set the VolumeAttributesClass used by this claim. If specified, the CSI driver will create or update the volume with the attributes defined in the corresponding VolumeAttributesClass. This has a different purpose than storageClassName, it can be changed after the claim is created. An empty string or nil value indicates that no VolumeAttributesClass will be applied to the claim. If the claim enters an Infeasible error state, this field can be reset to its previous value (including nil) to cancel the modification. If the resource referred to by volumeAttributesClass does not exist, this PersistentVolumeClaim will be set to a Pending state, as reflected by the modifyVolumeStatus field, until such as a resource exists. More info: https://kubernetes.io/docs/concepts/storage/volume-attributes-classes/

volumeMode defines what type of volume is required by the claim. Value of Filesystem is implied when not included in claim spec.

volumeName is the binding reference to the PersistentVolume backing this claim.

APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required.

Kind is the type of resource being referenced

Name is the name of resource being referenced

APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required.

Kind is the type of resource being referenced

Name is the name of resource being referenced

Namespace is the namespace of resource being referenced Note that when a namespace is specified, a gateway.networking.k8s.io/ReferenceGrant object is required in the referent namespace to allow that namespace's owner to accept the reference. See the ReferenceGrant documentation for details. (Alpha) This field requires the CrossNamespaceVolumeDataSource feature gate to be enabled.

Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

matchExpressions is a list of label selector requirements. The requirements are ANDed.

matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.

key is the label key that the selector applies to.

operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.

values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.

Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

spec defines the desired characteristics of a volume requested by a pod author. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims

accessModes contains the desired access modes the volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1

dataSource field can be used to specify either:

• An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot)
• An existing PVC (PersistentVolumeClaim) If the provisioner or an external controller can support the specified data source, it will create a new volume based on the contents of the specified data source. When the AnyVolumeDataSource feature gate is enabled, dataSource contents will be copied to dataSourceRef, and dataSourceRef contents will be copied to dataSource when dataSourceRef.namespace is not specified. If the namespace is specified, then dataSourceRef will not be copied to dataSource.

dataSourceRef specifies the object from which to populate the volume with data, if a non-empty volume is desired. This may be any object from a non-empty API group (non core object) or a PersistentVolumeClaim object. When this field is specified, volume binding will only succeed if the type of the specified object matches some installed volume populator or dynamic provisioner. This field will replace the functionality of the dataSource field and as such if both fields are non-empty, they must have the same value. For backwards compatibility, when namespace isn't specified in dataSourceRef, both fields (dataSource and dataSourceRef) will be set to the same value automatically if one of them is empty and the other is non-empty. When namespace is specified in dataSourceRef, dataSource isn't set to the same value and must be empty. There are three important differences between dataSource and dataSourceRef:

• While dataSource only allows two specific types of objects, dataSourceRef allows any non-core object, as well as PersistentVolumeClaim objects.
• While dataSource ignores disallowed values (dropping them), dataSourceRef preserves all values, and generates an error if a disallowed value is specified.
• While dataSource only allows local objects, dataSourceRef allows objects in any namespaces. (Beta) Using this field requires the AnyVolumeDataSource feature gate to be enabled. (Alpha) Using the namespace field of dataSourceRef requires the CrossNamespaceVolumeDataSource feature gate to be enabled.

resources represents the minimum resources the volume should have. Users are allowed to specify resource requirements that are lower than previous value but must still be higher than capacity recorded in the status field of the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources

selector is a label query over volumes to consider for binding.

storageClassName is the name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1

volumeAttributesClassName may be used to set the VolumeAttributesClass used by this claim. If specified, the CSI driver will create or update the volume with the attributes defined in the corresponding VolumeAttributesClass. This has a different purpose than storageClassName, it can be changed after the claim is created. An empty string or nil value indicates that no VolumeAttributesClass will be applied to the claim. If the claim enters an Infeasible error state, this field can be reset to its previous value (including nil) to cancel the modification. If the resource referred to by volumeAttributesClass does not exist, this PersistentVolumeClaim will be set to a Pending state, as reflected by the modifyVolumeStatus field, until such as a resource exists. More info: https://kubernetes.io/docs/concepts/storage/volume-attributes-classes/

volumeMode defines what type of volume is required by the claim. Value of Filesystem is implied when not included in claim spec.

volumeName is the binding reference to the PersistentVolume backing this claim.

APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required.

Kind is the type of resource being referenced

Name is the name of resource being referenced

APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required.

Kind is the type of resource being referenced

Name is the name of resource being referenced

Namespace is the namespace of resource being referenced Note that when a namespace is specified, a gateway.networking.k8s.io/ReferenceGrant object is required in the referent namespace to allow that namespace's owner to accept the reference. See the ReferenceGrant documentation for details. (Alpha) This field requires the CrossNamespaceVolumeDataSource feature gate to be enabled.

Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

matchExpressions is a list of label selector requirements. The requirements are ANDed.

matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.

key is the label key that the selector applies to.

operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.

values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.

Arbiter determines if the zone contains the arbiter used for stretch cluster mode

Name is the name of the zone

VolumeClaimTemplate is the PVC template

Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

spec defines the desired characteristics of a volume requested by a pod author. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims

accessModes contains the desired access modes the volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1

dataSource field can be used to specify either:

• An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot)
• An existing PVC (PersistentVolumeClaim) If the provisioner or an external controller can support the specified data source, it will create a new volume based on the contents of the specified data source. When the AnyVolumeDataSource feature gate is enabled, dataSource contents will be copied to dataSourceRef, and dataSourceRef contents will be copied to dataSource when dataSourceRef.namespace is not specified. If the namespace is specified, then dataSourceRef will not be copied to dataSource.

dataSourceRef specifies the object from which to populate the volume with data, if a non-empty volume is desired. This may be any object from a non-empty API group (non core object) or a PersistentVolumeClaim object. When this field is specified, volume binding will only succeed if the type of the specified object matches some installed volume populator or dynamic provisioner. This field will replace the functionality of the dataSource field and as such if both fields are non-empty, they must have the same value. For backwards compatibility, when namespace isn't specified in dataSourceRef, both fields (dataSource and dataSourceRef) will be set to the same value automatically if one of them is empty and the other is non-empty. When namespace is specified in dataSourceRef, dataSource isn't set to the same value and must be empty. There are three important differences between dataSource and dataSourceRef:

• While dataSource only allows two specific types of objects, dataSourceRef allows any non-core object, as well as PersistentVolumeClaim objects.
• While dataSource ignores disallowed values (dropping them), dataSourceRef preserves all values, and generates an error if a disallowed value is specified.
• While dataSource only allows local objects, dataSourceRef allows objects in any namespaces. (Beta) Using this field requires the AnyVolumeDataSource feature gate to be enabled. (Alpha) Using the namespace field of dataSourceRef requires the CrossNamespaceVolumeDataSource feature gate to be enabled.

resources represents the minimum resources the volume should have. Users are allowed to specify resource requirements that are lower than previous value but must still be higher than capacity recorded in the status field of the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources

selector is a label query over volumes to consider for binding.

storageClassName is the name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1

volumeAttributesClassName may be used to set the VolumeAttributesClass used by this claim. If specified, the CSI driver will create or update the volume with the attributes defined in the corresponding VolumeAttributesClass. This has a different purpose than storageClassName, it can be changed after the claim is created. An empty string or nil value indicates that no VolumeAttributesClass will be applied to the claim. If the claim enters an Infeasible error state, this field can be reset to its previous value (including nil) to cancel the modification. If the resource referred to by volumeAttributesClass does not exist, this PersistentVolumeClaim will be set to a Pending state, as reflected by the modifyVolumeStatus field, until such as a resource exists. More info: https://kubernetes.io/docs/concepts/storage/volume-attributes-classes/

volumeMode defines what type of volume is required by the claim. Value of Filesystem is implied when not included in claim spec.

volumeName is the binding reference to the PersistentVolume backing this claim.

APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required.

Kind is the type of resource being referenced

Name is the name of resource being referenced

APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required.

Kind is the type of resource being referenced

Name is the name of resource being referenced

Namespace is the namespace of resource being referenced Note that when a namespace is specified, a gateway.networking.k8s.io/ReferenceGrant object is required in the referent namespace to allow that namespace's owner to accept the reference. See the ReferenceGrant documentation for details. (Alpha) This field requires the CrossNamespaceVolumeDataSource feature gate to be enabled.

Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

matchExpressions is a list of label selector requirements. The requirements are ANDed.

matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.

key is the label key that the selector applies to.

operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.

values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.

Enabled determines whether to create the prometheus rules for the ceph cluster. If true, the prometheus types must exist or the creation will fail. Default is false.

Ceph exporter configuration

ExternalMgrEndpoints points to an existing Ceph prometheus exporter endpoint

ExternalMgrPrometheusPort Prometheus exporter port

Interval determines prometheus scrape interval

Whether to disable the metrics reported by Ceph. If false, the prometheus mgr module and Ceph exporter are enabled. If true, the prometheus mgr module and Ceph exporter are both disabled. Default is false.

Port is the prometheus server port

Whether host networking is enabled for CephExporter. If not set, the network settings from CephCluster.spec.networking will be applied.

Only performance counters greater than or equal to this option are fetched

Time to wait before sending requests again to exporter server (seconds)

The Hostname of this endpoint

The IP of this endpoint. May not be loopback (127.0.0.0/8 or ::1), link-local (169.254.0.0/16 or fe80::/10), or link-local multicast (224.0.0.0/24 or ff02::/16).

Optional: Node hosting this endpoint. This can be used to determine endpoints local to a node.

Reference to object providing the endpoint.

API version of the referent.

If referring to a piece of an object instead of an entire object, this string should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. For example, if the object reference is to a container within a pod, this would take on a value like: "spec.containers{name}" (where "name" refers to the name of the container that triggered the event) or if no container name is specified "spec.containers[2]" (container with index 2 in this pod). This syntax is chosen only to have some well-defined way of referencing a part of an object.

Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names

Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/

Specific resourceVersion to which this reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency

UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids

AddressRanges specify a list of CIDRs that Rook will apply to Ceph's 'public_network' and/or 'cluster_network' configurations. This config section may be used for the "host" or "multus" network providers.

Settings for network connections such as compression and encryption across the wire.

DualStack determines whether Ceph daemons should listen on both IPv4 and IPv6

HostNetwork to enable host network. If host networking is enabled or disabled on a running cluster, then the operator will automatically fail over all the mons to apply the new network settings.

IPFamily is the single stack IPv6 or IPv4 protocol

Enable multiClusterService to export the Services between peer clusters

Provider is what provides network connectivity to the cluster e.g. "host" or "multus". If the Provider is updated from being empty to "host" on a running cluster, then the operator will automatically fail over all the mons to apply the "host" network settings.

Selectors define NetworkAttachmentDefinitions to be used for Ceph public and/or cluster networks when the "multus" network provider is used. This config section is not used for other network providers.

Valid keys are "public" and "cluster". Refer to Ceph networking documentation for more: https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/

Refer to Multus network annotation documentation for help selecting values: https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/docs/how-to-use.md#run-pod-with-network-annotation

Rook will make a best-effort attempt to automatically detect CIDR address ranges for given network attachment definitions. Rook's methods are robust but may be imprecise for sufficiently complicated networks. Rook's auto-detection process obtains a new IP address lease for each CephCluster reconcile. If Rook fails to detect, incorrectly detects, only partially detects, or if underlying networks do not support reusing old IP addresses, it is best to use the 'addressRanges' config section to specify CIDR ranges for the Ceph cluster.

As a contrived example, one can use a theoretical Kubernetes-wide network for Ceph client traffic and a theoretical Rook-only network for Ceph replication traffic as shown: selectors: public: "default/cluster-fast-net" cluster: "rook-ceph/ceph-backend-net"

Cluster defines a list of CIDRs to use for Ceph cluster network communication.

Public defines a list of CIDRs to use for Ceph public network communication.

Compression settings for the network connections.

Encryption settings for the network connections.

Whether to require msgr2 (port 3300) even if compression or encryption are not enabled. If true, the msgr1 port (6789) will be disabled. Requires a kernel that supports msgr2 (kernel 5.11 or CentOS 8.4 or newer).

Whether to compress the data in transit across the wire. The default is not set.

Whether to encrypt the data in transit across the wire to prevent eavesdropping the data on the network. The default is not set. Even if encryption is not enabled, clients still establish a strong initial authentication for the connection and data integrity is still validated with a crc check. When encryption is enabled, all communication between clients and Ceph daemons, or between Ceph daemons will be encrypted.

ClusterID uniquely identifies a cluster. It is used as a prefix to nslookup exported services. For example: ...svc.clusterset.local

Enable multiClusterService to export the mon and OSD services to peer cluster. Ensure that peer clusters are connected using an MCS API compatible application, like Globalnet Submariner.

CephX configures CephX key settings. More: https://docs.ceph.com/en/latest/dev/cephx/

KeyRotation defines options for rotation of OSD disk encryption keys.

KeyManagementService is the main Key Management option

CSI configures CephX key rotation settings for the Ceph-CSI daemons in the current Kubernetes cluster. CSI key rotation can affect existing PV connections, so take care when exercising this option.

Daemon configures CephX key settings for local Ceph daemons managed by Rook and part of the Ceph cluster. Daemon CephX keys can be rotated without affecting client connections.

RBDMirrorPeer configures CephX key settings of the rbd-mirror-peer user that is used for creating bootstrap peer token used connect peer clusters. Rotating the rbd-mirror-peer user key will update the mirror peer token. Rotation will affect any existing peers connected to this cluster, so take care when exercising this option.

KeepPriorKeyCountMax tells Rook how many prior keys to keep active. Generally, this would be set to 1 to allow for a migration period for applications. If desired, set this to 0 to delete prior keys after migration. This config only applies to prior keys that already exist. If PriorKeyCount is set to 2 while only a single key currently exists, only a single prior key will be kept, and the reported status will only indicate the actual number of prior keys, not necessarily a reflection of PriorKeyCount config here.

KeyGeneration specifies the desired CephX key generation. This is used when KeyRotationPolicy is KeyGeneration and ignored for other policies. If this is set to greater than the current key generation, relevant keys will be rotated, and the generation value will be updated to this new value (generation values are not necessarily incremental, though that is the intended use case). If this is set to less than or equal to the current key generation, keys are not rotated.

KeyRotationPolicy controls if and when CephX keys are rotated after initial creation. One of Disabled, or KeyGeneration. Default Disabled.

KeyGeneration specifies the desired CephX key generation. This is used when KeyRotationPolicy is KeyGeneration and ignored for other policies. If this is set to greater than the current key generation, relevant keys will be rotated, and the generation value will be updated to this new value (generation values are not necessarily incremental, though that is the intended use case). If this is set to less than or equal to the current key generation, keys are not rotated.

KeyRotationPolicy controls if and when CephX keys are rotated after initial creation. One of Disabled, or KeyGeneration. Default Disabled.

KeyGeneration specifies the desired CephX key generation. This is used when KeyRotationPolicy is KeyGeneration and ignored for other policies. If this is set to greater than the current key generation, relevant keys will be rotated, and the generation value will be updated to this new value (generation values are not necessarily incremental, though that is the intended use case). If this is set to less than or equal to the current key generation, keys are not rotated.

KeyRotationPolicy controls if and when CephX keys are rotated after initial creation. One of Disabled, or KeyGeneration. Default Disabled.

Enabled represents whether the key rotation is enabled.

Schedule represents the cron schedule for key rotation.

ConnectionDetails contains the KMS connection details (address, port etc)

TokenSecretName is the kubernetes secret containing the KMS token

Whether to allow updating the device class after the OSD is initially provisioned

Whether Rook will resize the OSD CRUSH weight when the OSD PVC size is increased. This allows cluster data to be rebalanced to make most effective use of new OSD space. The default is false since data rebalancing can cause temporary cluster slowdown.

BackfillFullRatio is the ratio at which the cluster is too full for backfill. Backfill will be disabled if above this threshold. Default is 0.90.

A regular expression to allow more fine-grained selection of devices on nodes across the cluster

A regular expression to allow more fine-grained selection of devices with path names

List of devices to use as storage devices

FlappingRestartIntervalHours defines the time for which the OSD pods, that failed with zero exit code, will sleep before restarting. This is needed for OSD flapping where OSD daemons are marked down more than 5 times in 600 seconds by Ceph. Preventing the OSD pods to restart immediately in such scenarios will prevent Rook from marking OSD as up and thus peering of the PGs mapped to the OSD. User needs to manually restart the OSD pod if they manage to fix the underlying OSD flapping issue before the restart interval. The sleep will be disabled if this interval is set to 0.

FullRatio is the ratio at which the cluster is considered full and ceph will stop accepting writes. Default is 0.95.

Migration handles the OSD migration

NearFullRatio is the ratio at which the cluster is considered nearly full and will raise a ceph health warning. Default is 0.85.

The maximum number of OSDs to update in parallel.

Whether to always schedule OSDs on a node even if the node is not currently scheduleable or ready

OSDStore is the backend storage type used for creating the OSDs

Whether to consume all the storage devices found on a machine

PersistentVolumeClaims to use as storage

A user confirmation to migrate the OSDs. It destroys each OSD one at a time, cleans up the backing disk and prepares OSD with same ID on that disk

A regular expression to allow more fine-grained selection of devices on nodes across the cluster

A regular expression to allow more fine-grained selection of devices with path names

List of devices to use as storage devices

ResourceRequirements describes the compute resource requirements.

Whether to consume all the storage devices found on a machine

PersistentVolumeClaims to use as storage

Claims lists the names of resources, defined in spec.resourceClaims, that are used by this container.

This field depends on the DynamicResourceAllocation feature gate.

This field is immutable. It can only be set for containers.

Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

Name must match the name of one entry in pod.spec.resourceClaims of the Pod where this field is used. It makes that resource available inside a container.

Request is the name chosen for a request in the referenced claim. If empty, everything from the claim is made available, otherwise only the result of this request.

Standard object's metadata. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

spec defines the desired characteristics of a volume requested by a pod author. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#persistentvolumeclaims

accessModes contains the desired access modes the volume should have. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#access-modes-1

dataSource field can be used to specify either:

• An existing VolumeSnapshot object (snapshot.storage.k8s.io/VolumeSnapshot)
• An existing PVC (PersistentVolumeClaim) If the provisioner or an external controller can support the specified data source, it will create a new volume based on the contents of the specified data source. When the AnyVolumeDataSource feature gate is enabled, dataSource contents will be copied to dataSourceRef, and dataSourceRef contents will be copied to dataSource when dataSourceRef.namespace is not specified. If the namespace is specified, then dataSourceRef will not be copied to dataSource.

dataSourceRef specifies the object from which to populate the volume with data, if a non-empty volume is desired. This may be any object from a non-empty API group (non core object) or a PersistentVolumeClaim object. When this field is specified, volume binding will only succeed if the type of the specified object matches some installed volume populator or dynamic provisioner. This field will replace the functionality of the dataSource field and as such if both fields are non-empty, they must have the same value. For backwards compatibility, when namespace isn't specified in dataSourceRef, both fields (dataSource and dataSourceRef) will be set to the same value automatically if one of them is empty and the other is non-empty. When namespace is specified in dataSourceRef, dataSource isn't set to the same value and must be empty. There are three important differences between dataSource and dataSourceRef:

• While dataSource only allows two specific types of objects, dataSourceRef allows any non-core object, as well as PersistentVolumeClaim objects.
• While dataSource ignores disallowed values (dropping them), dataSourceRef preserves all values, and generates an error if a disallowed value is specified.
• While dataSource only allows local objects, dataSourceRef allows objects in any namespaces. (Beta) Using this field requires the AnyVolumeDataSource feature gate to be enabled. (Alpha) Using the namespace field of dataSourceRef requires the CrossNamespaceVolumeDataSource feature gate to be enabled.

resources represents the minimum resources the volume should have. Users are allowed to specify resource requirements that are lower than previous value but must still be higher than capacity recorded in the status field of the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#resources

selector is a label query over volumes to consider for binding.

storageClassName is the name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1

volumeAttributesClassName may be used to set the VolumeAttributesClass used by this claim. If specified, the CSI driver will create or update the volume with the attributes defined in the corresponding VolumeAttributesClass. This has a different purpose than storageClassName, it can be changed after the claim is created. An empty string or nil value indicates that no VolumeAttributesClass will be applied to the claim. If the claim enters an Infeasible error state, this field can be reset to its previous value (including nil) to cancel the modification. If the resource referred to by volumeAttributesClass does not exist, this PersistentVolumeClaim will be set to a Pending state, as reflected by the modifyVolumeStatus field, until such as a resource exists. More info: https://kubernetes.io/docs/concepts/storage/volume-attributes-classes/

volumeMode defines what type of volume is required by the claim. Value of Filesystem is implied when not included in claim spec.

volumeName is the binding reference to the PersistentVolume backing this claim.

APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required.

Kind is the type of resource being referenced

Name is the name of resource being referenced

APIGroup is the group for the resource being referenced. If APIGroup is not specified, the specified Kind must be in the core API group. For any other third-party types, APIGroup is required.

Kind is the type of resource being referenced

Name is the name of resource being referenced

Namespace is the namespace of resource being referenced Note that when a namespace is specified, a gateway.networking.k8s.io/ReferenceGrant object is required in the referent namespace to allow that namespace's owner to accept the reference. See the ReferenceGrant documentation for details. (Alpha) This field requires the CrossNamespaceVolumeDataSource feature gate to be enabled.

Limits describes the maximum amount of compute resources allowed. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

Requests describes the minimum amount of compute resources required. If Requests is omitted for a container, it defaults to Limits if that is explicitly specified, otherwise to an implementation-defined value. Requests cannot exceed Limits. More info: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

matchExpressions is a list of label selector requirements. The requirements are ANDed.

matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.

key is the label key that the selector applies to.

operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.

values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.

Provider-specific device configuration

Count is the number of devices in this set

Whether to encrypt the deviceSet

Name is a unique identifier for the set

Portable represents OSD portability across the hosts

ResourceRequirements describes the compute resource requirements.

Scheduler name for OSD pod placement

TuneSlowDeviceClass Tune the OSD when running on a slow Device Class

TuneFastDeviceClass Tune the OSD when running on a fast Device Class

VolumeClaimTemplates is a list of PVC templates for the underlying storage devices