Ceph S3

To log into the Ceph control pod:

West Control Pod:

kubectl exec -it -n rook $(kubectl get pods -n rook --selector=app=rook-ceph-tools --output=jsonpath={.items..metadata.name}) -- bash

East Control Pod:

kubectl exec -it -n rook-east $(kubectl get pods -n rook-east --selector=app=rook-ceph-tools --output=jsonpath={.items..metadata.name}) -- bash

Central Control Pod:

kubectl exec -it -n rook-central $(kubectl get pods -n rook-central --selector=app=rook-ceph-tools --output=jsonpath={.items..metadata.name}) -- bash

Adding Users

Once logged into the Ceph control pod, run the command to add a user:

West Pool:

radosgw-admin --rgw-realm=nautiluss3 --rgw-zone=nautiluss3 --rgw-zonegroup=nautiluss3 user create --uid <uid> --display-name "<name>" --email "<email>"

East Pool:

radosgw-admin --rgw-realm=easts3 --rgw-zone=easts3 --rgw-zonegroup=easts3 user create --uid <uid> --display-name "<name>" --email "<email>"

Central Pool:

radosgw-admin --rgw-realm=centrals3 --rgw-zone=centrals3 --rgw-zonegroup=centrals3 user create --uid <uid> --display-name "<name>" --email "<email>"

HA realm (from the West control pod; the user replicates to all zones):

radosgw-admin --rgw-realm=ha --rgw-zonegroup=ha --rgw-zone=ha-west user create --uid <uid> --display-name "<name>" --email "<email>"

The access_key and secret_key is in the output from the above command.

If the request is from the Nautilus Support channel, use the user’s CiLogon id/Authentik username as the uid, name as name and the email address as email.

Deleting Users’ Buckets

Step 1: Link the Bucket

radosgw-admin bucket link --uid=USER --bucket=BUCKET

• Replace USER with the admin’s ID.
• Replace BUCKET with the name of the user’s bucket.

Step 2: Delete All Objects in the Bucket

rclone purge S3:bucket-name --checkers-128 --progress

• --checkers 128: Number of simultaneous checks.
• --progress: Shows progress.

Step 3: If the bucket failed to be removed (some files are corrupted), exec into the tools pod and run:

radosgw-admin bucket rm --bucket=bucket-name --purge-objects --bypass-gc

This command is slower than rclone, but deletes broken stuff too.

Cleaning up

List the buckets sorted by number of files:

radosgw-admin bucket limit check | jq -r '[.[] | .user_id as $uid | .buckets[] | {user: $uid, bucket_name: .bucket, num_objects: .num_objects}] | sort_by(.num_objects) | .[] | "\(.num_objects) \(.user)/\(.bucket_name)"' | sort -nr

Largest can be purged if need space, empty ones are probably abandoned.

Listing the buckets that need to be resharded

There’s a limit to the automatic reshard, after that needs manual. WARN is fine.

radosgw-admin bucket limit check | jq '.[] as $user | $user.buckets[] | select(.fill_status != "OK") | {user_id: $user.user_id, bucket: .bucket, fill_status: .fill_status}'

Finding orphans

rgw-orphan-list tool

rgw-orphan-list is an experimental tool that will run the below commands and help finding orphans. It did npt find all orphans for me, thus this guide.

Deep exploration of S3 pools

Listing bucket instances

This will provide the bucket instances - concrete physical instance of a bucket used for metadata.

radosgw-admin metadata list bucket.instance --max-entries=10000 \
    | jq -r '.keys | .[]' > bucket_instances.txt

Getting bucket markers

This will provide the bucket markers - those seem to be a modified bucket instances, and will be used further instead of instances.

radosgw-admin bucket stats | jq -r '.[] | "\(.bucket):\(.marker)"' > bucket_markers.txt

Listing objects by bucket marker.

This will get objects from the data pool - including all orphan objects, that might be not referenced by index pool. This takes tens of GB, so better put on a linstor volume.

rados ls -p nautiluss3.rgw.buckets.data > rados.intermediate

Get the numbers of objects by bucket marker

This is slow - it needs to sort millions of lines. Also make sure you have enough space in /tmp to sort.

awk -F_ '{print $1}' rados.intermediate | sort | \
    uniq -c | sort -nr > buckets_numbers.txt

Finding orphaned bucket instances (by marker) in the rados objects list from data pool

comm -23 \
    <(awk '{print $2}' buckets_numbers.txt | sort) \
    <(awk -F: '{print $2}' bucket_markers.txt | sort)

Do the above and get the number of orphaned objects per orphaned bucket instance

comm -23 \
  <(awk '{print $2}' buckets_numbers.txt | sort) \
  <(awk -F: '{print $2}' bucket_markers.txt | sort) \
  | grep -F -f - buckets_numbers.txt

Finding the total number of orphaned files

comm -23 \
  <(awk '{print $2}' buckets_numbers.txt | sort) \
  <(awk -F: '{print $2}' bucket_markers.txt | sort) \
  | grep -F -f - buckets_numbers.txt \
  | awk '{for(i=1;i<=NF;i++) sum+=$i} END {print sum}'

For each missing marker, get the files in a separate file for review

grep --binary-file=text <one_of_markers> rados.intermediate > <one_of_markers>.from.rados.intermediate

Check the object info directly from the data pool

rados -p nautiluss3.rgw.buckets.data stat object_id

Remove the orphaned files for one marker

cat <marker_from_above>.from.rados.intermediate | xargs -I{} -P16 rados rm -p nautiluss3.rgw.buckets.data "{}"

Multisite HA realm

A fourth RGW realm, ha, spans all three Ceph clusters as one zonegroup with three zones: ha-west in rook (master), ha-central in rook-central, ha-east in rook-east. Every bucket in it replicates to all three sites, and s3-ha.nrp-nautilus.io fails over across them, west first; the endpoints are listed on the user page. It is for mission-critical NRP services: the S3 token portal does not issue keys for it, so admins create its users (Adding Users).

• nautiluss3, centrals3 and easts3 are untouched. The realm shares only the RADOS clusters and .rgw.root; its users, buckets, pools (ha-<zone>.rgw.*) and daemons are separate.
• Each zone stores 3+2 erasure-coded data and all three keep a copy, so every byte costs about 5x raw capacity. The ceiling is the smallest free data pool of the three (west, about 127 TiB in September 2026). There are no quotas; gate by credential issuance.
• Bucket and user create/delete go to the master zone. If west is down, object reads and writes fail over, but those metadata operations fail until west returns or another zone is promoted.
• s3-ha.nrp-nautilus.io is the rook/ha-s3 ingress. Its backend Service, ha-s3-zones, has no endpoints on purpose: every server is listed in the config-backend annotation. ha-west takes the traffic, and ha-central and ha-east are backup servers, used in that order while west is down. Each of the three is probed through its zone’s ha-zone-health service (the addr and port on its server line). The ha-*-plain servers probe the RGW directly and are a last resort, used only while all three gated servers are down. A server leaves rotation after 2 failed probes (10 seconds); central and east come back after 6 good ones (30 seconds), west after 60 (5 minutes), so that sync from central can catch up first. GET and HEAD give up after 120 seconds of server silence and are retried; uploads keep the 6-hour timeout.
• ha-zone-health (a Deployment, a ConfigMap holding zone_health.py, and a Service in each of rook, rook-central and rook-east, running on the nodes of that zone’s HA RGW pods) answers 200 only when every PG of the zone’s ha-<zone>.* pools is active, read from the zone’s mgr metrics after a 20-second grace for peering, and the zone’s RGW serves GET /healthcheck/ok, down after 3 failures in a row. That object is public, in bucket healthcheck owned by user healthcheck, and replicated like any other; do not delete that bucket or user. GET /status on port 8080 shows the pool counts and the reason. The addr on each gated server line is that zone’s ha-zone-health ClusterIP, so if the Service is recreated, update the line.
• The ingress, ha-s3-zones and ha-zone-health are changed live; nothing applies them from a repository. prp/rook holds an exported copy of them and of the rest of the Rook configuration, written by its tools/export.py: after a change, run it and commit the result.
• On 2026-10-28 the OSDs of the two ucr hosts of rook stayed down for six hours after a reboot, because their OSD pods could not schedule past leftover NoSchedule taints (Node Management). 2 of the 32 PGs of ha-west.rgw.buckets.data (3+2, min_size 4, one shard per host) lost a shard on each host and went inactive. The probe then in use read only healthcheck/ok, which still answered, so large objects such as container image layers hung while west stayed in rotation; that is why each zone is now gated by its PG state. The per-zone rook-ceph-rgw-ha.<namespace>:7480 URLs are single-zone; only s3-ha fails over.
• Deleting west’s RADOS copy of the probe head (rados -p ha-west.rgw.buckets.data rm <healthcheck bucket marker>_ok) also drops west, through both its gated and its plain probe; a new PUT of healthcheck/ok through s3-ha lands on the active zone and syncs back.
• The resharding zonegroup feature is enabled, so dynamic resharding works; each zone’s two RGWs have a PodDisruptionBudget rook-ceph-rgw-ha (minAvailable: 1).

Hold a zone out of rotation, and bring it back:

kubectl -n rook scale deploy/ha-zone-health --replicas=0   # west leaves rotation within about 10 seconds
kubectl -n rook scale deploy/ha-zone-health --replicas=2   # west takes traffic back after 5 healthy minutes

Use -n rook-central or -n rook-east for the other zones; with all three at 0, the plain servers serve, west first. Before scaling west back up, check that ceph pg dump_stuck inactive on west lists nothing and that radosgw-admin sync status on west reports data is caught up with source for both other zones, because writes made during the failover land on central.

From the West control pod, radosgw-admin sync status --rgw-realm=ha shows sync state (run it on west: on central and east it waits on the master zone and hangs while west is degraded), and radosgw-admin zonegroup get --rgw-realm=ha --rgw-zonegroup=ha must show sync_from_all: true on every zone.