Ceph S3

To log into the Ceph control pod:

West Control Pod:

kubectl exec -it -n rook $(kubectl get pods -n rook --selector=app=rook-ceph-tools --output=jsonpath={.items..metadata.name}) -- bash

East Control Pod:

kubectl exec -it -n rook-east $(kubectl get pods -n rook-east --selector=app=rook-ceph-tools --output=jsonpath={.items..metadata.name}) -- bash

Central Control Pod:

kubectl exec -it -n rook-central $(kubectl get pods -n rook-central --selector=app=rook-ceph-tools --output=jsonpath={.items..metadata.name}) -- bash

Adding Users

Once logged into the Ceph control pod, run the command to add a user:

West Pool:

radosgw-admin --rgw-realm=nautiluss3 --rgw-zone=nautiluss3 --rgw-zonegroup=nautiluss3 user create --uid <uid> --display-name "<name>" --email "<email>"

East Pool:

radosgw-admin --rgw-realm=easts3 --rgw-zone=easts3 --rgw-zonegroup=easts3 user create --uid <uid> --display-name "<name>" --email "<email>"

Central Pool:

radosgw-admin --rgw-realm=centrals3 --rgw-zone=centrals3 --rgw-zonegroup=centrals3 user create --uid <uid> --display-name "<name>" --email "<email>"

HA realm (from the West control pod; the user replicates to all zones):

radosgw-admin --rgw-realm=ha --rgw-zonegroup=ha --rgw-zone=ha-west user create --uid <uid> --display-name "<name>" --email "<email>"

The access_key and secret_key is in the output from the above command.

If the request is from the Nautilus Support channel, use the user’s CiLogon id/Authentik username as the uid, name as name and the email address as email.

Deleting Users’ Buckets

Step 1: Link the Bucket

radosgw-admin bucket link --uid=USER --bucket=BUCKET

• Replace USER with the admin’s ID.
• Replace BUCKET with the name of the user’s bucket.

Step 2: Delete All Objects in the Bucket

rclone purge S3:bucket-name --checkers-128 --progress

• --checkers 128: Number of simultaneous checks.
• --progress: Shows progress.

Step 3: If the bucket failed to be removed (some files are corrupted), exec into the tools pod and run:

radosgw-admin bucket rm --bucket=bucket-name --purge-objects --bypass-gc

This command is slower than rclone, but deletes broken stuff too.

Cleaning up

List the buckets sorted by number of files:

radosgw-admin bucket limit check | jq -r '[.[] | .user_id as $uid | .buckets[] | {user: $uid, bucket_name: .bucket, num_objects: .num_objects}] | sort_by(.num_objects) | .[] | "\(.num_objects) \(.user)/\(.bucket_name)"' | sort -nr

Largest can be purged if need space, empty ones are probably abandoned.

Listing the buckets that need to be resharded

There’s a limit to the automatic reshard, after that needs manual. WARN is fine.

radosgw-admin bucket limit check | jq '.[] as $user | $user.buckets[] | select(.fill_status != "OK") | {user_id: $user.user_id, bucket: .bucket, fill_status: .fill_status}'

Finding orphans

rgw-orphan-list tool

rgw-orphan-list is an experimental tool that will run the below commands and help finding orphans. It did npt find all orphans for me, thus this guide.

Deep exploration of S3 pools

Listing bucket instances

This will provide the bucket instances - concrete physical instance of a bucket used for metadata.

radosgw-admin metadata list bucket.instance --max-entries=10000 \
    | jq -r '.keys | .[]' > bucket_instances.txt

Getting bucket markers

This will provide the bucket markers - those seem to be a modified bucket instances, and will be used further instead of instances.

radosgw-admin bucket stats | jq -r '.[] | "\(.bucket):\(.marker)"' > bucket_markers.txt

Listing objects by bucket marker.

This will get objects from the data pool - including all orphan objects, that might be not referenced by index pool. This takes tens of GB, so better put on a linstor volume.

rados ls -p nautiluss3.rgw.buckets.data > rados.intermediate

Get the numbers of objects by bucket marker

This is slow - it needs to sort millions of lines. Also make sure you have enough space in /tmp to sort.

awk -F_ '{print $1}' rados.intermediate | sort | \
    uniq -c | sort -nr > buckets_numbers.txt

Finding orphaned bucket instances (by marker) in the rados objects list from data pool

comm -23 \
    <(awk '{print $2}' buckets_numbers.txt | sort) \
    <(awk -F: '{print $2}' bucket_markers.txt | sort)

Do the above and get the number of orphaned objects per orphaned bucket instance

comm -23 \
  <(awk '{print $2}' buckets_numbers.txt | sort) \
  <(awk -F: '{print $2}' bucket_markers.txt | sort) \
  | grep -F -f - buckets_numbers.txt

Finding the total number of orphaned files

comm -23 \
  <(awk '{print $2}' buckets_numbers.txt | sort) \
  <(awk -F: '{print $2}' bucket_markers.txt | sort) \
  | grep -F -f - buckets_numbers.txt \
  | awk '{for(i=1;i<=NF;i++) sum+=$i} END {print sum}'

For each missing marker, get the files in a separate file for review

grep --binary-file=text <one_of_markers> rados.intermediate > <one_of_markers>.from.rados.intermediate

Check the object info directly from the data pool

rados -p nautiluss3.rgw.buckets.data stat object_id

Remove the orphaned files for one marker

cat <marker_from_above>.from.rados.intermediate | xargs -I{} -P16 rados rm -p nautiluss3.rgw.buckets.data "{}"

Multisite HA realm

A fourth RGW realm, ha, spans all three Ceph clusters as one zonegroup with three zones: ha-west in rook (master), ha-central in rook-central, ha-east in rook-east. Every bucket in it replicates to all three sites, and s3-ha.nrp-nautilus.io fails over across them, west first; the endpoints are listed on the user page. It is for mission-critical NRP services: the S3 token portal does not issue keys for it, so admins create its users (Adding Users).

• nautiluss3, centrals3 and easts3 are untouched. The realm shares only the RADOS clusters and .rgw.root; its users, buckets, pools (ha-<zone>.rgw.*) and daemons are separate.
• Each zone stores 3+2 erasure-coded data and all three keep a copy, so every byte costs about 5x raw capacity. The ceiling is the smallest free data pool of the three (west, about 127 TiB in September 2026). There are no quotas; gate by credential issuance.
• Bucket and user create/delete go to the master zone. If west is down, object reads and writes fail over, but those metadata operations fail until west returns or another zone is promoted.
• s3-ha.nrp-nautilus.io probes GET /healthcheck/ok on every zone (a public object in bucket healthcheck, owned by user healthcheck, replicated like any other); a zone whose data pool cannot serve it is dropped within 15 seconds. Do not delete that bucket or user.
• The probe reads one object, so it catches a zone whose storage is gone, not a partial OSD outage that leaves that object’s PG active. The per-zone rook-ceph-rgw-ha.<namespace>:7480 URLs are single-zone; only s3-ha fails over.
• To test failover or hold west out of rotation, delete west’s RADOS copy of the probe head (rados -p ha-west.rgw.buckets.data rm <healthcheck bucket marker>_ok); a new PUT of healthcheck/ok through s3-ha lands on the active zone and syncs back.
• The resharding zonegroup feature is enabled, so dynamic resharding works; each zone’s two RGWs have a PodDisruptionBudget rook-ceph-rgw-ha (minAvailable: 1).

From the West control pod, radosgw-admin sync status --rgw-realm=ha shows sync state, and radosgw-admin zonegroup get --rgw-realm=ha --rgw-zonegroup=ha must show sync_from_all: true on every zone.