Backup costs can creep up when daily archives accumulate in object storage. Unnecessarily expensive storage classes, missing lifecycle rules, and forgotten temporary folders can all add to the bill.
This guide covers rclone backups to S3 and Backblaze B2, including encryption, version retention, archive classes and recovery tests. Examples assume GNU/Linux, an installed rclone release and completed, validated backup artifacts. They use AWS S3 and the native B2 backend; those backends have different authentication and encryption options. Replace sample paths, names and credentials for your test environment.
What rclone does in an object-storage backup workflow
rclone transfers files across storage providers. copy leaves destination-only files in place but can update changed files with the same name. sync makes the destination match the source, including deletions. Historical backups therefore need unique artifact names and deliberate retention; a mirror can propagate an accidental source deletion. Preview a sync with --dry-run in a separate mirror prefix before considering it for live use.
Start with the recovery requirement: which files, database state and application version must return together, how much data loss is acceptable, and how long can recovery take? rclone cannot make a changing source tree transactionally consistent. Produce a database-aware backup or consistent snapshot first, then transfer the completed result.
Encryption protects different parts of the workflow, while lifecycle controls how objects age. S3 can transition objects between storage classes; native B2 lifecycle uses age-based hiding and deletion of versions. Neither provider automatically identifies which daily backup should become your retained monthly snapshot.
Plan encryption and access needed during a restore
Client-side encryption, server-side encryption and TLS
SSE encrypts data within the provider’s storage system. TLS protects transport. An rclone crypt remote encrypts data locally before upload and decrypts it locally after download, with separately configured filename behavior. Use the layers that fit the requirement and preserve the keys, any crypt salt and configuration for as long as the encrypted backups remain needed. SSE does not conceal plaintext from the provider.
S3 encryption modes and their dependencies
AWS S3 encrypts new uploads with SSE-S3 by default. Other S3-compatible services can have different defaults and supported options. Explicit rclone flags must agree with bucket policy; an explicit AES256 request is not a request to use the bucket’s KMS key.
SSE-S3 uses S3-managed keys. rclone’s --s3-server-side-encryption AES256 requests this mode. Changing a bucket’s encryption default does not rewrite objects that already exist; document the encryption of the actual backup objects, not just today’s default.
SSE-KMS uses AWS KMS and its key policies. rclone uses --s3-server-side-encryption aws:kms with --s3-sse-kms-key-id. The key must be available and usable by the restore identity. Uploads need kms:GenerateDataKey, downloads need kms:Decrypt, and multipart uploads require both; ordinary S3 permissions alone do not provide key access.
SSE-C requires customer-provided keys wherever the provider and bucket permit it. rclone has customer-key options, but this is a separate key-management design rather than a drop-in default. Supply secrets through protected configuration or a secret system, not visible command arguments. Losing a required key makes its encrypted backup unusable.
For a bucket-default workflow, omit conflicting per-upload encryption settings and verify a new object’s actual encryption after the test upload. A default setting is not a substitute for a policy enforcing the required mode. Keep encryption configuration, access roles and key recovery instructions in the runbook.
Test downloads with the identity that will perform disaster recovery, including from a fresh machine. Do not assume that an identity allowed to upload can decrypt or restore. Keep an independent route to required keys when the original server is unavailable.
Native B2 encryption without mixing backend flags
For the native B2 examples, enable SSE-B2 as the bucket’s default encryption in the Backblaze console or API, then verify a newly uploaded object. It applies to new uploads or copies, not existing objects; an explicitly selected SSE-C mode has different key requirements. Native B2 has its own SSE-C settings. Do not pass S3 encryption flags to the native backend. The Backblaze encryption guide documents both APIs and their differences.
Record which bucket default and object-level mode were in force when each backup was written. Store any customer-provided key separately from the backup data and restrict its access. A change to the bucket default does not retroactively change earlier versions.
Lifecycle rules and version retention
S3 transitions, expiration and old versions
A retention policy can prevent temporary or obsolete backups from accumulating indefinitely. Apply rules to explicit prefixes such as prod/logs/, prod/daily/ and prod/monthly/, and check existing objects before enabling a rule. Age alone does not establish whether an object is no longer needed.
S3 lifecycle actions operate on objects and versions, not on the logical dependency chain of a database backup. Keep all required manifests, base backups and logs together for the intended recovery window.
A transition changes storage class. GLACIER_IR is Glacier Instant Retrieval; GLACIER is Glacier Flexible Retrieval; DEEP_ARCHIVE is Glacier Deep Archive. They have different access and cost behavior. The AWS storage classes overview describes those differences.
Expiration in an unversioned bucket schedules permanent object removal. In a versioning-enabled bucket, expiration of the current object generally creates a delete marker and leaves a noncurrent version. It does not automatically remove all billable historical data, and execution is asynchronous rather than an exact-time deletion guarantee.
Noncurrent-version expiration needs its own policy. Expired delete markers and incomplete multipart uploads also require appropriate handling. Object Lock or legal holds may prevent deletion. Inspect the effective policy and object state instead of assuming that a current-object expiration removed every version.
Setting --s3-storage-class chooses a class on upload; lifecycle can change it later. Standard-IA has a 30-day minimum storage charge and a minimum billable object size. Intelligent-Tiering has monitoring and object-size considerations. Starting with Standard and measuring actual access can be more useful than assuming any one class always saves money.
B2 hiding and deletion are age-based controls
B2 lifecycle uses fileNamePrefix with ages such as daysFromUploadingToHiding and daysFromHidingToDeleting. These age-based rules do not provide a “keep the last N versions” control. A replaced version becomes hidden, so its deletion age can start when a newer version is uploaded, rather than at its original upload date.
The native B2 backend normally hides files on deletion. hard_delete=true permanently deletes the addressed version; it does not enable versioning or automatically purge all historical versions. Keep the default false for these examples and review lifecycle separately. Object Lock can protect particular versions under its configured retention or legal hold, but merely enabling the bucket feature does not prove every object is protected.
Choose Glacier classes against recovery time and total cost
Minimum duration and access behavior
Glacier Instant Retrieval supports immediate GET access and has a 90-day minimum storage duration. Flexible Retrieval also has a 90-day minimum but requires an archive restore before downloading. Deep Archive requires a restore too and has a 180-day minimum. Deleting, overwriting or transitioning sooner can still incur the remaining minimum-duration charge.
Evaluate one explicit schedule against the required recovery window. For example, daily objects could remain in Standard for 30 days, enter Glacier Instant Retrieval and expire at day 150, giving that class 120 days. A separate monthly copy could enter Deep Archive at day 30 and expire at day 365. These are illustrative policies, not savings or compliance guarantees; monthly selection must be performed by a separate job.
Request archive retrieval before a download
rclone can upload into an archive class or allow lifecycle to transition the object. For Flexible Retrieval and Deep Archive, use the S3 backend restore operation to request temporary readable data and wait for completion before copy or copyto. Glacier Instant Retrieval does not need this preparation. The lifetime of the temporary restored copy is different from the retention of the archived object.
Standard retrieval is typically 3–5 hours for Flexible Retrieval and within 12 hours for Deep Archive; Deep Archive Bulk is typically within 48 hours and has no Expedited option. Large datasets can take longer. Retrieval requests, temporary storage, download and rebuilding the application add time or cost beyond the archive retrieval itself. Measure the full operation against your RTO.
Build a repeatable artifact and upload workflow
Use separate daily and monthly namespaces
Give each completed backup a unique name containing its timestamp or job ID, for example app-YYYYMMDDTHHMMSSZ.tar.zst. Stage unfinished output outside the directory watched by the upload job. Only move it into /srv/backups/ready after the producer and integrity checks succeed.
Use prod/daily/ for routine artifacts and prod/monthly/ for deliberately selected monthly copies. A lifecycle prefix does not choose the last successful daily of the month. The selection job should verify that backup, copy it with a unique monthly name, verify the copy and record it in the catalog.
Configure credentials and named remotes
For these examples, use the following remote definitions in the file shown by rclone config file, preserving any existing configuration. Restrict the file to the backup account. S3 runtime credentials require env_auth=true and no competing explicit keys in the remote. Native B2 uses a separate credential mapping:
[backup-s3]
type = s3
provider = AWS
env_auth = true
region = eu-central-1
server_side_encryption = AES256
[backup-b2]
type = b2
hard_delete = false
Supply AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, plus the matching AWS_SESSION_TOKEN for temporary credentials, securely to the process. Native B2 uses RCLONE_B2_ACCOUNT for the application key ID and RCLONE_B2_KEY for the application key. The AWS bucket must already exist in eu-central-1. A scheduled job must receive the same protected configuration and runtime credentials; it does not inherit an interactive terminal’s environment.
The source directory below contains only complete, validated artifacts. A tar archive of a changing database directory is not a database backup. For applications such as WordPress, coordinate the database export with the file snapshot. Preserve a checksum manifest and the application restore procedure alongside the catalog.
Copy the completed artifacts to the S3 daily prefix. The example requests SSE-S3; align it with your actual encryption policy. Existing destination-only objects remain, but same-name changes can be overwritten:
rclone copy /srv/backups/ready backup-s3:mybucket/prod/daily --checksum --transfers 4 --checkers 8 --s3-storage-class STANDARD
With bucket default SSE-B2 enabled, the native B2 equivalent uses the backend’s own credentials and no S3-specific flags:
rclone copy /srv/backups/ready backup-b2:my-b2-bucket/prod/daily --transfers 4 --checkers 8
Require a successful transfer exit code and an appropriate verification result before recording the remote copy as available. The source producer’s failure must fail the whole job. If streaming with rcat is needed, Bash pipefail catches a failed producer, but a partial remote object may remain. Keep it outside the usable-backup catalog and verify it before promotion; the original input stream cannot simply be replayed for a whole-upload retry.
Review an explicit policy before enabling it
For log archives kept for a year, this separate example transitions prod/logs/ to Standard-IA at object age 30 days and Flexible Retrieval at 90 days, with expiration at 365 days. That differs from the earlier daily Instant Retrieval example and must target its own prefix. New or updated S3 lifecycle policies normally leave objects smaller than 128 KB untransitioned; transition requests and archive metadata may cost more than they save.
{
"Rules": [
{
"ID": "prod-logs",
"Filter": {
"Prefix": "prod/logs/"
},
"Status": "Enabled",
"Transitions": [
{
"Days": 30,
"StorageClass": "STANDARD_IA"
},
{
"Days": 90,
"StorageClass": "GLACIER"
}
],
"Expiration": {
"Days": 365
}
}
]
}For a B2 daily prefix, one illustrative policy hides objects after 45 days from upload and deletes them 7 days after hiding. Replaced versions may have become hidden earlier, and lifecycle processing is not instantaneous. Keep the monthly prefix outside that rule and avoid overlapping broader rules that shorten its effective retention. Copy a chosen, verified daily into the monthly prefix before its original copy expires.
Common problems to prevent
Encryption, but keep your keys where your future self can find them
A restore identity can have S3 read permission yet fail on KMS decryption. Check both IAM and key policy, key availability and the correct region/account. Verify the actual recovery identity on a fresh machine; do not assume that an upload-only test established the download path.
For B2 SSE-C, retain the customer key independently and keep its version mapping with the object catalog. Bucket default SSE-B2 removes that customer-key dependency but still requires working account access. Neither mode replaces a tested application restore.
Storage class and lifecycle assumptions
If yesterday’s backup must be usable within minutes, do not make a Flexible Retrieval or Deep Archive request the first step in that recovery. Consider how much recent history needs immediate access and whether a separate archived copy serves the long-term requirement. The minimum storage charge still applies even if lifecycle removes an object sooner.
Multipart uploads and small files
Bundling small files can reduce requests, while increasing the amount that must be read for one-file recovery. Compression and large multipart buffers also consume CPU and RAM. S3’s multipart buffer budget depends on transfers, upload concurrency and chunk size; B2 large uploads have their own chunk buffers. Increase concurrency only after measuring headroom, not as an unconditional performance fix.
Checksum strategy and verification
rclone check compares sizes and usable common hashes; inspect its report for missing hash support. S3 ETags are not universally MD5 hashes, particularly for multipart or some encrypted objects. Native B2 commonly uses SHA1, so a cross-provider S3/B2 comparison may lack a shared hash. Use a trusted checksum manifest or a selected --download check, accepting its read/transfer costs. Size-only equality is not content equality.
Network hiccups and concurrency
Start with modest transfer and checker counts, then monitor throughput and memory. --fast-list trades RAM for fewer listing requests where supported. rclone bandwidth limits are bytes per second: 50M means 50 MiB/s, not 50 Mbit/s. Keep retries enabled where appropriate and alert on jobs that still fail or miss the completion deadline.
Test recovery from a fresh environment
A useful drill restores a known backup into an isolated VM, validates the expected data and starts the application without affecting production. Include the database, files, required software versions and controlled secrets. Keep outbound integrations and scheduled application jobs disabled until validation is complete.
Record the artifact and object version, encryption mode, identity, archive class, retrieval priority, elapsed retrieval/download/rebuild times and validation result. Test the credentials and keys as well as the files. Keep a known working recovery path while changing lifecycle or encryption settings.
Map those measurements to a disaster recovery plan with RTO, RPO and runbooks. A successful object download is only one part of restoring a working service.
Upload and restore examples with consistent paths
A single completed artifact to S3
Select a completed, locally verified artifact from the catalog; app-YYYYMMDD.tar.zst below is a placeholder for its real unique name. This direct-object example uses the same prod/daily/ layout as the earlier directory copy.
rclone copyto /srv/backups/ready/app-YYYYMMDD.tar.zst backup-s3:mybucket/prod/daily/app-YYYYMMDD.tar.zst --s3-storage-class STANDARD
Do not continually re-upload expired artifacts from an unlimited local ready directory. Coordinate local retention and upload inventory with remote expiration, retaining whatever independent copies the recovery policy requires. Monthly selection and copy must complete before the chosen daily expires; a namespace alone does not make a snapshot.
The same artifact to native B2
Enable and verify default SSE-B2 before the new upload, then apply the reviewed daily-prefix hide/delete policy. The bucket’s existing objects are not encrypted retroactively. Record any monthly copy separately, outside the daily rule’s scope.
rclone copyto /srv/backups/ready/app-YYYYMMDD.tar.zst backup-b2:my-b2-bucket/prod/daily/app-YYYYMMDD.tar.zst
After copying, verify that the intended file is present and matches the source. Lifecycle controls retention; it does not certify that an archive is complete or a database backup is consistent. Confirm object-level retention if Object Lock is part of the design.
An archived object and an immediately readable object
For a Flexible Retrieval or Deep Archive object selected from the catalog, first preview the narrowly filtered retrieval request. Replace the placeholder name. Request retrieval only after checking the preview, then wait for restore status to show completion:
rclone backend restore backup-s3:mybucket/prod/archives/ --include '/app-YYYYMMDD.tar.zst' -o priority=Standard -o lifetime=2 --dry-run
# Repeat the preceding command without --dry-run after checking the preview.
# Status lists the directory and ignores filters; locate app-YYYYMMDD.tar.zst.
rclone backend restore-status backup-s3:mybucket/prod/archives/
# Only after retrieval of that object is complete:
rclone copyto backup-s3:mybucket/prod/archives/app-YYYYMMDD.tar.zst /tmp/restore/app.tar.zst
For an immediately readable daily object, download directly. Prepare an isolated destination, compare the trusted checksum, inspect/extract the archive and validate the application. The path below matches the upload example exactly:
rclone copyto backup-s3:mybucket/prod/daily/app-YYYYMMDD.tar.zst /tmp/restore/app-daily.tar.zst
Choose flags for their actual tradeoffs
Listing, concurrency and encryption settings
Use --fast-list only when its lower listing-request count is worth the extra memory. It is an optimization to measure, not a backup correctness setting.
On a small VPS, four transfers and eight checkers are illustrative starting values, not a performance promise. Monitor the complete backup producer and uploader together; a large archive compressor or database export may already consume much of the available memory.
For KMS, use --s3-server-side-encryption aws:kms and the correct full key ARN via --s3-sse-kms-key-id. These options select encryption for the upload; they do not grant IAM permissions or preserve the key.
--s3-storage-class chooses the initial class. If lifecycle will manage later transitions, upload to the class appropriate for the immediate restore requirement, usually Standard in these examples. Verify the resulting object rather than assuming a command flag overrode every policy.
Native B2 credentials and encryption options differ from S3-compatible B2 configuration. Keep the backend identity explicit in the runbook. For S3 behavior and flags, consult the rclone S3 documentation.
Compare the total cost of a retention plan
Calculate retained data and old versions, minimum-duration charges, requests, class transitions, archive metadata, retrieval and transfer. Include KMS requests when used. Compare at least a normal month and a full recovery exercise; cheaper stored GB can come with a slower or more expensive restore.
Measure representative objects and access patterns before changing policy across the full backup set. Small-file bundling and version cleanup can help, but both have tradeoffs: larger recovery reads for bundles and permanently lost historical recovery points for deletions.
Keep advanced designs recoverable
Cross-region copies, customer-managed KMS keys and separate retention policies can address specific needs. Document how each additional copy is created, verified and recovered, including its credentials and key access. Encryption does not compensate for missing backup dependencies, and replication alone is not a retained history policy.
Have another authorized operator follow the runbook without access to the source server. Record any missing dependency or permission, then repeat the failed part after fixing it. This tests whether the design can be operated when its original author is unavailable.
Understand Object Lock modes before relying on them
S3 and B2 Object Lock distinguish governance and compliance protection. Governance can be bypassed by identities with the appropriate permission or capability. A protected compliance-mode version cannot have its retention shortened or be deleted by users before expiry. Legal holds are separate. Enabling the bucket feature alone is insufficient: inspect the actual object retention, understand lifecycle interactions and test the intended access controls.
Criteria for a usable backup workflow
A completed workflow produces consistent artifacts, transfers and verifies them, retains the required independent history and can restore the application within the documented target. Recheck it after changes to encryption, credentials, lifecycle or archive classes.
FAQ
Does SSE replace rclone client-side encryption?
They address different boundaries. SSE encrypts within the provider; crypt encrypts before upload and requires recoverable client keys/configuration. TLS protects the transfer. Select the layers for the requirement and test their restore dependencies.
How long does a Glacier restore take?
Instant Retrieval objects support immediate GET. Flexible Retrieval and Deep Archive require retrieval first, ranging from minutes to hours according to class and priority; large datasets may take longer. Include download and application rebuild time in the measured RTO.
Can lifecycle delete a required backup?
Yes. Test explicit prefix and age rules, account for old versions and any database-backup dependencies, and preserve separately selected monthly copies. B2 hiding and deletion have different age clocks; S3 current expiration is different from noncurrent-version deletion.
