> ## Documentation Index
> Fetch the complete documentation index at: https://powersync-storage-v4.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Compacting Buckets

> Reduce storage and sync overhead by compacting and defragmenting data.

This allows clients to download incremental changes efficiently — only changed rows have to be downloaded. However, over time this history can grow large, causing new clients to potentially take a long time to download the initial set of data. To handle this, we compact the history of each bucket.

## Compacting

### PowerSync Cloud

The cloud-hosted version of PowerSync will automatically compact all buckets once per day.

You can manually trigger compacting from the [PowerSync Dashboard](https://dashboard.powersync.com/) or the [CLI](/tools/cli):

* **Dashboard**: Select your project and instance, go to the **Settings** view, and click the **Compact** button in the "Compact operation history" section.
* **CLI**: Run `powersync compact` against a linked Cloud instance. The CLI polls until the operation completes, with a default timeout of 30 minutes. Pass `--timeout=<minutes>` to override, or `--timeout=0` to wait indefinitely.

When using PowerSync Cloud, compact logs are available in the Dashboard's **Logs** view. See [Instance Logs](/maintenance-ops/monitoring-and-alerting#instance-logs).

[Defragmenting](/maintenance-ops/compacting-buckets#defragmenting) may still be required.

### Self-Hosted PowerSync

For self-hosted setups (PowerSync Open Edition & PowerSync Enterprise Self-Hosted Edition), the `compact` command in the Docker image can be used to compact all buckets. This can be run manually, or on a regular schedule using Kubernetes [CronJob](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/) or similar scheduling functionality.

On storage version 4, the experimental `compact --incremental-only` option runs the lower-cost incremental bucket and parameter-index compacting paths. You can schedule this mode more frequently. It skips bucket compacting for Sync Configs on older storage versions, so keep a regular full `compact` job if your instance still has configs on those versions.

[Defragmenting](/maintenance-ops/compacting-buckets#defragmenting) may still be required.

## Background

### Bucket Operations

Each bucket is an ordered list of `PUT`, `REMOVE`, `MOVE` and `CLEAR` operations. In normal operation, only `PUT` and `REMOVE` operations are created.

A simplified view of a bucket may look like this:

```bash theme={null}
(1, PUT, row1, <data>)
(2, PUT, row2, <data>)
(3, PUT, row1, <data>)
(4, REMOVE, row2)
```

### Compacting Step 1 - MOVE Operations

The first step of compacting involves `MOVE` operations. This just indicates that an operation is not needed anymore, since a later `PUT` or `REMOVE` operation replaces the row.

After this compact step, the bucket may look like this:

```bash theme={null}
(1, MOVE)
(2, MOVE)
(3, PUT, row1, <data>)
(4, REMOVE, row2)
```

This does not reduce the number of operations to download, but can reduce the amount of data to download.

### Compacting Step 2 - CLEAR Operations

The second step of compacting takes a sequence of `CLEAR`, `MOVE` and/or `REMOVE` operations at the start of the bucket, and replaces them all with a single `CLEAR` operation. The `CLEAR` operation indicates to the client that "this is the start of the bucket, delete any prior operations that you may have".

After this compacting step, the bucket may look like this:

```bash theme={null}
(2, CLEAR)
(3, PUT, row1, <data>)
(4, REMOVE, row2)
```

This reduces the number of operations for new clients to download in some cases.

The `CLEAR` operation can only remove operations at the start of the bucket, not in the middle of the bucket, which leads us to the next step.

### Defragmenting

There are cases that the above compacting steps cannot optimize efficiently. The key factor is that the oldest PUT operation in a bucket determines how much of the history can be compacted. This means:

1. If a row has never been updated since its initial creation, its original PUT operation remains at the start of the bucket
2. All operations that come after this oldest PUT cannot be fully compacted
3. This is particularly problematic when you have:
   * A small number of rarely-changed rows in the same bucket as frequently-updated rows
   * The rarely-changed rows' original PUT operations "block" compacting of the entire bucket
   * The frequently-updated rows continue to accumulate operations that can't be fully compacted

For example, imagine this sequence of statements:

```sql theme={null}
-- Insert a single row that rarely changes
INSERT INTO lists(name) VALUES('a');
-- Insert 50k rows that change frequently
INSERT INTO lists (name) SELECT 'b' FROM generate_series(1, 50000);
-- Delete those 50k rows, but keep 'a'
DELETE FROM lists WHERE name = 'b';
```

After compacting, the bucket looks like this:

```bash theme={null}
(1, PUT, row_1, <data>)  -- This original PUT blocks further compacting
(2, MOVE)
(3, MOVE)
...
(50001, MOVE)
(50002, REMOVE, row2)
(50003, REMOVE, row3)
...
(100001, REMOVE, row50000)
```

This is inefficient because:

1. The original PUT operation for row 'a' remains at the start
2. All subsequent operations can't be fully compacted
3. We end up with over 100k operations for what should be a simple bucket

To handle this case, we "defragment" the bucket by updating existing rows in the source database. This creates new PUT operations at the end of the bucket, allowing the compact steps to efficiently compact the entire history:

```sql theme={null}
-- Touch all rows to create new PUT operations
UPDATE lists SET name = name;
-- OR touch specific rows at the start of the bucket
UPDATE lists SET name = name WHERE name = 'a';
```

After defragmenting and compacting, the bucket looks like this:

```bash theme={null}
(100001, CLEAR)
(100002, PUT, row_1, <data>)
```

The bucket is now back to two operations, allowing new clients to sync efficiently.

<Note>
  Note: All rows in the bucket must be updated for this to be effective. If some rows are never updated, they will continue to block compacting of the entire bucket.
</Note>

<Tip>
  **Bucket Design Tip**: If you have a mix of frequently-updated and rarely-changed rows, consider splitting them into separate buckets. This prevents the rarely-changed rows from blocking compacting of the frequently-updated ones.
</Tip>

### When to Defragment

You should consider defragmenting your buckets when:

1. **High Operations-to-Rows Ratio**: If you notice that the number of operations significantly exceeds the number of rows in a bucket. You can inspect this using the [Sync Diagnostics Client](https://github.com/powersync-ja/powersync-js/tree/main/tools/diagnostics-app).
2. **Frequent Updates**: Tables that are frequently updated (e.g., status fields, counters, or audit logs)
3. **Large Data Churn**: Tables where you frequently insert and delete many rows

### Defragmenting Strategies

There are two mechanisms with different scopes:

1. **Update source rows, then compact.** Use this when you can identify the rows whose old `PUT` operations block compacting. You can target one table or subset of rows, but each update creates a new operation that clients must download. For recurring maintenance, schedule the updates with a tool such as `pg_cron`:

   ```sql theme={null}
   -- Daily defragmentation for high-churn tables
   UPDATE audit_logs SET last_updated = now() 
   WHERE last_updated < now() - interval '1 day';

   -- Weekly defragmentation for other tables
   UPDATE users SET last_updated = now() 
   WHERE last_updated < now() - interval '1 week';
   ```

2. **Force a full reprocess.** In the [PowerSync Dashboard](https://dashboard.powersync.com/), select your project and instance, open **Settings**, and use **Defragment** in the "Compact operation history" section. This action forces a new replication stream even when the deployment could otherwise use incremental reprocessing. It rebuilds all prepared sync data and makes clients download all subscribed data again. Use it for one-time cleanup when you intend to rebuild the entire Sync Config.

### Defragmenting Trade-offs

Both methods can significantly reduce the operation history, at the cost of existing clients downloading data again. Updating source rows and then compacting lets you control which rows are affected. The Dashboard action rebuilds and replaces the complete replication stream.

Key considerations:

1. **Frequency**: More frequent defragmentation means fewer operations per sync but more frequent re-syncs
2. **Scope**: Defragmenting all rows at once is more efficient but causes a larger sync cycle
3. **Monitoring**: Use the [Sync Diagnostics Client](https://github.com/powersync-ja/powersync-js/tree/main/tools/diagnostics-app) to track operations-to-rows ratio

## Sync Streams Deployments

With [incremental reprocessing](/sync/advanced/incremental-reprocessing) (currently in Beta; supported only with Sync Streams, a MongoDB source database, and MongoDB bucket storage), PowerSync reuses the prepared data for unchanged streams. This reduces reprocessing during the deployment and prevents clients from downloading unchanged data again. Reused buckets also keep their operation history, so a deployment no longer acts like defragmenting every bucket.

Without incremental reprocessing, a deployment reads all data selected by the Sync Config and creates new buckets for it. Clients then download that data again. Do not redeploy a Sync Config for routine history cleanup. Use compacting, targeted source-row updates, or the Dashboard's full-reprocess **Defragment** action according to the scope of cleanup you need.

## Technical Details

See the [documentation](https://github.com/powersync-ja/powersync-service/blob/main/docs/storage/compacting-operations.md) in the `powersync-service` repo for more technical details on compacting.