How TimescaleDB compresses time-series data

TimescaleDB Compression: Hypercore and Columnar Storage with up to 98% Ratio in PostgreSQL TimescaleDB can achieve compression of up to 98% for typical time-series data. Compressing time-series data requires a fundamentally different approach than the general-purpose algorithms used in OLTP databases. In TimescaleDB this is handled by the hypercore engine — a hybrid row-columnar engine that uses specialized algorithms: delta encoding, delta-of-delta, Gorilla XOR and run-length encoding. This article explains how it works and how to configure compression so that you actually achieve that ratio. TimescaleDB compression - how it differs from PostgreSQL TOAST PostgreSQL has a built-in mechanism called TOAST (The Oversized-Attribute Storage Technique), but TimescaleDB compression solves a fundamentally different problem. TOAST deals with individual large values (long strings, jsonb, bytea), whereas TimescaleDB compression optimizes cross-row patterns in time-series data. The two mechanisms are complementary, not competing — TimescaleDB even uses TOAST internally as a fallback for certain data types. PostgreSQL uses a fixed “page size”, typically 8 kB, and does not allow tuples to span multiple pages. For that reason, when field values are very large, the data must be compressed and/or split across multiple physical rows. | Feature | TOAST (vanilla PostgreSQL) | TimescaleDB hypercore | |---|---|---| | Design goal | Individual values > 2 KB | Cross-row patterns in time-series | | Trigger | Row exceeds TOAST_TUPLE_THRESHOLD (~2 KB) | Per-chunk policy (e.g. older than 7 days) | | Supported types | Variable-length only (text , jsonb , bytea , numeric ) | All data types | | Algorithms | pglz (default), lz4 (since PG14, opt-in) | Combination: delta encoding, delta-of-delta, simple-8b, run-length encoding, XOR-based, dictionary compression | | Compression granularity | Per value (1 value = 1 byte stream) | Per batch (~1000 rows together) | | Exploiting data structure | No - treats values as opaque bytes | Yes - exploits numeric structure, monotonicity, repetition | | Typical ratio for sensor floats | ~1.0× (no compression) | 10-20× | | Typical ratio for timestamps | ~1.0× (no compression - fixed-length type) | 50-100× (delta-of-delta for regular intervals) | | Typical ratio for text | 2-3× (general-purpose LZ) | 5-10× (dictionary + RLE if repetitive) | The table shows the scale of the difference. For a typical IoT workload with floats and timestamps — i.e. the columns TOAST does not compress at all — TimescaleDB reaches a ratio of 10-100×, because it is built for this type of data. The Hypercore engine and columnar compression In TimescaleDB, compression is handled by an engine called hypercore — a hybrid row-columnar engine in which new data lands in Postgres row-based chunks (fast INSERTs and UPDATEs), while older chunks are automatically converted to a columnar, compressed format. Analytical queries that read this compressed data read fewer bytes and run faster. This conversion enables compression of up to 98%, which significantly lowers storage costs in projects with long data retention. Unlike traditional row-based storage, where data is stored sequentially by row, columnar storage organizes and compresses data by column. As a result, queries can fetch only the necessary fields in batches instead of scanning entire rows. What happens to the rows Converting a chunk groups rows into batches of up to 1000 and each batch becomes a single row in the compressed table, in which the columns are arrays. Each compressed batch: - Encapsulates columnar data in compressed arrays of up to 1000 values per column, stored as a single entry in the compressed table. - Uses a column-major format inside the batch, which enables efficient scans by colocating values of the same column and lets you select individual columns without reading the entire batch. - Applies advanced column-level compression techniques — run-length encoding, delta encoding, Gorilla compression — reducing storage and improving I/O. Source: https://www.tigerdata.com/docs/learn/deep-dive/whitepaper#data-model An example of compression using delta encoding: | time | machine_id | sensor_type | value | |---|---|---|---| | 12:00:00 | MACHINE_001 | temp | 72.5 | | 12:00:00 | MACHINE_001 | speed | 2.0 | | 12:00:05 | MACHINE_001 | temp | 72.7 | | 12:00:05 | MACHINE_001 | speed | 2.1 | | 12:00:10 | MACHINE_001 | temp | 72.4 | | 12:00:10 | MACHINE_001 | speed | 2.4 | With delta encoding you only need to store how much each value changed relative to the previous data point, which means smaller values to store. After the first row, you can represent the following rows using less information, for example: | time | machine_id | sensor_type | value | |---|---|---|---| | 12:00:00 | MACHINE_001 | temp | 72.5 | | 0 seconds | MACHINE_001 | speed | 2.0 | | 5 seconds | MACHINE_001 | temp | +0.2 | | 0 seconds | MACHINE_001 | speed | +0.1 | | 5 seconds | MACHINE_001 | temp | -0.3 | | 0 seconds | MACHINE_001 | speed | +0.3 | In time-series data it is often the case that certain values repeat for some period. For example, if you have a temperature sensor that reads 72.5 degrees for 10 minutes, then suddenly rises to 73.0 degrees and stays there for another 10 minutes, you can use delta-of-delta encoding. If the interval is constant (e.g. always 5 seconds), the delta-of-delta is 0 and can be stored in a very small number of bits. | time | machine_id | sensor_type | value | |---|---|---|---| | 12:00:00 | MACHINE_001 | temp | 72.5 | | +5 seconds | MACHINE_001 | temp | +0.2 | | 0 seconds | MACHINE_001 | temp | -0.3 | | 0 seconds | MACHINE_001 | temp | +0.3 | | 0 seconds | MACHINE_001 | temp | -0.1 | Delta encoding works great for numeric values that change by small amounts, but time-series data also often contains columns in which the same value repeats across many consecutive rows — for example machine_id , sensor_type or device status. In such cases run-length encoding (RLE) is used, which instead of storing the same value repeatedly stores it once together with the number of repetitions. Data before compression: | time | machine_id | sensor_type | value | |---|---|---|---| | 12:00:00 | MACHINE_001 | temp | 72.5 | | 12:00:05 | MACHINE_001 | temp | 72.7 | | 12:00:10 | MACHINE_001 | temp | 72.4 | | 12:00:15 | MACHINE_001 | temp | 72.6 | | 12:00:20 | MACHINE_001 | temp | 72.5 | After applying RLE to the machine_id and sensor_type columns: | machine_id | sensor_type | |---|---| | MACHINE_001 × 5 | temp × 5 | Instead of five copies of the string MACHINE_001 (~55 bytes) we store a single value plus a counter (~15 bytes). With millions of rows sharing the same machine_id value, the savings are enormous. In the end it will look like this: | column | technique | representation after compression | |---|---|---| time | delta-of-delta | 12:00:00 , +5s , 0 , 0 , 0 | machine_id | run-length encoding | MACHINE_001 × 5 | sensor_type | run-length encoding | temp × 5 | value | delta encoding | 72.5 , +0.2 , -0.3 , +0.2 , -0.1 | There are many other methods used in TimescaleDB, which you can read about in the official documentation. Compression is not “one size fits all” — TimescaleDB picks the algorithm per column type, which is key to understanding why the ratio varies so much between schemas: - Integers, timestamps, booleans and integer-like types — a combination of delta encoding, delta-of-delta, simple-8b and run-length encoding. Delta-of-delta produces small numbers (for regular intervals — all zeros), and simple-8b then physically packs those small numbers into a few bits per value. A similar approach (delta-of-delta for timestamps) is used by Facebook’s Gorilla algorithm. - Columns without many repetitions (e.g. floats from temperature and vibration measurements) — XOR-based compression (based on Gorilla) with a touch of dictionary compression. XORing neighboring floats yields a result with a long run of leading and trailing zeros when the values are similar — then you only need to store the middle “significant” bits instead of the full 64. - JSONB — two layers: first dictionary (when values repeat), and if there are no repetitions, a fallback to PostgreSQL TOAST ( pglz by default,lz4 if configured). - Everything else (strings, more unusual types) — dictionary compression. The dictionary indexes also go through simple-8b + RLE, so the compression is two-stage. That is why sensor_type in the form 'TEMPERATURE'/'SPEED'/'PRESSURE' compresses brilliantly (a 3-element dictionary plus RLE on the indexes), monotonically increasing time drops to almost zero bytes per value, while a high-entropy column such as a per-row UUID will be much worse — a dictionary helps little, because every value is unique, so the dictionary is just as large as the original data. TimescaleDB detects this case and simply does not use a dictionary then. segmentby and orderby — the most important parameters These are the two parameters you have to choose deliberately, because they determine how rows are grouped into batches before compression. segmentby — the column whose values are shared across an entire batch (e.g.machine_id orsensor_id ). The value is stored once per batch, not as an array. In addition, the planner uses segmentby metadata to skip entire batches that do not match theWHERE clause.orderby — the sort order inside the batch (usuallytime DESC ). Sorting by time gives delta encoding and delta-of-delta their maximum advantage — neighboring values are close to each other, so the differences are small and pack into a few bits. ALTER TABLE iot_sensor_data SET ( timescaledb.orderby = 'time DESC', timescaledb.segmentby = 'machine_id' ); Queries with a WHERE machine_id = '...' AND time BETWEEN ... filter on a table configured this way can be an order of magnitude faster than without segmentby , because the planner skips other machines’ batches based on metadata — without touching the data itself. TimescaleDB packs rows into batche