DataStax Help Center

DTCS and the effect of timestamp resolution on compaction

Overview

When setting the timestamp_resolution for Date Tiered Compaction Strategy (DTCS) settings, it is important to understand why writes also need to set the same timestamp resolution. Failure to do so can lead to a unmanageable number of sstables which can in turn lead to compaction and repair problems.

Symptoms

A high number of very small sstables many with dates that are older than expected is a possible sign of this being a problem. Repair jobs might begin to fail. In some cases the node might fail with a too many open files error or even in some cases a java.lang.OutOfMemoryError: Java heap space

In cases where this has been observed the number of sstables have been in the order of several hundreds of thousands, not a smaller amount like several thousand.

Verifying the timestamps

The easiest way to detect if this might be a problem is simply to check the tables using DTCS and verify what the timestamp_resolution is set to, then compare this to what we see in our sstables with the sstablemetadata tool.

Working example

First let’s create a table and set the compaction subproperties:

CREATE TABLE markc.dtcs_example (
    key text PRIMARY KEY,
    col1 int,
    col2 int
) WITH bloom_filter_fp_chance = 0.01
    AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
    AND comment = ''
    AND compaction = {'timestamp_resolution': 'MILLISECONDS', 'class': 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy', 'base_time_seconds': '60'}
    AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
    AND dclocal_read_repair_chance = 0.1
    AND default_time_to_live = 0
    AND gc_grace_seconds = 864000
    AND max_index_interval = 2048
    AND memtable_flush_period_in_ms = 0
    AND min_index_interval = 128
    AND read_repair_chance = 0.0
    AND speculative_retry = '99.0PERCENTILE';

Add some values - default timestamp

cqlsh:markc> insert into dtcs_example (key, col1 , col2 ) VALUES ( 'k1', 100, 200);
cqlsh:markc> insert into dtcs_example (key, col1 , col2 ) VALUES ( 'k2', 101, 201);

Flush and check

Next we flush using nodetool flush and check the sstablemetadata on the sstable. For example:

# sstablemetadata /var/lib/cassandra/data/markc/dtcs_example-
b8f79c71a29911e68874c9b2b14e96bf/markc-dtcs_example-ka-1-Data.db
SSTable: /var/lib/cassandra/data/markc/dtcs_example-b8f79c71a29911e68874c9b2b14e96bf/markc-dtcs_example-ka-1
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Bloom Filter FP chance: 0.010000
Minimum timestamp: 1478269374033947
Maximum timestamp: 1478269383924958
SSTable max local deletion time: 2147483647
Compression ratio: 0.5113636363636364
Estimated droppable tombstones: 0.0
<...truncated...>

Using normal command line tools we can determine the min / max timestamps we have:

A microsecond timestamp will typically have 16 digits, millisecond 13 digits and second 10 digits, here we have 16 digits and so we take off 6 digits (if we had left all the digits on, the date would have come out far in the future and this would be incorrect) 

# date -d @1478269374
Fri Nov  4 14:22:54 UTC 2016

Add some values - millisecond timestamp

So we have verified above our sstable dates are written with a microsecond timestamp. However we need a millisecond timestamp because of our resolution. So, lets insert some records with a timestamp:

cqlsh:markc> insert into dtcs_example (key, col1 , col2 ) VALUES ( 'k3', 100, 200) USING TIMESTAMP 1478269400999;
cqlsh:markc> insert into dtcs_example (key, col1 , col2 ) VALUES ( 'k4', 100, 200) USING TIMESTAMP 1478269401999;

Flush and check

Again we flush using nodetool flush and check the sstablemetadata on the sstable. For example:

# sstablemetadata /var/lib/cassandra/data/markc/dtcs_example-b8f79c71a29911e68874c9b2b14e96bf/markc-dtcs_example-ka-2-Data.db
SSTable: /var/lib/cassandra/data/markc/dtcs_example-b8f79c71a29911e68874c9b2b14e96bf/markc-dtcs_example-ka-2
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Bloom Filter FP chance: 0.010000
Minimum timestamp: 1478269400999
Maximum timestamp: 1478269401999
SSTable max local deletion time: 2147483647
Compression ratio: 0.5284090909090909
Estimated droppable tombstones: 0.0
<...truncated...>

Removing 3 digits this time we get an accurate date

# date -d @1478269400
Fri Nov  4 14:23:20 UTC 2016

Conclusion

So sstable markc-dtcs_example-ka-1 has a microsecond timestamp but markc-dtcs_example-ka-2 has a millisecond timestamp, the former sstable will not compact correctly with the table settings since and will probably fall outside of the max_sstable_age_days parameter and never be considered for compaction. The latter sstable will compact as expected.

Therefore, if the timestamp resolution is of a different order than the actual timestamp of the inserts then these tables might never be compacted leading to the situation outlined above with unmanageable amounts of sstables which in turn can lead to other problems.

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk