Problem Symptoms:
1. Any of the following:
- Can't delete a record in cassandra
- A record that was updated is not showing the updated data
- A record that is inserted isn't showing up in cassandra
and
2. There were no errors when the delete/insert/update was done.
Problem Cause:
Any of the below:
Cause 1: You have already inserted a record with the same primary key as what you are now trying to insert/update/delete, and the timestamp of that record is a date/time from the future.
You can check to see if this is your case by doing a SELECT for that record and using the writetime() function to see the timestamp.
Example:
We have a table defined as:
CREATE TABLE thanhks1.users1 (
id int PRIMARY KEY,
fname text,
lname text
) ;
id int PRIMARY KEY,
fname text,
lname text
) ;
cqlsh:thanhks1> select id,fname,writetime(lname) from users1;
id | fname | writetime(lname)
----+---------+------------------
2 | barry | 1740734891000000
3 | charlie | 1582743609569828
(2 rows)
id | fname | writetime(lname)
----+---------+------------------
2 | barry | 1740734891000000
3 | charlie | 1582743609569828
(2 rows)
From the above, we know that we won't be able to delete or overwrite barry's user record from the users1 table as we normally would because that record has a writetime (timestamp) from the future (1740734891000000 = Feb 28, 2025).
Solution for not being able to overwrite/delete data with a future timestamp:
Use "USING TIMESTAMP" in your delete OR overwrite, with a timestamp that's greater than the existing timestamp (1740734891000000 in our example above), eg
cqlsh:thanhks1> delete from users1 using timestamp 1740734891000001 where id = 2;
After that, we'll have:
cqlsh:thanhks1> select id,fname,writetime(lname) from users1;
id | fname | writetime(lname)
----+---------+------------------
3 | charlie | 1582743609569828
id | fname | writetime(lname)
----+---------+------------------
3 | charlie | 1582743609569828
(1 rows)
NOTE that if write a tombstone with a future timestamp, you can "block" new writes for the primary key that was deleted using the future timestamp. To "unblock" writes that are blocked by tombstones with a future timestamp, see Cause 3 and its solution further down in this article.
Cause 2: Another insert or update occurred at almost exactly the same time you did your insert/update. That other insert/update is what's seen in query results when you do a SELECT from the table.
In this case, it could be 1 of 2 things that caused the problem:
2a. If you have multiple driver clients, or multiple instances of the same driver client, running on different servers/different app servers, and 1 insert/update is done using 1 driver client while the other insert/update is done using the other client AND the driver clients user a "newer" version of a datastax driver, then:
The write timestamp (if you don't explicitly supply one when doing the insert/update) is generated client-side. This means that the different client systems (app servers) are required to have their clocks synced in order for the insert that was done most recently to "be seen" in the database/in your query results.
(Check your driver documentation to see if it will use client-side generated timestamps automatically if/when you do not specify the write timestamp in your insert/update/delete)
Solution for problem caused by client-side generated timestamps:
Synchronize the clocks on your client systems/app servers where the driver clients are running. NTP is recommended for clock synchronization. Alternatively, modify your driver client code to use server-side timestamps instead (see your driver documentation for details on how to do this) AND make sure that all dse server clocks are synced via NTP. Beware that, even if/when dse servers all have their clocks synced, you can still end up with out-of-order writes, if/when a request from one client, sent at time T0, doesn't reach a dse node until after another request from another client that was sent at time T0+1 (or T0 + some number greater than 1).
Synchronize the clocks on your client systems/app servers where the driver clients are running. NTP is recommended for clock synchronization. Alternatively, modify your driver client code to use server-side timestamps instead (see your driver documentation for details on how to do this) AND make sure that all dse server clocks are synced via NTP. Beware that, even if/when dse servers all have their clocks synced, you can still end up with out-of-order writes, if/when a request from one client, sent at time T0, doesn't reach a dse node until after another request from another client that was sent at time T0+1 (or T0 + some number greater than 1).
2b. If you have multiple driver clients, or multiple instances of the same driver client, running on different servers/different app servers, and 1 insert/update is done using 1 driver client while the other insert/update is done using the other client AND the driver clients user an "older" version of a datastax driver, then:
The write timestamp (if you don't explicitly supply one when doing the insert/update) is generated server-side, by the dse node that is coordinating the query (the coordinator node). If dse server clocks are not synced, an insert/update that is most recent may be "lost" (written to cassandra, but not seen in query results) because the coordinator clock is "fast" so the insert/update gets an earlier timestamp than another insert/update that actually arrived at a different coordinator node earlier. OR, even if dse server clocks are synced, the most recent insert/update may be "lost" if it arrives at its coordinator node faster/earlier than an insert/update that was actually issued earlier but arrived later (due to network latency, for example).
(Check your driver documentation to see if it will use server-side generated timestamps automatically if/when you do not specify the write timestamp in your insert/update/delete)
Solution for problem caused by server-side generated timestamps:
Synchronize the dse server clocks using NTP and consider using a newer version of the driver you're using so that you can get automatic client-side generated timestamps (keeping in mind that, with client-side timestamps, you also have to have clocks synchronized on client systems/app servers) instead of server-side generated timestamps.
Cause 3. You have deleted a record using a timestamp from the future; thus, you have a tombstone (for the primary key you are trying to write to) whose timestamp is a future date/time.
In this case, you won't be able to write a new record for the primary key in question until the tombstone is expired and/or purged.
Solution for not being able to insert data due to having a tombstone with a future timestamp:
If your tombstone timestamp isn't that far into the future, you may be able to wait until it expires.
If you don't want to wait for the tombstone to expire on its own, you can force it out by setting gc_grace_seconds=0 and doing a manual compaction to purge the tombstone. JUST BEWARE that having gc_grace_seconds=0 doesn't give time for any tombstones to propagate to replica nodes, so you run the risk of data resurrection. To minimize this risk, make the change to gc_grace_seconds & do the compaction at a time when there is no or low write activity to the table, then change gc_grace_seconds back to what you were using once you've finished purging the tombstones.
Further Reading:
- Mark Curtis's Support Blog: https://academy.datastax.com/support-blog/dude-where%E2%80%99s-my-data
- Cassandra jira https://issues.apache.org/jira/browse/CASSANDRA-6178