This article relates to failures in Cassandra read operations as a result of a defect with the
AbstractEndpointSnitch class in early versions of Cassandra.
Some read operations fail resulting in an
IllegalArgumentException. Here is a sample error reported in the
ERROR [Native-Transport-Requests:44] 2015-12-14 10:59:00,641 ErrorMessage.java (line 210) Unexpected exception during request java.lang.IllegalArgumentException: Comparison method violates its general contract! at java.util.TimSort.mergeHi(TimSort.java:868) at java.util.TimSort.mergeAt(TimSort.java:485) at java.util.TimSort.mergeCollapse(TimSort.java:410) at java.util.TimSort.sort(TimSort.java:214) at java.util.TimSort.sort(TimSort.java:173) at java.util.Arrays.sort(Arrays.java:659) at java.util.Collections.sort(Collections.java:217) at org.apache.cassandra.locator.AbstractEndpointSnitch.sortByProximity(AbstractEndpointSnitch.java:49) at org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithScore(DynamicEndpointSnitch.java:153) at org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximityWithBadness(DynamicEndpointSnitch.java:171) at org.apache.cassandra.locator.DynamicEndpointSnitch.sortByProximity(DynamicEndpointSnitch.java:147) ...
This is due to a known defect in CASSANDRA-8448 where the snitch endpoint scores change while being sorted. Originally thought to have been solved, it was later discovered that the fix was incomplete and was not resolved until CASSANDRA-9519.
Issue was fixed in the following releases:
- Cassandra 2.0.17 (DataStax Enterprise 4.6.11+)
- Cassandra 2.1.9 (DataStax Enterprise 4.7.4+, 4.8.1+)
- Cassandra 2.2.0
It is possible to suppress the error by forcing the JVM to use a legacy sort methodology.
Follow the steps below to implement a temporary workaround for clusters experiencing the issue.
Step 1 - For the first node in the cluster, add the following JVM option in
Step 2 - Restart DSE on the node.
Step 3 - Repeat steps 1 and 2 one node at a time until all nodes in the cluster have been reconfigured.
As stated above, the workaround provided is not intended as a fix. For optimal operation of the cluster, upgrade to the following DataStax Enterprise releases which contain the fix:
- DataStax Enterprise 4.6.11+
- DataStax Enterprise 4.7.6+
- DataStax Enterprise 4.8.3+
Cassandra JIRA - CASSANDRA-9519 CASSANDRA-8448 not fixed