DataStax Help Center

Example: Upgrade from 4.0.3 to 4.8.4 - Encrypted Table Issues - Proof of concept

Summary

During an upgrade, the following error happened when trying to query an encrypted table after upgrading from 4.0.3 to 4.0.7 to 4.8.4

WARN [SharedPool-Worker-1] 2016-03-11 00:14:12,509 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
java.lang.RuntimeException: java.lang.RuntimeException: com.datastax.bdp.cassandra.crypto.KeyAccessException: java.io.IOException: Couldn't decrypt input

The reason why this is happening is that the correct steps to upgrade the cluster were not followed.

A good way to understand how upgrading DataStax enterprise with an encrypted table can be done doing a small "Proof of Concept" and single nodes for each version of the software.

The reason why the 4.0.3 > 4.0.7 > 4.8.4 is because this is the mandatory path to do an upgrade to from 4.0.3 to 4.8.4.

  • Get the tarball install for 4.0.3, 4.0.7 and 4.8.4
  • Untar each file in /opt/DSE/
  • You will now see 3 folders
    • /opt/DSE/4.0.3
    • /opt/DSE/4.0.7
    • /opt/DSE/4.8.4

DSE 4.0.3 Configuration

  • Change the cassandra.yaml for a one node cluster
  • Start DSE 4.0.3 using ./dse cassandra command
  • Create the following keyspace:
    • create KEYSPACE "test" WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};
  • Create the following encrypted table:
    • CREATE TABLE "test"."Friends" (
      key bigint,
      column1 bigint,
      value text,
      PRIMARY KEY (key, column1)
      ) WITH COMPACT STORAGE
      AND CLUSTERING ORDER BY (column1 ASC)
      AND bloom_filter_fp_chance = 0.01
      AND comment = ''
      AND compaction = {'min_threshold': '2', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
      AND compression = {'chunk_length_kb': '128', 'secret_key_strength': '128', 'sstable_compression': 'org.apache.cassandra.io.compress.EncryptingSnappyCompressor', 'cipher_algorithm': 'AES/ECB/PKCS5Padding'}
      AND dclocal_read_repair_chance = 0.0
      AND default_time_to_live = 0
      AND gc_grace_seconds = 86400
      AND memtable_flush_period_in_ms = 0
      AND read_repair_chance = 0.1
      AND speculative_retry = 'NONE';
  • Insert some data in the table
    • INSERT INTO "Friends" ( key, column1, value ) VALUES ( 9068605111399150616, 9068605111399150616, 'wee' ) ;
    • INSERT INTO "Friends" ( key, column1, value ) VALUES ( 9068605111399150617, 9068605111399150617, 'wee' ) ;
  • Drain the node using the command
    • nodetool drain
  • Shutdown the node
  • You will notice that the following file was create in the following folder:
    • /etc/dse/conf/data_encryption_keys   
      • This is the encryption key that was created after you have created the encrypted table

 DSE 4.0.7 Configuration and upgrade from 4.0.3

  • Change the cassandra.yaml for a one node cluster
  • Copy the file
    • /etc/dse/conf/data_encryption_keys   for
    • /etc/dse/conf/system_key
    • For version 4.0.7 the key name was changed to system_key
  • Start DSE 4.0.3 using ./dse cassandra command
  • Create the following keyspace:
    • create KEYSPACE "test" WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};
  • Create the following encrypted table:
    • CREATE TABLE "test"."Friends" (
      key bigint,
      column1 bigint,
      value text,
      PRIMARY KEY (key, column1)
      ) WITH COMPACT STORAGE
      AND CLUSTERING ORDER BY (column1 ASC)
      AND bloom_filter_fp_chance = 0.01
      AND comment = ''
      AND compaction = {'min_threshold': '2', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
      AND compression = {'chunk_length_kb': '128', 'secret_key_strength': '128', 'sstable_compression': 'org.apache.cassandra.io.compress.EncryptingSnappyCompressor', 'cipher_algorithm': 'AES/ECB/PKCS5Padding'}
      AND dclocal_read_repair_chance = 0.0
      AND default_time_to_live = 0
      AND gc_grace_seconds = 86400
      AND memtable_flush_period_in_ms = 0
      AND read_repair_chance = 0.1
      AND speculative_retry = 'NONE';
  • Move the sstables from the 4.0.3 folder for the keyspace test and table Friends to the 4.0.7 folder
  • Run the sstableloader command:
    • ./sstableloader -v -d 127.0.0.1 ....../data/test/Friends
  • Query the table to see if the sstables have been loaded properly
    • use keyspace test;
    • Select * from "test"."Friends";
      • If you get data, this means your sstableloader of the encrypted table worked
  • Before shutting down 4.0.7 and moving to 4.8.4, query the the following keyspace and table:
    • use dse_system;
    • Select * from "dse_system.encrypted_keys;
    • Copy and paste the result onto a textpad
  • Shutdown DSE

DSE 4.8.4 Configuration and upgrade from 4.0.7

  • Change the cassandra.yaml for a one node cluster
  • 4.8.4 will reuse the encryption jey
    • /etc/dse/conf/system_key
  • Start DSE 4.8.4 using ./dse cassandra command
  • Create the following keyspace:
    • create KEYSPACE "test" WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};
  • Create the following encrypted table:
    • CREATE TABLE "test"."Friends" (
      key bigint,
      column1 bigint,
      value text,
      PRIMARY KEY (key, column1)
      ) WITH COMPACT STORAGE
      AND CLUSTERING ORDER BY (column1 ASC)
      AND bloom_filter_fp_chance = 0.01
      AND comment = ''
      AND compaction = {'min_threshold': '2', 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
      AND compression = {'chunk_length_kb': '128', 'secret_key_strength': '128', 'sstable_compression': 'org.apache.cassandra.io.compress.EncryptingSnappyCompressor', 'cipher_algorithm': 'AES/ECB/PKCS5Padding'}
      AND dclocal_read_repair_chance = 0.0
      AND default_time_to_live = 0
      AND gc_grace_seconds = 86400
      AND memtable_flush_period_in_ms = 0
      AND read_repair_chance = 0.1
      AND speculative_retry = 'NONE';
  • Move the sstables from the 4.0.7 folder for the keyspace test and table Friends to the 4.8.4 folder
  • Insert the key_id and value you obtained from the 4.0.7 dse_system.encrypted_keys table in the 4.8.4 environment using the following command
    • Use dse_system;
    • INSERT INTO dse_system.encrypted_keys (key_file, cipher, strength, key_id, key) VALUES ( 'system_key', 'AES', 128, KEY_ID, 'KEY_VALUE');
  • Run the sstableloader command:
    • ./sstableloader -v -d 127.0.0.1 ....../data/test/Friends
  • Upgrade the sstables
    • nodetool upgradesstables command
  • Query the table to see if the sstables have been loaded properly
    • use keyspace test;
    • Select * from "test"."Friends";
      • If you get data, this means your sstableloader of the encrypted table worked for 4.8.4

 

 

 

Was this article helpful?
1 out of 1 found this helpful
Have more questions? Submit a request

Comments

Powered by Zendesk