OpsCenter unable to connect to S3 bucket using default AWS credential provider chain

Summary

This article discusses an authentication issue when OpsCenter connects to an S3 bucket using the default AWS credential provider chain. 

Applies to

• DataStax OpsCenter 6.5.2 or earlier
• DataStax OpsCenter 6.1.9 or earlier

Symptoms

Attempts to add Amazon S3 as a backup location in OpsCenter with AWS Credential Provider chain as the credentials source fails with the following error:

Location validation error: Unable to authenticate against AWS with the provided key and secret.

Cause

OpsCenter uses the AWS SDK for Java to access AWS infrastructure services such as S3. For AWS environments which use EC2 instance profile credentials (i.e. IAM role credentials which exist in the EC2 instance metadata), OpsCenter allows the use of these credentials when backing up to an S3 bucket without users having to explicitly provide API keys (OPSC-5161).

When OpsCenter tries to get the AWS key and secret from the credentials object returned by the DefaultAWSCredentialsProviderChain instance, the key and secret get cloned into a BasicAWSCredentials object so that OpsCenter can override the default toString method to prevent the credentials from accidentally "leaking" into logs. In some situations, the resulting BasicAWSCredentials object is malformed (OPSC-14939) and authentication against S3 fails with the error above.

Workaround

If using an affected version of OpsCenter, temporarily switch to supplying the AWS credentials when adding an Amazon S3 backup location.

Solution

OPSC-14939 has been fixed in OpsCenter 6.5.3 and OpsCenter 6.1.10.

Upgrade to the latest version of OpsCenter to take advantage of all the new improvements and latest fixes.

See also

The AWS SDK retrieves the EC2 instance credentials and OpsCenter saves them in a credentials object which gets cached in memory. However, instance profile credentials are temporary and the associated session token eventually expire which can cause backups to fail (OPSC-15138). See Expired AWS credentials result in failed OpsCenter backups to S3 location for details.