Trying to start opscenter agent with error "datastax-agent dead but pid file exists"

Summary

Sometimes when starting the opscenter agents from a service, the startup might not succeed

Symptoms

When starting the service the user might see the following on the command line after they start the service:

$ sudo service datastax-agent status
datastax-agent dead but pid file exists

Cause

The datastax agent monitor process may still be running even though the agent process is not. Typically you would see both:

root 32404 1 0 00:39 pts/0 00:00:00 /bin/bash /usr/share/datastax-agent/bin/datastax_agent_monitor
496 32426 1 5 00:39 ? 00:01:27 /usr/java/jdk1.7.0_67/bin/java -Xmx128M -Djclouds.mpu.parts.magnitude=100000 -Djclouds.mpu.parts.size=16777216 -Dopscenter.ssl.trustStore=/var/lib/datastax-agent/ssl/agentKeyStore -Dopscenter.ssl.keyStore=/var/lib/datastax-agent/ssl/agentKeyStore -Dopscenter.ssl.keyStorePassword=opscenter -Dagent-pidfile=/var/run/datastax-agent/datastax-agent.pid -Dlog4j.configuration=file:/etc/datastax-agent/log4j.properties -Djava.security.auth.login.config=/etc/datastax-agent/kerberos.config -jar datastax-agent-5.2.0-standalone.jar /var/lib/datastax-agent/conf/address.yaml

Solution

Ensure the datastax_agent_monitor process is killed and then restart the datastax-agent service

To kill a process, the command is usually:

$ kill <pid>

For example:

$ kill 32404 

In some cases where the process still remains running, using a kill -9 will stop it immediately. For example:

$ kill -9 32404