Starting MySQL. ERROR! The server quit without updating PID file - how we fixed it.
Archive - Originally posted on "The Horse's Mouth" - 2015-05-06 14:50:20 - Graham Ellis
Our MySQL databases are backed up frequently, and I've arranged for our server to email me in the event of a failure. From our /bin/bash script which is run by crontab:
/usr/local/mysql/bin/mysqldump -ufgw -psummat -h127.0.0.1 fgw > /home/backups/$HOUR/fgw.sql
if [ "$?" -eq 0 ]; then
echo "Success $HOUR fgw" > /dev/null
else
echo "Mysqldump encountered a problem $HOUR fgw" | mail -s "FGW BAD dump" graham@wellho.net
fi
I got an email a couple of hours ago ... and it turned out that the MySQL server had halted. Not sure why, which frightens me, but I'm working from a cruise boat with a poor connection and the important thing was to fix it. Trying to restart:
[root@Server51076 init.d]# ./mysql.server start
Starting MySQL. ERROR! The server quit without updating PID file (/usr/local/mysql/data/Server51076.uk2net.com.pid).
[root@Server51076 init.d]#
A quick trip to Google took me on to a couple of articles - which described different problems, or OS Specific things. Several referred to "Google Hell" on this subject, so let me add my own 2 cents worth in case you've arrived here from Google Hell!
It appears that what's happened is that the MySQL daemon has stopped catastrophically for some reason. It's confused into thinkig it should be referring to a PID file which didn't exist, and because it's crashed, it's refusing to restart. And it's not got any process to tidy us if I try to run server stop scripts. Thus the irritation above.
My solution to get running ...
Ensure that /etc/my.cnf contains: [mysqld]
innodb_force_recovery = 1
And restart the server as this will overstep the problem above, and also recover the databases; error recovery levels of 1 or 2 are likely to let you recover (almost) everything we're told, and I have the previous good backup anyway, so I can restore if I need to.
Then stop the server once it's forced the recovery, remove the innodb_force_recovery assignment, and restart it. And - well - it worked for me. I'm monitoring carefully, and posting this to my blog partly as a test and partly in the hope it may provide a helpful clue to others in the future.