Friday, June 27, 2014

UPDATED!!! Puppet Dashboard: All Tasks are "Pending" and all Nodes "Unresponsive"

Well it seems that the previous Solution, was a temporary one. :(
So I kept debugging a little bit and I found this error on the delayed_job log (/usr/share/puppet-dashboard/log/delayed_job.log) :

... 2014-06-13T12:05:40+0200: [Worker(delayed_job.1 host:pmaster01 pid:21516)] Report.create_from_yaml_file failed with ActiveRecord::StatementInvalid: Mysql::Error: Data too long for column 'details' at row 1: INSERT INTO `delayed_job_failure...

I "googled" a little bit and I found out this solution that I hope this time is a permanent one ;)

mysql> use dashboard_production;
mysql> describe delayed_job_failures;
+------------+--------------+------+-----+---------+----------------+
| Field      | Type         | Null | Key | Default | Extra          |
+------------+--------------+------+-----+---------+----------------+
| id         | int(11)      | NO   | PRI | NULL    | auto_increment |
| summary    | varchar(255) | YES  |     | NULL    |                |
| details    | text         | YES  |     | NULL    |                |
| read       | tinyint(1)   | NO   |     | 0       |                |
| created_at | datetime     | YES  |     | NULL    |                |
| updated_at | datetime     | YES  |     | NULL    |                |
| backtrace  | text         | YES  |     | NULL    |                |
+------------+--------------+------+-----+---------+----------------+
7 rows in set (0.00 sec)

mysql> 
mysql> ALTER TABLE delayed_job_failures MODIFY details BLOB;
Query OK, 0 rows affected (0.51 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> 

Also, we will need to change the Type on the "report_logs":

mysql> ALTER TABLE report_logs MODIFY message VARCHAR(65536);
Query OK, 46574 rows affected, 2 warnings (0.97 sec)
Records: 46574  Duplicates: 0  Warnings: 2

mysql> describe report_logs;
+-----------+--------------+------+-----+---------+----------------+
| Field     | Type         | Null | Key | Default | Extra          |
+-----------+--------------+------+-----+---------+----------------+
| id        | int(11)      | NO   | PRI | NULL    | auto_increment |
| report_id | int(11)      | NO   | MUL | NULL    |                |
| level     | varchar(255) | YES  |     | NULL    |                |
| message   | mediumtext   | YES  |     | NULL    |                |
| source    | text         | YES  |     | NULL    |                |
| tags      | text         | YES  |     | NULL    |                |
| time      | datetime     | YES  |     | NULL    |                |
| file      | text         | YES  |     | NULL    |                |
| line      | int(11)      | YES  |     | NULL    |                |
+-----------+--------------+------+-----+---------+----------------+
9 rows in set (0.00 sec)

mysql> 

;----

Previous Solution (19/05/2014):

I just update our Puppet Master Node to the latest Puppet Version: 3.6.0 and after the Upgrade, everything seemed fine, regardless some Ruby Warnings, until I noticed that ALL the Nodes on the Puppet Dashboard were listed as: "Unresponsive" and all Tasks where "Pending".
I have to confess, it took me a while to find a proper solution on Google, until I got it.

If this is your case, you can run the following Steps on the node where you're running the Puppet Dashboard:

1- Stop the Dashboard Workers

2- Execute these Steps:
cd /usr/share/puppet-dashboard/
l spool/
rm -v spool/*
rake jobs:clear RAILS_ENV=production

3- Start again the Dashboard Workers.

This is a Solution that at least for now, it worked for me.

I hope it helps!