Friday, June 27, 2014

Rotate Puppet-Dashboard delayed_job.log

There was a time where the delayed_job.log of the Puppet-Dashboard grew up to 8GB, so I had to find a solution, and this is it:

Edit the file: /etc/logrotate.d/puppet-dashboard and add the following lines:

# Puppet-Dashboard logs:
/usr/share/puppet-dashboard/log/delayed_job.log {
  daily
  rotate 7
  missingok
  compress
  delaycompress
  notifempty
  copytruncate
}
root@server:~# 

The, the logfiles will look like:

root@server:~# l /usr/share/puppet-dashboard/log/
total 1043256
drwxr-xr-x  2 www-data www-data       4096 Jun 27 05:10 ./
drwxr-xr-x 18 root     root           4096 Jun  4  2013 ../
-rw-r--r--  1 root     root         481078 Jun 27 11:33 delayed_job.log
-rw-r--r--  1 root     root        1811627 Jun 27 05:10 delayed_job.log.1
-rw-r--r--  1 root     root          78097 Jun 26 05:10 delayed_job.log.2.gz
root@server:~# 

A note for "copytruncate":
Truncate  the  original log file in place after creating a copy, instead of moving the old log file and optionally creating a new one,  It  can be used when some program can not be told to close its logfile and thus might continue writing (appending)  to  the previous log file forever.  Note that there is a very small time slice between copying the file and truncating it, so  some  logging  data  might be lost.  When this option is used, the create option will have no effect, as the old log file stays in  place.

Hope it helps! ;)

UPDATED!!! Puppet Dashboard: All Tasks are "Pending" and all Nodes "Unresponsive"

Well it seems that the previous Solution, was a temporary one. :(
So I kept debugging a little bit and I found this error on the delayed_job log (/usr/share/puppet-dashboard/log/delayed_job.log) :

... 2014-06-13T12:05:40+0200: [Worker(delayed_job.1 host:pmaster01 pid:21516)] Report.create_from_yaml_file failed with ActiveRecord::StatementInvalid: Mysql::Error: Data too long for column 'details' at row 1: INSERT INTO `delayed_job_failure...

I "googled" a little bit and I found out this solution that I hope this time is a permanent one ;)

mysql> use dashboard_production;
mysql> describe delayed_job_failures;
+------------+--------------+------+-----+---------+----------------+
| Field      | Type         | Null | Key | Default | Extra          |
+------------+--------------+------+-----+---------+----------------+
| id         | int(11)      | NO   | PRI | NULL    | auto_increment |
| summary    | varchar(255) | YES  |     | NULL    |                |
| details    | text         | YES  |     | NULL    |                |
| read       | tinyint(1)   | NO   |     | 0       |                |
| created_at | datetime     | YES  |     | NULL    |                |
| updated_at | datetime     | YES  |     | NULL    |                |
| backtrace  | text         | YES  |     | NULL    |                |
+------------+--------------+------+-----+---------+----------------+
7 rows in set (0.00 sec)

mysql> 
mysql> ALTER TABLE delayed_job_failures MODIFY details BLOB;
Query OK, 0 rows affected (0.51 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> 

Also, we will need to change the Type on the "report_logs":

mysql> ALTER TABLE report_logs MODIFY message VARCHAR(65536);
Query OK, 46574 rows affected, 2 warnings (0.97 sec)
Records: 46574  Duplicates: 0  Warnings: 2

mysql> describe report_logs;
+-----------+--------------+------+-----+---------+----------------+
| Field     | Type         | Null | Key | Default | Extra          |
+-----------+--------------+------+-----+---------+----------------+
| id        | int(11)      | NO   | PRI | NULL    | auto_increment |
| report_id | int(11)      | NO   | MUL | NULL    |                |
| level     | varchar(255) | YES  |     | NULL    |                |
| message   | mediumtext   | YES  |     | NULL    |                |
| source    | text         | YES  |     | NULL    |                |
| tags      | text         | YES  |     | NULL    |                |
| time      | datetime     | YES  |     | NULL    |                |
| file      | text         | YES  |     | NULL    |                |
| line      | int(11)      | YES  |     | NULL    |                |
+-----------+--------------+------+-----+---------+----------------+
9 rows in set (0.00 sec)

mysql> 

;----

Previous Solution (19/05/2014):

I just update our Puppet Master Node to the latest Puppet Version: 3.6.0 and after the Upgrade, everything seemed fine, regardless some Ruby Warnings, until I noticed that ALL the Nodes on the Puppet Dashboard were listed as: "Unresponsive" and all Tasks where "Pending".
I have to confess, it took me a while to find a proper solution on Google, until I got it.

If this is your case, you can run the following Steps on the node where you're running the Puppet Dashboard:

1- Stop the Dashboard Workers

2- Execute these Steps:
cd /usr/share/puppet-dashboard/
l spool/
rm -v spool/*
rake jobs:clear RAILS_ENV=production

3- Start again the Dashboard Workers.

This is a Solution that at least for now, it worked for me.

I hope it helps!