Bug #615

Interfacechecks switches to UNKNOWN state after restart of ICINGA

Added by crsp almost 4 years ago. Updated over 3 years ago.

Status:ClosedStart date:07/10/2010
Priority:NormalDue date:
Assignee:dnsmichi% Done:

0%

Category:Performance
Target version:-
Icinga Version: OS Version:

Description

As requested by Dnsmichi, i open an issue with a summary of the things we talked about in the IRC Channel on Friday 09.07.10 about 3pm.
Description of the bug(?):
Scenario: 2 Boxes, one hosts Icinga (HP DL360 G5, 4GB RAM, Quadcore CPU, Debian Lenny 64bit), the other one hosts the MySQL (HP DL360 G3, 2GB RAM, 2 CPUs, Debian Lenny 64bit), both connected with Gigabit to the same Switch.
I upgrated from Icinga 1.0.1 and applied the fix upgrade-1.0.2-fix-object-relations.sql for the mysql as well as the mysql-upgrade-1.0.2.sql.
I use IDOUTILS with MySQL. When i restart or reload Icinga, the mysql process on the mysql-host uses 60 to 100% CPU for about one minute and the Icinga box has normal load.
In 1.0.1 the shell prompt was shown very fast after hitting return, now with 1.0.2 it takes about half a minute. The load of both boxes during that "restart period" is the same as in 1.0.1.
The most annoying thing is, that most of the interface checks (i use check_iftraffic.pl) that are running during this "restart period" are going UNKNOWN until the next checkrun.
I had this issue with RC1 and 1.0.1 when i ran Icinga and the Mysql DB on the same machine (the DL360 G5). But it was gone after moving the DB to a separate host. Now with 1.0.2 it is back. I have about 3645 objects in the icinga_objects table.

History

#1 Updated by dnsmichi almost 4 years ago

so to speak the db is on a seperate host, but not ido2db itsself right? you could consider setting up

g5 icinga core with idomod+tcp socket
g3 ido2db+tcp socket with mysql rdbms

Maybe the altinity patch for extended hashslots was not that good for your setup. but i did not change that much in idoutils between those 2 releases except the database.. hmm.

#2 Updated by crsp almost 4 years ago

You're so right, this is exactly my setup:

g5 icinga core with idomod+tcp socket
g3 ido2db+tcp socket with mysql rdbms

#3 Updated by dnsmichi over 3 years ago

hm, what are your config_output_options in idomod.cfg?

#4 Updated by crsp over 3 years ago

it's set to 1: config_output_options=1

#5 Updated by dnsmichi over 3 years ago

hm, show me

select * from icinga_instances;

select count(*) as cnt from icinga_objects where is_ative=0;

select count(*) as cnt from icinga_objects join icinga_instances on icinga_objects.instance_id=icinga_instances.instance_id where icinga_objects.is_ative=0 and icinga_instances.instance_name='YOUR CURRENT INSTANCE NAME IN IDOMOD.CFG';

#6 Updated by crsp over 3 years ago

here you are:

select * from icinga_instances;

--------------------------+----------------------+ | instance_id | instance_name | instance_description |
--------------------------+----------------------+ | 1 | default | |
--------------------------+----------------------+
1 row in set (0.01 sec)

select count(*) as cnt from icinga_objects where is_ative=0;

ERROR 1054 (42S22): Unknown column 'is_ative' in 'where clause'

select count(*) as cnt from icinga_objects join icinga_instances on icinga_objects.instance_id=icinga_instances.instance_id where icinga_objects.is_ative=0 and icinga_instances.instance_name='YOUR CURRENT INSTANCE NAME IN IDOMOD.CFG';

mysql> select count(*) as cnt from icinga_objects join icinga_instances on icinga_objects.instance_id=icinga_instances.instance_id where icinga_objects.is_ative=0 and icinga_instances.instance_name='default';
ERROR 1054 (42S22): Unknown column 'icinga_objects.is_ative' in 'where clause'

#7 Updated by dnsmichi over 3 years ago

hach gottchen bisschen mitdenken :P

typo => is_ative == is_active

(show create table icinga_objects)

#8 Updated by crsp over 3 years ago

Mitdenken nu auch noch? :P
Ich vertraue immer dem Entwickler! :)

Aber hier:


mysql> select * from icinga_instances;
+-------------+---------------+----------------------+
| instance_id | instance_name | instance_description |
+-------------+---------------+----------------------+
|           1 | default       |                      |
+-------------+---------------+----------------------+
1 row in set (0.00 sec)

mysql>select count(*) as cnt from icinga_objects where is_active=0;

+-----+
| cnt |
+-----+
|  68 |
+-----+
1 row in set (0.01 sec)

mysql>select count(*) as cnt from icinga_objects join icinga_instances on icinga_objects.instance_id=icinga_instances.instance_id where icinga_objects.is_active=0 and icinga_instances.instance_name='default';

+-----+
| cnt |
+-----+
|  68 |
+-----+
1 row in set (0.02 sec)

#9 Updated by dnsmichi over 3 years ago

  • Status changed from New to Feedback

i can't reproduce that. can you recheck with 1.0.3 or the latest git master?

#10 Updated by crsp over 3 years ago

Updated today to the latest stuff from GIT. so far not too bad!
a restart produces about 40% load on the Mysql host. this is ways less than some time ago.
i test some more days, otherwise i think we are done.

mni tnx!!!

#11 Updated by dnsmichi over 3 years ago

ok very nice :)

#12 Updated by dnsmichi over 3 years ago

  • Status changed from Feedback to Closed

before i'll close the ticket - make sure you are using some way of ramdisk for pnp/perfdata. and put the mysql data stuff on an extended volume (like /var/lib), not on the same as /var/log and icinga.log reside.

but i think you'll find out yourself. closed for now.

Also available in: Atom PDF