[[monitor]]
This is an old revision of the document!
monitor
– Main.JehanProcaccia - 17 Dec 2007
Objectives and tools
The objectives is to collect data to monitor (cpu, memory, application
data exp here with openldap …) and to generate a graph that can be viewed by a web browser. We'll fetch data with snmp using `net-snnmp' : http://www.ne-snmp.org/, which will retreive a particular value based on a specific snmp oid. `MRTG' http://people.ee.ethz.ch/~oetiker/webtools/mrtg/mrtg.html will fetch these snmpd datas and archive (log) them in the ` rrd ' format, archives from witch a cgi-bin script will generate the web graphs.
Snmp
Configure the snmp server through the ` snmpd.conf ' file and test.
Configuration
$ vi /etc/snmp/snmpd.conf # First, map the community name "public" into a "security name" # sec.name source community ##com2sec notConfigUser default public com2sec local localhost secret com2sec mynetwork 192.168.21.0/24 secret com2sec mynetwork 192.168.50.0/24 secret # Second, map the security name into a group name: # groupName securityModel securityName ##group notConfigGroup v1 notConfigUser ##group notConfigGroup v2c notConfigUser group MyRWGroup v1 local group MyRWGroup v2c local group MyRWGroup usm local group MyROGroup v1 mynetwork group MyROGroup v2c mynetwork group MyROGroup usm mynetwork # Third, create a view for us to let the group have rights to: # name incl/excl subtree mask(optional) #view systemview included system #Grande ouverture! sur la racine des oid view all included .1 # Finally, grant the group read-only access to the systemview view. # group context sec.model sec.level prefix read write notif #access notConfigGroup "" any noauth exact systemview none none access MyROGroup "" any noauth exact all none none access MyRWGroup "" any noauth exact all all none syslocation Unknown (edit /etc/snmp/snmpd.conf) syscontact Root <root@localhost> (configure /etc/snmp/snmp.local.conf)
Tests
Start the snmpd server and configure the system so that it start at
reboot. (exp from RedHat commands !)
$ /etc/init.d/snmpd start Starting snmpd: [ OK ] $ chkconfig --level 345 snmpd on $ chkconfig --list | grep snmpd snmpd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
Use snmp tools to check that everything works fine.
$ snmpget -c secret -v 1 localhost .1.3.6.1.4.1.2021.10.1.5.1 UCD-SNMP-MIB::laLoadInt.1 = INTEGER: 59
MRTG/RRD
Configuration
Creation of a configuration file that creates the "targets" ->
`Target[name]', followed by the 2 oids that we want to have on the same graph, the rest represent the options … scale, legend names, title, cf → http://people.ee.ethz.ch/~oetiker/webtools/mrtg/reference.html for details.
$ vi /etc/mrtg/corbeau.cfg # #################### # Global Configuration # #################### Language: french LogFormat: rrdtool # Where should the logfiles, and webpages be created? WorkDir: /var/www/html/mrtg/serveur/ WriteExpires: Yes IconDir: /mrtg/ LoadMIBs: /usr/share/snmp/mibs/UCD-SNMP-MIB.txt PageTop[^]: <H1>corbeau</H1><HR> #PageTop[$]: Contact Peter Norton if you have any questions<HR> PageFoot[^]: <i>Page managed by <a href="mailto:jehan.procaccia@int-evry.fr">\ Jehan Procaccia</a></i> Target[corbeauload]: .1.3.6.1.4.1.2021.10.1.3.1&.1.3.6.1.4.1.2021.10.1.3.3:\ secret@corbeau.int-evry.fr Directory[corbeauload]: corbeau MaxBytes[corbeauload]: 1000 Title[corbeauload]: load on server corbeau Options[corbeauload]: gauge, growright, nopercent PageTop[corbeauload]: <h2>load on server corbeau</h2> YLegend[corbeauload]: load on server corbeau ShortLegend[corbeauload]: n LegendI[corbeauload]: Load (1mn) LegendO[corbeauload]: Load (15mn) Legend1[corbeauload]: on the latest minute Legend2[corbeauload]: on 15 minutes Target[corbeauswap]: .1.3.6.1.4.1.2021.4.3.0&.1.3.6.1.4.1.2021.4.4.0:\ secret@corbeau.int-evry.fr Directory[corbeauswap]: corbeau MaxBytes[corbeauswap]: 100000000 Title[corbeauswap]: swap on server corbeau Options[corbeauswap]: gauge, growright, nopercent PageTop[corbeauswap]: <h2>swap on server corbeau</h2> YLegend[corbeauswap]: swap on server corbeau ShortLegend[corbeauswap]: ko LegendI[corbeauswap]: total swap LegendO[corbeauswap]: available swap Legend1[corbeauswap]: total swap Legend2[corbeauswap]: available swap Target[corbeaumem]: .1.3.6.1.4.1.2021.4.5.0&.1.3.6.1.4.1.2021.4.6.0:\ secret@corbeau.int-evry.fr::5::2: Directory[corbeaumem]: corbeau MaxBytes[corbeaumem]: 100000000 Title[corbeaumem]: RAM on server corbeau Options[corbeaumem]: gauge, growright, nopercent PageTop[corbeaumem]: <h2>RAM on server corbeau</h2> YLegend[corbeaumem]: RAM on server corbeau ShortLegend[corbeaumem]: ko LegendI[corbeaumem]: Total RAM corbeau LegendO[corbeaumem]: Available RAM corbeau Legend1[corbeaumem]: total RAM Legend2[corbeaumem]: available RAM Target[corbeaueth0]: .1.3.6.1.2.1.2.2.1.10.2&.1.3.6.1.2.1.2.2.1.16.2:\ secret@corbeau.int-evry.fr::4::2: Directory[corbeaueth0]: corbeau MaxBytes[corbeaueth0]: 10000000 Title[corbeaueth0]: if eth0 corbeau Options[corbeaueth0]: growright, nopercent, bits PageTop[corbeaueth0]: <h2>trafic eth0 corbeau</h2> YLegend[corbeaueth0]: trafic eth0 corbeau ShortLegend[corbeaueth0]: b/s LegendI[corbeaueth0]: bits IN LegendO[corbeaueth0]: bits OUT Legend1[corbeaueth0]: IN eth0 Legend2[corbeaueth0]: OUT eth0 Target[tcp_established]: .1.3.6.1.2.1.6.9.0&.1.3.6.1.2.1.6.9.0:\ secret@localhost Directory[tcp_established]: corbeau MaxBytes[tcp_established]: 100 AbsMax[tcp_established]: 110 Title[tcp_established]: Linux # of established TCP Connections PageTop[tcp_established]: Linux # of established TCP Connections YLegend[tcp_established]: # conn ShortLegend[tcp_established]: # conn Legend1[tcp_established]: # of established TCP connections LegendI[tcp_established]: # of established TCP connections Options[tcp_established]: growright, integer, gauge, noinfo, nopercent
Launch
First launch:
$ /usr/bin/mrtg /etc/mrtg/corbeau.cfg $ ls -ltra /var/www/html/mrtg/serveur/corbeau/ total 812 drwxr-xr-x 3 root root 4096 Jan 13 19:40 .. drwxr-xr-x 2 jehan root 4096 Jan 13 23:12 . -rw-r--r-- 1 root root 94660 Jan 14 11:27 tcp_established.rrd -rw-r--r-- 1 root root 94660 Jan 14 11:27 corbeauswap.rrd -rw-r--r-- 1 root root 94660 Jan 14 11:27 corbeaumem.rrd -rw-r--r-- 1 root root 94660 Jan 14 11:27 corbeauload.rrd -rw-r--r-- 1 root root 94660 Jan 14 11:27 corbeaueth0.rrd
Automatic launch every 5mn with cron:
$ cat /etc/cron.d/mrtg 0-59/5 * * * * root /usr/bin/mrtg /etc/mrtg/corbeau.cfg
Check the content of rrd files.
$ rrdtool fetch corbeauload.rrd AVERAGE .... 1042539300: 1.3033333333e-01 3.7020000000e-01 1042539600: 2.8000000000e-01 3.4000000000e-01 1042539900: 6.7866666667e-01 3.1010000000e-01 1042540200: 3.1030000000e-01 2.7456666667e-01 1042540500: 1.6000000000e-01 2.0000000000e-01 1042540800: nan nan
index HTML file
Creation of an index.html file, which will be the gate to the other
(per target) graph files.
$ indexmaker /etc/mrtg/corbeau.cfg --output /var/www/html/mrtg/serveur/corbeau/index.html
cgi-bin file
Final display is made by default through the cgi-bin ` 14all.cgi '
file, if we used the above `indexmaker' command. In that case the file has to be copied to the right destination, and configured to read the right mrtg file:
$ rpm -qli mrtg | grep 14all.cgi /usr/share/doc/mrtg-2.9.17/contrib/14all/14all.cgi $ cp /usr/share/doc/mrtg-2.9.17/contrib/14all/14all.cgi /var/www/cgi-bin/ $ vi /var/www/cgi-bin/14all.cgi #$cfgfile = '/home/mrtg/mrtg.cfg'; $cfgfile = '/etc/mrtg/corbeau.cfg';
Scripts for application datas
Moreover the predefined oid values, we can monitor any kind of data
generated by scripts. Here we'll take the example of monitoring an openldap server. For this we'll fetch the data from the backend-monitor backend provided by openldap 2.1.X versions. Snmp associated with a perl script, will provide the data to MRTG.
back monitor
Openldap configuration.
$ vi /etc/openldap/slapd.conf
database monitor
access to dn.subtree=cn=monitor
by dn.exact=cn=admin,dc=int-evry,dc=fr write
by dn.subtree=dc=int-evry,dc=fr read
by * none
Test with openldap shell tools.
$ ldapsearch -x -b "cn=Total,cn=connections,cn=monitor" -D "cn=admin,dc=int-evry,dc=fr" -W description -LLL Enter LDAP Password: dn: cn=Total,cn=Connections,cn=Monitor description: 95
Monitor Script
Vincent Mathieu (Vincent.Mathieu AT univ-nancy2.fr) wrote a script that
gets values from ldap queries on the monitor backend. I modified it to enable the retrieve of single value data to cope with MRTG/RRD witch need a single numeric value to generates logs (arhives) and graph them.
The script: cf
http://www.int-evry.fr/mci/user/procacci/Files/monitor-jp.pl
Example, console display of openldap stats, followed by a single value
request example for mrtg (cf script header for details of usage)
$ /root/monitor-jp.pl TotConnexions : 99 CurrentConnexions : 6 ReadWaiters : 6 WriteWaiters : 0 TotBytes : 296129 TotEntries : 2630 TotPDU : 5149 TotOperations : 2582 TotAdd : 0 TotBind : 99 TotCompare : 0 TotDelete : 0 TotModify : 2279 TotSearch : 147 TotUnbind : 63 $ /root/monitor-jp.pl 1 mrtg valTotConnexions 100
Monitor script for openldap 2.2.X
The monitor database schema has change from 2.1.X to 2.2.X. A very
kind contributor (Spyridon.Iliopoulos AT Physik.Uni-Muenchen.DE ) has change the original `monitor-jp.pl' script to make it work on openldap 2.2.X monitor database.
Here it is:
http://www.int-evry.fr/mci/user/procacci/Files/monitor-jp-2.2.X.pl
Perl Pakages
This new version of the monitor script includes an enhancement, it
allows SALS authentificated connexions (optionnaly, just comment out the start-tls line), so it needs new perl packages
[root@corbeau ~] $ yum install perl-Authen-SASL perl-IO-Socket-SSL ... Installed: perl-Authen-SASL 2.08-1.1.fc2.dries.i386 perl-IO-Socket-SSL 0.95-1.rhfc2.at.noarch Dep Installed: perl-Net_SSLeay.pm 1.25-2.rhfc2.at.i386 perl-Digest-HMAC 1.01-12.noarch perl-Digest-SHA1 2.07-4.i386
LdapSearch monitor database
Here's an exemple of `ldapsearch' under 2.2.X
[root@corbeau /usr/local/openldap-2.2.17-1] $ ./etc/rc.d/init.d/ldap start Starting slapd: [ OK ] $ ldapsearch -x -b "cn=Search,cn=Operations,cn=Monitor" -D "cn=admin,dc=int-evry,dc=fr" -W monitoredInfo monitorCounter -LLL Enter LDAP Password: dn: cn=Search,cn=Operations,cn=Monitor monitorOpInitiated: 39 monitorOpCompleted: 38
Browse the monitor database with the script
$ perl /tmp/monitor-jp-2.2.X.pl TotConnexions : 12 CurrentConnexions : 1 NumThreads : ReadWaiters : 1 WriteWaiters : 0 TotBytes : 17819 TotEntries : 184 TotPDU : 244 TotOperations : 1 TotAdd : 0 TotBind : 11 TotCompare : 0 TotDelete : 0 TotModify : 0 TotSearch : 54 TotUnbind : 10
Search with mrtg output
$ perl /tmp/monitor-jp-2.2.X.pl 1 mrtg valTotConnexions 13
Snmpd configuration
Associate the script to snmpd
$ grep monitor-jp /etc/snmp/snmpd.conf exec monitor-jp.pl /usr/bin/perl /root/monitor-jp.pl 1 mrtg valTotConnexions exec monitor-jp.pl /usr/bin/perl /root/monitor-jp.pl 1 mrtg valTotSearch $ /etc/init.d/snmpd restart Stopping snmpd: [ OK ] Starting snmpd: [ OK ]
MRTG configuration
The first script `snmpd.conf' is accessible through the oid
`.1.3.6.1.4.1.2021.8.1.101.1', the second with `…101.2', etc …
$ snmpget -c secret -v 1 localhost .1.3.6.1.4.1.2021.8.1.101.1 UCD-SNMP-MIB::extOutput.1 = STRING: 104
Now we can define an ` MRTG target' for these oids:
$ grep Slapd /etc/mrtg/corbeau.cfg Target[corbeauSlapdCx]: .1.3.6.1.4.1.2021.8.1.101.1\ &.1.3.6.1.4.1.2021.8.1.101.2:secret@localhost Directory[corbeauSlapdCx]: corbeau MaxBytes[corbeauSlapdCx]: 100000 Title[corbeauSlapdCx]: Connexion/search LDAP Options[corbeauSlapdCx]: growright, nopercent, perminute PageTop[corbeauSlapdCx]: <h2>Number of LDAP connexions Search </h2> YLegend[corbeauSlapdCx]: LDAP Cx ShortLegend[corbeauSlapdCx]: Cx LegendI[corbeauSlapdCx]: Cx LegendO[corbeauSlapdCx]: Search Legend1[corbeauSlapdCx]: Connexions Legend2[corbeauSlapdCx]: Search
snmp tools
net-snmp shell commands can help to debug mrtg. For example, we can
check snmp variable queries, check oids etc …
$ snmpget -v1 -c secret corbeau \ .iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets .2 IF-MIB::ifInOctets.2 = Counter32: 4300648 get the numeric oid: $ snmpget -v1 -c secret corbeau \ .iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets .2 -On .1.3.6.1.2.1.2.2.1.10.2 = Counter32: 4306487 get the oid as a string: $ snmpget -v1 -c secret corbeau .1.3.6.1.2.1.2.2.1.10.2 -Of .iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets .2 \ = Counter32: 4311842 or: $ snmptranslate .1.3.6.1.2.1.2.2.1.10.2 -Of .iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets .2 cf man snmpcmd for common options to snmp* utils.
Contrib
contrib sur le forum cacti a ce sujet :
On a donc un tracé des mailq complet et efficasse sur:
URL sur le forum cacti pour debuger
snmp translate tools cisco
rrdtool
il se peut qu'un datasource (fichier rrd) sature suivant la façon dont-il a été crée a verifier avec un rrdtool info nom_fichier_rrd
j'avais posté a ce sujet sur le forum cacti http://forums.cacti.net/viewtopic.php?p=116900#116900
extrait:
I also had to modify the data-template for deferred and hold to increase the max number from 2000 to 20000 (x10 !) that increase is available for new data sources, but for previous one , you have to tune the rrd file, exemple: bash-3.1# rrdtool info smtp1_sodome_incoming_732.rrd | grep deferred ds[deferred].max = 2.0000000000e+03 ds[deferred].last_ds = “14913”
bash-3.1# rrdtool tune smtp1_sodome_incoming_732.rrd --maximum deferred:20000 bash-3.1# rrdtool info smtp1_sodome_incoming_732.rrd | grep deferred ds[deferred].max = 2.0000000000e+04 ds[deferred].last_ds = "14746" ds[deferred].value = 1.4746000000e+04
Now it work for “long queues” …. !
