– Main.JehanProcaccia - 17 Dec 2007
The objectives is to collect data to monitor (cpu, memory, application
data exp here with openldap …) and to generate a graph that can be viewed by a web browser. We'll fetch data with snmp using `net-snnmp' : http://www.ne-snmp.org/, which will retreive a particular value based on a specific snmp oid. `MRTG' http://people.ee.ethz.ch/~oetiker/webtools/mrtg/mrtg.html will fetch these snmpd datas and archive (log) them in the ` rrd ' format, archives from witch a cgi-bin script will generate the web graphs.
Configure the snmp server through the ` snmpd.conf ' file and test.
$ vi /etc/snmp/snmpd.conf # First, map the community name "public" into a "security name" # sec.name source community ##com2sec notConfigUser default public com2sec local localhost secret com2sec mynetwork 192.168.21.0/24 secret com2sec mynetwork 192.168.50.0/24 secret # Second, map the security name into a group name: # groupName securityModel securityName ##group notConfigGroup v1 notConfigUser ##group notConfigGroup v2c notConfigUser group MyRWGroup v1 local group MyRWGroup v2c local group MyRWGroup usm local group MyROGroup v1 mynetwork group MyROGroup v2c mynetwork group MyROGroup usm mynetwork # Third, create a view for us to let the group have rights to: # name incl/excl subtree mask(optional) #view systemview included system #Grande ouverture! sur la racine des oid view all included .1 # Finally, grant the group read-only access to the systemview view. # group context sec.model sec.level prefix read write notif #access notConfigGroup "" any noauth exact systemview none none access MyROGroup "" any noauth exact all none none access MyRWGroup "" any noauth exact all all none syslocation Unknown (edit /etc/snmp/snmpd.conf) syscontact Root <root@localhost> (configure /etc/snmp/snmp.local.conf)
Start the snmpd server and configure the system so that it start at
reboot. (exp from RedHat commands !)
$ /etc/init.d/snmpd start Starting snmpd: [ OK ] $ chkconfig --level 345 snmpd on $ chkconfig --list | grep snmpd snmpd 0:off 1:off 2:off 3:on 4:on 5:on 6:off
Use snmp tools to check that everything works fine.
$ snmpget -c secret -v 1 localhost .1.3.6.1.4.1.2021.10.1.5.1 UCD-SNMP-MIB::laLoadInt.1 = INTEGER: 59
Creation of a configuration file that creates the "targets" ->
`Target[name]', followed by the 2 oids that we want to have on the same graph, the rest represent the options … scale, legend names, title, cf → http://people.ee.ethz.ch/~oetiker/webtools/mrtg/reference.html for details.
$ vi /etc/mrtg/corbeau.cfg # #################### # Global Configuration # #################### Language: french LogFormat: rrdtool # Where should the logfiles, and webpages be created? WorkDir: /var/www/html/mrtg/serveur/ WriteExpires: Yes IconDir: /mrtg/ LoadMIBs: /usr/share/snmp/mibs/UCD-SNMP-MIB.txt PageTop[^]: <H1>corbeau</H1><HR> #PageTop[$]: Contact Peter Norton if you have any questions<HR> PageFoot[^]: <i>Page managed by <a href="mailto:jehan.procaccia@int-evry.fr">\ Jehan Procaccia</a></i> Target[corbeauload]: .1.3.6.1.4.1.2021.10.1.3.1&.1.3.6.1.4.1.2021.10.1.3.3:\ secret@corbeau.int-evry.fr Directory[corbeauload]: corbeau MaxBytes[corbeauload]: 1000 Title[corbeauload]: load on server corbeau Options[corbeauload]: gauge, growright, nopercent PageTop[corbeauload]: <h2>load on server corbeau</h2> YLegend[corbeauload]: load on server corbeau ShortLegend[corbeauload]: n LegendI[corbeauload]: Load (1mn) LegendO[corbeauload]: Load (15mn) Legend1[corbeauload]: on the latest minute Legend2[corbeauload]: on 15 minutes Target[corbeauswap]: .1.3.6.1.4.1.2021.4.3.0&.1.3.6.1.4.1.2021.4.4.0:\ secret@corbeau.int-evry.fr Directory[corbeauswap]: corbeau MaxBytes[corbeauswap]: 100000000 Title[corbeauswap]: swap on server corbeau Options[corbeauswap]: gauge, growright, nopercent PageTop[corbeauswap]: <h2>swap on server corbeau</h2> YLegend[corbeauswap]: swap on server corbeau ShortLegend[corbeauswap]: ko LegendI[corbeauswap]: total swap LegendO[corbeauswap]: available swap Legend1[corbeauswap]: total swap Legend2[corbeauswap]: available swap Target[corbeaumem]: .1.3.6.1.4.1.2021.4.5.0&.1.3.6.1.4.1.2021.4.6.0:\ secret@corbeau.int-evry.fr::5::2: Directory[corbeaumem]: corbeau MaxBytes[corbeaumem]: 100000000 Title[corbeaumem]: RAM on server corbeau Options[corbeaumem]: gauge, growright, nopercent PageTop[corbeaumem]: <h2>RAM on server corbeau</h2> YLegend[corbeaumem]: RAM on server corbeau ShortLegend[corbeaumem]: ko LegendI[corbeaumem]: Total RAM corbeau LegendO[corbeaumem]: Available RAM corbeau Legend1[corbeaumem]: total RAM Legend2[corbeaumem]: available RAM Target[corbeaueth0]: .1.3.6.1.2.1.2.2.1.10.2&.1.3.6.1.2.1.2.2.1.16.2:\ secret@corbeau.int-evry.fr::4::2: Directory[corbeaueth0]: corbeau MaxBytes[corbeaueth0]: 10000000 Title[corbeaueth0]: if eth0 corbeau Options[corbeaueth0]: growright, nopercent, bits PageTop[corbeaueth0]: <h2>trafic eth0 corbeau</h2> YLegend[corbeaueth0]: trafic eth0 corbeau ShortLegend[corbeaueth0]: b/s LegendI[corbeaueth0]: bits IN LegendO[corbeaueth0]: bits OUT Legend1[corbeaueth0]: IN eth0 Legend2[corbeaueth0]: OUT eth0 Target[tcp_established]: .1.3.6.1.2.1.6.9.0&.1.3.6.1.2.1.6.9.0:\ secret@localhost Directory[tcp_established]: corbeau MaxBytes[tcp_established]: 100 AbsMax[tcp_established]: 110 Title[tcp_established]: Linux # of established TCP Connections PageTop[tcp_established]: Linux # of established TCP Connections YLegend[tcp_established]: # conn ShortLegend[tcp_established]: # conn Legend1[tcp_established]: # of established TCP connections LegendI[tcp_established]: # of established TCP connections Options[tcp_established]: growright, integer, gauge, noinfo, nopercent
First launch:
$ /usr/bin/mrtg /etc/mrtg/corbeau.cfg $ ls -ltra /var/www/html/mrtg/serveur/corbeau/ total 812 drwxr-xr-x 3 root root 4096 Jan 13 19:40 .. drwxr-xr-x 2 jehan root 4096 Jan 13 23:12 . -rw-r--r-- 1 root root 94660 Jan 14 11:27 tcp_established.rrd -rw-r--r-- 1 root root 94660 Jan 14 11:27 corbeauswap.rrd -rw-r--r-- 1 root root 94660 Jan 14 11:27 corbeaumem.rrd -rw-r--r-- 1 root root 94660 Jan 14 11:27 corbeauload.rrd -rw-r--r-- 1 root root 94660 Jan 14 11:27 corbeaueth0.rrd
Automatic launch every 5mn with cron:
$ cat /etc/cron.d/mrtg 0-59/5 * * * * root /usr/bin/mrtg /etc/mrtg/corbeau.cfg
Check the content of rrd files.
$ rrdtool fetch corbeauload.rrd AVERAGE .... 1042539300: 1.3033333333e-01 3.7020000000e-01 1042539600: 2.8000000000e-01 3.4000000000e-01 1042539900: 6.7866666667e-01 3.1010000000e-01 1042540200: 3.1030000000e-01 2.7456666667e-01 1042540500: 1.6000000000e-01 2.0000000000e-01 1042540800: nan nan
Creation of an index.html file, which will be the gate to the other
(per target) graph files.
$ indexmaker /etc/mrtg/corbeau.cfg --output /var/www/html/mrtg/serveur/corbeau/index.html
Final display is made by default through the cgi-bin ` 14all.cgi '
file, if we used the above `indexmaker' command. In that case the file has to be copied to the right destination, and configured to read the right mrtg file:
$ rpm -qli mrtg | grep 14all.cgi /usr/share/doc/mrtg-2.9.17/contrib/14all/14all.cgi $ cp /usr/share/doc/mrtg-2.9.17/contrib/14all/14all.cgi /var/www/cgi-bin/ $ vi /var/www/cgi-bin/14all.cgi #$cfgfile = '/home/mrtg/mrtg.cfg'; $cfgfile = '/etc/mrtg/corbeau.cfg';
Moreover the predefined oid values, we can monitor any kind of data
generated by scripts. Here we'll take the example of monitoring an openldap server. For this we'll fetch the data from the backend-monitor backend provided by openldap 2.1.X versions. Snmp associated with a perl script, will provide the data to MRTG.
Openldap configuration.
$ vi /etc/openldap/slapd.conf
database monitor
access to dn.subtree=cn=monitor
by dn.exact=cn=admin,dc=int-evry,dc=fr write
by dn.subtree=dc=int-evry,dc=fr read
by * none
Test with openldap shell tools.
$ ldapsearch -x -b "cn=Total,cn=connections,cn=monitor" -D "cn=admin,dc=int-evry,dc=fr" -W description -LLL Enter LDAP Password: dn: cn=Total,cn=Connections,cn=Monitor description: 95
Vincent Mathieu (Vincent.Mathieu AT univ-nancy2.fr) wrote a script that
gets values from ldap queries on the monitor backend. I modified it to enable the retrieve of single value data to cope with MRTG/RRD witch need a single numeric value to generates logs (arhives) and graph them.
The script: cf
http://www.int-evry.fr/mci/user/procacci/Files/monitor-jp.pl
Example, console display of openldap stats, followed by a single value
request example for mrtg (cf script header for details of usage)
$ /root/monitor-jp.pl TotConnexions : 99 CurrentConnexions : 6 ReadWaiters : 6 WriteWaiters : 0 TotBytes : 296129 TotEntries : 2630 TotPDU : 5149 TotOperations : 2582 TotAdd : 0 TotBind : 99 TotCompare : 0 TotDelete : 0 TotModify : 2279 TotSearch : 147 TotUnbind : 63 $ /root/monitor-jp.pl 1 mrtg valTotConnexions 100
The monitor database schema has change from 2.1.X to 2.2.X. A very
kind contributor (Spyridon.Iliopoulos AT Physik.Uni-Muenchen.DE ) has change the original `monitor-jp.pl' script to make it work on openldap 2.2.X monitor database.
Here it is:
http://www.int-evry.fr/mci/user/procacci/Files/monitor-jp-2.2.X.pl
This new version of the monitor script includes an enhancement, it
allows SALS authentificated connexions (optionnaly, just comment out the start-tls line), so it needs new perl packages
[root@corbeau ~] $ yum install perl-Authen-SASL perl-IO-Socket-SSL ... Installed: perl-Authen-SASL 2.08-1.1.fc2.dries.i386 perl-IO-Socket-SSL 0.95-1.rhfc2.at.noarch Dep Installed: perl-Net_SSLeay.pm 1.25-2.rhfc2.at.i386 perl-Digest-HMAC 1.01-12.noarch perl-Digest-SHA1 2.07-4.i386
Here's an exemple of `ldapsearch' under 2.2.X
[root@corbeau /usr/local/openldap-2.2.17-1] $ ./etc/rc.d/init.d/ldap start Starting slapd: [ OK ] $ ldapsearch -x -b "cn=Search,cn=Operations,cn=Monitor" -D "cn=admin,dc=int-evry,dc=fr" -W monitoredInfo monitorCounter -LLL Enter LDAP Password: dn: cn=Search,cn=Operations,cn=Monitor monitorOpInitiated: 39 monitorOpCompleted: 38
$ perl /tmp/monitor-jp-2.2.X.pl TotConnexions : 12 CurrentConnexions : 1 NumThreads : ReadWaiters : 1 WriteWaiters : 0 TotBytes : 17819 TotEntries : 184 TotPDU : 244 TotOperations : 1 TotAdd : 0 TotBind : 11 TotCompare : 0 TotDelete : 0 TotModify : 0 TotSearch : 54 TotUnbind : 10
$ perl /tmp/monitor-jp-2.2.X.pl 1 mrtg valTotConnexions 13
Associate the script to snmpd
$ grep monitor-jp /etc/snmp/snmpd.conf exec monitor-jp.pl /usr/bin/perl /root/monitor-jp.pl 1 mrtg valTotConnexions exec monitor-jp.pl /usr/bin/perl /root/monitor-jp.pl 1 mrtg valTotSearch $ /etc/init.d/snmpd restart Stopping snmpd: [ OK ] Starting snmpd: [ OK ]
The first script `snmpd.conf' is accessible through the oid
`.1.3.6.1.4.1.2021.8.1.101.1', the second with `…101.2', etc …
$ snmpget -c secret -v 1 localhost .1.3.6.1.4.1.2021.8.1.101.1 UCD-SNMP-MIB::extOutput.1 = STRING: 104
Now we can define an ` MRTG target' for these oids:
$ grep Slapd /etc/mrtg/corbeau.cfg Target[corbeauSlapdCx]: .1.3.6.1.4.1.2021.8.1.101.1\ &.1.3.6.1.4.1.2021.8.1.101.2:secret@localhost Directory[corbeauSlapdCx]: corbeau MaxBytes[corbeauSlapdCx]: 100000 Title[corbeauSlapdCx]: Connexion/search LDAP Options[corbeauSlapdCx]: growright, nopercent, perminute PageTop[corbeauSlapdCx]: <h2>Number of LDAP connexions Search </h2> YLegend[corbeauSlapdCx]: LDAP Cx ShortLegend[corbeauSlapdCx]: Cx LegendI[corbeauSlapdCx]: Cx LegendO[corbeauSlapdCx]: Search Legend1[corbeauSlapdCx]: Connexions Legend2[corbeauSlapdCx]: Search
net-snmp shell commands can help to debug mrtg. For example, we can
check snmp variable queries, check oids etc …
$ snmpget -v1 -c secret corbeau \ .iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets .2 IF-MIB::ifInOctets.2 = Counter32: 4300648 get the numeric oid: $ snmpget -v1 -c secret corbeau \ .iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets .2 -On .1.3.6.1.2.1.2.2.1.10.2 = Counter32: 4306487 get the oid as a string: $ snmpget -v1 -c secret corbeau .1.3.6.1.2.1.2.2.1.10.2 -Of .iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets .2 \ = Counter32: 4311842 or: $ snmptranslate .1.3.6.1.2.1.2.2.1.10.2 -Of .iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets .2 cf man snmpcmd for common options to snmp* utils.
contrib sur le forum cacti a ce sujet :
On a donc un tracé des mailq complet et efficasse sur:
URL sur le forum cacti pour debuger
il se peut qu'un datasource (fichier rrd) sature suivant la façon dont-il a été crée a verifier avec un rrdtool info nom_fichier_rrd
j'avais posté a ce sujet sur le forum cacti http://forums.cacti.net/viewtopic.php?p=116900#116900
extrait:
I also had to modify the data-template for deferred and hold to increase the max number from 2000 to 20000 (x10 !) that increase is available for new data sources, but for previous one , you have to tune the rrd file, exemple: bash-3.1# rrdtool info smtp1_sodome_incoming_732.rrd | grep deferred ds[deferred].max = 2.0000000000e+03 ds[deferred].last_ds = “14913”
bash-3.1# rrdtool tune smtp1_sodome_incoming_732.rrd --maximum deferred:20000 bash-3.1# rrdtool info smtp1_sodome_incoming_732.rrd | grep deferred ds[deferred].max = 2.0000000000e+04 ds[deferred].last_ds = "14746" ds[deferred].value = 1.4746000000e+04
Now it work for “long queues” …. !
recuperer l'archive ( mysql_stats.zip) et suivre le README.TXT en gros :
autoriser l'acces "stat" sur mysql GRANT PROCESS ON * TO cactiuser@'cactihost.com' IDENTIFIED by 'cactipasswd';
ici au final
GRANT PROCESS ON *.* TO cactiuser@'mon.cactihost.fr' IDENTIFIED by 'secret';
charger les template xml via l'import de l'interface cacti
copier le script dans les scripts de cacti /usr/share/cacti/scripts/mysql_stats.php
enfin creer les graphs via l'interface web de cacti
recuperation des sources
[root@mon ~]# wget https://www.percona.com/downloads/percona-monitoring-plugins/percona-monitoring-plugins-1.1.8/binary/redhat/7/x86_64/percona-cacti-templates-1.1.8-1.noarch.rpm [root@mon ~]# rpm -Uvh percona-cacti-templates-1.1.8-1.noarch.rpm --nodeps Scripts are installed to /usr/share/cacti/scripts Templates are installed to /usr/share/cacti/resource/percona
avec la fonction d'import de cacti :
Cacti has imported the following items for the Template: CDEF [succès] Percona Turn Into Bits CDEF [nouveau] [succès] Percona Negate CDEF [updated] Table: cdef, Column: name, New Value: 'Percona Negate CDEF', Old Value: 'X Negate CDEF' [succès] Percona Disk Write Time per IO Request CDEF [nouveau] [succès] Percona Disk Write Time per IO Request Negate CDEF [nouveau] [succès] Percona Disk Read Time per IO Request CDEF [nouveau] GPRINT Preset [succès] Percona GNU/Linux Server Checksum 611ca2224c24387fc25a62daf58371a2 [nouveau] [succès] Percona GNU/Linux Server Version t1.1.8:s1.1.8 [nouveau] [succès] Percona Normal [updated] Table: graph_templates_gprint, Column: name, New Value: 'Percona Normal', Old Value: 'X Normal' Métode d'entrée [succès] Percona Get Proc Stats/Interrupts IM [nouveau] [succès] Percona Get Proc Stats/Context Switches IM [nouveau] [succès] Percona Get Proc Stats/Forks IM [nouveau] [succès] Percona Get Proc Stats/CPU Usage IM [nouveau] [succès] Percona Get W/Load Average IM [nouveau] [succès] Percona Get W/Number of Users IM [nouveau] [succès] Percona Get Memory/Memory IM [nouveau] [succès] Percona Get Disk Stats/Disk Operations IM [nouveau] [succès] Percona Get Disk Stats/Disk Sectors Read/Written IM [nouveau] [succès] Percona Get Disk Stats/Disk Read/Write Time (ms) IM [nouveau] [succès] Percona Get Disk Stats/Disk Read/Write Time per IO Request (ms) IM [nouveau] [succès] Percona Get Disk Stats/Disk Elapsed IO Time (ms) IM [nouveau] [succès] Percona Get Disk Stats/Disk IOPS IM [nouveau] [succès] Percona Get Disk Space/Disk Space IM [nouveau] [succès] Percona Get Network Traffic/Network Traffic IM [nouveau] [succès] Percona Get Network Errors/Network Errors IM [nouveau] [succès] Percona Get Network Connection States/Network Connection States IM [nouveau] [succès] Percona Get Swap Usage/Swap Usage IM [nouveau] Data Template [succès] Percona Interrupts DT [nouveau] [succès] Percona Context Switches DT [nouveau] [succès] Percona Forks DT [nouveau] [succès] Percona CPU Usage DT [nouveau] [succès] Percona Load Average DT [nouveau] [succès] Percona Number of Users DT [nouveau] [succès] Percona Memory DT [nouveau] [succès] Percona Disk Operations DT [nouveau] [succès] Percona Disk Sectors Read/Written DT [nouveau] [succès] Percona Disk Read/Write Time (ms) DT [nouveau] [succès] Percona Disk Read/Write Time per IO Request (ms) DT [nouveau] [succès] Percona Disk Elapsed IO Time (ms) DT [nouveau] [succès] Percona Disk IOPS DT [nouveau] [succès] Percona Disk Space DT [nouveau] [succès] Percona Network Traffic DT [nouveau] [succès] Percona Network Errors DT [nouveau] [succès] Percona Network Connection States DT [nouveau] [succès] Percona Swap Usage DT [nouveau] Modèle de graphique [succès] Percona Interrupts GT [nouveau] [succès] Percona Context Switches GT [nouveau] [succès] Percona Forks GT [nouveau] [succès] Percona CPU Usage GT [nouveau] [succès] Percona Load Average GT [nouveau] [succès] Percona Number of Users GT [nouveau] [succès] Percona Memory GT [nouveau] [succès] Percona Disk Operations GT [nouveau] [succès] Percona Disk Sectors Read/Written GT [nouveau] [succès] Percona Disk Read/Write Time (ms) GT [nouveau] [succès] Percona Disk Read/Write Time per IO Request (ms) GT [nouveau] [succès] Percona Disk Elapsed IO Time (ms) GT [nouveau] [succès] Percona Disk IOPS GT [nouveau] [succès] Percona Disk Space GT [nouveau] [succès] Percona Network Traffic GT [nouveau] [succès] Percona Network Errors GT [nouveau] [succès] Percona Network Connection States GT [nouveau] [succès] Percona Swap Usage GT [nouveau] Device Template [succès] Percona GNU/Linux Server HT [nouveau]
sur un client a monitorer
[root@jap webapps]# useradd -u 1111 cacti [root@jap webapps]# passwd cacti Changement de mot de passe pour l'utilisateur cacti. Nouveau mot de passe : secret pass </code>
on partage la clé ssh du serveur “mon” avec le client
[root@mon ~]# ssh-copy-id -i /etc/cacti/id_rsa.pub cacti@jap.int.fr
ouvrir le firewall sur le client
# firewall-cmd --permanent --add-rich-rule 'rule family="ipv4" source address="155.155.2.0/24" service name="snmp" log prefix="snmp-155.155.2.0/24" accept' # firewall-cmd --reload
et enable snmp au boot
# systemctl enable snmpd.service Created symlink from /etc/systemd/system/multi-user.target.wants/snmpd.service to /usr/lib/systemd/system/snmpd.service.
scripts locaux
[root@sym1 snmp]# scp -r local-snmp-cronjob postfix-script-stats uloganalyser uloganalyser-plugin root@sym2.int.fr:/etc/snmp [root@sym1 cron.d]# scp local-snmp root@sym2.int.fr:/etc/cron.d/
repertoire dediés aux logs mail pour les scripts
[root@sym2]# mkdir /var/local/snmp
parallel link