This shows you the differences between two versions of the page.
| Next revision | Previous revision | ||
|
docpublic:reseaux:services:monitor [2011/06/10 07:16] PROCACCIA created |
docpublic:reseaux:services:monitor [2018/05/21 18:08] (current) procacci@tem-tsp.eu [postfix] |
||
|---|---|---|---|
| Line 5: | Line 5: | ||
| - | %TOC% | + | ===== |
| - | + | ||
| - | ---# Objectives and tools | + | |
| | | ||
| Line 20: | Line 18: | ||
| | | ||
| - | 2 Snmp | + | ===== Snmp ===== |
| - | *=*=*=* | + | |
| | | ||
| | | ||
| - | ---## Configuration | + | ==== |
| | | ||
| - | << | + | <code> |
| $ vi / | $ vi / | ||
| | | ||
| Line 72: | Line 69: | ||
| syscontact Root < | syscontact Root < | ||
| | | ||
| - | >> | + | </code> |
| | | ||
| | | ||
| - | 2.2 Tests | + | ==== Tests ==== |
| - | ========== | + | |
| Start the snmpd server and configure the system so that it start at | Start the snmpd server and configure the system so that it start at | ||
| reboot. (exp from RedHat commands !) | reboot. (exp from RedHat commands !) | ||
| - | << | + | <code> |
| $ / | $ / | ||
| Starting snmpd: | Starting snmpd: | ||
| Line 86: | Line 82: | ||
| $ chkconfig --list | grep snmpd | $ chkconfig --list | grep snmpd | ||
| snmpd | snmpd | ||
| - | >> | + | </code> |
| | | ||
| Use snmp tools to check that everything works fine. | Use snmp tools to check that everything works fine. | ||
| - | << | + | <code> |
| $ snmpget -c secret -v 1 localhost .1.3.6.1.4.1.2021.10.1.5.1 | $ snmpget -c secret -v 1 localhost .1.3.6.1.4.1.2021.10.1.5.1 | ||
| UCD-SNMP-MIB:: | UCD-SNMP-MIB:: | ||
| - | >> | + | </code> |
| | | ||
| | | ||
| - | 3 | + | ===== |
| - | *=*=*=*=*=* | + | |
| - | + | ||
| - | | + | |
| | | ||
| - | 3.1 Configuration | + | ==== |
| - | ================== | + | |
| | | ||
| `Target[name]', | `Target[name]', | ||
| Line 109: | Line 101: | ||
| -> http:// | -> http:// | ||
| details. | details. | ||
| - | << | + | <code> |
| $ vi / | $ vi / | ||
| | | ||
| Line 202: | Line 194: | ||
| Options[tcp_established]: | Options[tcp_established]: | ||
| | | ||
| - | >> | + | </code> |
| | | ||
| | | ||
| - | 3.2 Launch | + | ==== Launch |
| - | =========== | + | |
| | | ||
| First launch: | First launch: | ||
| - | << | + | <code> |
| $ / | $ / | ||
| $ ls -ltra / | $ ls -ltra / | ||
| Line 225: | Line 216: | ||
| -rw-r--r-- | -rw-r--r-- | ||
| corbeaueth0.rrd | corbeaueth0.rrd | ||
| - | >> | + | </code> |
| | | ||
| Automatic launch every 5mn with cron: | Automatic launch every 5mn with cron: | ||
| - | << | + | <code> |
| $ cat / | $ cat / | ||
| 0-59/5 * * * * root / | 0-59/5 * * * * root / | ||
| - | >> | + | </code> |
| | | ||
| Check the content of rrd files. | Check the content of rrd files. | ||
| - | << | + | <code> |
| $ rrdtool fetch corbeauload.rrd AVERAGE | $ rrdtool fetch corbeauload.rrd AVERAGE | ||
| .... | .... | ||
| Line 243: | Line 234: | ||
| 1042540500: 1.6000000000e-01 2.0000000000e-01 | 1042540500: 1.6000000000e-01 2.0000000000e-01 | ||
| 1042540800: nan nan | 1042540800: nan nan | ||
| - | >> | + | </code> |
| | | ||
| | | ||
| - | 3.3 index HTML file | + | ==== index HTML file ==== |
| - | ==================== | + | |
| | | ||
| Creation of an index.html file, which will be the gate to the other | Creation of an index.html file, which will be the gate to the other | ||
| (per target) graph files. | (per target) graph files. | ||
| - | << | + | <code> |
| $ indexmaker / | $ indexmaker / | ||
| / | / | ||
| - | >> | + | </code> |
| | | ||
| | | ||
| - | 3.4 | + | ==== cgi-bin file ==== |
| - | ================= | + | |
| Final display is made by default through the cgi-bin ` 14all.cgi ' | Final display is made by default through the cgi-bin ` 14all.cgi ' | ||
| file, if we used the above `indexmaker' | file, if we used the above `indexmaker' | ||
| has to be copied to the right destination, | has to be copied to the right destination, | ||
| right mrtg file: | right mrtg file: | ||
| - | << | + | <code> |
| $ rpm -qli mrtg | grep 14all.cgi | $ rpm -qli mrtg | grep 14all.cgi | ||
| / | / | ||
| Line 273: | Line 262: | ||
| #$cfgfile = '/ | #$cfgfile = '/ | ||
| $cfgfile = '/ | $cfgfile = '/ | ||
| - | >> | + | </code> |
| | | ||
| | | ||
| - | 4 | + | ===== |
| - | *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*= | + | |
| | | ||
| Line 288: | Line 276: | ||
| | | ||
| - | 4.1 back monitor | + | ==== back monitor |
| - | ================= | + | |
| | | ||
| Openldap configuration. | Openldap configuration. | ||
| - | << | + | <code> |
| $ vi / | $ vi / | ||
| | | ||
| Line 300: | Line 287: | ||
| by dn.subtree=dc=int-evry, | by dn.subtree=dc=int-evry, | ||
| by * none | by * none | ||
| - | >> | + | </code> |
| | | ||
| Test with openldap shell tools. | Test with openldap shell tools. | ||
| - | << | + | <code> |
| $ ldapsearch -x -b " | $ ldapsearch -x -b " | ||
| " | " | ||
| Line 309: | Line 296: | ||
| dn: cn=Total, | dn: cn=Total, | ||
| description: | description: | ||
| - | >> | + | </code> |
| | | ||
| | | ||
| - | 4.2 Monitor Script | + | ==== |
| - | =================== | + | |
| | | ||
| - | Vincent Mathieu (Vincent.Mathieu@univ-nancy2.fr) wrote a script that | + | Vincent Mathieu (Vincent.Mathieu |
| gets values from ldap queries on the monitor backend. I modified it to | gets values from ldap queries on the monitor backend. I modified it to | ||
| enable the retrieve of single value data to cope with MRTG/RRD witch | enable the retrieve of single value data to cope with MRTG/RRD witch | ||
| Line 324: | Line 310: | ||
| Example, console display of openldap stats, followed by a single value | Example, console display of openldap stats, followed by a single value | ||
| request example for mrtg (cf script header for details of usage) | request example for mrtg (cf script header for details of usage) | ||
| - | << | + | <code> |
| $ / | $ / | ||
| TotConnexions : 99 | TotConnexions : 99 | ||
| Line 344: | Line 330: | ||
| $ / | $ / | ||
| 100 | 100 | ||
| - | >> | + | </code> |
| | | ||
| | | ||
| - | 4.3 | + | ==== Monitor script for openldap 2.2.X ==== |
| - | ====================================== | + | |
| | | ||
| The monitor database schema has change from 2.1.X to 2.2.X. A very | The monitor database schema has change from 2.1.X to 2.2.X. A very | ||
| Line 359: | Line 344: | ||
| | | ||
| - | 4.3.1 | + | ==== Perl Pakages |
| - | ------------------- | + | |
| This new version of the monitor script includes an enhancement, | This new version of the monitor script includes an enhancement, | ||
| allows SALS authentificated connexions (optionnaly, | allows SALS authentificated connexions (optionnaly, | ||
| start-tls line), so it needs new perl packages | start-tls line), so it needs new perl packages | ||
| - | << | + | <code> |
| [root@corbeau ~] | [root@corbeau ~] | ||
| $ yum install perl-Authen-SASL perl-IO-Socket-SSL | $ yum install perl-Authen-SASL perl-IO-Socket-SSL | ||
| Line 372: | Line 356: | ||
| Dep Installed: | Dep Installed: | ||
| perl-Digest-HMAC 1.01-12.noarch perl-Digest-SHA1 2.07-4.i386 | perl-Digest-HMAC 1.01-12.noarch perl-Digest-SHA1 2.07-4.i386 | ||
| - | >> | + | </code> |
| | | ||
| | | ||
| - | 4.3.2 | + | ==== |
| - | ---------------------------------- | + | |
| | | ||
| Here's an exemple of `ldapsearch' | Here's an exemple of `ldapsearch' | ||
| - | << | + | <code> |
| [root@corbeau / | [root@corbeau / | ||
| $ ./ | $ ./ | ||
| Line 391: | Line 374: | ||
| monitorOpInitiated: | monitorOpInitiated: | ||
| monitorOpCompleted: | monitorOpCompleted: | ||
| - | >> | + | </code> |
| | | ||
| | | ||
| - | 4.3.3 | + | ==== Browse the monitor database with the script |
| - | -------------------------------------------------- | + | |
| | | ||
| - | << | + | <code> |
| $ perl / | $ perl / | ||
| TotConnexions : 12 | TotConnexions : 12 | ||
| Line 416: | Line 398: | ||
| TotSearch : 54 | TotSearch : 54 | ||
| TotUnbind : 10 | TotUnbind : 10 | ||
| - | >> | + | </code> |
| | | ||
| | | ||
| - | 4.3.4 | + | ==== Search with mrtg output |
| - | ------------------------------ | + | |
| | | ||
| - | << | + | <code> |
| $ perl / | $ perl / | ||
| 13 | 13 | ||
| - | >> | + | </code> |
| | | ||
| | | ||
| - | 4.4 Snmpd configuration | + | ===== Snmpd configuration |
| - | ======================== | + | |
| | | ||
| - | << | + | <code> |
| $ grep monitor-jp / | $ grep monitor-jp / | ||
| exec monitor-jp.pl / | exec monitor-jp.pl / | ||
| Line 442: | Line 422: | ||
| Stopping snmpd: | Stopping snmpd: | ||
| Starting snmpd: | Starting snmpd: | ||
| - | >> | + | </code> |
| | | ||
| | | ||
| - | 4.5 MRTG configuration | + | ==== MRTG configuration |
| - | ======================= | + | |
| The first script `snmpd.conf' | The first script `snmpd.conf' | ||
| `.1.3.6.1.4.1.2021.8.1.101.1', | `.1.3.6.1.4.1.2021.8.1.101.1', | ||
| - | << | + | <code> |
| $ snmpget -c secret -v 1 localhost .1.3.6.1.4.1.2021.8.1.101.1 | $ snmpget -c secret -v 1 localhost .1.3.6.1.4.1.2021.8.1.101.1 | ||
| UCD-SNMP-MIB:: | UCD-SNMP-MIB:: | ||
| - | >> | + | </code> |
| | | ||
| Now we can define an ` MRTG target' | Now we can define an ` MRTG target' | ||
| - | << | + | <code> |
| $ grep Slapd / | $ grep Slapd / | ||
| Target[corbeauSlapdCx]: | Target[corbeauSlapdCx]: | ||
| Line 471: | Line 450: | ||
| Legend1[corbeauSlapdCx]: | Legend1[corbeauSlapdCx]: | ||
| Legend2[corbeauSlapdCx]: | Legend2[corbeauSlapdCx]: | ||
| - | >> | + | </code> |
| | | ||
| | | ||
| - | 5 System configuration and packages used | + | ===== snmp tools ===== |
| - | *=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=* | + | |
| - | + | ||
| - | For information, | + | |
| - | + | ||
| - | + | ||
| - | 5.1 Fedora core 2 | + | |
| - | ================== | + | |
| - | + | ||
| - | << | + | |
| - | $ cat / | + | |
| - | Fedora Core release 2 (Tettnang) | + | |
| - | $ uname -a | + | |
| - | Linux corbeau 2.6.7-1.494.2.2 #1 Tue Aug 3 09:39:58 EDT 2004 i686 i686 | + | |
| - | i386 GNU/Linux | + | |
| - | $ rpm -qa | egrep " | + | |
| - | openldap-2.2.17-1 | + | |
| - | net-snmp-5.1.1-2 | + | |
| - | openldap-servers-2.2.17-1 | + | |
| - | mrtg-2.10.5-3 | + | |
| - | net-snmp-utils-5.1.1-2 | + | |
| - | nss_ldap-217-1 | + | |
| - | openldap-clients-2.2.17-1 | + | |
| - | rrdtool-1.0.49-17.rhfc2.at | + | |
| - | >> | + | |
| - | + | ||
| - | + | ||
| - | + | ||
| - | 5.2 Redhat 8 | + | |
| - | ============= | + | |
| - | + | ||
| - | << | + | |
| - | $ cat / | + | |
| - | Red Hat Linux release 8.0 (Psyche) | + | |
| - | $ uname -a | + | |
| - | Linux corbeau.int-evry.fr 2.4.18-14 #1 Wed Sep 4 13:35:50 EDT 2002 \ | + | |
| - | i686 i686 i386 GNU/Linux | + | |
| - | $ rpm -qa | egrep " | + | |
| - | nss_ldap-198-3 | + | |
| - | net-snmp-5.0.6-8.80.2 | + | |
| - | net-snmp-utils-5.0.6-8.80.2 | + | |
| - | openldap-servers-2.1.11-2 | + | |
| - | rrdtool-1.0.39-1.8.0 | + | |
| - | openldap-2.1.11-2 | + | |
| - | openldap-clients-2.1.11-2 | + | |
| - | php-ldap-4.2.2-8.0.5 | + | |
| - | openldap-devel-2.1.11-2 | + | |
| - | mrtg-2.9.17-8 | + | |
| - | >> | + | |
| - | + | ||
| - | + | ||
| - | + | ||
| - | 6 snmp tools | + | |
| - | *=*=*=*=*=*=* | + | |
| | | ||
| check snmp variable queries, check oids etc ... | check snmp variable queries, check oids etc ... | ||
| - | << | + | <code> |
| $ snmpget -v1 -c secret corbeau \ | $ snmpget -v1 -c secret corbeau \ | ||
| .iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets | .iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets | ||
| Line 559: | Line 485: | ||
| | | ||
| cf man snmpcmd for common options to snmp* utils. | cf man snmpcmd for common options to snmp* utils. | ||
| - | >> | + | </code> |
| | | ||
| | | ||
| - | ---## Contrib | + | ===== Contrib |
| contrib sur le forum cacti a ce sujet : | contrib sur le forum cacti a ce sujet : | ||
| - | http:// | + | * http:// |
| On a donc un tracé des mailq complet et efficasse sur: | On a donc un tracé des mailq complet et efficasse sur: | ||
| - | ---## snmp translate tools cisco | + | URL sur le forum cacti pour debuger |
| + | |||
| + | * http:// | ||
| + | * http:// | ||
| + | * http:// | ||
| + | |||
| + | ===== snmp translate tools cisco ===== | ||
| http:// | http:// | ||
| + | |||
| + | ===== rrdtool ====== | ||
| + | |||
| + | il se peut qu'un datasource (fichier rrd) sature suivant la façon dont-il a été crée a verifier avec un rrdtool info nom_fichier_rrd | ||
| + | |||
| + | j' | ||
| + | |||
| + | extrait: | ||
| + | |||
| + | I also had to modify the data-template for deferred and hold to increase the max number from 2000 to 20000 (x10 !) | ||
| + | that increase is available for new data sources, but for previous one , you have to tune the rrd file, exemple: | ||
| + | bash-3.1# rrdtool info smtp1_sodome_incoming_732.rrd | grep deferred | ||
| + | ds[deferred].max = 2.0000000000e+03 | ||
| + | ds[deferred].last_ds = " | ||
| + | |||
| + | < | ||
| + | bash-3.1# rrdtool tune smtp1_sodome_incoming_732.rrd --maximum deferred: | ||
| + | |||
| + | bash-3.1# rrdtool info smtp1_sodome_incoming_732.rrd | grep deferred | ||
| + | ds[deferred].max = 2.0000000000e+04 | ||
| + | ds[deferred].last_ds = " | ||
| + | ds[deferred].value = 1.4746000000e+04 | ||
| + | </ | ||
| + | |||
| + | Now it work for "long queues" | ||
| + | |||
| + | ===== apache tomcat ====== | ||
| + | |||
| + | http:// | ||
| + | |||
| + | JMX | ||
| + | |||
| + | http:// | ||
| + | |||
| + | ===== Mysql ===== | ||
| + | |||
| + | * http:// | ||
| + | * http:// | ||
| + | |||
| + | recuperer l' | ||
| + | |||
| + | < | ||
| + | autoriser l' | ||
| + | ici au final | ||
| + | GRANT PROCESS ON *.* TO cactiuser@' | ||
| + | charger les template xml via l' | ||
| + | copier le script dans les scripts de cacti / | ||
| + | enfin creer les graphs via l' | ||
| + | </ | ||
| + | |||
| + | ==== postfix ==== | ||
| + | |||
| + | ==== percona ==== | ||
| + | |||
| + | |||
| + | * https:// | ||
| + | * https:// | ||
| + | |||
| + | recuperation des sources | ||
| + | |||
| + | < | ||
| + | [root@mon ~]# wget https:// | ||
| + | |||
| + | [root@mon ~]# rpm -Uvh percona-cacti-templates-1.1.8-1.noarch.rpm --nodeps | ||
| + | |||
| + | Scripts are installed to / | ||
| + | Templates are installed to / | ||
| + | </ | ||
| + | |||
| + | |||
| + | avec la fonction d' | ||
| + | |||
| + | < | ||
| + | Cacti has imported the following items for the Template: | ||
| + | |||
| + | CDEF | ||
| + | |||
| + | [succès] Percona Turn Into Bits CDEF [nouveau] | ||
| + | [succès] Percona Negate CDEF [updated] | ||
| + | Table: cdef, Column: name, New Value: ' | ||
| + | [succès] Percona Disk Write Time per IO Request CDEF [nouveau] | ||
| + | [succès] Percona Disk Write Time per IO Request Negate CDEF [nouveau] | ||
| + | [succès] Percona Disk Read Time per IO Request CDEF [nouveau] | ||
| + | GPRINT Preset | ||
| + | |||
| + | [succès] Percona GNU/Linux Server Checksum 611ca2224c24387fc25a62daf58371a2 [nouveau] | ||
| + | [succès] Percona GNU/Linux Server Version t1.1.8: | ||
| + | [succès] Percona Normal [updated] | ||
| + | Table: graph_templates_gprint, | ||
| + | Métode d' | ||
| + | |||
| + | [succès] Percona Get Proc Stats/ | ||
| + | [succès] Percona Get Proc Stats/ | ||
| + | [succès] Percona Get Proc Stats/Forks IM [nouveau] | ||
| + | [succès] Percona Get Proc Stats/CPU Usage IM [nouveau] | ||
| + | [succès] Percona Get W/Load Average IM [nouveau] | ||
| + | [succès] Percona Get W/Number of Users IM [nouveau] | ||
| + | [succès] Percona Get Memory/ | ||
| + | [succès] Percona Get Disk Stats/Disk Operations IM [nouveau] | ||
| + | [succès] Percona Get Disk Stats/Disk Sectors Read/ | ||
| + | [succès] Percona Get Disk Stats/Disk Read/Write Time (ms) IM [nouveau] | ||
| + | [succès] Percona Get Disk Stats/Disk Read/Write Time per IO Request (ms) IM [nouveau] | ||
| + | [succès] Percona Get Disk Stats/Disk Elapsed IO Time (ms) IM [nouveau] | ||
| + | [succès] Percona Get Disk Stats/Disk IOPS IM [nouveau] | ||
| + | [succès] Percona Get Disk Space/Disk Space IM [nouveau] | ||
| + | [succès] Percona Get Network Traffic/ | ||
| + | [succès] Percona Get Network Errors/ | ||
| + | [succès] Percona Get Network Connection States/ | ||
| + | [succès] Percona Get Swap Usage/Swap Usage IM [nouveau] | ||
| + | Data Template | ||
| + | |||
| + | [succès] Percona Interrupts DT [nouveau] | ||
| + | [succès] Percona Context Switches DT [nouveau] | ||
| + | [succès] Percona Forks DT [nouveau] | ||
| + | [succès] Percona CPU Usage DT [nouveau] | ||
| + | [succès] Percona Load Average DT [nouveau] | ||
| + | [succès] Percona Number of Users DT [nouveau] | ||
| + | [succès] Percona Memory DT [nouveau] | ||
| + | [succès] Percona Disk Operations DT [nouveau] | ||
| + | [succès] Percona Disk Sectors Read/ | ||
| + | [succès] Percona Disk Read/Write Time (ms) DT [nouveau] | ||
| + | [succès] Percona Disk Read/Write Time per IO Request (ms) DT [nouveau] | ||
| + | [succès] Percona Disk Elapsed IO Time (ms) DT [nouveau] | ||
| + | [succès] Percona Disk IOPS DT [nouveau] | ||
| + | [succès] Percona Disk Space DT [nouveau] | ||
| + | [succès] Percona Network Traffic DT [nouveau] | ||
| + | [succès] Percona Network Errors DT [nouveau] | ||
| + | [succès] Percona Network Connection States DT [nouveau] | ||
| + | [succès] Percona Swap Usage DT [nouveau] | ||
| + | Modèle de graphique | ||
| + | |||
| + | [succès] Percona Interrupts GT [nouveau] | ||
| + | [succès] Percona Context Switches GT [nouveau] | ||
| + | [succès] Percona Forks GT [nouveau] | ||
| + | [succès] Percona CPU Usage GT [nouveau] | ||
| + | [succès] Percona Load Average GT [nouveau] | ||
| + | [succès] Percona Number of Users GT [nouveau] | ||
| + | [succès] Percona Memory GT [nouveau] | ||
| + | [succès] Percona Disk Operations GT [nouveau] | ||
| + | [succès] Percona Disk Sectors Read/ | ||
| + | [succès] Percona Disk Read/Write Time (ms) GT [nouveau] | ||
| + | [succès] Percona Disk Read/Write Time per IO Request (ms) GT [nouveau] | ||
| + | [succès] Percona Disk Elapsed IO Time (ms) GT [nouveau] | ||
| + | [succès] Percona Disk IOPS GT [nouveau] | ||
| + | [succès] Percona Disk Space GT [nouveau] | ||
| + | [succès] Percona Network Traffic GT [nouveau] | ||
| + | [succès] Percona Network Errors GT [nouveau] | ||
| + | [succès] Percona Network Connection States GT [nouveau] | ||
| + | [succès] Percona Swap Usage GT [nouveau] | ||
| + | Device Template | ||
| + | |||
| + | [succès] Percona GNU/Linux Server HT [nouveau] | ||
| + | </ | ||
| + | |||
| + | |||
| + | sur un client a monitorer | ||
| + | |||
| + | [root@jap webapps]# useradd -u 1111 cacti | ||
| + | [root@jap webapps]# passwd cacti | ||
| + | Changement de mot de passe pour l' | ||
| + | Nouveau mot de passe : secret pass | ||
| + | </ | ||
| + | |||
| + | on partage la clé ssh du serveur " | ||
| + | |||
| + | < | ||
| + | [root@mon ~]# ssh-copy-id -i / | ||
| + | </ | ||
| + | |||
| + | ouvrir le firewall sur le client | ||
| + | |||
| + | < | ||
| + | # firewall-cmd --permanent --add-rich-rule 'rule family=" | ||
| + | # firewall-cmd --reload | ||
| + | |||
| + | </ | ||
| + | |||
| + | et enable snmp au boot | ||
| + | |||
| + | < | ||
| + | # systemctl enable snmpd.service | ||
| + | Created symlink from / | ||
| + | </ | ||
| + | |||
| + | |||
| + | ==== postfix ==== | ||
| + | |||
| + | scripts locaux | ||
| + | |||
| + | < | ||
| + | [root@sym1 snmp]# scp -r local-snmp-cronjob postfix-script-stats uloganalyser uloganalyser-plugin root@sym2.int.fr:/ | ||
| + | [root@sym1 cron.d]# scp local-snmp root@sym2.int.fr:/ | ||
| + | </ | ||
| + | |||
| + | repertoire dediés aux logs mail pour les scripts | ||
| + | |||
| + | < | ||
| + | [root@sym2]# | ||
| + | |||
| + | |||
| + | </ | ||
| + | ===== php weathermap ===== | ||
| + | |||
| + | parallel link | ||
| + | |||
| + | http:// | ||
| + | |||