[[Perfs Réseaux]]
Trace:
Show pageOld revisions
AdminRecent ChangesSitemap

* ancien site

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision 		Previous revision
docpublic:reseaux:infra:perf [2011/03/03 09:09]
PROCACCIA created 		docpublic:reseaux:infra:perf [2011/06/18 14:45] (current)
PROCACCIA
Line 1: Line 1:
-===== Perf =====+===== Perfs Réseaux =====
  
-===== références =====+===== Références =====
  
-http://www.sans.org/reading_room/whitepapers/detection/capturing-10g-1g-traffic-correct-settings_33043 +  * http://www.sans.org/reading_room/whitepapers/detection/capturing-10g-1g-traffic-correct-settings_33043 
-http://www.redhat.com/promo/summit/2008/downloads/pdf/Thursday/Mark_Wagner.pdf +  * http://www.redhat.com/promo/summit/2008/downloads/pdf/Thursday/Mark_Wagner.pdf  
 +  * http://acs.lbl.gov/~jason/tcpdump_advanced_filters.txt 
 +  * http://www.wains.be/pub/networking/tcpdump_advanced_filters.txt
  
 ===== Plateforme Evry ===== ===== Plateforme Evry =====
  
-==== Test Localhost ====+==== Serveur 1 ====
  
-Pour tester les performance locale de la machine+<code> 
 +olympie 
 +PowerEdge R610 
 +memory size: 16GiB DIMM Synchronous 1333 MHz (0.8 ns) 
 +16 cores Intel(R) Xeon(R) CPU           E5540  @ 2.53GHz 
 +[root@olympie ~]# ethtool -i eth6 
 +driver: bnx2x 
 +version: 1.52.1-6 
 +firmware-version: BC:5.0.13 PHY:0aa0:0406 
 +bus-info: 0000:05:00.1 
 +</code> 
 + 
 +==== Serveur 2 ==== 
 + 
 +<code> 
 +gates 
 +product: PowerEdge 2950 
 +8 cores Intel(R) Xeon(R) CPU           X5460  @ 3.16GHz 
 +memory size: 16GiB Synchronous 667 MHz (1.5 ns) 
 +[root@gates ~]# ethtool -i eth4 
 +driver: bnx2x 
 +version: 1.52.1-6 
 +firmware-version: BC:5.0.13 PHY:0aa0:0406 
 +bus-info: 0000:0a:00.0 
 +</code> 
 + 
 +==== Switch 10G ==== 
 + 
 +C6509 avec carte 8 ports 10G 
 + 
 +<code> 
 +Supervisor Engine 720 (Active)         WS-SUP720-BASE 
 +CEF720 8 port 10GE with DFC            WS-X6708-10GE 
 + 
 +System image file is "disk1:s72033-adventerprisek9_wan-mz.122-33.SXH8.bin" 
 +Version 12.2(33)SXH8, RELEASE SOFTWARE (fc1) 
 +</code> 
 + 
 +=====Test Localhost ===== 
 + 
 +Pour tester les performances locales de la machine
  
 <code> <code>
Line 35: Line 77:
 > [ ID] Interval       Transfer     Bandwidth > [ ID] Interval       Transfer     Bandwidth
 > [  3]  0.0-10.0 sec  14.0 GBytes  12.0 Gbits/sec > [  3]  0.0-10.0 sec  14.0 GBytes  12.0 Gbits/sec
->+ 
 +</code>
  
 ==== Caracteristiques carte réseau ==== ==== Caracteristiques carte réseau ====
Line 59: Line 102:
 >     Link detected: yes >     Link detected: yes
 </code> </code>
 +
 +==== Serveur n°2 ====
 +
 +<code>
 +[root@gates ~]# ethtool eth4
 +Settings for eth4:
 +    Supported ports: [ FIBRE ]
 +    Supported link modes:   1000baseT/Full
 +                               10000baseT/Full
 +    Supports auto-negotiation: Yes
 +    Advertised link modes:  10000baseT/Full
 +    Advertised auto-negotiation: No
 +    Speed: 10000Mb/s
 +    Duplex: Full
 +    Port: FIBRE
 +    PHYAD: 17
 +    Transceiver: internal
 +    Auto-negotiation: off
 +    Supports Wake-on: d
 +    Wake-on: d
 +    Current message level: 0x00000000 (0)
 +    Link detected: yes 
 +</code>
 +
 +=====  Test Back2Back =====
 +
 +Test réseau  "back2back" (fibre directe entre les 2 serveurs)  sans optimisation
 +
 +<code>
 +[root@olympie ~]# iperf -c 157.159.33.76 -i 1
 +------------------------------------------------------------
 +Client connecting to 157.159.33.76, TCP port 5001
 +TCP window size: 16.0 KByte (default)
 +------------------------------------------------------------
 +[  3] local 157.159.33.77 port 33022 connected with 157.159.33.76 port 5001
 +[ ID] Interval       Transfer     Bandwidth
 +[  3]  0.0- 1.0 sec   289 MBytes  2.42 Gbits/sec
 +[  3]  1.0- 2.0 sec   284 MBytes  2.38 Gbits/sec
 +[  3]  2.0- 3.0 sec   287 MBytes  2.41 Gbits/sec
 +[  3]  3.0- 4.0 sec   277 MBytes  2.33 Gbits/sec
 +[  3]  4.0- 5.0 sec   286 MBytes  2.40 Gbits/sec
 +[  3]  5.0- 6.0 sec   290 MBytes  2.43 Gbits/sec
 +[  3]  6.0- 7.0 sec   295 MBytes  2.47 Gbits/sec
 +[  3]  7.0- 8.0 sec   298 MBytes  2.50 Gbits/sec
 +[  3]  8.0- 9.0 sec   334 MBytes  2.80 Gbits/sec
 +[  3]  9.0-10.0 sec   341 MBytes  2.86 Gbits/sec
 +[  3]  0.0-10.0 sec  2.91 GBytes  2.50 Gbits/sec
 +
 +2eme
 +[  3]  0.0-10.0 sec  3.53 GBytes  3.03 Gbits/sec
 +
 +
 +[root@olympie ~]# date
 +ven fév 18 10:44:35 CET 2011
 +
 +[root@gates ~]# iperf -s -B 157.159.33.76 -i 1
 +------------------------------------------------------------
 +Server listening on TCP port 5001
 +Binding to local address 157.159.33.76
 +TCP window size: 85.3 KByte (default)
 +------------------------------------------------------------
 +[  4] local 157.159.33.76 port 5001 connected with 157.159.33.77 port 33022
 +[ ID] Interval       Transfer     Bandwidth
 +[  4]  0.0- 1.0 sec   284 MBytes  2.38 Gbits/sec
 +[  4]  1.0- 2.0 sec   285 MBytes  2.39 Gbits/sec
 +[  4]  2.0- 3.0 sec   287 MBytes  2.41 Gbits/sec
 +[  4]  3.0- 4.0 sec   277 MBytes  2.33 Gbits/sec
 +[  4]  4.0- 5.0 sec   287 MBytes  2.41 Gbits/sec
 +[  4]  5.0- 6.0 sec   288 MBytes  2.42 Gbits/sec
 +[  4]  6.0- 7.0 sec   295 MBytes  2.48 Gbits/sec
 +[  4]  7.0- 8.0 sec   299 MBytes  2.51 Gbits/sec
 +[  4]  8.0- 9.0 sec   335 MBytes  2.81 Gbits/sec
 +[  4]  9.0-10.0 sec   340 MBytes  2.86 Gbits/sec
 +[  4]  0.0-10.0 sec  2.91 GBytes  2.50 Gbits/sec 
 +</code>
 +
 +autre jours .. deux fois plus rapide !?
 +
 +<code>
 +[root@olympie ~]# date
 +mar fév 22 16:50:01 CET 2011
 +[root@olympie ~]# iperf -c 157.159.33.76 -i 1
 +------------------------------------------------------------
 +Client connecting to 157.159.33.76, TCP port 5001
 +TCP window size: 16.0 KByte (default)
 +------------------------------------------------------------
 +[  3] local 157.159.33.77 port 38359 connected with 157.159.33.76 port 5001
 +[ ID] Interval       Transfer     Bandwidth
 +[  3]  0.0- 1.0 sec   664 MBytes  5.57 Gbits/sec
 +[  3]  1.0- 2.0 sec   672 MBytes  5.64 Gbits/sec
 +[  3]  2.0- 3.0 sec   682 MBytes  5.72 Gbits/sec
 +[  3]  3.0- 4.0 sec   674 MBytes  5.66 Gbits/sec
 +[  3]  4.0- 5.0 sec   674 MBytes  5.65 Gbits/sec
 +[  3]  5.0- 6.0 sec   682 MBytes  5.72 Gbits/sec
 +[  3]  6.0- 7.0 sec   676 MBytes  5.67 Gbits/sec
 +[  3]  7.0- 8.0 sec   676 MBytes  5.67 Gbits/sec
 +[  3]  8.0- 9.0 sec   680 MBytes  5.70 Gbits/sec
 +[  3]  0.0-10.0 sec  6.60 GBytes  5.67 Gbits/sec
 +
 +[root@gates ~]# iperf -s -B 157.159.33.76 -i 1
 +
 +[  4]  0.0-10.0 sec  6.60 GBytes  5.66 Gbits/sec 
 +</code>
 +
 +
 +==== MTU 9000 no iptables ====
 +
 +En reseau avec une MTU de 9000 et arret du firewall iptables sur les 2 serveurs
 +
 +<code>
 +[root@gates ~]# ifconfig eth4
 +eth4      Link encap:Ethernet  HWaddr 00:10:18:7E:FE:DC  
 +          inet adr:157.159.33.76  Bcast:157.159.33.255  Masque:255.255.255.0
 +          adr inet6: fe80::210:18ff:fe7e:fedc/64 Scope:Lien
 +          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
 +          RX packets:10293908329 errors:1048345038 dropped:0 overruns:1048287173 frame:57865
 +          TX packets:1347839 errors:0 dropped:0 overruns:0 carrier:0
 +          collisions:0 lg file transmission:1000 
 +          RX bytes:52506411514898 (47.7 TiB)  TX bytes:98160218 (93.6 MiB)
 +          Interruption:169 Mémoire:d3000000-d37fffff 
 +[root@olympie ~]# ifconfig eth6
 +eth6      Link encap:Ethernet  HWaddr 00:10:18:7E:FE:C6  
 +          inet adr:157.159.33.77  Bcast:157.159.33.255  Masque:255.255.255.0
 +          adr inet6: fe80::210:18ff:fe7e:fec6/64 Scope:Lien
 +          UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1
 +          RX packets:1347669 errors:170 dropped:0 overruns:170 frame:0
 +          TX packets:11342253367 errors:0 dropped:0 overruns:0 carrier:0
 +          collisions:0 lg file transmission:1000 
 +          RX bytes:98194478 (93.6 MiB)  TX bytes:52506885555906 (47.7 TiB)
 +          Interruption:98 Mémoire:dd000000-dd7fffff 
 +</code>
 +
 +Test TCP
 +
 +<code>
 +[root@gates ~]# date
 +jeu mar  3 10:25:04 CET 2011
 +[root@gates ~]# iperf -s -i 1 
 +------------------------------------------------------------
 +Server listening on TCP port 5001
 +TCP window size: 85.3 KByte (default)
 +------------------------------------------------------------
 +[  4] local 157.159.33.76 port 5001 connected with 157.159.33.77 port 48163
 +[ ID] Interval       Transfer     Bandwidth
 +[  4]  0.0- 1.0 sec  1018 MBytes  8.54 Gbits/sec
 +[  4]  1.0- 2.0 sec  1.15 GBytes  9.86 Gbits/sec
 +[  4]  2.0- 3.0 sec  1.15 GBytes  9.90 Gbits/sec
 +[  4]  3.0- 4.0 sec  1.15 GBytes  9.87 Gbits/sec
 +[  4]  4.0- 5.0 sec  1.15 GBytes  9.88 Gbits/sec
 +[  4]  5.0- 6.0 sec   936 MBytes  7.85 Gbits/sec
 +[  4]  6.0- 7.0 sec  1.06 GBytes  9.14 Gbits/sec
 +[  4]  7.0- 8.0 sec  1.15 GBytes  9.89 Gbits/sec
 +[  4]  8.0- 9.0 sec  1.15 GBytes  9.88 Gbits/sec
 +[  4]  9.0-10.0 sec  1.15 GBytes  9.84 Gbits/sec
 +[  4]  0.0-10.0 sec  11.0 GBytes  9.46 Gbits/sec
 +
 +[root@olympie ~]# iperf -c 157.159.33.76  -i 1
 +------------------------------------------------------------
 +Client connecting to 157.159.33.76, TCP port 5001
 +TCP window size: 27.8 KByte (default)
 +------------------------------------------------------------
 +[  3] local 157.159.33.77 port 48163 connected with 157.159.33.76 port 5001
 +[ ID] Interval       Transfer     Bandwidth
 +[  3]  0.0- 1.0 sec  1021 MBytes  8.57 Gbits/sec
 +[  3]  1.0- 2.0 sec  1.15 GBytes  9.89 Gbits/sec
 +[  3]  2.0- 3.0 sec  1.15 GBytes  9.88 Gbits/sec
 +[  3]  3.0- 4.0 sec  1.15 GBytes  9.87 Gbits/sec
 +[  3]  4.0- 5.0 sec  1.15 GBytes  9.89 Gbits/sec
 +[  3]  5.0- 6.0 sec   939 MBytes  7.87 Gbits/sec
 +[  3]  6.0- 7.0 sec  1.06 GBytes  9.11 Gbits/sec
 +[  3]  7.0- 8.0 sec  1.15 GBytes  9.88 Gbits/sec
 +[  3]  8.0- 9.0 sec  1.15 GBytes  9.89 Gbits/sec
 +[  3]  9.0-10.0 sec  1.15 GBytes  9.85 Gbits/sec
 +[  3]  0.0-10.0 sec  11.0 GBytes  9.47 Gbits/sec
 +</code>
 +
 +==== Tests UDP ====
 +
 +Avec les parametres recommandés par l'IN2P3
 +
 +<code>
 +[root@gates ~]# iperf -s -u -w 32M -i 1 -B 157.159.33.76
 +------------------------------------------------------------
 +Server listening on UDP port 5001
 +Binding to local address 157.159.33.76
 +Receiving 1470 byte datagrams
 +UDP buffer size:  256 KByte (WARNING: requested 32.0 MByte)
 +------------------------------------------------------------
 +[  3] local 157.159.33.76 port 5001 connected with 157.159.33.77 port 5001
 +[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
 +[  3]  0.0- 1.0 sec  79.2 MBytes   664 Mbits/sec   0.004 ms 7023/63520 (11%)
 +[  3]  1.0- 2.0 sec  78.9 MBytes   662 Mbits/sec   0.006 ms 7249/63518 (11%)
 +[  3]  2.0- 3.0 sec  79.6 MBytes   667 Mbits/sec   0.004 ms 6763/63518 (11%)
 +[  3]  3.0- 4.0 sec  79.9 MBytes   670 Mbits/sec   0.003 ms 6514/63518 (10%)
 +[  3]  4.0- 5.0 sec  79.9 MBytes   670 Mbits/sec   0.004 ms 6553/63518 (10%)
 +[  3]  5.0- 6.0 sec  79.6 MBytes   668 Mbits/sec   0.004 ms 6705/63518 (11%)
 +[  3]  6.0- 7.0 sec  79.6 MBytes   668 Mbits/sec   0.006 ms 6629/63415 (10%)
 +[  3]  7.0- 8.0 sec  79.6 MBytes   667 Mbits/sec   0.013 ms 6575/63334 (10%)
 +[  3]  8.0- 9.0 sec  79.7 MBytes   669 Mbits/sec   0.014 ms 6595/63471 (10%)
 +[  3]  0.0-10.0 sec   796 MBytes   668 Mbits/sec   0.008 ms 66674/634704 (11%)
 +[  3]  0.0-10.0 sec  1 datagrams received out-of-order
 +
 +[root@olympie ~]# iperf -u -c 157.159.33.76 -w32M -t 10 -l9170 -b10G -B 157.159.33.77 -P1
 +------------------------------------------------------------
 +Client connecting to 157.159.33.76, UDP port 5001
 +Binding to local address 157.159.33.77
 +Sending 9170 byte datagrams
 +UDP buffer size:  256 KByte (WARNING: requested 32.0 MByte)
 +------------------------------------------------------------
 +[  3] local 157.159.33.77 port 5001 connected with 157.159.33.76 port 5001
 +[ ID] Interval       Transfer     Bandwidth
 +[  3]  0.0-10.0 sec  5.42 GBytes  4.66 Gbits/sec
 +[  3] Sent 634705 datagrams
 +[  3] Server Report:
 +[  3]  0.0-10.0 sec   796 MBytes   668 Mbits/sec   0.007 ms 66674/634704 (11%)
 +[  3]  0.0-10.0 sec  1 datagrams received out-of-order
 +
 +</code>
 +
 +==== Optimisations ====
 +
 +
 +<code>
 +Le 23/02/2011 09:43, Guillaume a écrit :
 +> Hello Jehan,
 +>
 +> - Attention aux buffers
 +> > TCP window size: 27.8 KByte (default)
 +> > UDP buffer size: 256 KByte (WARNING: requested 32.0 MByte)
 +>
 +> Les paramètres donnés à iperf sont non prioritaires par rapport aux paramètres sysctl, donc même si tu demande w32M ça n'est pas forcément utilisé. Je ne me souvient plus exactement quels paramètres kernel sont en jeu, mais peut être:
 +> net.core.rmem_max = 16777216
 +> net.core.wmem_max = 16777216
 +> net.ipv4.tcp_rmem = 4096 87380 16777216
 +> net.ipv4.tcp_wmem = 4096 65536 16777216
 +> (les paramètres tcp sont aussi pour UDP)
 +>
 +> - Il me semble avoir lu une histoire de problème de timestamp sur certains tests iperf en UDP: Chaque paquet part avec un timestamp et donc si tu met un seul thread ça fait un goulot d'appel système. Peut être tester avec 10 thread à 1G (options -P10 -b1G).
 +>
 +> - Les valeurs retournées par les tests iperf sont souvent fantaisistes, surtout quand ça marche moyen. Plutôt utiliser le monitoring des interfaces de l'équipement réseau en face.
 +>
 +> Guillaume 
 +</code>
 +
 +
 +Augmentation du nombre de thread a 8 pour parralleliser les transferts
 +
 +<code>
 +[root@olympie ~]# iperf -u -c 157.159.33.76 -w32M -t 10 -l9170 -b10G -B 157.159.33.77 -P8
 +bind failed: Address already in use
 +bind failed: Address already in use
 +bind failed: Address already in use
 +bind failed: Address already in use
 +bind failed: Address already in use
 +bind failed: Address already in use
 +bind failed: Address already in use
 +------------------------------------------------------------
 +Client connecting to 157.159.33.76, UDP port 5001
 +Binding to local address 157.159.33.77
 +Sending 9170 byte datagrams
 +UDP buffer size:  256 KByte (WARNING: requested 32.0 MByte)
 +------------------------------------------------------------
 +[ 10] local 157.159.33.77 port 33392 connected with 157.159.33.76 port 5001
 +[  3] local 157.159.33.77 port 5001 connected with 157.159.33.76 port 5001
 +[  5] local 157.159.33.77 port 39469 connected with 157.159.33.76 port 5001
 +[  4] local 157.159.33.77 port 36797 connected with 157.159.33.76 port 5001
 +[  7] local 157.159.33.77 port 58107 connected with 157.159.33.76 port 5001
 +[  6] local 157.159.33.77 port 48790 connected with 157.159.33.76 port 5001
 +[  8] local 157.159.33.77 port 38680 connected with 157.159.33.76 port 5001
 +[  9] local 157.159.33.77 port 57872 connected with 157.159.33.76 port 5001
 +[ ID] Interval       Transfer     Bandwidth
 +[ 10]  0.0-10.0 sec  1.44 GBytes  1.23 Gbits/sec
 +[ 10] Sent 168182 datagrams
 +[  3]  0.0-10.0 sec  1.44 GBytes  1.24 Gbits/sec
 +[  3] Sent 168473 datagrams
 +[  5]  0.0-10.0 sec  1.44 GBytes  1.23 Gbits/sec
 +[  5] Sent 168061 datagrams
 +[  4]  0.0-10.0 sec  1.44 GBytes  1.23 Gbits/sec
 +[  4] Sent 168243 datagrams
 +[  7]  0.0-10.0 sec  1.43 GBytes  1.23 Gbits/sec
 +[  7] Sent 167800 datagrams
 +[  6]  0.0-10.0 sec  1.44 GBytes  1.23 Gbits/sec
 +[  6] Sent 168212 datagrams
 +[  8]  0.0-10.0 sec  1.43 GBytes  1.23 Gbits/sec
 +[  8] Sent 167895 datagrams
 +[  9]  0.0-10.0 sec  1.43 GBytes  1.23 Gbits/sec
 +[  9] Sent 167909 datagrams
 +[SUM]  0.0-10.0 sec  11.5 GBytes  9.86 Gbits/sec
 +</code>
 +
 +
 +{{:docpublic:reseaux:infra:2010-03-03-iperf-olympie-gates-tcp1-udp2.png|2010-03-03-iperf-olympie-gates-tcp1-udp2.png }}
 +
docpublic/reseaux/infra/perf.1299143375.txt.gz · Last modified: 2011/03/03 09:09 by PROCACCIA
Show pageOld revisions
[unknown link type]Back to top
CC Attribution-Noncommercial-Share Alike 4.0 International
www.chimeric.de Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0