Perfs Réseaux
Références
Plateforme Evry
Serveur 1
olympie PowerEdge R610 memory size: 16GiB DIMM Synchronous 1333 MHz (0.8 ns) 16 cores Intel(R) Xeon(R) CPU E5540 @ 2.53GHz [root@olympie ~]# ethtool -i eth6 driver: bnx2x version: 1.52.1-6 firmware-version: BC:5.0.13 PHY:0aa0:0406 bus-info: 0000:05:00.1
Serveur 2
gates product: PowerEdge 2950 8 cores Intel(R) Xeon(R) CPU X5460 @ 3.16GHz memory size: 16GiB Synchronous 667 MHz (1.5 ns) [root@gates ~]# ethtool -i eth4 driver: bnx2x version: 1.52.1-6 firmware-version: BC:5.0.13 PHY:0aa0:0406 bus-info: 0000:0a:00.0
Switch 10G
C6509 avec carte 8 ports 10G
Supervisor Engine 720 (Active) WS-SUP720-BASE CEF720 8 port 10GE with DFC WS-X6708-10GE System image file is "disk1:s72033-adventerprisek9_wan-mz.122-33.SXH8.bin" Version 12.2(33)SXH8, RELEASE SOFTWARE (fc1)
Test Localhost
Pour tester les performances locales de la machine
Le 18/02/2011 10:46, jehan procaccia a écrit : > olympie localhost , test de perf de la machine > > [root@olympie ~]# iperf -s -B 127.0.0.1 -w 30MB > ------------------------------------------------------------ > Server listening on TCP port 5001 > Binding to local address 127.0.0.1 > TCP window size: 256 KByte (WARNING: requested 30.0 MByte) > ------------------------------------------------------------ > [ 4] local 127.0.0.1 port 5001 connected with 127.0.0.1 port 34256 > [ ID] Interval Transfer Bandwidth > [ 4] 0.0-10.0 sec 14.0 GBytes 12.0 Gbits/sec > > > [root@olympie ~]# iperf -c 127.0.0.1 -w 30MB > ------------------------------------------------------------ > Client connecting to 127.0.0.1, TCP port 5001 > TCP window size: 256 KByte (WARNING: requested 30.0 MByte) > ------------------------------------------------------------ > [ 3] local 127.0.0.1 port 34256 connected with 127.0.0.1 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-10.0 sec 14.0 GBytes 12.0 Gbits/sec
Caracteristiques carte réseau
> [root@olympie ~]# ethtool eth6 > Settings for eth6: > Supported ports: [ FIBRE ] > Supported link modes: 1000baseT/Full > 10000baseT/Full > Supports auto-negotiation: Yes > Advertised link modes: 10000baseT/Full > Advertised auto-negotiation: No > Speed: 10000Mb/s > Duplex: Full > Port: FIBRE > PHYAD: 16 > Transceiver: internal > Auto-negotiation: off > Supports Wake-on: d > Wake-on: d > Current message level: 0x00000000 (0) > Link detected: yes
Serveur n°2
[root@gates ~]# ethtool eth4
Settings for eth4:
Supported ports: [ FIBRE ]
Supported link modes: 1000baseT/Full
10000baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10000baseT/Full
Advertised auto-negotiation: No
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 17
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000000 (0)
Link detected: yes
Test Back2Back
Test réseau “back2back” (fibre directe entre les 2 serveurs) sans optimisation
[root@olympie ~]# iperf -c 157.159.33.76 -i 1 ------------------------------------------------------------ Client connecting to 157.159.33.76, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 157.159.33.77 port 33022 connected with 157.159.33.76 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 289 MBytes 2.42 Gbits/sec [ 3] 1.0- 2.0 sec 284 MBytes 2.38 Gbits/sec [ 3] 2.0- 3.0 sec 287 MBytes 2.41 Gbits/sec [ 3] 3.0- 4.0 sec 277 MBytes 2.33 Gbits/sec [ 3] 4.0- 5.0 sec 286 MBytes 2.40 Gbits/sec [ 3] 5.0- 6.0 sec 290 MBytes 2.43 Gbits/sec [ 3] 6.0- 7.0 sec 295 MBytes 2.47 Gbits/sec [ 3] 7.0- 8.0 sec 298 MBytes 2.50 Gbits/sec [ 3] 8.0- 9.0 sec 334 MBytes 2.80 Gbits/sec [ 3] 9.0-10.0 sec 341 MBytes 2.86 Gbits/sec [ 3] 0.0-10.0 sec 2.91 GBytes 2.50 Gbits/sec 2eme [ 3] 0.0-10.0 sec 3.53 GBytes 3.03 Gbits/sec [root@olympie ~]# date ven fév 18 10:44:35 CET 2011 [root@gates ~]# iperf -s -B 157.159.33.76 -i 1 ------------------------------------------------------------ Server listening on TCP port 5001 Binding to local address 157.159.33.76 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ [ 4] local 157.159.33.76 port 5001 connected with 157.159.33.77 port 33022 [ ID] Interval Transfer Bandwidth [ 4] 0.0- 1.0 sec 284 MBytes 2.38 Gbits/sec [ 4] 1.0- 2.0 sec 285 MBytes 2.39 Gbits/sec [ 4] 2.0- 3.0 sec 287 MBytes 2.41 Gbits/sec [ 4] 3.0- 4.0 sec 277 MBytes 2.33 Gbits/sec [ 4] 4.0- 5.0 sec 287 MBytes 2.41 Gbits/sec [ 4] 5.0- 6.0 sec 288 MBytes 2.42 Gbits/sec [ 4] 6.0- 7.0 sec 295 MBytes 2.48 Gbits/sec [ 4] 7.0- 8.0 sec 299 MBytes 2.51 Gbits/sec [ 4] 8.0- 9.0 sec 335 MBytes 2.81 Gbits/sec [ 4] 9.0-10.0 sec 340 MBytes 2.86 Gbits/sec [ 4] 0.0-10.0 sec 2.91 GBytes 2.50 Gbits/sec
autre jours .. deux fois plus rapide !?
[root@olympie ~]# date mar fév 22 16:50:01 CET 2011 [root@olympie ~]# iperf -c 157.159.33.76 -i 1 ------------------------------------------------------------ Client connecting to 157.159.33.76, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 157.159.33.77 port 38359 connected with 157.159.33.76 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 664 MBytes 5.57 Gbits/sec [ 3] 1.0- 2.0 sec 672 MBytes 5.64 Gbits/sec [ 3] 2.0- 3.0 sec 682 MBytes 5.72 Gbits/sec [ 3] 3.0- 4.0 sec 674 MBytes 5.66 Gbits/sec [ 3] 4.0- 5.0 sec 674 MBytes 5.65 Gbits/sec [ 3] 5.0- 6.0 sec 682 MBytes 5.72 Gbits/sec [ 3] 6.0- 7.0 sec 676 MBytes 5.67 Gbits/sec [ 3] 7.0- 8.0 sec 676 MBytes 5.67 Gbits/sec [ 3] 8.0- 9.0 sec 680 MBytes 5.70 Gbits/sec [ 3] 0.0-10.0 sec 6.60 GBytes 5.67 Gbits/sec [root@gates ~]# iperf -s -B 157.159.33.76 -i 1 [ 4] 0.0-10.0 sec 6.60 GBytes 5.66 Gbits/sec
MTU 9000 no iptables
En reseau avec une MTU de 9000 et arret du firewall iptables sur les 2 serveurs
[root@gates ~]# ifconfig eth4
eth4 Link encap:Ethernet HWaddr 00:10:18:7E:FE:DC
inet adr:157.159.33.76 Bcast:157.159.33.255 Masque:255.255.255.0
adr inet6: fe80::210:18ff:fe7e:fedc/64 Scope:Lien
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:10293908329 errors:1048345038 dropped:0 overruns:1048287173 frame:57865
TX packets:1347839 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 lg file transmission:1000
RX bytes:52506411514898 (47.7 TiB) TX bytes:98160218 (93.6 MiB)
Interruption:169 Mémoire:d3000000-d37fffff
[root@olympie ~]# ifconfig eth6
eth6 Link encap:Ethernet HWaddr 00:10:18:7E:FE:C6
inet adr:157.159.33.77 Bcast:157.159.33.255 Masque:255.255.255.0
adr inet6: fe80::210:18ff:fe7e:fec6/64 Scope:Lien
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:1347669 errors:170 dropped:0 overruns:170 frame:0
TX packets:11342253367 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 lg file transmission:1000
RX bytes:98194478 (93.6 MiB) TX bytes:52506885555906 (47.7 TiB)
Interruption:98 Mémoire:dd000000-dd7fffff
Test TCP
[root@gates ~]# date jeu mar 3 10:25:04 CET 2011 [root@gates ~]# iperf -s -i 1 ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ [ 4] local 157.159.33.76 port 5001 connected with 157.159.33.77 port 48163 [ ID] Interval Transfer Bandwidth [ 4] 0.0- 1.0 sec 1018 MBytes 8.54 Gbits/sec [ 4] 1.0- 2.0 sec 1.15 GBytes 9.86 Gbits/sec [ 4] 2.0- 3.0 sec 1.15 GBytes 9.90 Gbits/sec [ 4] 3.0- 4.0 sec 1.15 GBytes 9.87 Gbits/sec [ 4] 4.0- 5.0 sec 1.15 GBytes 9.88 Gbits/sec [ 4] 5.0- 6.0 sec 936 MBytes 7.85 Gbits/sec [ 4] 6.0- 7.0 sec 1.06 GBytes 9.14 Gbits/sec [ 4] 7.0- 8.0 sec 1.15 GBytes 9.89 Gbits/sec [ 4] 8.0- 9.0 sec 1.15 GBytes 9.88 Gbits/sec [ 4] 9.0-10.0 sec 1.15 GBytes 9.84 Gbits/sec [ 4] 0.0-10.0 sec 11.0 GBytes 9.46 Gbits/sec [root@olympie ~]# iperf -c 157.159.33.76 -i 1 ------------------------------------------------------------ Client connecting to 157.159.33.76, TCP port 5001 TCP window size: 27.8 KByte (default) ------------------------------------------------------------ [ 3] local 157.159.33.77 port 48163 connected with 157.159.33.76 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 1021 MBytes 8.57 Gbits/sec [ 3] 1.0- 2.0 sec 1.15 GBytes 9.89 Gbits/sec [ 3] 2.0- 3.0 sec 1.15 GBytes 9.88 Gbits/sec [ 3] 3.0- 4.0 sec 1.15 GBytes 9.87 Gbits/sec [ 3] 4.0- 5.0 sec 1.15 GBytes 9.89 Gbits/sec [ 3] 5.0- 6.0 sec 939 MBytes 7.87 Gbits/sec [ 3] 6.0- 7.0 sec 1.06 GBytes 9.11 Gbits/sec [ 3] 7.0- 8.0 sec 1.15 GBytes 9.88 Gbits/sec [ 3] 8.0- 9.0 sec 1.15 GBytes 9.89 Gbits/sec [ 3] 9.0-10.0 sec 1.15 GBytes 9.85 Gbits/sec [ 3] 0.0-10.0 sec 11.0 GBytes 9.47 Gbits/sec
Tests UDP
Avec les parametres recommandés par l'IN2P3
[root@gates ~]# iperf -s -u -w 32M -i 1 -B 157.159.33.76 ------------------------------------------------------------ Server listening on UDP port 5001 Binding to local address 157.159.33.76 Receiving 1470 byte datagrams UDP buffer size: 256 KByte (WARNING: requested 32.0 MByte) ------------------------------------------------------------ [ 3] local 157.159.33.76 port 5001 connected with 157.159.33.77 port 5001 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0- 1.0 sec 79.2 MBytes 664 Mbits/sec 0.004 ms 7023/63520 (11%) [ 3] 1.0- 2.0 sec 78.9 MBytes 662 Mbits/sec 0.006 ms 7249/63518 (11%) [ 3] 2.0- 3.0 sec 79.6 MBytes 667 Mbits/sec 0.004 ms 6763/63518 (11%) [ 3] 3.0- 4.0 sec 79.9 MBytes 670 Mbits/sec 0.003 ms 6514/63518 (10%) [ 3] 4.0- 5.0 sec 79.9 MBytes 670 Mbits/sec 0.004 ms 6553/63518 (10%) [ 3] 5.0- 6.0 sec 79.6 MBytes 668 Mbits/sec 0.004 ms 6705/63518 (11%) [ 3] 6.0- 7.0 sec 79.6 MBytes 668 Mbits/sec 0.006 ms 6629/63415 (10%) [ 3] 7.0- 8.0 sec 79.6 MBytes 667 Mbits/sec 0.013 ms 6575/63334 (10%) [ 3] 8.0- 9.0 sec 79.7 MBytes 669 Mbits/sec 0.014 ms 6595/63471 (10%) [ 3] 0.0-10.0 sec 796 MBytes 668 Mbits/sec 0.008 ms 66674/634704 (11%) [ 3] 0.0-10.0 sec 1 datagrams received out-of-order [root@olympie ~]# iperf -u -c 157.159.33.76 -w32M -t 10 -l9170 -b10G -B 157.159.33.77 -P1 ------------------------------------------------------------ Client connecting to 157.159.33.76, UDP port 5001 Binding to local address 157.159.33.77 Sending 9170 byte datagrams UDP buffer size: 256 KByte (WARNING: requested 32.0 MByte) ------------------------------------------------------------ [ 3] local 157.159.33.77 port 5001 connected with 157.159.33.76 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 5.42 GBytes 4.66 Gbits/sec [ 3] Sent 634705 datagrams [ 3] Server Report: [ 3] 0.0-10.0 sec 796 MBytes 668 Mbits/sec 0.007 ms 66674/634704 (11%) [ 3] 0.0-10.0 sec 1 datagrams received out-of-order
Optimisations
Le 23/02/2011 09:43, Guillaume a écrit : > Hello Jehan, > > - Attention aux buffers > > TCP window size: 27.8 KByte (default) > > UDP buffer size: 256 KByte (WARNING: requested 32.0 MByte) > > Les paramètres donnés à iperf sont non prioritaires par rapport aux paramètres sysctl, donc même si tu demande w32M ça n'est pas forcément utilisé. Je ne me souvient plus exactement quels paramètres kernel sont en jeu, mais peut être: > net.core.rmem_max = 16777216 > net.core.wmem_max = 16777216 > net.ipv4.tcp_rmem = 4096 87380 16777216 > net.ipv4.tcp_wmem = 4096 65536 16777216 > (les paramètres tcp sont aussi pour UDP) > > - Il me semble avoir lu une histoire de problème de timestamp sur certains tests iperf en UDP: Chaque paquet part avec un timestamp et donc si tu met un seul thread ça fait un goulot d'appel système. Peut être tester avec 10 thread à 1G (options -P10 -b1G). > > - Les valeurs retournées par les tests iperf sont souvent fantaisistes, surtout quand ça marche moyen. Plutôt utiliser le monitoring des interfaces de l'équipement réseau en face. > > Guillaume
Augmentation du nombre de thread a 8 pour parralleliser les transferts
[root@olympie ~]# iperf -u -c 157.159.33.76 -w32M -t 10 -l9170 -b10G -B 157.159.33.77 -P8 bind failed: Address already in use bind failed: Address already in use bind failed: Address already in use bind failed: Address already in use bind failed: Address already in use bind failed: Address already in use bind failed: Address already in use ------------------------------------------------------------ Client connecting to 157.159.33.76, UDP port 5001 Binding to local address 157.159.33.77 Sending 9170 byte datagrams UDP buffer size: 256 KByte (WARNING: requested 32.0 MByte) ------------------------------------------------------------ [ 10] local 157.159.33.77 port 33392 connected with 157.159.33.76 port 5001 [ 3] local 157.159.33.77 port 5001 connected with 157.159.33.76 port 5001 [ 5] local 157.159.33.77 port 39469 connected with 157.159.33.76 port 5001 [ 4] local 157.159.33.77 port 36797 connected with 157.159.33.76 port 5001 [ 7] local 157.159.33.77 port 58107 connected with 157.159.33.76 port 5001 [ 6] local 157.159.33.77 port 48790 connected with 157.159.33.76 port 5001 [ 8] local 157.159.33.77 port 38680 connected with 157.159.33.76 port 5001 [ 9] local 157.159.33.77 port 57872 connected with 157.159.33.76 port 5001 [ ID] Interval Transfer Bandwidth [ 10] 0.0-10.0 sec 1.44 GBytes 1.23 Gbits/sec [ 10] Sent 168182 datagrams [ 3] 0.0-10.0 sec 1.44 GBytes 1.24 Gbits/sec [ 3] Sent 168473 datagrams [ 5] 0.0-10.0 sec 1.44 GBytes 1.23 Gbits/sec [ 5] Sent 168061 datagrams [ 4] 0.0-10.0 sec 1.44 GBytes 1.23 Gbits/sec [ 4] Sent 168243 datagrams [ 7] 0.0-10.0 sec 1.43 GBytes 1.23 Gbits/sec [ 7] Sent 167800 datagrams [ 6] 0.0-10.0 sec 1.44 GBytes 1.23 Gbits/sec [ 6] Sent 168212 datagrams [ 8] 0.0-10.0 sec 1.43 GBytes 1.23 Gbits/sec [ 8] Sent 167895 datagrams [ 9] 0.0-10.0 sec 1.43 GBytes 1.23 Gbits/sec [ 9] Sent 167909 datagrams [SUM] 0.0-10.0 sec 11.5 GBytes 9.86 Gbits/sec
