I tried fiddling around w & w/o checkpoints and locks and it seems the whole performance overhead is due to copying. Since ASPLOS, after getting a new version of memcpy, as compared to native 750, I am getting fluctuating results 420-635 for FGFT. But with two/three threads I am seeing consistent 680 aggregate. With a powerful machine, I am seeing consistent results now. The fluctuation in single thread is gone. Just to reiterate a mail I sent before ASPLOS deadline, our high CPU comes from disabling interrupts during the whole copyin and copy out which drivers up CPU utilization on one thread significantly (even when compared to partial isolation). I still am not sure why this is happening - I did re-test device locks and interrupts and they are indeed correct. Anyways, I moved to adsl-q08 and now I am getting extremely consistent results. The copy overhead is significantly lower. The new results are: One thread ========= Native: 843 2.5% CPU (7% performance loss) FGFT: 784 3.4% CPU FGFT 1/2: 811 2.84% Two thread: ========== Native: 843 FGFT: 783 CPU util unchanged Three thread: =========== Native: ~843 FGFT: ~784 CPU util 0.01 % higher. ================================================================================= One thread Native ============== 847 root@adsl-q08 ipc_drivers]# netperf -H 128.105.104.101 -c -C MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 128.105.104.101 (128.105.104.101) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.06 843.52 2.53 27.63 0.985 2.685 [root@adsl-q08 ipc_drivers]# netperf -H 128.105.104.101 -c -C MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 128.105.104.101 (128.105.104.101) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.05 843.60 2.51 27.46 0.977 2.668 [root@adsl-q08 ipc_drivers]# [root@adsl-q08 ipc_drivers]# [root@adsl-q08 ipc_drivers]# netperf -H 128.105.104.101 -c -C MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 128.105.104.101 (128.105.104.101) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.05 843.50 2.54 28.16 0.986 2.736 One thread FGFT ============= [root@adsl-q08 ipc_drivers]# netperf -H 128.105.104.101 -c -C MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 128.105.104.101 (128.105.104.101) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.05 784.45 3.41 27.96 1.424 2.921 [root@adsl-q08 ipc_drivers]# netperf -H 128.105.104.101 -c -C MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 128.105.104.101 (128.105.104.101) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.06 784.21 3.43 27.44 1.435 2.868 [root@adsl-q08 ipc_drivers]# netperf -H 128.105.104.101 -c -C MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 128.105.104.101 (128.105.104.101) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.06 784.62 3.36 27.36 1.403 2.857 Two threads Native ============== MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 128.105.104.101 (128.105.104.101) port 0 AF_INET [root@adsl-q08 ~]# MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 128.105.104.101 (128.105.104.101) port 0 AF_INET [root@adsl-q08 ~]# Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.09 434.88 2.60 28.08 1.963 5.290 Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.10 408.56 2.58 28.22 2.071 5.664 Two threads FGFT ============== [root@adsl-q08 ipc_drivers]# MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 128.105.104.101 (128.105.104.101) port 0 AF_INET [root@adsl-q08 ipc_drivers]# Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.11 402.19 3.63 28.19 2.962 5.749 Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.11 380.18 3.64 28.19 3.137 6.082 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 128.105.104.101 (128.105.104.101) port 0 AF_INET [root@adsl-q08 ipc_drivers]# [root@adsl-q08 ipc_drivers]# Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.12 405.96 3.58 27.27 2.895 5.510 Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.12 377.42 3.54 27.25 3.071 5.916 Three threads Native ================ root@adsl-q08 ~]# for i in 1 2 Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.12 264.45 2.60 28.33 3.219 8.781 Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.14 294.71 2.62 28.47 2.909 7.916 Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.15 284.26 2.64 28.44 3.041 8.206 ---- [root@adsl-q08 ~]# MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 128.105.104.101 (128.105.104.101) port 0 AF_INET MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 128.105.104.101 (128.105.104.101) port 0 AF_INET MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 128.105.104.101 (128.105.104.101) port 0 AF_INET [root@adsl-q08 ~]# [root@adsl-q08 ~]# Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.12 288.73 2.67 28.68 3.034 8.146 Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.15 263.44 2.68 28.57 3.346 8.901 Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.17 291.55 2.68 28.71 3.014 8.074 Three threads FGFT =============== [root@adsl-q08 ~]# Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.16 268.46 3.72 27.95 4.537 8.533 Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.17 268.72 3.72 27.95 4.532 8.523 Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.15 246.75 3.70 27.93 4.916 9.280 ======= [root@adsl-q08 ipc_drivers]# MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 128.105.104.101 (128.105.104.101) port 0 AF_INET [root@adsl-q08 ipc_drivers]# [root@adsl-q08 ipc_drivers]# Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.16 271.07 3.64 28.15 4.406 8.514 Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.18 271.64 3.63 28.19 4.386 8.506 Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.15 240.14 3.60 28.09 4.921 9.602 ======================================== One thread (1/2) ============= [root@adsl-q08 ipc_drivers]# netperf -H 128.105.104.101 -c -C MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 128.105.104.101 (128.105.104.101) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.06 811.60 2.91 28.63 1.175 2.890 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 128.105.104.101 (128.105.104.101) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.07 811.08 2.93 27.90 1.184 2.819 [root@adsl-q08 ipc_drivers]# [root@adsl-q08 ipc_drivers]# netperf -H 128.105.104.101 -c -C MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 128.105.104.101 (128.105.104.101) port 0 AF_INET Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % S % S us/KB us/KB 87380 655360 655360 10.05 811.49 2.84 26.97 1.148 2.722 100 threads ==========- for i in {1..100} ; do netperf -H 128.105.104.101 -c -C $>> 100t_native.txt & done @adsl-q08 q08_perf_cameraready]# cat 100t_native2.txt | grep "87" | awk '{ sum+=$5;} END {printf "aggr %f ", sum;}' aggr 941.700000 for i in {1..100} ; do netperf -H 128.105.104.101 -c -C $>> 100t_fgft2.txt & done aggr 860.280000