所用工具
- tcpdump:linux下分析网络的一个好用的终端工具
- psysh:php的一个终端执行工具，可以很方便的在终端执行php代码
- curl:php的curl库
tcpdump报文中Flags代表报文类型:
S (SYN), F (FIN), P (PUSH), R (RST), U (URG), W (ECN CWR), E (ECN-Echo) or ‘.’ (ACK), or ‘none’ if no flags are set.
分析背景
最近在了解swoole，因为公司的技术栈是php，所以之前用的java那一套在普通的php开发上完全用不了。为了让php有更好的性能，更广泛的使用场景，找了找相关资料，了解到swoole这个库。决定使用之后，项目中有个地方需要使用到http连接池，所以需要使用curl，但是curl虽然强大，但是参数实在是太多了，里面的实现细节也只能查看c的源代码才知道。所以只能曲线救国，通过tcpdump了解curl在http请求上的工作机制
tcpdump的使用方法是:
案例分析
案例一
只执行一次exec，等待一段时间，强制关闭终端进程
<?php
context = curl_init();
curl_setopt(context,CURLOPT_URL,"http://www.izuqun.com");
curl_exec($context);
整个过程的报文如下
`192.168.1.107.44664 > 116.62.25.128.http`: Flags [S], cksum 0x4bdf (correct), seq 3687066847, win 29200, options [mss 1460,sackOK,TS val 5931640 ecr 0,nop,wscale 7], length 0
16:28:54.014280 IP (tos 0x20, ttl 52, id 0, offset 0, flags [DF], proto TCP (6), length 60)
`116.62.25.128.http > 192.168.1.107.44664`: Flags [S.], cksum 0xbcb5 (correct), seq 4174849842, ack 3687066848, win 28960, options [mss 1460,sackOK,TS val 3844377305 ecr 5931640,nop,wscale 7], length 0
16:28:54.014304 IP (tos 0x0, ttl 64, id 44712, offset 0, flags [DF], proto TCP (6), length 52)
`192.168.1.107.44664 > 116.62.25.128.http`: Flags [.], cksum 0x5bba (correct), ack 1, win 229, options [nop,nop,TS val 5931643 ecr 3844377305], length 0
16:28:54.014334 IP (tos 0x0, ttl 64, id 44713, offset 0, flags [DF], proto TCP (6), length 105)
`192.168.1.107.44664 > 116.62.25.128.http`: Flags [P.], cksum 0xee60 (correct), seq 1:54, ack 1, win 229, options [nop,nop,TS val 5931643 ecr 3844377305], length 53: HTTP, length: 53
GET / HTTP/1.1
Host: www.izuqun.com
Accept: *\*
16:28:54.023036 IP (tos 0x20, ttl 52, id 44520, offset 0, flags [DF], proto TCP (6), length 52)
`116.62.25.128.http > 192.168.1.107.44664`: Flags [.], cksum 0x5b85 (correct), ack 54, win 227, options [nop,nop,TS val 3844377307 ecr 5931643], length 0
16:28:54.023061 IP (tos 0x20, ttl 52, id 44521, offset 0, flags [DF], proto TCP (6), length 314)
`116.62.25.128.http > 192.168.1.107.44664`: Flags [P.], cksum 0xba4d (correct), seq 1:263, ack 54, win 227, options [nop,nop,TS val 3844377307 ecr 5931643], length 262: HTTP, length: 262
HTTP/1.1 200 OK
Server: nginx/1.12.2
Date: Mon, 11 Dec 2017 08:28:54 GMT
Content-Type: text/html
Content-Length: 1243
Last-Modified: Mon, 11 Dec 2017 02:35:33 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5a2deef5-4db"
Accept-Ranges: bytes
16:28:54.023076 IP (tos 0x0, ttl 64, id 44714, offset 0, flags [DF], proto TCP (6), length 52)
`192.168.1.107.44664 > 116.62.25.128.http`: Flags [.], cksum 0x5a73 (correct), ack 263, win 237, options [nop,nop,TS val 5931645 ecr 3844377307], length 0
16:28:54.024214 IP (tos 0x20, ttl 52, id 44522, offset 0, flags [DF], proto TCP (6), length 1295)
`116.62.25.128.http > 192.168.1.107.44664`: Flags [P.], cksum 0x870b (correct), seq 263:1506, ack 54, win 227, options [nop,nop,TS val 3844377307 ecr 5931643], length 1243: HTTP
16:28:54.024236 IP (tos 0x0, ttl 64, id 44715, offset 0, flags [DF], proto TCP (6), length 52)
`192.168.1.107.44664 > 116.62.25.128.http`: Flags [.], cksum 0x5585 (correct), ack 1506, win 256, options [nop,nop,TS val 5931645 ecr 3844377307], length 0
1~3报文：tcp的三次握手，
4~5报文：curl发起的get请求、服务器的ack回应
6~9报文：服务器对get请求的返回、我的机器的内核的ack，因为报文长度是超出了最大的限制，所以分成两次发送
稍等一会儿，又有新的报文
`116.62.25.128.http > 192.168.1.107.44664`: Flags [F.], cksum 0x161d (correct), seq 1506, ack 54, win 227, options [nop,nop,TS val 3844393567 ecr 5931645], length 0
16:29:59.097967 IP (tos 0x0, ttl 64, id 44716, offset 0, flags [DF], proto TCP (6), length 52)
`192.168.1.107.44664 > 116.62.25.128.http`: Flags [.], cksum 0xd672 (correct), ack 1507, win 256, options [nop,nop,TS val 5947914 ecr 3844393567], length 0
这是服务器发起的tcp连接关闭的报文，我的机器响应了，但是没有主动再次发送Fin
主动关闭psysh进程，出现大量报文
192.168.1.107.44664 > 116.62.25.128.http: Flags [F.], cksum 0x8765 (correct), seq 54, ack 1507, win 256, options [nop,nop,TS val 5968150 ecr 3844393567], length 0
16:31:20.253999 IP (tos 0x0, ttl 64, id 44718, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.107.44664 > 116.62.25.128.http: Flags [F.], cksum 0x8730 (correct), seq 54, ack 1507, win 256, options [nop,nop,TS val 5968203 ecr 3844393567], length 0
16:31:20.465940 IP (tos 0x0, ttl 64, id 44719, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.107.44664 > 116.62.25.128.http: Flags [F.], cksum 0x86fb (correct), seq 54, ack 1507, win 256, options [nop,nop,TS val 5968256 ecr 3844393567], length 0
16:31:20.890008 IP (tos 0x0, ttl 64, id 44720, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.107.44664 > 116.62.25.128.http: Flags [F.], cksum 0x8691 (correct), seq 54, ack 1507, win 256, options [nop,nop,TS val 5968362 ecr 3844393567], length 0
16:31:21.737935 IP (tos 0x0, ttl 64, id 44721, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.107.44664 > 116.62.25.128.http: Flags [F.], cksum 0x85bd (correct), seq 54, ack 1507, win 256, options [nop,nop,TS val 5968574 ecr 3844393567], length 0
16:31:23.437992 IP (tos 0x0, ttl 64, id 44722, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.107.44664 > 116.62.25.128.http: Flags [F.], cksum 0x8414 (correct), seq 54, ack 1507, win 256, options [nop,nop,TS val 5968999 ecr 3844393567], length 0
16:31:26.833967 IP (tos 0x0, ttl 64, id 44723, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.107.44664 > 116.62.25.128.http: Flags [F.], cksum 0x80c3 (correct), seq 54, ack 1507, win 256, options [nop,nop,TS val 5969848 ecr 3844393567], length 0
16:31:33.633977 IP (tos 0x0, ttl 64, id 44724, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.107.44664 > 116.62.25.128.http: Flags [F.], cksum 0x7a1f (correct), seq 54, ack 1507, win 256, options [nop,nop,TS val 5971548 ecr 3844393567], length 0
16:31:47.218007 IP (tos 0x0, ttl 64, id 44725, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.107.44664 > 116.62.25.128.http: Flags [F.], cksum 0x6cdb (correct), seq 54, ack 1507, win 256, options [nop,nop,TS val 5974944 ecr 3844393567], length 0
16:32:14.417998 IP (tos 0x0, ttl 64, id 44726, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.107.44664 > 116.62.25.128.http: Flags [F.], cksum 0x524b (correct), seq 54, ack 1507, win 256, options [nop,nop,TS val 5981744 ecr 3844393567], length 0
这些报文都是我的机器主动发送的关闭报文，但是服务器也不再响应
案例二
完整的使用curl_init和curl_close，显式关闭连接
<?php
context = curl_init();
curl_setopt(context,CURLOPT_URL,"http://www.izuqun.com");
curl_exec(context);
curl_close(context);
报文如下
192.168.1.107.44684 > 116.62.25.128.http: Flags [S], cksum 0x469f (correct), seq 2378041097, win 29200, options [mss 1460,sackOK,TS val 6028159 ecr 0,nop,wscale 7], length 0
16:35:20.090006 IP (tos 0x20, ttl 52, id 0, offset 0, flags [DF], proto TCP (6), length 60)
116.62.25.128.http > 192.168.1.107.44684: Flags [S.], cksum 0xd6a7 (correct), seq 3548165201, ack 2378041098, win 28960, options [mss 1460,sackOK,TS val 3844473825 ecr 6028159,nop,wscale 7], length 0
16:35:20.090050 IP (tos 0x0, ttl 64, id 64631, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.107.44684 > 116.62.25.128.http: Flags [.], cksum 0x75ac (correct), ack 1, win 229, options [nop,nop,TS val 6028162 ecr 3844473825], length 0
16:35:20.090105 IP (tos 0x0, ttl 64, id 64632, offset 0, flags [DF], proto TCP (6), length 105)
192.168.1.107.44684 > 116.62.25.128.http: Flags [P.], cksum 0x0853 (correct), seq 1:54, ack 1, win 229, options [nop,nop,TS val 6028162 ecr 3844473825], length 53: HTTP, length: 53
GET / HTTP/1.1
Host: www.izuqun.com
Accept: */*
16:35:20.099626 IP (tos 0x20, ttl 52, id 47030, offset 0, flags [DF], proto TCP (6), length 52)
116.62.25.128.http > 192.168.1.107.44684: Flags [.], cksum 0x7576 (correct), ack 54, win 227, options [nop,nop,TS val 3844473828 ecr 6028162], length 0
16:35:20.099655 IP (tos 0x20, ttl 52, id 47031, offset 0, flags [DF], proto TCP (6), length 314)
116.62.25.128.http > 192.168.1.107.44684: Flags [P.], cksum 0xda41 (correct), seq 1:263, ack 54, win 227, options [nop,nop,TS val 3844473828 ecr 6028162], length 262: HTTP, length: 262
HTTP/1.1 200 OK
Server: nginx/1.12.2
Date: Mon, 11 Dec 2017 08:35:20 GMT
Content-Type: text/html
Content-Length: 1243
Last-Modified: Mon, 11 Dec 2017 02:35:33 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5a2deef5-4db"
Accept-Ranges: bytes
16:35:20.099670 IP (tos 0x0, ttl 64, id 64633, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.107.44684 > 116.62.25.128.http: Flags [.], cksum 0x7464 (correct), ack 263, win 237, options [nop,nop,TS val 6028164 ecr 3844473828], length 0
16:35:20.099680 IP (tos 0x20, ttl 52, id 47032, offset 0, flags [DF], proto TCP (6), length 1295)
116.62.25.128.http > 192.168.1.107.44684: Flags [P.], cksum 0xa0fc (correct), seq 263:1506, ack 54, win 227, options [nop,nop,TS val 3844473828 ecr 6028162], length 1243: HTTP
16:35:20.099686 IP (tos 0x0, ttl 64, id 64634, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.107.44684 > 116.62.25.128.http: Flags [.], cksum 0x6f76 (correct), ack 1506, win 256, options [nop,nop,TS val 6028164 ecr 3844473828], length 0
这边的报文一开始和案例一完全一致，但是在我使用curl_close()显式关闭curl后，并没有发送任何关闭tcp连接的报文。查了一下资料，在php7.0中，如果$context任何有全局引用，即使使用curl_close()显式关闭，也不会关闭连接。
案例3
完整的使用curl_init和curl_close，显式关闭连接，并且多次执行exec
16:38:26.590098 IP (tos 0x0, ttl 64, id 18826, offset 0, flags [DF], proto TCP (6), length 60)
192.168.1.107.44714 > 116.62.25.128.http: Flags [S], cksum 0x520f (correct), seq 3672832041, win 29200, options [mss 1460,sackOK,TS val 6074787 ecr 0,nop,wscale 7], length 0
16:38:26.604823 IP (tos 0x20, ttl 52, id 0, offset 0, flags [DF], proto TCP (6), length 60)
116.62.25.128.http > 192.168.1.107.44714: Flags [S.], cksum 0xf795 (correct), seq 3884635295, ack 3672832042, win 28960, options [mss 1460,sackOK,TS val 3844520454 ecr 6074787,nop,wscale 7], length 0
16:38:26.604870 IP (tos 0x0, ttl 64, id 18827, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.107.44714 > 116.62.25.128.http: Flags [.], cksum 0x969a (correct), ack 1, win 229, options [nop,nop,TS val 6074790 ecr 3844520454], length 0
16:38:26.604904 IP (tos 0x0, ttl 64, id 18828, offset 0, flags [DF], proto TCP (6), length 105)
192.168.1.107.44714 > 116.62.25.128.http: Flags [P.], cksum 0x2941 (correct), seq 1:54, ack 1, win 229, options [nop,nop,TS val 6074790 ecr 3844520454], length 53: HTTP, length: 53
GET / HTTP/1.1
Host: www.izuqun.com
Accept: */*
16:38:26.616651 IP (tos 0x20, ttl 52, id 28329, offset 0, flags [DF], proto TCP (6), length 52)
116.62.25.128.http > 192.168.1.107.44714: Flags [.], cksum 0x9664 (correct), ack 54, win 227, options [nop,nop,TS val 3844520457 ecr 6074790], length 0
16:38:26.616674 IP (tos 0x20, ttl 52, id 28330, offset 0, flags [DF], proto TCP (6), length 314)
116.62.25.128.http > 192.168.1.107.44714: Flags [P.], cksum 0xf829 (correct), seq 1:263, ack 54, win 227, options [nop,nop,TS val 3844520457 ecr 6074790], length 262: HTTP, length: 262
HTTP/1.1 200 OK
Server: nginx/1.12.2
Date: Mon, 11 Dec 2017 08:38:26 GMT
Content-Type: text/html
Content-Length: 1243
Last-Modified: Mon, 11 Dec 2017 02:35:33 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5a2deef5-4db"
Accept-Ranges: bytes
16:38:26.616687 IP (tos 0x0, ttl 64, id 18829, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.107.44714 > 116.62.25.128.http: Flags [.], cksum 0x9551 (correct), ack 263, win 237, options [nop,nop,TS val 6074793 ecr 3844520457], length 0
16:38:26.616701 IP (tos 0x20, ttl 52, id 28331, offset 0, flags [DF], proto TCP (6), length 1295)
116.62.25.128.http > 192.168.1.107.44714: Flags [P.], cksum 0xc1ea (correct), seq 263:1506, ack 54, win 227, options [nop,nop,TS val 3844520457 ecr 6074790], length 1243: HTTP
16:38:26.616708 IP (tos 0x0, ttl 64, id 18830, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.107.44714 > 116.62.25.128.http: Flags [.], cksum 0x9063 (correct), ack 1506, win 256, options [nop,nop,TS val 6074793 ecr 3844520457], length 0
16:38:28.241841 IP (tos 0x0, ttl 64, id 18831, offset 0, flags [DF], proto TCP (6), length 105)
192.168.1.107.44714 > 116.62.25.128.http: Flags [P.], cksum 0x2174 (correct), seq 54:107, ack 1506, win 256, options [nop,nop,TS val 6075199 ecr 3844520457], length 53: HTTP, length: 53
GET / HTTP/1.1
Host: www.izuqun.com
Accept: */*
16:38:28.253636 IP (tos 0x20, ttl 52, id 28332, offset 0, flags [DF], proto TCP (6), length 314)
116.62.25.128.http > 192.168.1.107.44714: Flags [P.], cksum 0xeede (correct), seq 1506:1768, ack 107, win 227, options [nop,nop,TS val 3844520867 ecr 6075199], length 262: HTTP, length: 262
HTTP/1.1 200 OK
Server: nginx/1.12.2
Date: Mon, 11 Dec 2017 08:38:28 GMT
Content-Type: text/html
Content-Length: 1243
Last-Modified: Mon, 11 Dec 2017 02:35:33 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5a2deef5-4db"
Accept-Ranges: bytes
16:38:28.253689 IP (tos 0x0, ttl 64, id 18832, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.107.44714 > 116.62.25.128.http: Flags [.], cksum 0x8be1 (correct), ack 1768, win 276, options [nop,nop,TS val 6075202 ecr 3844520867], length 0
16:38:28.253704 IP (tos 0x20, ttl 52, id 28333, offset 0, flags [DF], proto TCP (6), length 1295)
116.62.25.128.http > 192.168.1.107.44714: Flags [P.], cksum 0xb8a1 (correct), seq 1768:3011, ack 107, win 227, options [nop,nop,TS val 3844520867 ecr 6075199], length 1243: HTTP
16:38:28.253713 IP (tos 0x0, ttl 64, id 18833, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.107.44714 > 116.62.25.128.http: Flags [.], cksum 0x86f3 (correct), ack 3011, win 295, options [nop,nop,TS val 6075202 ecr 3844520867], length 0
多次curl_exec()会复用之前的连接，但是curl_close()依然是不能关闭连接的。但当我使用unset($context)时，观察到了以下报文：
192.168.1.107.49938 > 116.62.25.128.http: Flags [F.], cksum 0x8407 (correct), seq 54, ack 1506, win 256, options [nop,nop,TS val 274723 ecr 3860448314], length 0
10:20:58.408173 IP (tos 0x20, ttl 52, id 25537, offset 0, flags [DF], proto TCP (6), length 52)
116.62.25.128.http > 192.168.1.107.49938: Flags [F.], cksum 0x5ca5 (correct), seq 1506, ack 55, win 227, options [nop,nop,TS val 3860458424 ecr 274723], length 0
10:20:58.408218 IP (tos 0x0, ttl 64, id 34603, offset 0, flags [DF], proto TCP (6), length 52)
192.168.1.107.49938 > 116.62.25.128.http: Flags [.], cksum 0x5c85 (correct), ack 1507, win 256, options [nop,nop,TS val 274726 ecr 3860458424], length 0
我的机器主动关闭连接发出Fin，服务器ack并发送Fin,我的机器也ack。两边顺利的完成了连接的关闭。
所以当前可以得出以下几点:
1.$context = curl_init(), $context是可以复用的，复用$context后，多次执行curl_exec()可以避免tcp的3次握手和4次断开的过程。
2.curl_close()在上面的试验中，并不能正常的关闭连接，这是因为在psysh的执行环境中，$context一直是在当前的作用域，会一直保持着变量的引用。而php7.0中，$context如果保持引用，curl_close则不会关闭连接，而unset($context)则会关闭。
3.服务器主动关闭连接后发出Fin，我的机器虽然响应了ack，但是没有主动发送Fin，这会不会导致服务器大量的fin_wait2？
第三点是很重要的，如果不解决会大量占用服务器资源。
所以再做一个实验，实验过程如下：
- 我的机器初始化大量的curl_init()，然后不释放，等待服务器关闭。使用netstat查看服务器是否会出现大量的fin_wait2。
使用netstat -an|awk '/tcp/ {print $6}'|sort|uniq -c可以快速查看各种tcp连接状态的统计。
初始情况为：
20 CLOSE_WAIT
63 ESTABLISHED
10 LISTEN
psysh中执行以下脚本
$contextArr = array();
for($i = 0; $i < 1000; $i++){
$contextArr[$i] = curl_init();
curl_setopt($contextArr[$i],CURLOPT_URL,"http://www.izuqun.com");
curl_exec($contextArr[$i]);
}
脚本执行完后，服务器状态变化如下：
20 CLOSE_WAIT
1058 ESTABLISHED
5 FIN_WAIT2
10 LISTEN
2 TIME_WAIT
20 CLOSE_WAIT
58 ESTABLISHED
1001 FIN_WAIT2
10 LISTEN
20 CLOSE_WAIT
63 ESTABLISHED
10 LISTEN
可以看到，中间有1000个FIN_WAIT2状态，说明我们的猜想是正确的。