HTTP Illustrated
In TCP/IP Illustrated, Richard Stevens describe the TCP/IP protocol stack by generating interesting situations and looking at what happens on the wire with tcpdump. Don't look for pretty illustrations, "illustrated" here means "by examples".
This is an excellent set of books, a bit dated but the technique lives on. Learning this stuff by just reading the RFCs must be painful. For some reason the RFC editors want the official documents to be plain ASCII with manual page breaks. For the nostalgic it must feel good to read document in 2006 that look exactly like a bunch of typewritten pages from 1966. The RFC editor even recommends using troff to "typeset" the document... Let's hope that they hear about Docbook one of these days. OK Docbook is not that nice, we can blame XML's verbosity for most of the problems and the rest of the blame goes to the extremely fragile tool chain that is considered "standard" on most GNU/Linux distributions. But at least with Docbook you can hack your own tool chain and you'll end up with stuff that don't look like crap.
In the mean time fooling around with the protocols is fun. I had this friend who really liked to send emails in raw SMTP by telneting to his mail server. I'm still not sure if it was because he like that or because XFree86 was still compiling on his gentoo box (or was it a LFS?). Nevertheless, now I understand why he managed to get the big bucks as a sysadmin while I wasted my time coding reporting code for balance sheets. I'd link to his website but he tells me that he don't have one anymore, it had something to do with compilations thats never finished...
So I started to telnet to port 80 on my favorite webservers. HTTP is a bit verbose so I soon opted to hack a micro HTTP client. Thanks to the socket module I can do that in a real small python script:
#!/usr/bin/python
import socket as s
import sys
CHUCK = 8000
if len(sys.argv) != 2:
print "USAGE: %s host" % sys.argv[0]
sys.exit()
host = sys.argv[1]
sock = s.socket(s.AF_INET, s.SOCK_STREAM)
sock.connect((host, 80))
sock.send("GET / HTTP/1.1\n")
sock.send("Connection: close\n")
sock.send("Host: %s\n\n" % host)
done = False
chunks = []
while not done:
buf = sock.recv(CHUCK)
chunks.append(buf)
print str(buf)
print "** read %4d bytes" % len(buf)
if len(buf) == 0:
done = True
print "DONE"
# headers are sometimes interesting
for l in "".join(chunks).split("\n"):
if not l.strip():
break
print l
Anything interesting that wasn't expected? You bet! First of
all the W3C pretend that you should always be able to use HTTP 0.9
to some extent. In 0.9 you can fetch foo.html with
GET foo.html\n\n and that's it: not headers or funny
stuff. Well forget about that, no one cares about retro
compatibility and they'll close the connection as soon as you try.
This isn't really a problem, just a really check between what you
read in textbooks and what happen on the wire. And then there is
this persistent connection thingy. If you are a web browser that's
probably a good thing but if you are just fooling around, you really
want to get rid of it: Connection: close\n
All respectable webservers won't care if you prepend
www or not. Half of then redirect to www
if you don't prepend, the other half redirect to the straight domain
of you prepend. A tiny fraction really don't care and spit out the
content right away. Another tiny fraction really want you
to prepend and yell at you if you don't. I must be missing
something because www seems obsolete to me.
You can learn a lot about the structure of generating code on the webserver just by looking at how the stuff is sent to you:
$ python mini-web.py www.uqam.ca ** read 1440 bytes ** read 1440 bytes ** read 1440 bytes ** read 45 bytes ** read 1440 bytes ** read 1440 bytes ** read 1332 bytes ** read 1440 bytes ** read 1440 bytes ** read 727 bytes ** read 0 bytes DONE
Here the server spit out the content in three bursts. With close inspection we see that the bursts corresponds to the usual web page division: common head, inner content, common footer. But why chucks of 1440 bytes? Here is a hint: I'm connected with DSL. Here is another hint, tcpdump has this to say:
# tcpdump port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
11:53:33.437125 IP ogre.33972 > girofle2.telecom.uqam.ca.www:
S 2059998486:2059998486(0) win 5840
<mss 1460,sackOK,timestamp 468230511 0,nop,wscale 2>
11:53:33.481900 IP girofle2.telecom.uqam.ca.www > ogre.33972:
S 3362836129:3362836129(0) ack 2059998487 win 25920
<nop,nop,timestamp 3450314929 468230511,
nop,wscale 0,nop,nop,sackOK,mss 1460>
11:53:33.481948 IP ogre.33972 > girofle2.telecom.uqam.ca.www:
. ack 1 win 1460
<nop,nop,timestamp 468230556 3450314929>
11:53:33.482148 IP ogre.33972 > girofle2.telecom.uqam.ca.www:
P 1:16(15) ack 1 win 1460
<nop,nop,timestamp 468230556 3450314929>
11:53:33.518047 IP girofle2.telecom.uqam.ca.www > ogre.33972:
. ack 16 win 25920
<nop,nop,timestamp 3450314933 468230556>
11:53:33.518102 IP ogre.33972 > girofle2.telecom.uqam.ca.www:
P 16:53(37) ack 1 win 1460
<nop,nop,timestamp 468230592 3450314933>
11:53:33.568422 IP girofle2.telecom.uqam.ca.www > ogre.33972:
. ack 53 win 25920
<nop,nop,timestamp 3450314938 468230592>
11:53:33.624884 IP girofle2.telecom.uqam.ca.www > ogre.33972:
. 1:1441(1440) ack 53 win 25920
<nop,nop,timestamp 3450314943 468230592>
11:53:33.625055 IP ogre.33972 > girofle2.telecom.uqam.ca.www:
. ack 1441 win 2184
<nop,nop,timestamp 468230699 3450314943>
11:53:33.629372 IP girofle2.telecom.uqam.ca.www > ogre.33972:
P 1441:2881(1440) ack 53 win 25920
<nop,nop,timestamp 3450314943 468230592>
11:53:33.629448 IP ogre.33972 > girofle2.telecom.uqam.ca.www:
. ack 2881 win 2908
<nop,nop,timestamp 468230704 3450314943>
11:53:33.634043 IP girofle2.telecom.uqam.ca.www > ogre.33972:
P 2881:4321(1440) ack 53 win 25920
<nop,nop,timestamp 3450314943 468230592>
Specifically, look at the SYN packet (ogre is my workstation): mss is 1460 and a TCP header is 20 bytes when there are no options. Everything adds up and we notice that splitting the packets is slow enough for the script to have time to print to the console before another packet is read. From this we can conjecture that a bigger mss would probably crank-up throughput significantly (or that I need a faster link). Initial handshake took 50ms which isn't that much but this is still something that is saved for free from subsequent GETs with persistent connections.
Anything else? Not really, except maybe that there are Easter
eggs in the Slashdot http headers. Yahoo.ca replies a 404 on /
about 30% of the time. I guess these guys don't care about
monitoring... All of this is a lot of fun isn't it? How I wish I
had done that before my trip to Silicon Valley!
;
)