How HTTP proxying works in YMSG9 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Yahoo supports proxying via HTTP. The following explains how it works. Proxying in HTTP can be achieved simply by providing the full URL in the initial GET/POST header line, as opposed to just the document/query. The proxy will then truncate this into its regular form and contact the required host on behalf of the web client. All messages for Yahoo IM should be routed towards http.pager.yahoo.com:80. Yahoo uses POST methods exclusively for its messaging. Other URL's are contacted to fetch address books and other relate services - these are not covered in this text, although they too will have to be proxied. HTTP is a stateless protocol, with connections of a short lifespan. This means the underlying network mechanics have to change quite a bit from regular socket based comms. Luckily the actual content of the 'packets' differs only a little from the socket based system. (In this document I'll refer to single YMSG messages as 'packets'. This is to avoid confusion with 'messages', which refers to text messages of packet type 0x06. The term 'packet' does not refer to a TCP/IP packet - although most YMSG messages are indeed small enough to be encoded inside a single TCP/IP packet.) Yahoo cannot deliver 'packets' to the client unless the client sends a HTTP request. For this reason, packets are collected and delivered in batches as payloads on HTTP POST requests. A single outgoing packet (there is always only one) can be met by zero or more incoming packets on the return stream. Let's kick off with an example, by logging into the Yahoo service under HTTP. All network traffic should be directed towards your HTTP proxy server. We start with an outgoing HTTP message: >> POST http://http.pager.yahoo.com:80/notify HTTP/1.0 >> Content-length: 64 >> User-Agent: Mozilla/4.5 [en] (X11; U; FreeBSD 2.2.8-STABLE i386) >> Host: http.pager.yahoo.com >> >> 59 4d 53 47 09 00 c8 00 00 2c 00 57 00 00 00 00 00 00 00 00 YMSG.....,.W........ >> 30 c0 80 74 65 73 74 79 61 68 6f 6f c0 80 31 c0 80 74 65 73 0..testyahoo..1..tes >> 74 79 61 68 6f 6f c0 80 32 34 c0 80 31 30 34 39 31 33 39 38 tyahoo..24..10491398 >> 38 32 c0 80 82.. Note here three things: 1) The cookie is missing (we don't get this until we receive a LIST packet) 2) Tag 24: starts as zero, although does not appear to get reset for each session, so if you log out and log in again it begins where the previous session left off. A successful login will change this value (see below). 3) It's vital to check the header line to ensure a healthy 200 return code. We get back the following on our input stream: << HTTP/1.0 200 OK << Date: Mon, 31 Mar 2003 19:44:44 GMT << P3P: policyref="http://p3p.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi P << UBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE GOV" << Connection: close << Content-Type: text/plain << << 01 00 00 00 59 4d 53 47 00 00 00 00 00 33 00 57 00 00 00 01 ....YMSG.....3.W.... << 7a 20 00 00 31 c0 80 74 65 73 74 79 61 68 6f 6f c0 80 39 34 z ..1..testyahoo..94 << c0 80 31 39 59 33 78 4c 55 31 78 4d 5f 78 78 78 78 5f 78 45 ..19Y3xLU1xM_xxxx_xE << 6a 58 61 77 2d 2d c0 80 31 33 c0 80 30 c0 80 32 34 c0 80 31 jXaw--..13..0..24..1 << 30 34 39 31 33 39 38 38 32 c0 80 049139882.. Notice the first four bytes are the number of 'packets' encoded into the body, in little endian form. In this case, only one packet. The connection is now broken. To reply we make a new connection and send: >> POST http://http.pager.yahoo.com:80/notify HTTP/1.0 >> Content-length: 113 >> User-Agent: Mozilla/4.5 [en] (X11; U; FreeBSD 2.2.8-STABLE i386) >> Host: http.pager.yahoo.com >> >> 59 4d 53 47 09 00 c8 00 00 5d 00 54 00 00 00 00 7a 20 00 00 YMSG.....].T....z .. >> 30 c0 80 74 65 73 74 79 61 68 6f 6f c0 80 36 c0 80 72 7a 57 0..testyahoo..6..rzW >> 78 78 78 78 4c 51 5a 48 53 4e 4a 6a 49 53 54 78 78 78 78 2d xxxxLQZHSNJjISTxxxx- >> 2d c0 80 39 36 c0 80 78 78 78 78 4d 48 74 51 4e 78 44 4a 41 -..96..xxxxMHtQNxDJA >> 72 65 39 56 59 48 30 34 41 2d 2d c0 80 32 c0 80 31 c0 80 31 re9VYxxxx--..2..1..1 >> c0 80 74 65 73 74 79 61 68 6f 6f c0 80 ..testyahoo.. And we get back, on the input stream: << HTTP/1.0 200 OK << Date: Mon, 31 Mar 2003 19:44:50 GMT << P3P: policyref="http://p3p.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE GOV" << Connection: close << Content-Type: text/plain << << 02 00 00 00 59 4d 53 47 00 00 00 00 02 51 00 55 00 00 00 00 ....YMSG.....Q.U.... << 7a 20 00 00 38 37 c0 80 74 65 73 74 67 72 6f 75 70 3a 73 6f z ..87..testgroup:so << 6d 65 67 75 79 0a c0 80 38 38 c0 80 6d 79 66 72 69 65 6e 64 meguy...88..myfriend << c0 80 38 39 c0 80 74 65 73 74 79 61 68 6f 6f c0 80 35 39 c0 ..89..testyahoo..59. << 80 59 09 76 3d 31 26 6e 3d 61 6e 67 39 64 70 31 78 78 78 78 .Y.v=1&n=ang9dp1xxxx << 38 39 26 6c 3d 39 30 6c 30 61 38 33 33 6f 78 78 78 78 3d 6d 89&l=90l0a833oxxxx=m << 31 73 32 6f 33 32 30 31 33 30 30 30 31 32 33 26 6a 62 3d 32 1s2o32013000123&jb=2 << 37 7c 34 37 7c 26 72 3d 61 6f 26 6c 67 3d 75 73 26 69 6e 74 7|47|&r=ao&lg=us&int << 6c 3d 75 73 3b 20 65 78 70 69 72 65 73 3d 54 68 75 2c 20 31 l=us; expires=Thu, 1 << 35 20 41 70 72 20 32 30 31 30 20 32 30 3a 30 30 3a 30 30 20 5 Apr 2010 20:00:00 << 47 4d 54 3b 20 70 61 74 68 3d 2f 3b 20 64 6f 6d 61 69 6e 3d GMT; path=/; domain= << 2e 79 61 68 6f 6f 2e 63 6f 6d c0 80 35 39 c0 80 54 09 7a 3d .yahoo.com..59..T.z= << 79 71 4a 69 2e 41 79 77 65 69 2e 41 41 5a 43 31 78 78 78 78 yqJi.Aywei.AAZC1xxxx << 63 47 54 54 6a 55 47 4e 6a 59 33 4e 44 41 31 4d 78 78 78 78 cGTTjUGNjY3NDA1Mxxxx << 51 2d 2d 26 61 3d 51 41 45 26 73 6b 3d 44 41 41 78 78 78 78 Q--&a=QAE&sk=DAAxxxx << 71 64 33 36 4b 51 52 26 64 3d 63 32 77 42 54 31 78 78 78 78 qd36KQR&d=c2wBT1xxxx << 31 55 52 58 64 4e 65 6d 4e 35 54 6e 70 4e 4d 30 78 78 78 78 1URXdNemN5TnpNM0xxxx << 30 42 59 51 46 52 51 55 55 42 64 47 6c 77 41 55 78 78 78 78 0BYQFRQUUBdGlwAUxxxx << 67 7a 51 77 46 36 65 67 46 35 63 55 70 70 4c 6b 78 78 78 78 gzQwF6egF5cUppLkxxxx << 45 2d 3b 20 65 78 70 69 72 65 73 3d 54 68 75 2c 20 31 35 20 E-; expires=Thu, 15 << 41 70 72 20 32 30 31 30 20 32 30 3a 30 30 3a 30 30 20 47 4d Apr 2010 20:00:00 GM << 54 3b 20 70 61 74 68 3d 2f 3b 20 64 6f 6d 61 69 6e 3d 2e 79 T; path=/; domain=.y << 61 68 6f 6f 2e 63 6f 6d c0 80 35 39 c0 80 43 09 6d 67 3d 31 ahoo.com..59..C.mg=1 << c0 80 31 35 33 c0 80 39 c0 80 39 30 c0 80 31 c0 80 33 c0 80 ..153..9..90..1..3.. << 74 65 73 74 79 61 68 6f 6f c0 80 31 30 30 c0 80 30 c0 80 31 testyahoo..100..0..1 << 30 31 c0 80 c0 80 31 30 32 c0 80 c0 80 39 33 c0 80 38 36 34 01....102....93..864 << 30 30 c0 80 31 34 39 c0 80 38 58 62 48 77 65 75 78 78 78 78 00..149..8XbHweuxxxx << 30 73 78 42 75 64 59 30 72 4f 41 2d 2d c0 80 31 35 30 c0 80 0sxBudY0rOA--..150.. << 30 71 4d 61 62 52 73 67 39 56 76 31 77 79 5f 46 78 78 78 78 0qMabRsg9Vv1wy_Fxxxx << 4c 77 2d 2d c0 80 31 35 31 c0 80 76 2e 4f 47 68 78 78 78 78 Lw--..151..v.OGhxxxx << 68 6a 52 36 33 44 6f 67 31 63 2e 51 67 2d 2d c0 80 59 4d 53 hjR63Dog1c.Qg--..YMS << 47 00 00 00 00 00 2d 00 01 00 00 00 00 7a 20 00 00 30 c0 80 G.....-......z ..0.. << 74 65 73 74 79 61 68 6f 6f c0 80 31 c0 80 74 65 73 74 79 61 testyahoo..1..testya << 68 6f 6f c0 80 32 34 c0 80 31 30 34 39 31 33 39 38 39 30 c0 hoo..24..1049139890. << 80 00 .. Notice we got back two packets, as signified by the 02 00 00 00 at the beginning of the payload. There are two things to note here: 1) The cookie data - from now on all HTTP requests must carry a cookie field, or Yahoo will return a "Bad cookie" error. 2) Tag 24 has changed in the LOGON packet. From now on, all our messages will carry this new value. So, we are now successfully logged into Yahoo via HTTP. From now on, whenever we send a packet we will read back zero or more packets from the stream. (The four byte count is always present, even if it is zero.) What happens if we don't send a message or otherwise need to send an outgoing packet for a long time? Yahoo gets around this by using IDLE packets. If nothing has been sent for thirty seconds, the client sends an IDLE packet, simply to give the server an oppotunity to send the packets which have gathered and need delivering since its last contact with the client. >> POST http://http.pager.yahoo.com:80/notify HTTP/1.0 >> Content-length: 64 >> User-Agent: Mozilla/4.5 [en] (X11; U; FreeBSD 2.2.8-STABLE i386) >> Host: http.pager.yahoo.com >> Cookie: Y=v=1&n=ang9dp1xxxx89&l=90l0a383o/o&p=m1s2o32013001234&jb=27|47|&r=ao&lg=us&intl=us; T=z=yqJi.Aywei.AAZC1Q57jcGTTjUGNjY3NDA1MDQwNQ--&a=QAE&sk=DAAQRtYqd36KQR&d=c2wBxxxJQk1URXdxxxx5TnpNxxxLS0BYQFRxxxxdGlwAUI3MjgzQwF6exxxxUppLkFnV0E- >> >> 59 4d 53 47 09 00 c8 00 00 2c 00 05 00 00 00 00 7a 20 00 00 YMSG.....,......z .. >> 31 c0 80 74 65 73 74 79 61 68 6f 6f c0 80 30 c0 80 74 65 73 1..testyahoo..0..tes >> 74 79 61 68 6f 6f c0 80 32 34 c0 80 31 30 34 39 31 33 39 38 tyahoo..24..10491398 >> 39 30 c0 80 90.. Note: 1) The cookie is now present, both Y and T cookies values. 2) The 24 tag has changed - now using the value we got from LOGON. << HTTP/1.0 200 OK << Date: Mon, 31 Mar 2003 19:45:57 GMT << P3P: policyref="http://p3p.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE GOV" << Connection: close << Content-Type: text/plain << << 00 00 00 00 .... Yahoo tells us there's nothing to deliver. And that's it! Apart from a brief comment to say that some packets Yahoo does not like to receive via HTTP. For example, typically one might send an ISBACK packet after login, but the HTTP service provided by Yahoo spits this back as an error. There are other quirks and oddities, but further experimentation is left to the reader :-)