Proposed Accept-Encoding header filter change.

[Graycode]

(Dec. 30, 2010 03:15 PM)JJoe Wrote:  

(Dec. 30, 2010 08:31 AM)whenever Wrote:  

(Dec. 29, 2010 05:21 PM)JJoe Wrote:  However, Opera still sends x-gzip and I found an app that added support years after its creators considered x-gzip to be deprecated. Why?

Opera users suffer pain for its sending x-gzip.

Changed. Opera will likely argue that incorrectly configured servers aren't their problem.

The Opera browser has some unique quirks, among them is its indication of 'deflate' before 'gzip'. That part isn't new in version 11, it's caused occasional problems over many versions. The Opera developers love to scream "Standards" in your face but I've had more than one support thread disappear after asking for "common sense" to be applied or suggesting they don't comprehend a section of some RFC. I used to be a big supporter but more recently they've made that impossible.

(Dec. 30, 2010 03:15 PM)JJoe Wrote:  

(Dec. 30, 2010 08:31 AM)whenever Wrote:  

(Dec. 29, 2010 05:21 PM)JJoe Wrote:  Am also considering forcing gzip only when deflate and gzip are present and or throwing a flag for deflate compressed files.

Doesn't put gzip first fix the issue?

Not necessarily. I think the server can still choose to send deflate.
"gzip,x-gzip,deflate" is supposed to mean 'use one of these, gzip is best choice"

Yes, the server can still choose deflate in that situation. But most seem to use gzip, especially when it's listed first. Unfortunately there isn't a way to tell the server whether to use RFC1950 or RFV1951 deflate.

Note too that 'x-gzip' is just a synonym for 'gzip'. In general, it seems servers haven't bothered with it because most browsers (other than Opera) don't mention it in their 'Accept-Encoding'.

The problem site that triggered this thread (www-31.ibm.com/ etc) encoded some data using RFC1950 deflate specification when used by Opera. Other browsers aren't impacted because the server picks up on their having 'gzip' first. IE and other Microsoft products do not support the RFC1950 deflate format but they do accommodate the RFC1951 format.

As you mentioned in the other thread, people had apparently overcome some 'deflate' decompression problems by looking for a specific 2-byte prefix value. The technical details are in section 2.2 of RFC1950. When a server uses a 32K window size for deflation, and if there is no preset dictionary, and if it uses a common default compression level, then the 1st 2 bytes will be 0x78 and 0x9C. So, when encountering 'deflate' decompression issues with software like IE, some people decided to look for those particular 2 bytes and skip over them as a way to emulate RFC1951 raw inflation. That seems to be what Scott may have done with Proxo. One thing I don't understand though is what they'd do for the trailing CRC bytes that RFC1950 has but RFC1951 doesn't.
http://connect.microsoft.com/VisualStudi...df-streams
http://www.subbu.org/blog/2008/03/ie7-deflate-or-not

Well the particular IBM server in question uses RFC1950 but with a 4K compression window instead of the more common 32K window. Its first 2 bytes come out 0x48 and 0xC7, so "These aren't the droids you're looking for" and Proxo's decompression failed. IMHO that's not necessarily Proxo's fault since IE and other apps would also fail to decompress it if presented with that situation. I see it as more Opera's fault due to its less-than-stellar choice of header values, or somewhat the server's fault for using a compression method that even everyone's IE couldn't decypher.

A side effect of a compressed stream is that they can aid in detecting the end of a server's response. When the server does not use 'chunked' and does not provide an explicit 'Content-Length', then the decompression process can provide an end-of-content signal for the stream. That can be important when there's full-bore pipelining (as only Opera does by default). That's the reason I question what happens to the 4 bytes of CRC on the tail when using a special 2-byte prefix to detect RFC1950, dropping those bytes, and then treating the remaining stream as RFC1951.

(Dec. 30, 2010 03:15 PM)JJoe Wrote:  The defaults shouldn't break anything important.

Probably not, yet it may only be useful on rare occasions. The issue identified involves a quirky browser like Opera, a server that's willing to choose 'deflate' over 'gzip' compression, and for that server to have chosen RFC1950 vs. RFC1951 deflate format, and for that server to use a compression window that doesn't yield the magic 2-byte prefix detection that's apparently already in Proxomitron.