Post Reply 
ProxHTTPSProxyMII: Development
Dec. 30, 2014, 09:25 PM (This post was last modified: Dec. 30, 2014 09:25 PM by JJoe.)
Post: #151
RE: ProxHTTPSProxyMII: Reloaded
(Dec. 30, 2014 04:34 PM)GunGunGun Wrote:  This may apply to Proxomitron too, because I think Proxomitron also decompress gzip before apply filter.

With the Proxomitron or a direct connection, I get no reply or an empty file. I suspect this is due to an incorrect referer header. Can we have the url of the parent page?

I don't have Privoxy here (or much Privoxy experience).
Add Thank You Quote this message in a reply
Dec. 31, 2014, 04:30 AM
Post: #152
RE: ProxHTTPSProxyMII: Reloaded
(Dec. 30, 2014 09:25 PM)JJoe Wrote:  
(Dec. 30, 2014 04:34 PM)GunGunGun Wrote:  This may apply to Proxomitron too, because I think Proxomitron also decompress gzip before apply filter.

With the Proxomitron or a direct connection, I get no reply or an empty file. I suspect this is due to an incorrect referer header. Can we have the url of the parent page?

I don't have Privoxy here (or much Privoxy experience).

Yes, that file is an empty file, but its original file size is 32bytes and fill up with space character. But If I use Privoxy and MITM 1.2, the file's file size after decode gzip is 1bytes only.
Add Thank You Quote this message in a reply
Jan. 04, 2015, 02:51 AM
Post: #153
RE: ProxHTTPSProxyMII: Reloaded
(Dec. 31, 2014 04:30 AM)GunGunGun Wrote:  Yes, that file is an empty file, but its original file size is 32bytes and fill up with space character. But If I use Privoxy and MITM 1.2, the file's file size after decode gzip is 1bytes only.

This is my Privoxy output with ProxHTTPSProxyMII 1.2:

Code:
2015-01-04 10:24:07.192 0000135c Request: www.blogger.com/dyn-css/authorization.css?targetBlogID=14267103&zx=96fd4534-c0b9-42d2-acbc-ccfe659f6435
2015-01-04 10:24:07.472 0000135c Re-Filter: Need to de-chunk first
2015-01-04 10:24:07.472 0000135c Re-Filter: De-chunking successful. Shrunk from 32 to 21
2015-01-04 10:24:07.485 0000135c Re-Filter: Decompression successful. Old size: 21, new size: 1.
2015-01-04 10:24:07.485 0000135c Re-Filter: filtering www.blogger.com/dyn-css/authorization.css?targetBlogID=14267103&zx=96fd4534-c0b9-42d2-acbc-ccfe659f6435 (size 1) with '123' produced 0 hits (new size 1).

It's the Privoxy who does the decompress. It doesn't seem to be a ProxHTTPSProxyMII problem.

The strange thing is if I change to ProxHTTPSProxyMII 1.1 which doesn't stream, Privoxy got more bytes.

Code:
2015-01-04 10:39:44.681 00001b14 Request: www.blogger.com/dyn-css/authorization.css?targetBlogID=14267103&zx=96fd4534-c0b9-42d2-acbc-ccfe659f6435
2015-01-04 10:39:44.985 00001b14 Re-Filter: Decompression successful. Old size: 250, new size: 297.
2015-01-04 10:39:44.986 00001b14 Re-Filter: filtering www.blogger.com/dyn-css/authorization.css?targetBlogID=14267103&zx=96fd4534-c0b9-42d2-acbc-ccfe659f6435 (size 297) with '123' produced 0 hits (new size 297).
Add Thank You Quote this message in a reply
Jan. 10, 2015, 09:31 AM (This post was last modified: Jan. 10, 2015 02:34 PM by GunGunGun.)
Post: #154
RE: ProxHTTPSProxyMII: Reloaded
(Jan. 04, 2015 02:51 AM)whenever Wrote:  
(Dec. 31, 2014 04:30 AM)GunGunGun Wrote:  Yes, that file is an empty file, but its original file size is 32bytes and fill up with space character. But If I use Privoxy and MITM 1.2, the file's file size after decode gzip is 1bytes only.

This is my Privoxy output with ProxHTTPSProxyMII 1.2:

Code:
2015-01-04 10:24:07.192 0000135c Request: www.blogger.com/dyn-css/authorization.css?targetBlogID=14267103&zx=96fd4534-c0b9-42d2-acbc-ccfe659f6435
2015-01-04 10:24:07.472 0000135c Re-Filter: Need to de-chunk first
2015-01-04 10:24:07.472 0000135c Re-Filter: De-chunking successful. Shrunk from 32 to 21
2015-01-04 10:24:07.485 0000135c Re-Filter: Decompression successful. Old size: 21, new size: 1.
2015-01-04 10:24:07.485 0000135c Re-Filter: filtering www.blogger.com/dyn-css/authorization.css?targetBlogID=14267103&zx=96fd4534-c0b9-42d2-acbc-ccfe659f6435 (size 1) with '123' produced 0 hits (new size 1).

It's the Privoxy who does the decompress. It doesn't seem to be a ProxHTTPSProxyMII problem.

The strange thing is if I change to ProxHTTPSProxyMII 1.1 which doesn't stream, Privoxy got more bytes.

Code:
2015-01-04 10:39:44.681 00001b14 Request: www.blogger.com/dyn-css/authorization.css?targetBlogID=14267103&zx=96fd4534-c0b9-42d2-acbc-ccfe659f6435
2015-01-04 10:39:44.985 00001b14 Re-Filter: Decompression successful. Old size: 250, new size: 297.
2015-01-04 10:39:44.986 00001b14 Re-Filter: filtering www.blogger.com/dyn-css/authorization.css?targetBlogID=14267103&zx=96fd4534-c0b9-42d2-acbc-ccfe659f6435 (size 297) with '123' produced 0 hits (new size 297).

I think you can fix that problem by using the old method (fully download and cache and then forward to browser/proxy server) for text file, clearly we can use mimetype (text/css, text/html) to detect text file or just use RegEx something like /.*?(?:\.css|\.js). But Mimetype always better, you can detect Mimetype using Content-Type header or maybe Python will have better method.

That problem is serious because it can make page stop loading, but seem it apply only to css, js, html, text file in general but not video, swf so I think will need a fix.

Also I hope we will have MimeType support for [BYPASS URL], I think better then use Mimetype video/ instead *.mp4, *.avi.. and much reliable.


And can you add a option to disable logging ? I think disable logging will improve performance.

Yeah, I think Privoxy report more bytes because the file get decompressed, so I think no problem. Gzip is a compression method like rar, zip, so probably after content get decompressed it must increase file size.
Add Thank You Quote this message in a reply
Jan. 10, 2015, 04:50 PM
Post: #155
RE: ProxHTTPSProxyMII: Reloaded
(Dec. 31, 2014 04:30 AM)GunGunGun Wrote:  its original file size is 32bytes and fill up with space character.

Why am I unable to find this file?
My Internet Explorer and Opera without any proxies (a direct connection) get a file that contains only a line feed.

What do you see with a direct connection?

https://www.blogger.com/dyn-css/authoriz...fe659f6435

(Dec. 31, 2014 04:30 AM)GunGunGun Wrote:  
(Dec. 30, 2014 09:25 PM)JJoe Wrote:  
(Dec. 30, 2014 04:34 PM)GunGunGun Wrote:  This may apply to Proxomitron too, because I think Proxomitron also decompress gzip before apply filter.

With the Proxomitron or a direct connection, I get no reply or an empty file. I suspect this is due to an incorrect referer header. Can we have the url of the parent page?

I don't have Privoxy here (or much Privoxy experience).

Yes, that file is an empty file, but its original file size is 32bytes and fill up with space character. But If I use Privoxy and MITM 1.2, the file's file size after decode gzip is 1bytes only.
Add Thank You Quote this message in a reply
Jan. 10, 2015, 05:56 PM (This post was last modified: Jan. 11, 2015 08:27 AM by whenever.)
Post: #156
RE: ProxHTTPSProxyMII: Reloaded
Quote: What do you see with a direct connection?

I'm sure, without any kind of proxy I get a blank file.
Add Thank You Quote this message in a reply
Jan. 11, 2015, 08:26 AM
Post: #157
RE: ProxHTTPSProxyMII: Development
(Jan. 10, 2015 09:31 AM)GunGunGun Wrote:  That problem is serious because it can make page stop loading

Could you provide more examples?

(Jan. 10, 2015 09:31 AM)GunGunGun Wrote:  Also I hope we will have MimeType support for [BYPASS URL], I think better then use Mimetype video/ instead *.mp4, *.avi.. and much reliable.

Mime type is available only after a request is made and responded. It would add another layer before the real request is made, which I don't think it as a good idea.

(Jan. 10, 2015 09:31 AM)GunGunGun Wrote:  And can you add a option to disable logging ? I think disable logging will improve performance.

It won't improve much.

(Jan. 10, 2015 09:31 AM)GunGunGun Wrote:  Yeah, I think Privoxy report more bytes because the file get decompressed

I mean the bytes before decompressed. "Old size: 21" vs "Old size: 250".
Add Thank You Quote this message in a reply
Jan. 11, 2015, 10:38 AM (This post was last modified: Jan. 14, 2015 08:10 AM by GunGunGun.)
Post: #158
RE: ProxHTTPSProxyMII: Development
Quote: Could you provide more examples?

Can you try this page and open Developer Tools -> Network, I download it first time ok, but when I navigate to Address bar and press Enter to reload page, the page cannot show because that css cannot get downloaded.
http://siderite.blogspot.com/2013/05/adb...ction.html

Here is another example like the css above: https://translate.googleapis.com/transla...c=2&mode=1

This link will download a file named f.txt, but if we filter it using Privoxy, it take forever, but just disable filter on it or disable Privoxy, everything ok again. It is a component of Google Translate API from Google Chrome.

And I rarely see url downloaded with full file size since I've updated to 1.2:
[Image: SLdHuRq.png]
[Image: cxwZXpL.png]

200 - , and 404 - without file size. Still a few requests contain file size but it is rare.

Another solution is we can detect if response is gzip or not by checking the Content-Encoding header, if not then stream, if gzip then old method 1.1.
Yeah streaming method is really nice, it solve zip, rar, youtube videoplayback problem but seem Privoxy always decompress content to filter/block it as soon as possible (if we keep gzip and download the whole file, we cannot filter it and send it to browser), so I think not Privoxy problem.

Quote: Mime type is available only after a request is made and responded. It would add another layer before the real request is made, which I don't think it as a good idea.
Yeah, I think so, but there is file that lack extension in url example:

http://example.com/image.php=1231231231

This is an image, sure, there is a lot of image like that (if you use VBulletin 4 forum, user's avatar using this technology, and clearly they can use this technology with video, all other format like Youtube with /videoplayback and a bunch of text after but it is video), but extension is PHP because there is server side technology, I think RegEx support for [BYPASS URL] is really needed in this case.
Add Thank You Quote this message in a reply
Jan. 14, 2015, 09:27 AM
Post: #159
RE: ProxHTTPSProxyMII: Development
(Jan. 11, 2015 10:38 AM)GunGunGun Wrote:  This link will download a file named f.txt, but if we filter it using Privoxy, it take forever, but just disable filter on it or disable Privoxy, everything ok again.

Doesn't this seem to be a Privoxy issue?

Privoxy is not bug free. On mail.live.com with web messenger enabled (background ping), I get many connection abort warning in the ProxHTTPSProxyMII log window with Privoxy. They all are gone once I switch to Proxomitron.

(Jan. 11, 2015 10:38 AM)GunGunGun Wrote:  And I rarely see url downloaded with full file size since I've updated to 1.2:

Versions prior to 1.2 cache the content so file size is always available. 1.2 does stream and it takes the value from "Content-Length" header which is not always available. The missing of the value doesn't mean you will get the file broken.

(Jan. 11, 2015 10:38 AM)GunGunGun Wrote:  Another solution is we can detect if response is gzip or not by checking the Content-Encoding header, if not then stream, if gzip then old method 1.1.

Stream or not doesn't change the content encoding at all, so what's the point?

(Jan. 11, 2015 10:38 AM)GunGunGun Wrote:  but seem Privoxy always decompress content to filter/block it as soon as possible (if we keep gzip and download the whole file, we cannot filter it and send it to browser), so I think not Privoxy problem.

I'm curious how did you get that conclusion.

(Jan. 11, 2015 10:38 AM)GunGunGun Wrote:  I think RegEx support for [BYPASS URL] is really needed in this case.

It supports wildcards already, which I think is basically enough. Even if not, both Privoxy and Proxomitron can bypass URL based on regex like syntax too.
Add Thank You Quote this message in a reply
Jan. 14, 2015, 05:59 PM (This post was last modified: Jan. 14, 2015 06:00 PM by JJoe.)
Post: #160
RE: ProxHTTPSProxyMII: Development
(Jan. 14, 2015 09:27 AM)whenever Wrote:  
(Jan. 11, 2015 10:38 AM)GunGunGun Wrote:  This link will download a file named f.txt, but if we filter it using Privoxy, it take forever, but just disable filter on it or disable Privoxy, everything ok again.

Doesn't this seem to be a Privoxy issue?

Privoxy is not bug free.

Privoxy has had problems with chunked content in the past, http://www.google.com/search?q=privoxy+chunked .

There was a fix mentioned but I wonder if it considered very small or empty files.

Also found this bug report, http://sourceforge.net/p/ijbswa/bugs/707/ , from 2007-11-07 that was closed due to lack of response.

Michel Julier Wrote:Privoxy seems to block a website whose pages are sent with "Transfer-Encoding: chunked" in the headers.
Dialog between Mozilla and the server (summarized), without using Privoxy:
> GET / HTTP/1.1
< HTTP/1.1 302 Moved Temporarily
< Location: /admin/
< Transfer-Encoding: chunked
> GET /admin/ HTTP/1.1
When using Privoxy, the last request does not happen, instead the web browser stays still, waiting for something.

http://sourceforge.net/p/ijbswa/bugs/707/#ddd3

Fabian Keil Wrote:...
The problem seems to be that
http://courrier.net4all.ch/ doesn't
follow the HTTP/1.1 standard and ignores
Privoxy's request to close the connection
when the request has been fulfilled.
When filtering is enabled, Privoxy currently
buffers everything it gets until the server
closes the connection to signal that it's done.
Only then Privoxy executes the actual filters
and delivers the content.
If the server is broken and ignores Privoxy's
"Connection: close" header, Privoxy continues
buffering until the connection eventually times
out and gets closed by the server. This may take
several minutes and requires some patience ...
Nowadays most browsers support connection
keep-alive, which explains why the site works
for you without Privoxy.
You can work around the problem with:
{+downgrade-http-version \ }
courrier.net4all.ch/
I assume disabling all content filters would
work as well, but I didn't test that.
Add Thank You Quote this message in a reply
Jan. 15, 2015, 01:47 AM
Post: #161
RE: ProxHTTPSProxyMII: Development
That's year 2007 and Privoxy didn't do HTTP/1.1 right at that time: HTTP/1.1 doesn't need to close the connection to signal finish of the content transfer.

That should already be fixed over these many years.

Here comes the version 1.3 of ProxHTTPSProxyMII:

+ Each request has a number ranged from 001 to 999 for reference. 000 is reserved for SSL requests not MITMed like those in [BLACKLIST] and [SSL Pass-Thru] sections.
+ Log window now displays the length of the bytes submitted in POST method


Attached File(s)
.zip  ProxHTTPSProxyMII_py 1.3.zip (Size: 210.19 KB / Downloads: 870)
Add Thank You Quote this message in a reply
Jan. 15, 2015, 07:36 AM (This post was last modified: Jan. 15, 2015 07:48 AM by GunGunGun.)
Post: #162
RE: ProxHTTPSProxyMII: Development
I fixed ProxHTTPSProxy 1.1 to make it reloads config correctly and tweak a little bit like 1.2+, I fix it because combine with Privoxy, I cannot load a few sites, example: http://siderite.blogspot.com/2013/05/adb...ction.html (the authorization.css take forever to load, I cannot believe, just a css can make the whole page freeze, in my knowledge, only js can make site unloadable).

Code:
http://tny.cz/1afebb15
Add Thank You Quote this message in a reply
Jan. 15, 2015, 10:05 AM
Post: #163
RE: ProxHTTPSProxyMII: Development
Great! That's the benefit to open source the code.

However, isn't it a better way to just bypass the problematical URL, either in ProxHTTPSProxyMII or in Privoxy?
Add Thank You Quote this message in a reply
Jan. 15, 2015, 01:22 PM (This post was last modified: Jan. 15, 2015 04:41 PM by GunGunGun.)
Post: #164
RE: ProxHTTPSProxyMII: Development
Quote: However, isn't it a better way to just bypass the problematical URL, either in ProxHTTPSProxyMII or in Privoxy?

I always want to filter as much content as possible, but this problem break about 3-4 sites that I always visit (unloadable). And bypass in Privoxy seem impossible, because ProxHTTPSProxy is parent proxy and Privoxy is just child that need ProxHTTPSProxy to work, maybe I'm not smart enough to figure but I tried many times already.

Im going to add proxy authentication and redirect feature to ProxHTTPSProxy, do you have any advice for me, maybe I should learn something from Another Proxy.

The reason is I want to use ProxHTTPSProxy with my private proxy on some site, if I use my proxy with Privoxy, it do nothing because ProxHTTPS is the parent proxy, I think I will use Proxy-Authorization header with a base64 encode function to encode username and password to base64, and I also want to add redirect feature mainly because I want to redirect https to http because there are sites that use both https and http like Facebook static-a, external.. and some Google service... Location header is a good choice.

Found the website: https://urllib3.readthedocs.org/en/latest/helpers.html
I will learn Python because I still want to write my own video accelerator.
Add Thank You Quote this message in a reply
Jan. 16, 2015, 02:38 AM (This post was last modified: Jan. 16, 2015 06:32 AM by whenever.)
Post: #165
RE: ProxHTTPSProxyMII: Development
(Jan. 15, 2015 01:22 PM)GunGunGun Wrote:  And bypass in Privoxy seem impossible

I haven't tried, but I think -filter should could be used to prevent some URLs from being filtered.

(Jan. 15, 2015 01:22 PM)GunGunGun Wrote:  Im going to add proxy authentication and redirect feature to ProxHTTPSProxy, do you have any advice for me, maybe I should learn something from Another Proxy.

Redirect is a 301/302 response with a Location header pointing to another URL. I had no experience with proxy authentication but it should not be hard.

(Jan. 15, 2015 01:22 PM)GunGunGun Wrote:  I will learn Python because I still want to write my own video accelerator.

Great! It's like that I learn Python to write ProxHTTPSProxy. You may also look at Go, which is very promising!
Add Thank You Quote this message in a reply
Post Reply 


Forum Jump: