Post Reply 
ProxHTTPSProxyMII: Reloaded
Dec. 07, 2018, 03:40 PM
Post: #301
RE: ProxHTTPSProxyMII: Reloaded
SOLVED!

I've made some progress, after my "random thought" last night... Using the Dev Tools panel in Firefox, I loaded https://www.google.com. Here are the request headers and the Privoxy results log:

Code:
Host: www.google.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Pragma: no-cache
Cache-Control: no-cache

2018-12-07 10:12:40.177 00002080 Request: www.google.com/
2018-12-07 10:12:40.323 00002080 Re-Filter: Need to de-chunk first
2018-12-07 10:12:40.330 00002080 Re-Filter: De-chunking successful. Shrunk from 63991 to 63978
2018-12-07 10:12:40.337 00002080 Re-Filter: filtering www.google.com/ (size 63978) with 'removeanchortargets' produced 0 hits (new size 63978).
2018-12-07 10:12:40.343 00002080 Re-Filter: filtering www.google.com/ (size 63978) with 'removeemptyalts' produced 0 hits (new size 63978).
2018-12-07 10:12:40.348 00002080 Re-Filter: filtering www.google.com/ (size 63978) with 'TEST' produced 0 hits (new size 63978).

Note that none of the three filters had any matches.

I then made exactly the same request again, with one alteration to the headers:

Code:
Host: www.google.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
[b]Accept-Encoding: gzip[/b]
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
Pragma: no-cache
Cache-Control: no-cache

2018-12-07 10:13:17.670 00002080 Request: www.google.com/
2018-12-07 10:13:17.815 00002080 Re-Filter: Need to de-chunk first
2018-12-07 10:13:17.823 00002080 Re-Filter: De-chunking successful. Shrunk from 69356 to 69335
2018-12-07 10:13:17.834 00002080 Re-Filter: Decompression successful. Old size: 69335, new size: 219348.
2018-12-07 10:13:17.854 00002080 Re-Filter: filtering www.google.com/ (size 219348) with 'removeanchortargets' produced 1 hits (new size 219333).
2018-12-07 10:13:17.859 00002080 Re-Filter: filtering www.google.com/ (size 219333) with 'removeemptyalts' produced 0 hits (new size 219333).
2018-12-07 10:13:17.866 00002080 Re-Filter: filtering www.google.com/ (size 219333) with 'TEST' produced 230 hits (new size 219793).

Now we see that two of the filters DID have matches, and the filtering worked! What changed? This "Decompression successful. Old size: 69335, new size: 219348" was not done in the first request, so Privoxy was trying to filter compressed content. Either deflate or br. With that in mind, I checked the options for Firefox, to see if it allows configurations of what encodings it supports, and found under about:config this option: "network.http.accept-encoding.secure;gzip, deflate, br". I removed "br" from the end, and now my HTTPS filters work!!

So I guess that Privoxy simply does not support Brotli encoding.
Add Thank You Quote this message in a reply
[-] The following 1 user says Thank You to Quaraxkad for this post:
referrer
Dec. 17, 2018, 08:08 PM
Post: #302
RE: ProxHTTPSProxyMII: Reloaded
Quote:Is it possible that Firefox and Chrome are requesting pages in a compression/encoding/whatever format that Privoxy can't understand?
"prevent-compression" in privoxy used?
Add Thank You Quote this message in a reply
[-] The following 1 user says Thank You to vlad_s for this post:
referrer
Dec. 18, 2018, 12:16 AM
Post: #303
RE: ProxHTTPSProxyMII: Reloaded
(Dec. 17, 2018 08:08 PM)vlad_s Wrote:  "prevent-compression" in privoxy used?

Do you mean "enable-compression"? That option is disabled by default, and as far as I know only sets whether or not data is compressed *by* Privoxy before sending it to the browser/client.
Add Thank You Quote this message in a reply
Dec. 18, 2018, 12:41 AM (This post was last modified: Dec. 18, 2018 12:52 AM by referrer.)
Post: #304
RE: ProxHTTPSProxyMII: Reloaded
'prevent-compression' can do the job.
Code:
8.5.30. prevent-compression
Typical use:

Ensure that servers send the content uncompressed, so it can be passed through filters.
Effect:

Removes the Accept-Encoding header which can be used to ask for compressed transfer.

Example usage (sections):# Selectively turn off compression, and enable a filter
#
{ +filter{tiny-textforms} +prevent-compression }
# Match only these sites
.google.
sourceforge.net
sf.net

# Or instead, we could set a universal default:
#
{ +prevent-compression }
/ # Match all sites

# Then maybe make exceptions for broken sites:
#
{ -prevent-compression }
.compusa.com/
Or add a header filter to remove just the 'br' part.
Code:
CLIENT-HEADER-FILTER: NO-BR Remove Accept-Encoding Brotli.
#www.google.com
#Header: scan: Accept-Encoding: gzip, deflate, br
[email protected]^(Accept-Encoding:).*@$1 gzip, [email protected]
{ +client-header-filter{NO-BR} }
.google.
Privoxy's TODO said the 'prevent-compression' action will be removed.
Add Thank You Quote this message in a reply
Dec. 18, 2018, 12:52 AM
Post: #305
RE: ProxHTTPSProxyMII: Reloaded
Ahh, I see. I have all of the default actions and filters disabled, I wasn't aware of that one. As soon as I recognized the problem, my alternative solution if Firefox didn't have that "network.http.accept-encoding.secure" option was going to be a client-header-filter to remove Brotli.
Add Thank You Quote this message in a reply
Dec. 19, 2018, 02:36 AM
Post: #306
RE: ProxHTTPSProxyMII: Reloaded
I hope I'm not the only one having this issue. Its nothing breaking fortunately, just very disruptive. After awhile of browsing, there will be moments where loading a site will temporarily stop, as if it hit a snag or something. Trying to reload or access any other sight will yield the same result, the browser is trying to load it but is unable to as if your internet connection was being disrupted. However, once you click abort all active connections via proxomitron, any website that is stuck loading will suddenly snap out of it and will proceed to finish.

Its as if you are giving proxomitron a defibrillation when it stops as you hit the abort connections button. This is something that has to be done almost every couple of minutes of regular web browsing, and as I'm sure you can imagine it gets pretty frustrating.

Is this a known problem? I hope I was clear with my explanation.
Add Thank You Quote this message in a reply
Dec. 19, 2018, 05:13 AM
Post: #307
RE: ProxHTTPSProxyMII: Reloaded
(Dec. 19, 2018 02:36 AM)Hydl Wrote:  I hope I'm not the only one having this issue. Its nothing breaking fortunately, just very disruptive. After awhile of browsing, there will be moments where loading a site will temporarily stop, as if it hit a snag or something. Trying to reload or access any other sight will yield the same result, the browser is trying to load it but is unable to as if your internet connection was being disrupted. However, once you click abort all active connections via proxomitron, any website that is stuck loading will suddenly snap out of it and will proceed to finish.

Have this issue too.
It seems sometimes proxhttpsproxy didn't close the connection.
In proxifier connections window I find many connections keep by proxhttpsproxy.
Proxomitron's abort button cannot close all of them.
When this happens I always use the 'reload' button in 'proxhttpsproxy Launcher'.
Add Thank You Quote this message in a reply
Dec. 19, 2018, 12:41 PM
Post: #308
RE: ProxHTTPSProxyMII: Reloaded
What Proxomitron version are you using? Do you have a multiple-core machine? Even the latest one Scott released contains a few multithreading bugs which would not have been noticeable on the single-core machines widely used in his time. You can try setting Proxomitron's affinity to a single core (right-click in Task Manager to select Set Affinity) and see if the problem disappears or reduces in frequency.

You may also want to give Proxomitron Reborn a try, either behind ProxHTTPSProxy or using its own SSL filtering capabilities.
Add Thank You Quote this message in a reply
Dec. 20, 2018, 01:08 AM (This post was last modified: Dec. 20, 2018 03:06 AM by Hydl.)
Post: #309
RE: ProxHTTPSProxyMII: Reloaded
Hello Amy. I am using version Naoko 4.5 (2003-6-1). I do have a multi core CPU and after setting both Proxomitron and ProxHTTPSProxyMII to use one thread only, the problem as you predicted reduces in frequency. I am able to browse a lot more but will eventually need to abort all connections.

I think its terrific that you are developing Proxomitron Reborn, and I wish you all the best with it. Though unfortunately the problem persists when using it. Again its not as frequent, but it still happens.
Add Thank You Quote this message in a reply
Dec. 22, 2018, 09:54 AM
Post: #310
RE: ProxHTTPSProxyMII: Reloaded
(Dec. 18, 2018 12:41 AM)referrer Wrote:  Privoxy's TODO said the 'prevent-compression' action will be removed.
It is a pity that they cut such functionality. In general, with the dominance of https-sites, developers should think about their wonderful proxy, which will lose more and more unique opportunities.
Add Thank You Quote this message in a reply
Dec. 23, 2018, 09:25 PM (This post was last modified: Dec. 23, 2018 09:26 PM by vlad_s.)
Post: #311
RE: ProxHTTPSProxyMII: Reloaded
Error log:
Code:
python3[3481]: 127.0.0.1 - - [24/Dec/2018 00:13:48] code 400, message Bad request syntax ('CNT 1 CON 260')
python3[3481]: 127.0.0.1 - - [24/Dec/2018 00:13:48] "CNT 1 CON 260" 400 -
Appears when viewing the site cont.ws, specifically this:
Code:
https://coms-bwxszt8x.cont.ws/socket.io/?EIO=3&transport=websocket
Forced to add to [SSL Pass-Thru]
Add Thank You Quote this message in a reply
Dec. 24, 2018, 04:47 AM
Post: #312
RE: ProxHTTPSProxyMII: Reloaded
^ That's MSNP, not HTTP. No wonder it's confused. Not everything that establishes an SSL tunnel through an HTTP proxy with CONNECT is going to send HTTP through that tunnel.
Add Thank You Quote this message in a reply
Jan. 04, 2019, 10:41 PM (This post was last modified: Jan. 04, 2019 10:42 PM by vlad_s.)
Post: #313
RE: ProxHTTPSProxyMII: Reloaded
After updating Ubuntu 16.04 to Ubuntu 18.04, a message is constantly being sent in the syslog:
Code:
----------------------------------------
python3[3340]: Exception happened during processing of request from ('127.0.0.1', 50838)
python3[3340]: Traceback (most recent call last):
python3[3340]:   File "/usr/lib/python3.6/socketserver.py", line 651, in process_request_thread
python3[3340]:     self.finish_request(request, client_address)
python3[3340]:   File "/usr/lib/python3.6/socketserver.py", line 361, in finish_request
python3[3340]:     self.RequestHandlerClass(request, client_address, self)
python3[3340]:   File "/usr/lib/python3.6/socketserver.py", line 721, in __init__
python3[3340]:     self.handle()
python3[3340]:   File "/usr/lib/python3.6/http/server.py", line 418, in handle
python3[3340]:     self.handle_one_request()
python3[3340]:   File "/opt/ProxHTTPSProxyMII/ProxyTool.py", line 115, in handle_one_request
python3[3340]:     BaseHTTPRequestHandler.handle_one_request(self)
python3[3340]:   File "/usr/lib/python3.6/http/server.py", line 406, in handle_one_request
python3[3340]:     method()
python3[3340]:   File "/opt/ProxHTTPSProxyMII/ProxHTTPSProxy.py", line 171, in do_CONNECT
python3[3340]:     ssl_sock = ssl.wrap_socket(self.connection, keyfile=dummycert, certfile=dummycert
python3[3340]:   File "/usr/lib/python3.6/ssl.py", line 1158, in wrap_socket
python3[3340]:     ciphers=ciphers)
python3[3340]:   File "/usr/lib/python3.6/ssl.py", line 817, in __init__
python3[3340]:     self.do_handshake()
python3[3340]:   File "/usr/lib/python3.6/ssl.py", line 1077, in do_handshake
python3[3340]:     self._sslobj.do_handshake()
python3[3340]:   File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
python3[3340]:     self._sslobj.do_handshake()
python3[3340]: OSError: [Errno 0] Error
python3[3340]: ----------------------------------------
Everything seems to be working. But the messages are straining, because go constantly.
Add Thank You Quote this message in a reply
Jan. 12, 2019, 02:40 PM (This post was last modified: Jan. 12, 2019 02:43 PM by vlad_s.)
Post: #314
RE: ProxHTTPSProxyMII: Reloaded
I use the decentraleyes option from here. Redirection is done on my own web server (apache) via https. So that there are no unnecessary questions, I'm in [SSL No-Verify]
added it as 192.168.2.1. Everything was fine until the upgrade to Ubuntu 18.04. After the upgrade, I will get an error for my server:
Code:
502: HTTPError

The following error occurred while trying to access https://192.168.2.1/

HTTPSConnectionPool(host='192.168.2.1', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)'),))
At the time I removed 192.168.2.1 from [SSL No-Verify] and I see an error that the browser usually reports that the site is not reliable, that it is not properly configured. Such a site can be added to the exception. After that, the site works, so my site on HTTPS is working fine, because page after that opened. What is the reason that my site added to [SSL No-Verify] stopped working?
Add Thank You Quote this message in a reply
Jan. 12, 2019, 03:10 PM
Post: #315
RE: ProxHTTPSProxyMII: Reloaded
The problem was in the version urllib3, version 1.22. At 1.20 and the last which is available for me 1.24.1 this error is not present.
Add Thank You Quote this message in a reply
Post Reply 


Forum Jump: