ProxHTTPSProxyMII: Reloaded

[vlad_s]

(May. 07, 2019 04:27 PM)JJoe Wrote:  Something (firewall, anti-virus, Privoxy, etc.) is blocking POST requests.

No, this does not always happen. For example, with startpage.com, I just once again perform in the browser "refresh" and everything is fine.

[JJoe]

(May. 07, 2019 07:11 PM)vlad_s Wrote:  For example, with startpage.com, I just once again perform in the browser "refresh" and everything is fine.

I have made this happen.

(May. 06, 2019 04:08 PM)vlad_s Wrote:  [18:19] 661 [F] ('Connection aborted.', ResponseNotReady('Request-sent',)) on "POST http://startpage.com/do/metasearch.pl"

http://startpage.com/do/metasearch.pl does not exist.
By now, your browser should automatically change this request to https.

Are you removing or modifying HTTP Strict-Transport-Security response headers or values, https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security?

Does http://startpage.com/ return a redirect to https or go directly to https://startpage.com/?

[pepak]

(May. 07, 2019 04:27 PM)JJoe Wrote:  Are you having this problem?

Yes. Usually if I am too fast with the next POST. With some sites it occurs more often that othes, e.g. logging in to https://forum.delphi.cz/ frequently leads to the error. Unfortunately, I wasn't able to figure out the cause, all the leads I had turned out to be false :-(

If someone wants to give it a try, I am attaching the file that I was trying to use. Just import it into the main script.

[vlad_s]

JJoe
startpage.com works fine. The error occurs once when viewing several (more than 10) pages in this search engine. Sometimes, when trying to find selected text through the right mouse button, a page with an error opens, instead of startpage.com. If you just press F5 in the browser, then the problem is usually solved. Most often, a problem arises when POST requests, for example, on the response page in a forum, when it is sent, this error may also occur, but not at another time. I generally do not exactly believe that the error is due to ProxHTTPSProxy. Here, too, there is a button "Post Reply", but here I have not yet seen the error. I think we have different reasons for the error with pepak.

[vlad_s]

Here is a typical error:
ProxHTTPSProxyMII:

Code:

[18:54] 835 ProxHTTPSProxyMII RearProxy/v1.5 [Errno 32] Broken pipe
[18:54] 837 ProxHTTPSProxyMII RearProxy/v1.5 [Errno 32] Broken pipe
[18:54] 839 ProxHTTPSProxyMII RearProxy/v1.5 [Errno 32] Broken pipe
[18:55] 841 [F] ('Connection aborted.', ResponseNotReady('Request-sent',)) on "POST http://nnmclub.to/forum/posting.php"
[18:55] 841 ProxHTTPSProxyMII RearProxy/v1.5 [Errno 32] Broken pipe
[18:57] 846 ProxHTTPSProxyMII RearProxy/v1.5 [Errno 32] Broken pipe
[18:57] 847 ProxHTTPSProxyMII RearProxy/v1.5 [Errno 32] Broken pipe
[18:57] 848 [F] ('Connection aborted.', ResponseNotReady('Request-sent',)) on "POST http://nnmclub.to/forum/posting.php"
[18:57] 848 ProxHTTPSProxyMII RearProxy/v1.5 [Errno 32] Broken pipe
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 53830)
Traceback (most recent call last):
  File "/usr/lib/python3.6/socketserver.py", line 651, in process_request_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python3.6/socketserver.py", line 361, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python3.6/socketserver.py", line 721, in __init__
    self.handle()
  File "/usr/lib/python3.6/http/server.py", line 418, in handle
    self.handle_one_request()
  File "/opt/ProxHTTPSProxyMII/ProxyTool.py", line 115, in handle_one_request
    BaseHTTPRequestHandler.handle_one_request(self)
  File "/usr/lib/python3.6/http/server.py", line 406, in handle_one_request
    method()
  File "/opt/ProxHTTPSProxyMII/ProxHTTPSProxy.py", line 171, in do_CONNECT
    ssl_sock = ssl.wrap_socket(self.connection, keyfile=dummycert, certfile=dummycert, server_side=True)
  File "/usr/lib/python3.6/ssl.py", line 1158, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
OSError: [Errno 0] Error
----------------------------------------

Privoxy (debug 1024)

Code:

2019-05-08 18:50:12.506 7fd0d2195700 Crunch: Blocked: http://counter.yadro.ru/hit?t52.10;rhttps%3A//nnmclub.to/forum/viewtopic.php%3Ft%3D1300644%26start%3D
2019-05-08 18:50:56.056 7fd0b2ffd700 Crunch: Forwarding failed: http://tracker.tfile.co/scrape.php?uk=LIryP9C276&&info_hash=
2019-05-08 18:53:25.894 7fd0d45bb700 Crunch: No server data received: http://open.touki.ru/scrape.php?info_hash=
2019-05-08 18:55:39.161 7fd0d2996700 Crunch: No server data received: http://open.touki.ru/announce.php?info_hash=
2019-05-08 18:55:59.442 7fd0b2ffd700 Crunch: No server data received: http://open.touki.ru/announce.php?info_hash=
2019-05-08 18:56:25.766 7fd0b37fe700 Crunch: Connection failure: http://tracker.openbittorrent.com/scrape?info_hash=
2019-05-08 18:57:31.742 7fd0d2996700 Crunch: Forwarding failed: http://tracker.openbittorrent.com/announce?info_hash=
2019-05-08 18:57:35.526 7fd0d35b9700 Crunch: Blocked: http://yastatic.net/pcode/adfox/loader.js
2019-05-08 18:57:35.545 7fd0d35b9700 Crunch: Blocked: http://www.google-analytics.com/ga.js
2019-05-08 18:57:35.549 7fd0d1193700 Crunch: Blocked: http://mc.yandex.ru/metrika/watch.js
2019-05-08 18:57:36.062 7fd0d45bb700 Crunch: Blocked: http://mc.yandex.ru/metrika/watch.js
2019-05-08 18:57:36.141 7fd0d1193700 Crunch: Blocked: http://assets.ipv6.nnm-club.ws/forum/images/spacer.gif
2019-05-08 18:57:36.316 7fd0d2195700 Crunch: Blocked: http://www.google-analytics.com/ga.js
2019-05-08 18:57:36.357 7fd0d35b9700 Crunch: Blocked: http://ssl.google-analytics.com/ga.js
2019-05-08 18:57:36.696 7fd0d3dba700 Crunch: Redirected: http://ajax.googleapis.com/ajax/libs/jquery/1.5.2/jquery.min.js
2019-05-08 18:57:37.543 7fd0b1ffb700 Crunch: Blocked: http://counter.yadro.ru/hit?t52.10;rhttps%3A//nnmclub.to/forum/viewtopic.php%3Ft%3D1300644%26start%3D
2019-05-08 18:57:54.150 7fd0d1994700 Crunch: Blocked: http://yastatic.net/pcode/adfox/loader.js
2019-05-08 18:57:54.194 7fd0d1994700 Crunch: Blocked: http://assets.ipv6.nnm-club.ws/forum/images/spacer.gif
2019-05-08 18:57:54.200 7fd0b3fff700 Crunch: CGI Call: http://config.privoxy.org/send-banner?type=auto
2019-05-08 18:57:54.207 7fd0d1994700 Crunch: Blocked: http://yastatic.net/share2/share.js
2019-05-08 18:57:54.208 7fd0b1ffb700 Crunch: Blocked: http://mc.yandex.ru/metrika/watch.js
2019-05-08 18:57:54.210 7fd0b3fff700 Crunch: Blocked: http://www.google-analytics.com/ga.js
2019-05-08 18:57:54.474 7fd0b3fff700 Crunch: Blocked: http://yastatic.net/pcode/adfox/loader.js
2019-05-08 18:57:54.666 7fd0b1ffb700 Crunch: Blocked: http://mc.yandex.ru/metrika/watch.js
2019-05-08 18:57:54.669 7fd0b3fff700 Crunch: Blocked: http://yastatic.net/share2/share.js
2019-05-08 18:57:54.700 7fd0b1ffb700 Crunch: Blocked: http://www.google-analytics.com/ga.js
2019-05-08 18:57:54.724 7fd0b1ffb700 Crunch: Blocked: http://ssl.google-analytics.com/ga.js
2019-05-08 18:57:55.001 7fd0b1ffb700 Crunch: Blocked: http://assets.ipv6.nnm-club.ws/forum/images/avatars/19868724364bcd5b1715951.gif
2019-05-08 18:57:55.009 7fd0d1994700 Crunch: Blocked: http://assets.ipv6.nnm-club.ws/forum/images/avatars/19133439095c291f0a4b5b7.gif
2019-05-08 18:57:55.012 7fd0d3dba700 Crunch: Blocked: http://assets.ipv6.nnm-club.ws/forum/images/avatars/17991207484fb11db3404af.gif
2019-05-08 18:57:55.042 7fd0d3dba700 Crunch: Blocked: http://counter.yadro.ru/hit?t52.10;rhttps%3A//nnmclub.to/;s2048*1152*24;uhttps%3A//nnmclub.to/forum/v
2019-05-08 18:58:02.389 7fd0b3fff700 Crunch: Blocked: http://yastatic.net/pcode/adfox/loader.js
2019-05-08 18:58:02.410 7fd0b1ffb700 Crunch: Blocked: http://www.google-analytics.com/ga.js

As you can see, the request was not blocked. And there is nothing else there, because it's just a router. An error occurs not often, but steadily.

[vlad_s]

Here is a typical error:
ProxHTTPSProxyMII:

Code:

[18:54] 835 ProxHTTPSProxyMII RearProxy/v1.5 [Errno 32] Broken pipe
[18:54] 837 ProxHTTPSProxyMII RearProxy/v1.5 [Errno 32] Broken pipe
[18:54] 839 ProxHTTPSProxyMII RearProxy/v1.5 [Errno 32] Broken pipe
[18:55] 841 [F] ('Connection aborted.', ResponseNotReady('Request-sent',)) on "POST http://nnmclub.to/forum/posting.php"
[18:55] 841 ProxHTTPSProxyMII RearProxy/v1.5 [Errno 32] Broken pipe

Privoxy (debug 1024)

Code:

2019-05-08 18:50:12.506 7fd0d2195700 Crunch: Blocked: http://counter.yadro.ru/hit?t52.10;rhttps%3A//nnmclub.to/forum/viewtopic.php%3Ft%3D1300644%26start%3D
2019-05-08 18:50:56.056 7fd0b2ffd700 Crunch: Forwarding failed: http://tracker.tfile.co/scrape.php?uk=LIryP9C276&&info_hash=
2019-05-08 18:53:25.894 7fd0d45bb700 Crunch: No server data received: http://open.touki.ru/scrape.php?info_hash=
2019-05-08 18:55:39.161 7fd0d2996700 Crunch: No server data received: http://open.touki.ru/announce.php?info_hash=
2019-05-08 18:55:59.442 7fd0b2ffd700 Crunch: No server data received: http://open.touki.ru/announce.php?info_hash=
2019-05-08 18:56:25.766 7fd0b37fe700 Crunch: Connection failure: http://tracker.openbittorrent.com/scrape?info_hash=

As you can see, the request was not blocked. And there is nothing else there, because it's just a router. An error occurs not often, but steadily.
While trying to edit, even then an error occurred:
https://ibb.co/MGjTk9n

Code:

[19:05] 176 ProxHTTPSProxyMII RearProxy/v1.5 [Errno 32] Broken pipe
[19:09] 180 [F] ('Connection aborted.', ResponseNotReady('Request-sent',)) on "POST http://www.prxbx.com/forums/xmlhttp.php?action=edit_post&do=update_post
[19:09] 180 ProxHTTPSProxyMII RearProxy/v1.5 [Errno 32] Broken pipe
[19:10] 182 ProxHTTPSProxyMII RearProxy/v1.5 [Errno 32] Broken pipe

[vlad_s]

After updating urllib3 to the latest version (>= 1.25), I get an error for my local server:

Code:

[20:25] 014 [R][D] "GET https://192.168.2.1/" HTTPSConnectionPool(host='192.168.2.1', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)'),))
[20:25] 015 [R][D] "GET https://192.168.2.1/favicon.ico" HTTPSConnectionPool(host='192.168.2.1', port=443): Max retries exceeded with url: /favicon.ico (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:847)'),))

And it seems that in the config.ini file the [SSL No-Verify] section starts to be ignored, in which I just made an entry like 192.168.2.1:

Code:

[SSL No-Verify]
192.168.2.1

[rogern]

I have installed it again with privoxy just to see that in firefox as you can make individual adjustments before goes live.
SEC_ERROR_BAD_SIGNATURE.

As Vlad_S i have urlib 1.25 and proxHTTPS Proxy 1.4 ( i Belive) and running ubuntu 18.04.LTS
Privoxy is 3.0.28

But let say i visit youtube.com it redirect to https://youtube.com:8443/manage ?
The only aplication wich i know listen and use port https:8443 is unifi controller.(lighttpd)

And :8443/manage/site/(session)/dashboard is then dashboard for unifi controller.
I can ask if it is possible to change the port in Unifi Controller if that helps.

Code:

[12:21] 320 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[12:21] 321 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[12:21] 322 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[12:21] 323 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[12:21] 324 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[12:21] 325 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[12:23] 000 "[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:852)" while trying to establish local SSL tunnel for [raw.githubusercontent.com:443]

[JJoe]

(Jul. 10, 2019 01:07 PM)rogern Wrote:  As Vlad_S i have urlib 1.25 and proxHTTPS Proxy 1.4 ( i Belive) and running ubuntu 18.04.LTS
Privoxy is 3.0.28

I think, vlad_s is using v1.5, https://prxbx.com/forums/showthread.php?tid=2191 .

(Jul. 10, 2019 01:07 PM)rogern Wrote:  But let say i visit youtube.com it redirect to https://youtube.com:8443/manage ?
The only aplication wich i know listen and use port https:8443 is unifi controller.(lighttpd)

And :8443/manage/site/(session)/dashboard is then dashboard for unifi controller.
I can ask if it is possible to change the port in Unifi Controller if that helps.

Code:

[12:21] 320 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[12:21] 321 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[12:21] 322 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[12:21] 323 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[12:21] 324 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[12:21] 325 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[12:23] 000 "[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:852)" while trying to establish local SSL tunnel for [raw.githubusercontent.com:443]

I don't see a redirect at youtube. Could this be Privoxy filter error?

raw.githubusercontent.com works for me. Your cert error may be fixed by using v1.5.

[rogern]

I only see 1.5 for windows is there some place to download for linux?

Edit: You did link to it , but one thing i see malwarebytes complain about your site.
I guess it is a false/positive flag only let you know

[rogern]

I think there is the problem for me.

Code:

root@ubuntu-pihole:~# netstat -tulpn | grep LISTEN
tcp        0      0 127.0.0.1:27117         0.0.0.0:*               LISTEN      2134/bin/mongod
tcp        0      0 0.0.0.0:8079            0.0.0.0:*               LISTEN      1482/python3.6
tcp        0      0 0.0.0.0:8880            0.0.0.0:*               LISTEN      2011/java
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      2011/java
tcp        0      0 10.0.0.89:80            0.0.0.0:*               LISTEN      1411/apache2
tcp        0      0 10.0.0.128:80           0.0.0.0:*               LISTEN      1133/lighttpd
tcp        0      0 0.0.0.0:8081            0.0.0.0:*               LISTEN      1482/python3.6
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      1549/pihole-FTL
tcp        0      0 10.0.0.128:53           0.0.0.0:*               LISTEN      1549/pihole-FTL
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1308/sshd
tcp        0      0 0.0.0.0:8443            0.0.0.0:*               LISTEN      2011/java
tcp        0      0 10.0.0.128:8000         0.0.0.0:*               LISTEN      1411/apache2
tcp        0      0 0.0.0.0:6789            0.0.0.0:*               LISTEN      2011/java
tcp        0      0 127.0.0.1:4711          0.0.0.0:*               LISTEN      1549/pihole-FTL
tcp        0      0 0.0.0.0:8843            0.0.0.0:*               LISTEN      2011/java

So i wonder then if it is possible to install it in docker, but the docker ip is.
First time trying docker on ubuntu or docker ever

Code:

IP address for docker0: 172.17.0.1

Wich is out of my subnet 10.0.0.1/24,10.10.10.1/24

[rogern]

Well i changed in privoxy and proxHTTPS ports
Now i have gone further but get now an HTTP Status 400 – Bad Request

Code:

root@ubuntu-pihole:/usr/local/bin/ProxHTTPSProxy# ./ProxHTTPSProxy.py
============================================================================
ProxHTTPSProxyMII v1.5 (urllib3/1.25.3)

  FrontServer  : localhost:8879
  RearServer   : localhost:8881
  ParentServer : http://10.0.0.89:8118
  Proxomitron  : http://10.0.0.89:8880
============================================================================
[14:15] 002 [D] "GET http://digi.no/favicon.ico" 404 796
^[[A[14:16] 003 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[14:16] 004 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[14:16] 005 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[14:16] 006 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[14:16] 007 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[14:16] 008 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[14:17] 011 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[14:17] 012 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[14:17] 013 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[14:17] 014 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[14:17] 015 [D] "GET http://detectportal.firefox.com/success.txt" 404 796

But why GET http://detectportal.firefox.com/success.txt" 404 796?

[vlad_s]

(Jul. 11, 2019 01:34 PM)rogern Wrote:  Well i changed in privoxy and proxHTTPS ports
Now i have gone further but get now an HTTP Status 400 – Bad Request

Code:

root@ubuntu-pihole:/usr/local/bin/ProxHTTPSProxy# ./ProxHTTPSProxy.py
============================================================================
ProxHTTPSProxyMII v1.5 (urllib3/1.25.3)

  FrontServer  : localhost:8879
  RearServer   : localhost:8881
  ParentServer : http://10.0.0.89:8118
  Proxomitron  : http://10.0.0.89:8880
============================================================================
[14:15] 002 [D] "GET http://digi.no/favicon.ico" 404 796
^[[A[14:16] 003 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[14:16] 004 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[14:16] 005 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[14:16] 006 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[14:16] 007 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[14:16] 008 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[14:17] 011 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[14:17] 012 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[14:17] 013 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[14:17] 014 [D] "GET http://detectportal.firefox.com/success.txt" 404 796
[14:17] 015 [D] "GET http://detectportal.firefox.com/success.txt" 404 796

But why GET http://detectportal.firefox.com/success.txt" 404 796?

Forgot nothing? Just to be sure, as I have done:

Code:

[GENERAL]
ProxAddr = http://localhost:8118
FrontPort = 8079
RearPort = 8081
# DefaultProxy = http://127.0.0.1:8118

Privoxy, config:

Code:

...
actionsfile proxhttpsproxy.action
...
filterfile proxhttpsproxy.filter
...
listen-address 127.0.0.1:8118
...

proxhttpsproxy.action

Code:

{+client-header-tagger{tagger4https}}
/
{+forward-override{forward 127.0.0.1:8081}}
TAG:.*?ProxHTTPSProxyMII

proxhttpsproxy.filter

Code:

CLIENT-HEADER-TAGGER: tagger4https Tagget for ProxHTTPSProxyMII.
s@^.*Tagged:.*ProxHTTPSProxyMII.*FrontProxy.*$@$0@i

I have a problem, a couple of sites do not work, https://just.ru and https://www.onlinetrade.ru
In privoxy added (action):

Code:

{-block \
-crunch-incoming-cookies \
-crunch-outgoing-cookies \
-fast-redirects \
-filter \
-hide-referrer \
-prevent-compression \
-overwrite-last-modified \
}
.onlinetrade.ru/

Does not help. curl -vk https://www.onlinetrade.ru loads the page, but some kind of looping happens to the browser, after several cycles, https://www.onlinetrade.ru appears in the address bar of the browser instead of https://www.onlinetrade.ru/? and so on to infinity. If I add to

Code:

[SSL Pass-Thru]
*.onlinetrade.ru

then the page will somehow load, but as if the styles have not loaded. Without a proxy, it works fine.

[amy]

Looking at that page, there is some sort of tracking/fingerprinting JavaScript on it, which I did not run but it looks very suspicious to me --- stuff like that reminds me of hacked sites where a malware script has been injected.

[rogern]

Can confirm that my problems did go away.
Though it loads pages a lot slower than pihole, maybe it is because i do have both enabled.
Just firefox though wich is used for the moment with proxhttps and privoxy.