Post Reply 
ProxHTTPSProxyMII: Development
Jul. 22, 2019, 02:12 AM
Post: #241
RE: ProxHTTPSProxyMII: Development
(Jul. 20, 2019 06:48 PM)vlad_s Wrote:  Why are hop-by-hop headers (line 233) removed from the ProxyTool.py file?

The HTTP/1.1 specifications require proxies to remove hop-by-hop headers.

https://tools.ietf.org/html/draft-ietf-h...on-7.1.3.1 Wrote:Hop-by-hop header fields, which are meaningful only for a single
transport-level connection, and are not stored by caches or
forwarded by proxies.
Add Thank You Quote this message in a reply
Jul. 22, 2019, 07:04 PM (This post was last modified: Jul. 22, 2019 07:05 PM by vlad_s.)
Post: #242
RE: ProxHTTPSProxyMII: Development
Well, I have a bad understanding of all this, but I will explain what I asked for. Privoxy still puts these headers for clients, it’s visible in the log, but in fact they don’t. Does Privoxy violate standards?
Some sites, such as tjournal.ru, vc.ru, require a protocol change (the Upgrade header), in this form in the developer console I get an error HTTP 403 (it is generated by the site, not Privoxy). I modified this line in ProxyTool.py and finally saw instead of HTTP 403 in HTTP 101. I don’t know what the site developers were trying to change the protocol, but it didn’t work. So I decided to ask.
Add Thank You Quote this message in a reply
Jul. 23, 2019, 06:08 AM
Post: #243
RE: ProxHTTPSProxyMII: Development
This will take some more study. The short story so far is...

tjournal.ru and vc.ru are trying to use the WebSocket protocol.

I'm not sure our proxies can or should handle WebSocket.

Although the specs do not allow proxies to "forward" hop-by-hop headers, proxies can add their own hop-by-hop headers.
We need to send or allow (at least)

Code:
Upgrade: WebSocket
Connection: Upgrade

Did the sites work correctly after you made the changes to ProxyTool.py?

Can Privoxy filter the WebSocket data?
https://sourceforge.net/p/ijbswa/mailman/message/34532990/
Add Thank You Quote this message in a reply
Jul. 23, 2019, 12:03 PM
Post: #244
RE: ProxHTTPSProxyMII: Development
WebSocket is actually one of the things I tried to filter. I had an experimental version of Proxomitorn Reborn that attempted to tunnel WS connections.

The big difference between WS and HTTP is that it's truly bidirectional instead of "half duplex"; either side can send at any time and one side can send multiple times before the other side does (if ever). A typical proxy's "read client, filter, send to server; read server, filter, send to client" loop will work only in very limited cases and make other cases look like a hang.
Add Thank You Quote this message in a reply
Jul. 23, 2019, 05:34 PM (This post was last modified: Jul. 23, 2019 05:40 PM by vlad_s.)
Post: #245
RE: ProxHTTPSProxyMII: Development
After the changes in ProxyTool.py, the above listed sites work without changes. Does Privoxy support WebSocket? The developer’s response is rather vague. Here is what is recorded in the Privoxy log:
Code:
...
2019-07-23 20:14:15.328 7fe55effd700 Header: scan: GET http://tjournal.ru/chan/system HTTP/1.1
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Accept: */*
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Host: tjournal.ru
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.5,en;q=0.3
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Accept-Encoding: gzip, deflate, br
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Sec-WebSocket-Version: 13
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Origin: https://tjournal.ru
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Sec-WebSocket-Extensions: permessage-deflate
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Sec-WebSocket-Key: xLQdEBcAZbVdlHZTmiHG4g==
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Connection: keep-alive, Upgrade
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Cookie: pushVisitsCount=45;fingerprint=f794e8cfdfc384g67de; adblock-state=1; audio_
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Pragma: no-cache
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Cache-Control: no-cache
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Upgrade: websocket
2019-07-23 20:14:15.329 7fe55effd700 Header: scan: Tagged: ProxHTTPSProxyMII FrontProxy/v1.5 Python/3.7.3:397
2019-07-23 20:14:15.329 7fe56d32f700 Header: scan: GET http://static-osnova.gcdn.co/static/build/tjournal.ru/dropdown.7ed182cc2aa8236bff77.min.js HT
2019-07-23 20:14:15.329 7fe56d32f700 Header: scan: Accept: */*
2019-07-23 20:14:15.329 7fe56d32f700 Header: scan: Host: static-osnova.gcdn.co
...
2019-07-23 20:14:15.360 7fe55effd700 Header: Keeping the client header 'Connection: keep-alive, Upgrade' around. The server connection will be kept alive if possible.
...
2019-07-23 20:14:15.400 7fe55effd700 Header: scan: HTTP/1.1 101 Switching Protocols
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Server: nginx
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Date: Tue, 23 Jul 2019 17:14:15 GMT
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Content-Length: 0
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Connection: upgrade
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Sec-WebSocket-Extensions: permessage-deflate
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Access-Control-Allow-Credentials: true
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Access-Control-Allow-Origin: https://tjournal.ru
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Sec-WebSocket-Accept: sGlz46w7kC9gyLQiIHLN6IA81u4=
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Upgrade: websocket
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Strict-Transport-Security: max-age=31536000; preload
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: X-XSS-Protection: 1; mode=block
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: X-Content-Type-Options: nosniff
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Referrer-Policy: origin
2019-07-23 20:14:15.401 7fe55effd700 Header: scan: Content-Security-Policy: default-src * data: 'unsafe-eval' 'unsafe-inline'
2019-07-23 20:14:15.406 7fe55effd700 Header: Adding: Proxy-Connection: keep-alive
...

What I changed in ProxyTool.py to avoid misunderstandings was:
Code:
except (ssl.SSLEOFError, ssl.SSLError) as e:
            logger.error(Fore.RED + Style.BRIGHT + "[SSLError]")
            self.send_error(417, message="Exception %s" % str(e.__class__), explain=str(e))

    def purge_headers(self, headers):
        "Remove hop-by-hop headers that shouldn't pass through a Proxy"
        for name in ["Connection", "Keep-Alive", "Upgrade",
                     "Proxy-Connection", "Proxy-Authenticate"]:
            del headers[name]

    def purge_write_headers(self, headers):
        self.purge_headers(headers)
        for key, value in headers.items():
            self.send_header(key, value)
        self.end_headers()

changes:
Code:
except (ssl.SSLEOFError, ssl.SSLError) as e:
            logger.error(Fore.RED + Style.BRIGHT + "[SSLError]")
            self.send_error(417, message="Exception %s" % str(e.__class__), explain=str(e))

    def purge_headers(self, headers):
        "Remove hop-by-hop headers that shouldn't pass through a Proxy"
        for name in [""]:
            del headers[name]

    def purge_write_headers(self, headers):
        self.purge_headers(headers)
        for key, value in headers.items():
            self.send_header(key, value)
        self.end_headers()
235 line.
Add Thank You Quote this message in a reply
Jul. 26, 2019, 10:16 PM
Post: #246
RE: ProxHTTPSProxyMII: Development
This is what I see in the ProxHTTPSProxyMII logs after enabling the Upgrade header:
Code:
...
[01:06] 624 [D] "GET https://tjournal.ru/chan/content-108174" 101 0
127.0.0.1 - - [27/Jul/2019 01:06:38] code 400, message Bad HTTP/0.9 request type ('Á©J+ÕÞà}\x9f\x10')
127.0.0.1 - - [27/Jul/2019 01:06:38] "Á©J+ÕÞà}......a¢....çêú.áo.zÿwfc.lÈbüJ/.Cyì..ÜJ...1nv" 400 -
[01:06] 625 [D] "GET https://leonardo.osnova.io/b72c9bb3-5240-82b3-feaf-3b7fba1d0419/-/resize/48/" 200 598
[01:06] 626 [D] "GET https://tjournal.ru/chan/content-108199" 101 0
[01:07] 627 [D] "GET https://leonardo.osnova.io/05fcbd49-02d1-cf13-f121-ee47f4a0e09a/-/scale_crop/1300x819/center/-/format/webp/
127.0.0.1 - - [27/Jul/2019 01:07:08] code 400, message Bad HTTP/0.9 request type ('\x88\x80O\x8d')
127.0.0.1 - - [27/Jul/2019 01:07:08] "..O.      L" 400 -
127.0.0.1 - - [27/Jul/2019 01:07:22] code 400, message Bad request syntax ('\x88\x80\x05¡¤§')
127.0.0.1 - - [27/Jul/2019 01:07:22] "...¡¤§" 400 -
127.0.0.1 - - [27/Jul/2019 01:08:48] code 400, message Bad request version ('260')
127.0.0.1 - - [27/Jul/2019 01:08:48] "CNT 1 CON 260" 400 -
...
Add Thank You Quote this message in a reply
Post Reply 


Forum Jump: