The Un-Official Proxomitron Forum

Full Version: ProxHTTPSProxyMII: Development
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
0.8a should fix the urllib3 problem.

You already help a lot. Does that mean I could be free from fixing problems of the script for the next few days? Smile!
(Jun. 11, 2014 05:14 AM)whenever Wrote: [ -> ]Does that mean I could be free from fixing problems of the script for the next few days? Smile!

Ahhh! The silver lining to my clouds. Wink

(Jun. 11, 2014 05:14 AM)whenever Wrote: [ -> ]0.8a should fix the urllib3 problem.

It does. Smile!

The Proxomitron shows unclosed connections accumulating. Do you see the same?
When I open mail.yahoo.com there is a GET to "sb.scorecardresearch.com/p?" the response is a 302 to "sb.scorecardresearch.com/p2?" that hangs and does not close.
After I use the Proxomitron to abort the connection, the browser again requests "sb.scorecardresearch.com/p?".

Got to go.
(Jun. 11, 2014 05:22 PM)JJoe Wrote: [ -> ]The Proxomitron shows unclosed connections accumulating. Do you see the same?

I saw that too. Just had not got time to look into it.

The hang happened between Proxomitron and the rear server, right? If so, I think it is a problem similar to http://prxbx.com/forums/showthread.php?t...7#pid17607

What if you change line 102 from:

Code:
if r.status == 304:

to:

Code:
if r.status in (302, 304):
(Jun. 12, 2014 01:11 AM)whenever Wrote: [ -> ]
Code:
if r.status in (302, 304):

That fixed some. Changing it to

Code:
if r.status in (204, 302, 304):

fixed the rest. Cheers

Hope it was ok to zip and upload as ProxHTTPSProxy 0.8b
and it is very quick! Cool

Should anybody still have problems with https at yahoo...
In your Proxomitron's bypass list there may be an entry like

Code:
# Bypasses for Yahoo messenger (ok to remove if you don't use it)
[^/]++(edit|address|pager|cs[bc]).yahoo.com/

When this entry matches the rear ProxHTTPSProxy server may be bypassed. Yahoo may fail to load and the browser may report a 'redirection error'.
This may happen after submitting login info.

Also, I have been using a very simple cfg, "TestProxHTTPSProxy1.cfg", to help find the source of problems. There are a few simple filters and no lists.

Code:
##
## Proxomitron Config File
##

[Global]
Enable = TRUE
FreezeGIF = FALSE
FilterHTML = TRUE
FilterHeadersOut = TRUE
FilterHeadersIn = TRUE
EnableProxy = FALSE
EnableAutoRun = FALSE
ForceTextures = FALSE
NoTextures = FALSE
SysTray = TRUE
UseSSLeay = FALSE
NoCmdURL = TRUE
PriorityBoost = TRUE
BypassURL = "$OHDR(Tagged:Proxomitron FrontProxy/*) $SETPROXY(127.0.0.1:8081)(^)"
Port = 8080
MaxCapURL = 100

[Blocklists]

[HTTP headers]
In = FALSE
Out = TRUE
Key = "Accept-encoding: Allow webpage encoding (out)"
Match = "*"
Replace = "gzip, deflate"

In = TRUE
Out = FALSE
Key = "Cache-Control: always cache (in)"
Match = "*"


[Patterns]
Name = "New HTML filter"
Active = TRUE
Limit = 2
Match = "\1$STOP()"
Replace = "\1"


[Proxies]
OpenLog = TRUE

>127.0.0.1:8081 ProxHTTPSProxyR
(Jun. 12, 2014 02:47 AM)JJoe Wrote: [ -> ]
Code:
if r.status in (204, 302, 304):

fixed the rest. Cheers

Nice job! Thumbs Up

0.8c added colorful output. It depends on module colorama.

Code:
c:\Python34\Scripts>pip install colorama
(Jun. 13, 2014 09:42 AM)whenever Wrote: [ -> ]0.8c added colorful output. It depends on module

Ahhh, that is better and green. Smile!

I was still seeing hangs at yahoo on "content-length: 0" responses. Including hangs on 200 responses.

So I commented out

Code:
# Proxomitron hang for 304 response if we don't close the connection
# Maybe a bug?
# https://code.google.com/p/chromium/issues/detail?id=9875
if r.status in (204, 302, 304):
    self.close_connection = 1

and added

Code:
# Hang between Proxomitron and Rear Proxy for "content-length: 0" response if we don't close the connection
# ???
if data == None:
    self.close_connection = 1

Hangs are resolved and connections closed.
However, things seem slower and I'm seeing EOF errors with each "content-length: 0" response

Code:
[21:18:55] "GET https://login.yahoo.com/config/verify?.done=https%3a//us-mg.mail.yahoo.com/neo/b/launch%3f.rand=2 HTTP/1.1" 302 0
EOF occurred in violation of protocol (_ssl.c:1636) on GET /config/verify?.done=https%3a//us-mg.mail.yahoo.com/neo/b/launch%3f.rand=2 HTTP/1.1
Code:
[21:19:22] "GET https://us-mg.mail.yahoo.com/neo/b/dimensions?noFlush&width=1349&height=658 HTTP/1.1" 200 0
EOF occurred in violation of protocol (_ssl.c:1636) on GET /neo/b/dimensions?noFlush&width=1349&height=658 HTTP/1.1

Is my code correct?

I see 'Content-Length' mentioned in proxytool.py but it will take some more study before I completely understand it.
(Jun. 16, 2014 02:37 AM)JJoe Wrote: [ -> ]
Code:
# Hang between Proxomitron and Rear Proxy for "content-length: 0" response if we don't close the connection
# ???
if data == None:
    self.close_connection = 1

Hangs are resolved and connections closed. However, things seem slower ...

"data" is request body and most of the time it is None provided the http method is not POST.

Your code disable the connection reuse for most of the time, that might be the cause of the slow down.

"r.data " is response body and "self.body_length" is its length. The code could be changed to

Code:
if self.body_length == 0:
    self.close_connection = 1

(Jun. 16, 2014 02:37 AM)JJoe Wrote: [ -> ]I'm seeing EOF errors with each "content-length: 0" response

I am not sure if it helps but according to http://stackoverflow.com/questions/14102...-occurred, Let's give it a test by changing line 146 from

Code:
sslparams = dict(cert_reqs="REQUIRED", ca_certs="cacert.pem", ssl_version="SSLv23")

to

Code:
sslparams = dict(cert_reqs="REQUIRED", ca_certs="cacert.pem", ssl_version="TLSv1")

(Jun. 16, 2014 02:37 AM)JJoe Wrote: [ -> ]I see 'Content-Length' mentioned in proxytool.py but it will take some more study before I completely understand it.

Persistent connection needs 'Content-Length' value to to indicate the transfer-length of the message-body. proxytool.py updates this header with the right value.
(Jun. 16, 2014 08:38 AM)whenever Wrote: [ -> ]Your code disable the connection reuse for most of the time,

D'oh! Yep... I added a flag to check. I think the slowdown was mostly an internet connection issue, however.

(Jun. 16, 2014 08:38 AM)whenever Wrote: [ -> ]"r.data " is response body and "self.body_length" is its length.

And I wondered where whenever found this... D'oh!D'oh!
And then I wondered how I missed it.

Closing on "self.body_length == 0:" has fixed my sign in problem.
Sometimes I see a stall. When this happens the Proxomitron's log will show

Code:
Socket Error 10053 for request flush

(Jun. 16, 2014 08:38 AM)whenever Wrote: [ -> ]Let's give it a test by changing line 146...

I am still seeing the errors but TLSv1 is quicker and appears to use less cpu.

Time to go again.

Have fun.
(Jun. 16, 2014 10:00 PM)JJoe Wrote: [ -> ]
Code:
Socket Error 10053 for request flush

It seems Proxomitron hadn't expected that we closed the connection. Let's be a little more polite by changing line 119 of proxytool.py from

Code:
# Reuse the connection if the Client asks that
if not self.close_connection:
    headers["Connection"] = "Keep-Alive"

to

Code:
# Connection management
if self.close_connection:
    headers["Connection"] = "close"
else:
    headers["Connection"] = "Keep-Alive"
(Jun. 17, 2014 10:06 AM)whenever Wrote: [ -> ]Let's be a little more polite by changing line 119 of proxytool.py from

I just saw and logged it, after changing line 119. Maybe some context will help.

Some of these problems seem to happen only while web page filtering is enabled.

For testing, I have added another instance of the Proxomitron.
Looks like FrontProxHTTPSProxy>firstProxomitron>secondProxomitron>RearProxHTTPSProxy.
firstProxomitron only logs, no user filters. secondProxomitron may filter.

A request for the yahoo mail page after signing in looks like:

Code:
********firstProxomitron*********
+++GET 3+++
Using Proxy - 127.0.0.1:8080
GET http://us-mg.mail.yahoo.com/neo/b/launch?.rand=9 HTTP/1.1
Accept-Encoding: gzip,deflate,lzma,sdch
Host: us-mg.mail.yahoo.com
Referer: https://login.yahoo.com/?mbsw=d&.src=ym&.intl=us
Cookie:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.8
Tagged: Proxomitron FrontProxy/0.8d Python/3.4.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.132 Safari/537.36 OPR/21.0.1432.57 (Edition Campaign 37)
Connection: keep-alive
********/firstProxomitron********

********secondProxomitron********
+++GET 3+++
Using Proxy - 127.0.0.1:8081
GET http://us-mg.mail.yahoo.com/neo/b/launch?.rand=9 HTTP/1.1
Accept-Encoding: gzip, deflate
Host: us-mg.mail.yahoo.com
Referer: https://login.yahoo.com/?mbsw=d&.src=ym&.intl=us
Cookie:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.8
Tagged: Proxomitron FrontProxy/0.8d Python/3.4.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.132 Safari/537.36 OPR/21.0.1432.57 (Edition Campaign 37)
Connection: keep-alive
** 3 Socket Error 10053 for request flush **

+++RESP 3+++
HTTP/1.1 200 OK
Date: Tue, 17 Jun 2014 21:21:14 GMT
P3P: policyref="http://info.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE LOC GOV"
Expires: Wed, 19 Apr 1970 17:30:00 GMT
Pragma: no-cache
Vary: Accept-Encoding
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip
Age: 1
Server: ATS
Via: http/1.1 r13.ycpi.ne1.yahoo.net (ApacheTrafficServer/4.0.2 [cMsSf ])
Content-Length: 13864
Connection: Keep-Alive
Match 3: New HTML filter
********/secondProxomitron********

********firstProxomitron********
+++RESP 3+++
HTTP/1.1 200 OK
Date: Tue, 17 Jun 2014 21:21:14 GMT
P3P: policyref="http://info.yahoo.com/w3c/p3p.xml", CP="CAO DSP COR CUR ADM DEV TAI PSA PSD IVAi IVDi CONi TELo OTPi OUR DELi SAMi OTRi UNRi PUBi IND PHY ONL UNI PUR FIN COM NAV INT DEM CNT STA POL HEA PRE LOC GOV"
Expires: Wed, 19 Apr 1970 17:30:00 GMT
Pragma: no-cache
Vary: Accept-Encoding
Content-Type: text/html; charset=utf-8
Age: 1
Server: ATS
Via: http/1.1 r13.ycpi.ne1.yahoo.net (ApacheTrafficServer/4.0.2 [cMsSf ])
Transfer-Encoding: chunked
********/firstProxomitron********

Summary:
secondProxomitron user header filter changes "Accept-Encoding" header from "gzip,deflate,lzma,sdch" to "gzip, deflate"
** 3 Socket Error 10053 for request flush **, this does not always happen.
"Content-Encoding", "Content-Length" and Connection headers are removed but not by user filters. This is what I remember and expected.
"Transfer-Encoding: chunked" header is added, as expected.

Not shown:
When the secondProxomiton is not filtering, only the Connection header is missing in firstProxomitron log window.

So could the switch to "Transfer-Encoding: chunked" be a problem?

I'll be looking at \k next.
(Jun. 18, 2014 01:26 AM)JJoe Wrote: [ -> ]Maybe some context will help

I mean being polite because it is not a necessary. In HTTP/1.1 you could close a connection without sending "Connection: close" first.

(Jun. 18, 2014 01:26 AM)JJoe Wrote: [ -> ]So could the switch to "Transfer-Encoding: chunked" be a problem?

I don't so. It's the normal behavior of a filtering proxy.

(Jun. 18, 2014 01:26 AM)JJoe Wrote: [ -> ]** 3 Socket Error 10053 for request flush **, this does not always happen.

When that happen, does it break the web page?

I saw you mentioned it after applying the "if self.body_length == 0: " fix. Do you see it before that?

v0.8d is attached to make sure we are on the same boat. It should solve the EOF errors problem.
(Jun. 18, 2014 11:25 AM)whenever Wrote: [ -> ]
(Jun. 18, 2014 01:26 AM)JJoe Wrote: [ -> ]** 3 Socket Error 10053 for request flush **, this does not always happen.

When that happen, does it break the web page?

No. It does eventually load. ProxHTTPSProxy does work. Smile! I have been using it. Thank you!

(Jun. 18, 2014 11:25 AM)whenever Wrote: [ -> ]I saw you mentioned it after applying the "if self.body_length == 0: " fix. Do you see it before that?

I don't remember. I will check.

Have fun.
(Jun. 18, 2014 01:45 PM)JJoe Wrote: [ -> ]
(Jun. 18, 2014 11:25 AM)whenever Wrote: [ -> ]I saw you mentioned it after applying the "if self.body_length == 0: " fix. Do you see it before that?

I don't remember. I will check.

No, I do not see it. When not using the fix, the browser hangs before I would expect to see the socket error alert. The URL is the result of a redirect. Directly requesting the URL does not cause an alert.

I am not sure the alert is a problem. I may need to find another connection or ISP to test with.

(Jun. 18, 2014 01:26 AM)JJoe Wrote: [ -> ]I'll be looking at \k next.

I do believe \k is causing

Code:
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 65221)
Traceback (most recent call last):
  File "C:\Python34\lib\socketserver.py", line 609, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python34\lib\socketserver.py", line 344, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python34\lib\socketserver.py", line 665, in __init__
    self.handle()
  File "C:\Python34\lib\http\server.py", line 400, in handle
    self.handle_one_request()
  File "C:\Python34\lib\http\server.py", line 368, in handle_one_request
    self.raw_requestline = self.rfile.readline(65537)
  File "C:\Python34\lib\socket.py", line 371, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [WinError 10054] An existing connection was forcibly close
d by the remote host
----------------------------------------

These exceptions go away after removing \k from my filters. The socket error remains, however.

(Jun. 18, 2014 11:25 AM)whenever Wrote: [ -> ]v0.8d is attached to make sure we are on the same boat. It should solve the EOF errors problem.

My ProxHTTPSProxy's log window is exception and error free, for now. Wink
Thumbs UpCheersHailCool
(Jun. 18, 2014 05:56 PM)JJoe Wrote: [ -> ]I am not sure the alert is a problem. I may need to find another connection or ISP to test with.

I saw "Socket Error 10053 for request flush" too, even its self.body_length is > 0, so that connection was not flagged to be closed by our code, and it later got its response normally.

So I am not sure how it is related to the "if self.body_length == 0" fix, but I did see it only after applying that fix.

(Jun. 18, 2014 01:26 AM)JJoe Wrote: [ -> ]I do believe \k is causing
...
These exceptions go away after removing \k from my filters. The socket error remains, however.

Those exceptions are expected for \k. I would leave it as is since it was raised beyond the scope of our code.

JJoe, would you like to write a new readme and update the instructions on configuring the script to work with Proxomitron? I would look into building an exe version.
(Jun. 19, 2014 04:05 AM)whenever Wrote: [ -> ]JJoe, would you like to write a new readme and update the instructions on configuring the script to work with Proxomitron?

Yes, I'm collecting info for that now. I may start another thread.

You might consider another name for this new proxy.

First, I want to retest the socket error. I still had ProxHTTPSProxy in the chain.

Then I need to go to bed.
Pages: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Reference URL's