Post Reply 
ProxHTTPSProxy, a Proxomitron SSL Helper Program
May. 31, 2010, 03:11 AM
Post: #76
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 31, 2010 02:21 AM)whenever Wrote:  Graycode, is it normal a "/" come after "CONNECT" method? How could that happened? I think Proxo doesn't get the host:port where to establish the tunnel so it popped up the error message box.

No. The CONNECT method is supposed to be only for proxies, and proxied requests should have the host-port as part of the request line.

Non-proxied request (something a web server would get)
Code:
GET / HTTP/1.1
Host: www.example.com

Proxied request (something a proxy would get)
Code:
GET http://www.example.com/ HTTP/1.1
Host: www.example.com

For a CONNECT given to a proxy, it's normal for the request to contain host and port while the 'Host:' header does not contain the port by some browsers.
Code:
CONNECT www.example.com:443 HTTP/1.1
Host: www.example.com

Did your Python proxy get the headers you showed? Is Proxomitron just showing that format in its logging, or did it really send those headers to your proxy? If Proxo knows the communication is going to be sent through your proxy the it should have sent proxy formatting.

I'm confused.
Add Thank You Quote this message in a reply
May. 31, 2010, 03:19 AM
Post: #77
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 31, 2010 01:30 AM)JJoe Wrote:  Perhaps "continuing problem with httplib multiple set-cookie headers" at http://bugs.python.org/issue1660009 ?

A quick fix for that problem.


Attached File(s)
.zip  HTTPSProxy 0.1a.zip (Size: 1.85 KB / Downloads: 692)
.zip  ProxHTTPSProxy 0.2d.zip (Size: 1.99 KB / Downloads: 700)
Add Thank You Quote this message in a reply
May. 31, 2010, 03:38 AM
Post: #78
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 31, 2010 03:11 AM)Graycode Wrote:  Did your Python proxy get the headers you showed? Is Proxomitron just showing that format in its logging, or did it really send those headers to your proxy? If Proxo knows the communication is going to be sent through your proxy the it should have sent proxy formatting.

I'm confused.

I am not logging headers but I use host-port to build the https url to be 307 redirected in ProxHTTPSProxy or to be fetched in HTTPSProxy. I hadn't seen exceptions about that yet.

Proxo is not showing that format in its logging. For normal communication, it does log "CONNECT http://www.example.com:443 HTTP/1.1".

The host-port is so important that I don't think the browser will get it wrong. I am also confused why Proxo doesn't get it right. It's even not going into SSL layer yet.
Add Thank You Quote this message in a reply
May. 31, 2010, 04:21 AM
Post: #79
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 31, 2010 03:19 AM)whenever Wrote:  
(May. 31, 2010 01:30 AM)JJoe Wrote:  Perhaps "continuing problem with httplib multiple set-cookie headers" at http://bugs.python.org/issue1660009 ?

A quick fix for that problem.

With that and a ssl_sock.settimeout of .5 sec, I've logged in to yahoo.

Thanks
Add Thank You Quote this message in a reply
May. 31, 2010, 09:09 AM
Post: #80
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 31, 2010 03:38 AM)whenever Wrote:  The host-port is so important that I don't think the browser will get it wrong. I am also confused why Proxo doesn't get it right. It's even not going into SSL layer yet.

I just tried to telnet to Proxo's listening port and issued "CONNECT / HTTP/1.1" command manually. It didn't trigger the error message box but responded with below text, which indicated Proxo did have detection for malformed command:

Code:
HTTP/1.1 403 Connection refused
Connection: close

I forgot to add that when the error was raised, the error message box was keeping poping up and the Proxo log window was keeping outputing "CONNECT / HTTP/1.1". It seems something went wrong within Proxo internal and it was keeping trying.

(May. 31, 2010 04:21 AM)JJoe Wrote:  With that and a ssl_sock.settimeout of .5 sec, I've logged in to yahoo.

A longer timeout value will make the proxy block longer. I don't know what value should be reasonable. It is local communication between the browser and the proxy, which I had thought could be done very quickly.
Add Thank You Quote this message in a reply
May. 31, 2010, 05:12 PM (This post was last modified: May. 31, 2010 05:27 PM by Graycode.)
Post: #81
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
Are you using ProxHTTPSProxy or the other HTTPSProxy?

At the bottom of do_CONNECT in HTTPSProxy it seems to be closing the socket immediately after doing the shutdown(). Using shutdown for write should make the socket layer send a FIN after the last packet is sent (flushed), then the other side should ACK to close the socket, and final clean closure determined by a read of 0 bytes.

It may need to be something like:
Code:
# socket.SHUT_WR == 1
ssl_sock.shutdown(1)
ssl_sock.settimeout(6)   ## override previous use of settimeout(0.1)
while True:      ## wait for other side to ACK shutdown() closure
    if not ssl_sock.read()
        break
ssl_sock.close()

I've specified a 6 second timeout, don't know how much data will need to hit the wire, be processed by the other side and have the other side ACK to the FIN. When things work as they should, doing that adds no unnecessary delay. Considered using settimeout(0) or setblocking(True) for more permanent wait, maybe that's better. Something should be specified otherwise the previous settimeout(0.1) or (0.5) will still be in effect, and that may be insufficient to get all the data flushed back to Proxo.

There's a lot of things I don't know or don't understand about Python. What you've been able to accomplish is not easy or simple.
Add Thank You Quote this message in a reply
May. 31, 2010, 08:32 PM
Post: #82
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 31, 2010 09:09 AM)whenever Wrote:  
(May. 31, 2010 04:21 AM)JJoe Wrote:  With that and a ssl_sock.settimeout of .5 sec, I've logged in to yahoo.

A longer timeout value will make the proxy block longer. I don't know what value should be reasonable. It is local communication between the browser and the proxy, which I had thought could be done very quickly.

It looks like it has to be long enough to allow interaction with the SSL warning dialogs, "Allow for Session"? If I wait to click more than ssl_sock.settimeout, I have to reload the page.

(May. 31, 2010 05:12 PM)Graycode Wrote:  Are you using ProxHTTPSProxy or the other HTTPSProxy?

HTTPSProxy.

Thanks
Add Thank You Quote this message in a reply
Jun. 01, 2010, 02:59 AM
Post: #83
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 31, 2010 05:12 PM)Graycode Wrote:  It may need to be something like:
...

That's what I do in ProxHTTPSProxy to solve the reset issue. It seems not needed in HTTPSProxy because there is no pending readable data. Anyway, being strict is better. I added the code to HTTPSProxy too.

(May. 31, 2010 05:12 PM)Graycode Wrote:  Something should be specified otherwise the previous settimeout(0.1) or (0.5) will still be in effect, and that may be insufficient to get all the data flushed back to Proxo.

I forgot to set the socket back to blocking mode. Banging Head
Fixed.

(May. 31, 2010 08:32 PM)JJoe Wrote:  It looks like it has to be long enough to allow interaction with the SSL warning dialogs, "Allow for Session"? If I wait to click more than ssl_sock.settimeout, I have to reload the page.

Well, even 1 hour is not long enough. The socket should block forever until we finish interaction with the SSL warning dialogs. Should be fixed in version 0.1b.


Attached File(s)
.zip  HTTPSProxy 0.1b.zip (Size: 2.1 KB / Downloads: 676)
Add Thank You Quote this message in a reply
Jun. 01, 2010, 04:03 AM
Post: #84
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 01, 2010 02:59 AM)whenever Wrote:  I forgot to set the socket back to blocking mode. Banging Head
Fixed.

I think you are doing great. This isn't... It's suppose to be some kind of fun. Wink Don't forget to play with the kid.

(Jun. 01, 2010 02:59 AM)whenever Wrote:  
(May. 31, 2010 08:32 PM)JJoe Wrote:  It looks like it has to be long enough to allow interaction with the SSL warning dialogs, "Allow for Session"? If I wait to click more than ssl_sock.settimeout, I have to reload the page.

Well, even 1 hour is not long enough. The socket should block forever until we finish interaction with the SSL warning dialogs. Should be fixed in version 0.1b.

Appears to be fixed. I do still need a ssl_sock.settimeout of .2 to get all the yahoo cookies.

I'm testing at https://ssl.scroogle.org/ , https://login.yahoo.com/ , https://bugzilla.mozilla.org/ , https://developer.mozilla.org/ .

What do you see at https://developer.mozilla.org/ ?
I see

Code:
New Message Log Window....

+++GET 1+++
Using Proxy - 127.0.0.1:8081
CONNECT https://developer.mozilla.org:443/ HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.20) Gecko/20081217 Firefox/2.0.0.20
Proxy-Connection: keep-alive
Host: developer.mozilla.org
Loaded: OpenSSL 0.9.8 05 Jul 2005

+++SSL:GET 1+++
Using Proxy - 127.0.0.1:8081
SSL cipher TLSv1 AES256-SHA (256 bits)
GET https://developer.mozilla.org:443/ HTTP/1.1
Host: developer.mozilla.org
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.20) Gecko/20081217 Firefox/2.0.0.20
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: keep-alive
HTTP/1.0 200 Connection established
HTTP/1.0 Proxy-agent: HTTPSProxy/0.1b Python/2.6.5
SSL Verify: [1:60900360] error number 60900360

and

Code:
Exception happened during processing of request from ('127.0.0.1', 59345)
Traceback (most recent call last):
  File "C:\Python26\lib\SocketServer.py", line 558, in process_request_thread
    self.finish_request(request, client_address)
  File "C:\Python26\lib\SocketServer.py", line 320, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Python26\lib\SocketServer.py", line 615, in __init__
    self.handle()
  File "C:\Python26\lib\BaseHTTPServer.py", line 329, in handle
    self.handle_one_request()
  File "C:\Python26\lib\BaseHTTPServer.py", line 323, in handle_one_request
    method()
  File "C:\Users\E3\Programs\HTTPSProxy 0.1\HTTPSProxy.py", line 110, in do_CONN
ECT
    resp = opener.open(req)
  File "C:\Python26\lib\urllib2.py", line 397, in open
    response = meth(req, response)
  File "C:\Python26\lib\urllib2.py", line 510, in http_response
    'http', request, response, code, msg, hdrs)
  File "C:\Python26\lib\urllib2.py", line 429, in error
    result = self._call_chain(*args)
  File "C:\Python26\lib\urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "C:\Python26\lib\urllib2.py", line 605, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "C:\Python26\lib\urllib2.py", line 391, in open
    response = self._open(req, data)
  File "C:\Python26\lib\urllib2.py", line 409, in _open
    '_open', req)
  File "C:\Python26\lib\urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "C:\Python26\lib\urllib2.py", line 1161, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "C:\Python26\lib\urllib2.py", line 1134, in do_open
    r = h.getresponse()
  File "C:\Python26\lib\httplib.py", line 986, in getresponse
    response.begin()
  File "C:\Python26\lib\httplib.py", line 391, in begin
    version, status, reason = self._read_status()
  File "C:\Python26\lib\httplib.py", line 355, in _read_status
    raise BadStatusLine(line)
BadStatusLine
----------------------------------------

Loads without HTTPSProxy.
Add Thank You Quote this message in a reply
Jun. 01, 2010, 07:22 AM
Post: #85
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 01, 2010 04:03 AM)JJoe Wrote:  Don't forget to play with the kid.

Don't worry. It is not taking my family time. Smile!

I am thinking replacing urllib2 with httplib module for lower level operations.

urllib2 follows redirects by default, which may cause issue. For example, the browser doesn't know a 301/302 redirect happened (they should know); if a login page set cookies with a redirect, the browser won't see it.

Give me some time to work on it. Let's wait and see if it could solve the https://developer.mozilla.org/ issue.
Add Thank You Quote this message in a reply
Jun. 01, 2010, 04:12 PM
Post: #86
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
Well, I decided to go more lower level, so here is a just working socket version. It seems faster and worked with https://developer.mozilla.org/.

Have fun and report issue please.

I might won't be able to respond until Friday. Go to bed now.


Attached File(s)
.zip  HTTPSProxy socket alpha.zip (Size: 1.75 KB / Downloads: 675)
Add Thank You Quote this message in a reply
Jun. 01, 2010, 07:01 PM
Post: #87
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
This HTTPSProxy works at all sites checked. Smile!

And now the log shows outgoing headers. What next? Wink

You might want to update this topic's first post. The link is to ProxHTTPSProxy 0.1b.

I will be leaving tomorrow. Expect to be back Thursday or Friday.
Add Thank You Quote this message in a reply
Jun. 01, 2010, 11:46 PM
Post: #88
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 01, 2010 07:01 PM)JJoe Wrote:  This HTTPSProxy works at all sites checked. Smile!

Terrific! You & whenever have done a super job! Thumbs Up
Add Thank You Quote this message in a reply
Jun. 02, 2010, 02:09 AM
Post: #89
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(Jun. 01, 2010 11:46 PM)Graycode Wrote:  Terrific! You & whenever have done a super job! Thumbs Up

Strange, I have had the same thought. You & whenever... Wink

Thank you for sharing your time and knowledge.
Add Thank You Quote this message in a reply
Jun. 04, 2010, 03:43 PM (This post was last modified: Jun. 04, 2010 03:43 PM by whenever.)
Post: #90
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
Well, you all have helped a lot. Thanks!

Here is the version 0.3 for playing around. Cheers

What's new:
Code:
- Rewrite ProxHTTPSProxy with socket module too, faster than urllib2 version
- Put ProxHTTPSProxy and HTTPSProxy into one file
  half_ssl = 0: old HTTPSProxy + http request handler
  half_ssl = 1: socket version of ProxHTTPSProxy
- Parent proxy now supports both http and socks5
- Debug mode to display outgoing and incoming http headers (default off, debug = 1 to turn on)

Please test and report issue. Have fun!


Attached File(s)
.zip  ProxHTTPSProxy 0.3.zip (Size: 7.7 KB / Downloads: 540)
Add Thank You Quote this message in a reply
Post Reply 


Forum Jump: