Post Reply 
ProxHTTPSProxy, a Proxomitron SSL Helper Program
May. 25, 2010, 08:27 AM (This post was last modified: May. 25, 2010 08:31 AM by whenever.)
Post: #61
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 25, 2010 05:32 AM)JJoe Wrote:  Scroogle search with KMeleon set to

Great finding!

It is what I had worried about in the 0.1 version, where I put below comments in the code:

Quote:# Read 512 bytes to extract url path
# Will it cause issue not reading all data?

This might be the same reason why sometimes the browser can't receive the 307 redirect during your initial tests under win 7.

I finally could reproduce the issue on my machine. Please let me know if the version 0.2b fixes the issue on your machine.

(May. 25, 2010 05:32 AM)JJoe Wrote:  what about http://ssl.scroogle.org:443/sslnote.html?

It's a issue produced by ProxHTTPSProxy's redirecting, maybe we'd better fix it in ProxHTTPSProxy so it can be used without Proxo?

On the other hand, did you notice under some situations the referer header was not resent after 307 redirect?

(May. 25, 2010 05:32 AM)JJoe Wrote:  Couldn't a simple script detect us? document.location.href?

Yes, it can. I have already met a site doing that.


Attached File(s)
.zip  ProxHTTPSProxy 0.2b.zip (Size: 1.88 KB / Downloads: 774)
Add Thank You Quote this message in a reply
May. 27, 2010, 03:53 PM
Post: #62
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 25, 2010 08:27 AM)whenever Wrote:  This might be the same reason why sometimes the browser can't receive the 307 redirect during your initial tests under win 7.

Now you are just trying to make me feel better. Wink

I'll have to get back to you.

Later

Have fun
Add Thank You Quote this message in a reply
May. 28, 2010, 04:49 AM (This post was last modified: May. 28, 2010 04:51 AM by JJoe.)
Post: #63
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 25, 2010 08:27 AM)whenever Wrote:  Please let me know if the version 0.2b fixes the issue on your machine.

I believe it is fixed or much better. I would have expected to see the error by now.

(May. 25, 2010 08:27 AM)whenever Wrote:  
(May. 25, 2010 05:32 AM)JJoe Wrote:  what about http://ssl.scroogle.org:443/sslnote.html?

It's a issue produced by ProxHTTPSProxy's redirecting, maybe we'd better fix it in ProxHTTPSProxy so it can be used without Proxo?

If I understand, the referer header could be modified by ProxHTTPSProxy. This would only hide us from the less determined. The browser still has the 'incorrect' info and a script could access it and detect us. Proxo or ProxHTTPSProxy could, I think, add some code to hide or correct the 'incorrect' info. Having ProxHTTPSProxy add the script (a slippery slope) might complicate things for ProxHTTPSProxy's author and Proxo users.
Hiding is hard to do anyway.
I think.

(May. 25, 2010 08:27 AM)whenever Wrote:  without Proxo?

Isn't "ProxHTTPSProxy, a Proxomitron SSL Helper Program"? Wink

(May. 25, 2010 08:27 AM)whenever Wrote:  On the other hand, did you notice under some situations the referer header was not resent after 307 redirect?

I have not noticed a missing header but my set adds one by default. I haven't had time to modify things to really analize ProxHTTPSProxy. Even if I did (I will), I don't use the internet like I did when Scott was with us. I've forgotten a lot and some has changed.

So... I see 8 downloads of ProxHTTPSProxy 0.2b.zip. I download my uploads to make sure they work. Whenever, JJoe, ProxRocks, Graycode (not a User of the Proxomitron), and 4 unknowns. How about some feedback? Need some help getting it running? Just open another thread, if you are worried about spoiling this one. Please include 'ProxHTTPSProxy' in the title.

@Whenever,
Do you have a thread for this at your forum?
Can you host ssl pages and forms?
Is is ok to announce at the Prox-List?
Add Thank You Quote this message in a reply
May. 28, 2010, 06:56 AM
Post: #64
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 28, 2010 04:49 AM)JJoe Wrote:  If I understand, the referer header could be modified by ProxHTTPSProxy.

Yes, most headers even the POST body could be modified.

(May. 28, 2010 04:49 AM)JJoe Wrote:  This would only hide us from the less determined. The browser still has the 'incorrect' info and a script could access it and detect us.

I don't know a script could detect that too. What I see most is the server side detecting the referer info.

(May. 28, 2010 04:49 AM)JJoe Wrote:  Proxo or ProxHTTPSProxy could, I think, add some code to hide or correct the 'incorrect' info.

We'd better let Proxo do the job which it is good at. I suppose most users won't learn python just to add some code to ProxHTTPProxy. Needless to say it is impossible to modify the code if they select to use the exe build.

(May. 28, 2010 04:49 AM)JJoe Wrote:  Isn't "ProxHTTPSProxy, a Proxomitron SSL Helper Program"? Wink

Well, I think it might could be used with Graycode's proxy too. I remember Graycode is not going to filter https in his proxy.

(May. 28, 2010 04:49 AM)JJoe Wrote:  I haven't had time to modify things to really analize ProxHTTPSProxy.

I am currently using HttpFox to watch the http headers, which is handy.

(May. 28, 2010 04:49 AM)JJoe Wrote:  Do you have a thread for this at your forum?

Not yet. There are not experienced users like here.

(May. 28, 2010 04:49 AM)JJoe Wrote:  Can you host ssl pages and forms?

Hosting ssl needs unique ip which costs $3.95 monthly. If it is only for testing purpose, we can set up a local https server easily: http://nginx.org/en/docs/http/configurin...rvers.html

(May. 28, 2010 04:49 AM)JJoe Wrote:  Is is ok to announce at the Prox-List?

Sure. I uploaded an exe version here in case somebody doesn't like to install python.
Add Thank You Quote this message in a reply
May. 28, 2010, 01:14 PM
Post: #65
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 28, 2010 04:49 AM)JJoe Wrote:  Whenever, JJoe, ProxRocks, Graycode (not a User of the Proxomitron), and 4 unknowns. How about some feedback? Need some help getting it running? Just open another thread, if you are worried about spoiling this one. Please include 'ProxHTTPSProxy' in the title.

my apologies for the delays on my feedback...
the day after i installed everything and "started" my testing, the office sent me out of town for a couple of days...

since i was already "half way" to the folks house, i added a couple vacation days to catch an out-of-state visit...

i'll be back to "my" computer next Wednesday/Thursday...


my "initial" tests 'seemed to' have everything working, but i can't say as i'm 100% positive of that Sad
Add Thank You Quote this message in a reply
May. 28, 2010, 01:15 PM
Post: #66
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 28, 2010 06:56 AM)whenever Wrote:  I uploaded an exe version here in case somebody doesn't like to install python.

many thanks!
Add Thank You Quote this message in a reply
May. 28, 2010, 06:04 PM
Post: #67
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 28, 2010 04:49 AM)JJoe Wrote:  ... Graycode (not a User of the Proxomitron) ... How about some feedback?

I'm still thinking about this, but didn't want distract whenever with 'opinion' too much. But since you asked, here's a few things:

1. SSL Closure
I've reviewed my (very old) coding for OpenSSL, and the closure issue was raised a lot. Scan through the HTTPS RFC 2818 and you'll see premature SSL closure avoidance mentioned many times.

Within the Python there is commented out:
Code:
# shutdown cause a hard-to-hanle Exception on ssl_sock.read() (closed?)
# socket.SHUT_WR == 1
#ssl_sock.shutdown(1)
I think that shudown() should be used, assuming it means the same in Python as it does in C sockets. Shutting down for writing tells the local socket layer to send a FIN after any pending outbound data has been sent. At that point the client's socket layer should end the connection, which will be known by the receipt of 0 bytes. I looked a bit for an OpenSSL "flush" method, but couldn't spot it and wouldn't know if Python incorporates that anyway.


2. Non-SSL on port 443
TCP port 443 is supposed to be used only for SSL. You've already encountered Polipo refusing non-SSL through that port, and you may encounter that with other proxies or applications. Currently the code is using port 443 for both SSL and for non-SSL. One of my very early strategies was to use a different port to "key in" on what wasn't really encrypted data.

Consider having an option to substitute a different port, for example port 30443. The 307 redirection could send the user to http: on that port instead of :443.

From Proxo you would forward any requests for SSL or for port :30443 to ProxHTTPSproxy. I'm not so well versed in Proxomitron, hopefully it can do that.

The option could have additional benefit in debugging. For example http://some-ssl-server:30443/stuff.htm should become visible as plain text through ProxHTTPSproxy even without consideration of 'proxcert.pem' or 307 redirection.

Sorry about my lack of Python knowledge, but here is what I'm trying to suggest:
Code:
## alternate port option for processing 307-redirected SSL
alt_port = ':30443'

Code:
def do_METHOD(self):
    method = self.command
    https_url = 'https://%s' % self.path.split('http://')[1]
    if (alt_port != ''):        ## substitution port option
        https_url = https_url.replace(alt_port,':443',1)
Normally I'd say to modify the Host: header as well but I see you're dropping that header. I assume that's a Python UrlLib reason.

Code:
def do_CONNECT(self):
    host_port = self.path
    if (alt_port != ''):        ## substitution port option
        host_port = host_port.replace(':443',alt_port,1)


3. Do you really need the 307 Redirect?
This is the biggest thing because here I might suggest that whenever goes on a wild goose chase.

Consider the original objective:
(May. 19, 2010 06:56 AM)whenever Wrote:  One of the purposes I decided to learn Python Programming Language is to write a proxy which can do https interception to solve Proxomitron's SSL issue, so here comes the ProxHTTPSProxy, a Proxomitron SSL Helper Program. Wink

If you want to expand into a full-blown proxy that's one thing. You did get around Proxo's SSL issue. But then maybe it goes astray and introduces unforseen issues.

(I think) Proxo's SSL problem went away with your use of 'proxcert.pem'. That certificate would have been generated simply and does not contain the many extensions that have been added to (Open)SSL. With that certificate, Proxo and its SSLEAY libraries should stay happy because the newer SSL extensions aren't in there.

The problem (I think) arises when servers present newer more complicated certificates with extensions that the old SSLEAY versions didn't know about.

Maybe you should look into abandoning the 307 completely. In its place, collect those decrypted request headers (and any data) very similar to what's now done within do_METHOD. Don't issue a 307, but instead implement a data tunnel whereby you're handling all the (newer) SSL for the target server side and pumping back SSL to Proxo using the (old) proxcert.pem. You could probably implement a single-use tunnel by forcing a 'Connection: Close' on the server's response headers being given back to Proxo.

Let Proxomitron use its SSLEAY. Let it manage the in/out HTTP headers and filter the data. But defend it from newer incompatible SSL methods.

Proxo(using SSLEAY) --- (proxcert.pem) ProxHTTPS (any SSL method) --- secure servers.

So, I'm talking about an SSL-only solution. One that only has the do_CONNECT() and does not need (or does not have) the do_METHOD().

Maybe that's not realistic or technically possible. Or maybe it wouldn't really resolve anything.
Add Thank You Quote this message in a reply
May. 28, 2010, 07:35 PM
Post: #68
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 28, 2010 06:56 AM)whenever Wrote:  Hosting ssl needs unique ip which costs $3.95 monthly. If it is only for testing purpose, we can set up a local https server easily: http://nginx.org/en/docs/http/configurin...rvers.html

Local is fine.

(May. 28, 2010 06:56 AM)whenever Wrote:  Sure. I uploaded an exe version here in case somebody doesn't like to install python.

How was it generated?
Add Thank You Quote this message in a reply
May. 29, 2010, 03:45 PM (This post was last modified: May. 29, 2010 03:47 PM by whenever.)
Post: #69
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 28, 2010 06:04 PM)Graycode Wrote:  I'm still thinking about this, but didn't want distract whenever with 'opinion' too much.

Your comments are very helpful and always make ProxHTTPSProxy's version up. Don't hesitate to share your ideas with us.

(May. 28, 2010 06:04 PM)Graycode Wrote:  I think that shudown() should be used

I hadn't realized it is so important but I finally worked out a way to use it without raising the exception.

JJoe, could you please see if the 0.2c works on your system?

(May. 28, 2010 06:04 PM)Graycode Wrote:  TCP port 443 is supposed to be used only for SSL. You've already encountered Polipo refusing non-SSL through that port, and you may encounter that with other proxies or applications.

Consider having an option to substitute a different port, for example port 30443.

Polipo is not refusing non-SSL through port 443. What JJoe met is a bug of the 307 redirecting which has been addressed since version 0.2b (I hope so).

Will your proxy refuse non-SSL content on port 443?

Http and SSL can be configured to listen on any port, though 80 & 443 are mostly used. So far I don't see the importance to substitute a different port. If it causes problem in the future or most of others would like to have it changed, I will do it.

(May. 28, 2010 06:04 PM)Graycode Wrote:  The problem (I think) arises when servers present newer more complicated certificates with extensions that the old SSLEAY versions didn't know about.

The SSL error arises randomly on the SAME site I often visited. Could this eliminate the possibility you are talking about?

(May. 28, 2010 07:35 PM)JJoe Wrote:  How was it generated?

cx_Freeze, much easier than py2exe.


Attached File(s)
.zip  ProxHTTPSProxy 0.2c.zip (Size: 1.89 KB / Downloads: 712)
Add Thank You Quote this message in a reply
May. 29, 2010, 07:37 PM
Post: #70
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 29, 2010 03:45 PM)whenever Wrote:  I hadn't realized it is so important but I finally worked out a way to use it without raising the exception.

Great. HTTPS works with blocks of data, usually getting queued up by the SSL layer, so proper closure is more important than when using regular sockets. It seems like a factor that Python's SSL implementation should have been aware of and perhaps should have taken care of it internally.

Quote:Will your proxy refuse non-SSL content on port 443?

Yes in its default configuration. There's options to allow it, so it's not an issue.

The ports where a proxy's SSL CONNECT method are allowed is much more important in terms of real security, but that's not relevant here.

Quote:So far I don't see the importance to substitute a different port.

That's fine. It might be desirable at some point in the future, or maybe never. It's a concept I was playing with back when I had partial SSL in the proxy.

Quote:The SSL error arises randomly on the SAME site I often visited. Could this eliminate the possibility you are talking about?

Not really. Newer certificate extensions may be triggering SSLEAY issues that only occasionally manifest into the visible problem. If it's related to block size assumptions, uninitialized memory when doing extensions, etc. then sometimes SSLEAY may work, sometimes die, or could sometimes just not produce the correct output. The randomness of the error showing up implies to me that may be the case.

By providing only simple certificate methods that were commonly used long ago (proxcert.pem) then SSLEAY should not encounter anything new that it didn't already know how to handle well. I could be wrong though!
Add Thank You Quote this message in a reply
May. 29, 2010, 07:39 PM (This post was last modified: May. 29, 2010 10:55 PM by JJoe.)
Post: #71
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 29, 2010 03:45 PM)whenever Wrote:  JJoe, could you please see if the 0.2c works on your system?

I believe it does.

(May. 29, 2010 03:45 PM)whenever Wrote:  
(May. 28, 2010 06:04 PM)Graycode Wrote:  You've already encountered Polipo refusing non-SSL through that port,

Polipo is not refusing non-SSL through port 443. What JJoe met is a bug of the 307 redirecting which has been addressed since version 0.2b (I hope so).

I think my Polipo is simply refusing non-SSL through 443.
In the log that follows, Proxo was set to use Polipo on port 8123

Quote:+++GET 1752+++
Using Proxy - 127.0.0.1:8123
GET http://ssl.scroogle.org:443/ HTTP/1.1
Host: ssl.scroogle.org:443
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.20) Gecko/20081217 Firefox/2.0.0.20
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Cache-Control: max-age=0
Connection: keep-alive
Browser reload detected...

+++RESP 1752+++
HTTP/1.1 403 Forbidden port
Connection: keep-alive
Date: Sat, 29 May 2010 19:00:57 GMT
Content-Type: text/html
Content-Length: 468
Pragma: no-cache

Quote:403 Forbidden port

The following error occurred while trying to access http://ssl.scroogle.org:443/:

403 Forbidden port
Generated Sat, 29 May 2010 14:00:57 Central Daylight Time by Polipo on E3-PC:8123.

EDIT:
I should add that at the moment this does not matter. ProxHTTPSProxy is 'correcting' the address before Polipo sees it. Which is probably what whenever meant to say.
Add Thank You Quote this message in a reply
May. 30, 2010, 03:47 PM
Post: #72
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 29, 2010 07:39 PM)JJoe Wrote:  I should add that at the moment this does not matter. ProxHTTPSProxy is 'correcting' the address before Polipo sees it. Which is probably what whenever meant to say.

Yes, you won't get that '403 Forbidden port' error if you chain polipo after ProxHTTPSProxy.

(May. 28, 2010 06:04 PM)Graycode Wrote:  (I think) Proxo's SSL problem went away with your use of 'proxcert.pem'. That certificate would have been generated simply and does not contain the many extensions that have been added to (Open)SSL. With that certificate, Proxo and its SSLEAY libraries should stay happy because the newer SSL extensions aren't in there.
...
Maybe you should look into abandoning the 307 completely.
...
Let Proxomitron use its SSLEAY. Let it manage the in/out HTTP headers and filter the data. But defend it from newer incompatible SSL methods.

I agree the new method is much better than the 307 method if it works as you guessed. I made a just seems working version in a hurry. Please report if it doesn't work.


Attached File(s)
.zip  HTTPSProxy 0.1.zip (Size: 1.72 KB / Downloads: 683)
Add Thank You Quote this message in a reply
May. 30, 2010, 06:48 PM (This post was last modified: May. 30, 2010 09:28 PM by JJoe.)
Post: #73
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 30, 2010 03:47 PM)whenever Wrote:  Please report if it doesn't work.

It appears to work as well as the other but I can't complete the login at https://login.yahoo.com/ while using either. Can anybody?
I'll have to investigate later.

TIA

EDIT:
Should there be a separate thread for HTTPSProxy?
Add Thank You Quote this message in a reply
May. 31, 2010, 01:30 AM (This post was last modified: May. 31, 2010 02:13 AM by JJoe.)
Post: #74
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 30, 2010 06:48 PM)JJoe Wrote:  I can't complete the login at https://login.yahoo.com/ while using either. Can anybody?
I'll have to investigate later.

I suspect the set-cookie header.

Without HTTPSProxy, I see multiple set-cookie headers in the Proxomitron's log window, each contains one cookie, and the cookies (B,F,Y,PH,T,SSL) are set.

With HTTPSProxy, I see one set-cookie header, usually it contains only the B cookie, once it contained all 6 cookies separated by commas (I think), (I think) only the B cookie has been set.

I have increased "ssl_sock.settimeout". No joy.

EDIT:
Perhaps "continuing problem with httplib multiple set-cookie headers" at http://bugs.python.org/issue1660009 ?
Add Thank You Quote this message in a reply
May. 31, 2010, 02:21 AM (This post was last modified: May. 31, 2010 02:23 AM by whenever.)
Post: #75
RE: ProxHTTPSProxy, a Proxomitron SSL Helper Program
(May. 30, 2010 06:48 PM)JJoe Wrote:  It appears to work as well as the other but I can't complete the login at https://login.yahoo.com/ while using either. Can anybody?

I can't login to yahoo either. I hadn't figured out why but during my test I came cross the ssl start error again. Banging Head

The good thing is I think I might find out what triggered that error. When the error happened the Proxo log window show:

Code:
+++GET 19594+++
CONNECT / HTTP/1.1
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.9.9
Proxy-Connection: keep-alive
Host: mail.google.com
Referer: http://slashdot.org/search/referrer-karma.php?q=Big+Bang
+++CLOSE 19594+++

Graycode, is it normal a "/" come after "CONNECT" method? How could that happened? I think Proxo doesn't get the host:port where to establish the tunnel so it popped up the error message box.

Another ssl error I often see is ssl shutdown error.

I think these two kinds of errors are supposed to happen BEFORE and AFTER the ssl certificate thing, so HTTPSProxy might couldn't be helpful.

(May. 30, 2010 06:48 PM)JJoe Wrote:  Should there be a separate thread for HTTPSProxy?

Just do it if you think it help to keep topics organized.


Attached File(s)
.png  ssl_start_error.png (Size: 7.77 KB / Downloads: 633)
Add Thank You Quote this message in a reply
Post Reply 


Forum Jump: