ProxHTTPSProxyMII: Reloaded
|
May. 09, 2018, 02:48 PM
Post: #211
|
|||
|
|||
RE: ProxHTTPSProxyMII: Reloaded | |||
May. 09, 2018, 04:09 PM
(This post was last modified: May. 10, 2018 06:05 PM by ryszardzonk.)
Post: #212
|
|||
|
|||
RE: ProxHTTPSProxyMII: Reloaded
(May. 09, 2018 02:48 PM)JJoe Wrote: Did you point browser to use ProxHTTPSProxyMII front server at 3129? Yes I did make sure of that. Also checked behaviour on Windows 10 & Edge (setting up proxy here was a bit tricky as it is done in system not in the browser) with same result just somewhat different certificate installation process. Than I was thinking maybe it has something to do with the software versions I use in Gentoo Linux server. - dev-lang/python-3.5.5-r1 - dev-python/colorama-0.3.9 - dev-python/pyopenssl-17.5.0 - dev-python/PySocks-1.6.7 (there is pysocks 1.6.8 available upstream) - dev-python/urllib3-1.22 Than it hit me. I started ProxHTTPSProxyMII in debug mode and so errors in the log and to my surprise few properly working GETS. It turned out that in browser when I clicked certain website for the first time I got an error. For second time it worked. Code: [17:55] 000 "[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:2091)" while trying to establish local SSL tunnel for [forum.openstreetmap.org:443] So it seems on the first try ProxHTTPSProxyMII saves required certificate but does not use it and sets up proper connection on the next try when it already has required certificate on the disk... uffff... Anyways and awaiting that fix for using certificate the first time too EDIT: I failed to get working whole chain Squid / ProxHTTPSProxyMII / privoxy / ProxHTTPSProxyMII and get number of SSL errors So far I have been able to get working either: - Squid (http&https) / privoxy (http only) - ProxHTTPSProxyMII / privoxy / ProxHTTPSProxyMII It might be that squid by doing its own ssl mangling somehow will not work with ProxHTTPSProxyMII which does recreate ssl certificates too. Maybe the solution could be installing that CA.crt somewhere in the system for squid? EDIT2: I tried like 100 different combinations for squid-3.5.27 parent proxy cache_peer & sslproxy_flags which would help accepting ssl connection from ProxHTTPSProxyMII to squid like Code: sslflags=DONT_VERIFY_PEER,DONT_VERIFY_DOMAIN Code: 2018/05/10 19:25:18 kid1| Error negotiating SSL on FD 12: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol (1/-1/0) while response from ProxHTTPSProxyMII to squids request looked as follows Code: 127.0.0.1 - - [10/May/2018 20:02:06] code 400, message Bad request syntax ('\x16\x03\x01\x013\x01\x00\x01/\x03\x03¡1d\xad¿²\x81C¯\x0f{\x8câøÛÑxñ\x94úc3y\x97î²xå=Ü-\x87\x00\x00¬À0À,À(À$À\x14À') Is it even possible to connect the two together? |
|||
May. 11, 2018, 02:42 AM
Post: #213
|
|||
|
|||
RE: ProxHTTPSProxyMII: Reloaded
(May. 09, 2018 04:09 PM)ryszardzonk Wrote: So it seems on the first try ProxHTTPSProxyMII saves required certificate but does not use it and sets up proper connection on the next try when it already has required certificate on the disk... uffff... I haven't noticed this. I have rarely seen the connection timing out before the address is resolved on the first visit. The cached certificate might help but it has been too rare to study. When this happens on Windows the ProxHTTPSProxyMII log shows something like Quote:(Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x05C67630>: Failed to establish a new connection: [Errno 11001] getaddrinfo failed',)) This is more of a missing feature of urllib3 than a bug. The time limit can be increased. (May. 09, 2018 04:09 PM)ryszardzonk Wrote: It might be that squid by doing its own ssl mangling somehow will not work with ProxHTTPSProxyMII which does recreate ssl certificates too. Maybe the solution could be installing that CA.crt somewhere in the system for squid? Browser>>ProxHTTPSProxyMII front>>Privoxy>>ProxHTTPSProxyMII Rear>>Squid Assuming that you haven't disabled verification in ProxHTTPSProxyMII, did you add Squid's certificate to ProxHTTPSProxyMII certificate store (cacert.pem), like you added ProxHTTPSProxyMII's to the browser's? If the chain is as above, you could disable verification in ProxHTTPSProxyMII and certificate creation in Squid. Then ProxHTTPSProxyMII would hide the mitm from the browser and Squid would verify the site's certificates. (May. 09, 2018 04:09 PM)ryszardzonk Wrote: EDIT2: What about: Code: sslcipher=... The list of valid SSL ciphers to use when connecting Because, the errors below seem to show a failure to match protocols and/or cyphers. https://www.ssllabs.com/ssltest/viewMyClient.html may help. (May. 09, 2018 04:09 PM)ryszardzonk Wrote: (May. 09, 2018 04:09 PM)ryszardzonk Wrote: Is it even possible to connect the two together? Probably JJoe Wrote:I think I understand but I haven't actually done it. |
|||
May. 11, 2018, 05:23 PM
(This post was last modified: May. 11, 2018 05:26 PM by ryszardzonk.)
Post: #214
|
|||
|
|||
RE: ProxHTTPSProxyMII: Reloaded
One thing I noticed for the past two weeks I started fooling with caching/filtering SSL stuff is that is like steeping on the mine field. Fixing one problem leads to 2 more and the chain seems infinite. Having that said now to the fun part
(May. 11, 2018 02:42 AM)JJoe Wrote: I haven't noticed this. I have rarely seen the connection timing out before the address is resolved on the first visit. The cached certificate might help but it has been too rare to study.I do not notice much of a delay browsing the web pages on my i3 Ironlake server for it to be timing issue. If it is however than any idea what I might change to increase time for a website to react properly? Urllib on the other hand has over 100 open bugs. Some of the include certificates and one even certificates and squid. https://github.com/urllib3/urllib3/issues/1384 https://github.com/urllib3/urllib3/issues/476 (May. 11, 2018 02:42 AM)JJoe Wrote: Assuming that you haven't disabled verification in ProxHTTPSProxyMII, did you add Squid's certificate to ProxHTTPSProxyMII certificate store (cacert.pem), like you added ProxHTTPSProxyMII's to the browser's?I have not disabled verification in ProxHTTPSProxyMII, and it was very good idea to add my local Squid certificate to cacert.pem, but unfortunetly it did not work. My other idea was to use newest beta version of squid 4.0.24 with patch as it was to help with ssl, but that did not work either ( and yes I did update squid.conf to changes introduced in version 4) (May. 11, 2018 02:42 AM)JJoe Wrote: What about:I tried few more of those including those that disable some tls options like tls-options=NO_TICKET, tls-min-version=1.2 but with no luck either. By using tls-min-version=1.2 I even managed ProxHTTPSProxyMII to throw an exception after which to my surprise it kept working Code: Exception happened during processing of request from ('127.0.0.1', 36972) (May. 11, 2018 02:42 AM)JJoe Wrote: Browser>>ProxHTTPSProxyMII front>>Privoxy>>ProxHTTPSProxyMII Rear>>Squid This sounds very reasonable and just might work. I did not try it so far as I wanted to check all options at current configuration cause there is one thing that might be problematic with it. When I use iptables to direct 443 network traffic to 3129 (ProxHTTPSProxyMII FrontPort) it does not work for me. ProxHTTPSProxyMII would not accept intercepted traffic as squid does with intercept flag for transparent proxy. |
|||
May. 11, 2018, 08:16 PM
(This post was last modified: May. 11, 2018 08:28 PM by vlad_s.)
Post: #215
|
|||
|
|||
RE: ProxHTTPSProxyMII: Reloaded
ryszardzonk
Yes, I use squid to transparently proxy. Here are all the configs: privoxy Code: user-manual /usr/share/doc/privoxy/user-manual proxhttpsproxy Code: ### The parent proxy has to support CONNECT method, if you want to proxy HTTPS requests squid Code: # ДоÑтуп Ð´Ð»Ñ Ð»Ð¾ÐºÐ°Ð»ÑŒÐ½Ð¾Ð¹ Ñети. Note the selection in the config squid. It can not access the upstream proxy by the same type, so there are two 127.0.0.1 and 127.0.0.2. HTTP CONNECT method must be disabled for port 8118, and for 8079 it is allowed, so I managed to separate the different traffic. Rules iptables: Code: *nat |
|||
May. 12, 2018, 05:40 AM
(This post was last modified: May. 12, 2018 08:36 AM by ryszardzonk.)
Post: #216
|
|||
|
|||
RE: ProxHTTPSProxyMII: Reloaded
@vlad_s
Thanks It indeed worked. The secret lies in way squid handles ssl trafffic. You example has just basic SSL proxying Code: acl step1 at_step SslBump1 while mine has full caching requiring Squid to do SSL certificate recreation Code: ssl_bump peek all SSL Bump has led to ssl errors in the browsers even after adding to them certificate for Squid and ProxHTTPSProxyMII. I guess full ssl traffic caching with connection to other proxy doing its own ssl recreation is still ahead of us hence even squid 4 has number of bugs open. One problem I have now is that my squid logs get flooded with security warnings about 5 of them a second for that site Code: 2018/05/12 07:29:58 kid1| SECURITY ALERT: Host header forgery detected on local=172.217.20.195:443 remote=192.168.101.182:59425 FD 17 flags=33 (local IP does not match any domain IP) I have added code prior to cache_peer settings for it to go direct and skip squid cache alltogether but without any luck. Code: # Problematic SSL Sites EDIT: Those security alerts come from Chrome on Windows which does its own security checks and is not satisfied with certificates issued by ProxHTTPSProxyMII. On Android when one uses Chrome to install certificate it is accepted for whole device including other applications. Futher investigation ongoing... EDIT2: Search did not take to long... here and here Real question is can anything be done about that? |
|||
May. 12, 2018, 08:59 AM
Post: #217
|
|||
|
|||
RE: ProxHTTPSProxyMII: Reloaded
Yes, I have a basic configuration of squid, from which it is required to provide a transparent proxy. Do you want to cache encrypted traffic? As I understand from other users, it does not make sense. I modified the squid https://habr.com/sandbox/99037/ and used the keys for compilation no-verify-header and max_url 16k.
|
|||
May. 12, 2018, 04:16 PM
Post: #218
|
|||
|
|||
RE: ProxHTTPSProxyMII: Reloaded
(May. 12, 2018 08:59 AM)vlad_s Wrote: Do you want to cache encrypted traffic? As I understand from other users, it does not make sense.Back in the day of unencrypted websites and 28.8/56K modems that is exactly what a Squid was for. These days as sites are more dynamic and news from the morning in the evening are no longer news as they are so yesterday and some have access to 1GB Internet speeds this is not a much of a necessity and I would take transparent proxying of Squid + ProxHTTPSProxyMII/privoxy any day over the Squid with full caching but no advert filtering. That does not mean I would not wish that would work too and there was a way for Squid and ProxHTTPSProxyMII to talk to each other nicely and use each others abilities to the greater extend. (May. 12, 2018 08:59 AM)vlad_s Wrote: I modified the squid https://habr.com/sandbox/99037/ Following those instructions I prepared patch for 4.0.24 Code: diff -Naur squid-4.0.24.old/src/client_side_request.cc squid-4.0.24/src/client_side_request.cc Unfortunately that no longer works with that version Code: x86_64-pc-linux-gnu-g++ -DHAVE_CONFIG_H -DDEFAULT_CONFIG_FILE=\"/etc/squid/squid.conf\" -DDEFAULT_SQUID_DATA_DIR=\"/usr/share/squid\" -DDEFAULT_SQUID_CONFIG_DIR=\"/etc/squid\" -I.. -I../include -I../lib -I../src -I../include -I../src -Wall -Wpointer-arith -Wwrite-strings -Wcomments -Wshadow -Woverloaded-virtual -Wno-deprecated-register -pipe -D_REENTRANT -march=native -O2 -pipe -fgcse-sm -fgcse-las -fgcse-after-reload -ftree-vectorize -fabi-version=0 -c -o clientStream.o clientStream.cc (May. 12, 2018 08:59 AM)vlad_s Wrote: and used the keys for compilation no-verify-header and max_url 16k.Please elaborate. I did not really understood that last part. |
|||
May. 12, 2018, 04:28 PM
Post: #219
|
|||
|
|||
RE: ProxHTTPSProxyMII: Reloaded
(May. 12, 2018 05:40 AM)ryszardzonk Wrote: EDIT: Those security alerts come from Chrome on Windows which does its own security checks and is not satisfied with certificates issued by ProxHTTPSProxyMII. How do I replicate? To check Chrome I used https://portableapps.com/apps/internet/google_chrome_portable which shows green lock and 'certificate valid issued by ProxHTTPSProxy CA' etc... |
|||
May. 13, 2018, 06:16 AM
(This post was last modified: May. 15, 2018 05:10 AM by ryszardzonk.)
Post: #220
|
|||
|
|||
RE: ProxHTTPSProxyMII: Reloaded
(May. 12, 2018 04:28 PM)JJoe Wrote:What I have said has been right but only partially. The issues appeared the same in Chrome and in Edge, but not on Firefox on Windows 10. Everything was due to certificate installation which behave differently on those programs.(May. 12, 2018 05:40 AM)ryszardzonk Wrote: EDIT: Those security alerts come from Chrome on Windows which does its own security checks and is not satisfied with certificates issued by ProxHTTPSProxyMII. What I did is: I do have apache running I edited CA.crt file to remove private key from it and placed it on the local www site. Then when clicking in Firefox (both Windows and Linux) 192.168.1.1/CA.crt it properly installed for that browser. It went fine also for Chrome in Android. Chrome and Edge on Windows 10 however did not use internal browser repository, but used systems Certificate Installation creator. According to default settings used (Automatically select the certificate store based on the type of certificate) the creator had everything installed properly. I do not know if system deleted it, misplaced or did not used, but the result has been those security checks and certificate verification errors. To fix it I installed it not in the default store by clicking the certificate, but with this steps http://community.lightspeedsystems.com/d...indows-10/ ProxHTTPSProxyMII certificate authority all the sudden started working for both Edge and Chrome. EDIT: It turned out that my biggest problem running ProxHTTPSProxyMII was that my serwer and my client machine where running with unsynchronized clocks and my client's clock was behind servers by about 35 seconds. That have led to the number of logs showing every single time new website was reached for browsing Quote:[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:2091)" while trying to establish local SSL tunnel for [younameit.com:443]and warnings about improper certificate for the website in the client's web browsers. Looking closely at one of the warnings I noticed that it was certificate that was created by ProxHTTPSProxyMII which few seconds later without me doing anything got accepted. Why? It was according to my client created certificate was from the future therefore not yet valid... |
|||
May. 14, 2018, 09:15 PM
Post: #221
|
|||
|
|||
RE: ProxHTTPSProxyMII: Reloaded
That article was related to squid 3.5.X, fixes errors "SECURITY ALERT: Host header forgery detected on..."
At the expense of other issues, I compiled the packages and installed them and unfortunately did not describe what changed there, only in the name of the archive of these deb packages indicated the compilation keys. I need to see the "rules" file, it's just in the virtual machine. Later I'll see. max_url - almost nothing does not affect, but when you edit the file user.action, the squid can give an error when saving this file. |
|||
May. 15, 2018, 04:07 AM
Post: #222
|
|||
|
|||
RE: ProxHTTPSProxyMII: Reloaded | |||
May. 15, 2018, 08:45 AM
(This post was last modified: May. 15, 2018 06:54 PM by ryszardzonk.)
Post: #223
|
|||
|
|||
RE: ProxHTTPSProxyMII: Reloaded
(May. 14, 2018 09:15 PM)vlad_s Wrote: That article was related to squid 3.5.X, fixes errors "SECURITY ALERT: Host header forgery detected on..."Yeah since than I decided to use squid-4.0.24-20180410 which is most recent available (seems less problematic than stock 4.0.24). After all it present config Squid does not do any certificate validity checking leaving it all to ProxHTTPSProxyMII nor mangles encrypted traffic with ssl_bump therefore breaking TLS is less likely and experimental patches not all that needed. (May. 15, 2018 04:07 AM)JJoe Wrote:There is more staff that might be useful for others if You want(May. 13, 2018 06:16 AM)ryszardzonk Wrote: EDIT: I have prepared installation scripts for Gentoo Linux which prepare whole chain to have ProxHTTPSProxyMII in transparent proxy mode thanks to squid - adblock2privoxy - program converting any adblock filter into one understood by privoxy (I hope some one will step up some day and update that PCRE version understood by privoxy so there would be no need for that step) - squid - installation script for transparent proxy in version 4.0.24+ not yet available in Portage (Gentoo's package system) - ProxHTTPSProxyMII - You know that one :P As for script itself python packages should install somewhat differently in Gentoo to be available for all versions of python which I do not know how to do, but the script is good enough and only thing it is missing is creating new user in system to have it running not as root. I might add that some time in the future. For now most problematic with using network wide transparent proxy and ssl filtering is Google which changed default policy to not accepting locally issued CA certificates and those android apps simply stopped working in most cases. Adding information to FAQ about need to have your phone rooted or have Magisk installed which would allow CA.crt be system accepted without rooting the phone might also be welcomed PS adblock2privoxy installation script might be added into haskell overlay EDIT: Worth to note is also that Google Chromecast refuses to work with local CA. There is no option in the Chromecast application to setup proxy nor to accept locally issued CA. |
|||
May. 16, 2018, 04:28 PM
(This post was last modified: May. 16, 2018 04:29 PM by vlad_s.)
Post: #224
|
|||
|
|||
RE: ProxHTTPSProxyMII: Reloaded
(Apr. 09, 2018 12:57 AM)JJoe Wrote: Try https://curl.haxx.se/docs/caextract.html for a more current file. How can I extract the public key? The certificate does not help with the link. I need to browse the site of https://uslugi.tatarstan.ru/, the browser Waterfox 55 does not, say SEC_ERROR_UNKNOWN_ISSUER, IE does not retrieve the key (the button is gray), MS Edge does not know how or I do not know how. |
|||
May. 16, 2018, 04:49 PM
(This post was last modified: May. 16, 2018 04:58 PM by vlad_s.)
Post: #225
|
|||
|
|||
RE: ProxHTTPSProxyMII: Reloaded
I found a certificate, here it is: https://www.tbs-certificates.co.uk/FAQ/e..._2018.html
Strange, but even fresh Mozilla Firefox 59 gives an error on this site (link above). |
|||
« Next Oldest | Next Newest »
|