Post Reply 
ProxHTTPSProxyMII: Development
Jul. 04, 2016, 02:14 PM
Post: #226
RE: ProxHTTPSProxyMII: Development
(Jul. 04, 2016 07:04 AM)whenever Wrote:  Here is the explanation of the error: http://stackoverflow.com/questions/11866...roken-pipe

You can just ignore the warning if it doesn't break the page loading.

BTW, please try to update your Python3 installation and related modules. It may not be related but I have seen several socket and http fixes in the recent update.

ok. thanks for the heads up.

i updated everythin but still get that "maximum number of open connections reached". can i increase them somewhere or where is the limit? what limits me? Sad
Add Thank You Quote this message in a reply
Jul. 05, 2016, 01:43 AM
Post: #227
RE: ProxHTTPSProxyMII: Development
(Jul. 04, 2016 02:14 PM)kik0s Wrote:  i updated everythin but still get that "maximum number of open connections reached".

Could you please post the full log of the message?
Add Thank You Quote this message in a reply
Jul. 05, 2016, 09:02 AM
Post: #228
RE: ProxHTTPSProxyMII: Development
(Jul. 05, 2016 01:43 AM)whenever Wrote:  
(Jul. 04, 2016 02:14 PM)kik0s Wrote:  i updated everythin but still get that "maximum number of open connections reached".

Could you please post the full log of the message?

no log. its just shows up in my browser.
Add Thank You Quote this message in a reply
Aug. 25, 2016, 09:24 PM
Post: #229
RE: ProxHTTPSProxyMII: Development
I'm looking for pointers on adding transparent proxy support to ProxHTTPSMII. On linux this basically involves creating the socket with 2 additional options - SOL_IP and IP_TRANSPARENT:
Code:
fd = socket(AF_INET, SOCK_STREAM, 0);
/* - 8< -*/
int value = 1;
setsockopt(fd, SOL_IP, IP_TRANSPARENT, &value, sizeof(value));
/* - 8< -*/
name.sin_family = AF_INET;
name.sin_port = htons(0xCAFE);
name.sin_addr.s_addr = htonl(0xDEADBEEF);
bind(fd, &name, sizeof(name));
See https://www.kernel.org/doc/Documentation...tproxy.txt

Why? Using the tproxy netfilter module, we can transparently redirect TLS traffic to a local port (ProxHTTPSMII), process it and send the responses back to the browser with zero user intervention.
This works great on a home router for when guests arrive without proxypac/manual proxy setting on their device browsers.

When I first found ProxMII, it was the perfect side-grade to Privoxy, without resorting to using something like Squid, which is a huge program to maintain, even without caching.
I created an issue on Github, and the author, whenever, was clear that he may not have time to add transparent proxy support.
I'm hoping someone here can lead me in the right direction on how I might try and hack this functionality in.
It literally appears requiring adding 2 more socket options when you create the socket, specifically the Front Server socket. I could be horribly wrong, I merely write shell scripts.

I've looked at https://github.com/wheever/ProxHTTPSProx...oxyTool.py and https://docs.python.org/3.4/library/socketserver.html and it's not clear where Python sets socket options or if this is going to be as trivial as setting those options somewhere before
Code:
socket.socket(socket.AF_INET, socket.SOCK_STREAM)
If it's not, how much an effort would this be? I'd be happy to donate some or perhaps a bunch of us could donate for this feature.

Also of interest while I was looking up help for this is from - https://github.com/wheever/ProxHTTPSProx...PSProxy.py All the python programmers told me not to use http.server

Thanks for this great software. Any pointers are appreciated.
Add Thank You Quote this message in a reply
Sep. 12, 2016, 10:41 PM (This post was last modified: Sep. 12, 2016 10:41 PM by kik0s.)
Post: #230
RE: ProxHTTPSProxyMII: Development
(Aug. 25, 2016 09:24 PM)Strykar Wrote:  Why? Using the tproxy netfilter module, we can transparently redirect TLS traffic to a local port (ProxHTTPSMII), process it and send the responses back to the browser with zero user intervention.
This works great on a home router for when guests arrive without proxypac/manual proxy setting on their device browsers.

@stryker what do you mean by "zero user intervention" that? you still somehow need to specify a port for the ssl proxy. or do you want to put every request through a single port and then decide what kind of traffic that is?

@whenever i encountered another issue. when i get EOF errors the proxy freezes. the ports still appear open but no other request comes through. idk if its my setup but when i restart my proxhttps instance everything works fine again.
Add Thank You Quote this message in a reply
Sep. 13, 2016, 05:36 PM
Post: #231
RE: ProxHTTPSProxyMII: Development
(Sep. 12, 2016 10:41 PM)kik0s Wrote:  @stryker what do you mean by "zero user intervention" that? you still somehow need to specify a port for the ssl proxy. or do you want to put every request through a single port and then decide what kind of traffic that is?

I believe stryker wants an "intercepting proxy". See Juliusz Chroboczek post at https://sourceforge.net/p/polipo/mailman...506&page=1 .

Juliusz Chroboczek is the author of Polipo, which will no longer be maintained.
Add Thank You Quote this message in a reply
Sep. 14, 2016, 01:38 AM
Post: #232
RE: ProxHTTPSProxyMII: Development
(Sep. 12, 2016 10:41 PM)kik0s Wrote:  @stryker what do you mean by "zero user intervention" that? you still somehow need to specify a port for the ssl proxy. or do you want to put every request through a single port and then decide what kind of traffic that is?

Yes, the router/linux machine on which ProxHTTPSProxyMII is running, would transparently redirect traffic on port 443 (HTTPS) so nothing needs to be set in the browser proxy settings. This is also achievable via DHCP proxypac, but is a pain as it requires an HTTP server listening on port 80 to serve the actual wpad.dat to clients.

There are many ways of intercepting HTTP/HTTPS traffic. Transparent Proxying or TPROXY is a linux solution that does not involve redirecting TLS traffic which would break TLS.
Squid is a very popular caching HTTP/S proxy, you can review the different forms of interception here - http://wiki.squid-cache.org/ConfigExamples/Intercept
My approach is based on TPROXY as explained here - http://wiki.squid-cache.org/Features/Tproxy4
Add Thank You Quote this message in a reply
Sep. 14, 2016, 01:46 AM (This post was last modified: Sep. 14, 2016 01:48 AM by Strykar.)
Post: #233
RE: ProxHTTPSProxyMII: Development
(Sep. 13, 2016 05:36 PM)JJoe Wrote:  I believe stryker wants an "intercepting proxy".

Interesting read, I'm familiar with Polipo since the Tor group preferred it via Privoxy long back. The argument for Polipo not being a transparent interception proxy hold some water. After all it was designed with users seeking privacy/anonymity via Tor. However they aren't points that are valid reasons for not enabling interception in ProxHTTPSProxyMII.

ProxHTTPSProxyMII is by very nature designed to do interception. Unless the website uses certificate pinning, you are breaking TLS and performing a MITM without the average user knowing it.
Asking the user to choose a port for SSL proxy or providing it via DHCP proxypac isn't an improvement over transparent interception. Indeed, in some use cases like home guest wifi networks, it's often a pain.
Add Thank You Quote this message in a reply
Sep. 16, 2016, 08:32 PM
Post: #234
RE: ProxHTTPSProxyMII: Development
(Sep. 12, 2016 10:41 PM)kik0s Wrote:  @whenever i encountered another issue. when i get EOF errors the proxy freezes. the ports still appear open but no other request comes through. idk if its my setup but when i restart my proxhttps instance everything works fine again.

someone else with such an erreror? i guess its because of some pip packages got updated. on the privoxy forum there a russisn guy with the same problem Smile!
Add Thank You Quote this message in a reply
Sep. 16, 2016, 11:31 PM
Post: #235
RE: ProxHTTPSProxyMII: Development
(Sep. 16, 2016 08:32 PM)kik0s Wrote:  on the privoxy forum there a Russian guy with the same problem Smile!

Ahhh.

(Sep. 16, 2016 08:32 PM)kik0s Wrote:  someone else with such an error?

I am not sure.
I am using the exe on Windows 10 Home with Opera portable.
But maybe...

Back in June, I changed ISPs and routers.
Old service provider and old router were IPV4 only.
New service provider randomly assigned IPV6 or IPV4 public address.
New router's default setting is to support IPV4 or IPV6, and assign local IPV6 addresses.

New router, IPV4 public address, and ProxHTTPSProxyMII broke the internet. I don't remember the details.
New router, IPV6 public address, and ProxHTTPSProxyMII worked great.

My solution was to disable IPV6 in the new router's setup.
New router set to only allow IPV4, IPV4 public address, and ProxHTTPSProxyMII works great.
My guess was that one of the python packages (urllib3?) fails with IPV4 when the router shows IPV6 is possible.

What happens if you, set your router to only allow IPV4?
If you can...
Add Thank You Quote this message in a reply
Sep. 19, 2016, 11:31 PM
Post: #236
RE: ProxHTTPSProxyMII: Development
(Sep. 16, 2016 11:31 PM)JJoe Wrote:  
(Sep. 16, 2016 08:32 PM)kik0s Wrote:  on the privoxy forum there a Russian guy with the same problem Smile!

Ahhh.

(Sep. 16, 2016 08:32 PM)kik0s Wrote:  someone else with such an error?

I am not sure.
I am using the exe on Windows 10 Home with Opera portable.
But maybe...

Back in June, I changed ISPs and routers.
Old service provider and old router were IPV4 only.
New service provider randomly assigned IPV6 or IPV4 public address.
New router's default setting is to support IPV4 or IPV6, and assign local IPV6 addresses.

New router, IPV4 public address, and ProxHTTPSProxyMII broke the internet. I don't remember the details.
New router, IPV6 public address, and ProxHTTPSProxyMII worked great.

My solution was to disable IPV6 in the new router's setup.
New router set to only allow IPV4, IPV4 public address, and ProxHTTPSProxyMII works great.
My guess was that one of the python packages (urllib3?) fails with IPV4 when the router shows IPV6 is possible.

What happens if you, set your router to only allow IPV4?
If you can...

looked into this but its not the ipv6 or ipv4 issue. i have ipv4 only so thats not the problem.
Add Thank You Quote this message in a reply
Apr. 19, 2018, 01:56 AM (This post was last modified: Apr. 20, 2018 01:40 AM by JJoe.)
Post: #237
RE: ProxHTTPSProxyMII: Development

.zip  ProxHTTPSProxyMII_py 1.5wipa.zip (Size: 167.12 KB / Downloads: 571)

Changes:
Added SubjectAltNames support for DNS and IP... No guarantees, warranties, etc., but it appears to work.
Common name will no longer use a leading '*'. This means less code but a larger cert folder.
__version__ updated


Have Fun
Add Thank You Quote this message in a reply
[-] The following 3 users say Thank You to JJoe for this post:
Thomas S., vlad_s, Faxopita
Apr. 21, 2018, 09:04 PM
Post: #238
RE: ProxHTTPSProxyMII: Development

.zip  ProxHTTPSProxyMII_py 1.5wipb.zip (Size: 167.22 KB / Downloads: 635)

Changes:
'*' returns to cert's Subject field due to some hosts using more than the 64 characters that are allowed. Example: 18cfdfd73150f69310ab-4d842a0601d0ae955a714605e7fb6d6f.ssl.cf2.rackcdn.com.


Have Fun
Add Thank You Quote this message in a reply
[-] The following 3 users say Thank You to JJoe for this post:
Thomas S., vlad_s, Faxopita
Jun. 17, 2018, 02:12 AM
Post: #239
RE: ProxHTTPSProxyMII: Development

.zip  ProxHTTPSProxyMII_py 1.5.zip (Size: 171.89 KB / Downloads: 625)

Docs and etc updated, 1.5wipb becomes 1.5
Add Thank You Quote this message in a reply
[-] The following 5 users say Thank You to JJoe for this post:
ryszardzonk, Thomas S., referrer, mizzmona, vlad_s
Jul. 20, 2019, 06:48 PM
Post: #240
RE: ProxHTTPSProxyMII: Development
Why are hop-by-hop headers (line 233) removed from the ProxyTool.py file? Seeing this, Privoxy tries to get them back.
Add Thank You Quote this message in a reply
Post Reply 


Forum Jump: