Post Reply 
Content-Type Issue?
Jun. 21, 2010, 01:27 PM (This post was last modified: Jun. 21, 2010 04:37 PM by Inferno.)
Post: #1
Content-Type Issue?
Hello!
I'm new to the forum and still a beginner with proxomitron. I'm working with the program for some days now and it seems to work very fine for simple filtering. Unfortunately there is a big problem on websites using AJAX. I mean the program does not filter the content! It just filters on the very first request but not on sub-requests. Here is a response header that refers to my problem:

Code:
HTTP/1.1 200 OK
Date: Mon, 21 Jun 2010 13:18:47 GMT
Server: Apache
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding,User-Agent
Content-Encoding: gzip
Content-Length: 6182
Keep-Alive: timeout=15, max=95
Connection: Keep-Alive
Content-Type: application/json

After hours of searching and finally much headache I guess the Content-Type is the real problem. As I'm already mentioned...I am not very experienced with proxomitron, but I think the program does NOT filter anything which has a "application/json" content type?!
Please correct me if I'm wrong. It would be great if someone of you can give me a hint or even a good solution :-)

Thanks in advance for your efforts!
Add Thank You Quote this message in a reply
Jun. 21, 2010, 03:31 PM
Post: #2
RE: Content-Type Issue?
This should help: http://prxbx.com/forums/showthread.php?tid=218
Add Thank You Quote this message in a reply
Jun. 21, 2010, 05:29 PM (This post was last modified: Jun. 21, 2010 06:12 PM by Inferno.)
Post: #3
RE: Content-Type Issue?
Thanks for that link, but it still doesn't work Sad. I used the following code:
Code:
[HTTP headers]
In = TRUE
Out = FALSE
Key = "Content-Type: application/json"
Match = "(application/json)\0"
Replace = "\0$FILTER(true)"

I'm not sure if this is the solution I am searching for. Let me explain what I mean. A JavaScript function sends a request to "update.php" and the server returns the given headers+content. It returns application/json data and proxomitron should replace for instance all the words "City" with "Foo".
Unfortunately it doesn't replace the words I want (nothing is replaced). The returned data seems not parsed and therefore ignored by proxo.

I just want to force proxo to parse (and replace) the returned content but I have no idea how to do this. Hope you can help me with that Thumbs Up
Add Thank You Quote this message in a reply
Jun. 21, 2010, 10:17 PM
Post: #4
RE: Content-Type Issue?
something else in your config could be "protecting" the application/json...
ie, if you're using a sidki-based config, you will have a text file called "Content-Types" that might be telling your config to NOT filter the app/json's...

"http header filters" are parsed in a particular order so if one filter says to filter the app/json and a filter that is parsed later says NOT to filter the app/json, then the filter will NOT filter the app/json because the "last" filter said not to...

therefore, make sure that your filter that says TO filter app/json is the LAST "content-type" filter...

the easiest way to do this is to "number" your filters so that Proxo then uses them "in sequential order"...

let's say that the "highest sequential" content-type filter in your config is "titled" 'Content-Type: 6c Log Third Party Scripts', you could then RENAME your app/json filter to 'Content-Type: 6d app/json' and then you should be good-to-go...
Add Thank You Quote this message in a reply
Jun. 21, 2010, 10:54 PM (This post was last modified: Jun. 21, 2010 10:57 PM by Inferno.)
Post: #5
RE: Content-Type Issue?
Thank you very much ProxRocks. I'm using the standard/basic proxomitron filters which come with the program. I don't use any custom made filters Smile!
I even unchecked ALL header and web-page filters except the one specific header filter from above. Web-page filter is set to replace all words "City" with "Foo". This works fine with standard text/html but not application/json. I disabled all other header and web-page filters via the GUI.
If thats the wrong way to disable everything else, please tell me Smile!

Your reply is very appreciated. Thanks again!
Add Thank You Quote this message in a reply
Jun. 21, 2010, 11:13 PM
Post: #6
RE: Content-Type Issue?
not sure how many of us even "remember" anything about the filters that "come with" the program...

Proxo is only as good as the config you feed it - have you tried feeding it a "recent" config?
Add Thank You Quote this message in a reply
Jun. 21, 2010, 11:25 PM (This post was last modified: Jun. 22, 2010 12:38 AM by Inferno.)
Post: #7
RE: Content-Type Issue?
To make things easier now....I just downloaded the sidki-config and the latest update too! It's now loaded into my proxo (sidki_2009-02-13.ptron). I see a LOT of filters now, but not sure what to do next. Could you tell me how to set the application/json filter now? That would be very nice Smile!

EDIT: Well ok...I added my filter to the sidki-config now:
Code:
[HTTP headers]
In = TRUE
Out = FALSE
Key = "Content-Type: 8 application/json"
Match = "(application/json)\0"
Replace = "\0$FILTER(true)"

It is still not working and I really don't understand why. Again, it works fine for text/html but not application/json. I have to admit that I'm close to give up Sad
Add Thank You Quote this message in a reply
Jun. 22, 2010, 12:33 AM
Post: #8
RE: Content-Type Issue?
You'd better give us a URL to test.
Add Thank You Quote this message in a reply
Jun. 22, 2010, 12:56 AM (This post was last modified: Jun. 22, 2010 01:15 AM by Inferno.)
Post: #9
RE: Content-Type Issue?
I'm sorry this is not possible because you would need to login my account. If you really need a URL, I'm going to look for a public website where this particular problem occurs too.
Here is the complete header info for the AJAX request (maybe helpful for you). I edited the cookie information and URL for security reasons:
Code:
+++GET 1496+++
GET /request/update.rq.php?sid=97cd48h8fz49fg7gfogiaifg7agr4w8&page=dist&action=switch&ID=10248&update=page HTTP/1.1
Host: abc.domain.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3
Accept: text/javascript, text/html, application/xml, text/xml, */*
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
X-Requested-With: XMLHttpRequest
X-Prototype-Version: 1.6.0.1
Referer: http://abc.domain.com/index.php?sid=97cd48h8fz49fg7gfogiaifg7agr4w8
Cookie: __utma=123456789....................
Connection: keep-alive

+++RESP 1496+++
HTTP/1.1 200 OK
Date: Tue, 22 Jun 2010 00:47:55 GMT
Server: Apache
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Vary: Accept-Encoding,User-Agent
Content-Encoding: gzip
Content-Length: 6192
Content-Type: application/json
+++CLOSE 1496+++

I tried almost everything without success...even replacing application/json with text/html, text/plain, text/json...
Referring to that thread http://prxbx.com/forums/showthread.php?t...pplication there are several default content type filters. Is there any chance to extend that default list with application/json?
Add Thank You Quote this message in a reply
Jun. 22, 2010, 02:59 AM (This post was last modified: Jun. 22, 2010 03:01 AM by JJoe.)
Post: #10
RE: Content-Type Issue?
This filter should throw a flag when it finds application/json and should then enable filtering.

Code:
[HTTP headers]
In = TRUE
Out = FALSE
Key = "Content-Type: filter application/json (in)"
Match = " [^/]+/json&$ALERT(application/json found)&$FILTER(true)(^)"

Try it with both configs.
Make sure their aren't any bypass list entries related to this site.
After you import or add the filter, you must click "Apply" or "OK".

(Jun. 22, 2010 12:56 AM)Inferno Wrote:  I tried almost everything without success...even replacing application/json with text/html, text/plain, text/json...

Was the change shown in the Proxomitron's log window?
Are you sure your web page filters are good? Maybe you are 'missing' the target.

HTH
Add Thank You Quote this message in a reply
Jun. 22, 2010, 12:18 PM (This post was last modified: Jun. 22, 2010 12:50 PM by Inferno.)
Post: #11
RE: Content-Type Issue?
(Jun. 22, 2010 02:59 AM)JJoe Wrote:  This filter should throw a flag when it finds application/json and should then enable filtering.

Code:
[HTTP headers]
In = TRUE
Out = FALSE
Key = "Content-Type: filter application/json (in)"
Match = " [^/]+/json&$ALERT(application/json found)&$FILTER(true)(^)"
Hello JJoe!
I tried your filter and with each click on the webpage I get an alert window "The Proxomitron Says... application/json found".

(Jun. 22, 2010 02:59 AM)JJoe Wrote:  Was the change shown in the Proxomitron's log window?
Are you sure your web page filters are good? Maybe you are 'missing' the target.

I was able to see how the Content-Type changed depending on what I set (application/json --> text/html or text/plain...). As a conclusion I am not missing the target. Furthermore the website stopped working with text/plain or text/html. If I change the Content-Type to something different the website seems to be frozen because the browser doesn't know what to do with application/json data that is marked as text/html.

Also your help is very appreciated JJoe Thumbs Up

EDIT:
Alert works great but it still doesn't replace the words I want. It replaces my words every time I do a complete page refresh in my browser but all the AJAX requests update only the content of a <div> box. This data seems not parsed. I don't understand this because every single GET request goes through proxo, right?
Add Thank You Quote this message in a reply
Jun. 22, 2010, 12:56 PM (This post was last modified: Jun. 22, 2010 12:58 PM by Inferno.)
Post: #12
RE: Content-Type Issue?
I just did a little test:
I sent the whole GET request manually via browser and the alert window pops up.
Code:
http://abc.domain.com/request/update.rq.php?sid=97cd48h8fz49fg7gfogiaifg7agr4w8&page=dist&action=switch&ID=10248&update=page
After that my Opera browser offers me to download the "update.rq.php". If I look into that file, it contains typical json data and escaped html. The content is completely untouched and nothing was replaced.
Add Thank You Quote this message in a reply
Jun. 22, 2010, 12:59 PM (This post was last modified: Jun. 22, 2010 01:09 PM by JJoe.)
Post: #13
RE: Content-Type Issue?
(Jun. 22, 2010 12:18 PM)Inferno Wrote:  I was able to see how the Content-Type changed depending on what I set (application/json --> text/html or text/plain...). As a conclusion I am not missing the target.

This means the Header filter is matching but not the Web page filters.
You should be able to see the web page filter's Match in the Proxomitrons log window or you could add a flag/alert to the Web page filter.

(Jun. 22, 2010 12:18 PM)Inferno Wrote:  I don't understand this because every single GET request goes through proxo, right?

Probably but it depends. Request seen in the Log window may be filtered.

Have you seen the actual AJAX html? Maybe it's not what you expect.
Do you see any filter's "Match" for the AJAX requests in the Proxomitron's log window?
Enabling "HTML Debug info" before the AJAX request may show something.

"HTML Debug info" > right click on the Proxomitron's log window.

HTH
Add Thank You Quote this message in a reply
Jun. 22, 2010, 01:02 PM
Post: #14
RE: Content-Type Issue?
(Jun. 22, 2010 12:56 PM)Inferno Wrote:  If I look into that file, it contains typical json data and escaped html. The content is completely untouched and nothing was replaced.

Do your filters match this code?
You can use the Proxomitron's test window.
Add Thank You Quote this message in a reply
Jun. 22, 2010, 01:23 PM (This post was last modified: Jun. 22, 2010 01:25 PM by Inferno.)
Post: #15
RE: Content-Type Issue?
(Jun. 22, 2010 12:59 PM)JJoe Wrote:  This means the Header filter is matching but not the Web page filters.
The web-page filter is matching. I can see the pink matching lines in the log window.

(Jun. 22, 2010 12:59 PM)JJoe Wrote:  Have you seen the actual AJAX html? Maybe it's not what you expect.
Do you see any filter's "Match" for the AJAX requests in the Proxomitron's log window?
Enabling "HTML Debug info" before the AJAX request may show something.

"HTML Debug info" > right click on the Proxomitron's log window.
Yes I can see it matching 3 times...as it should be.

(Jun. 22, 2010 12:59 PM)JJoe Wrote:  Do your filters match this code?
You can use the Proxomitron's test window.

Yes my filter matches 3 times. I can see the pink info in the log window and red <Match: myFilter >City</Match> hints in the HTML debug code.
Add Thank You Quote this message in a reply
Post Reply 


Forum Jump: