PhantomJS is a scriptable headless WebKit Browser. It is great for automatic website testing, site scraping, screenshot creation, page speed monitoring and many more things. Its Javascript API is pretty straightforward and easy to use. Beside some other projects, i wanted to use it to check how sites react to requests send along with a referrer. Some Sites use it to show you a welcome message, redirect you to special sites, show the latest visitors and their referrer (;-)) etc and i wanted to analyze this bevahiour on a larger scale. Also Piwik and Analytics use the document.referrer in their stats…

A message shown to people coming from another site

Example of a referrer based welcome message

PhantomJS (as of version 1.6.1, released Jul 24th 2012) offers a way to set custom request headers using

page.customHeaders = {...}

. It works great to the extent that it creates and sends the http request with your custom headers. It fails however to set the

document.referrer

value as accessible by Javascript on the website. So we would not see any Javascript based customizations based on the referrer with the current setup. Thats a pitty.

I created a workaround to fake the referrer with a phantomJS Script, and a patch that changes the behaviour of phantomJS. You can use both, depending on your needs. The Workaround includes an additional pageload for every requests and is not 100% reliable since its based on injecting a link to the (referring) page and simulating the click. This is probably not what you want for larger scale tests and scans.

You can find my basic fix on Github. It passes your custom Headers to the request object and thereby makes the referrer visible to the page: https://github.com/papoms/phantomjs/commit/6f738b06a02e2aaee1c18859b76132e5932334e0

And here is the script-only workaround for phantomJS. It Loads the page you want to show as the referrer. Injects a link using Javascript and triggers a MouseEvent to click the link. https://gist.github.com/3453137 and https://gist.github.com/3452369 for a CasperJS based version.

 

Share or like this article, to let me know if it helped you!

Please leave a comment if you have any questions or suggestions! You can also follow me on twitter.

  • Google Plus
  • Facebook
  • Twitter
  • Delicious
  • LinkedIn
  • StumbleUpon
  • Add to favorites
  • Email
  • RSS
 

4 Responses to Fake Referrer with PhantomJS

  1. Joey sagt:

    Hello! I am very interested in using this code. I don’t really know JS very well and mostly work with php. Could I possibly get an example page of this. One that you would use on your personal website or something. Just so I can see what needs to be changed and what all the values need to be and etc… It would be greatly appreciated!

    • Paul sagt:

      Hey Joey,

      i am not sure what you want to achieve. PhantomJS is not something that you can use “on your personal website or something” but an actual webbrowser used for things like automated testing of web applications. If you let me know what you want to do i might help anyways.

  2. Michael sagt:

    Thank you, this was exactly my issue and what I needed to implement.. I can’t believe it’s not good enough setting the custom header, you have to go as far as emulating the process… Much help, thanks again!

    -Michael

  3. Mateusz sagt:

    Hey Paul,
    I am trying to solve similar problem to yours from almost 6 houres. Your workaround fix works fine, but it is too much expensive for my software. Anyway, I can’t achieve the same effect with your phantomjs fork from github. It is not passing referrer. I really don’t know why. I should set only customHeaders and that’s all, yes? Can you present us some examples?

    Thanks in advance.

Hinterlasse eine Antwort

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind markiert *

Du kannst folgende HTML-Tags benutzen: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Email