by developers, for developers

Home
About
Events
Open Source
Vote for a Topic
Write for Techportal
Select a page

Manipulating HTTP with Charles Proxy

05 Mar 2013
Mar 5, 2013
Charles Proxy is exactly what its name implies; a proxy. This proxy is special however, as it is
specifically aimed at giving functionality that developers will need. It allows you to view, change
and replay the trac that passes through, and can handle SSL. Charles isnt free, but at $50 (at
the time of writing) for a license, it reaps returns from day one. This article will show you how to
start using Charles and covers techniques that will help you work with your web and mobile
applications; basically anything that works over HTTP.

Getting Set Up
You can download Charles Proxy from the project website: http://www.charlesproxy.com/.
Under the Documentation section you will find instructions for installing the tool onto
whichever platform you use (Charles supports Windows, Mac and Linux).

Firefox Extension
Charles oers a Firefox extension from its download page that makes it very easy to start using
your web applications with Charles immediately. Once you have the extension installed, simply
run Charles and then enable the extension from the Tools menu in Firefox. When a web request
is made in Firefox, the HTTP trac involved is routed through Charles. This is a great way to
get started and is the approach used to generate the first few examples in this article.

Inspecting Traffic in Charles

Charles isnt the only option for inspecting trac, but it is at least as fully featured as using
alternatives such as browser plugins such as REST Console or LiveHTTPHeaders, or capturing
trac using a tcpdump and/or Wireshark. Charles understands more types of trac than just
what comes through a browser, and has better support for storing, sharing, altering and
repeating requests than Wireshark.
By default in Charles, when you start to capture trac youll be looking at the Structure tab,
which shows all the requests that have been made, organised by URL. Each folder has a
domain, then there are subfolders for each element of the path. The view ends up looking
something like:

A more linear way to look at the trac is to use the Sequence tab, which will show something
like this:

Where to look next depends entirely what youre looking for. The filter box (below the sequence
of requests and above the detail section) is very smart, including the ability to configure it to
use a regular expression in its search, and will search all the fields shown in the grid. This
allows you to filter down the results to just requests with a given response or using a particular
verb, or to a particular domain or subdomain.
Once you have identified the request you want to see, check out the tabs in the bottom half of
the screen. These typically show the following options:
Overview: Some summary information about the request and response formats,
sizes, time taken, and so on. This can be useful when comparing similar requests.
Request: Full, detailed information about the request, headers, data sent. Look at
the dierent aspects using the tabs along the bottom of the window.
Response: Much like the request tab, the response tab has some very nice feaures,
in particular its ability to nicely represent data formats such as JSON which can be
viewed raw, pretty-printed, or parsed into a structure you can drill down into.
Summary: Shows all the requests that were made as a result of this request, such
as loading other assets to display the page, plus information about the overall size
and time of these. Mostly useful when requesting a root page which loads in other
elements such as images.
Chart: Also mostly useful when requesting a page that loads other elements, this
gives a timing diagram of things arriving as shown in the figure below:

Armed with this information, you can take a good look at whats taking the time in your
application, which pages are pulling in which other pages, and so on. Once you have identified
where a problem exists, we can use the following techniques to further diagnose or observe it.

Replay a Request
Charles oers the ability to repeat a specific request as many times as you want to. Either
right-click on the request to see the options and choose repeat, or press the blue circular
arrows button on the toolbar to repeat the request. As a debugging tool, this is much nicer than
having to fill in the same web form lots of times, or otherwise follow a tricky, multi-step process
to replicate a bug.
The right-click menu also oers the ability to repeat advanced, which allows the same
request to be made to the server multiple times, including the option to specify the
concurrency. If you are seeing weirdness under load or you suspect a race condition, then this
feature is your friend!

Sharing Requests
Its possible to import and export sessions with Charles, in HAR (HTTP ARchive) format, which
is supported by other tools as well. This is superbly handy when one developer is seeing a
problem and wants to show another one whats happening without the aforementioned tricky
process to reproduce the bug. Simply export your session from Charles and add it as an
attachment in the bug tracker; whether it is you or someone else that picks this up, it makes it
very easy to replicate the problem and be able to begin working on a fix.

Throttling Requests
This is a really good use of a local proxy; to throttle the trac going through your browser in
order to simulate a particular setup or network speed. For websites where it is likely that the
user is either using a mobile phone or is connected via a mobile data connection, this gives us,

as developers, insight into how the user experience will be in that scenario.
Charles oers a few options for speed restrictions to simulate dierent kinds of network
connection, shown in the screenshot below:

It is also possible to limit throttling to just a particular set of hosts, so if for example you know
that one particular 3rd party may provide slow connection speeds in some situations, then you
can narrow that down using these settings.
Throttling is a very important step in testing mobile applications, especially when working with
either mobile sites or with app simulators, since it gives a more realistic sense of the user
experience. I lead an open source project (http://joind.in) which is used mostly at conferences,
and conference wifi is usually fairly awful so I throttle down to old modem speeds to give
myself a sense of experience when developing that site.
Another feature of Charles which is relevant to this area of development is that it can disable
caching. This enables you to reproduce the experience that a new visitor to your site will see.
To achieve this, Charles changes the headers in the HTTP trac, a feature that we can take
direct advantage of ourselves.

Changing Requests and Responses

Sometimes, the request and responses that are made by an application arent exactly what we
need them to be. This can be useful for playing tricks such as redirecting trac from a mobile
app which assumes a particular URL for its API backend, or injecting a specific header to be
able to trace your request in the server logs or to enable debugging for requests coming in via
this route.
The rewrite functionality can be applied to all requests, or just a subset, and multiple rewrites
can be configured at one time. They are encapsulated into rule sets, so you can save the
settings of these, and then enable/disable them at will. For the rewrites themselves, there are
yet more choices; here is the input dialog for creating a rule, to give you an idea of whats
possible:

As you can see, the options here are endless. There are any number of reasons why you might
need to add things, but these are my most common use cases:
Injecting extra headers, either so that I can track a journey through the
site because the additional information gets added to a log file, or to
enable additional debugging.
Injecting response headers to see how a client would react on receiving
these from a server. HTTP supports the use of custom headers, so in
theory you can use any header you like so long as your server knows
how to understand it, and the RFC (see RFC 2616) recommends picking
a prefix. In practice, its common to use X- as the prefix for these
invented headers.
Modifying headers to alter a User-Agent header to chase down a
particular bug.
Setting Host headers easily the most-used, mostly when using other
machines or devices on a network to hit a local development virtual host
with a dierent domain name than expected by the application (more on
this in the next section showing how I develop with mobile devices
against my local development machine via Charles).
Adding a rule to do a find-and-replace on the body of a request or
response is also very useful at times, for example to rewrite links.

These are very useful tools and the biggest advantage is that you dont need to recompile or
reconfigure either the client or server to make these changes; Charles can intercept the trac
and make the changes on the fly. When developing a mobile app, where you might want to
change whether it hits live or development copies of the back end API for example, this is a
very quick and easy way to do it. It also protects against hardcoding these changes into your
application and then finding that they get erroneously committed and deployed.

Mobile Debugging and Re-routing

Charles can act as a proxy for other devices to route via the machine it runs on, allowing us to
observe and operate on trac to clients other than the Firefox plugin. To get set up is fairly
straightforward using the following steps.

First, check the IP address of your machine, and check or set which port Charles will listen on
by looking under Proxy, then Proxy Settings. By default this is 8888, and in this example my
machine has the IP 10.1.0.121.
Add either the machine hostname or IP to the device to proxy under the wifi settings for the
shared network. Im using an Android phone as Ive been looking at the updated Android app
for joind.in, although do be aware that some versions of Android (stock ROMs in the 2.x series,
also known as bug 1273) do not allow you to specify a proxy when adding network settings.

When you try to make a web request from the phone now, Charles will alert you that something
is trying to proxy through, and give you the opportunity to allow or deny the access:

While working with the mobile app, this is a very easy way to observe the trac being sent and
received, and to spot any errors that might be silently ignored or not handled by the application
(particularly as my application is in very early stages).
At this point we have the opportunity to perform all of the filtering and altering of trac that
were seen earlier when making web requests. In this scenario, sometimes I want to try the
mobile app against my local development version of the API. To do this, I have a rewrite that
simply routes anything for http://api.joind.in to http://api.joindin.local. This is in the hosts
file on my machine, so it knows that domain is on the local host. Here is the rewrite rule in
Charles:

When I tick and untick the joindin-api-dev box, I interrupt trac to the live API, and instead
send it to my local version of the API, which contains sample data and possibly local
modifications. This is such a powerful feature when doing mobile development, because the
data sources can be swapped without any need to restart the app, and certainly without
changing or rebuilding it. This tactic is useful both when the app cant be changed, and also
when its simpler to intercept its trac for testing.

Charles and SSL

In the interests of saving the best feature until last: Charles will also work with SSL trac.
SSL trac is usually very dicult to debug, by design, because nothing other than the client
and the server taking part in the transaction should be be able to see the trac. An encrypted
channel is established and everything is passed over it. So how can Charles intercept it?
What actually happens is that Charles executes a classic man in the middle attack. The
encrypted channel gets set up between Charles and the remote server using the real SSL
certificate. Charles then uses its own certificate to sign a second encrypted channel between
Charles and the browser or client. In order to protect you against man-in-the-middle attacks,
your browser wont accept a certificate that isnt from a trusted Certificate Authority unless you
specifically grant permission for it. This is fairly common on development servers where it
doesnt make sense to pay for an ocial SSL certificate, so you may have seen these types of
untrusted certificate warnings in such scenarios.
To use this feature, first of all you will need to enable Charles to proxy SSL trac, by ticking the
box in the Proxy Settings screen. You should also specify which domains should have their SSL
trac proxied.

As we already mentioned, Charles then intercepts the trac, using the remote sites correct
SSL certificate for the connection between there and Charles, then an untrusted one between
Charles and the client. This means that when you enable this in Charles, your browser will tell
you that there isnt a valid SSL certificate for the remote site.

Confirm the security exception and you can now proceed as normal with all the features
mentioned in the earlier sections of this post. Charles isnt the only tool to oer this
functionality, but its one of the easiest to get up and running with. When testing or debugging
for SSL setups, this is a very valuable addition to the toolbox, and one that I hope youll find
useful moving forward.

Charles, Web Debugging Proxy

This article aimed to give you an overview of some of the most useful aspects of Charles for
both web and mobile developers, as well as some pointers on how to actually apply the
techniques covered here. All the requests, settings and configuration in Charles can be
exported and shared, which is a huge help when using this tool across a team.

Other good products are available and you may also choose to make use of these. We also use
Curl for being able to make many dierent kinds of web requests and to easily share them by
pasting the commands into the bug trackers. Another alternative if you only need to inspect
(non-SSL) trac is Wireshark, as we mentioned earlier in the article. Both Curl and Wireshark
are free, multiplatform tools.
Charles tag line web debugging proxy is pretty accurate, making it indispensable in many
situations. If youve got a Charles tip to share, or want to tell us how you make use of the
techniques shown here, please do leave us a comment, wed love to hear from you!
+1144
Share

Tweet

Tags: browser, charles, curl, debugging, development, http, mobile, techniques, tools,
wireshark
by Lorna Mitchell/5 Comments

Related Posts
Profiling PHP Applications with XHGui
Tips for Developing Apps with PhoneGap
Building a Continuous Integration Server for PHP with Hudson
DPCRadio: The Art of Scalability

Home
About
Events
Open Source
Vote for a Topic
Write for Techportal
Follow us on Twitter
RSS

Search
Search site

Popular Posts
Testing Javascript: Get Started with Jasmine (1276)
Conference report: nodeconf ONE-SHOT London (1207)
Conference roundup: PHPNW 2014 (1196)
Manipulating HTTP with Charles Proxy (1144)
RESTFest 2014 and Women in Tech (1141)

Recent Posts
Testing Javascript: Get Started with Jasmine
Conference roundup: PHPNW 2014
RESTFest 2014 and Women in Tech

Conference report: nodeconf ONE-SHOT London

My top ten favourite PhpSpec limitations

Tags
agile apc

architecture atk bdd behat best practice caching challenge code competition

conference

database derick rethans development DPC dpc10 dpc11 dutch php conference frontend github
http ibuildings Javascript keynote

magento memcached mysql nosql performance

REST rob allen scalability Sebastian Bergmann

php phpunit profiling

symfony testing tipstricks tools unit testing xhprof zend zend framework

Archive
2014
2013
2012
2011
2010
2009
2008
2007
2006

Techportal is supported by The Inviqa Group

Inviqa is at the forefront of open source oering Web Development, Support, Training and
Consulting to enterprises in the UK and beyond.
info@inviqa.com

Session Digital is the leading Magento Enterprise partner in the UK, specialising in
providing eCommerce solutions to companies using Magento Enterprise.
info@sessiondigital.com

SensioLabsUK specialises in Symfony2 software and oers Training, Support and

Consultancy to organisations in the UK.
info@sensiolabs.co.uk
Legal Stu
Copyright - techPortal 2013
Privacy Policy
Have any queries?
techportal@inviqa.com