Debugging HTTP/HTTPS Issues With Charles and Fiddler

Most applications use some kind of remote data source. Whether it is a website, some RESTful services, or SOAP APIs, it all comes down to HTTP (Hypertext Transfer Protocol) that handles the requests. When building applications, developers often don’t worry what is happening at the network level. But as things tend to get more complex and unusual bugs appear, looking into the network level could be an invaluable time saver.

If you’ve ever been suspicious of your app consuming unusual amounts of data, having connection problems or you just want to get the best performance out of your app, here are some useful tips that might make you want to consider trying HTTP proxy tools the next time you come across issues related to network communication.

How to identify HTTP-based issues?

In order to identify HTTP-based issues the best solution is to capture data traffic of your app and look at individual requests or even raw packets being sent back and forth. In web-based apps we would probably look at the data traffic using some kind of inspector, like Chrome DevTools. But what about mobile apps? The best and simplest solution is to use an HTTP debugging proxy server tool.

In general, HTTP debugging proxy server tools can be used for a number of reasons, but they are great for capturing HTTP traffic and logging it for the user to review. Also, most modern tools are capable of decrypting HTTPS (HyperText Transfer Protocol Secure) traffic.

To understand how the decrypting is done, let’s take a closer look at how HTTPS works. In general, HTTPS appears in the URL when a website is secured by an SSL (Secure Sockets Layer) certificate (also known as Transport Layer Security or TLS).

Encounters with SSL usually go like this:

1. The client attempts to connect to a web service secured with SSL.
2. The client requests the web server to identify itself.
3. The server sends the client a copy of its SSL Certificate.
4. The client checks whether it trusts the SSL Certificate. If it does, it sends a message to the server.
5. The server sends back a digitally signed acknowledgement to start an SSL encrypted session.

The way these tools make HTTPS decrypting happen is by using self-signed SSL certificates to fake a certificate for the HTTPS endpoint, thus implementing a man-in-the-middle interception.That means that the computer running the proxy tool will be able to see all the data communication that is happening on the mobile device.

Fortunately, there are many great tools available for this purpose, but we will take a look at the best ones – Fiddler and Charles.

Let’s assume that we have an app and it has a simple problem. When a certain button is pressed, the app should open and view a PDF file that is located on a remote server. But nothing loads and displays on the screen, no errors are shown. Now let’s see how these tools can help us find the cause of the issue, starting with Fiddler.

Setting up our project

When it comes to setting up iOS and Android projects, there is nothing else you need to do for iOS projects; you might as well skip this part.

However, for Android projects, if your device is running Android 7.0 or newer, there are a few extra things to be done to ensure Android trusts the SSL Certificate. In order to make Android trust the certificate, Network Security Configuration needs to be specified for the application. There are many ways to do this, but the most effective is to specify the config file in your xml resources like this:

<?xml version="1.0" encoding="utf-8"?> 
<network-security-config> 
   <debug-overrides>
      <trust-anchors>
         <certificates src="system" /> 
         <certificates src="user"/>
      </trust-anchors> 
   </debug-overrides>
</network-security-config>

Now save it within your XML resource folder, like this, for example:

res/xml/network_security_config.xml

After the configuration file is created, let’s specify it with the android:networkSecurityConfig property within your AndroidManifest.xml file just like this:

<manifest ... >
   <application 
      android:allowBackup="true" 
      android:icon="@mipmap/ic_launcher" 
      android:label="@string/app_name" 
      android:theme="@style/AppTheme" 
      android:networkSecurityConfig="@xml/network_security_config"
      android:supportsRtl="true" 
      ... >
   </application> 
</manifest>

Now the project is ready for debugging. You can specify that this only applies to the debug builds of your application, so that the production builds use the default trust profile.

Fiddler

Fiddler (provided by Telerik) is a free web debugging proxy for any browser, system or platform. As an HTTP debugging tool, Fiddler is one of the best, especially for mobile platforms, because it’s easy to install and use. The installation package can be obtained from Telerik’s website. Setting up Fiddler is pretty straightforward, there are separate step-by-step guides provided by Telerik which explain how to install the root certificate which is the key component for the proxy to work.

• iOS setup guide
• Android setup guide

After setting up the device and installing certificates we are now ready to do some actual debugging. The most basic use of Fiddler is to run it in background on a computer and use your mobile device to reproduce the issue.

Here’s an example of some HTTP traffic recorded. We have our basic information of each request on the left hand side and detailed analysis as well as the request and response details available in different formats on the right hand side.

In our case, we needed to verify if the PDF file was downloaded successfully from the server. As we can see from the result status code (200), the request was made successfully and there is some data received. Now we can go further and inspect the data that was received and we can see that the Content-Length parameter appears to be 0, which means that the received file is empty. There might be many causes for this to happen, but this method convinced us pretty quickly to take a look at the file that gets returned from server and not to look for any failures in the client application.

Usually a lot of traffic goes through the device, so to make the inspecting process easier, we can use filters to only look at the endpoints and URLs that we need.

After inspecting we can save our session by clicking File → Save → All Sessions or Selected Sessions and view or share the session file later.

Charles

Charles is another great tool for all sorts of HTTP proxying. It’s so versatile that some people have tried to hack Facebook games with it, there is even a note in the support page of their website:

Please note: Charles is not intended or marketed as a tool for hacking Facebook games. Please do not contact me for support in this case.

You can download the installation package from their website. The installation process is fairly simple, just follow these instructions provided on their website. After installing Charles, install the SSL certificate following these instructions.

Charles is quite similar to Fiddler – you just run in the background on your computer. But we have to admit that Charles’ user interface is much more user friendly, with just the right things needed. Also, if you are using a Mac, this the go-to tool for you, since Fiddler doesn’t work very well on Macs.

This is the typical view that you will see, when working with Charles. The cool thing about this is that Charles structures requests by hosts. You can click and expand one host and  all requests will be listed under it, which makes the whole process more manageable and easier to use.

For our PDF problem, we can see the request response pretty clearly along with some timing data and overall stats what we would not see in the same window in Fiddler.

One of the very nice features of Charles is the option to limit bandwidth. By going to Proxy and selecting Throttle Settings… you can set up network throttling that allows you to simulate a variety of slower connection speeds and see how that affects the app.

For example, here are the stats that will be used to simulate a 3G network connection. This is available in Fiddler as well, but it must be configured from the script editor which can be more time consuming. For more information on setting up different network conditions you can read our blog post.

Tips and tricks

When troubleshooting network bugs, there are several components that can cause issues, so here is a useful set of steps to help us track down problems of this kind more easily:

• Did we send a request?
• Was the request correct?
• Did the server fail?
• Did the network fail?
• Did we receive a response?
• Was the response correct?
• Did the client failed to handle the response?

Most common issues that can be easily tracked down with HTTP proxy tools:

• Making calls to data sources and not receiving the expected results
• API errors (e.g. invalid payload formats)
• Excessive amounts of requests or not enough requests per action
• Requests sent in wrong or unexpected scenarios
• Receiving and sending too much data to endpoints when only some parts of the request are actually used

Conclusion

When it comes to debugging network related issues HTTP proxy tools can be real time savers. For the most part, Fiddler and Charles offer similar features. Although we only scratched the surface of what these tools can do, you can now explore them for yourself and find even more ways to improve your app. Without a doubt, Fiddler supports some advanced scenarios, but they may not be that easy to implement for the average user. One can go about creating plugins to exercise quite complex and esoteric scenarios by using the .NET Fiddler module, such as examples here, but let’s leave that for another article.

Feel free to reach out if you have any questions about this topic.