This post explains how to use HTTP monitor to intercept HTTP traffic on mobile with Fennec (Firefox for Android) installed.

Monitor HTTP traffic on mobile

HTTP Monitor is Firefox extension and so, you need Fennec installed on your mobile phone. Fennec is compatible with Android 2.2 and above devices.

If you have a compatible device go ahead and install Fennec (you need the Nightly channel).

In case of remote debugging you need to install HTTP Monitor extension on both sides: client and server. In this case, we want to monitor HTTP traffic on mobile (server) and check all results on desktop machine (client).

There are couple of preferences you need to set through about:config.

Server (mobile):
extensions.httpmonitor.serverMode set to true (you need to restart the browser)

Client (desktop):
extensions.httpmonitor.serverHost set to IP address of your phone (10.0.3.110 in my case).

In order to find IP address of your phone follow Settings > Wireless Controls > Wi-Fi settings and tap on the network you are connected to. It'll pop up a dialog with network status, speed, signal strength, security type and IP address.

As soon as you are all set (extension installed and preferences set) you can start Fennec on the phone and load a page. In my case I loaded mozilla.org.

Start also Firefox on your desktop machine and open HTTP Monitor console: Web Developer -> HTTP Monitor. There should be a Connect Me button at top left corner of the window.

Both the IP address and port number comes from preferences. Connect your phone and click the other: Select Tab button to pick a tab opened on your phone.

Now reload the tab on your phone and watch the result on the desktop!

Mobile HAR File

Here is an online HAR file showing mozilla.org page load on Samsung Galaxy Nexus:

Further Resources

• Source code
• Download (always use the latest release)
• Report an issue

We agreed that we want to have an effective and synchronized strategy when working on in-browser developer tools. One pillar of the plan is sharing APIs, components, modules, code maintenance, etc. to avoid duplicate development effort.

The first step is sharing an independent HTTP tracking component that is based on Firebug's Net panel and can be shared among all tools that want to intercept an analyze HTTP traffic. Firebug and (built-in tools in) Firefox will be its first clients.

So take a look at what we have managed to implement and demonstrate for our Toronto meeting last week.

HTTP Monitor

We have been hardworking on extracting the Net panel logic since the London meetup and result of the effort is an extension called HTTP Monitor. It's based on Firebug's Net panel and has exactly the same purpose - display HTTP activity of the selected tab to the user. This extension has very simple UI (a XUL window) with content corresponding to what you know from the Net panel.

The extension has two main parts: front-end and a back-end. Both independent and customizable so other tools can use, for example, only the back-end.

Especially tools that are used to automatically collect page performance data (often with Selenium) and don't need the UI can profit from having reliable, simple and built-in HTTP monitor API. Customization of the front-end will be based on Firefox themes (design) and Domplate (layout).

These two parts compose the entire HTTP Monitor component.

Local & Remote Monitoring

As the previous architecture diagram already indicates, support for remote HTTP monitoring is also covered in the concept. Here is a simple list of scenarios that are already supported by HTTP Monitor (beta phase):

1. Tracking HTTP activity of a browser tab running on the same machine.
2. Tracking HTTP activity of a browser tab running on another machine.
3. Tracking HTTP activity of a browser tab running on mobile

Note: HTTP Monitor extension must be installed on both sides (client/server), see instructions.

Communication between Firefox/Fennec instances is based on HTTP Monitor Remote Protocol (TCP/IP).

Using the extension should be fairly easy for those who are already familiar with the Net panel. The UI/UX is exactly the same, just available in separate XUL window. In order to open the HTTP console window pick Web Developer -> HTTP Monitor (you need to have the extension installed at this moment, it's not yet build in Firefox).

Remote HTTP tracking on Fennec requires Fennec Nightly build.

Further Resources

• Source code (GIT, pull requests welcome!)
• Wiki page
• Download (always use the latest release)
• Remote Protocol

Are you involved in page load performance area? Are you working on HTTP traffic analyzer tools? Do you want to see specific features as part of the HTTP Monitor component?

Then you could be involved in:

• Automation (NetExport)
• Export (HAR) (NetExport)
• Remoting

Stay tuned! (or watch the repo)

I had also great opportunity to lead a session about Firebug Tips & Tricks where I also mentioned some useful Firebug extensions. Check out my slides (en-US), but I am not sure how useful they are without me talking

We discussed many interesting topics and here is a brief summary of what stuck in my memory.

• There is Firefox Affiliates program for everyone who want to share Firefox with the world
• It would be great if US Mozilla dev folks could attend the next time and talk about Firefox related technologies.
• Pavel Cvrcek and Pavel Franc spoke about Mozilla in 2012: do not track, browser ID, apps, B2G, Firefox for Android
• Mozilla's mission is to promote openness, innovation and opportunity on the web.
• Jaroslav Souček from Seznam.cz spoke about Seznam Listicka (Firefox extension) and custom Firefox build
• Especially B2G got a lot of interest
• Transfer my mobile apps from my old phone to a new one easily (across devices)
• HTML5 & Video (btw. this post is quite related)
• And of course, Firebug and dev-tools built in Firefox.

If it sounds interesting to you, you should join us the next time! Follow Mozilla.cz to know when the next MozCamp CZ happens.

Photos provided by Jan Horák (Red Hat)

The Asynchronous Module Definition (AMD) API specifies a mechanism for defining modules such that the module and its dependencies can be asynchronously loaded.

One reason why Firebug uses AMD syntax is to support module sharing between Firebug and Firebug Lite. This post explains how to build Firebug extension based on AMD.

Hello AMD!

Let's start with an example extension. It'll be based on XUL Overlay for now to focus only on AMD in this post (next part of the tutorial will be about how to build a bootstrapped Firebug extension based on AMD).

The structure of the extension is following (it's just like standard XUL based extension, you can compare with the one from the first part):

Initialization Sequence

We are using XUL Overlay to load our extension. See how mainOverlay.xul and mainOverlay.js files look like.

mainOverlay.xul

The overlay loads only mainOverlay.js script file, which looks as follows:

The only thing the extension really needs to do is to register itself, by calling Firebug.registerExtension and Firebug will automatically look for a main module in the same directory to load it.

The first helloamd argument represents extension's name (extName) and is used to construct chrome URL for the main module as follows:

In fact, the loader maps extName to:

So, helloamd/myPanel module ID is automatically translated to:

(the loader also appends the js file extension)

The main module is using AMD syntax and is responsible for loading further extension modules. In our case we want to load a panel module.

You'll see in the next part of this tutorial, that the main module looks exactly the same for bootstrapped extensions. Just the Firebug registration won't be done in an overlay.

Panel Implementation

You should be already familiar with how to implement a new Firebug panel and Firebug model (Part I.) so, the rest of the extension should be very easy. The only difference is that we are using AMD syntax.

Get HelloAMD! Example Source

Full source of the example extension is available on GitHub.

The example shows also how to:

• Use Firebug tracing console API (FBTrace) (logging)
• Implement also a new Firebug module
• Register a custom stylesheet and use styles in a domplate template.
• Localize strings.

The feature is implemented by FireFontFamily, which is a Firebug extension created by Heather Arthur (also a member of the devtools team). So, you need to install it first if you are interested.

Simple feature description:

Highlights the rendered font-family from the list of font-families for a CSS 'font' or 'font-family' property in Firebug's CSS panel.

And couple of explanatory screenshots. The first one shows the Style side panel when FireFontFamily extension is not installed.

The other one shows the same when FireFontFamily is installed.

See that highlighted font, it's the one actually rendered.

So, for those who are interested at forking Firebug repository, issuing a pull request, developing Firebug extension, etc. here are some basic things to know before you start.

Firebug Repository

There are already several repositories in Firebug's account, but the one you are probably interested the most is firebug

https://github.com/firebug/firebug

The structure of the repo is following:

• extension Firebug extension directory
• tests Firebug automated test files and test harness - FBTest
• trace Firebug tracing console - FBTrace

Here is how to clone the repository (read only):

Run Firebug From Source

The extension directory contains Firebug extension files and so, you can run Firebug off of it. It's also the best way how to quickly check your code changes.

1. Locate your Firefox profile folder
2. Open extensions/ folder, create if it doesn't exist.
3. Create a new text file and put the full path to your development folder inside. (e.g. C:\firebug\extension\ or ~/firebug/extension/). Windows users should retain the OS' slash direction, and everyone should remember to include a closing slash and remove any trailing spaces.
4. Save the file with Firebug ID as it's name firebug@software.joehewitt.com

Now you can modify the code base, restart Firefox (with Firebug extension installed) and see your changes live.

Build Firebug XPI

You probably don't need this that often, but you can also easily build Firebug XPI package. All you need is Apache Ant installed on your machine.

The result XPI file should be located in ./release directory.

Push!

And now the most interesting part. You fixed a bug or implemented new API you need in your extension and you want us to pull your changes, commit into Firebug master branch and release! Let's see what you need to do.

Update: it's useful to create a separate branch for every contribution you make. (related to #4 and #5)

1. First, you need a GitHub account. It's simple and all the cool kids are already there
   
   
2. Fork Firebug repository (see how to fork a repo).
   
   
3. Clone your fork onto your local machine (your URL will be different):
   $ git clone git@github.com:janodvarko/firebug.git
4. Create a new branch, make your changes, stage/add modified files (e.g. firebug.js) and commit:
   $ cd firebug/extension
   $ git checkout -b myNewAPI
   # Make your changes in firebug.js file now
   $ git add content/firebug/firebug.js
   $ git commit -m "New API for my extension"
5. Push to origin (your public fork), the local branch myNewAPI to the remote branch myNewAPI:
   $ git push -u origin myNewAPI
6. Send a pull request. We'll review your changes and accept if all is OK!

Our release cycle is quick (a release at least every other week) so, your changes can be publicly available pretty soon!

Or Create a Patch

If you don't like github.com from some reason you can always create read-only clone (see at the beginning of this post), make your changes and generate a patch:

... or using svn:

Finally, create a new issue report (Firebug official issue list is still at the same location) and attach your patch. Again, we'll review your changes and accept if all is OK!

Further Resources

• Home: http://getfirebug.com/
• Blog: http://blog.getfirebug.com/
• Twitter: http://twitter.com/#!/firebugnews
• Discussion Group: https://groups.google.com/forum/?fromgroups#!forum/firebug
• Wiki: http://getfirebug.com/wiki
• Report an Issue: http://code.google.com/p/fbug/issues/list
• Firebug Extensions: http://getfirebug.com/wiki/index.php/Firebug_Extensions

The particular memory leak I have been looking for is called a zombie compartment. In my case, there is a document living in the memory even if the parent tab has been closed long time ago. Something is still referencing that document object so, it can't be released. The goal here is usually to find that referent. And this is when the CC heap debugging comes to rescue.

The rest of this post describes my extension CCDump that allows to dynamically analyze CC object graph.

Manual Analysis

CC heap dump log can be created manually using the following script (you can evaluate it in Firefox Error Console).

A file, something like: cc-edges-1.7104.log is created and path to it logged into the Error Console. This file contains text representation of the entire graph and you can use any text editor to search it.

Manual creation and search is tedious and you can easily spent hours and hours by repeating manual steps: open Firefox, reproduce a memory leak, execute the script to get the log, open the text file, search for zombie document, search for related objects, etc. - without any satisfying results.

That's why I decided to automate the entire process. Thanks to Olli Pettay who implemented a new API (introduced in Firefox 13, current Nightly) that allows to dynamically analyze the CC object graph from within an extension.

Dynamic Analysis

I implemented a bootstrapped (restart-less) extension that can be installed at the moment when you are experiencing a zombie compartment in your browser session - and immediately (without restart) analyze the current CC graph.

You can also use about:memory I order to see zombie compartments.

CCDump extension can be used to search within the object graph and execute queries to get the information you need.

You can download from AMO, get the latest build, or clone the source code (BSD License).

CCDump

After you install the extension, open: about:ccdump

You should see something as follows:

You mostly want to start by clicking on Run CC Analysis button in the Home tab. After you do it, Firefox cycle collector runs three times to clean up garbage and consequently once, with a listener, to collect entire CC graph of objects in the memory. The result object graph is immediately displayed as a table where each entry represents an object.

Every object has a context menu, just click the black triangle next to the green object label. There are three ways how to query the graph.

• Show Details - see detailed information about the object
• Show Roots - get roots that keep this object in the memory
• Show Graph - get all objects that are directly or indirectly related to the object (the result is displayed as an expandable tree).

Show Graph allows to dramatically reduce the object graph to see only related objects and search further only in the sub-graph result.

You can also search the graph. Use a search box located at the top right corner. Note that if there is a string in the box the search runs automatically just after analysis (I am typically keeping nsDocument text in the box).

There are some options:

• Find Zombie Documents - search for objects with nsDocument string in the name (i.e. for zombie content documents)
• Find Zombie HTTP Elements - search for objects with http string in the name (i.e. for zombie content nodes)
• Clear Search Results - trash the current search results and show the original table with entire graph.
• Case Sensitive - case sensitive search
• Use Regular Expression - you can also use regular expression to search the graph

The next screenshot shows object detail view:

• address unique hexadecimal address of the object. Always clickable thorough the UI
• refcount number of referents
• gcmarked if false CC is OK with freeing the object
• root if true the object is a root
• edges array of objects referenced by this object
• owners array of parent objects referencing this object

Where edges and owners represent relations in the graph.

Any feedback is welcome!

What could be yet improved in the UI?
What other queries could be made to the graph?
What other information in the CC graph would help to find and fix a memory leak?

Usage Scenario

My typical scenario when hunting down zombie compartments looks briefly as follows:

1. Run the browser, reproduce a memory leak
2. Open about:ccdump
3. Click Run CC Analysis to get the current object graph
4. Search for nsDocument to see a zombie document object
5. Show Graph for the zombie document
6. Search within the result graph for objects that keep it alive

The challenge is usually successful identification of an object in the graph and matching it with the corresponding object in the source code. Having more identity information would certainly help. Anyway, this problem should be dramatically reduced as soon as Bug 722749 i fixed - provide information about where JS object was allocated (source URL & line number).

Resources

Related resources you might find useful:

• CCDump Download: github.com/janodvarko/ccdump/downloads and AMO
• Source Code: github.com/janodvarko/ccdump
• Report issue/enhancement: github.com/janodvarko/ccdump/issues
• Cycle collector heap dump
• Common causes of memory leaks in extensions

Firebug UI resets every time the current browser tab is refreshed or navigated to another page - to show the new state. The consequence is that all Firebug panels trash the content (i.e. the Net or Console panel entries are lost).

This is fine since usually the user doesn't want to keep the entire history. But what in the case when you want to debug the transition process between page loads or redirects?

And this is exactly when the Persist button becomes practical...

Console Panel

Let's see a simple example that shows how to utilize the Persist button when debugging onUnload event.

This is a script on our test page

This is what the Console panel displays if you refresh the page (still the same even if you refresh several times):

The onUnload log is not displayed since immediately after the page is unloaded the Console panel is cleared - yep a new page is loaded.

And this is what you can see in the Console if you check the Persist button and refresh one more time.

The clear part is not happening now.

Net Panel

Let's use the same example page and see what the Persist button does in context of the Net panel.

Again, this is what the Net panel shows if you refresh the page:

And now with Persist mode enabled:

The previous entries (just one in this particular case) are kept under a special expandable entry at the top of the Net panel. You can refresh more times or even navigate to a different page and all the HTTP page load history is there.

Note, that the entire Net panel history (multiple page loads) can be exported using NetExport extension and displayed using HAR Viewer online tool (individual page loads are preserved and you can compare).

Do you think it would make sense to support the Persist feature on another panel too?

This feature is useful in cases where the developer needs to know when particular piece of Javascript code is executed relatively to the HTTP request made by web page. Especially in cases when Javascript is loaded through dynamically appended <script> tags or through XHR and evaluated using eval().

See the following screenshot that shows a time-stamp generated during a page load.

The time stamp is also displayed in the Console panel so, you can compare with other logs.

The rest of this post demonstrates this feature on couple of live examples in detail. So, if you are interested what you can do with Firebug 1.8, read further!

Script used in examples

Before we dive into details, let's define some basic functions used in the examples below.

• addExternalScript This function creates a <script> element and appends it into the document. The element asynchronously fetches the script from an external URL.
• sleep This functions freezes the JS execution for specified amount of time [ms].

Example 1

The first example shows basic usage of console.timeStamp() method. The preview below is the Net panel content (exported using NetExport Firebug extension) when the example page is loaded:

Script on the page looks like as follows:

The Net panel content (move mouse over the vertical lines to see detailed timing info.):

• Two HTTP requests displayed. One for page document (example.html) and one for attached CSS file (testcase.css).
• After the CSS file is loaded, script on the page is executed. This is depicted by the Script on the page executed time-stamp (lines produced by console.timeStamp use olive color).
• Javascript execution is frozen for 100 milliseconds.
• The page fires DOMContentLoad event.
• The page fires load event.

Example 2

This example demonstrates dynamically inserted <script> element.

Script on the page looks like as follows:

The example.js file:

• Three HTTP requests displayed. One for page document (example.html), one for attached CSS file (testcase.css) and one for dynamically inserted script element (example.js).
• Just like in the previous example, the first Script on the page executed time-stamp is generated when script on the page is executed.
• An external script is added, but it's loaded asynchronously so, we can see the External script added time-stamp immediately after Script on the page executed.
• The example.js file starts loading.
• The page fires DOMContentLoad event.
• The example.js file is loaded and its script evaluated. The result is new External script executed time-stamp.
• The external script freezes the page for 100 milliseconds.
• The page fires load event.

Timeline extent & request grouping

Let's see yet another example that explains how requests in the Net panel are grouped in order to preserve reasonable time-extent for the timeline.

In order to avoid endless extent for the waterfall diagram, the Net panel is grouping requests that belongs together (starting a new group if there is a pause in HTTP traffic). Where each group's waterfall starts again from the beginning of the graph. See also this post.

Created time stamps and page events lines are always rendered across request entries in the last group.

See this Net panel log (one group):

• There is a time stamp generated long time after the page load event. The timeline extent has been extended to display the time stamp.
• Both requests have been done in the same time period and so they belong to the same group.

See another Net panel log (two groups):

• Two requests executed to load the page (just like in the previous example). Both belong to the same group.
• There are another three requests fired some time later. These form a new group.
• There is a time stamp, rendered across requests in the last group.

Addendum

HTTP traffic timing data displayed in the Net panel comes from Firefox network layer (Necko). Unfortunately, the network layer uses different timer than Javascript Date object and these timers can provide a bit different results. Since the console.timeStamp() method is based on the JS Date object, you could notice sometimes that time-stamps is delayed about ~10-30 ms. This should be fixed as soon as bug 570341 in Firefox is fixed.

Of course, this button is only available when you are stopped at a breakpoint.

Lets go through real example step by step.

First, we need to create a breakpoint to break on.

We have created a breakpoint in a JS function that is called when you press a button on the example page. Let's pretend that execution of this function is not that simple and the developer needs to go through a complicated scenario to hit the breakpoint.

Click the button on the example page to hit our breakpoint.

As you can see within the Stack side-panel, there is several function calls on the stack. The bottom/oldest stack is the click event handler that has been executed upon our mouse click.

Since the debugger is sitting at the second console.log, the Console panel contains just one log.

Now let's imagine the developer wants to perform the execution again. Perhaps another function has been already called and it's necessary to debug that call too. This would require creating another breakpoint higher in the stack and re-execute.

Re-execution: instead of pressing the Continue button and so, be forced to repeate the user scenario again, let's click the Rerun button!

Yes, we are again at the same spot.

The only evidence that we previously stopped the execution at line 28 and re-ran the stack is the Console Panel that clearly shows only the Hello! text - twice, since the other log has never been reached!

Isn't that cool?

* Firebug 1.7 contains a little bug related to the stack frame list. There are couple more helper frames displayed (used to implement this feature) that should never be visible. Firebug 1.8 fixes it.