The HTTP Graph plugin

The HTTP Graph plugin provides real-time collection and visualization of HTTP traffic. Using the embeddable Membrane Router, details are extracted from the transaction headers and fed to Gephi for graphing and further analysis. This approach makes the relationships between clients, servers, and resources easily visible.

See the video in HD on Vimeo.

Nodes

There are 4 types of nodes: client, uri, host, domain.

Client: By default, the largest sized nodes with the source IP addresses of clients for labels. If you are the only one pointing to the plugin’s proxy, there will probably be only one of these nodes that says 127.0.0.1. Clients are linked to a domain node of ‘local’ to keep them together on the graph. Another function of the client node is to keep the graph anchored. You may find it interesting to use a filter in Gephi to hide the client type nodes to see a more “free-form” graph of the internet. If you do this, you may see large pieces float away because they didn’t link to the rest of the graph anymore!

URI: By default, the smallest sized nodes with no visible labels. These represent resources like HTML pages, images, javascript, or whatever other resources might be requested through the proxy. The size in bytes and the MIME (Content-Type) reported by the host when returning the resource is available so you can see what it is.

Host: For a given domain (.gephi.org, .google.com, etc.) there can be multiple hosts which serve the different resources. In some cases, you may see the same resource being served from multiple servers in a DNS-based load balancing system. Other interesting details about the underlying architecture of the sites you are viewing can be seen.

Domain: These nodes exist primarily to keep the related hosts close together on the graph. You may want to use a filter in Gephi for this type of node and hide them to see a different perspective.

Edges

HTTP and the web are defined by links, which are essentially directed graph edges, and these occur at the resource level. An HTML page resource will link to CSS, image, and other file resources, both on the same domain, and on remote domains. These inter-domain links are the glue that forms the structure of the world wide web.

Have fun!

~by phreakocious

Get the HTTP Graph plugin on the Plugins Center, or in Gephi go to Tools > Plugins > Available plugins.

Graph visualization on the web with Gephi and Seadragon

The project takes another big step forward and bring dynamic graph exploration on the web in one click from Gephi with the Seadragon Web Export plugin.

Mathieu Bastian and Julian Bilcke worked on a Seadragon export plugin. Directly export large graph pictures and put it on the web. Seadragon is pure Javascript and works on all modern browsers. As it uses images tiles (like Google Maps), there is no graph size limit.

Go to your Gephi installation and then to the Plugin Center (Tools > Plugin) to install the plugin. You can also download manually the plugin archive or get the source code.

/seadragon-samples/diseasome/seadragon.html

Sample with Diseasome Network dataset directly exported from Gephi

Communication about (large) graphs is currently limited because it’s not easy to put them on the web. Graph visualization has very much same aims as other types of visualization and need powerful web support. It’s a long time we are thinking about the best way to do this and found that there is no perfect solution. We need in the same time efficiency, interactivity and portability. The simpleness of making and hacking the system is also important, as we want developers to be able to improve it easily.

By comparing technologies we found that Seadragon is the best short-term solution, with minimum efforts and maximum results. It has however still a serious limitation: interactivity. No search and no click on nodes are possible for the moment. But as it is JS, I don’t see hurdles to add these features in the future, help needed.

The table below see our conclusions on technologies we are considering. We are very much eager to discuss it on the forum. As performance is the most important demand, WebGL is a serious candidate but development would require time and resources. We plan to start a WebGL visualization engine prototype next summer, for Google Summer of Code 2011, but we would like to discuss specifications with anyone interested and make this together.

	Portability	Efficiency	Effort	Interactivity
Flash
Java2D/Processing
Canvas (Processing.js/RaphaelJS)
WebGL
Seadragon

Figure: Comparing technologies able to display networks on the web.

How to use the plugin?

Install the plugin from Gephi, “Tools > Plugin” and find Seadragon Web Export. After restarting Gephi, the plugin is installed in the export menu. Load a sample network and try the plugin. Go to the Preview tab to configure the rendering settings like colors, labels and edges.

Export directly from Gephi Export menu

The settings asks for a valid directory where to export the files and the size of the canvas. Bigger is the canvas, more you can zoom in, but it takes longer time to generate and to load.

Export settings, configure the size of the image

Note that result on the local hard-drive can’t be viewed with Chrome, due to a bug. Run Chrome with “–allow-file-access-from-files” option to make it work.

Kudos to Microsoft Live Labs for this great library, released in Ms-PL open source license. Thank you to Franck Cuny for the CPAN Explorer project that inspired this plugin. Other interesting projects are GEXF Explorer, a Flash-based dynamic widget and gexf4js, load GEXF files into Protovis.