Plugin development gets new tools and opens-up to the community

Since the introduction of the Gephi Marketplace and tools such as the Plugins Bootcamp we’ve seen more and more plugins being developed. Even developers with little experience with Java give it a try and succeed in creating their first plugin. We want developers to be productive and make it as easy as possible to get started with plugin development and find help along the way. As the release of the 0.9 version is near, it’s time to review our plan on that matter and upcoming improvements. Here’s the summary:

  • The gephi-plugins base repository (i.e. repository plugin developers fork) is now using Maven for building and is simpler. It contains only 4 files versus 890 for the Ant-based system.
  • All Gephi modules are published on Maven central, making it very easy to inspect and extend.
  • Introduction of a custom Maven plugin designed to facilitate plugin development.
  • The submission and review of plugins will be entirely based of GitHub, making it more scalable and transparent.
  • A new online portal for plugins is coming up with an easier edit experience and new features.

From Ant to Maven

Before diving into plugins, let’s first review what has changed on how Gephi is compiled, built and packaged – as this directly affects plugins as well. Since the Gephi 0.8.2 version we have migrated our build system from Ant to Maven. This is in line with what the Netbeans Platform (i.e. which Gephi is based on) community recommends. It already has increased the level of automation we’re capable of as a result. The main benefits are (compared to Ant):

  • Maven is great at dependencies management. It’s now very clear what version of what library Gephi depends on, making it simpler to integrate. Dependencies are also downloaded automatically instead of being checked in the codebase
  • Unlike the Ant-based system, it’s independent from Netbeans. This allows developers not using Netbeans to develop Gephi and produce a build entirely from the command-line.
  • Gephi modules can now be placed on Maven Central (i.e. global repository where Maven finds its dependencies). This allows plugins to automatically find the Gephi dependencies online, reducing the manual steps at each Gephi upgrade.

Build assistant

There are a few critical steps we want to help plugin developers with and as a result started the development of a custom Maven plugin. This new tool will work behind the scenes when developers build their plugin. No installation or configuration is needed as it comes already as dependency of the gephi-plugins module. It already addresses common pain points and hope to automate more and more of the steps in the future. This is what it can do as of today:

  • Plugin validation: The assistant reviews the plugin configuration and metadata at each build. This allows for instance to check if the plugin depends on the correct Gephi version or remind the developer to define an author or license in its configuration.
  • Run Gephi with plugins: A single command allows to run Gephi with the plugins pre-installed. This makes testing faster than ever when developing plugins.
  • New plugin generator: A step-by-step command-line tool that creates the correct folder structure and configuration to get started.

In the future, we want to rely on this build assistant to further automate the process and for instance do easy migration or code generation. For instance, you could ask to generate a Layout plugin code and configuration. Afterwards, all needed would be to fill in the blanks in the code.

A new way to review and submit plugins

As the number of plugins grows, it’s important to have a clear process how plugins are reviewed and updated. We also want this process to be transparent and open to the community. So far, the process was based on the submission of the plugin binaries with a manual review done by the team. This helped us get where we are today, but we want to get it to the next level and propose to entirely move this process to GitHub – using the pull-request mechanism. This has multiple advantages, listed below:

  • Reviewing new/updated plugins can scale because any developer can read the code and contribute to the pull requests.
  • Developers are already asked to fork the gephi-plugins repository so submitting the plugin via GitHub is a natural extension to it.
  • There’s a clear history of each version, comment and what code has changed from one version to another.
  • It makes it easier to test plugins and detect issues before the plugin is approved.

As part of this migration, we’ll no longer add plugins with closed source code but all existing plugins for Gephi 0.8.2 will remain available. For security and stability reasons, it’s essential that each plugin’s code can be inspected before approval. In order for this to work, all existing plugins not already on GitHub or not forking the gephi-plugins repository will need to migrate. For those already set up, the migration will be easier but Ant-based plugins will still need to migrate to Maven.

To summarize, this is what the new 4-steps process looks like for developers:

gephi-new-plugin-development-process

In the current submission process we ask for additional information such as description, author or license as well as allow the upload of images. Going forward with GitHub, all of these data will directly be defined in the plugin’s configuration making it easier to update.

A new home for plugins (again)

Plugins are currently available online from the Gephi Marketplace, where users could also reach people providing teachings and support.  We have ideas on how to improve these community services and will be migrating them to a new architecture, starting with the plugins. We will tell you more about these changes in an upcoming post but for now our focus is on developing a new lightweight plugin portal that can directly be connected with the data source on GitHub.

Here is a preview of what it will look like for plugin pages:

new-plugin-frontend-preview

 

The content of this website will be automatically updated when plugins are published or updated. The way it works is with Travis CI (i.e. continuous integration platform) simply refreshing the JSON file after changes to the plugin repository on GitHub. Developers can even embed images and write the description in Markdown. This will remove entirely the need for plugin developers to login to the marketplace, update NBMs and metadata.

Migrating plugins

This new Maven-based repository along with the new submission process will be introduced with the Gephi 0.9 release. Let’s review what plugin developers need to know to bring their plugin to this new major version.

As with all major Gephi release, plugins compatibility needs to be evaluated as APIs may have changed. In fact, given this new version is based on an entirely redeveloped core it’s very likely code changes will be required. Hopefully, these changes will often be minor and actually simplify things (i.e less, more efficient code). Documentation will be published on these API changes and core developers will be available to answer questions as well.

Plugin developers will also get contacted regarding moving their code to GitHub with a step-by-step guide. We’re considering adding a migrate command to the new Gephi Maven plugin to facilitate the transition from Ant but that’s an unfunded project at the moment (if you’re interested contributing to that, please let us know). Stay tuned for details right after the release on the path to migration.

And again, thanks for all your hard work on bringing your ideas to life though new Gephi plugins!

 

Python Scripting Console for Gephi

The first release of the Gephi’s Python Console plugin is finally available for download. Through this plugin, you can execute queries and manipulate the graph structure by typing commands on a scripting console, making it a very powerful and concise tool to work with.

This console started as a joint proposal with the GUESS project which aimed at porting the Gython language as a console plugin for Gephi during the Google Summer of Code 2011 program. This project was mentored by Eytan Adar, from the GUESS project, and co-mentored by Mathieu Bastian, from Gephi.

After installation, the plugin can be accessed through the left slide on Gephi’s UI or through the Window > Console.

console

Among the features available through the console scripting language are:

  • Inspired by the Gython language from the GUESS project (i.e. additional operators for manipulating graphs).
  • Based on Jython 2.5.2, which implements the Python 2.5 specification.
  • Graph structure and attributes manipulation.
  • Filtering support.
  • Support for running layouts.
  • Export API support (i.e. generate PDF/GEXF/PNG exports from within the console).
  • Batch scripts support for automating tasks.
  • Loads third-party Python libraries.

For example, if you would like to filter the male persons from your social graph and make the result visible, you could simply type the following command:

visible = g.filter(gender == "male")

If you want set this subgraph nodes’ color to blue, you could just type:

g.filter(gender == "male").nodes.color = blue

Or, even better, if you want to color both the nodes and edges of this same subgraph, just type:

(gender == "male").color = blue

Implementing batch scripts for automating tasks is very easy. Just save the script as a file with the py extension and load it with the execfile function. For instance, a batch script for creating a random graph with 50 nodes would look like this:

scripting_random_graphs

And to run the script, you would call the following command on the console:

>>> execfile("C:\Users\user\Documents\script.py")

Installation

The Scripting Plugin can be installed through the Gephi’s Tools/Plug-ins menu:

  1. Go to the Tools > Plug-ins menu on Gephi’s user interface.
  2. Open the Available Plugins tab.
  3. Click the “Reload Catalog” button, to make sure you have an up-to-date catalog.
  4. Select the “Scripting Plugin” on the list and click the Install button.

After installation, the plugin can be accessed through the left slide on Gephi’s UI (as shown in the image below) or through the Window > Console menu.

Documentation

The console’s documentation at the wiki has instructions for downloading and installing the plugin.

Feedback and Contributing

This plugin has been developed by Luiz Ribeiro since the Google Summer of Code 2011 program and we’re really looking forward to see what you’ll be able to do with this new tool.

The plugin’s source code is hosted on GitHub and is open to your contributions. If you find any bugs or have requests or feature ideas make sure to open a new ticket on the Issues Tracker.

Gephi-Neo4j presentation at FOSDEM

FOSDEM_Brain_Logo_200Martin Škurla, developer of the Neo4j connector in Gephi and former Google Summer of Code student in 2010, gave a complete talk about his plug-in at the last FOSDEM conference in the workshop on Graph Processing. His talk covers the installation and showcase of visualization capabilities of Neo4j graph database in Gephi, as long as the exploration of general useful features of Gephi applying them on imported Neo4j database. Watch the video below, and comment it to ask additional questions!

The Neo4j plugin of Gephi is now officially supported by Neo Technology, Inc. If you have a software project involving Gephi, contact us at consortium@gephi.org to see how we can help you.

Gephi adds Neo4j graph database support, screencast

Neo4j is a powerful, award-wining graph database written in Java. It can store billions of nodes and relationships and allows very fast query/traversal. We release today a new version of the Neo4j Plugin supporting the latest 1.5 version of Neo4j. In Gephi, go to Tools > Plugins to install the plug-in.

The plugin let you visualize a graph stored in a Neo4j database and play with it. Features include full import, traversal, filter, export and lazy loading.

 


Neo4j Integration into Gephi from gephi on Vimeo.

The plug-in is officially supported by the Neo4j team and is open to contribution! The code is hosted on GitHub.

If you have suggestions please send them our way, we would love to hear your feedback! The forum is the best place for that.

ForceAtlas2, the new version of our home-brew Layout

The new version of the build-in layout ForceAtlas is now released. It is scaled for small to medium-size graphs, and is adapted to qualitative interpretation of graphs. The equations are the same as ForceAtlas 1, but there are more options and innovative optimizations that make it a very fast layout algorithm.

It is good enough to deal with very small graphs (10 nodes)  and fast enough to spatialize 10,000 nodes graphs in few minutes, with the same quality. If you have time, it can deal with even bigger graphs.

Update Gephi (Help > Check for Updates) to get this new layout.

Force Atlas 2:

  • Is a continuous algorithm, that allows you to manipulate the graph while it is rendering (a classic force-vector, like Fruchterman Rheingold, and unlike OpenOrd)
  • Has a linear-linear model (attraction and repulsion proportional to distance between nodes). The shape of the graph is between Früchterman & Rheingold’s layout and Noack’s LinLog.
  • Features a unique adaptive convergence speed that allows most graphs to converge more efficiently
  • Proposes summarized settings, focused on what impact the shape of the graph (scaling, gravity…). Default speed should be the good one.
  • Now features a Barnes Hut optimization (performance drops less with big graphs)

 

 

Force Atlas 2 features these settings:

  • Scaling: How much repulsion you want. More makes a more sparse graph.
  • Gravity: Attracts nodes to the center. Prevents islands from drifting away.
  • Dissuade Hubs: Distributes attraction along outbound edges. Hubs attract less and thus are pushed to the borders.
  • LinLog mode: Switch ForceAtlas’ model from lin-lin to lin-log (tribute to Andreas Noack). Makes clusters more tight.
  • Prevent Overlap: Use only when spatialized. Should not be used with “Approximate Repulsion”
  • Tolerance (speed): How much swinging you allow. Above 1 discouraged. Lower gives less speed and more precision.
  • Approximate Repulsion: Barnes Hut optimization: n² complexity to n.ln(n) ; allows larger graphs.
  • Approximation: Theta of the Barnes Hut optimization.
  • Edge Weight Influence: How much influence you give to the edges weight. 0 is “no influence” and 1 is “normal”.

 

 

Force Atlas 2 was created by Mathieu Jacomy at the Sciences Po Médialab (Paris), founding member of the Gephi Consortium.

Scientific graphs Generators plugin

Cezary Bartosiak and Rafał Kasprzyk just released the Complex Generators plugin, introducing many awaited scientific generators. These generators are extremely useful for scientists, as they help to simulate various real networks. They can test their models and algorithms on well-studied graph examples. For instance, the Watts-Strogatz generator creates networks as described by Duncan Watts in his Six Degrees book.

The plugin contains the following generators:

  • Balanced Tree
  • Barabasi Albert
  • Barabasi Albert Generalized
  • Barabasi Albert Simplified A
  • Barabasi Albert Simplified B
  • Erdos Renyi Gnm
  • Erdos Renyi Gnp
  • Kleinberg
  • Watts Strogatz Alpha
  • Watts Strogatz Beta

The plug-in can be installed directly from Gephi 0.8, from the Plugins menu.

The source code is available on Launchpad.

The HTTP Graph plugin

The HTTP Graph plugin provides real-time collection and visualization of HTTP traffic. Using the embeddable Membrane Router, details are extracted from the transaction headers and fed to Gephi for graphing and further analysis. This approach makes the relationships between clients, servers, and resources easily visible.


See the video in HD on Vimeo.

Nodes

There are 4 types of nodes: client, uri, host, domain.

Client: By default, the largest sized nodes with the source IP addresses of clients for labels. If you are the only one pointing to the plugin’s proxy, there will probably be only one of these nodes that says 127.0.0.1. Clients are linked to a domain node of ‘local’ to keep them together on the graph. Another function of the client node is to keep the graph anchored. You may find it interesting to use a filter in Gephi to hide the client type nodes to see a more “free-form” graph of the internet. If you do this, you may see large pieces float away because they didn’t link to the rest of the graph anymore!

URI: By default, the smallest sized nodes with no visible labels. These represent resources like HTML pages, images, javascript, or whatever other resources might be requested through the proxy. The size in bytes and the MIME (Content-Type) reported by the host when returning the resource is available so you can see what it is.

Host: For a given domain (.gephi.org, .google.com, etc.) there can be multiple hosts which serve the different resources. In some cases, you may see the same resource being served from multiple servers in a DNS-based load balancing system. Other interesting details about the underlying architecture of the sites you are viewing can be seen.

Domain: These nodes exist primarily to keep the related hosts close together on the graph. You may want to use a filter in Gephi for this type of node and hide them to see a different perspective.

Edges

HTTP and the web are defined by links, which are essentially directed graph edges, and these occur at the resource level. An HTML page resource will link to CSS, image, and other file resources, both on the same domain, and on remote domains. These inter-domain links are the glue that forms the structure of the world wide web.

Have fun!

~by phreakocious

Get the HTTP Graph plugin on the Plugins Center, or in Gephi go to Tools > Plugins > Available plugins.