Plugin development gets new tools and opens-up to the community

Since the introduction of the Gephi Marketplace and tools such as the Plugins Bootcamp we’ve seen more and more plugins being developed. Even developers with little experience with Java give it a try and succeed in creating their first plugin. We want developers to be productive and make it as easy as possible to get started with plugin development and find help along the way. As the release of the 0.9 version is near, it’s time to review our plan on that matter and upcoming improvements. Here’s the summary:

  • The gephi-plugins base repository (i.e. repository plugin developers fork) is now using Maven for building and is simpler. It contains only 4 files versus 890 for the Ant-based system.
  • All Gephi modules are published on Maven central, making it very easy to inspect and extend.
  • Introduction of a custom Maven plugin designed to facilitate plugin development.
  • The submission and review of plugins will be entirely based of GitHub, making it more scalable and transparent.
  • A new online portal for plugins is coming up with an easier edit experience and new features.

From Ant to Maven

Before diving into plugins, let’s first review what has changed on how Gephi is compiled, built and packaged – as this directly affects plugins as well. Since the Gephi 0.8.2 version we have migrated our build system from Ant to Maven. This is in line with what the Netbeans Platform (i.e. which Gephi is based on) community recommends. It already has increased the level of automation we’re capable of as a result. The main benefits are (compared to Ant):

  • Maven is great at dependencies management. It’s now very clear what version of what library Gephi depends on, making it simpler to integrate. Dependencies are also downloaded automatically instead of being checked in the codebase
  • Unlike the Ant-based system, it’s independent from Netbeans. This allows developers not using Netbeans to develop Gephi and produce a build entirely from the command-line.
  • Gephi modules can now be placed on Maven Central (i.e. global repository where Maven finds its dependencies). This allows plugins to automatically find the Gephi dependencies online, reducing the manual steps at each Gephi upgrade.

Build assistant

There are a few critical steps we want to help plugin developers with and as a result started the development of a custom Maven plugin. This new tool will work behind the scenes when developers build their plugin. No installation or configuration is needed as it comes already as dependency of the gephi-plugins module. It already addresses common pain points and hope to automate more and more of the steps in the future. This is what it can do as of today:

  • Plugin validation: The assistant reviews the plugin configuration and metadata at each build. This allows for instance to check if the plugin depends on the correct Gephi version or remind the developer to define an author or license in its configuration.
  • Run Gephi with plugins: A single command allows to run Gephi with the plugins pre-installed. This makes testing faster than ever when developing plugins.
  • New plugin generator: A step-by-step command-line tool that creates the correct folder structure and configuration to get started.

In the future, we want to rely on this build assistant to further automate the process and for instance do easy migration or code generation. For instance, you could ask to generate a Layout plugin code and configuration. Afterwards, all needed would be to fill in the blanks in the code.

A new way to review and submit plugins

As the number of plugins grows, it’s important to have a clear process how plugins are reviewed and updated. We also want this process to be transparent and open to the community. So far, the process was based on the submission of the plugin binaries with a manual review done by the team. This helped us get where we are today, but we want to get it to the next level and propose to entirely move this process to GitHub – using the pull-request mechanism. This has multiple advantages, listed below:

  • Reviewing new/updated plugins can scale because any developer can read the code and contribute to the pull requests.
  • Developers are already asked to fork the gephi-plugins repository so submitting the plugin via GitHub is a natural extension to it.
  • There’s a clear history of each version, comment and what code has changed from one version to another.
  • It makes it easier to test plugins and detect issues before the plugin is approved.

As part of this migration, we’ll no longer add plugins with closed source code but all existing plugins for Gephi 0.8.2 will remain available. For security and stability reasons, it’s essential that each plugin’s code can be inspected before approval. In order for this to work, all existing plugins not already on GitHub or not forking the gephi-plugins repository will need to migrate. For those already set up, the migration will be easier but Ant-based plugins will still need to migrate to Maven.

To summarize, this is what the new 4-steps process looks like for developers:

gephi-new-plugin-development-process

In the current submission process we ask for additional information such as description, author or license as well as allow the upload of images. Going forward with GitHub, all of these data will directly be defined in the plugin’s configuration making it easier to update.

A new home for plugins (again)

Plugins are currently available online from the Gephi Marketplace, where users could also reach people providing teachings and support.  We have ideas on how to improve these community services and will be migrating them to a new architecture, starting with the plugins. We will tell you more about these changes in an upcoming post but for now our focus is on developing a new lightweight plugin portal that can directly be connected with the data source on GitHub.

Here is a preview of what it will look like for plugin pages:

new-plugin-frontend-preview

 

The content of this website will be automatically updated when plugins are published or updated. The way it works is with Travis CI (i.e. continuous integration platform) simply refreshing the JSON file after changes to the plugin repository on GitHub. Developers can even embed images and write the description in Markdown. This will remove entirely the need for plugin developers to login to the marketplace, update NBMs and metadata.

Migrating plugins

This new Maven-based repository along with the new submission process will be introduced with the Gephi 0.9 release. Let’s review what plugin developers need to know to bring their plugin to this new major version.

As with all major Gephi release, plugins compatibility needs to be evaluated as APIs may have changed. In fact, given this new version is based on an entirely redeveloped core it’s very likely code changes will be required. Hopefully, these changes will often be minor and actually simplify things (i.e less, more efficient code). Documentation will be published on these API changes and core developers will be available to answer questions as well.

Plugin developers will also get contacted regarding moving their code to GitHub with a step-by-step guide. We’re considering adding a migrate command to the new Gephi Maven plugin to facilitate the transition from Ant but that’s an unfunded project at the moment (if you’re interested contributing to that, please let us know). Stay tuned for details right after the release on the path to migration.

And again, thanks for all your hard work on bringing your ideas to life though new Gephi plugins!

 

A month of Gephi Marketplace

 

guillaume_old160Computer Science & Engineering Graduate from the Compiègne University of Technology, Guillaume Ceccarelli has been Gephi’s SysAdmin since 2008 and Web Developer since 2012. When he’s not by day working in the financial world, he freelances and stays close to ongoing entrepreneurship ventures.

Four weeks ago, we launched the Gephi Marketplace.

Haven’t taken a look yet? Didn’t know it existed? Take a few minutes to check it out now! It’s right here, waiting for you.

Why the Gephi marketplace?

For a long time, we felt like we lacked a central point of access when it came to what everyone was doing to improve or build upon Gephi. On one hand, our forums allowed for discussions to happen but they didn’t really enable anyone to share their work easily or to find out about one another. On the other hand, the wiki as well as our main site, served more as official portals to spread the word or to explore knowledge around Gephi, but they didn’t seem like a good fit to showcase the work our community was doing.

A little while ago, we decided to fix that.

The spirit behind the marketplace is simple. When we started, our idea was:

  1. To provide Gephi users with easy means to find and learn more about what the community was building for and around Gephi
  2. To give creators and people working with Gephi a way to publish their work and make them known to the entire community, without the overhead of a complex and lengthy process

This meant creating the central point for Gephi Plugins and what we call Gephi Services, which eventually became the Gephi Marketplace.

Before the marketplace existed, plugins were submitted and downloaded through the old plugin center, which required authors and the Gephi staff to go through a tedious process, creating a lot of work and unnecessary frustration for everyone. In addition, the system didn’t allow authors to update their own plugin pages, or to submit a different plugin for different versions of the Gephi application, a feature which was long requested before we finally became in a position to deliver it to you.

A new world of plugins

Now, every plugin download, even from the Gephi Application itself go through the marketplace! It means that as soon as a plugin goes live on our new platform, it’s directly available to every Gephi user around the world. Even if you haven’t made a plugin yourself, chances are you’ve already used the marketplace without realizing it!

Speaking of around the world, here’s a geographic breakdown of Gephi plugin downloads within the past 30 days:

Screen-Shot-2013-02-17-at-6.27.59-PM

Isn’t it impressive? 1169 plugin downloads and every continent is represented! (ok, not every single one. Props to the first Gephi user who’ll download a plugin from Antartica!)

It makes us extremely proud to help and foster such a vibrant community of Gephi users around the world. Remember that this map is for the last 30 days alone!

Today, we have 38 plugins listed on the marketplace, and we’ve made sure you can post your creations as easily as possible. It takes no more than 5 minutes to first fill in the plugin upload form, and we usually get back to you within 24 hours, so you can make your work available to thousands in record time!

A world of services

As briefly mentioned before, we’re also listing Gephi Services on the marketplace.

Over the years, we’ve noticed that a community of professionals and enthusiasts started providing services to the community. We’ve seen people doing trainings on Gephi, others integrating Gephi into data analysis workflows, or even create studies using Gephi as their tool of choice. We thought it was about time these fine folks got easier to find!

Every service provided by the community, for free or with a fee, is welcome to be listed. We already have a few, and to be honest we would love to see more of them!

Listing the services you provide on the Gephi marketplace is easy, and like with listing plugins it’s completely free, regardless of what you choose to charge for your work. We’re not here to collect fees. We just want to help other people know about your work, and to help you easily be found and easily be reached.

Following the same simple form submission process you can use for plugins, you give us all the info on what you’ve got for the community, you click ‘upload’, we get back to you, and then you have yourself a listing!… Which you can then customize to your liking, and for which you can include every single bit of information you want, and even a great copy! So not only can you let everyone know about what it is that you do, but also how truly wonderful it is to work with you and your expertise. So?

And now back to you.

Any question? Feel free to comment below! We’ll be glad to help you get started. And if you’d like to be kept in the know of what’s newly getting posted on the Marketplace, the RSS feed is this way.

In the meantime, thank you for your continued trust and your use of Gephi as your Graph Exploration and Manipulation tool of choice! It is our humble pleasure to serve you all.

See you soon on the marketplace! 🙂

Python Scripting Console for Gephi

The first release of the Gephi’s Python Console plugin is finally available for download. Through this plugin, you can execute queries and manipulate the graph structure by typing commands on a scripting console, making it a very powerful and concise tool to work with.

This console started as a joint proposal with the GUESS project which aimed at porting the Gython language as a console plugin for Gephi during the Google Summer of Code 2011 program. This project was mentored by Eytan Adar, from the GUESS project, and co-mentored by Mathieu Bastian, from Gephi.

After installation, the plugin can be accessed through the left slide on Gephi’s UI or through the Window > Console.

console

Among the features available through the console scripting language are:

  • Inspired by the Gython language from the GUESS project (i.e. additional operators for manipulating graphs).
  • Based on Jython 2.5.2, which implements the Python 2.5 specification.
  • Graph structure and attributes manipulation.
  • Filtering support.
  • Support for running layouts.
  • Export API support (i.e. generate PDF/GEXF/PNG exports from within the console).
  • Batch scripts support for automating tasks.
  • Loads third-party Python libraries.

For example, if you would like to filter the male persons from your social graph and make the result visible, you could simply type the following command:

visible = g.filter(gender == "male")

If you want set this subgraph nodes’ color to blue, you could just type:

g.filter(gender == "male").nodes.color = blue

Or, even better, if you want to color both the nodes and edges of this same subgraph, just type:

(gender == "male").color = blue

Implementing batch scripts for automating tasks is very easy. Just save the script as a file with the py extension and load it with the execfile function. For instance, a batch script for creating a random graph with 50 nodes would look like this:

scripting_random_graphs

And to run the script, you would call the following command on the console:

>>> execfile("C:\Users\user\Documents\script.py")

Installation

The Scripting Plugin can be installed through the Gephi’s Tools/Plug-ins menu:

  1. Go to the Tools > Plug-ins menu on Gephi’s user interface.
  2. Open the Available Plugins tab.
  3. Click the “Reload Catalog” button, to make sure you have an up-to-date catalog.
  4. Select the “Scripting Plugin” on the list and click the Install button.

After installation, the plugin can be accessed through the left slide on Gephi’s UI (as shown in the image below) or through the Window > Console menu.

Documentation

The console’s documentation at the wiki has instructions for downloading and installing the plugin.

Feedback and Contributing

This plugin has been developed by Luiz Ribeiro since the Google Summer of Code 2011 program and we’re really looking forward to see what you’ll be able to do with this new tool.

The plugin’s source code is hosted on GitHub and is open to your contributions. If you find any bugs or have requests or feature ideas make sure to open a new ticket on the Issues Tracker.

Introducing the Gephi Plugins Bootcamp

We’re happy to announce a new tool for the community today: the Gephi Plugins Bootcamp. The bootcamp is a large set of plug-in examples to guide developers create Gephi plugins easily.

Gephi’s vision focuses on the platform and we want developers to be creative and successful. Gephi is built in a way it can be easily extended with plug-ins (layout, filters, io, preview, …) but it’s not always easy to know where to start. The bootcamp addresses this need and provide the environment and the examples to get started.

Want to create a new layout? The Grid Layout example shows how to read the graph and to change coordinates. A new filter? A new exporter? Check out the JPGExporter or the SQLite exporter examples. Below is the complete list of examples and we’ll add more soon upon requests.

Checkout the code on GitHub.

The README file contains the instructions to get the code and to run the examples.

Layout

Grid Layout Example Grid Layout
Place all nodes in a simple grid. Users can configure the size of the area and the speed.
Sorted Grid Layout Example Sorted Grid Layout
Same example as Grid Layout but users can sort nodes using an attribute column.

Filter

Transform to Undirected Example Transform to Undirected
Edge filter to remove mutual edges in a directed graph.
Top Nodes Filter Top nodes
Keep the top K nodes using an attribute column.
Filter Edge Crossing Example Remove Edge Crossing
Example of a complex filter implementation which removes edges until no crossing occurs.

Tool

FInd with Autocomplete Find with autocomplete
Tool with a autocomplete text field to find any node based on labels and zoom by it.
Add Nodes example Add Nodes
Listen to mouse clicks and adds nodes. Also adds edges if selecting other nodes.

Export

jpg export JPG Export
Vectorial export to the JPG image format. Contains a settings panel to set the width and height.
sqlite_export SQLite Database Export
Current graph export to a SQLite Database file. A new sub-menu is added in the Export menu and an example of a custom exporter is shown.

Preview

Highlight mutual edges example Highlight Mutual Edges
Colors differently mutual edges. Overwrites and extends the default edge renderer.
Glow Renderer example Glow Renderer
Adds a new renderer for node items which draws a glow effect around nodes.
Z-Ordering Example Node Z-ordering
Extends the default node builder by reordering the node items by size or any number columns. Also shows how to create complex Preview UI.

Importer

Matrix import example Matrix Market File Importer
File importer for the Matrix Market format. Large set of matrix file examples on Yifan Hu’s gallery.

Statistics

Count Self Loop example Count Self-Loop
Example of a statistics result at the global level. Simply counts the number of self-loop edges in the graph.
Average Euclidean Distance Example Average Euclidean Distance
Example of a per-node calculation. For a given node it calculates the average distance to others.

Plugins sub-menu

Submenus Example Test action
Simple action which displays a message and a dialog.
remove_self_loops Remove self loops
Action which accesses the graph and removes self-loops, if any.
Progress Example Using Progress and Cancel
Action which creates a long task and executes it with progress and cancel support.

Execute at startup

UI Ready Example When UI is ready
Do something when the UI finished loading.
workspace_events Workspace select events
Do something when a workspace is selected.

Processor

Initial Position Processor Example Initial Position
Set up the nodes’ initial position always the same. It calculates a hash with all nodes so the X/Y position is randomized always in the same way.

Panels

Panel Example New panel
Example of a new panel plugin set up at the layout position.

If you have any questions please send an email to the gephi-plugins [at] lists.gephi.org mailing list or stop by on the forum.

Gephi adds Neo4j graph database support, screencast

Neo4j is a powerful, award-wining graph database written in Java. It can store billions of nodes and relationships and allows very fast query/traversal. We release today a new version of the Neo4j Plugin supporting the latest 1.5 version of Neo4j. In Gephi, go to Tools > Plugins to install the plug-in.

The plugin let you visualize a graph stored in a Neo4j database and play with it. Features include full import, traversal, filter, export and lazy loading.

 


Neo4j Integration into Gephi from gephi on Vimeo.

The plug-in is officially supported by the Neo4j team and is open to contribution! The code is hosted on GitHub.

If you have suggestions please send them our way, we would love to hear your feedback! The forum is the best place for that.

Gephi 0.8 beta released

The latest beta version of Gephi has been released, download it for Windows, Mac OS and Linux platforms. This release focus on new features for both users and developers, and the new license unlocks opportunities for business. The Ranking and Preview modules have been completely rewritten in a modular way and can be now extended with plug-ins! Preview can now be extended in many ways, for instance group shapes or edge bundling. Moreover, continuous progress have been made on the dynamic network support and we release today the last big part: statistics over time, available from the Statistics module when the network is dynamic. Thanks to users who reported bugs, it’s the only way to fix them.

The team will now start developing the 0.9 version of Gephi (please consider joining us!) and integrate the latest Google Summer of Code projects, including a new timeline. We are also willing to help as much as possible plug-ins developers to get things done and improve documentation. We want to leverage the new Preview and will help newcomers to get started.

Because it’s a major release, changes are not deployed through the AutoUpdate, you need to download and install the new version. Plug-ins also need to be checked for compatibility. They will reappear on the Plugin Center in the coming days, as they are verified. Thanks for your patience.

Consult the release notes and the new Javadoc for more information.

Features highlight

Ranking Auto Apply Dynamic Ranking

Ranking now works with dynamic networks and it’s easy! When manipulating a dynamic network with the Timeline simply enable the ‘Auto Apply’ and the color/size is updated in real-time. To import a dynamic node size simply import a regular dynamic column in your GEXF and select it in Ranking like before.

PNG Export PNG Export

A new powerful PNG export has been added to the existing PDF and SVG export. One can create high-resolution network images with all the customization available in Preview. You can even create transparent background images!

New Preview New Preview

Major effort to completely rewrite the Preview module in a modular way. One can now create plug-ins for Preview! The new Preview includes new opacity options, a text outline, radius (customize edge start/end points) and a simplified list of properties.

Dynamic Metrics Dynamic Metrics

New dynamic metrics in the Statistics module: Dynamic Degree, Dynamic Node Count, Dynamic Edge Count and Dynamic Clustering Coefficient. Dynamic metrics are executed on a dynamic network and allow to analyze how network properties evolve over time.

New And Noteworthy

* Data Lab node merging
* ForceAtlas2 layout algorithm, with multi-thread option
* Node and Edge transparency in Preview
* Edge labels on curved edges in Preview
* Text outline now in Preview
* Database importer now supports time columns (start & end)
* DL Export (Thanks to Taras Klaskovsky)
* GML Export (Thanks to Taras Klaskovsky)
* NET Export (Thanks to Daniel Bernardes)
* K-core filter
* Inter and Intra partition filter
* Now supports SQLite databases
* Display the number of layout iterations in status bar when ended
* Recent Palette in Ranking
* Weighted degree now also for directed graphs

New localization (Go to Tools > Languages)

* Portuguese (Brazilian) (Thanks to Célio Faria Jr)
* Japanese (Thanks to Siro Kida and Koji Chono)

Performance

* Memory starvation manager, warns the user before running out of memory
* Less memory usage with attributes

Bug fixes

* Timeline need more precision when dealing with dates (bug 521937)
* Exception on range slider (bug 541808)
* Inconsistent label data from Overview to Preview (bug 660204)
* Statistics: sub-optimal modularity (bug 727701)
* Timeline cann’t drag if the two sliders moved to the left (bug 745476)
* Missing Polish characters when exporting to pdf (bug 746740)
* Edge selection color is not correct on OSX (bug 752300)
* Workspace name truncated, hard to read (bug 758578)
* Average degree cannot be switched to directed / undirected (bug 760454)
* Window->Favorites appears in 0.8 alpha (bug 764494)
* Disable ‘directed’ on metric settings if the graph is undirected (bug 771318)
* Timeline does not work (exception) (bug 774455)
* Layout properties can’t be saved in a language and loaded in another language (bug 783637)
* Preview: edge label not shown (bug 783868)
* Possible memory leak on Dynamic Range Filter (bug 784606)
* Edge attributes not saved in .gephi project file (bug 785268)
* Data Lab: Exception when selecting only one column for merging (bug 785269)
* Data Lab Import Spreadsheet should not ignore parallel edges (bug 785635)
* Data Laboratory: wrong edge type created (mutual instead of directed) (bug 787401)
* Data Lab: impossible to edit time intervals in a date format (bug 793163)
* Spelling of Proportionnal (bug 794358)
* Graphics errors when JOGL installed as a JRE/JDK extension (bug 799545)
* NPE if source/target is empty in GEXF import (bug 799574)
* Toolkit can’t open .gephi files (bug 802101)
* Resizing edge sizes changes edge weight values (bug 803763)
* Preview does not use node label settings from overview tab (bug 805763)
* Data Lab ‘Import Spreadsheet’ dialogue should accept other file types than .csv (bug 806798)
* Edge weights not imported from CSV matrix (bug 808078)
* Preview tab: no option to switch off node borders? (bug 808606)
* Gephi runs out of memory without warning the user (bug 811373)
* Counter-intuitive filename in Data export dialog (bug 814178)
* NullPointerException when creating newProjects too quickly (bug 817170)
* Nodes and edges Id attribute dictionary is not properly created when loading a .gephi file (bug 818181)
* Database driver doesn’t persist in Edge List Database import UI (bug 822316)
* NodeEqualNumberFilter does not work (bug 823038)
* Gephi does not build on JDK 7 (bug 823543)
* SVG node, edge export should include relevant node IDs as classes (bug 827706)
* Import Spreadsheet: need to trim column names (bug 829956)
* Layout list not sorted by name (bug 830149)
* In/Out degree metric is computed on the main graph instead of the visible graph (bug 830752)
* Layout is not giving the algorithm’s number of iterations (bug 831782)
* Banner height issues, need a fixed height (bug 834400)
* NPE when running ClusteringCoefficient on a filtered graph (bug 852799)
* Missing node properties from dot file (bug 855410)
* ‘The value column doesn’t exist’ error when opening a gephi file (bug 857595)
* Import fails for NET (Pajek) file with position/color data (bug 860825)
* GEXF export referes to v1.1 schema, should be v1.2 schema (bug 864484)

New Plug-ins documentation

Checkout the documentation for the newly created Preview module: HowTo write a Preview Renderer. Learn also how to extend the Data Laboratory features in a new tutorial.

New license

Gephi is now released in the dual license CDDL + GNU GPLv3. We abandon the GNU AGPL to offer new opportunities to reuse and integrate parts of Gephi in a full Open Source way. The dual license system means the possibility to choose to apply either the CDDL or the GNU GPLv3 when Gephi source code is integrated to a derivative work. When modified, original files of Gephi should always be published publicly so that the community benefit from the improvements. However, the CDDL license does not require to publish the whole work, so you can build commercial applications for free using Gephi source code!
The CDDL is a license created by Sun and approved by the Open Source Initiative. It is business-friendly. Read the Legal FAQs to learn more, and ask questions on the forum.

Contribute

It’s fun to contribute to an open-source project! Contribute whatever the time you can give: few minutes to  report a bug, some hours to fix one or to translate the user interface or more to create a plug-in. If you’re a student and looking for cool and challenging semester projects check out the Gephi Student Program or contact us.

Do Gephi technologies matter for your research or business? You can support us by donating to the Gephi Consortium, or becoming a member to have an impact on our roadmap.

Feel free to reach to us if you are willing to organize events (meetups, workshops, hackathon, etc.), we will support them.

First Gephi Plugin Developers Workshop on October, 6

gephi workshop

This is an announcement for the first Gephi Plugins Developers Workshop October 6, 2011 in Mountain View, California. Come and learn how to write your first Gephi plugin and ask questions. The workshop is organized by Mathieu Bastian, Gephi Architect and will be gratefully hosted by IMVU.

Gephi is a modular software and can be extended with plug-ins. Plug-ins can add new features like layout, filters, metrics, data sources, etc. or modify existing features. Gephi is written in Java so anything that can be used in Java can be packaged as a Gephi plug-in! Visit the Plugins Portal on the wiki and follow the tutorials to get started.

The workshop will start with a presentation of Gephi’s architecture and the different types of plugins that can be written with examples. Details about Gephi’s APIs, code examples and best practices will be presented in an interactive “live coding” way. The Gephi Toolkit will also be covered in details. The second part of the workshop will be dedicated to help individuals with their projects and answer questions.

Some of the best projects using or extending Gephi are developed in the Silicon Valley and we are looking forward helping the developer community. Please don’t hesitate to send us your ideas to maximize efficiency.

RSVP here