Rebuilding Gephi’s core for the 0.9 version

This is the first article about the future Gephi 0.9 version. Our objective is to prepare the ground for a future 1.0 release and focus on solving some of the most difficult problems. It all starts with the core of Gephi and we’re giving today a preview of the upcoming changes in that area. In fact, we’re rewriting the core modules from scratch to improve performance, stability and add new features. The core modules represent and store the graph and attributes in memory so it’s available to the rest of the application. Rewriting Gephi’s core is like replacing the engine of a truck and involves adapting a lot of interconnected pieces. Gephi’s current graph structure engine was designed in 2009 and didn’t change much in multiple releases. Although it’s working, it doesn’t have the level of quality we want for Gephi 1.0 and needs to be overhauled. The aim is to complete the new implementation and integrate it in the 0.9 version.

In November 2012, we started to develop a completely new in-memory graph structure implementation for Gephi based on what we’ve learnt over the years and our desire to design a solution that will last. The project code-name is GraphStore and we focus on four main things:

  • Performance: The graph structure is so important to the rest of the application that is has to be fast and memory efficient.
  • Stability: The new code will be the most heavily unit-tested in the history of Gephi.
  • Simplicity: The Graph API should be documented and easy to use for developers.
  • Openness: If possible, we want GraphStore to be used in other projects and keep the code free of Gephi-specific concepts.

Gephi is known to use a large amount of memory, especially for very large networks. We want to challenge ourselves and tackle this issue by redesigning the way graphs are encoded and stored. Besides memory usage, we carefully analyzed possible solutions to improve read/write performance and optimize the throughput. Stability and simplicity are like food and shelter, and whatever we try to do at Gephi should be simple to use and stable. As we’re going towards a 1.0 version, we’re putting more and more efforts to testing and code quality.

Since November 2012, we have been working on GraphStore separately from Gephi’s codebase and will start the integration fairly soon. The Graph API is very similar to the existing API. However, it isn’t entirely compatible and several core things changed like attributes, views or dynamic networks and will require a lot of work in some modules. On the other hand, because the GraphStore code is decoupled, it could be leveraged in other projects. For instance, it could serve as a Blueprints implementation as an alternative to TinkerGraph.

Graph structure

A graph (also called network) is a pair of a set of nodes and a set of edges. Edges can be undirected, or directed if the direction of the relation matters. Edges may also have weights to represent a value attached to the edges, like the strength of a connection or the flow capacity. Edges may also point to the same node (i.e. self-loops). Gephi currently supports these features, but they are not sufficient to describe the variety of problems graphs can be helpful with. Multigraphs permit several relationships between nodes and is for instance commonly used to represent RDF graphs. Multigraphs with properties (i.e. ability to attach any property to nodes and edges) have recently become the standard representation for graph databases.

The next version of Gephi will support multigraphs and therefore allow multiple edges between nodes to be imported. The rollout will be done in two phases. The first phase is to allow this new type of graph to be imported, filtered and exported. We will update the importers and add new options to support these graphs. The second phase is to update the visualization and the way multiple edges between nodes look like.

Hierarchical graphs

Since the 0.7 version released in 2009, Gephi has supported hierarchical graphs. Hierarchical graphs let the user group or ungroup nodes so it forms a tree. Nodes which contain other nodes are named meta-nodes and edges are collapsed into meta-edges. Groups obtained from clustering algorithm (e.g. modularity) could also easily be collapsed into meta-nodes in order to study the network at a higher level. We initially recognized the potential of this idea for network analysis and developed a hierarchy-enabled data structure. However, we realized we didn’t completely fulfill the vision by not providing all the tools to fully explore and manage hierarchical networks. Although the data structure allows it, the software still lacks many features to really make hierarchical networks explorable.

Recently, we are more focused on networks over time and plan to continue to do so. In the past years, users have shown steady and continuous interest in dynamic networks and we haven’t really seen a strong interest in hierarchical networks. Therefore, we propose to remove this feature from next releases. On the developer side, cutting this feature will greatly simplify the code and improve performance.

Dynamic networks

Networks that change over time are some of the most interesting to visualize and analyze. We have heavily invested in supporting this type of network, for instance by developing the Timeline component. However, dynamic graph support was added after the current graph structure implementation was conceived and therefore remains suboptimal and difficult to scale. Now that we have enough hindsight, we can rethink how this should be done and make it simpler.

One pain point is the way we decided to represent the time. Essentially, there are two ways to represent time for a particular node in a graph: timestamps or intervals. Timestamps are a list of points where the particular nodes exist and intervals have a beginning and an end. For multiple reasons, we thought intervals would be easier to manipulate and more efficient than a (possibly very large) set of timestamps. By talking to our users, we found that intervals are rarely used in real-world data. On the code side, we also found that it makes things much more complex and not that efficient at the end.

In future versions, we’ll remove support for intervals and add timestamps instead. We considered supporting both intervals and timestamps but decided that it would add too much complexity and confusion.

Graph structure internals

Graph structures design is an interesting problem to solve. The objective is quite simple, yet challenging: how to best represent an interconnected graph so it’s fast to query and compact in space? Also, how to keep it simple and serve a large number of features at the same time?

Graph storage

Our goal is to develop a thread-safe, in-memory graph structure implementation in Java suitable for real-time analysis. You may ask how this differs from a graph database or a distributed graph analysis package. In a few words, one can say the requirements are quite different.

Graph databases like Neo4j, OrientDB or Titan store the graph on local disk or in a cluster and are optimized for large graphs and large number of concurrent users. Typically, the networks are much larger than what can fit in memory and these databases mostly focus on answering traversal queries. In the environment where graph databases operate most of the needs can be converted in some sort of traversal query (e.g. friends of X, tweets of Y). Traversal queries are also the reason why graph databases scale to billions of nodes. Indeed, for each traversal, only a subset of the graph is accessed. This is quite different from Gephi, which by its nature of being an analysis software needs to access the complete graph. For instance, when a layout is running Gephi needs to read the X,Y position of each node as quickly as possible. Although reading from the disk can be very quick as well (e.g. GraphChi), it’s limited to sequential access and things become more complex that way.

Because of the real-time requirements, we want to keep our graph data in memory accessible at all time. However, we want to make it easy to connect to external data sources, and graph databases in particular.

Reducing overhead

In computer science, overhead is any combination of excess or indirect computation time, memory, bandwidth, or other resources that are required to attain a particular goal.

GraphStore heavily relies on Java primitives, arrays and efficient collections library like fastutil. We are reducing overhead by simply avoiding using too many Java objects, which are very costly. Instead of using maps, trees or lists, Nodes and Edges are stored in large arrays which can be dynamically resized in blocks. For instance, iterating over the graph should be extremely fast because the CPU caches array blocks. This may sounds obvious but performance optimizations are tricky in Java because of the JVM and the uncertainty of what makes a difference and what doesn’t. In his “Effective Java” book, Joshua Bloch writes “Don’t guess, measure” and that’s still true today. For our project, we rely on well-defined micro-benchmarks to see where the bottlenecks are and how to make our data-structure more cache-friendly and more compact in memory. When the graph contains millions of edges, every byte saved per edge can make a large difference at the end.

In terms of speed, we focused on optimizing the most common operations, which are iterating over all the elements and consult nodes’ neighbors. Typically, a layout algorithm needs to read the neighbors of every node at each iteration. Neighbors can’t simply be an unsorted list because of the removal complexity: to remove a node, you need to know where it is. The current Gephi graph structure uses a binary tree to store the node’s neighbors. Although the complexity is logarithmic, every node in the tree takes extra memory and logarithmic complexity is still suboptimal. After isolating the problem in a benchmark, we found that using a double linked-list is the best solution for our requirements and achieves a O(1) complexity, as it fulfills both a quick iteration and quick update. Here is a snapshot of the solution:

Every edge has 4 integer pointers to the next in/out predecessor and successors and a separate dictionary would help to find the right edge based on the source and destination pointers. Each node has a pointer to the first edge in the linked list (i.e the head). Node ids are integers (32 bits) so one can easily create a long->Edge dictionary by encoding the source and destination node into a single long number (64 bits). The diagram intentionally leaves out the multigraph support for simplicity. In reality, nodes can have multiple head pointers, one for each edge type. Each edge type is represented by a integer index.

Views

Views are one of the most useful aspects of Gephi’s graph structure and are mainly used behind the scenes in the Filter module. A view is a graph subset (i.e. a subgraph) which remains connected to the main structure, so if a node is removed from the graph, it’s removed from the views as well. For instance, when users create a ‘Degree Filter’, Gephi creates a view and removes all the nodes which don’t fulfill the degree threshold. Multiple views can co-exist at any time in the graph structure. In the current graph structure, a node tree complete copy is done for every view and we found that this can be very inefficient.

In the new version, the way views are implemented is very different and should yield to better performance. Instead of doing a copy of the nodes, we maintain bit-vectors for nodes and edges. Because these elements are stored in large arrays with a unique identifier, it’s easy to create and maintain a bit-vector. When developers obtain the ‘Graph’ object for a particular view, the bit-vectors are used behind the scenes to adapt iterators and accessors. This solution should make filtering for large graphs much quicker. One drawback is that whereas the current implementation copies and then trims the view, GraphStore work with bit-vectors but continues to access the complete graph. In other words, if the view represents only 1% of the original graph, it still needs to iterate over the 100% to find which elements are the 1%. Even though this sounds bad, our benchmarks show it’s a very fast operation and we win overall because of the reduced overhead of duplicating the graph. Moreover, we can introduce some caching later to optimize this further.

Inverted Index

When you’re using the Partition module in Gephi, you’re manipulating some sort of inverted index. Nodes and edges have properties like ‘gender’, ‘age’ or ‘country’, and these properties are contained within the nodes and edges objects. An index is a simple data structure which allows to retrieve the list of elements for a particular value. For instance, the partition module needs to know what is the number of ‘male’ or ‘female’ nodes for the ‘gender’ column. When the column is a number like ‘age’, it also needs to know what is the maximum and minimum value. Unlike the Ranking module and its auto-apply feature, the Partition module is not refreshed in real-time and therefore difficult to use when the graph is changing a lot. We have decided to invest in this feature for the future release and are building a real column inverted index in the graph structure. The index will simply keep track of which values exist for each column and which elements are holding this value. The index will be updated in real-time as elements are added, removed or updated.

The ability to quickly retrieve elements and counts based on specific values will be very useful in many different modules like Filters, Partition or Data Laboratory. New APIs will be added for developers to use the newly created index interface. As we’re working on attributes storage and manipulation, we’ll also merge the Attributes and Graph API because they are so interconnected that it doesn’t really make sense to have them separate. The interfaces that developers are familiar with like Table or Columns will remain the same.

Events

In software programming, events are a common way to inform other modules that something changed. In Gephi, we also use events to convey graph updates events to inform other modules about updated nodes or edges. In the new GraphStore, we’ll stop using events to transport graph modifications because of the large overhead due to the creation of event objects. Indeed, when 10K nodes are added to the graph, the existing structure literally creates 10K event objects and puts them in a queue. Although the event queue is compressing objects of the same type, the overhead to create, queue, send and destroy large amount of small Java objects is too large.

Instead of a push model (i.e. the emitter is pushing updates), we want to rather promote a pull model (i.e. the listener pull updates from time to time) for future releases. A similar system is already in place to link the graph and the visualization module and it has been working without a glitch. We’ll develop the tools to easily calculate graph differences between a listener module and the graph structure. By removing the bottleneck, write performance should greatly improve.

Timestamps

As said earlier, we’ll add timestamps support to represent dynamic networks. Instead of using a time interval, a timestamp array will be associated with nodes and edges. For element (node/edge) visibility, each timestamp represents the presence of the element at that time. For example if a network snapshot is collected every month for a year, each node will have up to 12 different timestamps. The timestamp itself is a real number and can therefore represents an epoch time but also any other value in a different context. For a dynamic attribute, the time+value is simply represented as a list of (time, value) pairs.

To support the timeline and dynamic networks algorithm, we’re developing an inverted index for timestamps so we can make time filtering very quick. One good thing about intervals is that it’s very easy to know if two intervals overlaps with each other. With a flat list of timestamps, one can’t avoid to go through the entire list. The index will essentially map timestamps to the nodes and edges elements in the graph and therefore solves this issue. The Interval tree implementation which we are currently using to store intervals is based on a binary tree and is very costly in memory because of all the Java objects overhead. Using simple arrays should reduce overhead and improve performance for large dynamic networks. When computing a dynamic network algorithm (ex: Clustering coefficient over time), we’re using a sliding window over the graph so the ability to quickly filter is critical as it impacts how fast the graph refreshes.

Saving/Loading

Saving and loading the graph structure into into/from a file (or a stream) is another critical feature. When a user saves a project in Gephi, the graph data structure is serialized in XML and compressed into a .gephi file. If you worked with project files in Gephi, you may have experienced corrupted files issues or errors when loading a file. We’ve done our best to fix these problems but some still remain. We’re rethinking how this should be done in GraphStore and are making a call to rewrite the code from scratch. Our approach will rely on a lot of unit tests to make sure the code is stable so we don’t repeat the same issues in future versions. Please note that this concerns the .gephi files only and existing importers (e.g. GEXF, GraphML) will remain the same.

Concerning the GraphStore serialization, we’re abandoning XML in favor of pure byte arrays. That should yield to better performance and reduced project file size. We’ll create a custom reader for previous Gephi versions so you can still open your existing projects. Other modules like Filters or Preview will continue to use XML as it’s working just fine.

Next steps

This is the first post about the Gephi 0.9 version and more will come soon. We’re excited about the current developments and hope to hear from you. Please join the gephi-dev mailing list to learn more about ongoing projects and contribute. We need your ideas!

Follow us on Twitter!

A month of Gephi Marketplace

 

guillaume_old160Computer Science & Engineering Graduate from the Compiègne University of Technology, Guillaume Ceccarelli has been Gephi’s SysAdmin since 2008 and Web Developer since 2012. When he’s not by day working in the financial world, he freelances and stays close to ongoing entrepreneurship ventures.

Four weeks ago, we launched the Gephi Marketplace.

Haven’t taken a look yet? Didn’t know it existed? Take a few minutes to check it out now! It’s right here, waiting for you.

Why the Gephi marketplace?

For a long time, we felt like we lacked a central point of access when it came to what everyone was doing to improve or build upon Gephi. On one hand, our forums allowed for discussions to happen but they didn’t really enable anyone to share their work easily or to find out about one another. On the other hand, the wiki as well as our main site, served more as official portals to spread the word or to explore knowledge around Gephi, but they didn’t seem like a good fit to showcase the work our community was doing.

A little while ago, we decided to fix that.

The spirit behind the marketplace is simple. When we started, our idea was:

  1. To provide Gephi users with easy means to find and learn more about what the community was building for and around Gephi
  2. To give creators and people working with Gephi a way to publish their work and make them known to the entire community, without the overhead of a complex and lengthy process

This meant creating the central point for Gephi Plugins and what we call Gephi Services, which eventually became the Gephi Marketplace.

Before the marketplace existed, plugins were submitted and downloaded through the old plugin center, which required authors and the Gephi staff to go through a tedious process, creating a lot of work and unnecessary frustration for everyone. In addition, the system didn’t allow authors to update their own plugin pages, or to submit a different plugin for different versions of the Gephi application, a feature which was long requested before we finally became in a position to deliver it to you.

A new world of plugins

Now, every plugin download, even from the Gephi Application itself go through the marketplace! It means that as soon as a plugin goes live on our new platform, it’s directly available to every Gephi user around the world. Even if you haven’t made a plugin yourself, chances are you’ve already used the marketplace without realizing it!

Speaking of around the world, here’s a geographic breakdown of Gephi plugin downloads within the past 30 days:

Screen-Shot-2013-02-17-at-6.27.59-PM

Isn’t it impressive? 1169 plugin downloads and every continent is represented! (ok, not every single one. Props to the first Gephi user who’ll download a plugin from Antartica!)

It makes us extremely proud to help and foster such a vibrant community of Gephi users around the world. Remember that this map is for the last 30 days alone!

Today, we have 38 plugins listed on the marketplace, and we’ve made sure you can post your creations as easily as possible. It takes no more than 5 minutes to first fill in the plugin upload form, and we usually get back to you within 24 hours, so you can make your work available to thousands in record time!

A world of services

As briefly mentioned before, we’re also listing Gephi Services on the marketplace.

Over the years, we’ve noticed that a community of professionals and enthusiasts started providing services to the community. We’ve seen people doing trainings on Gephi, others integrating Gephi into data analysis workflows, or even create studies using Gephi as their tool of choice. We thought it was about time these fine folks got easier to find!

Every service provided by the community, for free or with a fee, is welcome to be listed. We already have a few, and to be honest we would love to see more of them!

Listing the services you provide on the Gephi marketplace is easy, and like with listing plugins it’s completely free, regardless of what you choose to charge for your work. We’re not here to collect fees. We just want to help other people know about your work, and to help you easily be found and easily be reached.

Following the same simple form submission process you can use for plugins, you give us all the info on what you’ve got for the community, you click ‘upload’, we get back to you, and then you have yourself a listing!… Which you can then customize to your liking, and for which you can include every single bit of information you want, and even a great copy! So not only can you let everyone know about what it is that you do, but also how truly wonderful it is to work with you and your expertise. So?

And now back to you.

Any question? Feel free to comment below! We’ll be glad to help you get started. And if you’d like to be kept in the know of what’s newly getting posted on the Marketplace, the RSS feed is this way.

In the meantime, thank you for your continued trust and your use of Gephi as your Graph Exploration and Manipulation tool of choice! It is our humble pleasure to serve you all.

See you soon on the marketplace! 🙂

0.8.2 beta released

The latest version of Gephi has been released, download it for Windows, Mac OS and Linux platforms. This release captitalizes the bugfixes and stability improvements we have done over the last few months. It also greatly improves the Mac OS X compatibility with the Gatekeeper and Retina Display support. Gephi should now starts right away when double-clicking on the App with a Gatekeeper-enabled computer. However if you have an older version of Gephi on your computer, you should uninstall it and remove the user directory, see the installation instructions.

This release is the first one based on our new Continuous Integration environment. This new system makes it easy for developers to create a new release and for beta-testers to test an early version. Users eventually get a software which has been tested much more heavily and by a larger population compared to previous releases.

Plugins need to be checked for compatibility. They will reappear on the Plugin Center in the coming days, as they are verified. Thanks for your patience.

Consult the release notes and the new Javadoc for more information.

New and Noteworthy

* Data Laboratory’s time interval merge strategy now supports custom date format
* Improved parser for dynamic types. Literal strings are now supported.
* Add Retina Display support to the Mac OS X version

Bug fixes

* Filters ‘Duplicate’ does not work (Issue 176)
* Exporting SVG File throws DomException due to invalid stroke-widths (Issue 697)
* File name entered is lost when changing folder (Issue 463)
* Datalab: can’t export all columns (Issue 628)
* NullPointerException when importing from database (Issue 691)
* Filter on column created with regex causes crash (Issue 663)
* “Long cannot be cast to a String” when either exporting a graph or saving a gephi file (Issue 679)
* Mapping of Escape Keyboard Shortcut for “Save changes before closing?” dialog box (Issue 686)
* DataLab: filling edge weight column doesn’t work when dynamic (Issue 619)
* Spreadsheet import of dynamic data: support of “infinity” (Issue 631)
* Missing license headers (Issue 664)
* Spreadsheet import and self-loops (Issue 683)
* Timelime interval set in infinite loop (Issue 712)
* OutDegreeRange broken (Issue 651)
* Shortest path on filtered graphs fails (Issue 650)
* Weighted degree computation fails (all values 0) when a filter is applied (Issue 636)
* Dot parser fixes (Issue 621)
* Filtering leads to Null Pointer exception when saving (Issue 617)
* Partition percentages incorrectly composed across filters (Issue 637)
* Start/end attributes are always imported using DATETIME format (Issue 649)
* HeatMap / Shortest Path on undirected Graph wrongly paints / calculates (Issue 630)
* Typo fix connetion to connection (Issue 642)
* Timeline sparkline bug (Issue 615)
* Fixes calculation of clustering coefficient on graph (Issue 625)
* Unix timestamp support (Issue 612)
* GML loading cannot accept scientific notation for float-type edge property (Issue 300)
* GML export does not respect specs (Issue 604)
* GEXF export outputs incorrect files (Issue 570)
* Problems with export of PNG files (Issue 601)
* Edge pencil: unable to set edge directedness (Issue 549)
* Presets of “Preview settings” are incorrectly / not saved (Issue 575)
* Option “Time intervals as dates” in Timeline (Issue 613)
* 8.1 and 8.0 both freeze at start-up when on network (Issue 592)
* Data laboratory max columns unintuitive (Issue 590)
* Modularity with Edge Weight Causes Array Out-of-Bounds (Issue 577)

Do Gephi technologies matter for your research or business? You can support us by donating to the Gephi Consortium, or becoming a member to have an impact on our roadmap.

The next version will be a new serie (0.9) and will bring a strengthen core and new features. Stay tuned for that.

Feel free to reach to us if you are willing to organize events (meetups, workshops, hackathon, etc.), we will support them.

Continuous Integration at Gephi

We recently finished to deploy a continuous integration environment at Gephi and I’m excited to share some of the highlights.

The Gephi developer team has been hard at work to change the way we iterate and create releases at Gephi. Developer productivity has been an important theme for this year’s focus and we already made several improvements. At the end of last year we migrated our code to GitHub and improved the documentation. We then focused on plugin developers and made it really easy to create new plugins with the Plugin Bootcamp and the new gephi-plugins repository. Finally, we’re now introducing a completely automated build and release production system.

Our objective was to automate the way releases are created and tested. Previously, creating a release was a manual process and included error prone tasks like updating configuration files, unzipping translations in the right folder or creating installers. Open-source tools like Maven, Jenkins and Nexus can help to make this process seamless and always have the latest deliverables available.

Maven migration

We migrated our code base from Ant to Maven. Gephi is based on the Netbeans Platform and has more than 80 different modules with dependencies and third-party librairies. Maven makes it easy to manage a large number of dependencies and put all configuration parameters in one place. Maven has also a large number of plugins and is very well integrated in Netbeans and Eclipse IDE.

Highlights:

  • A full application package, all Javadocs and sources are now produced and uploaded online with a single command.
  • Dependencies are all defined in one place. It is also much easier to update to the latest version of the Netbeans Platform.
  • All library JARs are dependencies to Maven Central or 3rd party repositories. No library JARs are directly included in the sources anymore.
  • The Gephi project is now a standard multi-module Maven project. It can therefore be opened and built in Eclipse or IntelliJ, as well as Netbeans out of the box
  • It facilitates module reuse in other projects like the Gephi Toolkit. Any other project can easily depend on any (or all) Gephi modules.

Jenkins server

Jenkins is the continuous integration server we chose to automate building and testing Gephi. It is configured to build and test Gephi every night based on the latest version of the code on GitHub. If the build fails, developers are informed something needs to be fixed.

Highlights:

  • Fully automated build in a stable environment. If something is wrong, it must be the code.
  • In addition of Gephi itself, we’re also building the Gephi Toolkit every night. Eventually, we’ll be able to build and test plugins as well.
  • Artifacts produced are uploaded to Nexus.

Nexus

Nexus is a repository for artifacts, which could either be librairies Gephi is using or release binaires like the latest release. At any time, beta testers can download the nightly build and test new features. We just announced a new beta testing program, which couldn’t be possible without the availability of the nightly build.

Highlights:

  • All 3rd party librairies have been uploaded to Nexus. Maven is using Nexus as a source for librairies.
  • The nightly build packages are available for download.
  • It also hosts the latest set of NBMs and Javadocs.


We learnt a lot during this project and will continue to strengthen the developer and beta-tester environment to scale up Gephi development. So far, we’ve done the Maven migration on a separate GitHub repository but we’ll soon convert the main repository and soon after release a 0.8.2 Gephi version. We’ve created a new Continuous Integration section on the Dev Portal and documented this project.

Plugin development remains the same for now and all plugins should be compatible with the new code base. In the next few months we would like to bring continuous integration to plugin developers as well. Testing at scale a large number of plugins at each new Gephi version remains a challenge and we would like to improve that. Also, we’ve seen issues where different plugins use different version of the same library and eventually cause crashes. Stay tuned for some news on that.

In the next few weeks we’ll update the documentation at various places how to build Gephi and work with the code. Developers interested to try this new system out should follow the instructions on GitHub or reach to us on the developer mailing-list.

Last but not least, we would like to say kudos to Maven, Jenkins and Nexus contributors for their huge and excellent work!

Supporting the NodeLink.io project: intuitive tools enable investigative reporters to visualize networks

It’s been a long time that the Gephi team is looking to the web as a platform for network visualization:

  • The Gephi Toolkit enables the use of Gephi on servers.
  • The Seadragon Export plugin facilitates interactive publishing (quickly followed by great plugins from the community).
  • The GraphGL experiment has advanced our understanding of WebGL, a promising technology.
  • A Web gallery with easy publishing system from Gephi is under development.

But so far we focused on the Gephi software itself. Part of our team is now willing to create a novel online platform which will benefit from our experience in making Gephi. With a focus on network sketching, easy data gathering and publishing facilities across various devices, we aim at democratizing the investigation and reporting of real-world networks: organisations, lobbying, government spending, crime networks, financial and human migration flows, social media, and more.

We believe that network thinking is of tremendous importance in investigative journalism: the media industry is facing big challenges, yet citizens need comprehensive stories on the complex situations that shakes our societies at a high pace (e.g. the Euro debt crisis, the Arab Spring, etc.). Investigating these events takes time and journalists struggle with scientific tools like Gephi to explore data and produce clear visuals. We lack of a new tool, open to everyone, to investigate networks and reveal evidences easily.

Hence we propose to develop NodeLink.io, an online platform for investigative reporters which makes node-link data actionable by easily sketching and publishing network visuals on various media. Journalists will gain a powerful tool to explain complex stories, and citizens will gain a better understanding of the mechanics that rule our societies.

We are applying for the Knight News Challenge to support this project. The Knight Foundation aims at accelerating media innovation by funding breakthrough ideas in news and information. Watch the introduction below or go directly to read our short proposal (500 words), and leave us a comment here to claim your support. Proposals are evaluated on the light of these discussions as well, so please give us early feedback!

The GSoC rocket launched

GSoC_2012_logo_250pxThe Google Summer of Code 2012 has officially started! The results have been announced by Google. Congratulations to the students who join the Gephi project:

  • Eduardo Espinoza – Legend Module in Preview
  • Romain Yon – Cloud Gephi
  • Taras Klaskovsky – Force Directed Edge Bundling algorithm
  • Vikash Anand – Statistics Reports and HTML5 Charts
  • Min WU – Interconnect Graph Streaming API and GraphStream

You put a lot of attention on doing the bests applications and demonstrate great motivation in addition to strong technical skills. We are very excited to work with you guys!

This year we are also honored to count on world-class researchers as mentors: Yoann Pigné is an Assistant Professor at the university of Le Havre, France, and is a leader of the GraphStream project. We will co-mentor the Graph Streaming project with André Panisson, our former Google Summer of Code student. André got his Ph.D. recently, and authored the video of the Egyptian Revolution on Twitter. Finally, Christian Tominski, who mentored the Preview refactoring last year, will mentor the Force Directed Edge Bundling project. He is a Lecturer and Researcher at the Institute for Computer Science at the University of Rostock. He has authored several articles in the field of information visualization.

Former Google Summer of Code students will also mentor and advise students, like Luiz Ribeiro.

The Summer Timeline:

* Until May 21: Students get to know mentors, read documentation, get up to speed to begin working on their projects.
* May 21: Students starts to code
* July 13: Mid-term evaluation
* August 24: Final evaluation

0.8.1 beta released

The latest version of Gephi has been released, download it for Windows, Mac OS and Linux platforms. This release delivers a complete new timeline component with many awaited features. The timeline is at the heart of our dynamic network analysis (longitudinal networks) effort at Gephi and this feature is the result of last year’s Google Summer of Code. It gives the ability to explore a network over time in a very powerful way.

timeline_sparkline
New Timeline with embedded sparkline. Example with a Twitter user network.

The timeline new features include animation (play button), custom bounds and embedded sparkline. In the previous Gephi version we added the ability to run metrics over time. For instance, one can calculate the average degree over time or simply the number of nodes (like in the example above). With the new timeline it’s possible to display a dynamic attribute directly in the component. For example if you’re looking at a Twitter network you can plot the total number of tweets as a sparkline.

This release also ships with three new languages: Czech, Chinese and Russian. Special Thanks to our awesome localization team.

separator

We just started the Google Summer of Code with exciting projects! We’re looking for talented students this summer (apply here). The team will also work on the next major release of Gephi (0.9). If you have ideas or suggestions, please don’t hesitate to express then on our forum or through GitHub issues. Our roadmap is open and most ideas comes from users’ experience.

Because it’s a major release, changes are not deployed through the AutoUpdate, you need to download and install the new version. Plug-ins also need to be checked for compatibility. They will reappear on the Plugin Center in the coming days, as they are verified. Thanks for your patience.

Consult the release notes and the new Javadoc for more information.

New features

tineline_thumb New Timeline

Exciting new timeline with animation, custom bounds settings and embedded spakline. The timeline interval and bounds can now be set manually by the user. By hitting the play button the interval moves automatically. All parameters can be configured. The timeline date support has been completely rewritten, now showing dates and periods in a readable way.

local_scale_thumb Local/Global Ranking

Compare networks over time or with different filters is now easier as Ranking lets you decide whether the scale should be local or global. In local scale, sizes and colors depend on the visible sub-graph. In global scale (set by default) they depend on the whole graph, thus allowing comparisons over time.

Screen-shot-2012-03-25-at-3.39.56-PM Weighted community-detection

Louvain Modularity (Community detection) adds edge weight support and resolution setting.

New and Noteworthy

* Better Preview performance when hidden edges (good for large graphs)
* Self-loop filter. Remove self loops in the graph.

New localization (Go to Tools > Languages)

* Russian
* Chinese (China)
* Czech

Bug fixes

* Exception when deserializing a preview preset (Issue 554)
* ClassCastException when saving a gephi file after PDF export (Issue 552)
* Cannot Preserve Z-Coordinate in GEXF exporter (Issue 547)
* Gephi won’t start on Mac OS X Lion because of a missing JAWT library (Issue 542)
* Add a local/global scale button to Ranking (Issue 541)
* Spreadsheet import: edge weight not imported (Issue 540)
* NOT operator wrong with topology filters (Issue 539)
* Multiple partition filters not named: cannot tell them apart (Issue 537)
* AttributeContollerImpl should be renamed to AttributeControllerImpl (Issue 530)
* Timeline – Set Custom Bounds does not support DATETIME (Issue 529)
* Data Laboratory – Dynamic Attributes Edit function resets values (Issue 528)
* DL exporter fix (Issue 525)
* Validator of MySQL’s “host” field (Issue 521)
* Edge/Node attributes don’t support datetime format (Issue 518)
* Directed Graph cluster coefficient provides incorrect values in certain situations (Issue 517)
* NullPointer in GUI when defining range Filters (Issue 516)
* datalab: empty column fails edge csv import (Issue 515)
* updateDimensions in PreviewController does not take into account node size (Issue 514)
* Overview’s graph area does not redraw edges automatically (Issue 513)
* c+p correction: HITS hub chart was repeated twice in output. (Issue 510)
* database import from Postgres does not handle node color attribute (Issue 506)
* Incorrect Teradata JDBC url (Issue 505)
* [Datalab] Duplicated nodes on import spreadsheet (Issue 498)
* [Datalab] Failing silently to load attribute values on import spreadsheet (Issue 497)
* Attribute Columns Property editors don’t support dynamic numbers (Issue 494)
* Arrows not hidden when edges hidden on Preview (Issue 493)
* Some text appears garbled when loaded from a project file (Issue 488)
* Exception when adding a new Perspective plugin development (Issue 486)
* NullPointer Exception with Edge Labels (Issue 478)
* Filter range does not filter if min == max (Issue 477)
* Heatmap Tool Sometimes Doesn’t Work (Issue 472)
* deleting a field in data lab yields an error (Issue 468)
* Email imports can’t be cancelled (Issue 466)
* Invalid XML character error when loading a gephi file (Issue 465)
* Range Slider writes N/A if the min and max values are the same (Issue 464)
* datalab: exception on hiding a column used to sort the table (Issue 454)
* Can’t build without Netbeans installed (Issue 452)
* Auto-refresh not updated after changing the spline (Issue 448)
* AttributeRangeFilter returns main graph when nothing matches filter (Issue 435)
* Timeline need more precision when dealing with dates (Issue 202)
* Degree Filter Not Filtering as Subfilter in Tookit (Issue 69)
* UnsatisfiedLinkError on startup on Mac OS X (Issue 44)

Plugins development

Last november we migrated our source code from Launchpad to GitHub and have seen an increase of activity in plugin development. We also simplified the environment and wrote more documentation how to get started. We created a dedicated repository on GitHub and compiled a list of more than 20 different code examples on the Gephi Plugins Bootcamp.

Do Gephi technologies matter for your research or business? You can support us by donating to the Gephi Consortium, or becoming a member to have an impact on our roadmap.

Feel free to reach to us if you are willing to organize events (meetups, workshops, hackathon, etc.), we will support them.