pyvis and networkx

The networkx and pyvis libraries are used for generating and visualizing network data, respectively.

Pyvis is fundamentally a python wrapper around the popular Javascript visJS library. Networkx instead of is a pretty sophisticated package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.

>>> from pyvis.network import Network
>>> import networkx as nx
# generate generic network graph instance
>>> nx_graph = nx.Graph()
# add some nodes and edges
>>> nx_graph.nodes[1]['title'] = 'Number 1'
>>> nx_graph.nodes[1]['group'] = 1
>>> nx_graph.nodes[3]['title'] = 'I belong to a different group!'
>>> nx_graph.nodes[3]['group'] = 10
>>> nx_graph.add_node(20, size=20, title='couple', group=2)
>>> nx_graph.add_node(21, size=15, title='couple', group=2)
>>> nx_graph.add_edge(20, 21, weight=5)
>>> nx_graph.add_node(25, size=25, label='lonely', title='lonely node', group=3)
# instantiatet pyvis network
>>> nt = Network("500px", "500px")
# populates pyvis network from networkx instance
>>> nt.from_nx(nx_graph)
>>> nt.show("nx.html")

It took me a little to familiarise with the libraries’ concepts and to generate some basic graphs. So, the tutorials linked below are meant to provide some reusable code building blocks for working with these tools.

Once you get the hang of it though, the fun part begins. What are the best data variables to represent in the graph? What color coding strategy is making it easier to explore the data? How many nodes/edges to display? Can we add some interactivity to the visualizations? Check out the resulting visualizations below for more ideas.

Dataviz: concepts co-occurence network

The Building a concepts co-occurence network notebook shows how to turn document keywords extracted from ‘semantic web’ publications into a simple topic map – by virtue of their co-occurrence within the same documents.

See also the standalone html version of the interactive visualization: concepts_network_2020-08-05.html

Dataviz: Organizations Collaboration Network

The Building an Organizations Collaboration Network Diagram notebook shows how to use publications’ authors and GRID data to generate a network of collaborating research organizations.

See also the standalone html version of the interactive visualization: network_2_levels_grid.412125.1.html

Yes – that’s right – command line. So not for everyone. Also: this is bare bones and pre-alpha. So don’t expect wonders. It basically provides a simple interface for searching a folder full of PDFs. That’s all for now!

Key features (or lack of)

First impressions

Pretty good. Was concerned I was gonna miss things like collections or tags. But I found a workaround: first, identify the papers I am interested in. Then, create a folder in the same directory and symlink them in there (= create an alias).

It’s not quite like uncarved wood, but it definitely feels simple enough.

The project is available on Github. In a nutshell, DimCli helps people becoming productive with the powerful scholarly analytics API from Dimensions. See the video below for a quick taster of the functionalities available.

Background

I recenlty joined the Dimensions team, so needed a way to get to grips with their feature-rich API (official docs). So, building DimCli has been a fun way for me to dig into the logic of the Dimensions Search Language (DSL).

Plus, this project gave me a chance to learn more about two awesome Python technologies: JupyterLab and its magic commands, as well as the Python Prompt Toolkit library.

Features

In a nutshell, this is what DimCli has to offer:

Feedback

DimCli lives on Github, so for any feedback or bug reports, feel free to open an issue there.

Over the past month I’ve been working on a new version of Ontospy, which is now available for download on Pypi.

What’s new in this version

This update includes support for Python 3, plus various other improvements that make it easier to query semantic web vocabularies using OntoSpy’s interactive shell module. To find out more about Ontospy:



Here’s a short video showing a typical sessions with the OntoSpy repl:

What’s new in this release

The main new features of version 1.6.7:



Install/update simply by typing pip install ontospy -U in your terminal window (see this page for more info).

Coming up next

I’d really like to add more output visualisations e.g. VivaGraphJS or one of the JavaScript InfoVis Toolkit.

Probably even more interesting, I’d like to refactor the code generating visualisations so that it allows people to develop their own via a standard API and then publishing them on GitHub.

Lastly, more support for instance management: querying and creating instances from any loaded ontology.

Of course, any comments or suggestions are welcome as usual – either using the form below or via GitHub. Cheers!

Tip: make the script executable and add an alias for it in order to be able to call it from the command line easily.

The script relies on the handy crosscite.org API, which is one of the wonderful services provided by CrossRef.

As part of his CS education work at Google, Philip Guo has been developing an open-source educational tool called Online Python Tutor. This tool enables teachers and students to write Python programs directly in the web browser and then single-step forwards and backwards to visualize what the computer is doing as it executes those programs. The tool has already been used by over 100,000 people but has a lot of potential for advancement. Philip is actively seeking partnerships with educators at all grade levels to deploy and improve the Online Python Tutor tool. Visit the URL for more information on using the tool and how to get involved.

Create, Test, Share

Once you’ve created a program, you can also share it online via a url, or get a snippet of code that will let you embed it in your site. Which is pretty neat! For example:

I emailed the customer service people at Flickr, they promptly replied that it wasn’t their fault but most likely a bug with other apps I had previously authorised to edit my Flickr collection (e.g. iPhoto or Aperture). Bad news: apparently whatever happened now what’s lost is lost forever. Not much to my consolation, the same happened to other people, for example check this post or this post to see alternative versions of the problem from 2010 and 2007.

So I’ve suddenly realised the cloud isn’t that secure a place, as yet. It’s time to change strategy: use flickr for sharing and my local HD for backup!

The good news is that if you know a little programming you can download your entire Flickr collection without having to pay a cent, for example by using Python. There are a few free libraries out there for accessing the Flickr APIs, such as flickrpy and FlickrAPI. They both require you to fiddle a little with the code (at the very least, get a personalised passkey from Flickr and add it to the python program) in order to get what you want.

The one I’ve gone for instead is a little package called flickrtouchr, which is even easier to use. After downloading you just have to run it from the command line and it’ll begin browsing your whole Flickr collection and download pictures at the highest resolution available. I have more than 8000 photos, and it worked like a charm – beware though – it took more than 10 hours on my TalkTalk connection.

Thanks Dan@hivelogic.com for writing this code – couldn’t be asking for more!

[mac]@mike:~/Dropbox/code/python/_libs/dan-hivelogic-flickrtouchr-9ba645b>python flickrtouchr.py ~/Desktop/FlickrBackupFolder

In order to allow FlickrTouchr to read your photos and favourites
you need to allow the application. Please press return when you've
granted access at the following url (which should have opened
automatically).

http://api.flickr.com/services/auth/?api_key=e2245325378b5675b4af4e8cdb0564716fa9bd&perms=read&frob=8856734hhgbbhsksd19443-caa77e89367asbbhfa2ba-600258&api_sig=a4aasdbbnb345c7fb46bdd33cfa65ec17bb32a

Waiting for you to press return

Egypt 1 ... in set ... Sharm el Sheik, Dec 2011
Egypt 2 ... in set ... Sharm el Sheik, Dec 2011
Egypt 3 ... in set ... Sharm el Sheik, Dec 2011
Egypt 4 ... in set ... Sharm el Sheik, Dec 2011

..... etc….

1. Python libraries for working with Rdf

RdfLib http://www.rdflib.net/

RdfLib (download) is a pretty solid and extensive rdf-programming kit for python. It contains parsers and serializers for RDF/XML, N3, NTriples, Turtle, TriX and RDFa. The library presents a Graph interface which can be backed by any one of a number of store implementations, including, memory, MySQL, Redland, SQLite, Sleepycat, ZODB and SQLObject.

The latest release is RdfLib 3.0, although I have the feeling that many are still using the previous release, 2.4. One big difference between the two is that in 3.0 some libraries have been separated into another package (called rdfextras); among these libraries there’s also the one you need for processing sparql queries (the rdf query language), so it’s likely that you want to install that too.
A short overview of the difference between these two recent releases of RdfLib can be found here. The APIs documentation for RdfLib 2.4 is available here, while the one for RdfLib 3.0 can be found here. Finally, there are also some other (a bit older, but possibly useful) docs on the wiki.

Next thing, you might want to check out these tutorials:

RdfAlchemy http://www.openvest.com/trac/wiki/RDFAlchemy

The goal of RDFAlchemy (install | apidocs | usergroup) is to allow anyone who uses python to have a object type API access to an RDF Triplestore. In a nutshell, the same way that SQLAlchemy is an ORM (Object Relational Mapper) for relational database users, RDFAlchemy is an ORM (Object RDF Mapper) for semantic web users.

RdfAlchemy can also work in conjunction with other datastores, including rdflib, Sesame, and Jena. Support for SPARQL is present, although it seems less stable than the rest of the library.

Fuxi http://code.google.com/p/fuxi/

FuXi is a Python-based, bi-directional logical reasoning system for the semantic web. It requires rdflib 2.4.1 or 2.4.2 and it is not compatible with rdflib 3. FuXi aims to be the ‘engine for contemporary expert systems based on the Semantic Web technologies’. The documentation can be found here; it might be useful also to look at the user-manual and the discussion group.

In general, it looks as if Fuxi can offer a complete solution for knowledge representation and reasoning over the semantic web; it is quite sophisticated and well documented (partly via several academic articles). The downside is that to the end of hacking together a linked data application.. well Fuxi is probably just too complex and difficult to learn.

ORDF ordf.org

ORDF (download | docs) is the Open Knowledge Foundation‘s library of support infrastructure for RDF. It is based on RDFLib and contains an object-description mapper, support for multiple back-end indices, message passing, revision history and provenance, a namespace library and a variety of helper functions and modules to ease integration with the Pylons framework.

The current version of this library is 0.35. You can have a peek at some of its key functionalities by checking out the ‘Object Description Mapper‘ – an equivalent to what an Object-Relational Mapper would give you in the context of a relational database. The library seems to be pretty solid; for an example of a system built on top of ORDF you can see Bibliographica, an online open catalogue of cultural works.

Django-rdf http://code.google.com/p/django-rdf/

Django-RDF (download | faq | discussiongroup) is an RDF engine implemented in a generic, reusable Django app, providing complete RDF support to Django projects without requiring any modifications to existing framework or app source code. The philosophy is simple: do your web development using Django just like you’re used to, then turn the knob and – with no additional effort – expose your project on the semantic web.

Django-RDF can expose models from any other app as RDF data. This makes it easy to write new views that return RDF/XML data, and/or query existing models in terms of RDFS or OWL classes and properties using (a variant of) the SPARQL query language. SPARQL in, RDF/XML out – two basic semantic web necessities. Django-RDF also implements an RDF store using its internal models such as Concept, Predicate, Resource, Statement, Literal, Ontology, Namespace, etc. The SPARQL query engine returns query sets that can freely mix data in the RDF store with data from existing Django models.

The major downside of this library is that it doesn’t seem to be maintained anymore; the last release is from 2008, and there seem to be various conflicts with recent versions of Django. A real shame!

Djubby http://code.google.com/p/djubby/

Djubby (download | docs) is a Linked Data frontend for SPARQL endpoints for the Django Web framework, adding a Linked Data interface to any existing SPARQL-capable triple stores.

Djubby is quite inspired by Richard Cyganiak’s Pubby (written in Java): it provides a Linked Data interface to local or remote SPARQL protocol servers, it provides dereferenceable URIs by rewriting URIs found in the SPARQL-exposed dataset into the djubby server’s namespace, and it provides a simple HTML interface showing the data available about each resource, taking care of handling 303 redirects and content negotiation.

Redland http://librdf.org/

Redland (download | docs | discussiongroup) is an RDF library written in C and including several high-level language APIs providing RDF manipulation and storage. Redland makes available also a Python interface (intro | apidocs) that can be used to manipulate RDF triples.

This library seems to be quite complete and is actively maintained; only potential downside is the installation process. In order to use the python bindings you need to install the C library too (which in turns depends on other C libraries), so (depending on your programming experience and operating system used) just getting up and running might become a challenge.

SuRF http://packages.python.org/SuRF/

SuRF (install | docs) is an Object – RDF Mapper based on the RDFLIB python library. It exposes the RDF triple sets as sets of resources and seamlessly integrates them into the Object Oriented paradigm of python in a similar manner as ActiveRDF does for ruby.

Other smaller (but possibly useful) python libraries for rdf:

2. RDF Triplestores that are python-friendly

An important component of a linked-data application is the triplestore (that is, an RDF database): many commercial and non-commercial triplestores are available, but only a few offer out-of-the-box python interfaces. Here’s a list of them:

Allegro Graph http://www.franz.com/agraph/allegrograph/

AllegroGraph RDFStore is a high-performance, persistent RDF graph database. AllegroGraph uses disk-based storage, enabling it to scale to billions of triples while maintaining superior performance. Unfortunately, the official version of AllegroGraph is not free, but it is possible to get a free version of it (it limits the DB to 50 million triples, so although useful for testing or development it doesn’t seem a good solution for a production environment).

The Allegro Graph Python API (download | docs | reference) offers convenient and efficient access to an AllegroGraph server from a Python-based application. This API provides methods for creating, querying and maintaining RDF data, and for managing the stored triples.

Open Link Virtuoso http://virtuoso.openlinksw.com/

Virtuoso Universal Server is a middleware and database engine hybrid that combines the functionality of a traditional RDBMS, ORDBMS, virtual database, RDF, XML, free-text, web application server and file server functionality in a single system. Rather than have dedicated servers for each of the aforementioned functionality realms, Virtuoso is a “universal server”; it enables a single multithreaded server process that implements multiple protocols. The open source edition of Virtuoso Universal Server is also known as OpenLink Virtuoso.

Virtuoso from Python is intended to be a collection of modules for interacting with OpenLink Virtuoso from python. The goal is to provide drivers for `SQLAlchemy` and `RDFLib`. The package is installable from the Python Package Index and source code for development is available in a mercurial repository on BitBucket.

Sesame http://www.openrdf.org/

Sesame is an open-source framework for querying and analyzing RDF data (download | documentation). Sesame supports two query languages: SeRQL and Sparql. Sesame’s API differs from comparable solutions in that it offers a (stackable) interface through wich functionality can be added, and the storage engine is abstracted from the query interface (many other Triplestores can in fact be used through the Sesame API).

It looks as if the best way to interact with Sesame is by using Java; however there is also a pythonic API called pySesame. This is essentially a python wrapper for Sesame’s REST HTTP API, so the range of operations supported (Log in, Log out, Request a list of available repositories, Evaluate a SeRQL-select, RQL or RDQL query, Extract/upload/remove RDF from a repository) are somehow limited (for example, there does not seem to be any native SPARQL support).

Talis platform http://www.talis.com/platform/

The Talis Platform (faq | docs)is an environment for building next generation applications and services based on Semantic Web technologies. It is a hosted system which provides an efficient, robust storage infrastructure. Both arbitrary documents and RDF-based semantic content are supported, with sophisticated query, indexing and search features. Data uploaded on the Talis platform are organized into stores: a store is a grouping of related data and metadata. For convenience each store is assigned one or more owners who are the people who have rights to configure the access controls over that data and metadata. Each store provides a uniform REST interface to the data and metadata it manages.

Stores don’t come free of charge, but through the Talis Connected Commons scheme it is possible have quite large amounts of store space for free. The scheme is intended to support a wide range of different forms of data publishing. For example scientific researchers seeking to share their research data; dissemination of public domain data from a variety of different charitable, public sector or volunteer organizations; open data enthusiasts compiling data sets to be shared with the web community.

Good news for pythonistas too: pynappl is a simple client library for the Talis Platform. It relies on rdflib 3.0 and draws inspiration from other similar client libraries. Currently it is focussed mainly on managing data loading and manipulation of Talis Platform stores (this blog post says more about it).

4store http://4store.org/

4store (download | features | docs) is a database storage and query engine that holds RDF data. It has been used by Garlik as their primary RDF platform for three years, and has proved itself to be robust and secure.
4store’s main strengths are its performance, scalability and stability. It does not provide many features over and above RDF storage and SPARQL queries, but if your are looking for a scalable, secure, fast and efficient RDF store, then 4store should be on your shortlist.

4store offers a number of client libraries, among them there are two for python: first, HTTP4Store is a client for the 4Store httpd service – allowing for easy handling of sparql results, and adding, appending and deleting graphs. Second, py4s, although this seems to be a much more experimental library (geared towards multi process queries).
Furthemore, there is also an application for the Django web framework called django-4store that makes it easier to query and load rdf data into 4store when running Django. The application offers some support for constructing sparql-based Django views.

End of the survey.. have I missed out on something? Please let me know if I did – I’ll try to keep adding stuff to this list as I move on with my project work!