Parerga und Paralipomena

HTML5 Canvas Cookbook

mike — Wed, 08 Feb 2012 17:34:12 +0000

HTML5 Canvas Cookbook is a new publication from Packt publishing that discusses in details the new drawing functionalities the html5 canvas element makes available; in the last weeks I've been looking at this book in more details and since it's been a quite useful learning experience I wanted to mention it here too.

The book (which is available online here )is simple and well organized; it spends quite a bit of time on both introductory topics and more advanced ones, so it'll probably fit both the beginner and the more experienced programmer.

Here's a list of the chapters available:

CHAPTER 1: GETTING STARTED WITH PATHS AND TEXT
Introduction
Drawing a line
Drawing an arc
Drawing a Quadratic curve
Drawing a Bezier curve
Drawing a zigzag
Drawing a spiral
Working with text
Drawing 3D text with shadows
Unlocking the power of fractals: Drawing a haunted tree

CHAPTER 2: SHAPE DRAWING AND COMPOSITES
Introduction
Drawing a rectangle
Drawing a circle
Working with custom shapes and fill styles
Fun with Bezier curves: drawing a cloud
Drawing transparent shapes
Working with the context state stack to save and restore styles
Working with composite operations
Creating patterns with loops: drawing a gear
Randomizing shape properties: drawing a field of flowers
Creating custom shape functions: playing card suits
Putting it all together: drawing a jet

CHAPTER 3: WORKING WITH IMAGES AND VIDEOS
Introduction
Drawing an image
Cropping an image
Copying and pasting sections of the canvas
Working with video
Getting image data
Introduction to pixel manipulation: inverting image colors
Inverting video colors
Converting image colors to grayscale
Converting a canvas drawing into a data URL
Saving a canvas drawing as an image
Loading the canvas with a data URL
Creating a pixelated image focus

CHAPTER 4: MASTERING TRANSFORMATIONS
Introduction
Translating the canvas context
Rotating the canvas context
Scaling the canvas context
Creating a mirror transform
Creating a custom transform
Shearing the canvas context
Handling multiple transforms with the state stack
Transforming a circle into an oval
Rotating an image
Drawing a simple logo and randomizing its position, rotation, and scale

CHAPTER 5: BRINGING THE CANVAS TO LIFE WITH ANIMATION
Introduction
Creating an Animation class
Creating a linear motion
Creating acceleration
Creating oscillation
Oscillating a bubble
Swinging a pendulum
Animating mechanical gears
Animating a clock
Simulating particle physics
Creating microscopic life forms
Stressing the canvas and displaying the FPS

CHAPTER 6: INTERACTING WITH THE CANVAS: ATTACHING EVENT LISTENERS TO SHAPES AND REGIONS 
Introduction
Creating an Events Class
Working With Canvas Mouse Coordinates
Attaching Mouse Event Listeners to Regions
Attaching Touch Event Listeners to Regions on a Mobile Device
Attaching Event Listeners to Images
Dragging-And-Dropping Shapes
Dragging-And-Dropping Images
Creating an Image Magnifier
Creating a Drawing Application

CHAPTER 7: CREATING GRAPHS AND CHARTS
Introduction
Creating a pie chart
Creating a bar chart
Graphing equations
Plotting data points with a line chart

CHAPTER 8: SAVING THE WORLD WITH GAME DEVELOPMENT
Introduction
Creating sprite sheets for the heroes and enemies
Creating level images and boundary maps
Creating an Actor class for the hero and enemies
Creating a Level class
Creating a Health Bar class
Creating a Controller class
Creating a Model class
Creating a View class
Setting up the HTML document and starting the game

CHAPTER 9: INTRODUCING WEBGL
Introduction
Creating a WebGL wrapper to simplify the WebGL API
Creating a triangular plane
Rotating a triangular plane in 3D space
Creating a rotating cube
Adding textures and lighting
Creating a 3D world that you can explore

The last chapters about animations and game developments are probably the most interesting ones, especially they all include detailed walk-through of the techniques discussed. Here's for example an example from Chapter 5, 'Oscillating a bubble':

In a nutshell, this is what is going on (note that the 'animation' library is discussed in a previous chapter of the book):

<script src="animation.js"></script>
<script>
    window.onload = function(){
        // instantiate new animation object
        var anim = new Animation("myCanvas");
        var context = anim.getContext();
        var canvas = anim.getCanvas();
        
        anim.setDrawStage(function(){
            // update
            var widthScale = Math.sin(this.getTime() / 200) * 0.1 + 0.9;
            var heightScale = -1 * Math.sin(this.getTime() / 200) * 0.1 + 0.9;

            // clear
            this.clear();
            
            //draw
            context.beginPath();
            context.save();
            context.translate(canvas.width / 2, canvas.height / 2);
            context.scale(widthScale, heightScale);
            context.arc(0, 0, 65, 0, 2 * Math.PI, false);
            context.restore();
            context.fillStyle = "#8ED6FF";
            context.fill();
            context.lineWidth = 2;
            context.strokeStyle = "#555";
            context.stroke();
            
            context.beginPath();
            context.save();
            context.translate(canvas.width / 2, canvas.height / 2);
            context.scale(widthScale, heightScale);
            context.arc(-30, -30, 15, 0, 2 * Math.PI, false);
            context.restore();
            context.fillStyle = "white";
            context.fill();
        });
        
        anim.start();
    };
</script>

All in all, a book definitely worth reading!

Setting up the new ‘staticfiles’ app on Django 1.3

mike — Tue, 24 Jan 2012 10:26:09 +0000

Even if I've been using Django 1.3 for a while now, I've been holding off on some of it new features, such as the new way to handle static files via a new app called (guess what) staticfiles . Essentially, what the new static app does is allowing you to leave all static files within the /static directory of each of the django apps you're using. This is for development; when you're deploying and (most likely) want these files to be served via a (faster) separate server process, the staticfiles app helps you gather all of them into a predefined directory by using a handy shell command. That's the gist of it. I finally took a look at django.contrib.staticfiles last week, so here's a brief report on how to get it to work (p.s. a couple of related threads on stack overflow can be found here and here).

1. Set up the static and media paths settings

My previous settings looked like this:

# the site root is one level up from where settings.py is
SITE_ROOT = os.path.dirname(os.path.realpath(__file__)).rsplit('/', 1)[0]
MEDIA_ROOT = os.path.join(SITE_ROOT, 'static')
MEDIA_URL = '/static/'
ADMIN_MEDIA_PREFIX = '/static/adminmedia1.3/'

The media root contained all of my static files, some of which I had to copy in there manually, each time I added a new django app to my project. On the production server, the MEDIA_URL is mapped to a local path (in the apache conf settings) that is essentially a duplicate of the /static directory we have in the development server. The only difference, the static stuff is delivered directly by Apache, bypassing django (=so to make it faster). The new way of declaring these variables is this instead:

MEDIA_URL = '/media/uploads/'
STATIC_URL = '/media/static/'
ADMIN_MEDIA_PREFIX = '/media/static/admin/'

# Absolute path to the directory that holds media uploaded
MEDIA_ROOT = os.path.join(SITE_ROOT, 'media/uploads')

# physical location of extra static files in development server
STATICFILES_DIRS = (
    os.path.join(SITE_ROOT, 'media/static'),
)
# path used with "python manage.py collectstatic"
STATIC_ROOT = os.path.join(SITE_ROOT, 'media/static_apache')

Obviously on a production server, you will have to set up the required aliases in the apache conf file, so that MEDIA_URL and STATIC_URL are pointing at the right physical locations:

Alias /media/uploads/ /path/to/mysite.com/media/uploads/
Alias /media/static/ /path/to/mysite.com/media/static_apache/

The django docs explain the new approach with these words:

In previous versions of Django, it was common to place static assets in MEDIA_ROOT along with user-uploaded files, and serve them both at MEDIA_URL. Part of the purpose of introducing the staticfiles app is to make it easier to keep static files separate from user-uploaded files. For this reason, you need to make your MEDIA_ROOT and MEDIA_URL different from your STATIC_ROOT and STATIC_URL. The STATIC_ROOT directory is not necessary in the development server: in fact is only used if you call the collectstatic manangement command. It's not needed to add the directory to the STATICFILES_DIRS setting to serve your static files! Furthemore, the STATICFILES_DIRS variable tells Django of the location of static files which are not within specific applications. Mind that during the 'collection' operations also these files will be copied into the STATIC_ROOT directory.

2. Add context processors and the app

Add the required context processor in setting.py:

TEMPLATE_CONTEXT_PROCESSORS += 'django.core.context_processors.static'

Add also the new static app to your installed apps:

INSTALLED_APPS = (
....    
    'django.contrib.staticfiles',    
.....
)

Keep in mind that it's very likely that in your templates you will have to manually change all references to MEDIA_URL into STATIC_URL!

3. Url configuration

On your development server, this is what you would do:

from django.contrib.staticfiles.urls import staticfiles_urlpatterns
# ... the rest of your URLconf goes here ...
urlpatterns += staticfiles_urlpatterns()

This will inspect your STATIC_URL setting and wire up the view to serve static files accordingly. Don't forget to set the STATICFILES_DIRS setting appropriately to let django.contrib.staticfiles know where to look for files additionally to files in app directories.

WARNING: the staticfiles_urlpatterns helper function will only work if DEBUG is True and your STATIC_URL setting is neither empty nor a full URL such as http://static.example.com/ (more info here).

Finally, mind that in this new approach you will need to arrange for serving of files in MEDIA_ROOT yourself; staticfiles does not deal with user-uploaded files at all. You can, however, use django.views.static.serve() view for serving MEDIA_ROOT in development (for more info see Serving other directories).

if settings.LOCAL_SERVER:     # ===> static files on local machine    
    urlpatterns += patterns('', 
        (r'^media/uploads/(?P.*)$', 'django.views.static.serve', 
            {'document_root': settings.MEDIA_ROOT, 'show_indexes': True}),
        )

In the end, I conflated the two things into this code (ps I've added a variable called LOCAL_SERVER to quickly see which platform I'm on):

if settings.LOCAL_SERVER:     # ===> static files on local machine
    from django.contrib.staticfiles.urls import staticfiles_urlpatterns
    urlpatterns += staticfiles_urlpatterns()    
    urlpatterns += patterns('', 
        (r'^media/uploads/(?P.*)$', 'django.views.static.serve', 
            {'document_root': settings.MEDIA_ROOT, 'show_indexes': True}),
        )

4. On your production server

Easy: in your urlconf there's no need to do anything, as static urls are usually handled by Apache directly. However for that to happen what you have to do is collect all static files into the directory that apache is looking into, that is, the one specified with the STATIC_ROOT setting. This is how you do it:

python manage.py collectstatic

This shell command will a) look in the /static/ directory of each of the apps of yours INSTALLED_APPS setting. b) look in directories you specify in the STATICFILES_DIRS setting. …and copy whatever it finds into the STATIC_ROOT directory, as defined in your settings (ps: it'll preserve the original directory structure). That's all folks!

Towards a conceptual model for the domain of Sculpture

mike — Sat, 19 Nov 2011 14:44:09 +0000

For the next two years I'll be collaborating with the Art of Making project. The project investigates the processes involved in the carving of stone during the Roman period, in particular it aims at analysing them using the insights and understanding Peter Rockwell (son of Norman Rockwell) developed during his lifelong experience as a sculptor. Eventually we will present these results by means of a freely accessible online digital resource that guides users through examples of stone carving. In this post I just wanted to report on the very first discussions I had with the sculpture and art scholars I'm working with, to the purpose of creating a shared model for this domain. The project started this July, it is based at King’s College London and is funded by the Leverhulme Trust. I'm more involved with the digital aspects of the project, and as usual one of the first steps involved in the building of a digital resource (in particular, a database-backed digital resource) is the construction of a conceptual model that can represent the main types of things being dealt with. In other words, it is fundamental to identify which are the things our database and web-application should 'talk about'; later on, this model can be refined and extended so to become an abstract template of the data-manipulation tasks the software application must be capable of supporting (e.g. entering data into the system, searching and visualising them). Here's a nice example of the sculptures (a sarcophagus from Aphrodisias) that constitute our 'source' materials:

What are the key entities in the sculpture domain?

To this purpose, a few weeks ago we had a very productive brainstorming session aimed at fleshing out the main items of interest in the world of sculpture. This is a very first step towards the construction of a formal model for this domain; nonetheless, I think that we have already managed to pin down the key elements we're going to be dealing with in the next two years. Here's a list of the main objects we identified:

- People, such as craftsman's etc..
- Sculptures (of various kinds)
- Materials
- Tools
- Generic processes that are part of a sculpting project, such as quarrying and transport.
- More specific methods being used within a particular process, e.g. carving styles, or approaches to quarrying.
- Traditions, conceptualisations of the 'way of doing things' that, in turn, can inspire the way methods and processes are carried out nowadays.

We encoded the results of our discussions in a mind map for better readability, and also in order to use a technology that would make it easier to share our findings later on. I added it below.. (in case the interactive image doesn't work, you can find it here too).

Fleshing out the model a bit more

After a few weeks of work we did a reiteration of the conceptual map above. The good news was that it soon became evident to us that we got it quite right on the first round; that is, we didn't really feel like adding or removing anything from the map. On the other hand, we thought we should try to add some relations (= links, arcs) among the concepts (=bubbles) previously identified, so to characterize their semantics a bit more. I had a go at adding some relations first, and here's the result: I should specify that I have no knowledge whatsoever of the domain of sculpture, so the stuff I added to the map came out entirely from the (little) research I've been doing on the subject (on and off) during the last weeks. At the same time, also Will and Ben (the art historians I'm collaborating with) worked independently at the task of fleshing out the mind map with more relations. Needeless to say, what they came up with is way more dense and intricate than what I could have ever imagined! This is probably not surprising, as one would expect to see a significant difference between a non-expert's representation of a subject domain and another one which is instead created by experts. Still, it was interesting to see it happening with my own eyes! Here it is: The next step will be trying to reduce the (natural) complexity of the portion of the world we are representing to a more manageable size… then, formalize it, and start building our database based on that.. stay tuned for more!

Upgrading to Lion: tips and tricks

mike — Wed, 19 Oct 2011 14:01:18 +0000

I've finally decided to upgrade my mac operating system from Snow Leopard (10.6.8) to Lion (10.7.2). Not really for the new features such as Mission Control and LaunchPad (which although being pretty cool are not going to change my life) but mostly for the fact that the new OS is well integrated with iCloud. Sharing my iCal calendars, Address book contacts and other useful stuff across my apple computers and mobile devices is going to be a digital-life-changer, hopefully for the better. Unfortunately as usual an OS upgrade is never as smooth as I would like it to be; so here's a list of the problems I ran into (and still am), with pointers to solutions whenever I found one.

Tip: A useful place where to get info about Lion is here: http://osxdaily.com/tag/mac-os-x-10-7/

1. Wi-Fi disconnection issues

This is quite a bad one: my wifi connection wouldn't stay on for more than 2-3 minutes at times, making it impossible to do any continuative work online. After a lot of swearing and searching online, I found a good solution on this article on iLounge.com. The article offers many solutions, but the one that worked for me is to manually downgrading the Atheros WiFi driver which seems to be ill on Lion to the version provided with Snow Leopard (comment 53). Here's the recipy:

1. run kextstat | grep -i atheros if a line was returned, you are most likely using Atheros WiFi card and the related kext 2. In your home folder create two directories: cd && mkdir New Old 3. If you have a copy (time machine backup) or other system installed with Snow Leopard (as latest as possible), take a copy of /System/Library/Extensions/IO80211Family.kext and put it into ~/Old/ directory [P.S. if you don't know where to get that file, I've uploaded a copy here: IO80211Family.kext.zip] 4. (Back on to your main system) gain root access: sudo -s 5. Move: mv /System/Library/Extensions/IO80211Family.kext ~/New/ 6. Move cache file: mv /System/Library/Caches/com.apple.kext.caches/Startup/kernelcache ~/New/ 7. copy old kext: cp -r ~/Old/IO80211Family.kext /System/Library/Extensions/ 8. go to: cd /System/Library/Caches/com.apple.kext.caches/Startup/ 9. rebuild cache: kextcache -v 1 -a i386 -a x86_64 -mkext kernelcache /System/Library/Extensions 10. Using GUI Utilities->Disk Utility->Macintosh HD->Repair permissions 11. pray and reboot

2. Install the latest Xcode and Developer Tools

This is number 2 in the list, but really it should be number 1 if you are a software developer. The latest Xcode (4.2, get it on the apple store for free) includes all sorts of low and high level programming frameworks, from C compilers to Python interpreters. In other words, without reinstalling Xcode tons of software development environments won't run as usual! Installing Xcode is very easy, here's a nice tip to keep in mind though (as found here):

When installing XCode, for some unknown unholy reason, if you have not quit itunes, and itunes helper (see activity monitor) prior to starting the XCode installer, the install will hang. Do yourself a favor and kill it with fire.

Also, two more crucial things to keep in mind:

- Remember; the binary directory for the dev tools is in /Developer/usr/bin/ — this includes gcc-4.2 - Do yourself a favor, drop “export ARCHFLAGS="-arch x86_64"” into your .bash_profile.

The last point is extremely important; make sure you do that otherwise you'll easily run into all sorts of 32 vs 64 bits architectures conflicts that (if you're like me) you have no time to get into (more info here).

3. Fix your Python installation

On my iMac I couldn't run iPython anymore; this was surprisingly easy to fix, I just run again easy_install ipython from the command line and there you go iPython went back up (version 2.7, the default one with Lion). This didn't work on my other mac though (a slightly older MacBook); the easy_install ipython trick failed with a DistributionNotFound error; if this is the case, you probably have to re-install Apple's Developer Tools (see point 2 above). However before doing that you might want to consider this good tip which I found here:

1. Make a list of all the third-party libraries you currently have installed under Python 2.6, because you’ll have to reinstall all of them for 2.7 and you won’t have the 2.6 site-packages directory to refer to. 2. Update to Lion. 3. Update to Xcode 4.1. It’s free in the App Store.

Good, so I installed XCode 4 (= Apple's Developer Tools) but I still had a problem: for some reason the default python binary (/usr/bin/python) was still pointing at the 2.5 release, instead of the 2.7 that comes with Lion. This was easily fixed by issuing this command:

defaults write com.apple.versioner.python Version 2.7

Two final steps are then required:

a) Download the 2.7 version of setuptools and install it (e.g. sudo sh setuptools-0.6c11-py2.7.egg); b) re-install iPython via setuptools: sudo easy_install ipython

Obviously I'll have to manually install under 2.7 all the libraries I used to have on 2.6, but that can be done incrementally and thanks to easy_install it's also quite quick. For more discussions on python and Lion, check out these posts: Quick Python/Developer tips for OSX Lion and Python problems on Lion.

4. The Mercurial binaries disappeared

What the heck. You need to reinstall Mercurial using the package available here: http://mercurial.berkwood.com/ and Python problems on Lion

5. Omnigraffle can't save files anymore

My 5.0 copy of Omnigraffle loaded and let me create stuff just fine, but it won't save files anymore. I got an error message that looked like an internal error ("class bla bla can't be subclassed by bla bla bla") each time I tried to save my work. Bugger. Couldn't find a solution to this; I guess that getting the latest version (5.3 I think) will solve the issue (it' 70 dollars though).

6. Java runtime missing

Lion does not provide a Java runtime by default, so you need to install it separately. No worries: that's going to happen automatically as soon as you try to run a Java application; alternatively, you can do that manually here.

7. The Finder Sidebar Icons colours (well lack of..) are depressing

Why on earth haven't they made that a preference we can change? There's a way around it though, and it is described here. This is the gist of it:

- Download and install SIMBL, which you can get here - Download the ColorfulSidebar SIMBL plugin (direct link) or visit the developers home and mount the DMG file - Move the ColorfulSidebar.bundle into the following SIMBL plugin folder: ~/Library/Application Support/SIMBL/Plugins/ - Either login and logout of Mac OS X, or just kill the Finder through the Terminal to relaunch it: killall Finder

Unfortunately this solution won't stick after an OS reload; but it's enough to relaunch the Finder (killall Finder) to reload the plugin. P.S. If you can't access the ~/Library folder using Finder, check out point 12 below!

8. Safari Won't Reopen To HomePage

Apple changed the way new windows open. In Safari, if you set the "New windows open with" preference to "Homepage", then ALL new windows will now open with your homepage setting, including the initial startup window. In past versions, you could set that preference to "Same Page" while startup would auto open the homepage. Solution: go to System Preferences/General/ and uncheck the "Restore windows when quitting and re-opening apps" option.

9. The CD-TO utility disappeared

A small utility I've been using all the time is cdto, a mini application that opens a Terminal.app window cd'd to the front most finder window. That doesn't work anymore, but the good news is that as of Mac OS X Lion 10.7, Terminal includes exactly this functionality as a Service, which you can set by going to System Preferences > Keyboard > Keyboard Shortcuts > Services (check this useful blog post for more details). Also, a more in-depth discussion of the various options available can be found on stackOverflow. In addition, Lion Terminal will open a new terminal window if you drag a folder (or pathname) onto the Terminal application icon, and you can also drag to the tab bar of an existing window to create a new tab. Update: CDTO has a Lion version available too. Also, another free software that offers the same functionalities is Go2Shell.

10. Visor doesn't work anymore

Yes because Visor (a pumped-up version of the terminal application) relied on SIMBL, which is handled different on Lion. Not a problem, just download TotalTerminal, which is an updated and more stable version of Visor. For original Visor 2.2 users: TotalTerminal plugin is not injected into Terminal.app automatically like with SIMBL. You have to launch TotalTerminal.app to inject the plugin into Terminal.app. You might want to put TotalTerminal.app into Startup Items.

11. VmwareFusion won't work

VmwareFusion is a Virtual-PC software that people use to run windows on a mac; I was using an older version of it and it looks as if this is incompatible with the 64-bit architecture of the Lion operating system. Solution: nothing else than getting the latest version of VmwareFusion, which currently sells for 50 dollars. Bugger.

12. Customize the Finder

Get the Library folder to show again: by default, the Library folder in your home directory doesn't show in Finder. Just type this into your terminal app to fix it:

chflags nohidden ~/Library/

Show hidden files in finder: beware though, this might result in a lot of information being displayed which is not needed.

defaults write com.apple.finder AppleShowAllFiles -bool YES

Show full paths in finder: it's quite useful to know always where you are.

defaults write com.apple.finder _FXShowPosixPathInTitle -bool YES

.. and many more customisations are listed on http://secrets.blacktree.com/. Alternatively, you can customise your mac to the bone without touching the console by downloading a free utility software called TinkerTool.

13. Get the Old Apple Mail Back

The new Apple Mail interface is quite slick, but I still have some problems adapting to it. You can easily revert back to the old interface just by going to the Mail menu and choose Preferences. From there, click Viewing and you'll find the options you want right at the top.

14. .. ? (still looking for it!)

Seminar: Tagore digital editions and Bengali textual computing

mike — Thu, 13 Oct 2011 17:01:52 +0000

Professor Sukanta Chaudhuri yesterday gave a very interesting talk on the scope, methods and aims of 'Bichitra' (literally, 'the various'), the ongoing project for an online variorum edition of the complete works of Rabindranath Tagore in English and Bengali. The talk (part of this year's DDH research seminar) highlighted a number of issues I personally wasn't much familiar with, so in this post I'm summarising them a bit and then highlighting a couple of possible suggestions. Sukanta Chaudhuri is Professor Emeritus at Jadavpur University, Kolkata (Calcutta), where he was formerly Professor of English and Director of the School of Cultural Texts and Records. His core specializations are in Renaissance literature and in textual studies: he published The Metaphysics of Text from Cambridge University Press in 2010. He has also translated widely from Bengali into English, and is General Editor of the Oxford Tagore Translations. Rabindranath Tagore (1861 – 1941), the first nobel laureate of Asia, was arguably the most important icon of modern Indian Renaissance. This recent project on the electronic collation of Tagore texts, called 'the Bichitra project', is being developed as part of the national commemoration of the 150th birth anniversary of the poet (here's the official page). This is how the School of Cultural Texts and Records summarizes the project's scope:

The School is carrying out pioneer work in computer collation of Tagore texts and creation of electronic hypertexts incorporating all variant readings. The first software for this purpose in any Indian language, named “Pathantar” (based on the earlier version “Tafat”), has been developed by the School. Two pilot projects have been carried out using this software, for the play Bisarjan (Sacrifice) and the poetical collection Sonar Tari (The Golden Boat). The CD/DVDs contain all text files of all significant variant versions in manuscript and print, and their collation using the ”Pathantar” software. The DVD of Sonar Tari also contains image files of all the variant versions. These productions are the first output of the series “Jadavpur Electronic Tagore”. Progressing from these early endeavours, we have now undertaken a two-year project entitled "Bichitra" for a complete electronic variorum edition of all Tagores works in English and Bengali. The project is funded by the Ministry of Culture, Government of India, and is being conducted in collaboration with Rabindra-Bhavana, Santiniketan. The target is to create a website which will contain (a) images of all significant variant versions, in manuscript and print, of all Tagores works; (b) text files of the same; and (c) collation of all versions applying the "Pathantar" software. To this end, the software itself is being radically redesigned. Simultaneously, manuscript and print material is being obtained and processed from Rabindra-Bhavana, downloaded from various online databases, and acquired from other sources. Work on the project commenced in March 2011 and is expected to end in March 2013, by which time the entire output will be uploaded onto a freely accessible website.

A few interesting points

Tagore, as Sukanta noted, "wrote voluminously and revised extensively". From a DH point of view this means that creating a comprehensive digital edition of his works would require a lot of effort - much more than what we could easily pay people for, if we wanted to mark up all of this text manually. For this reason it is fundamental to find some type of semi-automatic methods for aligning and collating Tagore's texts, e.g. the ”Pathantar” software. Follows a screenshot of the current collation interface.

The Bengali language, which is used by Tagore, is widely spoken in the world (it is actually one of the most spoken languages, with nearly 300 million total speakers). However this language poses serious problems for a DH project. In particular, the writing system is extremely difficult to parse using traditional OCR technologies: its vowel graphemes are mainly realized not as independent letters but as diacritics attached to its consonant letters. Furthermore clusters of consonants are represented by different and sometimes quite irregular forms, thus learning to read is complicated by the sheer size of the full set of letters and letter combinations, numbering about 350 (from wikipedia).

One of the critical points that emerged during the discussion had to do with the visual presentation of the results of the collation software. Given the large volume of text editions they're dealing with, and the potential vast amount of variations between one edition and the others, a powerful and interactive visualization mechanism seems to be strongly needed. However it's not clear what are the possible approaches on this front..

Textual computing, Sukanta pointed out, is not as developed in India as it is in the rest of the world. As a consequence, in the context of the "Bichitra" project widely used approaches based on TEI and XML technologies haven't really been investigated enough. The collation software mentioned above obviously marks up the text in some way; however this markup remains hidden to the user and much likely it is not compatible with other standards. More work would thus be desirable in this area - in particular within the Indian continent.

Food for thought

On the visualization of the results of a collation. Some inspiration could be found in the type of visualizations normally used in version control software systems, where multiple and alternative versions of the same file must be tracked and shown to users. For example, we could think of the visualizations available on GitHub (a popular code-sharing site), which are described on this blog post and demonstrated via an interactive tool on this webpage. Here's a screenshot: The situation is striking similar - or not? Would it be feasible to reuse one of these approaches with textual sources? Another relevant visualization is the one used by popular file-comparison softwares (eg File Merge on a Mac) for showing differences between two files:

On using language technologies with Bengali. I did a quick tour of what's available online, and (quite unsurprisingly, considering the reputation Indian computer scientists have) found several research papers which seem highly relevant. Here's a few of them: - Asian language processing: current state-of-the-art [text] - Research report on Bengali NLP engine for TTS [text] - The Emile corpus, containing fourteen monolingual corpora, including both written and (for some languages) spoken data for fourteen South Asian languages [homepage] - A complete OCR system for continuous Bengali characters [text] - Parsing Bengali for Database Interface [text] - Unsupervised Morphological Parsing of Bengali [text]

On open-source softwares that appear to be usable with Bengali text. Not a lot of stuff, but more than enough to get started (the second project in particular seems pretty serious): - Open Bangla OCR - A BDOSDN (Bangladesh Open Source Development Network) project to develop a Bangla OCR - Bangla OCR project, mainly focused on the research and development of an Optical Character Recognizer for Bangla / Bengali script

Any comments and/or ideas?

Django B_project available on gitHub

mike — Wed, 12 Oct 2011 09:34:00 +0000

I've put together and shared on gitHub all the files needed to get started with a Django project - essentially this is a clean file structure usable to get going with a Django project nice and easy. The idea is that after downloading this, you just have to change a couple of names (see the README file for more details) and there you go you have a complete Django project ready to be ran and customized. The code is here: https://github.com/magicrebirth/b_project. Feel free to download or fork it as you like! Currently the B_project is set up to work with Django 1.3 and it has a few 'batteries' included (= commonly used apps):

MPTT: Utilities for implementing a modified pre-order traversal tree in django

REGISTRATION: A user-registration application for Django.

DJANGO EXTENSIONS: global custom management extensions for the Django Framework.

PICKLEFIELD: an implementation of a pickled object field.

The other things included in the framework are:

a settings.py file that calls localized settings specifications depending on parameters (eg dev server or live one)

a bunch of generic python utilities

a few 'enhanced' model classes

all that is needed for working with trees in the admin (as discussed here: Django admin and MPTT)

a javascript autocomplete framework, integrated in the admin (as discussed here: Autocomplete in Django)

some basic templates and templatetags

Enjoy!

Event: THATcamp Kansas and Digital Humanities Forum

mike — Wed, 28 Sep 2011 16:56:55 +0000

The THATcamp Kansas and Digital Humanities Forum happened last week at the Institute for Digital Research in the Humanities, which is part of the University of Kansas in beautiful Lawrence. I had the opportunity to be there and give a talk about some recent stuff I've been working on regarding digital prosopography and computer ontologies, so in this blog post I'm summing up a bit the things that caught my attention while at the conference. The event happened on September 22-24 and consistend of three separate things:

Bootcamp Workshops: a set of in-depth workshops on digital tools and other DH topics http://kansas2011.thatcamp.org/bootcamps/.

THATCamp: an “unconference” for technologists and humanists http://kansas2011.thatcamp.org/.

Representing Knowledge in the DH conference: a one-day program of panels and poster sessions (schedule | abstracts )

The workshop and THATcamp were both packed with interesting stuff, so I strongly suggest you take a look at the online documentation, which is very comprehensive. In what follows I'll instead highlight some of the contributed papers which a) I liked and b) I was able to attend (needless to say, this list matches only my individual preference and interests). Hope you'll find something of interest there too!

A (quite subjective) list of interesting papers

The Graphic Visualization of XML Documents, by David Birnbaum ( abstract ): a quite inspiring example of how to employ visualizations in order to support philological research in the humanities. Mostly focused on Russian texts and XML-oriented technologies, but its principles easily generalizable to other contexts and technologies.

Exploring Issues at the Intersection of Humanities and Computing with LADL, by Gregory Aist ( abstract ): the talk presented LADL, the Learning Activity Description Language, a fascinating software environment provides a way to "describe both the information structure and the interaction structure of an interactive experience", to the purpose of "constructing a single interactive Web page that allows for viewing and comparing of multiple source documents together with online tools".

Making the most of free, unrestricted texts–a first look at the promise of the Text Creation Partnership, by Rebecca Welzenbach ( abstract ): an interesting report on the pros and cons of making available a large repository of SGML/XML encoded texts from the Eighteenth Century Collections Online (ECCO) corpus.

The hermeneutics of data representation, by Michael Sperberg-McQueen ( abstract ): a speculative and challenging investigation of the assumptions at the root of any machine-readable representation of knowledge - and their cultural implications.

Breaking the Historian’s Code: Finding Patterns of Historical Representation, by Ryan Shaw ( abstract ): an investigation on the usage of natural language processing techniques to the purpose of 'breaking down' the 'code' of historical narrative. In particular, the sets of documents used are related to the civil rights movement, and the specific NLP techniques being employed are named entity recognition, event extraction, and event chain mining.

Employing Geospatial Genealogy to Reveal Residential and Kinship Patterns in a Pre-Holocaust Ukrainian Village, by Stephen Egbert.( abstract ): this paper showed how it is possible to visualize residential and kinship patterns in the mixed-ethnic settlements of pre-Holocaust Eastern Europe by using geographic information systems (GIS), and how these results can provide useful materials for humanists to base their work on.

Prosopography and Computer Ontologies: towards a formal representation of the ‘factoid’ model by means of CIDOC-CRM, by me and John Bradley ( abstract ): this is the paper I presented (shameless self plug, I know). It's about the evolution of structured prosopography (= the 'study of people' in history) from a mostly single-application and database-oriented scenario towards a more interoperable and linked-data one. In particular, I talked about the recent efforts for representing the notion of 'factoids' (a conceptual model normally used in our prosopographies) using the ontological language provided by CIDOC-CRM (a computational ontology commonly used in the museum community).

That's all! Any comments/further thoughts on such things?

First steps with Canvas and HTML5

mike — Thu, 25 Aug 2011 14:47:35 +0000

I've been postponing experimenting with HTML5 for quite a while, so today I finally set aside a few hours to play with it. This is a very simple example of the graphic effects you can create using javascript and the HTML5 canvas element..

Note that the examples below will work only on HTML5-compatible browsers, such as the latest versions of Chrome or Firefox..

1. Setting up a canvas and drawing some lines on it.

This is easily done using the moveTo and lineTo functions. So let's create a simple symmetrical geometrical shape..

Here's the source code:


<html>
<body>

<canvas id="myCanvas1" width="800" height="400" style="border:1px dashed #c3c3c3;">
Your browser does not support the canvas element.
canvas>

<script type="text/javascript">

var c=document.getElementById("myCanvas1");
var cxt=c.getContext("2d");
var i=0;

for (i=1;i<=5;i++)
{
cxt.moveTo(0, 400);
cxt.lineTo(100, i * 40);
cxt.lineTo(300, i * 60);
cxt.lineTo(400, i * 30);
cxt.lineTo(500, i * 60);
cxt.lineTo(700, i * 40);
cxt.lineTo(800, 400);
}

cxt.stroke();

script>

body>
html>

2. Creating a mirroring effect

Now we can increase the size of the canvas to 800 and replicate these lines at the bottom of the canvas so to achieve a 'mirroring' effect.. since we know the max Y value of the canvas (800), let's just add another loop that draws the same lines but inverts the Y position. This is easily achieved by subtracting the constant-dependent parameter from the max value of Y.

This is how the new source code looks like:

<canvas id="myCanvas2" width="800" height="800" style="border:1px dashed #c3c3c3;">
Your browser does not support the canvas element.
canvas>

<script type="text/javascript">

var c=document.getElementById("myCanvas2");
var cxt=c.getContext("2d");
var i=0;

for (i=1;i<=5;i++)
{
cxt.moveTo(0, 400);
cxt.lineTo(100, i * 40);
cxt.lineTo(300, i * 60);
cxt.lineTo(400, i * 30);
cxt.lineTo(500, i * 60);
cxt.lineTo(700, i * 40);
cxt.lineTo(800, 400);
}


for (i=5;i>=1;i--)
{
cxt.moveTo(0, 400);
cxt.lineTo(100, 800 - (i * 40));
cxt.lineTo(300, 800 - (i * 60));
cxt.lineTo(400, 800 - (i * 30));
cxt.lineTo(500, 800 - (i * 60));
cxt.lineTo(700, 800 - (i * 40));
cxt.lineTo(800, 400);
}

cxt.stroke();

script>

3. Adding more graphical interest

Finally, let's parametrize a bit more the construction of this geometrical pattern by including everything into another loop, and using this new counter to increment the points ordinate position. We'll add a new variable y and use it to run the external loop 20 times. Here's the result:

Not too bad uh? This last modification to the code looks like this:

<canvas id="myCanvas3" width="800" height="800" style="border:1px dashed #c3c3c3;">
Your browser does not support the canvas element.
canvas>

<script type="text/javascript">

var c=document.getElementById("myCanvas3");
var cxt=c.getContext("2d");
var i=0;
var y=0;

for (y=1;y<=20;y++)
{
    
    for (i=1;i<=5;i++)
    {
    cxt.moveTo(0, 400+y);
    cxt.lineTo(100, i * (40+y));
    cxt.lineTo(300, i * (60+y));
    cxt.lineTo(400, i * (30+y));
    cxt.lineTo(500, i * (60+y));
    cxt.lineTo(700, i * (40+y));
    cxt.lineTo(800, 400+y);
    }


    for (i=5;i>=1;i--)
    {
    cxt.moveTo(0, 400);
    cxt.lineTo(100, 800 - (i * (40+y)));
    cxt.lineTo(300, 800 - (i * (60+y)));
    cxt.lineTo(400, 800 - (i * (30+y)));
    cxt.lineTo(500, 800 - (i * (60+y)));
    cxt.lineTo(700, 800 - (i * (40+y)));
    cxt.lineTo(800, 400+y);
    }

}

cxt.stroke();

script>

Cool! I'd like to know more..

Yes, I agree. This opens up a new world for web-based graphics.. and I just scratched the surface of it! In particular I'd like to see how this type of graphical components can be used to compose visualizations in the digital humanities - provided more interactivity is added to them. Here are a couple if learning resources I found useful:

Learning the basics of HTML5 canvas on .Net magazine online

Canvas tutorial on Mozilla.org (more advanced)

Javascript Graphics and Effects Frameworks: useful document providing a list of the most common libraries for doing javascript-based graphics

Canvas and interactivity tutorial: a step by step discussion on how to create a 'breakout' game clone that you can play in your browser, using javascript and the canvas element

Creating an HTML 5 canvas painting application: nice tutorial that shows how to put together several canvas drawing techniques so to build a basic 'painting' application

21 Ridiculously Impressive HTML5 Canvas Experiments: a collection of some state-of-the-art HTML5 canvas-based experiments that will make you say, “Wow!”

Inspecting an ontology with RDFLib

mike — Mon, 18 Jul 2011 14:59:05 +0000

RDFLib (homepage) is a pretty solid and comprehensive rdf-programming kit for Python. In a previous post I already discussed what pythonic options are currently available out there for doing semantic web programming; after some more in depth testing I realized that Rdflib is the most accessible and complete of them all (in fact many of the available libraries are based on Rdflib's APIs). So.. here we go: in this post I'm giving an overview of some of the things you can do with this library. The Linked Data world is primarily made up of RDF, many would say, so the most important thing is being able to parse and extract information from this simple but versatile language. A quite well known mantra in this community is the 'a little semantics goes a long way', which expresses succinctly the idea that there's no need to fixate on the construction of large-scale CYC-like knowledge-based systems in order to get something going in an open-world scenario such as the web (of data). In other words, this idea suggests that (for now) it's enough to make your application spit out structured data using a standard data model (RDF, that is), and possibly connect your RDF dataset to other datasets in the 'cloud' by creating rdf-links. Once you've done that, you can take it easy and stop worrying about the data integration problems your RDF might generate, or the 'big picture'. Others will figure out how to use your data; it's an incremental approach, there will be some sort of snowball effect at some stage, semantic web enthusiasts seem to suggest. This and other arguments are a bit make-believe, I have to say; but at the same time they also do make some sense: unless we have some real stuff to play with out there on the data-web, not much will ever happen!

Hullo, RDFlib

After quickly ascertaining that it's not a total waste of time to work with RDF, it's now time to get practical and experiment a bit with RDFlib. This is a great python library for it lets you process RDF data very very easily. Example:

# open a graph

>>> import rdflib

>>> graph = rdflib.Graph()



# load some data

>>> graph.parse('http://dbpedia.org/resource/Semantic_Web')

<Graph identifier=pDZkOGzl0 (<class 'rdflib.graph.Graph'>)>

>>> len(graph)

98



# query the data

>>> list(graph)[:10]

[(rdflib.term.URIRef('http://dbpedia.org/resource/SUPER'), rdflib.term.URIRef('http://dbpedia.org/property/keywords'), rdflib.term.URIRef('http://dbpedia.org/resource/Semantic_Web')), (rdflib.term.URIRef('http://dbpedia.org/resource/Semantic_internet'), rdflib.term.URIRef('http://dbpedia.org/ontology/wikiPageRedirects'), rdflib.term.URIRef('http://dbpedia.org/resource/Semantic_Web')), (rdflib.term.URIRef('http://dbpedia.org/resource/SW'), rdflib.term.URIRef('http://dbpedia.org/ontology/wikiPageDisambiguates'), rdflib.term.URIRef('http://dbpedia.org/resource/Semantic_Web')), (rdflib.term.URIRef('http://dbpedia.org/resource/Semantic_integrity'), rdflib.term.URIRef('http://dbpedia.org/ontology/wikiPageRedirects'), rdflib.term.URIRef('http://dbpedia.org/resource/Semantic_Web')), (rdflib.term.URIRef('http://dbpedia.org/resource/Ontotext'), rdflib.term.URIRef('http://dbpedia.org/ontology/industry'), rdflib.term.URIRef('http://dbpedia.org/resource/Semantic_Web')), (rdflib.term.URIRef('http://mpii.de/yago/resource/Semantic_Web'), rdflib.term.URIRef('http://www.w3.org/2002/07/owl#sameAs'), rdflib.term.URIRef('http://dbpedia.org/resource/Semantic_Web')), (rdflib.term.URIRef('http://dbpedia.org/resource/Deborah_McGuinness'), rdflib.term.URIRef('http://dbpedia.org/ontology/knownFor'), rdflib.term.URIRef('http://dbpedia.org/resource/Semantic_Web')), (rdflib.term.URIRef('http://dbpedia.org/resource/The_semantic_web'), rdflib.term.URIRef('http://dbpedia.org/ontology/wikiPageRedirects'), rdflib.term.URIRef('http://dbpedia.org/resource/Semantic_Web')), (rdflib.term.URIRef('http://dbpedia.org/resource/Access-eGov'), rdflib.term.URIRef('http://dbpedia.org/property/keywords'), rdflib.term.URIRef('http://dbpedia.org/resource/Semantic_Web')), (rdflib.term.URIRef('http://dbpedia.org/resource/SOA4All'), rdflib.term.URIRef('http://dbpedia.org/property/keywords'), rdflib.term.URIRef('http://dbpedia.org/resource/Semantic_Web'))]



>>> for s, p, o in graph:

...     print s, "\n--- ", p, "\n------ ", o

... 

http://dbpedia.org/resource/SUPER 

---  http://dbpedia.org/property/keywords 

------  http://dbpedia.org/resource/Semantic_Web

http://dbpedia.org/resource/Semantic_internet 

---  http://dbpedia.org/ontology/wikiPageRedirects 

------  http://dbpedia.org/resource/Semantic_Web

http://dbpedia.org/resource/SW 

---  http://dbpedia.org/ontology/wikiPageDisambiguates 

------  http://dbpedia.org/resource/Semantic_Web

http://dbpedia.org/resource/Semantic_integrity 

---  http://dbpedia.org/ontology/wikiPageRedirects 

------  http://dbpedia.org/resource/Semantic_Web

http://dbpedia.org/resource/Ontotext 

---  http://dbpedia.org/ontology/industry 

------  http://dbpedia.org/resource/Semantic_Web

http://mpii.de/yago/resource/Semantic_Web 

---  http://www.w3.org/2002/07/owl#sameAs 

------  http://dbpedia.org/resource/Semantic_Web

http://dbpedia.org/resource/Deborah_McGuinness 

---  http://dbpedia.org/ontology/knownFor 

------  http://dbpedia.org/resource/Semantic_Web

http://dbpedia.org/resource/The_semantic_web 

---  http://dbpedia.org/ontology/wikiPageRedirects 

------  http://dbpedia.org/resource/Semantic_Web

http://dbpedia.org/resource/Access-eGov 

---  http://dbpedia.org/property/keywords 

------  http://dbpedia.org/resource/Semantic_Web

http://dbpedia.org/resource/SOA4All 

---  http://dbpedia.org/property/keywords 

------  http://dbpedia.org/resource/Semantic_Web

http://dbpedia.org/resource/DBpedia 

---  http://dbpedia.org/ontology/genre 

------  http://dbpedia.org/resource/Semantic_Web

http://dbpedia.org/resource/Tabulator 

---  http://dbpedia.org/ontology/genre 

------  http://dbpedia.org/resource/Semantic_Web

http://dbpedia.org/resource/Semantic_web 

---  http://dbpedia.org/ontology/wikiPageRedirects 

------  http://dbpedia.org/resource/Semantic_Web

http://en.wikipedia.org/wiki/Semantic_Web 

---  http://xmlns.com/foaf/0.1/primaryTopic 

------  http://dbpedia.org/resource/Semantic_Web

http://dbpedia.org/resource/Sesame_%28framework%29 

---  http://dbpedia.org/ontology/genre 

------  http://dbpedia.org/resource/Semantic_Web

http://dbpedia.org/resource/Mulgara_%28software%29 

---  http://dbpedia.org/ontology/genre 

------  http://dbpedia.org/resource/Semantic_Web

http://dbpedia.org/resource/James_Hendler 

---  http://dbpedia.org/ontology/knownFor 

------  http://dbpedia.org/resource/Semantic_Web

http://dbpedia.org/resource/Jena_%28framework%29 

---  http://dbpedia.org/ontology/genre 

------  http://dbpedia.org/resource/Semantic_Web

http://dbpedia.org/resource/JRDF_%28framework%29 

---  http://dbpedia.org/ontology/genre 

------  http://dbpedia.org/resource/Semantic_Web

http://dbpedia.org/resource/Nigel_Shadbolt 

---  http://dbpedia.org/ontology/knownFor 

------  http://dbpedia.org/resource/Semantic_Web

http://dbpedia.org/resource/Semantic_Web 

---  http://dbpedia.org/property/refimprove 

------  November 2009

http://dbpedia.org/resource/Semantic_Web 

---  http://dbpedia.org/property/wikiPageUsesTemplate 

------  http://dbpedia.org/resource/Template:Multiple_issues

http://dbpedia.org/resource/Semantic_Web 

---  http://www.w3.org/2000/01/rdf-schema#label 

------  Web sémantique

http://dbpedia.org/resource/Semantic_Web 

---  http://www.w3.org/2000/01/rdf-schema#label 

------  &#12475;マンティック・ウェブ

http://dbpedia.org/resource/Semantic_Web 

---  http://www.w3.org/2000/01/rdf-schema#label 

------  Semantic Web

http://dbpedia.org/resource/Semantic_Web 

---  http://www.w3.org/2000/01/rdf-schema#label 

------  Web semantico

http://dbpedia.org/resource/Semantic_Web 

---  http://www.w3.org/2000/01/rdf-schema#label 

------  Web semântica

http://dbpedia.org/resource/Semantic_Web 

---  http://www.w3.org/2000/01/rdf-schema#label 

------  Semantiska webben

http://dbpedia.org/resource/Semantic_Web 

---  http://www.w3.org/2000/01/rdf-schema#label 

------  Semantisches Web

http://dbpedia.org/resource/Semantic_Web 

---  http://www.w3.org/2000/01/rdf-schema#label 

------  Web semántica

http://dbpedia.org/resource/Semantic_Web 

---  http://www.w3.org/2000/01/rdf-schema#label 

------  &#35821;义网

http://dbpedia.org/resource/Semantic_Web 

---  http://www.w3.org/2000/01/rdf-schema#label 

------  Semanttinen Web

http://dbpedia.org/resource/Semantic_Web 

---  http://www.w3.org/2000/01/rdf-schema#label 

------  &#1057;емантическая паутина

http://dbpedia.org/resource/Semantic_Web 

---  http://dbpedia.org/property/tone 

------  November 2009

http://dbpedia.org/resource/Semantic_Web 

---  http://dbpedia.org/property/essayLike 

------  November 2009

http://dbpedia.org/resource/Semantic_Web 

---  http://dbpedia.org/property/date 

------  November 2009

http://dbpedia.org/resource/Semantic_Web 

---  http://dbpedia.org/ontology/abstract 

------  Con il termine web semantico, termine coniato dal suo ideatore, Tim Berners-Lee, si intende la trasformazione del World Wide Web in un ambiente dove i documenti pubblicati (pagine HTML, file, immagini, e così via) siano associati ad informazioni e dati che ne specifichino il contesto semantico in un formato adatto all'interrogazione, all'interpretazione e, più in generale, all'elaborazione automatica. Con l'interpretazione del contenuto dei documenti che il Web Semantico propugna, saranno possibili ricerche molto più evolute delle attuali, basate sulla presenza nel documento di parole chiave, ed altre operazioni specialistiche come la costruzione di reti di relazioni e connessioni tra documenti secondo logiche più elaborate del semplice link ipertestuale.

http://dbpedia.org/resource/Semantic_Web 

--- etc. etc etc................

Pretty straightforward uh? In a nutshell, what we've just done is: a) loading the RDF description of the 'Semantic Web' page on DBPedia (http://dbpedia.org/resource/Semantic_Web); b) showing the first 10 triples in that RDF graph; c) iterating through all the triples in the graph and printing them out in a format that reflects the subject-predicate-object structure of RDF. However we still don't know much about those data. Meaning: what is the abstract structure used to define them? Do they conform to some sound and thorough data-model or is it just some automatically-generated messy agglomerate of database records? In other words, what I want to know is, what's the ontology behind these data? How can I see it? Shall I reuse it (and thus endorse it) within my own application, or does my application require something slightly different? I'm probably biased here, cause I personally get much more satisfaction from creating and thinking about ontologies rather than fiddling with large quantities of rdf-xml triples. Still, I think that being able to evaluate the ontology a bunch of rdf refers to is of vital importance, in order to judge whether that RDF is what you're looking for or not, and how to best integrate it in your application. Long story short, I couldn't find anything in RdfLib that would let me print out the hierarchy tree of an ontology and other related information. So I thought, here's a good candidate-task for me to learn how to use the library better.

Inspecting an ontology using RDFLib

I created a small class called 'OntoInspector' that you can instantiate with an RDFS/OWL ontology and then query to find out basic information about that ontology. I know - all of this could have been done using one of the many (and constantly increasing) ontology editing tools - but hey this is all about learning isn't it? You can find all the source code on BitBucket. Feel free to get it and modify as needed. Also, I integrated this python toolkit within a django application that let you browse ontologies online (beware - it's just a hack really). This is called - surprise - OntoView, and it's accessible here. The first thing to do in our class definition is (obviously) loading up the RDFLib library. I've developed this using RDFlib 2.4, but recently tested it with 3.0 (the latest release available) and it all still works fine. By loading up the RDF and RDFS modules we'll have access to all the constants needed to query for classes and subclasses. Note that I added an OWL module as that is not part of RDFLib. You can find it in the source code, it's just a list of all predicates in the OWL vocabulary.

from rdflib import ConjunctiveGraph, Namespace, exceptions

from rdflib import URIRef, RDFS, RDF, BNode

import OWL

Now let's set up the basic structure of the OntoInspector class. In principle, an OntoInspector object should contain all the information necessary to query an ontology. An ontology is referred to using its URI, so that's all is needed for creating an instance of OntoInspector too:

class OntoInspector(object):

    """Class that includes methods for querying an RDFS/OWL ontology"""

        

    def __init__(self, uri, language=""):



        super(OntoInspector, self).__init__()



        self.rdfGraph = ConjunctiveGraph()

        try:

            self.rdfGraph.parse(uri, format="xml")

        except:

            try:

                self.rdfGraph.parse(uri, format="n3")

            except:

                raise exceptions.Error("Could not parse the file! Is it a valid RDF/OWL ontology?")

        finally:

            # let's cache some useful info for faster access

            self.baseURI = self.get_OntologyURI() or uri            

            self.allclasses = self.__getAllClasses(classPredicate)

            self.toplayer = self.__getTopclasses()

            self.tree = self.__getTree()





    def get_OntologyURI(self, ....):

        # todo

        pass



    def __getAllClasses(self, ....):

        # todo

        pass

        

    def __getTopclasses(self, ....):

        pass

        

    def __getTree(self, ....):

        # todo

        pass

As you can see the __init__ method tries to load the ontology file (which can be expressed in either rdf/xml or n3 format) and then sets up 4 class attributes. These attributes will contain some key information about the ontology: its URI, a list of all the classes available, the classes in the top layer and the main taxonomical tree of the ontology. We're now going to implement the methods needed to fill out these 4 attributes.

Getting the ontology URI

If we're dealing with an OWL ontology, it may be the case that the URI we have just used to retrieve the ontology file is not the 'official' URI of the ontology. In fact OWL provides a construct that can be used to 'state' which is the base URI of an ontology (essentially, this is equivalent to stating that an RDF resource has rdf:type http://www.w3.org/2002/07/owl#Ontology). So in the following method first we check if an URI of rdf:type OWL:Ontology exists, and return that if available (when we return None, the URI value defaults to the URI originally provided when creating the OntoInspector object - see the constructor code above):



def get_OntologyURI(self, return_as_string=True):

    """ 

    In [15]: [x for x in o.rdfGraph.triples((None, RDF.type, OWL.Ontology))]

    Out[15]: 

    [(rdflib.URIRef('http://purl.com/net/sails'),

      rdflib.URIRef('http://www.w3.org/1999/02/22-rdf-syntax-ns#type'),

      rdflib.URIRef('http://www.w3.org/2002/07/owl#Ontology'))]



    Mind that this will work only for OWL ontologies.

    In other cases we just return None, and use the URI passed at loading time



    """

    test = [x for x, y, z in self.rdfGraph.triples((None, RDF.type, OWL.Ontology))]

    if test:

        if return_as_string:

            return str(test[0])

        else:

            return test[0]

    else:

        return None

Extracting all the classes

Essentially, there are only two ways to define a class: you can either specify that an entity has a property RDF:type with value rdfs:Class, or that it has a property RDF:type with value owl:Class. Note that the owl:Class predicate is defined as a subclass of rdfs:Class. The rationale for having a separate OWL class construct lies in the restrictions on OWL DL (and thus also on OWL Lite), which imply that not all RDFS classes are legal OWL DL classes. In OWL Full these restrictions do not exist and therefore owl:Class and rdfs:Class are equivalent in OWL Full (more info here: http://www.w3.org/TR/owl-ref/, section 3.1). Thus, In order to retrieve all the classes defined in an ontology we can just query the RDF graph for triples that have this form: someURI - rdf:type - rdf:Class OR owl:Class . This approach will work in the majority of cases. However, things are complicated by the fact that people are sometimes sloppy when they define ontologies, or because they use different tools that automatically generate different styles of RDF code. For example, often an entity is defined as being an rdfs:subclassOf another entity, without explicitly declaring that both of them are (rdfs, or owl) classes; another common case is that one of classes mentioned in the domain/range values of properties (via the rdfs.domain and rdfs.range properties) but not declared explicitly. Since we want to be as comprehensive as possible when looking for *all* the classes present in an ontology, I added a couple of methods that deal with these borderline cases. If you don't want to include all of this stuff, you can still bypass these extra checks by using the classPredicate argument.

def __getAllClasses(self, classPredicate = "", removeBlankNodes = True):

    """  

    Extracts all the classes from a model

    We use the RDFS and OWL predicate by default; also, we extract non explicitly declared classes

    """

    rdfGraph = self.rdfGraph

    exit = []       

    if not classPredicate:          

        for s, v, o in rdfGraph.triples((None, RDF.type , OWL.Class)): 

            exit.append(s)

        for s, v, o in rdfGraph.triples((None, RDF.type , RDFS.Class)):

            exit.append(s)

        # this extra routine makes sure we include classes not declared explicitly

        # eg when importing another onto and subclassing one of its classes...

        for s, v, o in rdfGraph.triples((None, RDFS.subClassOf , None)):

            if s not in exit:

                exit.append(s)

            if o not in exit:

                exit.append(o)

        # this extra routine includes classes found only in rdfs:domain and rdfs:range definitions

        for s, v, o in rdfGraph.triples((None, RDFS.domain , None)):

            if o not in exit:

                exit.append(o)

        for s, v, o in rdfGraph.triples((None, RDFS.range , None)):

            if o not in exit:

                exit.append(o)

    else:

        if classPredicate == "rdfs" or classPredicate == "rdf":

            for s, v, o in rdfGraph.triples((None, RDF.type , RDFS.Class)):

                exit.append(s)

        elif classPredicate == "owl":

            for s, v, o in rdfGraph.triples((None, RDF.type , OWL.Class)): 

                exit.append(s)

        else:

            raise exceptions.Error("ClassPredicate must be either rdf, rdfs or owl")



    exit = remove_duplicates(exit)

    if removeBlankNodes:

        exit = [x for x in exit if not self.__isBlankNode(x)]

    return sort_uri_list_by_name(exit)

You probably noticed that there are a couple of other methods mentioned in the snippet above: they are used for checking if a URI is a BlankNode (which we're normally not interested in, when dealing with ontologies) and for other utility functions, such as sorting and removing duplicates from our list of classes. You'll find all the details about this stuff in the source code obviously.. Next, we want to be able to move around the ontology hierarchy. So we need methods to get super and sub classes from a given class. This is easily done by querying the graph for triples containing the rdfs.subClassOf predicate:

# methods for getting ancestores and descendants of classes: by default, we do not include blank nodes



def get_classDirectSupers(self, aClass, excludeBnodes = True):

    returnlist = []

    for s, v, o in self.rdfGraph.triples((aClass, RDFS.subClassOf , None)):

        if excludeBnodes:

            if not self.__isBlankNode(o):

                returnlist.append(o)

        else:

            returnlist.append(o)

    return sort_uri_list_by_name(remove_duplicates(returnlist)) 





def get_classDirectSubs(self, aClass, excludeBnodes = True):

    returnlist = []

    for s, v, o in self.rdfGraph.triples((None, RDFS.subClassOf , aClass)):

        if excludeBnodes:

            if not self.__isBlankNode(s):

                returnlist.append(s)

        else:

            returnlist.append(s)

    return sort_uri_list_by_name(remove_duplicates(returnlist))





def get_classAllSubs(self, aClass, returnlist = [], excludeBnodes = True):

    for sub in self.get_classDirectSubs(aClass, excludeBnodes):

        returnlist.append(sub)

        self.get_classAllSubs(sub, returnlist, excludeBnodes)

    return sort_uri_list_by_name(remove_duplicates(returnlist))





def get_classAllSupers(self, aClass, returnlist = [], excludeBnodes = True ):

    for ssuper in self.get_classDirectSupers(aClass, excludeBnodes):

        returnlist.append(ssuper)

        self.get_classAllSupers(ssuper, returnlist, excludeBnodes)

    return sort_uri_list_by_name(remove_duplicates(returnlist))





def get_classSiblings(self, aClass, excludeBnodes = True):

    returnlist = []

    for father in self.get_classDirectSupers(aClass, excludeBnodes):

        for child in self.get_classDirectSubs(father, excludeBnodes):

            if child != aClass:

                returnlist.append(child)

    return sort_uri_list_by_name(remove_duplicates(returnlist))

Getting the top layer

We're now all set for retrieving the classes at the top of the taxonomic hierarchy of our ontology, that is, its 'top-layer'. This can be done by reusing the get_classDirectSupers method previously defined, so to search for all classes that have no superclasses:

def __getTopclasses(self, classPredicate = ''):

    """ Finds the topclass in an ontology (works also when we have more than on superclass)



    """

    returnlist = []

    # gets all the classes

    for eachclass in self.__getAllClasses(classPredicate):

        x = self.get_classDirectSupers(eachclass)

        if not x:

            returnlist.append(eachclass)

    return sort_uri_list_by_name(returnlist)

Reconstructing the ontology tree

Now that we know which are the top classes in our taxonomy, we can parse the tree recursively using the get_classDirectSubs method defined above, and reconstruct the whole taxonomical structure of the ontology.

def __getTree(self, father=None, out=None):

    """ Reconstructs the taxonomical tree of an ontology, from the 'topClasses' (= classes with no supers, see below)

        Returns a dictionary in which each class is a key, and its direct subs are the values.

        The top classes have key = 0



        Eg.

        {'0' : [class1, class2], class1: [class1-2, class1-3], class2: [class2-1, class2-2]}

    """

    if not father:

        out = {}

        topclasses = self.toplayer

        out[0] = topclasses

        for top in topclasses:

            children = self.get_classDirectSubs(top)

            out[top] = children

            for potentialfather in children:

                self.__getTree(potentialfather, out)

        return out

    else:

        children = self.get_classDirectSubs(father)

        out[father] = children

        for ch in children:

            self.__getTree(ch, out)

That's it really. Given this abstract tree representation, it can be printed out differently depending on the context (html, command line) but the core will remain intact.

Wrapping up

The source code on BitBucket contains also other utilities I added, eg for handling class comments, namespaces, for nice-printing of classes' names, and for outputting the ontology tree as an image, using the Graphviz library (which needs to be installed separately). Here's an example of how OntoInspector can be used in the python interactive shell for inspecting the Friend Of A Friend (FOAF) ontology:

In [1]: from onto_inspector import *

In [2]: onto = OntoInspector("http://xmlns.com/foaf/spec/20100809.rdf")         

In [3]: onto.toplayer

Out[3]: 

[rdflib.URIRef('http://xmlns.com/foaf/0.1/Agent'),

 rdflib.URIRef('http://www.w3.org/2000/01/rdf-schema#Class'),

 rdflib.URIRef('http://www.w3.org/2004/02/skos/core#Concept'),

 rdflib.URIRef('http://xmlns.com/foaf/0.1/Document'),

 rdflib.URIRef('http://xmlns.com/foaf/0.1/LabelProperty'),

 rdflib.URIRef('http://www.w3.org/2000/01/rdf-schema#Literal'),

 rdflib.URIRef('http://www.w3.org/2000/10/swap/pim/contact#Person'),

 rdflib.URIRef('http://xmlns.com/foaf/0.1/Project'),

 rdflib.URIRef('http://www.w3.org/2003/01/geo/wgs84_pos#SpatialThing'),

 rdflib.URIRef('http://www.w3.org/2002/07/owl#Thing')]

In [4]: onto.printTree()

foaf:Agent

----foaf:Group

----foaf:Organization

----foaf:Person

rdfs:Class

http://www.w3.org/2004/02/skos/core#Concept

foaf:Document

----foaf:Image

----foaf:PersonalProfileDocument

foaf:LabelProperty

rdfs:Literal

http://www.w3.org/2000/10/swap/pim/contact#Person

----foaf:Person

foaf:Project

http://www.w3.org/2003/01/geo/wgs84_pos#SpatialThing

----foaf:Person

owl:Thing

----foaf:OnlineAccount

--------foaf:OnlineChatAccount

--------foaf:OnlineEcommerceAccount

--------foaf:OnlineGamingAccount



In [5]: document = onto.find_class_byname("document")

In [6]: document

Out[6]: 

[rdflib.URIRef('http://xmlns.com/foaf/0.1/Document'),

 rdflib.URIRef('http://xmlns.com/foaf/0.1/PersonalProfileDocument')]



In [7]: document = document[0]

In [8]: document

Out[8]: rdflib.URIRef('http://xmlns.com/foaf/0.1/Document')

In [9]: onto.get_classAllSubs(document)

Out[9]: 

[rdflib.URIRef('http://xmlns.com/foaf/0.1/Image'),

 rdflib.URIRef('http://xmlns.com/foaf/0.1/PersonalProfileDocument')]

In [10]: onto.get_classAllSupers(document)

Out[10]: []

In [11]: onto.get_classComment(document)

Out[11]: rdflib.Literal('A document.', language=None, datatype=None)

Any comments? As I said I'm still learning/improving this... so any feedback is welcome! You can also get in touch via the BitBucket page.

Workshop on Live Coding @ RMLL-11

mike — Mon, 11 Jul 2011 15:13:10 +0000

I just got back from Strasbourg (France) where I gave a talk about my experience with Livecoding and Impromptu at the at the Cultures et Arts Libres Workshop, part of the 2011 Libre Software Meeting. In a nutshell, livecoding is the process of writing software in realtime, as a form of improvised time-based art. Many thanks for the organizers for inviting me, it's been a quite rewarding experience. Here I'm posting the slides from the talk in case people want to follow up on the things I mentioned. The slides are very introductory, so I strongly encourage anyone interested to find out more about the world of Impromptu by following the links provided in the presentation.

Livecoding with impromptu

The other two livecoders who gave talks at the workshop were Marje Baalman and Dan Stowell; both of them do very interesting stuff with SuperCollider (another livecoding environment) so you better check them out too! By the way, after the workshop there was also a livecoding performance - but I'm going to report about that here..