The power of django’s Q objects

I’ve been spending some time today doing research on how to best use django’s Q objects. Then I did a bit of testing myself and decided to put this whole thing together for future’s sake, you know, for the time when I’ll find myself lost and in need some coding recipe.

First things first: the official documentation.

  • Django doc: Complex lookups with Q objects
    The main intro
  • Django doc: OR lookups
    More in-depth examples
  • Now… let’s get started by resurrecting the simple model used in the django Tutorial, Polls, and adding a few instances to play with:

    # ==> in models.py
    from django.db import models
    import datetime
    
    class Poll(models.Model):
            question = models.CharField(max_length=200)
            pub_date = models.DateTimeField('date published')
    
    
    # ==> add some instances from the shell
    >>> p = Poll(question='what shall I make for dinner', pub_date=datetime.date.today())
    >>> p.save()
    >>> p = Poll(question='what is your favourite meal?', pub_date=datetime.datetime(2009, 7, 19, 0, 0)) #one year ago
    >>> p.save()
    >>> p = Poll(question='how do you make omelettes?', pub_date=datetime.date.today())
    >>> p.save()
    
    

    What are Q Objects Handy for?

    In a nutshell, the main reason why you might want to use them is because you need to do complex queries, for example involving OR and AND statements a the same time.

    Imagine that you want to retrieve all polls whose question contains the word ‘dinner’ OR ‘meal’. The usual ‘filter’ option lets you do the following:

    >>> Poll.objects.filter(question__contains='dinner').filter(question__contains='meal')
    []
    
    

    But this is not really what we want, is it? We just got a concatenation of AND statements (= all constraints need to be satisfied), while we wanted an OR (=at least one of the constraints needs to be satisfied).
    This is where Q objects become useful:

    >>> from django.db.models import Q
    # a simple query
    >>> Poll.objects.filter(Q(question__contains='dinner'))
    [<Poll: what shall I make for dinner>]
    # two constraints linked by AND
    >>> Poll.objects.filter(Q(question__contains='dinner') & Q(question__contains='meal'))
    []
    # two constraints linked by OR - that's it!
    >>> Poll.objects.filter(Q(question__contains='dinner') | Q(question__contains='meal'))
    [<Poll: what shall I make for dinner>, <Poll: what is your favourite meal?>]

    Note that if you don’t specify any logical connector the Q sequence is implicitly interpreted as AND:

    # no logical connector is interpreted as AND
    >>> Poll.objects.filter(Q(question__contains='dinner'), Q(question__contains='meal'))
    []
    

    Things start getting more interesting when creating complex queries:

    # eg (A OR B) AND C:
    >>> Poll.objects.filter((Q(question__contains='dinner') | Q(question__contains='meal')) & Q(pub_date=datetime.date.today()))
    [<Poll: what shall I make for dinner>]
    

    Dynamic Query Building

    Now, it is likely that you want to build up queries like the ones above dynamically. This is another place where Q objects can save lots of time….for example, when constructing search engines or faceted browsers where the interface lets a user accumulate search filters in an incremental manner.

    One way to handle this situation is by creating lists of Q objects, and then combining them together using python’s operator and reduce methods:

    >>> import operator
    # create a list of Q objects
    >>> mylist = [Q(question__contains='dinner'), Q(question__contains='meal')]
    # OR
    >>> Poll.objects.filter(reduce(operator.or_, mylist))
    [<Poll: what shall I make for dinner>, <Poll: what is your favourite meal?>]
    # AND
    >>> Poll.objects.filter(reduce(operator.and_, mylist))
    []
    

    Now, if you’re building the query dynamically you probably won’t know in advance which are the filters you have to use. It is likely instead that you find yourself having to generate a list of Q objects programatically from a list of strings representing queries to your models:

    # string representation of our queries
    >>> predicates = [('question__contains', 'dinner'), ('question__contains', 'meal')]
    # create the list of Q objects and run the queries as above..
    >>> q_list = [Q(x) for x in predicates]
    >>> Poll.objects.filter(reduce(operator.or_, q_list))
    [<Poll: what shall I make for dinner>, <Poll: what is your favourite meal?>]
    >>> Poll.objects.filter(reduce(operator.and_, q_list))
    []
    
    # now let's append another filter to the query-strings..
    >>> predicates.append(('pub_date', datetime.date.today()))
    >>> predicates
    [('question__contains', 'dinner'), ('question__contains', 'meal'), ('pub_date', datetime.date(2010, 7, 19))]
    # .. and the results will change too
    >>> q_list = [Q(x) for x in predicates]
    >>> Poll.objects.filter(reduce(operator.or_, q_list))
    [<Poll: what shall I make for dinner>, <Poll: what is your favourite meal?>, <Poll: how do you make omelettes?>]
    >>> Poll.objects.filter(reduce(operator.and_, q_list))
    []
    

    Query Expansion and Q Objects

    Using query expansion syntax you could normally do things like this:

    >>> mydict = {'question__contains': 'omelette', 'pub_date' : datetime.date.today()}
    >>> Poll.objects.filter(**mydict)
    [<Poll: how do you make omelettes?>]
    

    Here you are basically sequencing AND statements which are built dynamically from some string values.

    A really cool feature of django ORM is that you can keep doing this even while using Q objects. In other words, you can delegate all the ‘normal’ stuff to the query expansion mechanism, while instead resorting to Q objects whenever you need more complex queries, such an OR statement.

    For example (remember to put the dictionary always in second position):

    # OR plus query expansion
    >>> Poll.objects.filter(reduce(operator.or_, q_list), **mydict)
    [<Poll: how do you make omelettes?>]
    # AND plus query expansion
    >>> Poll.objects.filter(reduce(operator.and_, q_list), **mydict)
    []
    

    In the first case, we have one result only because, although the OR constraint on q_list is would match more stuff by itself, **mydict is exploded as a series of constraints connected by AND, thus making [Poll: how do you make omelettes?] the only matchable object.
    In the second case instead there are no results at all simply because we are asking for a Poll instance that satisfies simultaneously all the constraints (which doesn’t exist)!

    Other Resources

    That’s all for now! As I said, I found quite a few very good posts about the subject online and I strongly advise you to check them out too in order to have a clearer picture of how Q objects work.

  • Adding Q objects in Django
    Shows another method to chain Q objects together using ‘add’
  • A Tip on Complex Django Queries
    Discusses how queries need to be written in order to produce the expected mySQL code
  • The power of Q
    Really nice article that overviews all the major features of Q objects (and, or, negation, picking)
  • Dynamic Django queries (or why kwargs is your friend)
    Overview of dynamic querying with django
  •  

    Share/Bookmark






    14 Responses to “The power of django’s Q objects”

    Hey thanks for writing this up – I didn’t know you could do that sort of things with Q objects, useful indeed!

    jordan spencer added these pithy words on Mar 30 11 at 2:35 pm

    Please can you try and reformat your code blocks so that they don’t run off the edge of their wrappers… that will make this page printable. Thanks!

    Derek added these pithy words on Sep 26 12 at 7:03 pm

    Hey ! Thanks for writing this.

    Omie added these pithy words on Oct 13 12 at 11:41 am

    [...] 0 0 http://www.michelepasin.org/blog/2010/07/20/the-power-of-djangos-q-objects/ – думаю, эта ссылка должна вам помочь, в частности [...]

    про фильтры added these pithy words on Oct 14 12 at 2:06 pm

    wow great reference, thanks!

    bastian added these pithy words on Oct 18 12 at 11:13 am

    [...] poked around on the internet, and I’ve found this, which seems like a good way to go about concatenating everything on the back-end, but I’m [...]

    Dynamically adding forms and handling the results added these pithy words on Dec 19 12 at 7:15 pm

    Genius! Thank you very much for a concise summary of the Q object with examples. I needed to dynamically build OR queries across a range of models without resorting to raw SQL. This did it nicely.

    Senor Jorge added these pithy words on Mar 09 13 at 2:59 pm

    its awesome!!
    thanks for your reference its very easy to understand the Q objects with simple examples!!!!it was quite very useful…i will keep it for future reference…..

    ajay added these pithy words on Mar 28 13 at 6:31 am

    Thanks so much for this great post. I was looking for a Dynamic was to produce OR-queries the last 5 hours before coming to your side. TnX!

    Tobias added these pithy words on Jun 02 13 at 10:19 pm

    Great post Michele, I love the use of `reduce`. Thanks!

    Matt Stevens added these pithy words on Jul 05 13 at 10:09 am

    Great post. Works like a charm!

    Marcel added these pithy words on Jul 26 13 at 2:51 pm

    Thank u so much bro, wel done, very helpful :D

    Sudeesh added these pithy words on Nov 12 13 at 1:33 pm

    Django OR Lookup Example: https://github.com/django/django/blob/master/tests/or_lookups/tests.py

    Japan Shah added these pithy words on Jun 18 14 at 3:22 pm

    Leave a Reply


    nine − = 0