/var/

Various programming stuff

Adding a delay to Django HTTP responses

Sometimes you’d like to make your Django views more slow by adding a fake delay. This may sound controversial (why would somebody want to make some of his views slower) however it is a real requirement, at least when developing an application.

For example, you may be using a REST API and you want to implement a spinner while your form is loading. However, usually when developing your responses will load so soon that you won’t be able to admire your spinner in all its glory! Also, when you submit a POST form (i.e a form that changes your data), it is advisable to disable your submit button so that when your users double click it the form won’t be submitted two times (it may seem strange to some people but this is a very common error that has bitten me many times; there are many users that think that they need to double click the buttons; thus I always disable my submit buttons after somebody clicks them); in this case you also need to make your response a little slower to make sure that the button is actually disabled!

I will propose two methods for adding this delay to your responses. One that will affect all (or most) your views using a middleware and another that you can add to any CBV you want using a mixin; please see my previous CBV guide for more on Django CBVs and mixins. For the middleware solution we’ll also take a quick look at what is the Django middleware mechanism and how it can be used to add functionality.

Using middleware

The Django middleware is a mechanism for adding your own code to the Django request / response cycle. I’ll try to explain this a bit; Django is waiting for an HTTP Request (i.e GET a url with these headers and these query parameters), it will parse this HTTP Request and prepare an HTTP Response (i.e some headers and a Payload). Your view will be the main actor for retrieving the HTTP response and returning the HTTP request. However, using this middleware mechanism Django allows you to enable other actors (the middleware) that will universally modify the HTTP request before passing it to your view and will also modify the view’s HTTP respone before sending it back to the client.

Actually, a list of middleware called … MIDDLEWARE is defined by default in the settings.py of all new Django projects; these are used to add various capabilities that are universally needed, for example session support, various security enablers, django message support and others. You can easily attach your own middleware to that list to add extra functionality. Notice that the order of the middleware in the MIDDLEWARE list actually matters. Middleware later in the list will be executed after the ones previous in the list; we’ll see some consequences of this later.

Now the time has come to take a quick look at how to implement a middleware, taken from the Django docs:

class SimpleMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response
        # One-time configuration and initialization.

    def __call__(self, request):
        # Code to be executed for each request before
        # the view (and later middleware) are called.

        response = self.get_response(request)

        # Code to be executed for each request/response after
        # the view is called.

        return response

Actually you can implement the middleware as a nested function however I prefer the classy version. The comments should be really enlightening: When your project is started the constructor (__init__) will be called once, for example if you want to read a configuration setting from the database then you should do it in the __init__ to avoid calling the database everytime your middleware is executed (i.e for every request). The __call__ is a special method that gets translated to calling this class instance as a function, i.e if you do something like:

sm = SimpleMiddleware()
sm()

Then sm() will execute the __call__; there are various similar python special methods, for example __len__, __eq__ etc

Now, as you can see the __call__ special method has four parts:

  • Code that is executed before the self.get_response() method is called; here you should modify the request object. Middleware will reach this point in the order they are listed.
  • The actual call to self.get_response()
  • Code that is executed after the self.get_response() method is called; here you should modify the response object. Middleware will reach this point in the reverse order they are listed.
  • Returning the response to be used by the next middleware

Notice that get_response will call the next middleware; while the get_response for the last middleware will actually call the view. Then the view will return a response which could be modified (if needed) by the middlewares in the opposite order of their definition list.

As an example, let’s define two simple middlewares:

class M1:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        print("M1 before response")
        response = self.get_response(request)
        print("M1 after response")
        return response

class M2:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        print("M2 before response")
        response = self.get_response(request)
        print("M2 after response")
        return response

When you define MIDDLEWARE = ['M1', 'M2'] you’ll see the following:

# Got the request
M1 before response
M2 before response
# The view is rendered to the response now
M2 after response
M1 after response
# Return the response

Please notice a middleware may not call self.get_response to continue the chain but return directly a response (for example a 403 Forbiden response).

After this quick introduction to how middleware works, let’s take a look at a skeleton for the time-delay middleware:

import time

class TimeDelayMiddleware(object):

    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        time.sleep(1)
        response = self.get_response(request)
        return response

This is really simple, I’ve just added an extra line to the previous middleware. This line adds a one-second delay to all responses. I’ve added it before self.get_response - because this delay does not depend on anything, I could have added it after self.get_response without changes in the behavior. Also, the order of this middleware in the MIDDLEWARE list doesn’t matter since it doesn’t depend on other middleware (it just needs to run to add the delay).

This middleware may have a little more functionality, for example to configure the delay from the settings or add the delay only for specific urls (by checking the request.path). Here’s how these extra features could be implemented:

import time
from django.conf import settings

class TimeDelayMiddleware(object):

    def __init__(self, get_response):
        self.get_response = get_response
        self.delay = settings.REQUEST_TIME_DELAY


    def __call__(self, request):
        if '/api/' in request.path:
            time.sleep(self.delay)
        response = self.get_response(request)
        return response

The above will add the delay only to requests whose path contains '/api'. Another case is if you want to only add the delay for POST requests by checking that request.method == 'POST'.

Now, to install this middleware, you can configure your MIDDLEWARE like this in your settings.py (let’s say that you have an application named core containing a module named middleware):

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',

    'core.middleware.TimeDelayMiddleware',
]

The other middleware are the default ones in Django. One more thing to consider is that if you have a single settings.py this middleware will be called; one way to override the delay is to check for settings.DEBUG and only call time.sleep when DEBUG == True. However, the proper way to do it is to have different settings for your development and production environments and add the TimeDelayMiddleware only to your development MIDDLEWARE list. Having different settings for each development is a common practice in Django and I totally recommend to use it.

Using CBVs

Another method to add a delay to the execution of a view is to implement a TimeDelayMixin and inherit your Class Based View from it. As we’ve seen in the CBV guide, the dispatch method is the one that is always called when your CBV is rendered, thus your TimeDelayMixin could be implemented like this:

import time

class TimeDelayMixin(object, ):

    def dispatch(self, request, *args, **kwargs):
        time.sleep(1)
        return super().dispatch(request, *args, **kwargs)

This is very simple (and you can use similar techniques as described for the middleware above to configure the delay time or add the delay only when settings.DEBUG == True etc) - to actually use it, just inherit your view from this mixin, f.e:

class DelayedSampleListView(TimeDelayMixin, ListView):
    model = Sample

Now whenever you call your DelayedSampleListView you’ll see it after the configured delay!

What is really interesting is that the dispatch method actually exists (and has the same functionality) also in Django Rest Framework CBVs, thus using the same mixin you can add the delay not only your normal CBVs but also your DRF API views!

Easy immutable objects in Javascript

With the rise of Redux and other similar Javascript frameworks (e.g Hyperapp) that try to be a little more functional (functional as in functional programming), a new problem was introduced to Javascript programmers (at least to those that weren’t familair with functional programming): How to keep their application’s state “immutable”.

Immutable means that the state should be an object that does not change (mutates) - instead of changing it, Redux needs the state object to be created from the beginning.

So when something happens in your application you need to discard the existing state object and create a new one from scratch by modifying and copying the previous state values. This is easy in most toy-apps that are used when introducing the concept, for example if your state is {counter: 0} then you could just define the reducer for the ADD action (i.e when the user clicks the + button) like this:

let reducer = (state={counter: 0}, action) => {
  switch (action.type) {
    case 'ADD': return { 'counter': state.counter+1 }
    default: return state
  }
}

Unfortunately, your application will definitely have a much more complex state than this!

In the following, I’ll do a quick introduction on how to keep your state objects immutable using modern Javascript techniques, I’ll present how complex it is to modify non-trivial immutable objects and finally I’ll give you a quick recipe for modifying your non-trivial immutable objects. If you want to play with the concepts I’ll introduce you can do it at a playground I’ve created on repl.it.

Please keep in mind that this article has been written for ES6 - take a look at my browserify with ES6 article to see how you can also use it in your projects with Browserify.

Also, if you’d like to see a non-toy React + Redux application or you’d like a gentle introduction to the concepts I talked about (state, reducers, actions etc) you can follow along my React Redux tutorial. This is a rather old article (considering how quickly the Javascript framework state change) but the basic concepts introduced there are true today.

Immutable objects

Let’s start our descent into avoiding mutations by supposing that you had something a little more complex than the initial example, for example your state was like this:

let state = {
    'counter': 0,
    'name': 'John',
    'age': 36
}

If you continued the same example then your ADD reducer would need to return something like this

return {
    'counter': state.counter+1,
    'name': state.name,
    'age': state.age
}

This gets difficult and error prone very soon - and what happens if you later need to add another attribute to your state?

The correct way to implement this would be to enumerate all properties of state except ‘counter’, copy them to a new object, and then assign counter+1 to the new object’s counter attribute. You could implement this by hand however, thankfully, there’s the Object.assign method! This method will copy all attributes from a list of objects to an object which will return as a result and is defined like this:

Object.assign(target, ...sources)

The target parameter is the object that will retrieve all attributes from sources (which is a variadic argument - you can have as many sources as you want - even 0; in this case the target will be returned). For a quick example, running:

let o = {'a': 1}
let oo = Object.assign(o, {'b': 2, 'c': 1}, {'c': 3})
console.log(oo, o===oo)

will return { a: 1, b: 2, c: 3 } true i.e the attributes ‘b’ and ‘c’ were copied to o and it was assigned to oo — notice that o and oo are the same object (thus o is modified now). Also, notice that the the attributes of objects to the right have priority over the attributes of the objects to the left ('c': 1 was overriden by 'c': 3).

As you should have guessed by now, you should never pass the state as the target but instead you should create a new object, thus the ADD reducer should return the following:

return Object.assign({}, state, {'counter': state.counter+1)

This means that it will create a new object which will copy all current attributes of state and increase the existing counter attribute.

I’d like to also add here that instead of using the Object.assign method you could use the spread syntax to more or less do the same. The spread syntax on an object takes this object’s attributes and outputs them as key-value dictionary pairs (for them to be used to initialize other objects). Thus, you can use the spread syntax to create an new object that has the same attributes of another object like this:

let newState = {...state}
// which is similar to
newState = Object.assign({}, state)

Of course you usually need to override some attributes, which can be passed directly to the newly created object, for example for the ADD reducer:

return {...state, 'counter': state.counter+1 }

Like Object.assign, you can have as many sources as you want in your spread syntax thus nothing stops you from using ... multiple times to copy the attributes of multiple objects for example you could define ADD like this:

return {...state, ...{'counter': state.counter+1 } }

The order is similar to Object.assign, i.e the attributes that follow will override the previous ones.

One final comment is that both Object.assign and copying objects with the spread syntax will do a “shallow” copy i.e it will copy only the outer object, not the objects its keys refer to. An example of this behavior is that if you run the following:

let a = {'val': 3 }
let x = {a }
let y = {...x}
console.log(x, y)
x['val2'] = 4
y['val2'] = 5
a['val'] = 33
console.log(x, y)

you’ll get:

{ a: { val: 3 } } { a: { val: 3 } }
{ a: { val: 33 }, val2: 4 } { a: { val: 33 }, val2: 5 }

i.e x and y got a different val2 attribute since they not the same object, however both x and y have a reference to the same a thus when it’s val attribute was changed this change appears to both x and y!

What the above means is that if you have a state object containing other objects (or arrays) you will also need to copy these children objects to keep your state immutable. We’ll see examples on this later.

Immutable arrays

One thing we haven’t talked about yet is what happens if there’s an array in the state, for example your state is let state=[] and you have and APPEND reducer that puts something in the end of that array. The naive (and wrong) way to do it is to call push directly to the state - this will mutate your state and is not be allowed!

You need to copy the array elements and the tool for this job is Array.slice. This methods takes two optional arguments (begin and end) that define the range of elements that will be copied; if you call it without arguments then it will copy the whole array. Using slice, your APPEND reducer can be like this:

let newState = state.slice()
newState.push('new element')
return newState

Also, you could use the Array.concat method which will return a new array by copying all the elements of its arguments

return state.concat(['new element'])

This will append new element to a new object that will have the elements of state (it won’t modify the existing state) and is easier if you have this exact requirement. The advantage of slice is that you can use it to add/remove/modify elements from any place in the original array. For example, here’s how you can add an element after the first element of an array:

let x = ['a', 'b', 'c' ]
let y = x.slice(0,1).concat(['second' ], x.slice(1,3))

Now y will be equal to [ 'a', 'second', 'b', 'c' ]. So the above will get the first (0-th) element from the x array and concat it with another element (second) and the remaining elements of x. Remember that x is not modifyied since concat will create a new array.

In a similar fashion to objects, instead of using concat it is much easier to use the spread syntax. The spread syntax for an array will output its elements one after the other for them to be used by other arrays. Thus, continuing from the previous example, [...x] will return a new array with the elements of x (so it is similar to x.slice() or x.concat()), thus to re-generate the previous example you’ll do something like

let y = y=[...x.slice(0,1), 'second', ...x.slice(1,3)]

All three of concat, slice or the spread syntax will do a shallow copy (similar to how Object.assign works) so the same conclusions from the previous section are true here: If you have arrays inside other arrays (or objects) you’ll need to copy the inner arrays recursively.

More complex cases

We’ll now take a look at some more complex cases and see how quickly it gets difficult because of the shallow copying. Let’s suppose that our state is the following:

const state = {
  'user': {
    'first_name': 'John',
    'last_name': 'Doe',
    'address': {
      'city': 'Athens',
      'country': 'Greece',
      'zip': '12345'
    }
  }
}

and we want to assign a group attribute to the state. This can be easily done with assign:

let groups = [{
    'name': 'group1'
}]

state = Object.assign({}, state, {
  'groups': groups
})

or spread:

state = {
  ...state, 'groups': groups
}

Notice that instead of 'groups': groups I could have used the shorthand syntax and written only groups and it would still work (i.e state = {...state, groups} is the same). In all cases, the resulting state will be:

{
  'user': {
    'first_name': 'John',
    'last_name': 'Doe',
    'address': {
      'city': 'Athens',
      'country': 'Greece',
      'zip': '12345'
    }
  },
  'groups': [{
    'name': 'group1'
  }]
}

From now on I’ll only use the spread syntax which is more compact.

Let’s try to change the user’s name. This is not as easy as the first example because we need to:

  • Create a new copy of the user object with the new first name
  • Create a new copy of the state object with the new user object created above

This can be done in two steps like this:

let user ={...state['user'], 'first_name': 'Jack'}
state = {...state, user}

or in one step like this:

state = {...state, 'user':{
  ...state['user'], 'first_name': 'Jack'}
}

The single step assignment is the combination of the two step described above. It is a little more complex but it saves typing and is prefered because it allows the reducer function to have a single expression. Now let’s try to modify the user’s zip code. We’ll do it in three steps first:

let address ={...state['user']['address'], 'zip': '54321'}
user ={...state['user'], address}
state = {...state, user}

And now in one:

state = {...state, 'user': {
  ...state['user'], 'address': {
    ...state['user']['address'], 'zip': 54321
  }
}}

Now, as can be seen in the above examples, modifying (without mutating) a compex state object is not very easy - it needs much thinking and is too error prone! This will be even more apparent when we also get the array modifications into the equation, for example by adding another two groups:

state = {
  ...state, groups: [
    ...state['groups'].slice(),
    {name: 'group2', id: 2},
    {name: 'group3', id: 3}
  ]
}

The above copies the existing state and assigns to it a new groups object by copying the existing groups and appending two more groups to that array! The state now will be:

{
  user: {
    first_name: 'Jack',
    last_name: 'Doe',
    address: { city: 'Athens', country: 'Greece', zip: 54321 }
  },
  groups: [
    { name: 'group1' },
    { name: 'group2', id: 2 },
    { name: 'group3', id: 3 }
  ]
}

As a final examply, how can we add the missing id attribute to the first group? Following the above techniques:

state = {
  ...state, groups: [
    {...state['groups'][0], 'id': 1},
    ...state['groups'].slice(1)
  ]
}

One more time what the above does?

  • Creates a new object and copies all existing properties of state to it
  • Creates a new array which assigns it to the new state’s groups attribute
  • For the first element of that array it copies all attributes of the first element of state[‘groups’] and assings it an id=1 attribute
  • For the remaining elements of that array it copies all elements of state[‘groups] after the first one

Now think what would happen if we had an even more complex state with 3 or 4 nested levels!

Immutability’s little helpers

As you’ve seen from the previous examples, using immutable objects is not as easy as seems from the toy examples. Actually, drilling down into complex immutable objects and returning new ones that have some values changed is a well-known problem in the functional world and has already a solution called “lenses”. This is a funny name but it more or less means that you use a magnifying lens to look at exactly the value you want and modify or retrieve it. The problem with lenses is that although they solve the problem I mention is that if you want to use them you’ll need to dive deep into functional programming and also you’ll need to include an extra library to your project (even if you only want this specific capability).

For completeness, here’s the the docs on lens from Ramda which is a well known Javascript functional library. This needs you to understand what is prop, what is assoc and then how to use the lens with view, set and over. For me, these are way too much things to remember for such a specific thing. Also, notice that the minified version of Ramda is around 45 kb which is not small. Yes, if I wanted to fully use Ramda or a similar library I’d be delighted to use all these techniques and include it as a dependency - however most people prefer to stick with more familiar (and more procedural) concepts.

The helpers I’m going to present here are more or less a poor man’s lens, i.e you will be able to use the basic functionality of a lens but…

  • without the peculiar syntax and
  • without the need to learn more functional concepts than what you’ll want and
  • without the need to include any more external dependencies

Pretty good deal, no?

In any case, a lens has two parts, a get and a set. The get will be used to drill down and retrieve a value from a complex object while the set will be used to drill down and assign a value to a complex object. The set does not modify the object but returns a new one. The get lens is not really needed since you can easily drill down to an object using the good old index syntax but I’ll include it here for completenes.

We’ll start with the get which seems easier. For this, I’ll just create a function that will take an object and a path inside that object as parameter and retrieve the value at that path. The path could be either a string of the form ‘a.0.c.d’ or an array [‘a’, ‘0’, ‘c’, ‘d’] - for numerical indeces we’ll consider an array at that point.

Thus, for the object {'a': [{'b': {'c': {'d': 32} }}]} when the lens getter is called with either 'a.0.b.c' or [‘a’, 0, ‘b’, ‘c’] as the path, it should return {'d': 32}.

To implement the get helper I will use a functional concept, reduce. I’ve already explained this concept in my previous react-redux tutorial so I urge you to read that article for more info. Using reduce we can apply one by one accumulatively the members of the path to the initial object and the result will be the value of that path. Here’s the implementation of pget (from property get):

const objgetter = (accumulator, currentValue) => accumulator[currentValue];
const pget = (obj, path) =>  (
    (typeof path === 'string' || path instanceof String)?path.split('.'):path
).reduce(objgetter, obj)

I have defined an objgetter reducer function that gets an accumulated object and the current value of the path and just returns the currentValue index of that accumulated object. Finally, for the get lens (named pget) I just check to see if the path is a string or an array (if it’s a string I split it on dots) and then I “reduce” the path using the objgetter defined above and starting by the original object as the initial value. To understand how it is working, let’s try calling it for an object:

const s1 = {'a': [{'b': {'c': {'d': 32} }}]}
console.log(pget(s1, ['a', 0, 'b', 'c']))

The above pget will call reduce on the passed array using the defined objgetter above as the reducer function and s1 as the original object. So, the reducer function will be called with the following values each time:

accumulator currentvalue
s1 'a'
s1['a'] 0
s1['a'][0] 'b'
s1['a'][0]['b'] 'c'
s1['a'][0]['b']['c']  

Thus the result will be exactly what we wanted {'d' :32}. An interesting thing is that it’s working fine without the need to differentiate between arrays and objects because of how index access [] works.

Continuing for the set lens (which will be more difficult), I’ll first represent a simple version that works only with objects and an array path but displays the main idea of how this will work: It uses recursion i.e it will call itself to gradually build the new object. Here’s how it is implemented:

const pset0 = (obj, path, val) => {
  let idx = path[0]

  if(path.length==1) {
    return {
      ...obj, [idx]: val
    }
  } else {
    let remaining = path.slice(1)
    return {
      ...obj,
      [idx]: pset0(...[obj[idx]], remaining, val)
    }
  }
}

As already explained, I have assumed that the path is an array of indeces and that the obj is a complex object (no arrays in it please); the function returns a new object with the old object’s value at the path be replaced with val. This function checks to see if the path has only one element, if yes it will assign the value to that attribute of the object it retrieve. If not, it will call itself recursively by skipping the current index and assign the return value to the current index of the curent object. Let’s see how it works for the following call:

const s2 = {a0: 0, a: {b0: 0, b: {c0: 0, c: 3}}}
console.log(pset0(s2, ['a', 'b', 'c'], 4))
# Call Call parameters Return
1 pset0(s2, [‘a’, ‘b’, ‘c’], 4) {…s2, [‘b’]: pset0(s2[‘a’], [‘b’, ‘c’], 4) }
2 pset0(s2[‘a’], [‘b’, ‘c’], 4) {…s2[‘a’], [‘c’]: pset0(s2[‘a’][‘b’], [‘c’], 4) }
3 pset0(s2[‘a’][‘b’], [‘c’], 4) {…s2[‘a’][‘b’], [‘c’]: 4}

Thus, the first time it will be called it will return a new object with the attributes of s2 but overriding its 'b' index with the return of the second call. The second call will return a new object with the attributes of s2['a'] but override it’s 'c' index with the return of the third call. Finally, the 3rd call will return an object with the attributes of s2['a']['b'] and setting the 'c' index to 4. The result will be as expected equal to:

{a0: 0, a: {b0: 0, b: {c0: 0, c: 4 }}}

Now that we’ve understood the logic we can extend the above function with the following extras:

  • support for arrays in the object using numerical indeces
  • support for array (['a', 'b']) or string path ('a.b') parameter
  • support for a direct value to set on the path or a function that will be applied on that value

Here’s the resulting set lens:

const pset = (obj, path, val) => {
  let parts = (typeof path === 'string' || path instanceof String)?path.split('.'):path
  const cset = (obj, cidx, val) => {
    let newval = val
    if (typeof val === "function") {
      newval = val(obj[cidx])
    }
    if(Array.isArray(obj)) {
      return [
        ...obj.slice(0, cidx*1),
        newval,
        ...obj.slice(cidx*1+1)
        ]
    } else {
      return {
        ...obj, [cidx]: newval
      }
    }
  }

  let pidx = parts[0]
  if(parts.length==1) {
    return cset(obj, pidx, val)
  } else {
    let remaining = parts.slice(1)
    return cset(obj, pidx, pset(obj[pidx], remaining, val))
  }
}

It may seem a little complex but I think it’s easy to be understood: The parts in the beginning will just check to see if the path is an array or a string and split the string to its parts. The cset function that follows is a local function that is used to make the copy of the object or array and set the new value. Here’s how it is working: It will first check to see if the val parameter is a function or a not. If it is a function it will apply this function to the object’s index to get the newvalue else it will just use val as the newvalue. After that it checks if the object it got is an array or not. If it is an array it will do the slice trick we saw before to copy the elements of the array except the newval which will put it at the index (notice that the index at that point must be numerical but that’s up to you to assert). If the current obj is not an array then it must be an object thus it uses the spread syntax to copy the object’s attributes and reassign the current index to newval.

The last part of pset is similar to the pset0 it just uses cset to do the new object/array generation instead of doing it in place like pset0 - as already explained, pset is called recursively until only one element remains on the path in which case the newval will be assigned to the current index of the current obj.

Let’s try to use pset for the following rather complex state:

let state2 = {
  'users': {
    'results': [
      {'name': 'Sera', 'groups': ['g1', 'g2', 'g3']},
      {'name': 'John', 'groups': ['g1', 'g2', 'g3']},
      {'name': 'Joe', 'groups': []}
    ],
    'pagination': {
      'total': 100,
      'perpage': 5,
      'number': 0
    }
  },
  'groups': {
    'results': [
    ]
    ,
    'total': 0
  }
}

Let’s call it three times one after the other to change various attributes:

let new_state2 = pset(
    pset(
        pset(
            pset(state2, "users.results.2.groups.0", 'aa'),
        "users.results.0.name", x=>x.toUpperCase()),
    "users.total", x=>x+1),
'users.results.1.name', 'Jack')

And here’s the result:

{
    "users": {
        "results": [{
            "name": "SERA",
            "groups": ["g1", "g2", "g3"]
        }, {
            "name": "Jack",
            "groups": ["g1", "g2", "g3"]
        }, {
            "name": "Joe",
            "groups": ["aa"]
        }],
        "pagination": {
            "total": 101,
            "perpage": 5,
            "number": 0
        }
    },
    "groups": {
        "results": [],
        "total": 0
    }
}

This should be self explanatory.

I’ve published the above immutable little helpers as an npm package: https://www.npmjs.com/package/poor-man-lens (yes I decided to use the poor man lens name instead of the immutable little helpers) - they are too simple and could be easily copied and pasted to your project but I’ve seen even smaller npm packages and I wanted to try to see if it is easy to publish a package to npm (answer: it is very easy - easier than python’s pip). Also there’s a github repository for these utils in case somebody wants to contribute anything or look at the source: https://github.com/spapas/poor-man-lens.

Notice that this package has been written in ES5 (and actually has a polyfil for Object.assign) thus you should probably be able to use it anywhere you want, even directly from the browser by directly including the pml.js file.

Conclusion

Using the above techniques you should be able to easily keep your state objects immutable. For simple cases you can stick to the spread syntax or Object.assign / Array.slice but for more complex cases you may want to consider either copying directly the pset and pget utils I explained above or just using the poor-man-lens npm package.

A comprehensive Django CBV guide

Class Based Views (CBV) is one of my favourite things about Django. During my first Django projects (using Django 1.4 around 6 years ago) I was mainly using functional views — that’s what the tutorial recommended then anyway. However, slowly in my next projects I started reducing the amount of functional views and embracing CBVs, slowly understanding their usage and usefulness. Right now, I more or less only use CBVs for my views; even if sometimes it seems more work to use a CBV instead of a functional one I know that sometime in the future I’d be glad that I did it since I’ll want to re-use some view functionality and CBVs are more or less the only way to have DRY views in Django.

I’ve heard various rants about them, mainly that they are too complex and difficult to understand and use, however I believe that they are not really difficult when you start from the basics. Even if it is a little work to become comfortable with the CBV logic, when they are used properly they will greatly improve your Django experience so it is definitely worth it.

Notice that to properly understand CBVs you must have a good understanding of how Python’s (multiple) inheritance and MRO work. Yes, this is a rather complex and confusing thing but I’ll try to also explain this as good as I can to the first chapter of this article so if you follow along you shouldn’t have any problems.

This guide has four parts:

  • A gentle introduction to how CBVs are working and to the the problems that do solve. For this we’ll implement our own simple Custom Class Based View variant and take a look at python’s inheritance model.
  • A high level overview of the real Django CBVs using CBV inspector as our guide.
  • A number of use cases where CBVs can be used to elegantly solve real world problems
  • Describing the usage of some of the previous use cases to a real Django application

I’ve implemented an accompanying project to this article which you can find at https://github.com/spapas/cbv-tutorial. This project has two separate parts. One that is the implementation of the Custom Class Based View variant to see how it is working and the other is the application that contains the usage of the various CBV use cases.

A gentle introduction to CBVs

In this part of the guide we’ll do a gentle introduction to how CBVs work by implementing our own class based views variant - along with it we’ll introduce and try to understand some concepts of python (multiple) inheritance and how it applies to CBVs.

Before continuing, let’s talk about the concept of the “view” in Django: Django is considered an MVT (Model View Template) framework - the View as conceived by Django is not the same as the MVC-View. A Django View is more or less a way to define the data that the Template (which is cloased to the MVC-View) will display, so the Django View (with the help of the Django Framework) is similar to the MVC-Controller.

In any case, traditionally a view in Django is a normal python function that takes a single parameter, the request object and must return a response object (notice that if the view uses request parameters for example the id of an object to be edited they will also be passed to the function). The responsibility of the view function is to properly parse the request parameters and construct the response object - as can be understood there is a lot of work that need to be done for each view (for example check if the method is GET or POST, if the user has access to that page, retrieve objects from the database, crate a context dict and pass it to the template to be rendered etc).

Now, since functional views are simple python functions it is not easy to override, reuse or extend their behaviour. There are more or less two methods for this: Use function decorators or pass extra parameters when adding the view to your urls. I’d like to point out here that there’s a third method for code-reuse: Extracting functionality to common and re-usable functions or classes that will be called from the functional views but this is not something specific to Django views but a general concept of good programming style which you should follow anyway.

The first one uses python decorators to create a functional view that wraps the initial one. The new view is called before the initial one, adds some functionality (for example check if the current user has access, modify request parameters etc), calls the initial one which will return a response object, modify the response if needed and then return that. This is how login_required works. Notice that by using decorators you can change things before and after the original view runs but you can’t do anything about the way the original view works.

For the second one (adding extra view parameters) you must write your function view in a way which allows it to be reused, for example instead of hard-coding the template name allow it to be passed as a parameter or instead of using a specific form class for a form make it configurable through a parameter. Then, when you add this function to your urls you will pass different parameters depending on how you want to configure your view. Using this method you can override the original function behaviour however there’s a limit to the number of parameters you can allow your function views to have and notice that these function views cannot be further overridden. The login authentication view (which is now deprecated in favour of a CBV one) is using this technique, for example you can pass it the template name that will be used, a custom authentication form etc.

It should be obvious that both these methods have severe limitations and do not allow you to be as DRY as you should be. When using the wrapped views you can’t actually change the functionality of the original view (since that original function needs to be called) but only do things before and after calling it. Also, using the parameters will lead to spaghetti code with multiple if / else conditions in order to take into account the various cases that may arise. All the above lead to very reduced re-usability and DRYness of functional views - usually the best thing you can do is to gather the common things in external normal python functions (not view functions) that could be re-used from other functional views as already discussed.

Class based views solve the above problem of non-DRY-ness by using the well known concept of OO inheritance: The view is defined from a class which has methods for implementing the view functionality - you inherit from that class and override the parts you want so the inherited class based view will use the overridden methods instead of the original ones. You can also create re-usable classes (mixins) that offer a specific functionality to your class based view by implementing some of the methods of the original class. Each one of your class based views can inherit its functionality from multiple mixins thus allowing you to define a single class for each thing you need and re-using it everywhere. Notice of course that this is possible only if the CBVs are properly implemented to allow overriding their functionality. We’ll see how this is possible in the next section.

Hand-made CBVs

To make things more clear we’ll start implementing our own class based views hierarchy. Here’s a rather naive first try:

class CustomClassView:
    context = []
    header = ''

    def __init__(self, **kwargs):
        self.kwargs = kwargs
        for (k,v) in kwargs.items():
            setattr(self, k, v)

    def render(self):
        return """
            <html>
                <body>
                    <h1>{header}</h1>
                    {body}
                </body>
            </html>
        """.format(
                header=self.header, body='<br />'.join(self.context),
            )

    @classmethod
    def as_view(cls, *args, **kwargs):
        def view(request, ):
            instance = cls(**kwargs)
            return HttpResponse(instance.render())

        return view

This class can be used to render a simple HTML template with a custom header and a list of items in the body (named context). There are two things to notice here: The __init__ method (which will be called as the object’s constructor) will assign all the keyword arguments (kwargs) it receives as instance attributes (for example CustomClassView(header='hello') will create an instance with 'hello' as its header attribute). The as_view is a class method (i.e it can be called directly on the class without the need to instantiate an object for example you can call CustomClassView.as_view() ) that defines and returns a traditional functional view (named view) that will be used to actually serve the view. The returned functional view is very simple - it just instantiates a new instance (object) of CustomClassView passing the kwargs it got in the constructor and then returns a normal HttpResponse with the instance’s render() result. This render() method will just output some html using the instance’s header and context to fill it.

Notice that the instance of the CustomClassView inside the as_view class method is not created using CustomClassView(**kwargs) but using cls(**kwargs) - cls is the name of the class that as_view was called on and is actually passed as a parameter for class methods (in a similar manner to how self is passed to instance methods). This is important to instantiate an object instance of the proper class.

For example, if you created a class that inherited from CustomClassView and called its as_view method then when you use the cls parameter to instantiate the object it will correctly create an object of the inherited class and not the base one (if on the other hand you had used CustomClassView(**kwargs) to instantiate the instance then the as_view method of the inheriting classes would instantiate instances of CustomClassView so inheritance wouldn’t really work!).

To add the above class method in your urls, just use its as_view() as you’d normally use a functional view:

from django.conf.urls import include, url
from . import views

urlpatterns = [
    url(r'^ccv-empty/$', views.CustomClassView.as_view(), name='ccv-empty'),
    # ... other urls
]

This doesn’t actually render anything since both header and context are empty on the created instance — remember that as_view returns a functional view that instantiates a CustomClassView objet and returns an HttpResponse filling it with the object’s render() reuslts. To add some output we can either create another class that inherits from CustomClassView or initialize the attributes from the constructor of the class (using the kwargs functionality described above).

The inherited class can just override the values of the attributes:

class InheritsCustomClassView(CustomClassView, ):
    header = "Hi"
    context = ['test', 'test2' ]

And then just add the inherited class to your urls as before:

url(r'^ccv-inherits/$', views.InheritsCustomClassView.as_view(), name='ccv-inherits'),

The as_view() method will create an instance of InheritsCustomClassView that has the values configured in the class as attributes and return its render() output as response.

The other way to configure the attributes of the class is to pass them to the as_view class method (which in turn will pass them to the instances constructor which will set the attributes in the instance). Here’s an example:

url(r'^ccv-with-values/$', views.CustomClassView.as_view(header='Hello', context=['hello', 'world', ], footer='Bye', ), name='ccv-with-values'),

The above will create a CustomClassView instance with the provided values as its attributes.

Although this method of configuration is used in normal django CBVs (for example setting the template_name in a TemplateView) I recommend you avoid using it because passing parameters to the as_view method pollutes the urls.py with configuration that (at least in my opinion) should not be there (and there’s no reason to have to take a look at both your urls.py and your views.py to understand the behavior of your views) and also, even for very simple views I know that after some time I’ll need to add some functionality that cannot be implemented by passing the parameters so I prefer to bite the bullet and define all my views as inherited classes so it will be easy for me to further customize them later (we’ll see how this is done in a second). Thus, even if you have

In any case, I won’t discuss passing parameters to the as_view method any more, so from now on any class based views I define will be added to urls py using ClassName.as_view() without any parameters to the as_view() class method.

Is this really DRY ?

Let’s now suppose that we wanted to allow our class based view to print something on the header even if no header is provided when you configure it. The only way to do it would be to re-define the render method like this:

def render(self):
    header=self.header if self.header else "DEFAULT HEADER"
    return """
        <html>
            <body>
                <h1>{header}</h1>
                {body}
            </body>
        </html>
    """.format(
            header=header, body='<br />'.join(self.context),
        )

This is definitely not the DRY way to do it because you would need to re-define the whole render method. Think what would happen if you wanted to print "ANOTHER DEFAULT HEADER" as a default header for some other view - once again re-defining render! In fact, the above CustomClassView is naively implemented because it does not allow proper customization through inheritance. The same problems for the header arise also when you need modify the body; for example, if you wanted to add an index number before displaying the items of the list then you’d need to again re-implement the whole render method.

If that was our only option then we could just stick to functional views. However, we can do much better if we define the class based view in such a way that allows inherited classes to override methods that define specific parts of the functionality. To do this the class-based-view must be properly implemented so each part of its functionality is implemented by a different method.

Here’s how we could improve the CustomClassView to make it more DRY:

class BetterCustomClassView(CustomClassView, ):
    def get_header(self, ):
        print ("Better Custom Class View")
        return self.header if self.header else ""

    def get_context(self , ):
        return self.context if self.context else []

    def render_context(self):
        context = self.get_context()
        if context:
            return '<br />'.join(context)
        return ""

    def render(self):
        return """
            <html>
                <body>
                    <h1>{header}</h1>
                    {body}
                </body>
            </html>
        """.format(
                header=self.get_header(), body=self.render_context(),
            )

So what happens here? First of all we inherit from ClassClassView to keep the as_view method which doesn’t need changing. Beyond this, the render uses methods (get_header and render_context) to retrieve the values from the header and the body - this means that we could re-define these methods to an inherited class in order to override what these methods will return. Beyond get_header and render_contex I’ve added a get_context method that is used by render_context to make this CBV even more re-usable. For example I may need to configure the context (add/remove items from the context i.e have a CBV that adds a last item with the number of list items to the list to be displayed). Of course this could be done from render_context but this means that I would need to define my new functionality (modifying the context items) and re-defining the context list formatting. It is much better (in my opinion always) to keep properly separated these things.

Now, the above is a first try that I created to mainly fulfil my requirement of having a default header and some more examples I will discuss later (and keep everything simple enough). You could extract more functionality as methods-for-overriding, for example the render method could be written like this:

def render(self):
    return self.get_template().format(
            header=self.get_header(), body=self.render_context(),
        )

and add a get_template method that will return the actual html template. There’s no hard rules here on what functionality should be extracted to a method (so it could be overridden) however I recommend to follow the YAGNI rule (i.e implement everything as normal and when you see that some functionality needs to be overridden then refactor your code to extract it to a separate method).

Let’s see an example of adding the default header functionality by overriding get_header:

class DefaultHeaderBetterCustomClassView(BetterCustomClassView, ):
    def get_header(self, ):
        return self.header if self.header else "DEFAULT HEADER"

Classes inheriting from DefaultHeaderBetterCustomClassView can choose to not actually define a header attribute so "DEFAULT HEADER" will be printed instead. Keep in mind that for DefaultHeaderBetterCustomClassView to be actually useful you’ll need to have more than one classes that need this default-header functionality (or else you could just set the header attribute of your class to "DEFAULT HEADER" - this is not user generated input, this is your source code!).

Re-using view functionality

We have come now to a crucial point in this chapter, so please stick with me. Let’s say that you have more than one class based views that contain a header attribute. You want to include the default header functionality on all of them so that if any view instantiated from these class based views doesn’t define a header the default string will be output (I know that this may be a rather trivial example but I want to keep everything simple to make following easy - instead of the default header the functionality you want to override may be adding stuff to the context or filtering the objects you’ll retrieve from the database).

To re-use this default header functionality from multiple classes you have two options: Either inherit all classes that need this functionality from DefaultHeaderBetterCustomClassView or extract the custom get_header method to a mixin and inherit from the mixin. A mixin is a class not related to the class based view hierarchy we are using - the mixin inherits from object (or from another mixin) and just defines the methods and attributes that need to be overridden. When the mixin is mixed with the ancestors of a class its functionality will be used by that class (we’ll see how shortly). So the mixin will only define get_header and not all other methods like render, get_context etc. Using the DefaultHeaderBetterCustomClassView is enough for some cases but for the general case of re-using the functionality you’ll need to create the mixin. Let’s see why:

Suppose that you have a base class that renders the header and context as JSON instead of the HTML template, something like this:

class JsonCustomClassView:
    def get_header(self, ):
        return self.header if self.header else ""

    def get_context(self, ):
        return self.context if self.context else []

    @classmethod
    def as_view(cls, *args, **kwargs):
        def view(request, ):
            instance = cls(**kwargs)
            return HttpResponse(json.dumps({
                'header': instance.get_header(),
                'context': instance.get_context(),
            }))

        return view

Notice that this class does not inherit from our previous hierarchy (i.e does not inherit from BetterCustomClassView) but from object since it provides its own as_view method. How could we re-use default header functionality in this class (without having to re-implement it)? One solution would be to create a class that inherits from both JsonCustomClassView and DefaultHeaderBetterCustomClassView using something like

# OPTION 1
class DefaultHeaderJsonCustomClassView(DefaultHeaderBetterCustomClassView, JsonCustomClassView):
    pass

# OR
# OPTION 2
class JsonDefaultHeaderCustomClassView(JsonCustomClassView, DefaultHeaderBetterCustomClassView):
    pass

What will happen here? Notice that the methods get_header and as_view exist in both ancestor classes! So which one will be used in each case? Actually, there’s a (rather complex) rule for that called MRO (Method Resolution Order). The MRO is also what can used to know which get_header and as_view will be used in each case in the previous example.

Interlude: An MRO primer

What is MRO? For every class that Python sees, it tries to create a list (MRO list) of ancestor classes containing that class as the first element and its ancestors in a specific order I’ll discuss in the next paragraph. When a method of an object of that specific class needs to be called, then the method will be searched in the MRO list (from the first element of the MRO list i.e. starting with the class it self) - when a class is found in the list that defines the method then that method instance (i.e. the method defined in this class) will be called and the search will stop (careful readers: I haven’t yet talked about super so please be patient).

Now, how is the MRO list created? As I explained, the first element is the class itself. The second element is the MRO of the leftmost ancestor of that object (so MRO will run recursively on each ancestor), the third element will be the MRO of the ancestor right next to the leftmost ancestor etc. There is one extra and important rule: When a class is found multiple times in the MRO list (for example if some elements have a common ancestor) then only the last occurrence in the list will be kept - so each class will exist only once in the MRO list. The above rule implies that the rightmost element in every MRO list will always be object - please make sure you understand why before continuing.

Thus, the MRO list for DefaultHeaderJsonCustomClassView defined in the previous section is (remember, start with the class to the left and add the MRO of each of its ancestors starting from the leftmost one): [DefaultHeaderJsonCustomClassView, DefaultHeaderBetterCustomClassView, BetterCustomClassView, CustomClassView, JsonCustomClassView, object], while for JsonDefaultHeaderCustomClassView is [JsonDefaultHeaderCustomClassView, JsonCustomClassView, DefaultHeaderBetterCustomClassView, BetterCustomClassView, CustomClassView, object]. What this means is that for DefaultHeaderJsonCustomClassView the CustomClassView.as_view() and DefaultHeaderBetterCustomClassView.get_header() (thus we will not get the JSON output) and for JsonDefaultHeaderCustomClassView the JsonCustomClassView.as_view() and JsonCustomClassView.get_header() will be used (so we won’t get the default header functionality) - i.e none of those two options will result to the desired behaviour.

Let’s try an example that has the same base class twice in the hierarchy (actually the previous examples also had a class twice in the hierarchy - object but let’s be more explicit). For this, we’ll create a DefaultContextBetterCustomClassView that returns a default context if the context is empty (similar to the default header functionality).

class DefaultContextBetterCustomClassView(BetterCustomClassView, ):
    def get_context(self, ):
        return self.context if self.context else ["DEFAULT CONTEXT"]

Now we’ll create a class that inherits from both DefaultHeaderBetterCustomClassView and DefaultContextBetterCustomClassView:

class DefaultHeaderContextCustomClassView(DefaultHeaderBetterCustomClassView, DefaultContextBetterCustomClassView):
    pass

Let’s do the MRO for the DefaultHeaderContextCustomClassView class:

Initially, the MRO will be the following:

Starting with the initial class
1. DefaultHeaderContextCustomClassView
Follows the leftmost class (DefaultHeaderBetterCustomClassView) MRO
2. DefaultHeaderContextCustomClassView, 3. BetterCustomClassView, 4. CustomClassView, 5. object
And finally the next class (DefaultContextBetterCustomClassView) MRO
6. DefaultContextBetterCustomClassView, 7. BetterCustomClassView, 8. CustomClassView, 9. object

Notice that classes BetterCustomClassView, CustomClassView and object are repeated two times (on place 3,4,5 and 7,8,9) thus only their last (rightmost) occurrences will be kept in the list. So the resulting MRO is the following (3,4,5 are removed):

[DefaultHeaderContextCustomClassView, DefaultHeaderBetterCustomClassView, DefaultContextBetterCustomClassView, BetterCustomClassView, CustomClassView, object]

One funny thing here is that the DefaultHeaderContextCustomClassView will actually work properly because the get_header will be found in DefaultHeaderBetterCustomClassView and the get_context will be found in DefaultContextBetterCustomClassView so this result to the correct functionality.

Yes it does work but at what cost? Do you really want to do the mental exercise of finding out the MRO for each class you define to see which method will be actually used? Also, what would happen if the DefaultHeaderContextCustomClassView class also had a get_context method defined (hint: that get_context would be used and the get_context of DefaultContextBetterCustomClassView would be ignored).

Before finishing this interlude, I’d like to make a confession: The Python MRO algorithm is not as simple as than the procedure I described. It uses an algorithm called C3 linearization which seems way too complex to start explaining or understanding if you not a CS student. What you’ll need to remember is that the procedure I described works fine in normal cases when you don’t try to do something stupid. Here’s a post that explains the theory more. However if you follow along my recommendations below you won’t have any problems with MRO, actually you won’t really need to use the MRO that much to understand the method calling hierarchy.

Using mixins for code-reuse

The above explanation of MRO should convince you that you should avoid mixing hierarchies of classes - if you are not convinced then wait until I introduce super() in the next section and I guarantee that you’ll be!

So, that’s why I propose implementing common functionality that needs to be re-used between classes only with mixins (hint: that’s also what Django does). Each re-usable functionality will be implemented in its own mixin; class views that need to implement that functionality will just inherit from the mixin along with the base class view. Each one of the view classes you define should inherit from one and only one other class view and any number of mixins you want. Make sure that the view class is rightmost in the ancestors list and the mixins are to the left of it (so that they will properly override its behaviour; remember that the methods of the ancestors to the left are searched first in the MRO list — and the methods of the defined class have of course the highest priority since it goes first in the MRO list).

Let’s try implementing the proposed mixins for a default header and context:

class DefaultHeaderMixin:
    def get_header(self, ):
        return self.header if self.header else "DEFAULT HEADER"

class DefaultContextMixin:
    def get_context(self, ):
        return self.context if self.context else ["DEFAULT CONTEXT"]

and all the proposed use cases using the base class view and the mixins:

class DefaultHeaderMixinBetterCustomClassView(mixins.DefaultHeaderMixin, BetterCustomClassView):
    pass

class DefaultContextMixinBetterCustomClassView(mixins.DefaultContextMixin, BetterCustomClassView):
    pass

class DefaultHeaderContextMixinBetterCustomClassView(mixins.DefaultHeaderMixin, mixins.DefaultContextMixin, BetterCustomClassView):
    pass

class JsonDefaultHeaderMixinCustomClassView(mixins.DefaultHeaderMixin, JsonCustomClassView):
    pass

I believe that the above definitions are self-documented and it is very easy to know which method of the resulting class will be called each time: Start from the main class and if the method is not found there continue from left to right to the ancestor list; since the mixins do only one thing and do it well you’ll know what each class does simply by looking at its definition.

The super situation

The final (and most complex) thing and extension I’d like to discuss for our custom class based views is the case where you want to use the functionality of more than one mixins for the same thing. For example, let’s suppose that we had a mixin that added some data to the context and a different mixing that added some different data to the context. Both would use the get_context method and you’d like to have the context data of both of them to your context. But this is not possible using the implementations above because when a get_context is found in the MRO list it will be called and the MRO search will finish there!

So how could we add the functionality of both these mixins to a class based view? This is the same problem as if we wanted to inherit from a mixin (or a class view) and override one of its methods but also call its parent (overridden) method for example to get its output and use it as the base of the output for the overridden method. Both these situations (re-use functionality of two mixins with the same method or re-use functionality from a parent method you override) are the same because what stays in the end is the MRO list. For example say we we had the following base class

class V:pass

and we wanted to override it either using mixins or by using normal inheritance.

When using mixins for example like this:

class M1:pass
class M2:pass
class MIXIN(M2, M1, V):pass

we’ll have the following MRO:

# MIXIN.mro()
# [MIXIN, M2, M1, V, object, ]

while when using inheritance like this:

class M1V(V):pass
class M2M1V(M1V):pass
class INHERITANCE(M2M1V):pass

we’ll have the following MRO:

# INHERITANCE.mro() # [INHERITANCE, M2M1V, M1V, V, object ]

As we can see in both cases the base class V is the last one (just next to object) and between this class and the one that needs the functionality (MIXIN in the first case and INHERITANCE in the second case) there are the classes that will define the extra functionality that needs to be re-used: M2 and M1 (start from left to right) in the first case and M2M1V and M1V (follow the inheritance hierarchy) in the second case. So in both cases when calling a method they will be searched the same way using the MRO list and when the method is found it will be executed and the search will stop.

But what if we needed to re-use some method from V (or from some other ancestor) and a class on the left of the MRO list has the same method? The answer, as you should have guessed by now if you have some Python knowledge is super().

The super method can be used by a class method to call a method of its ancestors respecting the MRO. Thus, running super().x() from a method instance will try to find method x() on the MRO ancestors of this instance even if the instance defines the “x()“ method i.e it will not search the first element of the MRO list. Notice that if the x() method does not exist in the headless-MRO chain you’ll get an attribute error. So, usually, you’ll can super().x() from inside the x() method to call your parent’s (as specified by the MRO list) same-named method and retrieve its output.

Let’s take a closer look at how super() works using a simple example. For this, we’ll define a method calld x() on all classes of the previous example:

class V:
    def x(self):
        print ("From V")

class M1:
    def x(self):
        super().x()
        print ("From M1")

class M2:
    def x(self):
        super().x()
        print ("From M2")

class MIXIN(M2, M1, V):
    def x(self):
        super().x()
        print ("From MIXIN")


class M1V(V):
    def x(self):
        super().x()
        print ("From M1V")

class M2M1V(M1V):
    def x(self):
        super().x()
        print ("From M2M1V")

class INHERITANCE(M2M1V):
    def x(self):
        super().x()
        print ("From INHERITANCE")

print ("MIXIN OUTPUT")
MIXIN().x()

print ("INHERITANCE OUTPUT")
INHERITANCE().x()

Here’s the output:

MIXIN OUTPUT
From V
From M1
From M2
From MIXIN
INHERITANCE OUTPUT
From V
From M1V
From M2M1V
From INHERITANCE

Notice when each message is printed: Because x() first calls its super() method and then it prints the message in both cases first the From V message is printed from the base class and then from the following classes in the hierarchy (as per the MRO) ending with the class of the instance (either MIXIN or INHERITANCE). Also the print order is the same in both cases as we’ve already explained. Please make sure you understand why the output is like this before continuing.

Using super in our hierarchy

Using super and mixins it is easy to mix and match functionality to create new classes. Of course, super can be used without mixins when overriding a method from a class you inherit from and want to also call your ancestor’s method.

Here’s how we could add a prefix to the header:

class HeaderPrefixMixin:
    def get_header(self, ):
        return "PREFIX: " + super().get_header()

and here’s how it could be used:

class HeaderPrefixBetterCustomClassView(mixins.HeaderPrefixMixin, BetterCustomClassView):
    header='Hello!'

This will retrieve the header from the ancestor and properly print the header displaying both PREFIX and Hello. What if we wanted to re-use the default header mixin? First let’s change DefaultHeaderMixin to properly use super():

class DefaultHeaderSuperMixin:
    def get_header(self, ):
        return super().get_header() if super().get_header() else "DEFAULT HEADER"
class HeaderPrefixDefaultBetterCustomClassView(mixins.HeaderPrefixMixin, mixins.DefaultHeaderSuperMixin, BetterCustomClassView):
    pass

Notice the order of the ancestor classes. The get_header() of HeaderPrefixMixin will be called which will call the get_header() of DefaultHeaderSuperMixin (which will call the get_header() of BetterCustomClassView returning None). So the result will be "PREFIX: DEFAULT HEADER". However if instead we had defined this class like

class HeaderPrefixDefaultBetterCustomClassView(mixins.DefaultHeaderSuperMixin, mixins.HeaderPrefixMixin, BetterCustomClassView):
    pass

the result would be "PREFIX: " (DEFAULT HEADER won’t be printed). Can you understand why?

One thing to keep in mind is that most probably you’ll need to call super() and return its output when you override a method. Even if you think that you don’t need to call it for this view or mixin, you may need it later from some other view or mixin that inherits from this view. Also notice that super() may not return anything but may have some side-effects in your class (for example set a self attribute) which you won’t get if you don’t call it!

For another example of super, let’s define a couple of mixins that add things to the context:

class ExtraContext1Mixin:
    def get_context(self, ):
        ctx = super().get_context()
        ctx.append('data1')
        return ctx


class ExtraContext2Mixin:
    def get_context(self, ):
        ctx = super().get_context()
        ctx.insert(0, 'data2')
        return ctx

The first one retrieves the ancestor context list and appends 'data1' to the it while the second one will insert 'data2' to the start of the list. To use these mixins just add them to the ancestor list of your class hierarchy as usually. One interesting thing to notice here is that because of how get_context is defined we’ll get the same output no matter the order of the mixins in the hierarchy since ExtraContext1Mixin will append data1 to the end of the context list and the ExtraContext2Mixin will insert data2 to the start of the context list.

class ExtraContext12BetterCustomClassView(mixins.ExtraContext1Mixin, mixins.ExtraContext2Mixin, BetterCustomClassView):
    pass

class ExtraContext21BetterCustomClassView(mixins.ExtraContext2Mixin, mixins.ExtraContext1Mixin, BetterCustomClassView):
    pass

If instead both of these mixins appended the item to the end of the list, then the output would be different depending on the ancestor order. Of course, since we’ve already defined HeaderPrefixMixin and DefaultHeaderSuperMixin nothing stops us from using all those mixins together!

class AllTogetherNowBetterCustomClassView(
        mixins.HeaderPrefixMixin,
        mixins.DefaultHeaderSuperMixin,
        mixins.ExtraContext1Mixin,
        mixins.ExtraContext2Mixin,
        BetterCustomClassView
    ):
    pass

This will have the desired behaviour of adding a prefix to the header, having a default header if not one was defined and adding the extra context from both mixins!

Testing all this

In the accompanying project at https://github.com/spapas/cbv-tutorial you can take a look at how this custom CBV hierarchy works by running it and taking a look at the core project (visit http://127.0.0.1:8001/non-django-cbv/). There you can take a look at the views.py and mixins.py to see all the views and mixins we’ve discussed in this chapter.

A high level overview of CBVs

After the previous rather long (but I hope gentle enough) introduction to implementing our own class based view hierarchy using inheritance, mixins, MRO, method overriding and super we can now start talking about the Django Class Based Views (CBVs). Our guide will be the CBV inspector application which displays all classes and mixins that Django CBVs are using along with their methods and attributes. Using this application and after reading this article you should be able to quickly and definitely know which method or attribute you need to define to each one of your mixins or views.

To use CBV inspector, just click on a class name (for example CreateView); you will immediately see its MRO ancestors, its list of attributes (and the ancestor class that defines each one) and finally a list of methods that this class and all its ancestors define. Of course when a method is defined by multiple classes the MRO ordering will be used - super is used when the functionality of the ancestor classes is also used. The CBV inspector (and our project) has Python 3 syntax. If you want to follow along with Python 2 (I don’t recommend it though since Django 2.0 only supports Python 3.x) use the following syntax to call super for method x():

super(ClassName, self).x()

this is the same as calling

super().x()

in Python 3.x.

Taking a look at the View

In any case, our travel starts from the central CBV class which is (intuitively) called … View!

This class is used as the base in Django’s CBV hierarchy (similar to how CustomClassView was used in our own hierarchy). It has only one attribute (http_method_names) and a very small number of methods. The most important method is the as_view class method (which is similar to the one we defined in the previous section). The as_view will instantiate an instance object of the View class (actually the class that inherits from View) and use this object to properly generate a functional view.

The View class cannot be used as it is but it must be inherited by a child class. The child class needs to define a method that has the same name as each http method that is supported - for example if only HTTP GET and HTTP POST are supported then the inherited class must define a get and a post method; these methods are called from the functional view through a method called dispatch and need to return a proper response object. So, we have two central methods here: The as_view class method that creates the object instance and returns its view function and dispatch that will call the proper named class method depending on the HTTP method (i.e post, get, put etc). One thing to keep from this discussion is that you shouldn’t ever need to mess with as_view but, because dispatch is the only instance method that is guaranteed to run every time the class based view will run, you will frequently need to override it especially to control access control.

As an example, we can implemented the BetterCustomClassView from the first section using View as its ancestor:

class DjangoBetterCustomClassView(View, ):
    header = ''
    context =''

    def get_header(self, ):
        return self.header if self.header else ""

    def get_context(self , ):
        return self.context if self.context else []

    def render_context(self):
        context = self.get_context()
        if context:
            return '<br />'.join(context)
        return ""

    def get(self, *args, **kwargs):
        resp = """
            <html>
                <body>
                    <h1>{header}</h1>
                    {body}
                </body>
            </html>
        """.format(
                header=self.get_header(), body=self.render_context(),
            )
        return HttpResponse(resp)

This method won’t print anything but of course it could use the mixins from before to have some default values:

class DefaultHeaderContextDjangoBetterCustomClassView(DefaultHeaderMixin, DefaultContextMixin, DjangoBetterCustomClassView):
    pass

Of course instead of using our mixins and render methods it would be much better to use the proper ones defined by Django - that’s what we’re going to do from now on I just wanted to make clear that there’s nothing special in Django’s CBV hierarchy and can be overridden as we’d like.

RedirectView and TemplateView

Continuing our tour of Django CBVs I’d like to talk a little about the classes that the CBV Inspector puts in the same level as View (GENERIC BASE): RedirectView and TemplateView. Both inherit directly from View and, the first one defines a get method that returns a redirect to another page while the latter one renders and returns a Django template in the get method.

The RedirectView inherits directly from view and has attributes like url (to use a static url) or pattern_name (to use one of the patterns define in your urls.py) to define where it should redirect. These attributes are used by the get_redirect_url which will generate the actual url to redirect to and can be overriden for example to redirect to a different location depending on the current user.

The TemplateView on the other hand inherits from View and two more classes (actually these are mixins) beyond View: TemplateResponseMixin and ContextMixin. If you take a look at them you’ll see that the TemplateResponseMixin defines some template-related attributes (most important being the template_name) and two methods: One that retrieves the template that will be used to render this View (get_template_names) and one that actually renders the template (render_to_response) using a TemplateResponse instance. The ContextMixin provides the get_context_data that is passed to the template to be rendered and should be overridden if you want to pass more context variables.

We can already see many opportunities of reusing and overriding functionality and improving our DRY score, for example: Create a catch all RedirectView that depending on the remainder of the url it will redirect to a different page, create a mixin that appends some things to the context of all CBVs using it, use dynamic templates based on some other condition (that’s actually what Detail/List/UpdateView are doing), render a template to a different output than Html (for example a text file) etc. I’ll try to present examples for these in the next section.

The FormView

The next view we’re going to talk about is FormView. This is a view that can be used whenever we want to display a form (not a form related to a Model i.e for Create/Update/Delete, for these cases there are specific CBVs we’ll see later). It is interesting to take a look at the list of its ancestors: TemplateResponseMixin, BaseFormView, FormMixin, ContextMixin, ProcessFormView and View. We are familiar with TemplateResponseMixin, ContextMixin and View but not with the others. Before discussing these classes let’s take a look at the FormView hierarchy, courtesy of http://ccbv.cco.uk and http://yuml.me:

FormView

The above diagram should make everything easier: The FormMixin inherits from ContextMixin and overrides its get_context_data method to add the form to the view. Beyond this, it adds some attributes and methods for proper form handling, for example the form_class (attribute when the form class will be the same always) and get_form_class() (method when the form class will be dynamic for example depending on the logged in user), initial and get_initial() (same logic as before for the form’s initial values), form_valid() and form_invalid() to define what should happen when the form is valid or invalid, get_form_kwargs to pass some keyword arguments to the form’s constructor etc. Notice that FormMixin does not define any form handling logic (i.e check if the form is valid and call its form_valid() method) — this logic is defined in the ProcessFormView which inherits from View and defines proper get() (just render the form) and post() (check if the form is valid and call form_valid else call form_invalid) methods.

One interesting here is to notice here is that Django defines both the FormMixin and ProcessFormView. The FormMixin offers the basic Form elements (the form class, initial data etc) and could be re-used in a different flow beyond the one offered by ProcessFormView (for example display the form as a JSON object instead of a Django template). On the other hand, ProcessFormView is required in order to define the get and post methods that are needed from the View. These methods can’t be overridden in the FormMixin since that would mean that the mixin would behave as a view!

Finally, the BaseFormView class is used to inherit from ProcessFormView and FormMixin. It does not do anything more than providing a base class that other classes that want to use the form functionality (i.e both the ProcessFormView and FormMixin) will inherit from.

The ListView and DetailView

Next in our Django CBV tour is the ListView. The ListView is used to render multiple objects in a template, for example in a list or table. Here’s a diagram of the class hierarchy (courtesy of http://ccbv.cco.uk and http://yuml.me):

ListView

The MultipleObjectMixin is used make a query to the database (either using a model or a queryset) and pass the results to the context. It also supports custom ordering (get_ordering()) and pagination (paginate_queryset()). However, the most important method of this mixin is get_queryset(). This method checks to see if the queryset or model attribute are defined (queryset will be checked first so it has priority if both are defined) and returns a queryset result (taking into account the ordering). This queryset result will be used by the get_context_data() method of this mixin to actually put it to the context by saving to a context variable named object_list. Notice that you can set the context_object_name attribute to add and extra another variable to the context with the queryset beyond object_list (for example if you have an ArticleLsitView you can set context_object_name = articles to be able to do {% for article in articles %} in your context instead of {% for article in object_list %}).

The MultipleObjectMixin can be used and overridden when we need to put multiple objects in a View. This mixin is inherited (along with View) from BaseListView that adds a proper get method to call get_context_data and pass the result to the template.

As we can also see, Django uses the MultipleObjectTemplateResponseMixin that inherits from TemplateResponseMixin to render the template. This mixin does some magic with the queryset or model to define a template name (so you won’t need to define it yourself) - that’s from where the app_label/app_model_list.html default template name is created.

Similar to the ListView is the DetailView which has the same class hierarchy as the ListView with two differences: It uses SingleObjectMixin instead of MultipleOjbectMixin, SingleObjectTemplateResponseMixin instead of MultipleObjectTemplateResponseMixin and BaseDetailView instead of BaseListView. The SingleObjectMixin will use the get_queryset() (in a similar manner to the get_queryset() of MultipleObjectMixin) method to return a single object (so all attributes and methods concerning ordering or pagination are missing) but instead has the get_object() method which will pick and return a single object from that queryset (using a pk or slug parameter). This object will be put to the context of this view by the get_context_data. The BaseDetailView just defines a proper get to call the get_context_data (of SingleObjectMixin) and finally the SingleObjectTemplateResponseMixin will automatically generate the template name (i.e generate app_label/app_model_detail.html).

The CreateView

The next Django CBV we’ll talk about is CreateView. This class is used to create a new instance of a model. It has a rather complex hierarchy diagram but we’ve already discussed most of these classes:

As we can see the CreateView inherits from BaseCreateView and SingleObjectTemplateResponseMixin. The SingleObjectTemplateResponseMixin is mainly used to define the template names that will be searched for (i.e app_label/app_model_form.html), while the BaseCreateView is used to combine the functionality of ProcessFormView (that handles the basic form workflow as we have already discussed) and ModelFormMixin. The ModelFormMixin is a rather complex mixin that inherits from both SingleObjectMixin and FormMixin. The SingleObjectMixin functionality is not really used by CreateView (since no object will need to be retrieved for the CreateView) however the ModelFormMixin is also used by UpdateView that’s why ModelFormMixin also inherits from it (to retrieve the object that will be updated).

ModelFormMixin mixin adds functionality for handling forms related to models and object instances. More specifically it adds functionality for: * creating a form class (if one is not provided) by the configured model / queryset. If you don’t provide the form class (by using the form_class attribute) then you need to configure the fields that the generated form will display by passing an array of field names through the fields attribute * overrides the form_valid in order to save the object instance of the form * fixes get_success_url to redirect to the saved object’s absolute_url when the object is saved * pass the current object to be updated (that was retrieving through the SingleObjectMixin) -if there is a current object- to the form as the instance attribute

The UpdateView and DeleteView

The UpdateView class is almost identical to the CreateView - the only difference is that UpdateView inherits from BaseUpdateView (and SingleObjectTemplateResponseMixin) instead of BaseCreateView. The BaseUpdateView overrides the get and post methods of ProcessFormView to retrieve the object (using SingleObjectMixin‘s get_object()) and assign it to an instance variable - this will then be picked up by the ModelFormMixin and used properly in the form as explained before. One thing I notice here is that it seems that the hierarchy would be better if the ModelFormMixin inherited only from FormMixin (instead of both from FormMixin and SingleObjectMixin) and BaseUpdateView inheriting from ProcessFormView, ModelForMixin and SingleObjectMixin. This way the BaseCreateView wouldn’t get the non-needed SingleObjectMixin functionality. I am not sure why Django is implemented this way (i.e the ModelFormMixin also inheriting from SingleObjectMixin thus passing this non-needed functionality to BaseCreateView) — if a reader has a clue I’d like to know it.

In any way, I’d like to also present the DeleteView which is more or less the same as the DetailView with the addition of the DeleteMixin in the mix. The DeleteMixin adds a post() method that will delete the object when called and makes success_url required (since there would be no object to redirect to after this view is posted).

Access control mixins

Another small hierarchy of class based views (actually these are all mixins) are the authentication ones which can be used to control access to a view. These are AcessMixin, LoginRequiredMixin, PermissionRequiredMixin and UserPassesTestMixin. The AccessMixin provides some basic functionality (i.e what to do when the user does not have access to the view, find out the login url to redirect him etc) and is used as a base for the other three. These three override the dispatch() method of View to check if the user has the specific rights (i.e if he has logged in for LoginRequiredMixin, if he has the defined permissions for PermissionRequiredMixin or if he passes the provided test in UserPassesTextMixin). If the user has the rights the view will proceed as normally (call super’s dispatch) else the access denied functionality from AccessMixin will be implemented.

Some other CBVs

Beyond the class based views I discussed in this section, Django also has a bunch of CBVs related to account views (LoginView, LogoutView, PasswordChangeView etc) and Dates (DateDetailView, YearArchiveView etc). I won’t go into detail about these since they follow the same concepts and use most of the mixins we’ve discussed before. Using the CBV Inspector you should be able to follow along and decide the methods you need to override for your needs.

Also, most well written Django packages will define their own CBVs that inherit from the Django CBVs - with the knowledge you acquired here you will be able to follow along on their source code to understand how everything works.

Real world use cases

In this section I am going to present a number of use cases demonstrating the usefulness of Django CBVs. In most of these examples I am going to override one of the methods of the mixins I discussed in the previous section. There are two methods you can use for integrating the following use cases to your application.

Create your own class inheriting from one of the Django CBVs and add to it directly the method to override. For example, if you wanted to override the get_queryset() method a ListView you would do a:

class GetQuerysetOverrideListView(ListView):
    def get_queryset(self):
        qs = super().get_queryset()
        return qs.filter(status='PUBLISHED')

This is useful if you know that you aren’t going to need the overriden get_queryset functionality to a different method and following the YAGNI principle (or if you know that even if you need it you could inherit from GetQuerysetOverrideListView i.e in another ListView). However, if you know that there may be more CBVs that would need their queryset filtered by status='PUBLISHED' then you should add a mixin that would be used by your CBVs:

class GetQuerysetOverrideMixin:
    def get_queryset(self):
        qs = super().get_queryset()
        return qs.filter(status='PUBLISHED')

class GetQuerysetOverrideListView(GetQuerysetOverrideMixin, ListView):
    pass

Now, one thing that needs some discussion here is that the method get_queryset is provided by a mixin (in fact it is provided by two mixins: MultipleObjectMixin for ListView and SingleObjectMixin for DetailView, UpdateView and DeleteView). Because of how MRO works, I won’t need to inherit GetQuerysetOverrideMixin from MultipleObjectMixin (or SingleObjectMixin but let’s ignore that for now) but I can just inherit from object and make sure that, as already discussed, put the mixin before (to the left) of the CBV. Notice that even if I had defined GetQuerysetOverrideMixin as GetQuerysetOverrideMixin(MultipleObjectMixin) the MultipleObjectMixin class would be found twice in the MRO list so only the rightmost instance would remain. So the MRO for both GetQuerysetOverrideMixin(object, ) and GetQuerysetOverrideMixin(MultipleObjectMixin) would be the same! Also, inheriting directly from object makes our GetQuerysetOverrideMixin more DRY since if it inherited from MultipleObjectMixin we’d need to create another version of it that would inherit from SingleObjectMixin; this is because get_queryset exists in both these mixins.

For some of the following use cases I am also going to use the following models for user generated content (articles and uploaded files):

STATUS_CHOICES = (
    ('DRAFT', 'Draft', ),
    ('PUBLISHED', 'Published', ),
    ('REMOVED', 'Removed', ),
)


class Category(models.Model):
    name = models.CharField(max_length=128, )

    def __str__(self):
        return self.name

    class Meta:
        permissions = (
            ("publisher_access", "Publisher Access"),
            ("admin_access", "Admin Access"),
        )


class AbstractGeneralInfo(models.Model):
    status = models.CharField(max_length=16, choices=STATUS_CHOICES, )
    category = models.ForeignKey('category', on_delete=models.PROTECT, )
    created_on = models.DateTimeField(auto_now_add=True, )
    created_by = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.PROTECT, related_name='%(class)s_created_by', )
    modified_on = models.DateTimeField(auto_now=True, )
    modified_by = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.PROTECT, related_name='%(class)s_modified_by', )

    owned_by = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.PROTECT, related_name='%(class)s_owned_by', )
    published_on = models.DateTimeField(blank=True, null=True)

    class Meta:
        abstract = True


class Article(AbstractGeneralInfo):
    title = models.CharField(max_length=128, )
    content = models.TextField()


class Document(AbstractGeneralInfo):
    description = models.CharField(max_length=128, )
    file = models.FileField()

All this can be found on the accompanying project https://github.com/spapas/cbv-tutorial on the djangocbv app (visit http://127.0.0.1:8001/djangocbv/).

Do something when a valid form is submitted

When a form is submitted and the form is valid the form_valid method of ModelForMixin (and FormMixin) will be called. This method can be overridden to do various things before (or after) the form is saved. For example, you may want have a field whose value is calculated from other fields in the form or you want to create an extra object. Let’s see a generic example of overriding a CreateView or UpdateView with comments:

def form_valid(self, form, ):
    # let's calculate a field value
    form.instance.calculated_field = form.cleaned_data['data1'] + form.cleaned_data['data2']

    # save the form by calling super().form_valid(); keep the return value - it is the value of get_success_url
    redirect_to = super().form_valid(form)

    # For Create or UpdateView, the just-saved object will be assigned to self.object
    logger.log("Created an object with id {0}".format(self.object.id)

    # return the redirect
    return redirect_to

This is rather complex so I’ll also explain it: The form_valid gets the actual form which, since is validated has a cleaned_data dictionary of values. This form also has an instance attribute which is the object that this form is bound to - notice that a normal Form won’t have an instance only a ModelForm. This can be used to modify the instance of this form as needed - before saving it. When you want to actually save the instance you call super().form_valid() passing it the modified form (and instance). This method does three things

  • It saves the instance to the database
  • It assigns the saved object to the object instance attribute (so you can refer to it by self.instance)
  • It uses get_redirect_url to retrieve the location where you should redirect after the form is submitted

Thus in this example we save redirect_to to return it also from our method also and then can use self.object.id to log the id of the current object.

On a more specific example, notice the Article and Document models which both inherit (abstract) from AbstractGeneralInfo have a created_by and a modified_by field. These fields have to be filled automatically from the current logged in user. Now, there are various options to do that but what I vote for is using an AuditableMixin as I have already described in my Django model auditing article.

To replicate the functionality we’ll create an AuditableMixin like this:

class AuditableMixin(object,):
    def form_valid(self, form, ):
        if not form.instance.created_by:
            form.instance.created_by = self.request.user
        form.instance.modified_by = self.request.user
        return super().form_valid(form)

This mixin can be used by both the create and update view of both Article and Document. So all four of these classes will share the same functionality. Notice that the form_valid method is overridden - the created_by of the form’s instance (which is the object that was edited, remember how ModelFormMixin works) will by set to the current user if it is null (so it will be only set once) while the modified_by will be set always to the current user. Finally we call super().form_valid and return its response so that the form will be actually saved and the redirect will go to the proper success url. To use it for example for the Article, CreateView should be defined like this:

class ArticleCreateView(AuditableMixin, CreateView):
    class Meta:
        model = Article

Change the queryset of the CBV

All CBVs that inherit from SingleObjectMixin or MultipleObjectMixin (ListView, DetailView, UpdateView and DeleteView) have a model and a queryset property that can be used (either one or the other) to define the queryset that will be used for querying the database for that CBVs results. This queryset can be further dynamically refined by overriding the get_queryset() method. What I usually do is that I define the model attribute and then override get_querset in order to dynamically modify the queryset.

For example, let’s say that I wanted to add a count of articles and documents per each category. Here’s how the CategoryListView could be done:

class CategoryListView(ExportCsvMixin, AdminOrPublisherPermissionRequiredMixin, ListView):
    model = Category
    context_object_name = 'categories'

    def get_queryset(self):
        qs = super().get_queryset()
        return qs.annotate(article_cnt=Count('article'), document_cnt=Count('document'))

Notice that I also use some more mixins for this ListView (they’ll be explained later). The get_queryset adds the annotation to the super() queryset (which will be Category.objects.all()). One final comment is that instead of this, I could have more or less the same functionality by implementing CategoryListView:

class CategoryListView(ExportCsvMixin, AdminOrPublisherPermissionRequiredMixin, ListView):
    context_object_name = 'categories'
    query = Category.objects.all().annotate(article_cnt=Count('article'), document_cnt=Count('document'))

This has the same functionality (return all categories with the number of articles and documents for each one) and saves some typing from overriding the get_queryset method. However as I said most of the time I use the model attribute and override the get_queryset method because it seems more explicit and descriptive to me and most of the time I’ll need to add some more filtering (based on the current user, based on some query parameter etc) that can only be implemented on the get_queryset.

Allow each user to list/view/edit/delete only his own items

Continuing from the previous example of modifying the queryset, let’s suppose that we want to allow each user to be able to list the items (articles and documents) he has created and view/edit/delete them. We also want to allow admins and publishers to view/edit everything.

Since the Article and Document models both have an owned_by element we can use use this to filter the results returned by get_queryset(). For example, here’s a mixin that checks if the current user is admin or publisher. If he is a publisher then he will just return the super() queryset. If however he is a simple user it will return only the results that are owned by him with qs.filter(owned_by=self.request.user).

class LimitAccessMixin:
    def get_queryset(self):
        qs = super().get_queryset()
        if self.request.user.has_perm('djangocbv.admin_access') or self.request.user.has_perm('djangocbv.publisher_access') :
            return qs
        return qs.filter(owned_by=self.request.user)

Another similar mixin that is used is the HideRemovedMixin that, for simple users, excludes from the queryset the objects that are removed:

class HideRemovedMixin:
    def get_queryset(self):
        qs = super().get_queryset()
        if self.request.user.has_perm('djangocbv.admin_access') or self.request.user.has_perm('djangocbv.publisher_access'):
            return qs
        return qs.exclude(status='REMOVED')

One thing that needs a little discussion is that for both of these mixins I am using get_queryset to implement access control to allow using the same mixin for views that inherit from both SingleObjectMixin and MultipleObjectMixin (since the get_queryset is used in both of them). This means that when a user tries to access an object that has not access to he’ll get a nice 404 error.

Beyond this, instead of filtering the queryset, for views inheriting from SingleObjectMixin (i.e DetailView, UpdateView and DeleteView) we could have overridden the get_object method to raise an access denied exception. Here’s how get_object could be overridden to raise a 403 Forbidden status when a user tries to access an object that does not belong to him:

from django.core.exceptions import PermissionDenied

def get_object(self, queryset=None):
    obj = super().get_object()
    if obj.owned_by=self.request.user:
        raise PermissionDenied
    return obj

Configure the form’s initial values from GET parameters

Sometimes we want to have a CreateView with some fields already filled. I usually implement this by passing the proper parameters to the URL (i.e by calling it as /create_view?category_id=2) and then using the following mixin to override the FormMixin get_initial method in order to return the form’s initial data from it:

class SetInitialMixin(object,):
    def get_initial(self):
        initial = super(SetInitialMixin, self).get_initial()
        initial.update(self.request.GET.dict())
        return initial

So if the /article_create url can be used to initialte the CreateView for the article, using /article_create?category_id=3 will show the CreateView with the Category with id=3 pre-selected in the category field!

Pass extra kwargs to the FormView form

This is a very common requirement. The form may need to be modified by an external condition, for example the current user or something that can be calculated from the view. Here’s a sample mixin that passes the current request (which also includes the user) to the form:

class RequestArgMixin:
    def get_form_kwargs(self):
        kwargs = super(RequestArgMixin, self).get_form_kwargs()
        kwargs.update({'request': self.request})
        return kwargs

Please notice that the form has to properly handle the extra kwarg in its constructor, before calling the super’s constructor. For example, here’s how a form that can accept the request could be implemented:

class RequestForm(forms.Form):
    def __init__(self, *args, **kwargs):
        self.request = kwargs.pop('request', None)
        super().__init__(*args, **kwargs)

We use pop to remove the request from the received kwargs and only then we call the parent constructor.

Add values to the context

To add values to the context of a CBV we override the get_context_data() method. Here’s a mixin that adds a list of categories to all CBVs using it:

class CategoriesContextMixin:
    def get_context_data(self, **kwargs):
        ctx = super().get_context_data(**kwargs)
        ctx['categories'] = Category.objects.all()
        return ctx

Notice that the mixin calls super to get the context data of its ancestors and appends to it. This mean that if we also had a mixin that f.e added the current logged in user to the context (this isn’t really needed since there’s a context processor for this but anyway) then when a CBV inherited from both of them then the data of both of them would be added to the context.

As a general comment there are three other methods the same functionality could be achieved:

  • Just override the get_context_data of the CBV you want to add extra data to its context
  • Add a template tag that will bring the needed data to the template
  • Use a context processor to bring the data to all templates

As can be understood, each of the above methods has certain advantages and disadvantages. For example, if the extra data will query the database then the context processor method will add one extra query for all page loads (even if the data is not needed). On the other hand, the template tag will query the database only on specific views but it makes debugging and reasoning about your template more difficult since if you have a lot of template tags you’ll have various context variables appearing from thing air!

One final comment is that overriding the get_context_data method will probably be the most common thing you’re going to do when using CBVs (you’ll definitely need to add things to the context) so try to remember the following 3 needed lines:

def get_context_data(self, **kwargs):
    ctx = super().get_context_data(**kwargs)
    # ... here we add stuff to the ctx
    return ctx

Add a simple filter to a ListView

For filtering I recommend using the excellent django-filter package as I’ve already presented in my essential Django package list. Here’s how a mixin can be created that adds a filter to the context:

class AddFilterMixin:
    filter_class = None

    def get_context_data(self, **kwargs):
        ctx = super().get_context_data(**kwargs)
        if not self.filter_class:
            raise NotImplementedError("Please define filter_class when using AddFilterMixin")
        filter = self.filter_class(self.request.GET, queryset=self.get_queryset())
        ctx['filter'] = filter
        if self.context_object_name:
            ctx[self.context_object_name] = filter.qs
        return ctx

Notice that the get_context_data checks to see if the filter_class attribute has been defined (if not it will raise a useful explanation). It will then instantiate the filter class passing it the self.request.GET and the current queryset (self.get_queryset()) - so for example any extra filtering you are doing to the queryset (for example only show content owned by the current user) will be also used. Finally, pass the filter to the context and assign the contect_object_name to the filtered queryset.

Here’s for example how this mixin is used for ArticleListView:

class ArticleListView(AddFilterMixin, ListView):
    model = Article
    context_object_name = 'articles'
    filter_class = ArticleFilter

And then just add the following to the article_list.html template:

<form method='GET'>
    {{ filter.form }}
    <input type='submit' value='Filter' />
</form>
{% for article in articles %}
    Display article info - only filtered articles will be here
{% endfor %}

Support for success messages

Django has a very useful messages framework which can be used to add flash messages to a view. A flash message is a message that persists in the sesion until it is viewed by the user. So, for example when a user edits an object and saves it, he’ll be redirected to the success page - if you have configured a flash message to inform the user that the save was ok then he’ll see this message once and then if he reloads the page it will be removed.

Here’s a mixin that can be used to support flash messages using Django’s message framework:

class SuccessMessageMixin:
    success_message = ''

    def get_success_message(self):
        return self.success_message

    def form_valid(self, form):
        messages.success(self.request, self.get_success_message())
        return super().form_valid(form)

This mixin overrides the form_valid and adds the message using get_success_message - this can be overriden if you want to have a dynamic message or just set the success_message attribute for a static message, for example something like this:

class SuccesMessageArticleCreateView(SuccessMessageMixin, CreateView):
    success_message = 'Object was created!'

    class Meta:
        model = Article

I’d like to once again point out here that since the super().form_valid(form) method is properly used then if a CBV uses multiple mixins that override form_valid (for example if your CBV overrides both SuccessMessageMixin and AuditableMixin then the form_valid of both will be called so you’ll get both the created_by/modified_by values set to the current user and the success message!

Notice that Django actually provides an implementation of a message mixin which can be used instead of the proposed implementation here (I didn’t know it until recently that’s why I am using this to some projects and I also present it here).

Implement a quick moderation

It is easy to implement some moderation to our model publishing. For example, let’s suppose that we only allow publishers to publish a model. Here’s how it can be done:

def form_valid(self, form):
    if form.instance.status != 'REMOVED':
        if self.request.user.has_perm('djangocbv.publisher_access'):
            form.instance.status = 'PUBLISHED'
        else:
            form.instance.status = 'DRAFT'

    return super().form_valid(form)

So, first of all we make sure that the object is not REMOVED (if it is remove it we don’t do anything else). Next we check if the current user has publisher_access if yes we change the object’s status to PUBLISHED - on any other case we change its status to DRAFT. Notice that this means that whenever a publisher saves the object it will be published and whenever a non-publisher saves it it will be made a draft. We then call our ancestor’s form_valid to save the object and return to success url.

I’d like to repeat here that this mixin, since it calls super, can work concurrently with any other mixins that override form_valid (and also call their super method of course), for example it can be used together with the audit (auto-fill created_by and moderated_by) and the success mixin we defined previously!

Allow access to a view if a user has one out of a group of permissions

For this we’ll need to use the authentication mixins functionality. We could implement this by overriding PermissionRequiredMixin or by overriding UserPassesTestMixin.

Using PermissionRequiredMixin is not very easy because the way it works it will allow access if the user has all permissions from the group (not only one as is the requirement). Of course you could override its has_permission method to change the way it checks if the user has the permissions (i.e make sure it has one permission instead of all):

class AnyPermissionRequiredMixin(PermissionRequiredMixin, ):
    def has_permission(self):
        perms = self.get_permission_required()
        return any(self.request.user.has_perm(perm) for perm in perms)

Also we could implement our mixin using UserPassesTestMixin as its base:

class AnyPermissionRequiredAlternativeMixin(UserPassesTestMixin):
    permissions = []

    def test_func(self):
        return any(self.request.user.has_perm(perm) for perm in self.permissions)

The functionality is very simple: If the user has one of the list of the configured permissions then the test will pass (so he’ll have access to the view). If instead the user has none of the permissions then he won’t be able to access the view.

Notice that for the above implementations we inherited from PermissionRequiredMixin or UserPassesTextMixin to keep their functionality - if we had inherited these mixins from object then we’d need to inherit our CBVs from both AnyPermissionRequiredMixin and PermissionRequiredMixin or AnyPermissionRequiredAlternativeMixin and UserPassesTestMixin (with the correct MRO order of course).

Now, the whole permission cheking functionality can be even more DRY. Let’s suppose that we know that there are a couple of views which should only be visible to users having either the app.admin or app.curator permission. Instead of inheriting all these views from AnyPermissionRequiredMixin and configuring the permissions list to each one, the DRY way to implement this is to add yet another mixin from which the CBVs will actually inhert:

class AdminOrPublisherPermissionRequiredMixin(AnyPermissionRequiredMixin):
    permissions = ['djangocbv.admin_access', 'djangocbv.publisher_access']

Disable a view based on some condition

There are times you want to disable a view based on an arbitrary condition - for example example make the view disabled before a specific date. Here’s a simple mixin that overrides dispatch to do this:

class DisabledDateMixin(object, ):
    the_date = datetime.date(2018,1,1)

    def dispatch(self, request, *args, **kwargs):
        if datetime.date.today() < the_date:
            raise PermissionDenied
        return super().dispatch(request, *args, **kwargs)

You can even disable a view completely in case you want to keep it in your urls.py using this mixin:

class DisabledDateMixin(object, ):
    def dispatch(self, request, *args, **kwargs):
        raise PermissionDenied

Output non-html views

I’ve written a whole article about this, please take a look at my Django non-HTML responses article.

Also, notice that is very easy to create a mixin that will output a view to PDF - I have already written an essential guide for outputting PDFs in Django so I am just going to refer you to this article for (much more) information!

Finally, let’s take a look at a generic Mixin that you can use to add CSV exporting capabilities to a ListView:

class ExportCsvMixin:
    def render_to_response(self, context, **response_kwargs):
        if self.request.GET.get('csv'):
            response = HttpResponse(content_type='text/csv')
            response['Content-Disposition'] = 'attachment; filename="export.csv"'

            writer = csv.writer(response)
            for idx, o in enumerate(context['object_list']):
                if idx == 0: # Write headers
                    writer.writerow(k for (k,v) in o.__dict__.items() if not k.startswith('_'))
                writer.writerow(v for (k,v) in o.__dict__.items() if not k.startswith('_'))

            return response
        return super().render_to_response(context, **response_kwargs)

As you can see this mixin overrides the render_to_response method. It will check if there’s a csv key to the GET queryset dictionary, thus the url must be called with ?csv=true or something similar. You can just add this link to your template:

<a class='button' href='?csv=true'>Export csv</a>

So if the view needs to be exported to CSV, it will create a new HttpResponse object with the correct content type. The next line will add a header that (Content-Disposition) will mark the response as an attachment and give it a default file name. We then crate a new csv.writer passing the just-created response as the place to write the csv. The for loop that follows enumerates the object_list value of the context (remember that this is added by the MultipleObjectMixin and contains the result of the ListView). It will then use the object’s __dict__ attribute to write the headers (for the first time) and then write the values of all objects.

As another simple example, let’s create a quick JSON output mixin for our DetailViews:

class JsonDetailMixin:
    def render_to_response(self, context, **response_kwargs):
        if self.request.GET.get('json'):
            response = HttpResponse(content_type='application/json')
            response.write(json.dumps(dict( (k,str(v)) for k,v in self.object.__dict__.items() )))
            return response
        return super().render_to_response(context, **response_kwargs)

If you add this to a view inheriting from DetailView and pass it the ?json=true query parameter you’ll get a JSON response!

Use one TemplateView for multiple html templates

Using a TemplateView you could display an html template without much problem just by settings the template attribute of your class. What if you wanted to have a single TemplateView that would display many templates based on the query path? Simple, just override get_template_names to return a different template based on the path. For example, using this view:

class DynamicTemplateView(TemplateView):
    def get_template_names(self):
        what = self.kwargs['what']
        return '{0}.html'.format(what)

You can render any template you have depending on the value of the what kwarg. To allow only specific template names you can either add a check to the above implementation (i.e that what is help or about) or you may do it to the urls.py if you use a regular expression. Thus, to only allow help.html and about.html to be rendered with this method add it to your urls like this:

re_path(r'^show/(?P<what>help|about)/', views.DynamicTemplateView.as_view(), name='template-show'),

Finally, to use it to render the help.html you’ll just call it like <a href=’{% url “template-show” “help” %}’>Help</a>

Notice that of course instead of creating the DynamicTemplateView you could just dump these html files in your static folder and return them using the static files functionality. However the extra thing that the DynamicTemplateView brings to you is that this is a full Django template thus you can use template tags, filters, your context variables, inherit from your site-base and even override get_context_data to add extra info to the template! All this is not possible with static files!

Implement a partial Ajax view

Overriding get_template_names can also be used to create a DRY Ajax view of your data! For example, let’s say that you have a DetailView for one of your models that has overriden the get_template_names like this:

def get_template_names(self):
    if self.request.is_ajax() or self.request.GET.get('ajax_partial'):
        return 'core/partial/data_ajax.html'
    return super().get_template_names()

and you have also defined a normal template for classic request response viewing and an ajax template that contains only the specific data for this instances (i.e it does not containg html body, headers, footers etc, only a <div> with the instance’s data). Notice I’m using either the is_ajax method or I directly passed GET value (ajax_partial) - this is needed because sometimes is_ajax is not working as expected (depending on how you’re going to do the request), also this way you can easily test the partial ajax view through your browser by passing it ?ajax_partia=true.

Using this technique you can create an Ajax view of your data just by requesting the DetailView through an Ajax call and dumping the response you get to a modal dialog (for example) - no need for fancy REST APIs. Also as a bonus, the classic DetailView will work normally, so you can have the Ajax view to give a summary of the instance’s data (i.e have a subset of the info on the Ajax template) and the normal view to display everything.

Add a dynamic filter and/or table to the context

If you have a lot of similar models you can add a mixin that dynamically creates tables and a filters for these models - take a look at my dynamic tables and filters for similar models article!

Configure forms for your views

As I’ve already explained if you are using a FormView you’ll need to set a form_class for your view (needed by FormMixin) while, for an Update or CreateView which use the ModelFormMixin you can either set the form-class or directly configure the instance’s fields that will be displayed to the form using the fields attribute.

For example, let’s say that you have a rather generic FormView that will display a different form depending on the user permissions. Here’s how you could do this to return a SuperForm if the current user is a superuser and a SimpleForm in other cases:

def get_form_class(self):
    if self.request.user.is_superuser:
        return SuperForm
    return SimpleForm

Display a different form for Create and Update

There are various ways you can do this (for example you can just declare a different form_class for your Create and UpdateView) but I think that the most DRY one, especially if the create and update form are similar is to pass an is_create argument to the form which it will then be used to properly configure the form.

Thus, on your CreateView you’ll add this get_form_kwargs:

def get_form_kwargs(self):
    kwargs = super(MyCreateView, self).get_form_kwargs()
    kwargs.update({'is_create': True})
    return kwargs

while on your UpdateView you’ll add this:

def get_form_kwargs(self):
    kwargs = super(MyUpdateView, self).get_form_kwargs()
    kwargs.update({'is_create': False})
    return kwargs

Please notice that the form has to properly handle the extra kwarg in its constructor as I’ve already explained previously.

Only allow specific HTTP methods for a view

Let’s say that you want to create an UnpublishView i.e a view that will change the status of your content to DRAFT. Since this view will change your model instance it must be called through POST, however you may not want to display an individual form for this view, just a button that when called will display a client-side (Javascript) prompt and if the user clicks it it will immediately do a POST request by submitting the form. The best way to create this is to just implement an UpdateView for your model and change its form valid to change the status to DRAFT, something like this:

def form_valid(self, form, ):
    form.instance.status = 'DRAFT'
    return super().form_valid(form)

Beyond this, you’ll need to add a fields = [] attribute to your UpdateView to denote that you won’t need to update any fields from the model (since you’ll update the status directly) and finally, to only allow this view to be called through an http POST method add the following attribute:

http_method_names = ['post',]

Create an umbrella View for multiple models

Let’s say that you have a couple of models (called Type1 and Type2 that are more or less the same and you want to quickly create a ListView for both of them but you’d like to create just one ListView and separate them by their url. Here’s how it could be done:

class UmbrellaListView(ListView):
    template_name='umbrella_list.html'

    def dispatch(self, request, *args, **kwargs):
        self.kind = kwargs['kind']
        if self.kind == 'type1':
            self.queryset = models.Type1.objects.all()
        elif self.kind == 'type2':
            self.queryset = models.Type2.objects.all()
        return super(UmbrellaListView, self).dispatch(request, *args, **kwargs)

Notice that for this to work properly you must setup your urls like this:

...
url(r'^list/(?P<kind>type1|type2)/$', UmbrellaListView.as_view() ) , name='umbrella_list' ),
...

A heavy CBV user project

In this small chapter I’d like to present a bunch of mixins and views that I’ve defined to the accompanying project (https://github.com/spapas/cbv-tutorial).

Let’s start with the mixins (I won’t show the mixins I’ve already talked about in the previous chapter):

class SetOwnerIfNeeded:
    def form_valid(self, form, ):
        if not form.instance.owned_by_id:
            form.instance.owned_by = self.request.user
        return super().form_valid(form)


class ChangeStatusMixin:
    new_status = None

    def form_valid(self, form, ):
        if not self.new_status:
            raise NotImplementedError("Please define new_status when using ChangeStatusMixin")
        form.instance.status = new_status
        return super().form_valid(form)


class ContentCreateMixin(SuccessMessageMixin,
                        AuditableMixin,
                        SetOwnerIfNeeded,
                        RequestArgMixin,
                        SetInitialMixin,
                        ModerationMixin,
                        LoginRequiredMixin):
    success_message = 'Object successfully created!'


class ContentUpdateMixin(SuccessMessageMixin,
                        AuditableMixin,
                        SetOwnerIfNeeded,
                        RequestArgMixin,
                        SetInitialMixin,
                        ModerationMixin,
                        LimitAccessMixin,
                        LoginRequiredMixin):
    success_message = 'Object successfully updated!'


class ContentListMixin(ExportCsvMixin, AddFilterMixin, HideRemovedMixin, ):
    pass


class ContentRemoveMixin(SuccessMessageMixin
                         AdminOrPublisherPermissionRequiredMixin,
                         AuditableMixin,
                         HideRemovedMixin,
                         ChangeStatusMixin,):
    http_method_names = ['post',]
    new_status = 'REMOVED'
    fields = []
    success_message = 'Object successfully removed!'


class ContentUnpublishMixin(SuccessMessageMixin
                            AdminOrPublisherPermissionRequiredMixin,
                            AuditableMixin,
                            UnpublishSuccessMessageMixin,
                            ChangeStatusMixin,):
    http_method_names = ['post',]
    new_status = 'DRAFT'
    fields = []
    success_message = 'Object successfully unpublished!'

The SetOwnerIfNeeded and ChangeStatusMixin are simple mixins that override form_valid to introduce some functionality before saving the object).

The mixins that follow are used to group functionality of other mixins together and will be inherited by the views. Thus, ContentCreateMixin has the mixin functionality needed to create something (for example an Article or a Document) i.e show a success message, add auditing information, set the object’s owner, pass the request to the form, set the form’s initial values, do some moderation and only allow logged in users. On a similar fashion, the ContentUpdateMixin collects the functionality needed to update something and is similar to ContentCreateMixin (with the difference that it also as the LimitAccessMixin to only allow simple users to edit their own content). The ContentListMixin adds functionality for export to CSV, simple filter and hiding removed things.

Finally, the ContentRemoveMixin and ContentUnpublishMixin are used to implement Views for removing and unpublishing an object. Both of them inherit from ChangeStatusMixin - one setting the new_status to REMOVED the other to DRAFT.

Notice that they share much functionality so I could remove both ContentRemoveMixin and ContentUnpublishMixin and add a single ContentChangeStatusMixin like this:

class ContentChangeStatusMixin(AdminOrPublisherPermissionRequiredMixin,
                            AuditableMixin,
                            UnpublishSuccessMessageMixin,
                            ChangeStatusMixin,):
    http_method_names = ['post',]
    fields = []

Thus the new_status attribute wouldn’t be there so the views inheriting from this ContentChangeStatusMixin (i.e *RemoveView and *UnpublishView) would need to define the new_status field themselves. This is definitely valid (and more DRY) but less explicit than the way I’ve implemented this - i.e you may wanted to not allow publishers to remove objects, only admins (so you could implement that differently in the get_queryset or dispatch method of ContentRemoveMixin and ContentUpdateMixin) this is easier if you have both the ContentRemoveMixin and ContentUnpublishMixin.

Now let’s take a look at the views:

class CategoryListView(ExportCsvMixin, AdminOrPublisherPermissionRequiredMixin, ListView):
    model = Category
    context_object_name = 'categories'

    def get_queryset(self):
        qs = super().get_queryset()
        return qs.annotate(article_cnt=Count('article'), document_cnt=Count('document'))


class CategoryCreateView(SuccessMessageMixin, AdminOrPublisherPermissionRequiredMixin, CreateView):
    model = Category
    fields = ['name']
    success_message = 'Category created!'
    success_url = reverse_lazy('category-list')


class CategoryUpdateView(SuccessMessageMixin, AdminOrPublisherPermissionRequiredMixin, UpdateView):
    model = Category
    fields = ['name']
    success_message = 'Category updated!'
    success_url = reverse_lazy('category-list')


class CategoryDetailView(CategoriesContextMixin, DetailView):
    model = Category
    context_object_name = 'category'

    def get_context_data(self, **kwargs):
        ctx = super().get_context_data(**kwargs)
        ctx['article_number'] = Article.objects.filter(category=self.object).count()
        ctx['document_number'] = Document.objects.filter(category=self.object).count()
        return ctx


class ArticleListView(ContentListMixin, ListView):
    model = Article
    context_object_name = 'articles'
    filter_class = ArticleFilter


class ArticleCreateView(ContentCreateMixin, CreateView):
    model = Article
    form_class = ArticleForm
    success_url = reverse_lazy('article-list')


class ArticleUpdateView(ContentUpdateMixin, UpdateView):
    model = Article
    form_class = ArticleForm
    success_url = reverse_lazy('article-list')


class ArticleDetailView(HideRemovedMixin, JsonDetailMixin, DetailView):
    model = Article
    context_object_name = 'article'

    def get_template_names(self):
        if self.request.is_ajax() or self.request.GET.get('partial'):
            return 'djangocbv/_article_content_partial.html'
        return super().get_template_names()


class ArticleRemoveView(ContentRemoveMixin, UpdateView):
    model = Article
    success_url = reverse_lazy('article-list')


class ArticleUnpublishView(ContentUnpublishMixin, UpdateView):
    model = Article
    success_url = reverse_lazy('article-list')


class DocumentListView(ContentListMixin, ListView):
    model = Document
    context_object_name = 'documents'
    filter_class = DocumentFilter


class DocumentCreateView(ContentCreateMixin, CreateView):
    model = Document
    form_class = DocumentForm
    success_url = reverse_lazy('document-list')


class DocumentUpdateView(ContentUpdateMixin, UpdateView):
    model = Document
    form_class = DocumentForm
    success_url = reverse_lazy('document-list')


class DocumentDetailView(HideRemovedMixin, JsonDetailMixin, DetailView):
    model = Document
    context_object_name = 'document'


class DocumentRemoveView(ContentRemoveMixin, UpdateView):
    model = Document
    success_url = reverse_lazy('document-list')


class DocumentUnpublishView(ContentUnpublishMixin, UpdateView):
    model = Document
    success_url = reverse_lazy('document-list')


class DynamicTemplateView(TemplateView):
    def get_template_names(self):
        what = self.kwargs['what']
        return '{0}.html'.format(what)

As you will see there are

  • 4 views related to categories (Create, Detail, Update and List)
  • 6 views related to articles (Create, Detail, Update, List, Unpublish and Remove) and
  • another 6 views related to documents (same as articles).

The views for Article and Document are more or less the same: They inherit from the corresponding mixin we defined previously (CreateContentMixin, UpdateContentMixin etc) add a redirect to their corresponding list view (I am using redirect_lazy there because redirect wouldn’t work because it will lead to a cyclic dependency between urls and views) and define their corresponding form and model. For the the DetailViews I don’t use a group mixin like the others but I just add the HideRemovedMixin and JsonDetailMixin directly to their ancestor list. This is to make clear that the group mixins (ContentCreateMixin etc) are optional and I could for example define ArticleCreateView like this:

class ArticleCreateView(SuccessMessageMixin,
                    AuditableMixin,
                    SetOwnerIfNeeded,
                    RequestArgMixin,
                    SetInitialMixin,
                    ModerationMixin,
                    LoginRequiredMixin,
                    CreateView):
    model = Article
    form_class = ArticleForm
    success_url = reverse_lazy('article-list')
    success_message = 'Object successfully created!'

What’s the advantage of using the ContentCreateMixin then? Well since the same mixins are used from DocumentCreateView I won’t need to re-define this list there. Also if for example I want to allow users that have a specific permission to create articles and document I will remove the LoginRequireMixin and add the AllowCreateContentMixin only to the ContentCreateMixin and not to both ArticleCreateView and DocumentCreateView (and I won’t be in danger of forgetting to change it somewhere). Of course all this depends on your requirements and how DRY you need to be.

The category related views are simpler and override their mixins directly. Finally there’s a DynamicTemplateView to display templates based on their filename as discussed previously.

Before continuing, please try to understand how much more DRY this project is when compared to using a traditional functional one (or if not using mixins). For example, there’s a RequestArgMixin that is used by all views that create/update content. If INHERITANCE didn’t use that mixin I’d need to re-define the same functionality (pass the current request to the form’s constructor) to 4 views (Article/Document Create/Update). Or for the AuditableMixin I’d need to remember to upgrade the created/modified by to 8 views (Article/Document Create/Update/Unpublish/Remove)!

Conclusion

The previous discussion should convince you how much more DRY your views will be when using CBVs and how much quicker will be to create your views. Also, if you followed closely the first and second chapters you should be able to understand everything that is needed for CBVs and be able to properly understand which method or attribute you need to override to implement some specific functionality. Finally, the list of examples in the third chapter should help you get started in all your CBV needs - if you have some specific question about CBVs or you’d like another use case added to the list feel free to ask and I’ll try to add it!

Easy downloading youtube videos and mp3s using youtube-dl and python

In this article I am going to present you with an easy (and advertisement/malware free) way to download videos from youtube, converting them to mp3 if needed. Also, I will give some more useful hints, for example how to download multiple mp3s using a script, how to break a long mp3 to same-length parts so you could quickly skip tracks when you play it in your stereo etc.

I am going to give specific instructions for Windows users - however everything I’ll propose should also be applicable to OSX or Linux users with minor modifications.

The tools we are going to use are:

  • youtube-dl which is a python library for downloading videos from youtube (and some other sites)
  • ffmpeg which is a passepartout video/audio editing library

Installation

To install youtube-dl I recommend installing it in your global python (2 or 3) package list using pip. Please read my previous article

to see how you should install and use Python 2 or 3 on Windows. Following the instructions from there, you can do the following to install youtube-dl in your global Python 3 packages:

py -3 -m pip install youtube-dl

To run youtube-dl you’ll write something like py -3 -m youtube_dl url_or_video_id (notice the underscore instead of dash since dashes are not supported in module names in python). For example try something like this py -3 -m youtube_dl YgtL4S7Hrwo and you’ll be rewarded with the 2016 Pycon talk from Guido van Rossum! If you find this a little too much too type don’t be afraid, I will give some hints later about how to improve this.

To upgrade your youtube-dl installation you should do something like this:

py -3 -m pip install -U youtube-dl

Notice that you must frequently upgrade your youtube-dl installation because sometimes youtube changes the requirements for viewing / downloading videos and your old version will not work. So if for some reason something is not correct when you use youtube-dl always try updating first.

If you wanted you could also create a virtual environment (see instructions on previously mentioned article) and install youtube-dl locally there using pip install youtube-dl however I prefer to install it on the global packages to be really easy for me to call it from a command prompt I open. Notice that if you install youtube-dl in a virtualenv, after you activate that virtualenv you’ll be able to run it just by typing youtube-dl.

Finally, If for some reason you don’t want want to mess with python and all this (or it just seems greek to you) then you may go on and directly download a youtube-dl windows executable. Just put it in your path and you should be good to go.

To install ffmpeg, I recommend downloading the Windows build from here (select the correct Windows architecture of your system and always static linking - the version doesn’t matter). This will get you a zip - we are mainly interested to the three files under the bin folder of that zip which should be copied to a directory under your path:

  • ffmpeg is the passepartout video converting toot that we are going to use
  • ffprobe will print some information about a file (about its container and video/audio streams)
  • ffplay will play the file — not really recommended there are better tools but it is invaluable for testing; if it can be played be ffplay then ffmpeg will be able to properly read your file

Notice I recommend copying things to a directory in your path. This is recommended and will save you from repeatedly typing the same things over and over. Also, later I will propose a bunch of DOS batch (.bat) files that can also be copied to that directory and help you even more in you youtube video downloading. To add a directory to the PATH, just press Windows+Pause Break, Advanced System Settings, Advanced, Environment Variables, edit the “Path” User variable (for your user) and append the directory there.

Using youtube-dl

As I’ve already explained before, to run youtube-dl you’ll either write something like py -3 -m youtube_dl (if you’ve installed it to your global python packages) or run youtube-dl if you’ve downloaded the pre-built exe or have installed it in a virtualenv. To save you from some keystrokes, you can create a batch file that will run and pass any more parameters to it, something like this:

py -3 -m youtube_dl %*

(the %* will capture the remaining command line) so to get the previous video just run getvideo YgtL4S7Hrwo (or getvideo https://www.youtube.com/watch?v=YgtL4S7Hrwo - works the same with the video id or the complete url).

One thing I’d like to mention here is that youtube-dl works fine with playlists and even channels. For example, to download all videos from PyCon 2017 just do this:

getvideo https://www.youtube.com/channel/UCrJhliKNQ8g0qoE_zvL8eVg/feed and you should see something like:

E:\>py -3 -m youtube_dl https://www.youtube.com/channel/UCrJhliKNQ8g0qoE_zvL8eVg/feed
[youtube:channel] UCrJhliKNQ8g0qoE_zvL8eVg: Downloading channel page
[youtube:playlist] UUrJhliKNQ8g0qoE_zvL8eVg: Downloading webpage
[download] Downloading playlist: Uploads from PyCon 2017
[youtube:playlist] UUrJhliKNQ8g0qoE_zvL8eVg: Downloading page #1
[youtube:playlist] playlist Uploads from PyCon 2017: Downloading 143 videos
[download] Downloading video 1 of 143
[youtube] AjFfsOA7AQI: Downloading webpage
[youtube] AjFfsOA7AQI: Downloading video info webpage
[youtube] AjFfsOA7AQI: Extracting video information
WARNING: Requested formats are incompatible for merge and will be merged into mkv.
[download] Destination: Final remarks and conference close  - Pycon 2017-AjFfsOA7AQI.f137.mp4
[download]   2.9% of 34.49MiB at 940.52KiB/s ETA 00:36

This is gonna take some time …

Now, youtube-dl has many options and can be configured with default values depending on your requirements. I won’t go into detail about these except on some things I usually use, if you need some help feel free to ask me.

When you download a video, youtube-dl will try to download the best quality possible for that video, however a video may have various different formats that can be queries by passing the option --list-formats to ffmpeg, for example here’s the output from the previously mentioned video:

E:\>getvideo YgtL4S7Hrwo --list-formats
[youtube] YgtL4S7Hrwo: Downloading webpage
[youtube] YgtL4S7Hrwo: Downloading video info webpage
[youtube] YgtL4S7Hrwo: Extracting video information
[info] Available formats for YgtL4S7Hrwo:
format code  extension  resolution note
249          webm       audio only DASH audio   53k , opus @ 50k, 15.14MiB
250          webm       audio only DASH audio   72k , opus @ 70k, 20.29MiB
171          webm       audio only DASH audio  111k , vorbis@128k, 29.42MiB
140          m4a        audio only DASH audio  130k , m4a_dash container, mp4a.40.2@128k, 38.38MiB
251          webm       audio only DASH audio  130k , opus @160k, 36.94MiB
278          webm       256x144    144p   58k , webm container, vp9, 30fps, video only, 11.01MiB
242          webm       426x240    240p   88k , vp9, 30fps, video only, 12.40MiB
160          mp4        256x144    144p  120k , avc1.4d400c, 30fps, video only, 33.64MiB
243          webm       640x360    360p  153k , vp9, 30fps, video only, 23.48MiB
134          mp4        640x360    360p  230k , avc1.4d401e, 30fps, video only, 28.91MiB
133          mp4        426x240    240p  260k , avc1.4d4015, 30fps, video only, 74.75MiB
244          webm       854x480    480p  289k , vp9, 30fps, video only, 39.31MiB
135          mp4        854x480    480p  488k , avc1.4d401f, 30fps, video only, 56.43MiB
247          webm       1280x720   720p  945k , vp9, 30fps, video only, 102.45MiB
136          mp4        1280x720   720p 1074k , avc1.4d401f, 30fps, video only, 116.72MiB
17           3gp        176x144    small , mp4v.20.3, mp4a.40.2@ 24k
36           3gp        320x180    small , mp4v.20.3, mp4a.40.2
43           webm       640x360    medium , vp8.0, vorbis@128k
18           mp4        640x360    medium , avc1.42001E, mp4a.40.2@ 96k
22           mp4        1280x720   hd720 , avc1.64001F, mp4a.40.2@192k (best)

As you can see, each has an id and defines an extension (container) and info about its video and audio stream. You can download a specific format by using the -f command line otpion. For example , to download the audio-only format with the worst audio quality use C:\Users\serafeim>getvideo YgtL4S7Hrwo -f 249. Notice that there are formats with audio ony and other formats with vide only. To download the worst format possible (resulting in the smallest file size of course ) you can pass the -f worst command line (there’s also a -f best command line which is used by default).

Another thing I’d like to point out here is that you can define an output template using the -o option that will format the name of the output file of your video using the provided options. There are many examples in the docs so I won’t go into any more details here.

Another cool option is the -a that will help you download all videos from a file. For example, if you have a file named videos.txt with the following contsnts:

AjFfsOA7AQI
3dDtACSYVx0
G17E4Muylis

running getvideo -a videos.txt -f worst

will get you all three videos in their worst quality. If you don’t want to create files then you can use something like this:

for %i in (AjFfsOA7AQI 3dDtACSYVx0 G17E4Muylis) do getvideo %i -f worst

and it will run getvideo for all three files.

Some more options I’d like to recommend using are:

  • --restrict-filenames to avoid strange filenames
  • --ignore-errors to ignore errors when download multiple files (from a playlist or a channel) - this is really useful because if you have a play with missing items youtube-dl will stop downloading the remaining files when it encounters the missing one

If you want to always use these options you may add them to your configuration file (C:\Users\<user name>\youtube-dl.conf) or to the getvideo.bat defined above i.e getvideo.bat will be:

py -3 -m youtube_dl --restrict-filenames --ignore-errors %*

Extracting mp3s

The next step in this trip is to understand how to extract mp3s from videos that are downloaded from youtube. If you’ve payed attention you’d know that by now you can download audio-only formats from youtube - however they are in a format called DASH which most probably is not playable by your car stereo (DASH is specialized for streaming audio through HTTP).

Thus, the proper way to get mp3s is to post-process the downloaded file using ffmpeg to convert it to mp3. This could be done manually (by doing something ffmpeg -i input out.mp3 — ffmpeg is smart enough to know how to convert per extension) but thankfully youtube-dl offers the -x (and friend) parameters to make this automatic. Using -x tells youtube-dl to extract the audio from the video (notice that youtube-dl is smart enough to download one of the audio-only formats so you don’t have ). Using -x alone may result in a different audio format (for example .ogg) so to force conversion to mp3 you should also add the --audio-format mp3 parameter. Thus, to download an mp3 you can use the following command line (continuing from the previous examples):

py -3 -m youtube_dl --restrict-filenames --ignore-errors -x --audio-format mp3  AjFfsOA7AQI

or even better, create a getmp3.bat batch file that will be used to retrieve an mp3:

py -3 -m youtube_dl --restrict-filenames --ignore-errors -x --audio-format mp3 %1

Please notice that also in this case youtube-dl is smart enough to download an audio-only format thus you won’t need to select it by hand using -f to save bandwith.

Splitting the mp3 file to parts

Some people would like to split their large mp3 files to same-length segments. Of course it would be better for the file to be split by silence to individual songs (if the file contains songs) but these methods usually don’t work that good so I prefer the same length segments. To do that using ffmpeg you just need to add the following parameters:

ffmpeg -i input.mp3 -segment_time 180 -f segment out.%03d.mp3"

The segment time is in seconds (so each segment will be 3 minutes) while the output files will have a name like out.001.mp3, out.002.mp3 etc.

What if you’d like to make the segmentation automatic? For this, I recommend writing a batch file with two commands - one to download the mp3 and a second one to call ffmpeg to segment the file. Notice that you could use the --postprocessor-args ARGS command line parameter to pass the required arguments to youtube-dl so it will be done in one command however I’d like to have a little more control thus I prefer two commands (if you decide to use --postprocessor-args ARGS keep in mind that args must be inside double quotes “”).

Since we are going to use two commands, we need to feed the output file of youtube-dl to ffmpeg and specify a name for the ffmpeg output file-segments. The easiest way to do that is to just pass two parameters to the batch file - one for the video to download and one for its name. Copy the following to a file named getmp3seg.bat:

py -3 -m youtube_dl %1 -x --audio-format mp3 --audio-quality 128k -o %2.%%(ext)s"
ffmpeg -i %2.mp3 -segment_time 180 -f segment %2.%%03d.mp3
del %2.mp3

You can then call it like this: getmp3seg AjFfsOA7AQI test. The first line will download and covert the video to mp3 and put it in a file named test.mp3 (the %2 is the test, the %% is used to escape the % and the %(ext)s is the extensions - this is needed if you use something like -o %2.mp3 youtube-dl will be confused when trying to convert the file to mp3 and will not work). The 2nd line will segment the file to 180 second seconds (notice that here also we need to escape %) and the third line will delete the original mp3. This leaves us with the following 4 files (the video was around 10 minutes): test.000.mp3, test.001.mp3, test.002.mp3, test.003.mp3.

One final thing I’d like to present here is a (more complex) script that you can use to download a video and segmentize it only if it is more than 360 seconds. For this, we will also use the mp3info util which can be downloaded directly from the homepage and copied to the path. So copy the following to a script named getmp3seg2.bat:

@echo off

IF "%2"=="" GOTO HAVE_1

py -3 -m youtube_dl %1 -x --audio-format mp3 -o %2.%%(ext)s"

FOR /f %%i IN ('mp3info -p "%%S" %2.mp3') DO SET koko=%%i

IF %koko%  GTR 360 (
        ECHO greater than or equal to 360
        ffmpeg -i %2.mp3 -segment_time 180 -f segment %2.%%03d.mp3
        del %2.mp3
)  else (
   ECHO less than 360
)

GOTO :eof

:HAVE_1
ECHO Please call this file with video id and title

This is a little more complex - I’ll explain it quickly: @echo off is used to suppress non needed output. The IF following makes sure that you have two parameters. The next line downloads the file and converts it to mp3. The FOR loop is a little strange but it’s result will be to retrieve the output of mp3info -o "%S" title.mp3 (which is the duration in seconds of that mp3) and assign it to the koko variable. The next IF checks if koko is greater than (GTR) 360 seconds and if yes will run the conversion code we discussed before - else it will just output that it is less than 360 seconds.

Finally, there’s a GOTO: eof line to skip printing the error message when the batch is called with less than two parameters.

Using youtube-dl from python

Integrating with youtube-dl from python is easy. Of course, you could just go on and directly call the command line however you can have more control. The most important class is youtube_dl.YoutubeDL.YoutubeDL. You instantiate an object of this class class passing the parameters you’d like and call its download() instance method passing a list of urls. Here’s a small script that downloads the input video ids:

import sys
from youtube_dl import YoutubeDL

if __name__ == '__main__':
    if len(sys.argv) > 1:
        ydl_opts = {}
        ydl = YoutubeDL(ydl_opts)
        ydl.download(sys.argv[1:])
    else:
        print("Enter list of urls to download")
        exit(0)

Save it in a file named getvideo.py and run it like py -3 getvideo.py AjFfsOA7AQI 3dDtACSYVx0 G17E4Muylis to download all three videos!

Fixing your unicode names

The last thing I’d like to talk about concerns people that want to download videos with Unicode characters in their titles (for example Greek).

Let’s suppose that you want to download the file vFVNOaUPRow which is piano music from a well-know greek composer. If you get it without parameters (for example py -3 -m youtube_dl -x --audio-format mp3 vFVNOaUPRow) you’ll get the following output file: Ο Μάνος Χατζιδάκις. παίζει 11 κομμάτια στο πιάνο-vFVNOaUPRow.f247.mp3 (notice the greek characters) while, if you add the --restrict-filenames I mentioned before you’ll get _11-vFVNOaUPRow.f247.mp3 (notice that the greek characters have been removed since they are not safe).

So if you use the --restrict-filenames parameter you’ll get an output that contains only the video id (and any safe characters it may find) while if you don’t use it you’ll get the normal title of the video. However, most stereos do not display unicode characters properly so if I get this file to my car I’ll see garbage and I won’t be able to identify it — I will be able to listen it but not see its name!

To fix that, I propose transliterating the unicode characters using the unidecode library. Just install it using pip. Then you can the following script to rename all mp3 files in a directory to using english characters only:

import os, unidecode

if __name__ == '__main__':
    for file in os.listdir('.'):
        if file.endswith('mp3'):
            print("Renaming {0} to {1}".format(file, unidecode.unidecode(file)))
            os.rename(file, unidecode.unidecode(file))

Copy this to a file named transliterate.py and run it in a directory containing mp3 files (py -3 transliterate.py) to rename them to non-unicode characters.

Authentication for django-rest-framework with django-rest-auth

Introduction

Most of the times I need authentication with any REST APIs defined through django-rest-framework I will use SessionAuthentication method. This method uses the session cookie (which is set through the normal Django login and logout views) to check out if there’s an authenticated user and get his username. This method works only in the same session (browser window) as the one that actually did the login but this should be enough for most cases.

However, sometimes instead of using the normal Django login/logout views, you’ll want to authentication through REST end-points, for example for using them with SPAs (where you don’t want to use the traditional views for authentication but through REST end-points) or because you have implemented a mobile (or desktop) application that needs to authenticate with your script.

There are various ways this could be done but one of the simplest is using django-rest-auth. This project adds a number of REST end-points to your project that can be used for user login and registration (and even social login when combined with django-allauth). In the following I am going to write a simple tutorial on how to actually use django-rest-auth to authenticate with django-rest-framework using the provided REST end points and how to call a REST API as an authenticated user.

Before continuing with the tutorial, let’s take a look at what we’ll build here:

Our project

This is a single html page (styled with spectre.css) that checks if the user is logged in and either displays the login or logout button (using javascript). When you click the login you’ll get a modal in which you can enter your credentials which will be submitted through REST to the django-rest-auth endpoint and depending on the response will set a javascript variable (and a corresponding session/local storage key). Then you can use the “Test auth” button that works only on authenticated users and returns their username. Finally, notice that after you log out the “test auth” button returns a 403 access denied.

If you want to play with this project yourself, you can clone it here https://github.com/spapas/rest_authenticate. Just create a venv, install requirements, create a superuser and you should be good to go!

Some theory

After you log in with Django, your authentication information is saved to the “session”_. The session is a bucket of information that the Django application saves about your visit — to distinguish between different visitors a cookie with a unique value named sessionid will be used. So, your web browser will send this cookie with each page request thus allowing Django to know which bucket of information is yours (and if you’ve authenticated know who are you). This is not a Django related concept but a general one (supported by most if not all HTTP frameworks) and is used to add state to an otherwise stateless medium (HTTP).

Since the sessionid cookie is sent not only with traditional but also with Ajax request it can be used to authenticate REST requests after you’ve logged in. This is what is used by default in django-rest-framework and as I said in the introduction it is a very good solution for most use cases: You login to django and you can go ahead and call the REST API through Ajax; the sessionid cookie will be sent along with the request and you’ll be authenticated.

Now, although the session authentication is nice for using in browsers, you may need to access your API through a desktop or a mobile application where, setting the cookies yourself is not the optimal solution. Also, you may have an SPA that needs to access an API in a different domain; using using cookies for this is not easy - if possible at all.

For such cases, django-rest-framework offers a different authentication method called TokenAuthentication_. Using this method, each user of the Django application is correlated with a random string (Token) which is passed along with the request at its header thus the Django app can authenticate the user using this token! One thing that may seem strange is that since both the session cookie and a token are set through HTTP Headers why all the fuss about tokens? Why not just use the session cookie and be done with it. Well, there are various reasons - here’s a rather extensive article explaining some. Some of the reasons are that a token can be valid forever while the session is something ephemeral - beyond authorization information, sessions may keep various other data for a web application and are expired after some time to save space. Also, since tokens are used for exactly this (authentication) they are much easier to use and reason about. Finally, as I’ve already explained, sharing cookies by multiple sites is not something you’d like to do.

Installation & configuration

To install django-rest-auth just follow the instructions here i.e just add 'rest_framework', 'rest_framework.authtoken' and 'rest_auth' to your INSTALLED_APPS in settings.py and run migrate.

Since I won’t be adding any other apps to this project (no models are actually needed), I’ve added two directories static and templates to put static files and templates there. This is configured by adding the 'DIRS' attribte to TEMPLATES, like this:

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [
            os.path.join(BASE_DIR, 'templates'),
        ],
        // ...

and adding the STATICFILES_DIRS setting:

STATICFILES_DIRS = [
    os.path.join(BASE_DIR, "static"),
]

The remaining setting are the default as were created by django-admin startproject.

I have included the the following urls to urls.py:

urlpatterns = [
    path('admin/', admin.site.urls),
    path('test_auth/', TestAuthView.as_view(), name='test_auth', ),
    path('rest-auth/logout/', LogoutViewEx.as_view(), name='rest_logout', ),
    path('rest-auth/login/', LoginView.as_view(), name='rest_login', ),
    path('', HomeTemplateView.as_view(), name='home', ),
] + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)

These are: The django-admin, a test_auth view (that works only for authenticated users and returns their username), a view (LogoutViewEx) that overrides the rest-auth REST logout-view (I’ll explain why this is needed in a minute), the rest-auth REST login-view, the home template view (which is the only view implemented) and finally a mapping of your static files to the STATIC_URL.

The views

There are three views in this application - the HomeTemplateView, the TestAuthView and the LogoutViewEx view that overrides the normal LogoutView of django-rest-auth. The first one is a simple TemplateView that just displays an html page and loads the client side code - we’ll talk about it later in the front-side section.

The TestAuthView is implemented like this:

class TestAuthView(APIView):
    authentication_classes = (authentication.TokenAuthentication,)
    permission_classes = (permissions.IsAuthenticated,)

    def get(self, request, format=None):
        return Response("Hello {0}!".format(request.user))

This is very simple however I’d like to make a few comments about the above. First of all you see that I’ve defined authentication_classes and permission_classes. These options define

  • which method will be used for authenticating access to the REST view i.e finding out if the user requesting access has logged in and if yes what’s his username (in our case the TokenAuthentication will be used)
  • if the user is authorized (has permission) to call this REST view (in our case only authenticated users will be allowed)

The authentication and permission clases can be set globally in your settings.py using REST_FRAMEWORK['DEFAULT_AUTHENTICATION_CLASSES'] and REST_FRAMEWORK['DEFAULT_PERMISSION_CLASSES'] or defined per-class like this. If I wanted to have the same authentication and permission classes defined in my settings.py so I wouldn’t need to set these options per-class I’d add the following to my settings.py:

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.TokenAuthentication',
    ),
    'DEFAULT_PERMISSION_CLASSES': (
        'rest_framework.permissions.IsAuthenticated',
    ),
}

Finally, keep in mind that you haven’t defined these in your views or your settings, they will have the following default values:

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.SessionAuthentication',
        'rest_framework.authentication.BasicAuthentication'
    ),
    'DEFAULT_PERMISSION_CLASSES': (
        'rest_framework.permissions.AllowAny',
    ),
}

The above mean that if you don’t define authentication and permission classes anywhere then the REST views will use either session authentication (i.e the user has logged in normally using the Django login views as explained before) or basic authentication (the request provides the credentials in the header using traditional HTTP Basic authentication) and also that all users (logged in or not) will be allowed to call all APIs (this is probably not something you want).

The TokenAuthentication that we are using instead means that for every user there must be a valid token which will be provided for each request he does. The tokens are normal object instances of rest_framework.authtoken.models.Token and you can take a look at them (or even add one) through the Django admin (auth token - tokens). You can also even do whatever you normally would do to an object instance, for example:

>>> [ (x.user, x.key) for x in Token.objects.all()]
[(<User: root>, 'db4dcc1b9d00d1af74fb3cb41e1f9e673208485b')]

To authenticate with a token (using TokenAuthentication), you must add an extra header to your request with the format Authorization: Token token for example in the previous case root would add Authorization: Token db4dcc1b9d00d1af74fb3cb41e1f9e673208485b. To do this you’ll need something client-side code which we’ll see in the next section.

To do it with curl you can just do something like this:

curl http://127.0.0.1:8000/test_auth/ -H "Authorization:Token db4dcc1b9d00d1af74fb3cb41e1f9e673208485b"

Try it with a valid and invalid token and without providing a token at all and see the response each time.

So, django-rest-framework provides the model (Token) and the mechanism (add the extra Authentication header) for authentication with Tokens. What it does not provide is a simple way to create/remove tokens for users: This is where django-rest-auth comes to the rescue! Its login and logout REST views will automatically create (and delete) tokens for the users that are logging in. They will also authenticate the user normally (using sessions) - this means that if a user logs in using the login REST endpoint he’ll then be logged in normally to the site and be able to access non-REST parts of the site (for example the django-admin). Also, if the user logs in through the django-rest-auth REST end point and if you have are using SessionAuthentication to one of your views then he’ll be able to authenticate to these views without the need to pass the token (can you understand why?).

Finally, let’s take a look at the LogoutViewEx:

class LogoutViewEx(LogoutView):
    authentication_classes = (authentication.TokenAuthentication,)

This class only defines the authentication_classes attribute. Is this really needed? Well, it depends on you project. If you take a look at the source code of LogoutView (https://github.com/Tivix/django-rest-auth/blob/master/rest_auth/views.py#L99) you’ll see that it does not define authentication_classes. This, as we’ve already discussed, means that it will fall-back to whatever you have defined in the settings (or the defaults of django-rest-framework). So, if you haven’t defined anything in the settings then you’ll get the by default the SessionAuthentication and BasicAuthentication methods (hint: not the TokenAuthentication). This means that you won’t be able to logout when you pass the token (but will be able to logout from the web-app after you login - why?). So to make everything crystal and be able to reason better about the behavior I specifically define the LogoutViewEx to use the TokenAuthentication - that’s what you’d use if you developed a mobile or desktop app anyway.

The client side scripts

I’ve included all client-side code to a home.html template that is loaded from the HomeTemplateView. The client-side code has been implemented only with jQuery because I think this is the library that most people are familiar with - and is really easy to be understood even if you are not familiar with it. It more or less consists of four sections in html:

  • A user-is-logged-in section that displays the username and the logout button
  • A user-is-not-logged-in section that displays a message and the login button
  • A test-auth section that displays a button for calling the TestAuthView defined previously and outputs its response
  • The login modal

Here’s the html (using spectre.css for styling):

<div class="container grid-lg">
    <h2>Test</h2>
    <div class="columns" id="non-logged-in">
        <div class='column col-3'>
            You have to log-in!
        </div>
        <div class='column col-3'>
            <button class="btn btn-primary"  id='loginButton'>Login</button>
        </div>
    </div>
    <div class="columns" id="logged-in">
        <div class='column col-3'>
            Welcome <span id='span-username'></span>!
        </div>
        <div class='column col-3'>
            <button class="btn btn-primary"  id='logoutButton'>Logout</button>
        </div>
    </div>
    <hr />
    <div class="columns" id="test">
        <div class='column col-3'>
            <button class="btn btn-primary"  id='testAuthButton'>Test auth</button>
        </div>
        <div class='column col-9'>
            <div id='test-auth-response' ></div>
        </div>
    </div>
</div>

<div class="modal" id="login-modal">
    <a href="#close" class="modal-overlay close-modal" aria-label="Close"></a>
    <div class="modal-container">
        <div class="modal-header">
            <a href="#close" class="btn btn-clear float-right close-modal" aria-label="Close"></a>
            <div class="modal-title h5">Please login</div>
        </div>
        <div class="modal-body">
            <div class="content">
                <form>
                    {% csrf_token %}
                    <div class="form-group">
                        <label class="form-label" for="input-username">Username</label>
                        <input class="form-input" type="text" id="input-username" placeholder="Name">
                    </div>
                    <div class="form-group">
                        <label class="form-label" for="input-password">Password</label>
                        <input class="form-input" type="password" id="input-password" placeholder="Password">
                    </div>
                    <div class="form-group">
                        <label class="form-checkbox" for="input-local-storage">
                            <input type="checkbox" id="input-local-storage" /> <i class="form-icon"></i>  Use local storage (remember me)
                        </label>
                    </div>
                </form>
                <div class='label label-error mt-1 d-invisible' id='modal-error'>
                    Unable to login!
                </div>
            </div>
        </div>
        <div class="modal-footer">

            <button class="btn btn-primary" id='loginOkButton' >Ok</button>
            <a href="#close" class="btn close-modal" >Close</a>
        </div>
    </div>
</div>

The html is very simple and I don’t think I need to explain much - notice that the #logged-in and #non-logged-in sections are mutually exclusive (I use $.show() and $.hide() to show and hide them) but the #test section is always displayed so you’ll be able to call the test REST API when you are and are not authenticated. For the modal to be displayed you need to add an active class to its #modal container.

For the javascript, let’s take a look at some initialization stuff:

var g_urls = {
    'login': '{% url "rest_login" %}',
    'logout': '{% url "rest_logout" %}',
    'test_auth': '{% url "test_auth" %}',
};
var g_auth = localStorage.getItem("auth");
if(g_auth == null) {
    g_auth = sessionStorage.getItem("auth");
}

if(g_auth) {
    try {
        g_auth = JSON.parse(g_auth);
    } catch(error) {
        g_auth = null;
    }
}

var getCookie = function(name) {
    var cookieValue = null;
    if (document.cookie && document.cookie !== '') {
        var cookies = document.cookie.split(';');
        for (var i = 0; i < cookies.length; i++) {
            var cookie = jQuery.trim(cookies[i]);
            // Does this cookie string begin with the name we want?
            if (cookie.substring(0, name.length + 1) === (name + '=')) {
                cookieValue = decodeURIComponent(cookie.substring(name.length + 1));
                break;
            }
        }
    }
    return cookieValue;
};
var g_csrftoken = getCookie('csrftoken');

var initLogin = function() {
    if(g_auth) {
        $('#non-logged-in').hide();
        $('#logged-in').show();
        $('#span-username').html(g_auth.username);
        if(g_auth.remember_me) {
            localStorage.setItem("auth", JSON.stringify(g_auth));
        } else {
            sessionStorage.setItem("auth", JSON.stringify(g_auth));
        }
    } else {
        $('#non-logged-in').show();
        $('#logged-in').hide();
        $('#span-username').html('');
        localStorage.removeItem("auth");
        sessionStorage.removeItem("auth");
    }
    $('#test-auth-response').html("");
};

First of all, I define a g_urls window/global object that will keep the required REST URLS (login/logout and test auth). These are retrieved from Django using the {% url %} template tag and are not hard-coded. After that, I check to see if the user has authenticated before. Notice that because this is client-side code, I need to do that every time the page loads or else the JS won’t be initialized properly! The user login information is stored to an object named g_auth and contains two attributes: username, key (token) and remember_me.

To keep the login information I use either a key named auth to either the localStorage or the sessionStorage. The sessionStorage is used to save info for the current browser tab (not window) while the localStorage saves info for ever (until somebody deletes it). Thus, localStorage can be used for implementing a “remember me” functionality. Notice that instead of using the session/local storage I could instead integrate the user login information with the Django back-end. To do this I’d need to see if the current user has a session login and if yes pass his username and token to Javascript. These values would then be read by the login initialization code. I’m leaving this as an exercise for attentive readers.

Getting the login information from the session probably is a better solution for web-apps however I think that using the local or session storage emulate better a more general (and completely stateless) behaviour especially considering that the API may be used for mobible/desktop apps.

In any case, after you’ve initialized the g_auth object you’ll need to read the CSRF cookie. By default Django requires CSRF protection for all POST requests (we do a POST request for login and logout). What happens here is that for pages that may need to do a POST request, Django will set a cookie (CSRF cookie) in its initial response. You’ll need to read that cookie and submit its value along with the rest of your form fields when you do the POST. So the getCookie function is just used to set the g_csrftoken with the value of the CSRF cookie.

The final function we define here (which is called a little later) checks to see if there is login information and hides/displays the correct things in html. It will also set the local or session storage (depending on remember me value).

After that, we have some client side code that is inside the $() function which will be called after the page has completely loaded:

$(function () {
    initLogin();

    $('#loginButton').click(function() {
        $('#login-modal').addClass('active');
    });

    $('.close-modal').click(function() {
        $('#login-modal').removeClass('active');
    });

    $('#testAuthButton').click(function() {
        $.ajax({
            url: g_urls.test_auth,
            method: "GET",
            beforeSend: function(request) {
                if(g_auth) {
                    request.setRequestHeader("Authorization", "Token " + g_auth.key);
                }
            }
        }).done(function(data) {
            $('#test-auth-response').html("<span class='label label-success'>Ok! Response: " + data);
        }).fail(function(data) {
            $('#test-auth-response').html("<span class='label label-error'>Fail! Response: " + data.responseText + " (status: " + data.status+")</span>");
        });
    });


    // continuing below ...

The first thing happening here is to call the initLogin function to properly intiialize the page and then we add a couple of handlers to the click buttons of the #loginButton (which just displays the modal by adding the active class ), .close-modal class (there are multiple ways to close the modal thus I use a class which just removes that active class) and finally to the #testAuthButton. This button will do a GET request to the g_urls.test_auth we defined before. The important thing to notice here is that we add a beforeSend attribute to the $.ajax request which, if g_auth is defined adds an Authorization header with the token in the form that django-rest-framework TokenAuthentication expects and as we’ve already discussed above:

beforeSend: function(request) {
    if(g_auth) {
        request.setRequestHeader("Authorization", "Token " + g_auth.key);
    }
}

If this ajax call returns ok (done part) we just add the Ok to a green label else if there’s an error (fail part) we add the response text and status to a red label. You can try clicking the button and you see that only if you’ve logged in you will succeed in this call.

Let’s now take a look at the #loginOkbutton click handler (inside the modal):

$('#loginOkButton').click(function() {
    var username = $('#input-username').val();
    var password = $('#input-password').val();
    var remember_me = $('#input-local-storage').prop('checked');
    if(username && password) {
        console.log("Will try to login with ", username, password);
        $('#modal-error').addClass('d-invisible');
        $.ajax({
            url: g_urls.login,
            method: "POST",
            data: {
                username: username,
                password: password,
                csrfmiddlewaretoken: g_csrftoken
            }
        }).done(function(data) {
            console.log("DONE: ", username, data.key);
            g_auth = {
                username: username,
                key: data.key,
                remember_me: remember_me
            };
            $('#login-modal').removeClass('active');
            initLogin();
            // CAREFUL! csrf token is rotated after login: https://docs.djangoproject.com/en/1.7/releases/1.5.2/#bugfixes
            g_csrftoken = getCookie('csrftoken');
        }).fail(function(data) {
            console.log("FAIL", data);
            $('#modal-error').removeClass('d-invisible');
        });
    } else {
        $('#modal-error').removeClass('d-invisible');
    }
});

All three user inputs (username, password, remember_me) are read from the form and if both username and password have been defined an Ajax request will be done to the g_urls.login url. We pass username, password and g_csrftoken (as discussed before) as the request data. Now, if there’s an error (fail) I just display a generic message (by removing it’s d-invisible class) while, if the request was Ok I retrieve the key (token) from the response, initialize the g_auth object with the username, key and remember_me values and call initLogin to show the correct divs and save to the session/local storage.

It is important to keep in mind that with the line g_csrftoken = getCookie('csrftoken') we re-read the CSRF cookie. This is needed because, as you can see in the mentioned link in the comment, after Django logs in, the csrf cookie value is rotated for security reasons so it must be re-read here (or else the logout that is also a POST request will not work).

Finally, here’s the code for logout (still inside the $(function () {):

    $('#logoutButton').click(function() {
        console.log("Trying to logout");
        $.ajax({
            url: g_urls.logout,
            method: "POST",
            beforeSend: function(request) {
                request.setRequestHeader("Authorization", "Token " + g_auth.key);
            },
            data: {
                csrfmiddlewaretoken: g_csrftoken
            }
        }).done(function(data) {
            console.log("DONE: ", data);
            g_auth = null;
            initLogin();
        }).fail(function(data) {
            console.log("FAIL: ", data);
        });
    });

}); // End of $(function () {

The code here is very simple - just do a POST to the g_urls.logout and if everything is ok delete the g_auth values and call initLogin() to show the correct divs and remove the auth key from local/session storage. Notice that when you POST to the logout REST end-point, you need to also add the Authorization header with the token or else (since we’ve defined only TokenAuthentication for the authentication_classes for the LogoutViewEx class) there won’t be any way to correlate the request with the user and log him out!

Conclusion

Using the info presented on this article you should be able to properly login and logout to Django using REST and also call REST end-points as an authenticated used. I recommend using the curl utility to try to call the rest end point with various parameters to see the response. Also, you change the LogoutViewEx with the default django-rest-auth LogoutView and then try logging out through the web-app and through curl and see what happens when you try to access the test-auth end-point.

Finally, the above project can be easily modified to use SessionAuthentication instead of TokenAuthentication (so you won’t need django-rest-auth at all) - I’m leaving it as an exercise to the reader.