What will it do

The solution is more or less a self contained Elixir module named QueryFilterEx that can be used to declaratively filter your queries. To use that you’ll need to declare your filters using a simple array of maps. The filters should then be added in your form using a different input for each filter; then your queryset will be filtered with all the values you’ve added to the inputs using AND.

The module has a very simple API consisting of three functions:

• get_changeset_from_params(params, filters): Pass it the GET request parameters you got from your form and the declared filters array to return you a proper changeset (which you can then use to build your form in your html)
• make_filter_changeset(filters, params): This function actually generates the changeset using the filters and a Map of filter_name: value pairs (it is actually used by get_changeset_from_params)
• filter(query, changeset, filters): Filter the query using the previously created changeset and the declared filters array

You can find a sample of the technique presented in this article in my PHXCRD repository: https://github.com/spapas/phxcrd for example in the user_controller or authority_controller.

Preparing the query

In order to use the QueryFilterEx module you’ll need to properly “prepare” your Ecto query. By preparing I don’t mean a big deal just the fact that you’ll need to name all your relations (or at least name all the relations you’re going to use for filtering). This is very simple to do, for example for the following query:

from(a in Authority,
  join: ak in AuthorityKind,
  on: [id: a.authority_kind_id],
  preload: [authority_kind: ak]
)

you can name the relations by adding two as: atoms like this:

from(a in Authority, as: :authority,
  join: ak in AuthorityKind, as: :authority_kind,
  on: [id: a.authority_kind_id],
  preload: [authority_kind: ak]
)

So after each join: you’ll add a name for your joined relation (and also add a name for your initial relation). Please notice that you can use any name you want for these (not related to the schema names).

Declaring the filters

To declare the filters you’ll just add an array of simple Elixir maps. Each map must have the following fields:

• :name This is the name of the specific filter; it is mainly used in conjunction with the queryset and the form fields to set initial values etc
• :type This is the type of the specific filter; it should be a proper Ecto type like :string, :date, :integer etc. This is needed to properly cast the values and catch errors
• :binding This is the name of the relation this filter concerns which you defined in your query using :as (discussed in previous section)
• :field_name This is the actual name of the field you want to filter on
• :method How to filter on this field; I’ve defined a couple of methods I needed but you can implement anything you want

The methods I’ve implemented are the following:

• :eq Equality
• :ilike Field value starts with the input - ignore case
• :icontains Field value contains the input - ignore case
• :year Field is a date or datetime an its year is the same as the value
• :date Field is a datetime and its date part is equal to the value

Anything else will just be compared using = (same as :eq).

Integrating it with a controller

As an example let’s see how QueryFilterEx is integrated it with the phxcrd user_controller. The query I’d like to filter on is the following (see that everything I’ll need is named using :as:

from(u in User, as: :user,
  left_join: a in Authority, as: :authority,
  on: a.id == u.authority_id,
  left_join: up in UserPermission,
  on: up.user_id == u.id,
  left_join: p in Permission, as: :permission,
  on: up.permission_id == p.id,
  preload: [authority: a, permissions: p]
)

To declare the filters I like to create a module attribute ending with filters, something like @user_filters for example. Here’s the filters I’m going to use for user_controller:

@user_filters [
  %{name: :username, type: :string, binding: :user, field_name: :username, method: :ilike},
  %{name: :authority_name, type: :string, binding: :authority, field_name: :name, method: :icontains},
  %{name: :permission_name, type: :string, binding: :permission, field_name: :name, method: :ilike},
  %{name: :last_login_date, type: :date, binding: :user, field_name: :last_login, method: :date}
]

So it will check if the user.username and permission.name start with the passed value, authority.name contains the passed value and if the user.login_date (which is a datetime) is the same as the passed date value.

Finally, here’s the full code of the index controller:

def index(conn, params) do
  changeset = QueryFilterEx.get_changeset_from_params(params, @user_filters)

  users =
    from(u in User,
      as: :user,
      left_join: a in Authority, as: :authority,
      on: a.id == u.authority_id,

      left_join: up in UserPermission,
      on: up.user_id == u.id,
      left_join: p in Permission, as: :permission,
      on: up.permission_id == p.id,
      preload: [authority: a, permissions: p]
    )
    |> QueryFilterEx.filter(changeset, @user_filters)
    |> Repo.all()

  render(conn, "index.html", users: users, changeset: changeset)
end

It is very simple, it just uses the get_changeset_from_params method I discussed before to generate the changeset and then uses it to filter the query. Also please notice that it passes the changeset to the template to be properly rendered in the filter form.

The template

The template for the user index action is the following:

<%= form_for @changeset, AdminRoutes.user_path(@conn, :index), [method: :get, class: "filter-form", as: :filter],  fn f -> %>
  <%= label f, :username, gettext "Username" %>
  <%= text_input f, :username  %>

  <%= label f, :authority_name, gettext "Authority name" %>
  <%= text_input f, :authority_name  %>

  <%= label f, :permission_name, gettext "Permission name" %>
  <%= text_input f, :permission_name  %>

  <%= label f, :last_login_date, gettext "Last login date" %>
  <%= text_input f, :last_login_date  %>
  <%= error_tag f, :last_login_date %>

  <%= submit gettext("Filter"), class: "ml-5" %>
  <%= link gettext("Reset"), to: AdminRoutes.user_path(@conn, :index), class: "button button-outline ml-2" %>
<% end %>
<%= for user <- @users do %>
<!-- display the user info -->
<% end %>

Notice that it gets the @changeset and uses it to properly fill the initial values and display error messages. For this case I’ve only added an error_tag for the :last_login_date field, the others since are strings do not really need it since they will accept all values.

Also, the form method form must be :get since we only filter (not change anything) and I’ve passed the as: :filter option to the form_for to collect the parameters under the filter server side parameter (this can be anything you want and can be optionally be passed to QueryFilterEx.get_changeset_from_params to know which parameter the filters are collected on).

How does this work?

In this section I’ll try to explain exactly how the QueryFilterEx module works. Before continuing I want to thank the people at the Elixir forum and #elixir-lang Freenode IRC chat that helped me with understanding how to be able to create dynamic bindings.

So I’ll split this explanation in two parts: Explain QueryFilterEx.get_changeset_from_params and make_filter_changeset (easy) and then explain QueryFilterEx.filter (more difficult).

def get_changeset_from_params(params, filters, filter_name \\ "filter") do
  case params do
    %{^filter_name => filter_params} ->
      filters |> make_filter_changeset(filter_params)

    _ ->
      filters |> make_filter_changeset(%{})
  end
end

defp make_filter_keys(filters) do
  filters |> Enum.map(& &1.name)
end

defp make_filter_types(filters) do
  filters |> Enum.map(&{&1.name, &1.type}) |> Map.new()
end

def make_filter_changeset(filters, params) do
  data = %{}
  types = filters |> make_filter_types

  {data, types}
  |> Ecto.Changeset.cast(params, filters |> make_filter_keys)
  |> Map.merge(%{action: :insert})
end

def filter(query, changeset, filters) do
  changes = Map.fetch!(changeset, :changes)
  filters |> Enum.reduce(query, creat_where_clauses_reducer(changes))
end

defp creat_where_clauses_reducer(changes) do
  fn %{name: name, field_name: field_name, binding: binding, method: method}, acc ->
    case Map.fetch(changes, name) do
      {:ok, value} ->
        acc |> creat_where_clause(field_name, binding,  method, value)

      _ ->
        acc
    end
  end
end

defp creat_where_clause(acc, field_name, binding,  method, value) do
  case method do
    :eq -> acc |> where(
      [{^binding, t}],
      field(t, ^field_name) == ^value
    )
    :ilike -> acc |> where(
      [{^binding, t}],
      ilike(field(t, ^field_name), ^("#{value}%") )
    )
    :icontains -> acc |> where(
      [{^binding, t}],
      ilike(field(t, ^field_name), ^("%#{value}%") )
    )
    :year -> acc  |> where(
      [{^binding, t}],
      fragment("extract (year from ?) = ?", field(t, ^field_name), ^value)
    )
    :date -> acc  |> where(
      [{^binding, t}],
      fragment("? >= cast(? as date) and ? < (cast(? as date) + '1 day'::interval"), field(t, ^field_name), ^value, field(t, ^field_name), ^value)
    )
    _ -> acc |> where(
      [{^binding, t}],
      field(t, ^field_name) == ^value
    )

  end
end

acc |> where(
  [{^binding, t}],
  field(t, ^field_name) == ^value
)

from(u in User) |> where([u], u.name == "root") |> Repo.all()

from(u in User, as: :user) |> where([user: u], u.name == "root") |> Repo.all()

from(u in User, as: :user) |> where([{:user, u}], u.name == "root") |> Repo.all()

binding = :user
from(u in User, as: :user) |> where([{^binding, u}], u.name == "root") |> Repo.all()

binding = :user
field_name = :name
from(u in User, as: :user) |> where([{^binding, u}], field(u, ^field_name) == "root") |> Repo.all()

where(
  [{^binding, t}],
  fragment("? >= cast(? as date) and ? < (cast(? as date) + '1 day'::interval"), field(t, ^field_name), ^value, field(t, ^field_name), ^value)
)

field_name >= cast(value as date) AND field_name < (cast(value as date) + '1 day'::interval)

Conclusion

By using the QueryFilterEx module presented here you can very quickly declare the fields you want to filter on and the method you want to use for each field no matter if these fields are in the same schema or are accessed through joins. You can easily extend the functionality of the module by adding your own methods. The only extra thing you need to do is to just add names to your queries.

The schemas

For this tutorial, we’ll use two schemas: A User and an Authority. Each User belongs to an Authority (thus will have a foreign key to Authority; that’s what we want to set using the ajax select2). Here are the ecto schemas for these entities:

defmodule Phxcrd.Auth.Authority do
  use Ecto.Schema

  import Ecto.Changeset
  alias Phxcrd.Auth.User

  schema "authorities" do
    field :name, :string
    has_many :users, User, on_replace: :nilify
    timestamps()
  end

  @doc false
  def changeset(authority, attrs) do
    authority
    |> cast(attrs, [:name])
    |> validate_required([:name], message: "The field is required")
    |> unique_constraint(:name, message: "The name already exists!")
  end

  use Accessible
end

defmodule Phxcrd.Auth.User do
  use Ecto.Schema

  import Ecto.Changeset
  alias Phxcrd.Auth.Authority

  schema "users" do
    field :email, :string
    field :username, :string
    field :password_hash, :string
    field :password, :string, virtual: true

    belongs_to :authority, Authority

    timestamps()
  end

  @doc false
  def changeset(user, attrs) do
    user
    |> cast(attrs, [:username, :email, :authority_id])
    |> validate_required([:username, :email ])
  end

  use Accessible
end

Notice that both these entities are contained in the Auth context and were created using mix phx.gen.html; I won’t include the migrations here.

The search API

Let’s now take a look at the search api for Authority. I’ve added an ApiController which contains the following function:

def search_authorities(conn, params) do
    q = params["q"]

    authorities =
      from(a in Authority,
        where: ilike(a.name, ^"%#{q}%")
      )
      |> limit(20)
      |> Repo.all()

    render(conn, "authorities.json", authorities: authorities)
end

Notice that this retrieves a q parameter and makes an ilike query to Authority.name. It then passes the results to the view for rendering. Here’s the corresponding function for ApiView:

def render("authorities.json", %{authorities: authorities}) do
    %{results: Enum.map(authorities, &authority_json/1)}
  end

  def authority_json(a) do
    %{
      id: a.id,
      text: a.name
    }
end

Notice that select2 wants its results in a JSON struct with the following form {results: [{id: 1, name: "Authority 1"}]}.

To add this controller action to my routes I’ve added this to router.ex:

scope "/api", PhxcrdWeb do
    pipe_through :api

    get "/search_authorities", ApiController, :search_authorities
end

Thus if you visit http://127.0.0.1/search_authorities?q=A you should retrieve authorities containing A in their name.

The controller

Concenring the UserController I’ve added the following methods to it for creating and updating users:

def new(conn, _params) do
  changeset = Auth.change_user(%User{})
  render(conn, "new.html", changeset: changeset)
end

def create(conn, %{"user" => user_params}) do
  case Auth.create_user(user_params) do
    {:ok, user} ->
      conn
      |> put_flash(:info, "#{user.name} created!")
      |> redirect(to: Routes.user_path(conn, :show, user))

    {:error, changeset} ->
      render(conn, "new.html", changeset: changeset)
  end
end

def edit(conn, %{"id" => id}) do
  user = Auth.get_user!(id)
  changeset = Auth.change_user(user)
  render(conn, "edit.html", user: user, changeset: changeset)
end

def update(conn, %{"id" => id, "user" => user_params}) do
  user = Auth.get_user!(id)

  user_params = Map.merge(%{"authority_id" => nil}, user_params)

  case Auth.update_user(user, user_params) do
    {:ok, user} ->
      conn
      |> put_flash(:info, "User updated successfully.")
      |> redirect(to: Routes.user_path(conn, :show, user))

    {:error, %Ecto.Changeset{} = changeset} ->
      render(conn, "edit.html", user: user, changeset: changeset)
  end
end

Most of these are more or less the default things that mix phx.gen.html creates. One thing that may seem a strange here is the user_params = Map.merge(%{"authority_id" => nil}, user_params) line of update. What happens here is that I want to be able to clear the authority of a user (I’ll explain how in the next sections). If I do that then the user_params that is passed to update will not contain an authority_id key thus the authority_id won’t be changed at all (so even though I cleared it, it will keep its previous value after I save it). To fix that I set a default value of nil to authority_id; if the user has actually selected an authority from the form this will be overriden when merging the two maps. So the resulting user_params will always contain an authority_id key, either set to nil or to the selected authority.

Beyond that I wont’ go into detail explaining the above functions, but if something seems strange feel free to ask. I also won’t explain the Auth.* functions; all these are created by phoenix in the context module.

The view

The UserView module contains a simple but very important function:

def get_select_value(changeset, attr) do
  case changeset.changes[attr] do
    nil -> Map.get(changeset.data, attr)
    z -> z
  end
end

This functions gets two parameters: The changeset and the name of the attribute (:authority_id in our case). What it does is to first check if this attribute is contained in the changeset.changes; if yes it will return that value. If it isn’t contained in the changeset.changes then it will return the value of changeset.data for that attribute.

This is a little complex but let’s try to understand its logic: When you start editing a User you want to display the current authority of that instance. However, when you submit an edited user and retrieve an errored form (for example because you forgot to fill the username) you want to display the authority that was submitted in the form. So the changeset.changes contains the changes that were submitted just before while the changeset.data contain the initial value of the struct.

Update 02/07/2019: Please notice that instead of using the get_select_value I presented before you can use the Ecto.Changeset.get_field function that does exactly this! So get_select_value could be defined like this:

def get_select_value(changeset, attr) do
  changeset |> Ecto.Changeset.get_field(attr)
end

The form template

Both the :new and :edit actions include a common form.html.eex template:

<%= form_for @changeset, @action, fn f -> %>
  <%= if @changeset.action do %>
  <div class="alert alert-danger">
    <p><%= gettext("Problems while saving") %></p>
  </div>
  <% end %>
  <div class='row'>
    <div class='column'>
      <%= label f, :username %>
      <%= text_input f, :username %>
      <%= error_tag f, :username %>
    </div>
    <div class='column'>
      <%= label f, :email %>
      <%= text_input f, :email %>
      <%= error_tag f, :email %>
    </div>
  </div>

  <div class='row'>
    <div class='column'>
      <%= label f, :authority %>
      <%= select(f,
        :authority_id, [
          (with sv when not is_nil(sv) <- get_select_value(@changeset, :authority_id),
                                     a <- Phxcrd.Auth.get_authority!(sv), do: {a.name, a.id})
        ],
        style: "width: 100%")
        %>
      <%= error_tag f, :authority_id %>
    </div>

  </div>

  <div>
    <%= submit gettext("Save") %>
  </div>
<% end %>

This is a custom Phoenix form but it has the following addition which is more or less the meat of this article (along with the get_select_value function I explained before):

select(f, :authority_id, [
        (with sv when not is_nil(sv) <- get_select_value(@changeset, :authority_id),
                                   a <- Phxcrd.Auth.get_authority!(sv), do: {a.name, a.id})
      ],
      style: "width: 100%")

So this will create an html select element which will contain a single value (the array in the third parameter of select): The authority of that object or the authority that the user had submitted in the form. For this it uses get_select_value to retrieve the :authority_id and if it’s not nil it passes it to get_authority! to retrieve the actual authority and return a tuple with its name and id.

By default when you create a select element you’ll pass an array of all options in the third parameter, for example:

select(f, :authority_id, Phxcrd.Auth.list_authorities |> Enum.map(&{&1.name, &1.id}))

Of course this beats the purpose of using ajax since all options will be rendered.

The final step is to add the required custom javascript to convert that select to select2-with-ajax:

$(function () {
    $('#user_authority_id').select2({
      allowClear: true,
      placeholder: 'Select authority',
      ajax: {
        url: '<%= Routes.api_path(@conn, :search_authorities) %>',
        dataType: 'json',
        delay: 150,
        minimumInputLength: 2
      }
    });
})

The JS very rather simple; the allowClear option will display an x so that you can clear the selected authority while the ajax url will be that of the :search_authorities.

Conclusion

Although this article may seem a little long, as I’ve already mentioned the most important thing to keep is how to properly set the value that should be displayed in your select2 widget. Beyond that everything is a walk in the park by following the docs.

Introduction

Android offers a nice component named AutoCompleteTextView that can be used to auto-fill a text box from a list of values. In its simplest form, you just create an array adapter passing it a list of objects (that have a proper toString() method). Then you type some characters to the textbox and by default it will filter the results searching in the beginning of the backing object’s toString() result.

However there are times that you don’t want to look at the beginning of the string (because you want to look at the middle of the string) or you don’t want to just to search in toString() method of the object or you want to do some more fancy things in object output. For this you must override the ArrayAdapter and add a custom Filter.

Unfurtunately this isn’t as straightforward as I’d like and I couldn’t find a quick and easy tutorial on how it can be done.

So here goes nothing: In the following I’ll show you a very simple android application that will have the minimum viable custom filtered adapter implementation. You can find the whole project in github: https://github.com/spapas/CustomFilteredAdapeter but I am going to discuss everything here also.

The application

Just create a new project with an empty activity from Android Studio. Use kotlin as the language.

The layout

I’ll keep it as simple as possible:

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout
        xmlns:android="http://schemas.android.com/apk/res/android"
        xmlns:tools="http://schemas.android.com/tools"
        android:orientation="vertical"
        android:layout_width="match_parent" android:layout_height="match_parent"
        tools:context=".MainActivity">
    <TextView
            android:layout_width="match_parent" android:layout_height="wrap_content"
            android:text="Hello World!"
            android:textSize="32sp"
            android:textAlignment="center"/>
    <AutoCompleteTextView
            android:layout_marginTop="32dip"
            android:layout_width="match_parent" android:layout_height="wrap_content"
            android:id="@+id/autoCompleteTextView"/>
</LinearLayout>

You should just care about the AutoCompleteTextView with an id of autoCompleteTextView.

The backing data object

I’ll use a simple PoiDao Kotlin data class for this:

data class PoiDao(
    val id: Int,
    val name: String,
    val city: String,
    val category_name: String
)

I’d like to be able to search to both name, city and category_name of each object. To create a list of the pois to be used to the adapter I can do something like:

val poisArray = listOf(
    PoiDao(1, "Taco Bell", "Athens", "Restaurant"),
    PoiDao(2, "McDonalds", "Athens","Restaurant"),
    PoiDao(3, "KFC", "Piraeus", "Restaurant"),
    PoiDao(4, "Shell", "Lamia","Gas Station"),
    PoiDao(5, "BP", "Thessaloniki", "Gas Station")
)

The custom adapter

This will be an ArrayAdapter<PoiDao> implementing also the Filterable interface:

inner class PoiAdapter(context: Context, @LayoutRes private val layoutResource: Int, private val allPois: List<PoiDao>):
    ArrayAdapter<PoiDao>(context, layoutResource, allPois),
    Filterable {
    private var mPois: List<PoiDao> = allPois

    override fun getCount(): Int {
        return mPois.size
    }

    override fun getItem(p0: Int): PoiDao? {
        return mPois.get(p0)
    }

    override fun getItemId(p0: Int): Long {
        // Or just return p0
        return mPois.get(p0).id.toLong()
    }

    override fun getView(position: Int, convertView: View?, parent: ViewGroup?): View {
        val view: TextView = convertView as TextView? ?: LayoutInflater.from(context).inflate(layoutResource, parent, false) as TextView
        view.text = "${mPois[position].name} ${mPois[position].city} (${mPois[position].category_name})"
        return view
    }

    override fun getFilter(): Filter {
        // See next section
    }
}

You’ll see that we add an instance variable named mPois that gets initialized in the start with allPois (which is the initial list of all pois that is passed to the adapter). The mPois will contain the filtered results. Then, for getCount and getItem we return the corresponding valeus from mPois; the getItemId is used when you have an sqlite backed adapter but I’m including it here for completeness.

The getView will create the specific line for each item in the dropdown. As you’ll see the layout that is passed must have a text child which is set based on some of the attributes of the corresponding poi for each position. Notice that we can use whatever view layout we want for our dropdown result line (this is the layoutResource parameter) but we need to configure it (i.e bind it with the values of the backing object) here properly.

Finally we create a custom instance of the Filter, explained in the next section.

The custom filter

The getFilter creates an object instance of a Filter and returns it:

override fun getFilter(): Filter {
    return object : Filter() {
        override fun publishResults(charSequence: CharSequence?, filterResults: Filter.FilterResults) {
            mPois = filterResults.values as List<PoiDao>
            notifyDataSetChanged()
        }

        override fun performFiltering(charSequence: CharSequence?): Filter.FilterResults {
            val queryString = charSequence?.toString()?.toLowerCase()

            val filterResults = Filter.FilterResults()
            filterResults.values = if (queryString==null || queryString.isEmpty())
                allPois
            else
                allPois.filter {
                    it.name.toLowerCase().contains(queryString) ||
                    it.city.toLowerCase().contains(queryString) ||
                    it.category_name.toLowerCase().contains(queryString)
                }
            return filterResults
        }
    }
}

This object instance overrides two methods of Filter: performFiltering and publishResults. The performFiltering is where the actual filtering is done; it should return a FilterResults object containing a values attribute with the filtered values. In this method we retrieve the charSequence parameter and converit it to lowercase. Then, if this parameter is not empty we filter the corresponding elements of allPois (i.e name, city and category_name in our case) using contains. If the query parameter is empty then we just return all pois. Warning java developers; here the if is used as an expression (i.e its result will be assigned to filterResults.values).

After the performFiltering has finished, the publishResults method is called. This method retrieves the filtered results in its filterResults parameter. Thus it sets mPois of the custom adapter is set to the result of the filter operation and calls notifyDataSetChanged to display the results.

Using the custom adapter

To use the custom adapter you can do something like this in your activity’s onCreate:

override fun onCreate(savedInstanceState: Bundle?) {
    super.onCreate(savedInstanceState)
    setContentView(R.layout.activity_main)

    val poisArray = listOf(
        // See previous sections
    )
    val adapter = PoiAdapter(this, android.R.layout.simple_list_item_1, poisArray)
    autoCompleteTextView.setAdapter(adapter)
    autoCompleteTextView.threshold = 3

    autoCompleteTextView.setOnItemClickListener() { parent, _, position, id ->
        val selectedPoi = parent.adapter.getItem(position) as PoiDao?
        autoCompleteTextView.setText(selectedPoi?.name)
    }
}

We create the PoiAdapter passing it the poisArray and android.R.layout.simple_list_item_1 as the layout. That layout just contains a textview named text. As we’ve already discussed you can pass something more complex here. The thresold defined the number of characters that the user that needs to enter to do the filtering (default is 2).

Please notice that when the user clicks (selects) on an item of the dropdown we set the contents of the textview (or else it will just use the object’s toString() method to set it).

Description of the project

The project is very simple, you can just add a url and it will retrieve its content asynchronously and report its length. For the models, it just has a Task model which is used to provide information about what we want to the asynchronous task to do and retrieve the result:

from django.db import models

class Task(models.Model):
    created_on = models.DateTimeField(auto_now_add=True)
    url = models.CharField(max_length=128)
    url_length = models.PositiveIntegerField(blank=True, null=True)
    job_id = models.CharField(max_length=128, blank=True, null=True)
    result = models.CharField(max_length=128, blank=True, null=True)

It also has a home view that can be used to start new asynchronous tasks by creating a Task object with the url we got and passing it to the asynchronous task:

from django.views.generic.edit import FormView
from .forms import TaskForm
from .tasks import get_url_length
from .models import Task

import time
from django.db import transaction

class TasksHomeFormView(FormView):
    form_class = TaskForm
    template_name = 'tasks_home.html'
    success_url = '/'

    def form_valid(self, form):
        task = Task.objects.create(url=form.cleaned_data['url'])
        get_url_length.delay(task.id)
        return super(TasksHomeFormView, self).form_valid(form)

    def get_context_data(self, **kwargs):
        ctx = super(TasksHomeFormView, self).get_context_data(**kwargs)
        ctx['tasks'] = Task.objects.all().order_by('-created_on')
        return ctx

And finally the asynchronous job itself that retrieves the task from the database, requests its url and saves its length:

import requests
from .models import Task
from rq import get_current_job
from django_rq import job

@job
def get_url_length(task_id):
    jb = get_current_job()
    task = Task.objects.get(
        id=task_id
    )
    response = requests.get(task.url)
    task.url_length = len(response.text)
    task.job_id = jb.get_id()
    task.result = 'OK'
    task.save()

The above should be fairly obvious: The user visits the homepage and enters a url at the input. When he presses submit the view will create a new Task object with the url that the user entered and fire-off the get_url_length asynchronous job passing the task id of the task that was just created. It will then return immediately without waiting for the asynchronous job to complete. The user will need to refresh to see the result of his job; this is the usual behavior with async jobs.

The asynchronous job on the other hand will retrieve the task whose id got as a parameter from the database, do the work it needs to do and update the result when it is finished.

Unfortunately, the above simple setup will probably behave erratically by randomly throwing database related errors!

Cause of the problem

In the previous section I said probably because the erratic behavior is caused by a specific setting of your Django project; the ATOMIC_REQUESTS. This setting can be set on your database connection and if it is TRUE then each request will be atomic. This means that each request will be tied with a database transaction i.e a transaction will be started when your request starts and commited only when your requests finishes; if for some reason your request throws an error then the transaction will be rolled back. An example of this setting is:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
        'ATOMIC_REQUESTS': True,
    }
}

Now, in my opinion, ATOMIC_REQUESTS is a great thing to have because it makes everything much easier. I always set it to True to my projects because I don’t need to actually think about transactions and requests; I know that if there’s a problem in a request the whole transaction will be rolle back and no garbage will be left in the database. If on the other hand for some reason a request does not need to be tied to a transaction I just set it off for this specific transaction (using transaction.non_atomic_requests_). Please notice that by default the ATOMIC_REQUESTS has a False value which means that the database will be in autocommit mode meaning that every command will be executed immediately.

So although the ATOMIC_REQUESTS is great, it is actually the reason that there are problems with asynchronous tasks. Why? Let’s take a closer look at what the form_valid of the view does:

def form_valid(self, form):
    task = Task.objects.create(url=form.cleaned_data['url']) #1
    get_url_length.delay(task.id) #2
    return super(TasksHomeFormView, self).form_valid(form) #3

It creates the task in #1, fires off the asynchronous task in #2 and continues the execution of the view processing in #3. The important thing to understand here is that the transaction will be commited only after #3 is finished. This means that there’s a possibility that the asynchronous task will be started before #3 is finished thus it won’t find the task because the task will not be created yet(!) This is a little counter-intuitive but you must remember that the async task is run by a worker which is a different process than your application server; the worker may be able to start before the transaction is commited.

If you want to actually see the problem every time you can add a small delay between the start of the async task and the form_valid something like this:

def form_valid(self, form):
    task = Task.objects.create(url=form.cleaned_data['url'])
    get_url_length.delay(task.id)
    time.sleep(1)
    return super(TasksHomeFormView, self).form_valid(form)

This will make the view more slow so the asynchronous worker will always have time to start executing the task (and get the not found error). Also notice that if you had ATOMIC_REQUESTS: False the above code would work fine because the task would be created immediately (auto-commited) and the async job would be able to find it.

The solution

So how is this problem solved? Well it’s not that difficult now that you know what’s causing it!

One solution would be to set ATOMIC_REQUESTS to False but that would make all database commands auto-commit so you’ll lose request-transaction-tieing. Another solution would be to set ATOMIC_REQUESTS to True and disable atomic requests for the specific view that starts the asynchronous job using transaction.non_atomic_requests_. This is a viable solution however I don’t like it because I’d lose the comfort of transaction per request for this specific request and I would need to add my own transaction handling.

A third solution is to avoid messing with the database in your view and create the task object in the async job. Any parameters you want to pass to the async job would be passed directly to the async function. This may work fine in some cases but I find it more safe to create the task in the database before starting the async job so that I have better control and error handling. This way even if there’s an error in my worker and for some reason the async job never starts or it breaks before being able to handle the database, I will have the task object in the database because it will have been created in the view.

Is there anything better? Isn’t there a way to start the executing the async job after the transaction of the view is commited? Actually yes, there is! For this, transaction.on_commit comes to the rescue! This function receives a callback that will be called after the transaction is commited! Thus, to properly fix you project, you should change the form_valid method like this:

def form_valid(self, form):
    task = Task.objects.create(url=form.cleaned_data['url'])
    transaction.on_commit(lambda: get_url_length.delay(task.id))
    time.sleep(1)
    return super(TasksHomeFormView, self).form_valid(form)

Notice that I need to use lambda to create a callback function that will call get_url_length.delay(task.id) when the transaction is commited. Now even though I have the delay there the async job will start after the transaction is commited, ie after the view handler is finished (after the 1 second delay).

Conclusion

From the above you should be able to understand why sometimes you have problems when your async jobs use the database. To fix it you have various options but at least for me, the best solution is to start your async jobs after the transaction is commited using transaction.on_commit. Just change each async.job.delay(parameters) call to transaction.on_commit(lambda: async.job.delay(parameters)) and you will be fine!