A starter project

Let’s create a simple wagtail-starter project (this is for windows you should be able to easily follow the same steps in Unix like systems):

C:\progr\py3>mkdir wagtail-starter
C:\progr\py3>cd wagtail-starter
C:\progr\py3\wagtail-starter>py -3 -m venv venv
C:\progr\py3\wagtail-starter>venv\Scripts\activate
(venv) C:\progr\py3\wagtail-starter>pip install wagtail
(venv) C:\progr\py3\wagtail-starter>wagtail.exe start wagtail_starter
(venv) C:\progr\py3\wagtail-starter>cd wagtail_starter
(venv) C:\progr\py3\wagtail-starter\wagtail_starter>python manage.py migrate
(venv) C:\progr\py3\wagtail-starter\wagtail_starter>python manage.py createsuperuser
(venv) C:\progr\py3\wagtail-starter\wagtail_starter>python manage.py runserver

When you’ve finished all the above you should be able to go to http://127.0.0.1/ and see your homepage and then visit http://127.0.0.1/admin/ and login with your superuser. What we’d like to do is add a “Latest changes” link in the “Reports” admin section like this:

Implementing the latest-changes menu

(venv) C:\progr\py3\wagtail-starter\wagtail_starter>python manage.py startapp latest_changes

from django.shortcuts import render
from wagtail.admin.views.reports import PageReportView
from wagtail.core.models import UserPagePermissionsProxy
from wagtail.core.models import Page


class LatestChangesView(PageReportView):
    template_name = "reports/latest_changes.html"
    title = "Latest changes"
    header_icon = "date"

    def get_queryset(self):
        self.queryset = Page.objects.order_by("-last_published_at")
        return super().get_queryset()

    def dispatch(self, request, *args, **kwargs):
        if not UserPagePermissionsProxy(request.user).can_remove_locks():
            return permission_denied(request)
        return super().dispatch(request, *args, **kwargs)

from django.http import HttpResponse
from django.conf.urls import url
from wagtail.core import hooks
from .views import LatestChangesView

@hooks.register('register_admin_urls')
def urlconf_time():
  return [
    url(r'^latest_changes/$', LatestChangesView.as_view(), name='latest_changes'),
  ]

Add the template

To add the template we’ll need to create a folder named templates under our latest_changes app and then add a reports folder to it. Finally in that folder add a latest_changes.html. So the full path of the latest_changes.html should be: wagtail_starter\latest_changes\templates\reports\latest_changes.html:

{% extends 'wagtailadmin/reports/base_page_report.html' %}
{% load i18n %}
{% block listing %}
    {% include "reports/_list_latest.html" %}
{% endblock %}

{% block no_results %}
    <p>{% trans "No changes found." %}</p>
{% endblock %}

I’ve selected the reports subfolder just to be compatible with what wagtail does, you can just put latest_changes.html directly under templates; don’t forget to update the LatestChangesView defined before though! This template extends the base_page_report.html template that Wagtail provides for page reports. It also includes a snippet named reports/_list_latest.html" thus you also need to add a ``_list_latest.html file in the same folder with the following contents:

{% extends "wagtailadmin/pages/listing/_list_explore.html" %}

{% load i18n wagtailadmin_tags %}

{% block post_parent_page_headers %}
<tr>
<th>Title</th>
<th>Last update</th>
<th>Kind</th>
<th>Status</th>
<th>Owner / last publish / last edit</th>
</tr>
{% endblock %}

{% block page_navigation %}
    <td>
        {{ page.owner }} / {{ page.live_revision.user }} / {{ page.get_latest_revision.user }}
    </td>
{% endblock %}

Please notice that my _list_latest.html snippet extends the Wagtail provided _list_explore.html template and overrides some things that can be overriden from that file. If you want to do more changes you’ll need to copy over everything and change things as you wish instead of extending.

Also, keep in mind that because you added a templates folder you’ll need to restart your django development server.

Finally, if everything is ok until now you should be able to visit http://127.0.0.1:8000/admin/latest_changes/ and see your view! It will say “No changes found” if you’ve followed the steps here; just go to Pages - Home from the wagtail menu and edit that page (just save it). Now visit http://127.0.0.1:8000/admin/latest_changes/ again and behold! Your own latest changes view:

Displaying our menu item

The last piece of the puzzle missing is to actually display a menu item under the Reports menu of wagtail admin. For this we are going to use our friends, the wagtail hooks. So, change the wagtail_hooks.py file like this (I’m also including the code from adding the url):

from django.http import HttpResponse
from django.conf.urls import url
from django.urls import reverse
from wagtail.admin.menu import MenuItem
from wagtail.core import hooks
from wagtail.core.models import UserPagePermissionsProxy
from .views import LatestChangesView

@hooks.register('register_admin_urls')
def urlconf_time():
    return [
      url(r'^latest_changes/$', LatestChangesView.as_view(), name='latest_changes'),
    ]


class LatestChangesPagesMenuItem(MenuItem):
    def is_shown(self, request):
        return UserPagePermissionsProxy(request.user).can_remove_locks()


@hooks.register("register_reports_menu_item")
def register_latest_changes_menu_item():
    return LatestChangesPagesMenuItem(
        "Latest changes", reverse("latest_changes"), classnames="icon icon-date", order=100,
    )

The above code uses the register_reports_menu_item which is a hook that can be used to add a child specifically to the Reports menu item. Notice that it uses the LatestChangesPagesMenuItem which is a class that inherits from MenuItem; the only thing that is overriden there is the is_shown method so it will have the same permissions as the LatestChangesView we defined above so user that will see the menu item will also have permissions to display the view. Here’s the final menu item:

django-crispy-forms

The django-crispy-forms django package is a great way to properly format your forms. If you don’t already use it I totally recommend to check it out; it’s very useful and you’ll definitely love it if you are a heavy django forms user. It helps you properly layout your forms either implicitly or explicitly.

For explicitly laying out your forms you should add a FormHelper to your django form class and use the {% crispy %} template tag. You can use this to explicitly define your form layout with as much detail as you want since you have full control. I won’t go into more details about this since it’s explained thoroughly in the docs.

For implicitly laying out your forms, you will just use the |crispy template filter. This gets a normal django form (without any modifications) and converts it to a crispy form based on the template pack you are using. This works great for many situations however sometimes you’ll need to have more control over this without going to the extra effort to add a complete layout to each of your forms using the FormHelper.

So how do you resolve this? Enter the |as_crispy_field template filter. To use that filter you’ll need to add the {% load crispy_forms_tags %} lines to your template and then you can pass any one of your form’s fields to it so it will be properly “crispified”! Let’s see a quick example of adding the fields of a form in a <div class='col-md-6'> so they will be in two columns (using bootstrap):

<form class='form' method="POST">
    <div class='row'>
        {% for field in form %}
            <div class='col-md-6'>
              {{ field|as_crispy_field }}
            </div>
        {% endfor %}
        {% csrf_token %}
    </div>
    <input class='btn btn-primary' type='submit'>
</form>

So the above code enumerates all form fields and uses the |as_crispy_field to properly add the crispified information to it. If you want to re-use the above two column layout in multiple forms and be more dry you can create a template snippet and {% include %} it in the part of your code you want the form to be rendered.

django-widget-tweaks

Using the as_crispy_field is excellent however sometimes you may need even more control of your for fields, for example add an extra class (like form-control-lg) to your form controls. The answer to this is the django-widget-tweaks package: It enables you to easily modify form fields by adding classes, attributes etc to them from within your django templates.

To use it you need to add a {% load widget_tweaks %} to your templates. Then you’ll be able to use the |add_class form field to add a class to your form field. For example the previous example can be modified like this to have smaller controls:

<form class='form' method="POST">
    <div class='row'>
        {% for field in form %}
            <div class='col-md-6'>
              {{ field|add_class:"form-control-sm"|as_crispy_field }}
            </div>
        {% endfor %}
        {% csrf_token %}
    </div>
    <input class='btn btn-primary' type='submit'>
</form>

Please notice that I use both add_class and as_crispy_fields together; notice the order, the add_class needs to be before the as_crispy_field or you’ll get an error. This way the django form field will have the form-control-sm class and then be rendered as a crispy field.

Let’s now suppose that you need to more control over your fields. For example you need to add a class only to your select fields or even only to a particular field (depending on its name). To do that you can use the name and field.widget.input_type attributes of each for field. So, to make select fields smaller and with a warning background and fields that have a name of name or year larger you can use something like this:

<form class='form' method="POST">
    <div class='row'>
        {% for field in form %}
          {% if field.name == 'name' or field.name == 'year' %}
              {{ field|add_class:"form-control-lg"|as_crispy_field }}
          {% elif field.field.widget.input_type == 'select' %}
              {{ field|add_class:"form-control-sm"|add_class:"bg-warning"|as_crispy_field }}
          {% else %}
              {{ field|as_crispy_field }}
          {% endif %}
        {% endfor %}
        {% csrf_token %}
    </div>
    <input class='btn btn-primary' type='submit'>
</form>

Using the above techniques you should be able to quickly layout and format your form fields with much control! If you need something more I recommend going the FormLayout route I mentioned in the beginning.

What will it do

The solution is a couple of function that can be put inside the QueryFilterEx I mentioned in the previous article. Please make sure that you’ve completely read and understand this article before continuing here.

To use the dynamic sorting function you’ll need to declare the fields that would allow sorting using a simple array of strings. The sort fields should then be added as links to your phoenix page which will then pass an order_by=field_name query parameter to your controller.

The module has a very simple API consisting of a single function:

• sort_by_params(query, params, allowed_sort_fields): Pass it the query, the GET request parameters you got from your form and the declared sort fields array to return you a sorted query

You can find a sample of the technique presented in this article in my PHXCRD repository: https://github.com/spapas/phxcrd for example in the user_controller or authority_controller.

Preparing the query

In order to use dynamic sorting you’ll need to properly “prepare” your Ecto query by naming all your relations as I’ve already explained in the previous article.

Declaring the sort fields

To declare the sort fields you’ll just add an array of fields you’ll want to allow sorting on. Each field should have the form binding_name__field_name where binding_name is the name of the table you’ve declared in your query and field_name is the name of the field that the query will be sorted by. This is the way that the sort fields will also be declared in the phoenix html page. Django users will definitely remember the model__field convention.

Declaring the sort fields here and using them again in the html page may seem reduntant, however it is absolute necessary to declare a priori which fields are allowed because the sort http params will be received as strings and to be used in queries these strings will be converted to atoms. The number of atoms is finite (there’s an absolute limit of allowed atoms in an erlang program; if that limit is surpassed your program will crash) so you can’t allow the user to pass whatever he wants (so if the order_by parameter does not contain one of the fields you declare here then no strings will be converted to atoms).

Integrating with a controller

As an example let’s see how the dynamic sort fields will be integrated with the phxcrd user_controller. The query I’d like to filter on is the following (see that everything I’ll need is named using :as):

from(u in User, as: :user,
  left_join: a in Authority, as: :authority,
  on: a.id == u.authority_id,
  left_join: up in UserPermission,
  on: up.user_id == u.id,
  left_join: p in Permission, as: :permission,
  on: up.permission_id == p.id,
  preload: [authority: a, permissions: p]
)

To declare the sort fields I like to create a module attribute ending with sort_fields, something like @user_sort_fields for example. Here’s the sort fields I’m going to use for user_controller:

@user_sort_fields [
  "user__username", "user__name", "user__last_login"
]

So it will only allow the user.username, user.name and user.last_login fields for sorting. I could easily sort by authority.name or permission.name in a similar fashion.

Finally, here’s the full code of the index controller:

def index(conn, params) do
  changeset = QueryFilterEx.get_changeset_from_params(params, @user_filters)

  users =
    from(u in User,
      as: :user,
      left_join: a in Authority, as: :authority,
      on: a.id == u.authority_id,

      left_join: up in UserPermission,
      on: up.user_id == u.id,
      left_join: p in Permission, as: :permission,
      on: up.permission_id == p.id,
      preload: [authority: a, permissions: p]
    )
    |> QueryFilterEx.filter(changeset, @user_filters)
    |> QueryFilterEx.sort_by_params(params, @user_sort_fields)
    |> Repo.all()

  render(conn, "index.html", users: users, changeset: changeset)
end

Notice that this is exactly the same as the controller I discussed in the dynamic filters article with the addition of the QueryFilterEx.sort_by_params(params, @user_sort_fields) pipe to do the sorting.

The template

The template for the user index action is also the same with a couple of minor changes: Instead of using a static header for the table title I will use a link that will change the sorting order:

<thead>
  <tr>
    <th>
      <%= link gettext("Username"), to: create_order_url(@conn, "user__username") %>
    </th>
    <th>
      <%= link gettext("Name"), to: create_order_url(@conn, "user__name") %>
    </th>
    <th>First name</th>
    <th>Last name</th>
    <th>Email</th>
    <th>Am / Am phxcrd</th>
    <th>Kind</th>

    <th>
      <%= link gettext("Last login"), to: create_order_url(@conn, "user__last_login") %>
    </th>
    <th>Is enabled</th>

    <th></th>
  </tr>
</thead>

Notice that I just used the create_order_url function passing it the @conn and the sort field. This create_order_url function is implemented in a module I include in all my views and will properly add an order_by=field in the url (it will also add an order_by=-field if the same header is clicked twice). I will explain it more in the following sections.

Finally, please notice that if you use pagination and sorting you need to properly handle the order_by query parameter when creating the next-previous page links. Actually, there are three things competing on their url parameter dominance; I’d like to talk about that in the next interlude.

How does this work?

In this section I’ll try to explain exactly how the dynamic sort fields work.

So I’ll split this explanation in two parts: Explain create_order_url and then explain sort_by_params.

def create_order_url(conn, field_name, allowed_keys \\ ["filter"]) do
  Phoenix.Controller.current_url(conn, get_order_params(conn.params, allowed_keys, field_name))
end

defp get_order_params(params, allowed_keys, order_key) do
  params
  |> Map.take(allowed_keys ++ ["order_by"])
  |> Enum.map(fn {k, v} -> {String.to_atom(k), v} end)
  |> Map.new()
  |> Map.update(
    :order_by,
    order_key,
    &case &1 do
      "-" <> ^order_key -> order_key
      ^order_key -> "-" <> order_key
      _ -> "-" <> order_key
    end
  )
end

def sort_by_params(qs, %{"order_by" => "-" <> val}, allowed),
  do: do_sort_by_params(qs, val, :asc, allowed)

def sort_by_params(qs, %{"order_by" => val}, allowed),
  do: do_sort_by_params(qs, val, :desc, allowed)

def sort_by_params(qs, _, _), do: qs

defp do_sort_by_params(qs, val, ord, allowed) do
  if val in allowed do
    [binding, name] = val |> String.split("__") |> Enum.map(&String.to_atom/1)
    qs |> order_by([{^binding, t}], [{^ord, field(t, ^name)}])
  else
    qs
  end
end

Conclusion

By using the methods described here you can easily add a dynamic sorting to your queries through fields that may span relations just by creating a bunch of http GET links and passing them an order_by query parameter.

A quick HTTP primer

So, what happens during a form display and submission? To properly understand that you need to know what an HTTP request is. An HTTP request is the way the browser “requests” a URL from a web server. This is thoroughly explained in the HTTP protocol however, for our purposes here it should suffice to say that an HTTP request is a text message that is send from the browser/HTTP Client to the HTTP/web server. This message should have the following format:

METHOD PATH HTTP/VERSION
Header 1: Value 1
Header 2: Value 2

Request data

The METHOD can be one of various methods supported in the HTTP protocol like GET, POST, PUT, OPTIONS etc however the most popular and the ones we’ll talk about here are GET and POST. The PATH is the actual url of the “page” you want to view without the server part. So if you are visiting http://www.example.com/path/to/page.html the path parameter will have a value of /path/to/page.html. The HTTP/VERSION will contain the version of HTTP the client uses; usually it is something like HTTP/1.1. Finally, after that first line there’s a bunch of optional headers with various extra information the client wants to pass to the server (for example what encoding it supports, what’s the host it connects to etc).

Additionally, in case of a POST request the headers are followed by a blank line which in turn is followed by a chunk of “data” that the client passes to the server.

The server will then response back with a text message similar to this (HTTP Response):

HTTP/VERSION STATUS
Header 1: Value 1
Header 2: Value 2

Response data

The HTTP/VERSION will also be something like HTTP/1.1 while the STATUS will be the status of the response. This status is a 3-digit numeric value followed by a textual description of the status. There are various statuses that you can receive, however the statuses can be grouped by the number they start with like this:

• 1xx: Information; rarely used
• 2xx: Success; status 200 is the most common one
• 3xx: Redirection (browser must visit another page); either permanent or temporary
• 4xx: Client error; 404 is page not found (also access denied errors will be 40x)
• 5xx: Server error; something fishy happened to the server while responding

In this article we’ll mainly talk about 2xx and 3xx: A 200 OK answer is the most common one, it means that the HTTP request was completed successfully. A 302 FOUND request means that the browser should display a “different” path; that path will be provided in a Location: path header. When the browser receives a redirect it will do another GET request to retrieve the redirect to page.

Notice that the Headers and Data parts of the server reply may also be optional like for the client however they usually exist (especially with a 200 response; without the data the client won’t have anything to display).

So when a browser “requests” a page it will send an HTTP GET to the path. This happens all the time when we visit(click) links or entering urls to our browser. However, when we submit an HTML form the situation is a little more complex.

Form methods

An html form is a <form> tag that contains a bunch of <input> elements each one of which should have at least a name property. One of the inputs will usually be is a submit button.

Also, the form tag has two important attributes:

• action: Defines the url where the post will be submitted to; it can be omitted to submit the form to the current path
• method: Will be either GET or POST

Thus, a sample form is something like:

<form method="GET" action="form.php" >
  <input type='text' name='input1'>
  <input type='text' name='input2'>
  <input type='submit'>
</form>

So what are the differences between a <form method='GET'> and a <form method='POST'>?

• Submitting an HTML form will translate to either an HTTP GET request or an HTTP POST request to the server depending on the method attribute
• The data of a GET form will be encoded in the PATH of the HTTP request while the data of the POST form will be in the corresponding data part of the HTTP Request
• A form that is submitted with GET should be idempotent i.e it should not modify anything in the server; a form that is submitted with POST should modify something the server

So, the form we defined previously (that has a GET method) will issue the following HTTP request (if we fill the values value1 and value2 to the inputs):

GET /form.php?input1=value1&input2=value2 HTTP/1.1

One the other hand if the form had a POST method the HTTP request would be like this:

POST /form.php HTTP/1.1

input1=value1&input2=value2

Notice that in the first case the data is in the PATH in the 1st line of the request while in the second case it is passed in the data section. Also, in both cases the encoded data of the form is similar to input1=value1&input2=value2: it is a list of key=value pairs seperated with & where the name attribute of each input is used as the key.

Concerning the idempotency of the action; this is not something that the HTTP protocol can enforce but it relies on the developer to implement it. When a form will not change anything to the server then it should be implemented as a method=GET. For example when you have a form with a search box that just returns some results. On the other hand, when a form does change things for example when you create a new item in an application then it should be implemented as a method=POST.

The browser has a different behavior after a GET vs after a POST because it expects that when you do a GET request then it won’t matter if that request is repeated many times. One the other hand, the browser will try to prevent you from submitting a request many times (because something is changed in the server so the user must do it intentionally by for example pressing a button to submit the form).

Proper form handling

So, following the previous section we can now explain how to handle a form properly:

For a GET form we don’t have to do anything fancy; we just retrieve the parameter values and we display the data these parameters correspond to just like if we displayed any other page.

For a POST form however we need to be extra careful as to avoid re-duplication of data and have a good user experience: When a POST form is submitted we need first to make sure that the submitted parameters are valid (for example there are no missing required fields). If the form is not valid then we will return a status 200 OK explaining to the user what went wrong; we usually return the same page containing the initial form with the fields that had errors marked.

On the other hand, if the form was valid we need to do the actual action that the form corresponds to; for example insert something to the database. After this is finished we should return a redirect (302) to either a different or even the same page. This will result to the browsing doing a GET request to the page we redirect to so there would be no danger of the user refreshing the page and resubmitting the form. We should not return a 200 OK after a POST request because then the user would be able to press F5 to duplicate the previous POST request (and re-insert the data).

One extra thing that we need to consider is how should we inform the user that his action was successful after the form submission and redirection? As we said, we can’t return a 200 OK message so we can’t really “create” the response, we instead need to redirect to another page. A common practice for this is to use a “flash” message; this is offered by many web frameworks through specific functions but can be easily implemented. I’ll explain how in the next section.

Implementing flash messages

Before talking about the flash message I’d like to quickly explain what’s a cookie and a browser session in HTTP, because the flash message builds upon these concepts:

A cookie is a way for the server to tell the client to store some information to be re-used later. What happens is that the server returns an HTTP header line similar to Set-Cookie: cookie-name=cookie-value. When the client receives that header line it will pass an HTTP header line similar to Cookie:  cookie-name=cookie-value to all future requests so the server will know which value it had send to the client (there are various options for expiring cookies etc but they are not important here). This may seem like a primitive solution however because HTTP is a stateless protocol that’s the only way for the server to store information about a client. If you disable cookies completely in a browser then there won’t be a way for the server to remember you, for example you won’t be able to login anywhere!

A session is a better way to store info about the client that builds upon the cookies. What happens is that when a client visits a site for the first time (so it has no cookies for that particular site) the server will send back a cookie named session_id (or something like that) containing a very big random number. The server will save this session id number in a persistent storage (for example in a database or a text file) and will correlate that number with information for that particular client. When the client sends back that session_id cookie the server will fetch the correlated info for that particular client from the persistent storage (and may update them etc). This way the server can store whatever info it wants about a particular client. The server usually keeps a a dictionary (map) of key-values for each session.

Now, a flash message is some information (message) that should be displayed to the user once. For example a message like “Your form has been submitted!”.

A simple way to implement this is to add a message attribute to the session when you want to display the flash message. The next page that is displayed (irrelevantly if there’s a redirect involved) will check to see if there’s a message attribute to the session; if yes it will display the actual message and remove the message attribute from the session (so it won’t be displayed again).

Implementing the form submission

Following the above guidelines I’ll present here a typical, production-ready form submission for PHP. Some choices I’ve made:

• I am going to implement a POST form since a GET form doesn’t need any special handling
• The form handler will be the same PHP page as the one displaying the form. This is a usual thing to do, you check the HTTP method and either display the form as-is (if it is GET) or handle the submission (if it is POST)
• When the form is submitted successfully redirect to the same page and display a flash message
• Check for valid input and display the error message

So without further ado here’s the complete php code that will submit your form; store it in a file named test.php:

<!DOCTYPE HTML>
<html>

<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <meta http-equiv="X-UA-Compatible" content="ie=edge">
  <title>TEST FORM</title>
  <link href="https://unpkg.com/tailwindcss@^1.0/dist/tailwind.min.css" rel="stylesheet">
</head>

<body class="bg-gray-100">
  <div class="px-8 py-8 w-1/2 m-auto">

    <?php
    session_start();
    if ($_SESSION['message'] ?? '') {
      echo '<div class="rounded bg-green-300 px-2 py-2">'.$_SESSION['message'].'</div>';
      unset($_SESSION['message']);
    }

    $nameErr = $wsErr = $name = $comment = "";
    $formValid = true;

    if ($_SERVER["REQUEST_METHOD"] == "POST") {
      if (empty($_POST["name"])) {
        $nameErr = "Name is required";
        $formValid = false;
      } else {
        $name = $_POST["name"];
      }

      if (empty($_POST["comment"])) {
        $comment = "";
      } else {
        $comment = $_POST["comment"];
      }

      if ($formValid) {
        if (1 == rand(1, 2)) {
          $_SESSION['message'] = "Success! You have submitted the values: <b>" . $name . " / " . $comment . "</b>";
          header('Location: ./test.php');
          exit();
        } else {
          $wsErr = "Error while trying to submit the form!";
        }
      }
    }
    ?>

    <h1 class="text-4xl font-bold text-indigo-500">Test a PHP form!</h1>

    <form class='border border-blue-800 rounded p-2' method="POST">

      <?= $wsErr ? "<div class='text-red-600 py-3'>" . $wsErr . "</div>" : "" ?>

      <div class="p-1">
        <label for="name">Name:</label> <input class="border border-blue-800 rounded" id="name" type="text" name="name" value="<?= $name ?>">
        <span class="text-red-600">* <?= $nameErr; ?></span>
      </div>

      <div class="p-1">
        <label for="comment">Comment:</label> <textarea class="border border-blue-800 rounded" id="comment" name="comment" rows="5" cols="40"><?= $comment ?></textarea>
      </div>

      <input class="rounded px-2 my-4 py-2 bg-blue-800 text-gray-100" type="submit" name="submit" value="Save">
    </form>
  </div>
</body>
</html>

So how is that working? As we can see there’s the php code first and the html is following (with some sprinkles of php). The HTML code is rather simple: It will display a POST form with two inputs named name and comment. Notice that we pass the $name and $comment php variables as their values. It also has a submit button and will display the $wsErr variable if it is not null (which means that there was an error while submitting the data). The PHP code now first starts the session (i.e it passes the session_id cookie to the client if such a cookie does not exist) checks to see if there’s a message attribute to the session. If such a message exists it will display it in a rounded green panel and remove that from the session (so it won’t be displayed again next time):

session_start();
if ($_SESSION['message'] ?? '') {
  echo '<div class="rounded bg-green-300 px-2 py-2">'.$_SESSION['message'].'</div>';
  unset($_SESSION['message']);
}

After that there are some variable initializations and we check if the HTTP request is POST (if the request is GET we’ll just disply the HTML):

if ($_SERVER["REQUEST_METHOD"] == "POST") {

For each of the inputs we check if they are empty or not and assign their values to the corresponding php variable. If the name is empty then we’ll set $formValid = false; and add an error message since this field is required. Then, if $formValid is not false we can do the actual action (for example write to the database). I’ve simulated that using a coin-toss with rand (so there’s a 50% possibility that the action will fail). If the action “failed” then nothing happened in the database so we should return the same page with the wsErr variable containing the error.

However if the action is successful that means that the data has been inserted to the database so we’ll need to set the flash message and do the redirect (the name of the page containing the form is /test.php so we’ll redirect to it):

$_SESSION['message'] = "Success! You have submitted the values: <b>" . $name . " / " . $comment . "</b>";
header('Location: ./test.php');
exit();

The two commands above (header and exit) will do the actual redirect in php. Since the session contains the message it will be displayed after the redirect has finished!

Conclusion

So following the above tutorial, here’s what you should absolutely do to submit an HTML form:

• Use HTTP POST if the form is going to change data on the server; use HTTP GET otherwise (mainly for search/filter forms)
• When using POST: Redirect when the form is valid and the action on the server has finished successfully; never return a 200 OK status when you’ve changed things in the server (database)
• Use flash messages to pass information to the user after a redirect

What will it do

The solution is more or less a self contained Elixir module named QueryFilterEx that can be used to declaratively filter your queries. To use that you’ll need to declare your filters using a simple array of maps. The filters should then be added in your form using a different input for each filter; then your queryset will be filtered with all the values you’ve added to the inputs using AND.

The module has a very simple API consisting of three functions:

• get_changeset_from_params(params, filters): Pass it the GET request parameters you got from your form and the declared filters array to return you a proper changeset (which you can then use to build your form in your html)
• make_filter_changeset(filters, params): This function actually generates the changeset using the filters and a Map of filter_name: value pairs (it is actually used by get_changeset_from_params)
• filter(query, changeset, filters): Filter the query using the previously created changeset and the declared filters array

You can find a sample of the technique presented in this article in my PHXCRD repository: https://github.com/spapas/phxcrd for example in the user_controller or authority_controller.

Preparing the query

In order to use the QueryFilterEx module you’ll need to properly “prepare” your Ecto query. By preparing I don’t mean a big deal just the fact that you’ll need to name all your relations (or at least name all the relations you’re going to use for filtering). This is very simple to do, for example for the following query:

from(a in Authority,
  join: ak in AuthorityKind,
  on: [id: a.authority_kind_id],
  preload: [authority_kind: ak]
)

you can name the relations by adding two as: atoms like this:

from(a in Authority, as: :authority,
  join: ak in AuthorityKind, as: :authority_kind,
  on: [id: a.authority_kind_id],
  preload: [authority_kind: ak]
)

So after each join: you’ll add a name for your joined relation (and also add a name for your initial relation). Please notice that you can use any name you want for these (not related to the schema names).

Declaring the filters

To declare the filters you’ll just add an array of simple Elixir maps. Each map must have the following fields:

• :name This is the name of the specific filter; it is mainly used in conjunction with the queryset and the form fields to set initial values etc
• :type This is the type of the specific filter; it should be a proper Ecto type like :string, :date, :integer etc. This is needed to properly cast the values and catch errors
• :binding This is the name of the relation this filter concerns which you defined in your query using :as (discussed in previous section)
• :field_name This is the actual name of the field you want to filter on
• :method How to filter on this field; I’ve defined a couple of methods I needed but you can implement anything you want

The methods I’ve implemented are the following:

• :eq Equality
• :ilike Field value starts with the input - ignore case
• :icontains Field value contains the input - ignore case
• :year Field is a date or datetime an its year is the same as the value
• :date Field is a datetime and its date part is equal to the value

Anything else will just be compared using = (same as :eq).

Integrating it with a controller

As an example let’s see how QueryFilterEx is integrated it with the phxcrd user_controller. The query I’d like to filter on is the following (see that everything I’ll need is named using :as:

from(u in User, as: :user,
  left_join: a in Authority, as: :authority,
  on: a.id == u.authority_id,
  left_join: up in UserPermission,
  on: up.user_id == u.id,
  left_join: p in Permission, as: :permission,
  on: up.permission_id == p.id,
  preload: [authority: a, permissions: p]
)

To declare the filters I like to create a module attribute ending with filters, something like @user_filters for example. Here’s the filters I’m going to use for user_controller:

@user_filters [
  %{name: :username, type: :string, binding: :user, field_name: :username, method: :ilike},
  %{name: :authority_name, type: :string, binding: :authority, field_name: :name, method: :icontains},
  %{name: :permission_name, type: :string, binding: :permission, field_name: :name, method: :ilike},
  %{name: :last_login_date, type: :date, binding: :user, field_name: :last_login, method: :date}
]

So it will check if the user.username and permission.name start with the passed value, authority.name contains the passed value and if the user.login_date (which is a datetime) is the same as the passed date value.

Finally, here’s the full code of the index controller:

def index(conn, params) do
  changeset = QueryFilterEx.get_changeset_from_params(params, @user_filters)

  users =
    from(u in User,
      as: :user,
      left_join: a in Authority, as: :authority,
      on: a.id == u.authority_id,

      left_join: up in UserPermission,
      on: up.user_id == u.id,
      left_join: p in Permission, as: :permission,
      on: up.permission_id == p.id,
      preload: [authority: a, permissions: p]
    )
    |> QueryFilterEx.filter(changeset, @user_filters)
    |> Repo.all()

  render(conn, "index.html", users: users, changeset: changeset)
end

It is very simple, it just uses the get_changeset_from_params method I discussed before to generate the changeset and then uses it to filter the query. Also please notice that it passes the changeset to the template to be properly rendered in the filter form.

The template

The template for the user index action is the following:

<%= form_for @changeset, AdminRoutes.user_path(@conn, :index), [method: :get, class: "filter-form", as: :filter],  fn f -> %>
  <%= label f, :username, gettext "Username" %>
  <%= text_input f, :username  %>

  <%= label f, :authority_name, gettext "Authority name" %>
  <%= text_input f, :authority_name  %>

  <%= label f, :permission_name, gettext "Permission name" %>
  <%= text_input f, :permission_name  %>

  <%= label f, :last_login_date, gettext "Last login date" %>
  <%= text_input f, :last_login_date  %>
  <%= error_tag f, :last_login_date %>

  <%= submit gettext("Filter"), class: "ml-5" %>
  <%= link gettext("Reset"), to: AdminRoutes.user_path(@conn, :index), class: "button button-outline ml-2" %>
<% end %>
<%= for user <- @users do %>
<!-- display the user info -->
<% end %>

Notice that it gets the @changeset and uses it to properly fill the initial values and display error messages. For this case I’ve only added an error_tag for the :last_login_date field, the others since are strings do not really need it since they will accept all values.

Also, the form method form must be :get since we only filter (not change anything) and I’ve passed the as: :filter option to the form_for to collect the parameters under the filter server side parameter (this can be anything you want and can be optionally be passed to QueryFilterEx.get_changeset_from_params to know which parameter the filters are collected on).

How does this work?

In this section I’ll try to explain exactly how the QueryFilterEx module works. Before continuing I want to thank the people at the Elixir forum and #elixir-lang Freenode IRC chat that helped me with understanding how to be able to create dynamic bindings.

So I’ll split this explanation in two parts: Explain QueryFilterEx.get_changeset_from_params and make_filter_changeset (easy) and then explain QueryFilterEx.filter (more difficult).

def get_changeset_from_params(params, filters, filter_name \\ "filter") do
  case params do
    %{^filter_name => filter_params} ->
      filters |> make_filter_changeset(filter_params)

    _ ->
      filters |> make_filter_changeset(%{})
  end
end

defp make_filter_keys(filters) do
  filters |> Enum.map(& &1.name)
end

defp make_filter_types(filters) do
  filters |> Enum.map(&{&1.name, &1.type}) |> Map.new()
end

def make_filter_changeset(filters, params) do
  data = %{}
  types = filters |> make_filter_types

  {data, types}
  |> Ecto.Changeset.cast(params, filters |> make_filter_keys)
  |> Map.merge(%{action: :insert})
end

def filter(query, changeset, filters) do
  changes = Map.fetch!(changeset, :changes)
  filters |> Enum.reduce(query, creat_where_clauses_reducer(changes))
end

defp creat_where_clauses_reducer(changes) do
  fn %{name: name, field_name: field_name, binding: binding, method: method}, acc ->
    case Map.fetch(changes, name) do
      {:ok, value} ->
        acc |> creat_where_clause(field_name, binding,  method, value)

      _ ->
        acc
    end
  end
end

defp creat_where_clause(acc, field_name, binding,  method, value) do
  case method do
    :eq -> acc |> where(
      [{^binding, t}],
      field(t, ^field_name) == ^value
    )
    :ilike -> acc |> where(
      [{^binding, t}],
      ilike(field(t, ^field_name), ^("#{value}%") )
    )
    :icontains -> acc |> where(
      [{^binding, t}],
      ilike(field(t, ^field_name), ^("%#{value}%") )
    )
    :year -> acc  |> where(
      [{^binding, t}],
      fragment("extract (year from ?) = ?", field(t, ^field_name), ^value)
    )
    :date -> acc  |> where(
      [{^binding, t}],
      fragment("? >= cast(? as date) and ? < (cast(? as date) + '1 day'::interval"), field(t, ^field_name), ^value, field(t, ^field_name), ^value)
    )
    _ -> acc |> where(
      [{^binding, t}],
      field(t, ^field_name) == ^value
    )

  end
end

acc |> where(
  [{^binding, t}],
  field(t, ^field_name) == ^value
)

from(u in User) |> where([u], u.name == "root") |> Repo.all()

from(u in User, as: :user) |> where([user: u], u.name == "root") |> Repo.all()

from(u in User, as: :user) |> where([{:user, u}], u.name == "root") |> Repo.all()

binding = :user
from(u in User, as: :user) |> where([{^binding, u}], u.name == "root") |> Repo.all()

binding = :user
field_name = :name
from(u in User, as: :user) |> where([{^binding, u}], field(u, ^field_name) == "root") |> Repo.all()

where(
  [{^binding, t}],
  fragment("? >= cast(? as date) and ? < (cast(? as date) + '1 day'::interval"), field(t, ^field_name), ^value, field(t, ^field_name), ^value)
)

field_name >= cast(value as date) AND field_name < (cast(value as date) + '1 day'::interval)

Conclusion

By using the QueryFilterEx module presented here you can very quickly declare the fields you want to filter on and the method you want to use for each field no matter if these fields are in the same schema or are accessed through joins. You can easily extend the functionality of the module by adding your own methods. The only extra thing you need to do is to just add names to your queries.