/var/

Various programming stuff

Declarative Ecto query sorting

In a previous article I presented a method for declaring dynamic filters for your ecto queries. Continuing this article, I’ll present here a way to allow dynamic sorting for your queries using fields that may even span relations.

What will it do

The solution is a couple of function that can be put inside the QueryFilterEx I mentioned in the previous article. Please make sure that you’ve completely read and understand this article before continuing here.

To use the dynamic sorting function you’ll need to declare the fields that would allow sorting using a simple array of strings. The sort fields should then be added as links to your phoenix page which will then pass an order_by=field_name query parameter to your controller.

The module has a very simple API consisting of a single function:

  • sort_by_params(query, params, allowed_sort_fields): Pass it the query, the GET request parameters you got from your form and the declared sort fields array to return you a sorted query

You can find a sample of the technique presented in this article in my PHXCRD repository: https://github.com/spapas/phxcrd for example in the user_controller or authority_controller.

Preparing the query

In order to use dynamic sorting you’ll need to properly “prepare” your Ecto query by naming all your relations as I’ve already explained in the previous article.

Declaring the sort fields

To declare the sort fields you’ll just add an array of fields you’ll want to allow sorting on. Each field should have the form binding_name__field_name where binding_name is the name of the table you’ve declared in your query and field_name is the name of the field that the query will be sorted by. This is the way that the sort fields will also be declared in the phoenix html page. Django users will definitely remember the model__field convention.

Declaring the sort fields here and using them again in the html page may seem reduntant, however it is absolute necessary to declare a priori which fields are allowed because the sort http params will be received as strings and to be used in queries these strings will be converted to atoms. The number of atoms is finite (there’s an absolute limit of allowed atoms in an erlang program; if that limit is surpassed your program will crash) so you can’t allow the user to pass whatever he wants (so if the order_by parameter does not contain one of the fields you declare here then no strings will be converted to atoms).

Integrating with a controller

As an example let’s see how the dynamic sort fields will be integrated with the phxcrd user_controller. The query I’d like to filter on is the following (see that everything I’ll need is named using :as):

from(u in User, as: :user,
  left_join: a in Authority, as: :authority,
  on: a.id == u.authority_id,
  left_join: up in UserPermission,
  on: up.user_id == u.id,
  left_join: p in Permission, as: :permission,
  on: up.permission_id == p.id,
  preload: [authority: a, permissions: p]
)

To declare the sort fields I like to create a module attribute ending with sort_fields, something like @user_sort_fields for example. Here’s the sort fields I’m going to use for user_controller:

@user_sort_fields [
  "user__username", "user__name", "user__last_login"
]

So it will only allow the user.username, user.name and user.last_login fields for sorting. I could easily sort by authority.name or permission.name in a similar fashion.

Finally, here’s the full code of the index controller:

def index(conn, params) do
  changeset = QueryFilterEx.get_changeset_from_params(params, @user_filters)

  users =
    from(u in User,
      as: :user,
      left_join: a in Authority, as: :authority,
      on: a.id == u.authority_id,

      left_join: up in UserPermission,
      on: up.user_id == u.id,
      left_join: p in Permission, as: :permission,
      on: up.permission_id == p.id,
      preload: [authority: a, permissions: p]
    )
    |> QueryFilterEx.filter(changeset, @user_filters)
    |> QueryFilterEx.sort_by_params(params, @user_sort_fields)
    |> Repo.all()

  render(conn, "index.html", users: users, changeset: changeset)
end

Notice that this is exactly the same as the controller I discussed in the dynamic filters article with the addition of the QueryFilterEx.sort_by_params(params, @user_sort_fields) pipe to do the sorting.

The template

The template for the user index action is also the same with a couple of minor changes: Instead of using a static header for the table title I will use a link that will change the sorting order:

<thead>
  <tr>
    <th>
      <%= link gettext("Username"), to: create_order_url(@conn, "user__username") %>
    </th>
    <th>
      <%= link gettext("Name"), to: create_order_url(@conn, "user__name") %>
    </th>
    <th>First name</th>
    <th>Last name</th>
    <th>Email</th>
    <th>Am / Am phxcrd</th>
    <th>Kind</th>

    <th>
      <%= link gettext("Last login"), to: create_order_url(@conn, "user__last_login") %>
    </th>
    <th>Is enabled</th>

    <th></th>
  </tr>
</thead>

Notice that I just used the create_order_url function passing it the @conn and the sort field. This create_order_url function is implemented in a module I include in all my views and will properly add an order_by=field in the url (it will also add an order_by=-field if the same header is clicked twice). I will explain it more in the following sections.

Finally, please notice that if you use pagination and sorting you need to properly handle the order_by query parameter when creating the next-previous page links. Actually, there are three things competing on their url parameter dominance; I’d like to talk about that in the next interlude.

Interlude: HTTP GET parameter priority

Now, in an index page you will probably have three things all of which will want to put parameters to your urls to be activated:

  • Query filtering; this will put a filter query parameter to filter your query. Notice that because of how phoenix works (it allows maps in the query parameters) the filter can be a single query parameter but contain multiple filters (i.e the filter will be something like %{"key1" => "value1", "key2" => "value2"}
  • Order by: This will put an order_by query parameter to denote the field that the query will be sorted
  • Pagination: This will put an page query parameter to denote the current page

I like to give them a priority in the order I’ve listed them; when one of them is changed, it will clear the ones following it. So if the query filters are changed both the pagination and the order by fields will be cleared, if the order by field is changed then only the pagination field will be cleared but if the pagination field is changed both the query filters and the order by fields will be kept there.

I think that’s the best way to do it from an UX point of view; try to think about it and you’ll probably agree.

How does this work?

In this section I’ll try to explain exactly how the dynamic sort fields work.

So I’ll split this explanation in two parts: Explain create_order_url and then explain sort_by_params.

create_order_url

This function receives three parameters: The current @conn, the name of a field to sort by and an optional list of query parameters that need to be kept while creating the order by links. I’ve put this function in a ViewHelpers module that I am including to all my views (by adding an import PhxcrdWeb.ViewHelpers line to the PhxcrdWeb module).

Let’s take a look at the code:

def create_order_url(conn, field_name, allowed_keys \\ ["filter"]) do
  Phoenix.Controller.current_url(conn, get_order_params(conn.params, allowed_keys, field_name))
end

This doesn’t do much, it just uses the phoenix’s current_url that generates a new url to the current page, passing it a dictionary of http get parameters that should be appended to the url that are created through get_order_params. Notice that there’s an allowed_keys parameter that contains the query parameters that we need to keep after the sorting (see the previous interlude). By default I pass the filter query parameter so if theres a filter (check my previous article) it will keep it when sorting (but any pagination will be cleared; if I sort by a new field I want to go to the first page there’s no reason for me to keep seeing the page I was on before changing the order by).

The get_order_params receives the query parameters of the current connection (as a map), the allowed keys I mentioned before and the actual name of the field to sort on. This method is a little more complex:

defp get_order_params(params, allowed_keys, order_key) do
  params
  |> Map.take(allowed_keys ++ ["order_by"])
  |> Enum.map(fn {k, v} -> {String.to_atom(k), v} end)
  |> Map.new()
  |> Map.update(
    :order_by,
    order_key,
    &case &1 do
      "-" <> ^order_key -> order_key
      ^order_key -> "-" <> order_key
      _ -> "-" <> order_key
    end
  )
end

It only keeps the parameters in the allowed_keys list and the current order_by parameter (if there’s one) discarding everything else. It will then convert the keys of the map to atoms and put them in a new map. Finally, it will update the order_by field (if exists) either by switching the - in front of the field to declare asc/desc sorting or adding it for the field that was clicked. Actually the logic of that Map.update is the following:

  • If there’s no :order_by key then add it and assign the passed order_key
  • If the current value of :order_by is equal to order_key with or without a - then toggle the - (this happens when you click on a field that is already used for sorting)
  • If the current value of :order_by is anything else (i.e not the same as the order_key) then just change :order_by to -orderKey (this happens when there’s sorting but you click on a different field, not the one used for the sorting)

Notice that this juggling between map, list of keywords and then map again (using Enum.map and then Map.new etc) is needed because the query parameters are in a map with strings as keys form (%{"key" => "value"}) while the current_url function needs the query params in a map with atoms as keys form (%{key: "value"}).

sort_by_params

The sort_by_params method gets three parameters: The query that will be sorted, the existing http parameters map (so as to retrieve the order_by value) and the declared list of allowed sorting fields. Let’s take a look at it:

def sort_by_params(qs, %{"order_by" => "-" <> val}, allowed),
  do: do_sort_by_params(qs, val, :asc, allowed)

def sort_by_params(qs, %{"order_by" => val}, allowed),
  do: do_sort_by_params(qs, val, :desc, allowed)

def sort_by_params(qs, _, _), do: qs

This multi-legged function will only do something if there’s an order_by parameter in the http parameters (else it will just return the query as is) and will call do_sort_by_params passing it the received query, either :asc or :desc (depending if there’s a - in front of the value) and the received allowed fields list.

The do_sort_by_params makes sure that the passed parameter is in the allowed list and if yes it creates the atoms of the binding and field name (using String.to_atom) and does the actual sorting to the passed query:

defp do_sort_by_params(qs, val, ord, allowed) do
  if val in allowed do
    [binding, name] = val |> String.split("__") |> Enum.map(&String.to_atom/1)
    qs |> order_by([{^binding, t}], [{^ord, field(t, ^name)}])
  else
    qs
  end
end

The line qs |> order_by([{^binding, t}], [{^ord, field(t, ^name)}]) may seem a little complex but it has been thoroughly explained in the previous article.

Conclusion

By using the methods described here you can easily add a dynamic sorting to your queries through fields that may span relations just by creating a bunch of http GET links and passing them an order_by query parameter.

How to properly handle an HTML form

One of the simplest things somebody would like to do in a web application is to display a form to the user and then handle the data the application received from the form. This is a very common task however there are many loose ends and things that a new developer can do wrong. Recently, I tried to explain this to somebody however I couldn’t find a tutorial containing all the information I think it is important. So I decided to write it here.

To make it very easy to test it and understand it I decided to use PHP to handle the form submission. I must confess that I don’t like PHP and I never use it in production apps (unless I need to support an existing one). However, for such simple tasks and examples PHP is probably the easiest thing for a new developer to understand: Just create a .php file and put it in a folder in your apache htdocs; there now you can test the behavior!

This tutorial will be as comprehensive as possible and will try to explain all things that I feel that need explaining. However I won’t go into details about HTML or PHP syntax; you’ll need to know some things about them to understand what’s going on here; after all the important thing is to understand the big picture so you can re-implement them in your case (or understand why your web framework does what it does).

A quick HTTP primer

So, what happens during a form display and submission? To properly understand that you need to know what an HTTP request is. An HTTP request is the way the browser “requests” a URL from a web server. This is thoroughly explained in the HTTP protocol however, for our purposes here it should suffice to say that an HTTP request is a text message that is send from the browser/HTTP Client to the HTTP/web server. This message should have the following format:

METHOD PATH HTTP/VERSION
Header 1: Value 1
Header 2: Value 2

Request data

The METHOD can be one of various methods supported in the HTTP protocol like GET, POST, PUT, OPTIONS etc however the most popular and the ones we’ll talk about here are GET and POST. The PATH is the actual url of the “page” you want to view without the server part. So if you are visiting http://www.example.com/path/to/page.html the path parameter will have a value of /path/to/page.html. The HTTP/VERSION will contain the version of HTTP the client uses; usually it is something like HTTP/1.1. Finally, after that first line there’s a bunch of optional headers with various extra information the client wants to pass to the server (for example what encoding it supports, what’s the host it connects to etc).

Additionally, in case of a POST request the headers are followed by a blank line which in turn is followed by a chunk of “data” that the client passes to the server.

The server will then response back with a text message similar to this (HTTP Response):

HTTP/VERSION STATUS
Header 1: Value 1
Header 2: Value 2

Response data

The HTTP/VERSION will also be something like HTTP/1.1 while the STATUS will be the status of the response. This status is a 3-digit numeric value followed by a textual description of the status. There are various statuses that you can receive, however the statuses can be grouped by the number they start with like this:

  • 1xx: Information; rarely used
  • 2xx: Success; status 200 is the most common one
  • 3xx: Redirection (browser must visit another page); either permanent or temporary
  • 4xx: Client error; 404 is page not found (also access denied errors will be 40x)
  • 5xx: Server error; something fishy happened to the server while responding

In this article we’ll mainly talk about 2xx and 3xx: A 200 OK answer is the most common one, it means that the HTTP request was completed successfully. A 302 FOUND request means that the browser should display a “different” path; that path will be provided in a Location: path header. When the browser receives a redirect it will do another GET request to retrieve the redirect to page.

Notice that the Headers and Data parts of the server reply may also be optional like for the client however they usually exist (especially with a 200 response; without the data the client won’t have anything to display).

So when a browser “requests” a page it will send an HTTP GET to the path. This happens all the time when we visit(click) links or entering urls to our browser. However, when we submit an HTML form the situation is a little more complex.

Form methods

An html form is a <form> tag that contains a bunch of <input> elements each one of which should have at least a name property. One of the inputs will usually be is a submit button.

Also, the form tag has two important attributes:

  • action: Defines the url where the post will be submitted to; it can be omitted to submit the form to the current path
  • method: Will be either GET or POST

Thus, a sample form is something like:

<form method="GET" action="form.php" >
  <input type='text' name='input1'>
  <input type='text' name='input2'>
  <input type='submit'>
</form>

So what are the differences between a <form method='GET'> and a <form method='POST'>?

  • Submitting an HTML form will translate to either an HTTP GET request or an HTTP POST request to the server depending on the method attribute
  • The data of a GET form will be encoded in the PATH of the HTTP request while the data of the POST form will be in the corresponding data part of the HTTP Request
  • A form that is submitted with GET should be idempotent i.e it should not modify anything in the server; a form that is submitted with POST should modify something the server

So, the form we defined previously (that has a GET method) will issue the following HTTP request (if we fill the values value1 and value2 to the inputs):

GET /form.php?input1=value1&input2=value2 HTTP/1.1

One the other hand if the form had a POST method the HTTP request would be like this:

POST /form.php HTTP/1.1

input1=value1&input2=value2

Notice that in the first case the data is in the PATH in the 1st line of the request while in the second case it is passed in the data section. Also, in both cases the encoded data of the form is similar to input1=value1&input2=value2: it is a list of key=value pairs seperated with & where the name attribute of each input is used as the key.

Concerning the idempotency of the action; this is not something that the HTTP protocol can enforce but it relies on the developer to implement it. When a form will not change anything to the server then it should be implemented as a method=GET. For example when you have a form with a search box that just returns some results. On the other hand, when a form does change things for example when you create a new item in an application then it should be implemented as a method=POST.

The browser has a different behavior after a GET vs after a POST because it expects that when you do a GET request then it won’t matter if that request is repeated many times. One the other hand, the browser will try to prevent you from submitting a request many times (because something is changed in the server so the user must do it intentionally by for example pressing a button to submit the form).

Proper form handling

So, following the previous section we can now explain how to handle a form properly:

For a GET form we don’t have to do anything fancy; we just retrieve the parameter values and we display the data these parameters correspond to just like if we displayed any other page.

For a POST form however we need to be extra careful as to avoid re-duplication of data and have a good user experience: When a POST form is submitted we need first to make sure that the submitted parameters are valid (for example there are no missing required fields). If the form is not valid then we will return a status 200 OK explaining to the user what went wrong; we usually return the same page containing the initial form with the fields that had errors marked.

On the other hand, if the form was valid we need to do the actual action that the form corresponds to; for example insert something to the database. After this is finished we should return a redirect (302) to either a different or even the same page. This will result to the browsing doing a GET request to the page we redirect to so there would be no danger of the user refreshing the page and resubmitting the form. We should not return a 200 OK after a POST request because then the user would be able to press F5 to duplicate the previous POST request (and re-insert the data).

One extra thing that we need to consider is how should we inform the user that his action was successful after the form submission and redirection? As we said, we can’t return a 200 OK message so we can’t really “create” the response, we instead need to redirect to another page. A common practice for this is to use a “flash” message; this is offered by many web frameworks through specific functions but can be easily implemented. I’ll explain how in the next section.

Implementing flash messages

Before talking about the flash message I’d like to quickly explain what’s a cookie and a browser session in HTTP, because the flash message builds upon these concepts:

A cookie is a way for the server to tell the client to store some information to be re-used later. What happens is that the server returns an HTTP header line similar to Set-Cookie: cookie-name=cookie-value. When the client receives that header line it will pass an HTTP header line similar to Cookie:  cookie-name=cookie-value to all future requests so the server will know which value it had send to the client (there are various options for expiring cookies etc but they are not important here). This may seem like a primitive solution however because HTTP is a stateless protocol that’s the only way for the server to store information about a client. If you disable cookies completely in a browser then there won’t be a way for the server to remember you, for example you won’t be able to login anywhere!

A session is a better way to store info about the client that builds upon the cookies. What happens is that when a client visits a site for the first time (so it has no cookies for that particular site) the server will send back a cookie named session_id (or something like that) containing a very big random number. The server will save this session id number in a persistent storage (for example in a database or a text file) and will correlate that number with information for that particular client. When the client sends back that session_id cookie the server will fetch the correlated info for that particular client from the persistent storage (and may update them etc). This way the server can store whatever info it wants about a particular client. The server usually keeps a a dictionary (map) of key-values for each session.

Now, a flash message is some information (message) that should be displayed to the user once. For example a message like “Your form has been submitted!”.

A simple way to implement this is to add a message attribute to the session when you want to display the flash message. The next page that is displayed (irrelevantly if there’s a redirect involved) will check to see if there’s a message attribute to the session; if yes it will display the actual message and remove the message attribute from the session (so it won’t be displayed again).

Implementing the form submission

Following the above guidelines I’ll present here a typical, production-ready form submission for PHP. Some choices I’ve made:

  • I am going to implement a POST form since a GET form doesn’t need any special handling
  • The form handler will be the same PHP page as the one displaying the form. This is a usual thing to do, you check the HTTP method and either display the form as-is (if it is GET) or handle the submission (if it is POST)
  • When the form is submitted successfully redirect to the same page and display a flash message
  • Check for valid input and display the error message

So without further ado here’s the complete php code that will submit your form; store it in a file named test.php:

<!DOCTYPE HTML>
<html>

<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <meta http-equiv="X-UA-Compatible" content="ie=edge">
  <title>TEST FORM</title>
  <link href="https://unpkg.com/tailwindcss@^1.0/dist/tailwind.min.css" rel="stylesheet">
</head>

<body class="bg-gray-100">
  <div class="px-8 py-8 w-1/2 m-auto">

    <?php
    session_start();
    if ($_SESSION['message'] ?? '') {
      echo '<div class="rounded bg-green-300 px-2 py-2">'.$_SESSION['message'].'</div>';
      unset($_SESSION['message']);
    }

    $nameErr = $wsErr = $name = $comment = "";
    $formValid = true;

    if ($_SERVER["REQUEST_METHOD"] == "POST") {
      if (empty($_POST["name"])) {
        $nameErr = "Name is required";
        $formValid = false;
      } else {
        $name = $_POST["name"];
      }

      if (empty($_POST["comment"])) {
        $comment = "";
      } else {
        $comment = $_POST["comment"];
      }

      if ($formValid) {
        if (1 == rand(1, 2)) {
          $_SESSION['message'] = "Success! You have submitted the values: <b>" . $name . " / " . $comment . "</b>";
          header('Location: ./test.php');
          exit();
        } else {
          $wsErr = "Error while trying to submit the form!";
        }
      }
    }
    ?>

    <h1 class="text-4xl font-bold text-indigo-500">Test a PHP form!</h1>

    <form class='border border-blue-800 rounded p-2' method="POST">

      <?= $wsErr ? "<div class='text-red-600 py-3'>" . $wsErr . "</div>" : "" ?>

      <div class="p-1">
        <label for="name">Name:</label> <input class="border border-blue-800 rounded" id="name" type="text" name="name" value="<?= $name ?>">
        <span class="text-red-600">* <?= $nameErr; ?></span>
      </div>

      <div class="p-1">
        <label for="comment">Comment:</label> <textarea class="border border-blue-800 rounded" id="comment" name="comment" rows="5" cols="40"><?= $comment ?></textarea>
      </div>

      <input class="rounded px-2 my-4 py-2 bg-blue-800 text-gray-100" type="submit" name="submit" value="Save">
    </form>
  </div>
</body>
</html>

So how is that working? As we can see there’s the php code first and the html is following (with some sprinkles of php). The HTML code is rather simple: It will display a POST form with two inputs named name and comment. Notice that we pass the $name and $comment php variables as their values. It also has a submit button and will display the $wsErr variable if it is not null (which means that there was an error while submitting the data). The PHP code now first starts the session (i.e it passes the session_id cookie to the client if such a cookie does not exist) checks to see if there’s a message attribute to the session. If such a message exists it will display it in a rounded green panel and remove that from the session (so it won’t be displayed again next time):

session_start();
if ($_SESSION['message'] ?? '') {
  echo '<div class="rounded bg-green-300 px-2 py-2">'.$_SESSION['message'].'</div>';
  unset($_SESSION['message']);
}

After that there are some variable initializations and we check if the HTTP request is POST (if the request is GET we’ll just disply the HTML):

if ($_SERVER["REQUEST_METHOD"] == "POST") {

For each of the inputs we check if they are empty or not and assign their values to the corresponding php variable. If the name is empty then we’ll set $formValid = false; and add an error message since this field is required. Then, if $formValid is not false we can do the actual action (for example write to the database). I’ve simulated that using a coin-toss with rand (so there’s a 50% possibility that the action will fail). If the action “failed” then nothing happened in the database so we should return the same page with the wsErr variable containing the error.

However if the action is successful that means that the data has been inserted to the database so we’ll need to set the flash message and do the redirect (the name of the page containing the form is /test.php so we’ll redirect to it):

$_SESSION['message'] = "Success! You have submitted the values: <b>" . $name . " / " . $comment . "</b>";
header('Location: ./test.php');
exit();

The two commands above (header and exit) will do the actual redirect in php. Since the session contains the message it will be displayed after the redirect has finished!

Conclusion

So following the above tutorial, here’s what you should absolutely do to submit an HTML form:

  • Use HTTP POST if the form is going to change data on the server; use HTTP GET otherwise (mainly for search/filter forms)
  • When using POST: Redirect when the form is valid and the action on the server has finished successfully; never return a 200 OK status when you’ve changed things in the server (database)
  • Use flash messages to pass information to the user after a redirect

Declarative Ecto query filters

Continuing my Elixir journey I’d like to discuss here a method to implement one of my favorite Django features: Declarative query filters. This functionality is not a core Django feature but it is offered through the excellent django-filter package: Using this, you can create a Filter class which defines which fields are to be used for filtering the queryset and how each field will be queried (i.e using things like exact, like, year of date etc).

This is a functionality I am greatly missing in Elixir/phoenix so I’ve tried implementing it on my own. Of course, django-filter has various other capabilities that result from the implicit generation of things that Django offers like automatically creating the html for the declared fields, automatically declare the fields based on their types etc but such things are not supported by phoenix in any case so I won’t be trying in them here.

During my research I’ve seen a bunch of blog posts or packages about this thing however they didn’t properly support joins (i.e you could only filter on fields on a specific schema) or needed too much work to filter on joins (i.e define different filters for each part of the join). In the solution I’ll present here you’ll just define a filter for the specific query you need to filter no matter how many joins it has (just like in django-filters).

What will it do

The solution is more or less a self contained Elixir module named QueryFilterEx that can be used to declaratively filter your queries. To use that you’ll need to declare your filters using a simple array of maps. The filters should then be added in your form using a different input for each filter; then your queryset will be filtered with all the values you’ve added to the inputs using AND.

The module has a very simple API consisting of three functions:

  • get_changeset_from_params(params, filters): Pass it the GET request parameters you got from your form and the declared filters array to return you a proper changeset (which you can then use to build your form in your html)
  • make_filter_changeset(filters, params): This function actually generates the changeset using the filters and a Map of filter_name: value pairs (it is actually used by get_changeset_from_params)
  • filter(query, changeset, filters): Filter the query using the previously created changeset and the declared filters array

You can find a sample of the technique presented in this article in my PHXCRD repository: https://github.com/spapas/phxcrd for example in the user_controller or authority_controller.

Preparing the query

In order to use the QueryFilterEx module you’ll need to properly “prepare” your Ecto query. By preparing I don’t mean a big deal just the fact that you’ll need to name all your relations (or at least name all the relations you’re going to use for filtering). This is very simple to do, for example for the following query:

from(a in Authority,
  join: ak in AuthorityKind,
  on: [id: a.authority_kind_id],
  preload: [authority_kind: ak]
)

you can name the relations by adding two as: atoms like this:

from(a in Authority, as: :authority,
  join: ak in AuthorityKind, as: :authority_kind,
  on: [id: a.authority_kind_id],
  preload: [authority_kind: ak]
)

So after each join: you’ll add a name for your joined relation (and also add a name for your initial relation). Please notice that you can use any name you want for these (not related to the schema names).

Declaring the filters

To declare the filters you’ll just add an array of simple Elixir maps. Each map must have the following fields:

  • :name This is the name of the specific filter; it is mainly used in conjunction with the queryset and the form fields to set initial values etc
  • :type This is the type of the specific filter; it should be a proper Ecto type like :string, :date, :integer etc. This is needed to properly cast the values and catch errors
  • :binding This is the name of the relation this filter concerns which you defined in your query using :as (discussed in previous section)
  • :field_name This is the actual name of the field you want to filter on
  • :method How to filter on this field; I’ve defined a couple of methods I needed but you can implement anything you want

The methods I’ve implemented are the following:

  • :eq Equality
  • :ilike Field value starts with the input - ignore case
  • :icontains Field value contains the input - ignore case
  • :year Field is a date or datetime an its year is the same as the value
  • :date Field is a datetime and its date part is equal to the value

Anything else will just be compared using = (same as :eq).

Integrating it with a controller

As an example let’s see how QueryFilterEx is integrated it with the phxcrd user_controller. The query I’d like to filter on is the following (see that everything I’ll need is named using :as:

from(u in User, as: :user,
  left_join: a in Authority, as: :authority,
  on: a.id == u.authority_id,
  left_join: up in UserPermission,
  on: up.user_id == u.id,
  left_join: p in Permission, as: :permission,
  on: up.permission_id == p.id,
  preload: [authority: a, permissions: p]
)

To declare the filters I like to create a module attribute ending with filters, something like @user_filters for example. Here’s the filters I’m going to use for user_controller:

@user_filters [
  %{name: :username, type: :string, binding: :user, field_name: :username, method: :ilike},
  %{name: :authority_name, type: :string, binding: :authority, field_name: :name, method: :icontains},
  %{name: :permission_name, type: :string, binding: :permission, field_name: :name, method: :ilike},
  %{name: :last_login_date, type: :date, binding: :user, field_name: :last_login, method: :date}
]

So it will check if the user.username and permission.name start with the passed value, authority.name contains the passed value and if the user.login_date (which is a datetime) is the same as the passed date value.

Finally, here’s the full code of the index controller:

def index(conn, params) do
  changeset = QueryFilterEx.get_changeset_from_params(params, @user_filters)

  users =
    from(u in User,
      as: :user,
      left_join: a in Authority, as: :authority,
      on: a.id == u.authority_id,

      left_join: up in UserPermission,
      on: up.user_id == u.id,
      left_join: p in Permission, as: :permission,
      on: up.permission_id == p.id,
      preload: [authority: a, permissions: p]
    )
    |> QueryFilterEx.filter(changeset, @user_filters)
    |> Repo.all()

  render(conn, "index.html", users: users, changeset: changeset)
end

It is very simple, it just uses the get_changeset_from_params method I discussed before to generate the changeset and then uses it to filter the query. Also please notice that it passes the changeset to the template to be properly rendered in the filter form.

The template

The template for the user index action is the following:

<%= form_for @changeset, AdminRoutes.user_path(@conn, :index), [method: :get, class: "filter-form", as: :filter],  fn f -> %>
  <%= label f, :username, gettext "Username" %>
  <%= text_input f, :username  %>

  <%= label f, :authority_name, gettext "Authority name" %>
  <%= text_input f, :authority_name  %>

  <%= label f, :permission_name, gettext "Permission name" %>
  <%= text_input f, :permission_name  %>

  <%= label f, :last_login_date, gettext "Last login date" %>
  <%= text_input f, :last_login_date  %>
  <%= error_tag f, :last_login_date %>

  <%= submit gettext("Filter"), class: "ml-5" %>
  <%= link gettext("Reset"), to: AdminRoutes.user_path(@conn, :index), class: "button button-outline ml-2" %>
<% end %>
<%= for user <- @users do %>
<!-- display the user info -->
<% end %>

Notice that it gets the @changeset and uses it to properly fill the initial values and display error messages. For this case I’ve only added an error_tag for the :last_login_date field, the others since are strings do not really need it since they will accept all values.

Also, the form method form must be :get since we only filter (not change anything) and I’ve passed the as: :filter option to the form_for to collect the parameters under the filter server side parameter (this can be anything you want and can be optionally be passed to QueryFilterEx.get_changeset_from_params to know which parameter the filters are collected on).

How does this work?

In this section I’ll try to explain exactly how the QueryFilterEx module works. Before continuing I want to thank the people at the Elixir forum and #elixir-lang Freenode IRC chat that helped me with understanding how to be able to create dynamic bindings.

So I’ll split this explanation in two parts: Explain QueryFilterEx.get_changeset_from_params and make_filter_changeset (easy) and then explain QueryFilterEx.filter (more difficult).

QueryFilterEx.get_changeset_from_params and make_filter_changeset

This function generates a changeset using the GET request parameters and the list of declared filters. The create changeset is a schemaless one since it may contains fields of various schemas (or fields that are not even exist on a schema). To generate it it uses the cast/4 function passing it a {data, types} first parameter to generate the schemaless changeset. It has two public methods: get_changeset_from_params and make_filter_changeset. The get_changeset_from_params is the one we’ve used to integrate with the controller and is used to retrieve the filter parameters from the request parameters based on the collect parameter of the form we mentioned before (the as: :filter). If such parameters are found they will be passed to make_filter_changeset (or else it will pass an empty struct). Notice that the filter_name by default is "filter" but you can change it to anything you want.

def get_changeset_from_params(params, filters, filter_name \\ "filter") do
  case params do
    %{^filter_name => filter_params} ->
      filters |> make_filter_changeset(filter_params)

    _ ->
      filters |> make_filter_changeset(%{})
  end
end

The make_filter_changeset is the function that actually creates the schemaless changeset. To do that it uses two private functions that operate on the passed filters array: make_filter_keys to extract the :name field of each key filter and the make_filter_types to generate a Map of %{name: :type} as needed by the types of the {data, types} tuple passed to cast (the data is just an empty Map):

defp make_filter_keys(filters) do
  filters |> Enum.map(& &1.name)
end

defp make_filter_types(filters) do
  filters |> Enum.map(&{&1.name, &1.type}) |> Map.new()
end

def make_filter_changeset(filters, params) do
  data = %{}
  types = filters |> make_filter_types

  {data, types}
  |> Ecto.Changeset.cast(params, filters |> make_filter_keys)
  |> Map.merge(%{action: :insert})
end

One interesting thing here is the Map.merge(%{action: :insert}) that is piped to the generated changeset. This is needed to actually display the validation errors, if there’s no action to the changeset (and there won’t be since we aren’t going do any updates to the database with this changeset) then the casting errors won’t be displayed.

Please notice that although I use the get_changeset_from_params in my controller the important function here is the make_filter_changeset. The get_changeset_from_params is mainly used to retrieve the filter-related GET query parameter; however to use QueryFilterEx you can just create (however you want) a Map of filter_name: value pairs and pass it to make_filter_changeset to get the changeset.

QueryFilterEx.filter

The filter method gets three parameters. The query, the changeset (that was created with make_filter_changeset) and the declared filters. This function will then check all declared filters one by one and see if the changeset contains a change for this filter (i.e if the field has a value). If yes it will append a where/3 to the query based on the passed value of the changeset and the declared filter :method.

To do that it just uses Enum.reduce starting with the initial query as an accumulator and reducing on all the declared filters:

def filter(query, changeset, filters) do
  changes = Map.fetch!(changeset, :changes)
  filters |> Enum.reduce(query, creat_where_clauses_reducer(changes))
end

defp creat_where_clauses_reducer(changes) do
  fn %{name: name, field_name: field_name, binding: binding, method: method}, acc ->
    case Map.fetch(changes, name) do
      {:ok, value} ->
        acc |> creat_where_clause(field_name, binding,  method, value)

      _ ->
        acc
    end
  end
end

Notice that the creat_where_clauses_reducer function returns a function (the reducer) that reduce will use. This function checks to see if the current changes of the changeset contain the filter_name:. If yes it will pass the following values to the creat_where_clause function:

  • The accumulated query (acc)
  • The field_name:, :binding and :method values of the current filter
  • The value of the changes of the changeset

If the current filter_name is not contained in the changes then it just returns the accumulated query as it is.

Let’s now take a look at the creat_where_clause function:

defp creat_where_clause(acc, field_name, binding,  method, value) do
  case method do
    :eq -> acc |> where(
      [{^binding, t}],
      field(t, ^field_name) == ^value
    )
    :ilike -> acc |> where(
      [{^binding, t}],
      ilike(field(t, ^field_name), ^("#{value}%") )
    )
    :icontains -> acc |> where(
      [{^binding, t}],
      ilike(field(t, ^field_name), ^("%#{value}%") )
    )
    :year -> acc  |> where(
      [{^binding, t}],
      fragment("extract (year from ?) = ?", field(t, ^field_name), ^value)
    )
    :date -> acc  |> where(
      [{^binding, t}],
      fragment("? >= cast(? as date) and ? < (cast(? as date) + '1 day'::interval"), field(t, ^field_name), ^value, field(t, ^field_name), ^value)
    )
    _ -> acc |> where(
      [{^binding, t}],
      field(t, ^field_name) == ^value
    )

  end
end

This function is just a simple case that pipes the accumulated query to a different where clause depending on the method:. Let’s take a closer look at what happens when :method == :eq:

acc |> where(
  [{^binding, t}],
  field(t, ^field_name) == ^value
)

This may seem a little confusing so let’s take a look at a simple where first:

from(u in User) |> where([u], u.name == "root") |> Repo.all()

Nothing fancy here, now let’s add a named query:

from(u in User, as: :user) |> where([user: u], u.name == "root") |> Repo.all()

Notice that now we can declare that u is an alias for the users named binding. What if we used the tuples syntax for the user: u instead of the keyword one:

from(u in User, as: :user) |> where([{:user, u}], u.name == "root") |> Repo.all()

Yes this still works. What if we wanted to use a variable for the binding name in the where?

binding = :user
from(u in User, as: :user) |> where([{^binding, u}], u.name == "root") |> Repo.all()

I think it starts to make sense now, let’s finally use a variable for the field name also:

binding = :user
field_name = :name
from(u in User, as: :user) |> where([{^binding, u}], field(u, ^field_name) == "root") |> Repo.all()

So this is exactly how this works!

Beyond the :eq I’ve got the definitions for the other methods I described there, the most complex one is probably the :date which is something like:

where(
  [{^binding, t}],
  fragment("? >= cast(? as date) and ? < (cast(? as date) + '1 day'::interval"), field(t, ^field_name), ^value, field(t, ^field_name), ^value)
)

What this does is that it generates the following SQL fragment:

field_name >= cast(value as date) AND field_name < (cast(value as date) + '1 day'::interval)

You can add your own methods by adding more clauses to the case of the creat_where_clause function and following a similar pattern.

Conclusion

By using the QueryFilterEx module presented here you can very quickly declare the fields you want to filter on and the method you want to use for each field no matter if these fields are in the same schema or are accessed through joins. You can easily extend the functionality of the module by adding your own methods. The only extra thing you need to do is to just add names to your queries.

Phoenix forms integration with select2 and ajax

During the past months I’ve tried to implement a project using Elixir and the Phoenix framework. Old visitors of my blog will probably remember that I mainly use Django for back-end development but I decided to also give Phoenix a try.

My first impressions are positive but I don’t want to go into detail in this post; I’ll try to add a more extensive post comparing Elixir / Phoenix with Python / Django someday.

The problem that this particular post will try to explain is how to properly integrate a jQuery select2 dropdown ajax with autocomplete search to your Phonix Forms. This seems like a very common problem however I couldn’t find a proper solution anywhere in the internet. It seems that most people using Phoenix prefer to implement their autocompletes using SPA like functionality (react etc). Also I found this project that seems to be working, however it does not use select2 and I really didn’t like to mess with a different JS library for reasons that should be too obvious to most people.

So here we’ll implement a simple solution for allowing your foreign key value to be autocompleted through ajax using select2. The specific example is that you have a User that belongs to an Authority i.e user has a field named authority_id which is a foreign key to authority. We’ll add a functionality to the user edit form to select the authority using ajax-autocomplete.

Please notice that you can find a working version of this tutorial in my Phoenix Crud template project: https://github.com/spapas/phxcrd. This project contains various other functionality that I need but you should be able to test the user - authority integration by following the instructions there.

The schemas

For this tutorial, we’ll use two schemas: A User and an Authority. Each User belongs to an Authority (thus will have a foreign key to Authority; that’s what we want to set using the ajax select2). Here are the ecto schemas for these entities:

defmodule Phxcrd.Auth.Authority do
  use Ecto.Schema

  import Ecto.Changeset
  alias Phxcrd.Auth.User

  schema "authorities" do
    field :name, :string
    has_many :users, User, on_replace: :nilify
    timestamps()
  end

  @doc false
  def changeset(authority, attrs) do
    authority
    |> cast(attrs, [:name])
    |> validate_required([:name], message: "The field is required")
    |> unique_constraint(:name, message: "The name already exists!")
  end

  use Accessible
end
defmodule Phxcrd.Auth.User do
  use Ecto.Schema

  import Ecto.Changeset
  alias Phxcrd.Auth.Authority

  schema "users" do
    field :email, :string
    field :username, :string
    field :password_hash, :string
    field :password, :string, virtual: true

    belongs_to :authority, Authority

    timestamps()
  end

  @doc false
  def changeset(user, attrs) do
    user
    |> cast(attrs, [:username, :email, :authority_id])
    |> validate_required([:username, :email ])
  end

  use Accessible
end

Notice that both these entities are contained in the Auth context and were created using mix phx.gen.html; I won’t include the migrations here.

The search API

Let’s now take a look at the search api for Authority. I’ve added an ApiController which contains the following function:

def search_authorities(conn, params) do
    q = params["q"]

    authorities =
      from(a in Authority,
        where: ilike(a.name, ^"%#{q}%")
      )
      |> limit(20)
      |> Repo.all()

    render(conn, "authorities.json", authorities: authorities)
end

Notice that this retrieves a q parameter and makes an ilike query to Authority.name. It then passes the results to the view for rendering. Here’s the corresponding function for ApiView:

def render("authorities.json", %{authorities: authorities}) do
    %{results: Enum.map(authorities, &authority_json/1)}
  end

  def authority_json(a) do
    %{
      id: a.id,
      text: a.name
    }
end

Notice that select2 wants its results in a JSON struct with the following form {results: [{id: 1, name: "Authority 1"}]}.

To add this controller action to my routes I’ve added this to router.ex:

scope "/api", PhxcrdWeb do
    pipe_through :api

    get "/search_authorities", ApiController, :search_authorities
end

Thus if you visit http://127.0.0.1/search_authorities?q=A you should retrieve authorities containing A in their name.

The controller

Concenring the UserController I’ve added the following methods to it for creating and updating users:

def new(conn, _params) do
  changeset = Auth.change_user(%User{})
  render(conn, "new.html", changeset: changeset)
end

def create(conn, %{"user" => user_params}) do
  case Auth.create_user(user_params) do
    {:ok, user} ->
      conn
      |> put_flash(:info, "#{user.name} created!")
      |> redirect(to: Routes.user_path(conn, :show, user))

    {:error, changeset} ->
      render(conn, "new.html", changeset: changeset)
  end
end

def edit(conn, %{"id" => id}) do
  user = Auth.get_user!(id)
  changeset = Auth.change_user(user)
  render(conn, "edit.html", user: user, changeset: changeset)
end

def update(conn, %{"id" => id, "user" => user_params}) do
  user = Auth.get_user!(id)

  user_params = Map.merge(%{"authority_id" => nil}, user_params)

  case Auth.update_user(user, user_params) do
    {:ok, user} ->
      conn
      |> put_flash(:info, "User updated successfully.")
      |> redirect(to: Routes.user_path(conn, :show, user))

    {:error, %Ecto.Changeset{} = changeset} ->
      render(conn, "edit.html", user: user, changeset: changeset)
  end
end

Most of these are more or less the default things that mix phx.gen.html creates. One thing that may seem a strange here is the user_params = Map.merge(%{"authority_id" => nil}, user_params) line of update. What happens here is that I want to be able to clear the authority of a user (I’ll explain how in the next sections). If I do that then the user_params that is passed to update will not contain an authority_id key thus the authority_id won’t be changed at all (so even though I cleared it, it will keep its previous value after I save it). To fix that I set a default value of nil to authority_id; if the user has actually selected an authority from the form this will be overriden when merging the two maps. So the resulting user_params will always contain an authority_id key, either set to nil or to the selected authority.

Beyond that I wont’ go into detail explaining the above functions, but if something seems strange feel free to ask. I also won’t explain the Auth.* functions; all these are created by phoenix in the context module.

The view

The UserView module contains a simple but very important function:

def get_select_value(changeset, attr) do
  case changeset.changes[attr] do
    nil -> Map.get(changeset.data, attr)
    z -> z
  end
end

This functions gets two parameters: The changeset and the name of the attribute (:authority_id in our case). What it does is to first check if this attribute is contained in the changeset.changes; if yes it will return that value. If it isn’t contained in the changeset.changes then it will return the value of changeset.data for that attribute.

This is a little complex but let’s try to understand its logic: When you start editing a User you want to display the current authority of that instance. However, when you submit an edited user and retrieve an errored form (for example because you forgot to fill the username) you want to display the authority that was submitted in the form. So the changeset.changes contains the changes that were submitted just before while the changeset.data contain the initial value of the struct.

Update 02/07/2019: Please notice that instead of using the get_select_value I presented before you can use the Ecto.Changeset.get_field function that does exactly this! So get_select_value could be defined like this:

def get_select_value(changeset, attr) do
  changeset |> Ecto.Changeset.get_field(attr)
end

The form template

Both the :new and :edit actions include a common form.html.eex template:

<%= form_for @changeset, @action, fn f -> %>
  <%= if @changeset.action do %>
  <div class="alert alert-danger">
    <p><%= gettext("Problems while saving") %></p>
  </div>
  <% end %>
  <div class='row'>
    <div class='column'>
      <%= label f, :username %>
      <%= text_input f, :username %>
      <%= error_tag f, :username %>
    </div>
    <div class='column'>
      <%= label f, :email %>
      <%= text_input f, :email %>
      <%= error_tag f, :email %>
    </div>
  </div>

  <div class='row'>
    <div class='column'>
      <%= label f, :authority %>
      <%= select(f,
        :authority_id, [
          (with sv when not is_nil(sv) <- get_select_value(@changeset, :authority_id),
                                     a <- Phxcrd.Auth.get_authority!(sv), do: {a.name, a.id})
        ],
        style: "width: 100%")
        %>
      <%= error_tag f, :authority_id %>
    </div>

  </div>

  <div>
    <%= submit gettext("Save") %>
  </div>
<% end %>

This is a custom Phoenix form but it has the following addition which is more or less the meat of this article (along with the get_select_value function I explained before):

select(f, :authority_id, [
        (with sv when not is_nil(sv) <- get_select_value(@changeset, :authority_id),
                                   a <- Phxcrd.Auth.get_authority!(sv), do: {a.name, a.id})
      ],
      style: "width: 100%")

So this will create an html select element which will contain a single value (the array in the third parameter of select): The authority of that object or the authority that the user had submitted in the form. For this it uses get_select_value to retrieve the :authority_id and if it’s not nil it passes it to get_authority! to retrieve the actual authority and return a tuple with its name and id.

By default when you create a select element you’ll pass an array of all options in the third parameter, for example:

select(f, :authority_id, Phxcrd.Auth.list_authorities |> Enum.map(&{&1.name, &1.id}))

Of course this beats the purpose of using ajax since all options will be rendered.

The final step is to add the required custom javascript to convert that select to select2-with-ajax:

$(function () {
    $('#user_authority_id').select2({
      allowClear: true,
      placeholder: 'Select authority',
      ajax: {
        url: '<%= Routes.api_path(@conn, :search_authorities) %>',
        dataType: 'json',
        delay: 150,
        minimumInputLength: 2
      }
    });
})

The JS very rather simple; the allowClear option will display an x so that you can clear the selected authority while the ajax url will be that of the :search_authorities.

Conclusion

Although this article may seem a little long, as I’ve already mentioned the most important thing to keep is how to properly set the value that should be displayed in your select2 widget. Beyond that everything is a walk in the park by following the docs.

How to create a custom filtered adapter in Android

Introduction

Android offers a nice component named AutoCompleteTextView that can be used to auto-fill a text box from a list of values. In its simplest form, you just create an array adapter passing it a list of objects (that have a proper toString() method). Then you type some characters to the textbox and by default it will filter the results searching in the beginning of the backing object’s toString() result.

However there are times that you don’t want to look at the beginning of the string (because you want to look at the middle of the string) or you don’t want to just to search in toString() method of the object or you want to do some more fancy things in object output. For this you must override the ArrayAdapter and add a custom Filter.

Unfurtunately this isn’t as straightforward as I’d like and I couldn’t find a quick and easy tutorial on how it can be done.

So here goes nothing: In the following I’ll show you a very simple android application that will have the minimum viable custom filtered adapter implementation. You can find the whole project in github: https://github.com/spapas/CustomFilteredAdapeter but I am going to discuss everything here also.

The application

Just create a new project with an empty activity from Android Studio. Use kotlin as the language.

The layout

I’ll keep it as simple as possible:

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout
        xmlns:android="http://schemas.android.com/apk/res/android"
        xmlns:tools="http://schemas.android.com/tools"
        android:orientation="vertical"
        android:layout_width="match_parent" android:layout_height="match_parent"
        tools:context=".MainActivity">
    <TextView
            android:layout_width="match_parent" android:layout_height="wrap_content"
            android:text="Hello World!"
            android:textSize="32sp"
            android:textAlignment="center"/>
    <AutoCompleteTextView
            android:layout_marginTop="32dip"
            android:layout_width="match_parent" android:layout_height="wrap_content"
            android:id="@+id/autoCompleteTextView"/>
</LinearLayout>

You should just care about the AutoCompleteTextView with an id of autoCompleteTextView.

The backing data object

I’ll use a simple PoiDao Kotlin data class for this:

data class PoiDao(
    val id: Int,
    val name: String,
    val city: String,
    val category_name: String
)

I’d like to be able to search to both name, city and category_name of each object. To create a list of the pois to be used to the adapter I can do something like:

val poisArray = listOf(
    PoiDao(1, "Taco Bell", "Athens", "Restaurant"),
    PoiDao(2, "McDonalds", "Athens","Restaurant"),
    PoiDao(3, "KFC", "Piraeus", "Restaurant"),
    PoiDao(4, "Shell", "Lamia","Gas Station"),
    PoiDao(5, "BP", "Thessaloniki", "Gas Station")
)

The custom adapter

This will be an ArrayAdapter<PoiDao> implementing also the Filterable interface:

inner class PoiAdapter(context: Context, @LayoutRes private val layoutResource: Int, private val allPois: List<PoiDao>):
    ArrayAdapter<PoiDao>(context, layoutResource, allPois),
    Filterable {
    private var mPois: List<PoiDao> = allPois

    override fun getCount(): Int {
        return mPois.size
    }

    override fun getItem(p0: Int): PoiDao? {
        return mPois.get(p0)
    }

    override fun getItemId(p0: Int): Long {
        // Or just return p0
        return mPois.get(p0).id.toLong()
    }

    override fun getView(position: Int, convertView: View?, parent: ViewGroup?): View {
        val view: TextView = convertView as TextView? ?: LayoutInflater.from(context).inflate(layoutResource, parent, false) as TextView
        view.text = "${mPois[position].name} ${mPois[position].city} (${mPois[position].category_name})"
        return view
    }

    override fun getFilter(): Filter {
        // See next section
    }
}

You’ll see that we add an instance variable named mPois that gets initialized in the start with allPois (which is the initial list of all pois that is passed to the adapter). The mPois will contain the filtered results. Then, for getCount and getItem we return the corresponding valeus from mPois; the getItemId is used when you have an sqlite backed adapter but I’m including it here for completeness.

The getView will create the specific line for each item in the dropdown. As you’ll see the layout that is passed must have a text child which is set based on some of the attributes of the corresponding poi for each position. Notice that we can use whatever view layout we want for our dropdown result line (this is the layoutResource parameter) but we need to configure it (i.e bind it with the values of the backing object) here properly.

Finally we create a custom instance of the Filter, explained in the next section.

The custom filter

The getFilter creates an object instance of a Filter and returns it:

override fun getFilter(): Filter {
    return object : Filter() {
        override fun publishResults(charSequence: CharSequence?, filterResults: Filter.FilterResults) {
            mPois = filterResults.values as List<PoiDao>
            notifyDataSetChanged()
        }

        override fun performFiltering(charSequence: CharSequence?): Filter.FilterResults {
            val queryString = charSequence?.toString()?.toLowerCase()

            val filterResults = Filter.FilterResults()
            filterResults.values = if (queryString==null || queryString.isEmpty())
                allPois
            else
                allPois.filter {
                    it.name.toLowerCase().contains(queryString) ||
                    it.city.toLowerCase().contains(queryString) ||
                    it.category_name.toLowerCase().contains(queryString)
                }
            return filterResults
        }
    }
}

This object instance overrides two methods of Filter: performFiltering and publishResults. The performFiltering is where the actual filtering is done; it should return a FilterResults object containing a values attribute with the filtered values. In this method we retrieve the charSequence parameter and converit it to lowercase. Then, if this parameter is not empty we filter the corresponding elements of allPois (i.e name, city and category_name in our case) using contains. If the query parameter is empty then we just return all pois. Warning java developers; here the if is used as an expression (i.e its result will be assigned to filterResults.values).

After the performFiltering has finished, the publishResults method is called. This method retrieves the filtered results in its filterResults parameter. Thus it sets mPois of the custom adapter is set to the result of the filter operation and calls notifyDataSetChanged to display the results.

Using the custom adapter

To use the custom adapter you can do something like this in your activity’s onCreate:

override fun onCreate(savedInstanceState: Bundle?) {
    super.onCreate(savedInstanceState)
    setContentView(R.layout.activity_main)

    val poisArray = listOf(
        // See previous sections
    )
    val adapter = PoiAdapter(this, android.R.layout.simple_list_item_1, poisArray)
    autoCompleteTextView.setAdapter(adapter)
    autoCompleteTextView.threshold = 3

    autoCompleteTextView.setOnItemClickListener() { parent, _, position, id ->
        val selectedPoi = parent.adapter.getItem(position) as PoiDao?
        autoCompleteTextView.setText(selectedPoi?.name)
    }
}

We create the PoiAdapter passing it the poisArray and android.R.layout.simple_list_item_1 as the layout. That layout just contains a textview named text. As we’ve already discussed you can pass something more complex here. The thresold defined the number of characters that the user that needs to enter to do the filtering (default is 2).

Please notice that when the user clicks (selects) on an item of the dropdown we set the contents of the textview (or else it will just use the object’s toString() method to set it).