When you need to support user-uploaded files from Django (usually called media) you will probably use a FileField in your models. This is translated to a simple varchar (text) field in the database that contains a unique identifier for the file. This usually would be the path to the file, however this is not always the case!

What really happens is that Django has an underlying concept called a File Storage which is a class that has information on how to talk to the actual storage backend, and particularly how to translate the unique identifier stored on the db to an actual file object. By default Django stores files in the file system using the FileSystemStorage however it is possible to use different backends through an add-on (for example Amazon S3) or even write your own.

Each FileField can be configured to use a different storage backend by passing the storage parameter; if you don’t use this parameter then the default storage backend is used. So you can easily configure a FileField that would upload files to your filesystem and another one that would upload files to S3.

However, one thing that is not supported though is to use multiple storages for the same FileField depending on some parameter of the model instance. Unfortunately, in a recent project I had to do exactly that: We had a FileField on a model that contained hundreds of GBs of files stored on the filesystem; we wanted to be able to upload the files of new instances of that model on S3 but also wanted to keep the old files on the filesystem to avoid moving all these to S3 (which would result to a lot of downtime). I also wanted a way to be “flexible” on this i.e to be able to change again the storage backend for some instances if needed and definitely not move/copy all these files!

If you take a peek at the FileField options you’ll see that there’s a storage parameter that can be a callable. However this callable is initialized with the models and is not evaluated again until the app is restarted so it can’t be used to decide on the storage for each model instance.

The only thing that is evaluated each time a file is uploaded through the FileField is when upload_to is a function. This function receives the model instance and returns the path that the file will be uploaded to.

The idea is to use this upload_to function to return a different path depending on the model instance and then use a custom storage backend that will use the path to decide on the actual storage backend to use.

This is the code I ended up with for the upload_to function:

def file_upload_path(instance, filename):
    dt_str = app.created_on.strftime("%Y/%m/%d")
    file_storage = ""

    if instance.id >= settings.STORAGE_CHANGE_ID: 
        file_storage = settings.STORAGE_SELECTION_STR + "/"

    return "protected/{0}{1}/{2}/{3}".format(file_storage, dt_str, instance.id, filename)

class Model(models.Model):
    file = models.FileField(upload_to=file_upload_path)

What happens here is that I have a setting STORAGE_CHANGE_ID that is the id of the instance after which all instances will use the different storage backend. You can use whatever method you want here to decide on the storage that would be used; the only thing to keep in mind is to put the storage somewhere on the returned path.

I also have a setting STORAGE_SELECTION_STR that is the string that will be used in the path to differentiate the storage backend. The STORAGE_SELECTION_STR has the value of minios3 for this project.

Using this function the paths of the instances that are >= STORAGE_CHANGE_ID will be of the form protected/minios3/2021/04/11/1234/filename.ext while for the old files these will be of the form protected/2021/04/11/1234/filename.ext. Notice the minios3 string in between.

Of course this is not enough. We also need to tell Django to use the different storage backend for the new files. In order to do this we have to implement a custom storage class like this:

from django.core.files.storage import FileSystemStorage, Storage
from storages.backends.s3boto3 import S3Boto3Storage
from django.conf import settings


class FilenameBasedStorage(Storage):
    minio_choice = settings.STORAGE_SELECTION_STR

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def _open(self, name, mode="rb"):
        if self.minio_choice in name:
            return S3Boto3Storage().open(name, mode)
        else:
            return FileSystemStorage().open(name, mode)

    def _save(self, name, content):
        if self.minio_choice in name:
            return S3Boto3Storage().save(name, content)
        else:
            return FileSystemStorage().save(name, content)

    def delete(self, name):
        if self.minio_choice in name:
            return S3Boto3Storage().delete(name)
        else:
            return FileSystemStorage().delete(name)

    def exists(self, name):
        if self.minio_choice in name:
            return S3Boto3Storage().exists(name)
        else:
            return FileSystemStorage().exists(name)

    def size(self, name):
        if self.minio_choice in name:
            return S3Boto3Storage().size(name)
        else:
            return FileSystemStorage().size(name)

    def url(self, name):
        if self.minio_choice in name:
            return S3Boto3Storage().url(name)
        else:
            return FileSystemStorage().url(name)

    def path(self, name):
        if self.minio_choice in name:
            return S3Boto3Storage().path(name)
        else:
            return FileSystemStorage().path(name)

This class should be self explainable: It uses the settings.STORAGE_SELECTION_STR we mentioned above to decide which storage backend to use and then it forwards each method to the corresponding backend (either the filesystem storage or the S3 storage).

One thing to notice is that the django.core.files.storage.Storage class this class inherits from has more methods that can be implemented (and would raise if called without implementing them) however this implementation works fine for my needs.

One question some readers may have is what happens if the user uploads a file named test-minios3.pdf (i.e a file containing the STORAGE_SELECTION_STR). Well you may just as well ignore it; it will just be saved always on the minio-s3 storage backend. Or you can make sure to remove that string from the filename before saving it on the file_upload_path. I chose to ignore it since it doesn’t matter for my use case.

Finally, we need to tell Django to use this storage class for the file field. We can do this by adding it to the FileField like:

file = models.FileField(upload_to=file_upload_path, storage=FilenameBasedStorage)

or we can configure it on the DEFAULT_FILE_STORAGE setting (for Django < 4.2) or on the STORAGES dict (for Django >= 4.2).

I hope this helps someone else that needs to do something similar!

Over the past few years, there has been a surge in the popularity of frontend frameworks, such as React and Vue. While there are certainly valid use cases for these frameworks, I believe that they are often unnecessary, as most web applications can be adequately served by traditional request/response web pages without any frontend framework. The high usage of these frameworks is largely driven by FOMO and lack of knowledge about alternatives. However, using such frameworks can add unnecessary complexity to your project, as you now have to develop two projects in parallel (the frontend and the backend) and maintain two separate codebases.

That being said, I understand that some projects may require extra UX enhancements such as modals, navigation and form submissions without full page reloads, immediate form validation feedback, page fragment updates etc. If you want some of this functionality but do not want to hop on the JS framework train, you can use the Unpoly library.

Unpoly is similar to other libraries like intercooler, htmx or turbo however I find it to be the easiest to be used in the kind of projects I work on. These libraries allow you to write dynamic web applications with minimal changes to your existing server-side code.

In this guide, we’ll go over how to use Unpoly with Django. Specifically, we’ll cover the following topics:

• An unpoly demo
• Integrating unpoly with Django
• Navigation improvements
• Form improvements
• Modal improvements (layers)
• Integration with (some) django packages
• More advanced concepts

The unpoly demo

Unpoly provides a demo application written in Ruby. You can go on and play with it for a bit to understand what it offers compared to a traditional web app.

I’ve re-implemented this in Django so you can compare the code with a non-unpoly Django app. It can be found on https://github.com/spapas/django-unpoly-demo and the actual demo in Django is at: https://unpoly-demo.spapas.net or https://unpoly-demo.fly.dev/ (deployed on fly.io) or https://unpoly-demo.onrender.com/ (deployed on render.com; notice the free tier of render.com is very slow, this isn’t related to the app). The demo app uses an ephemeral database so the data may be deleted at any time.

Try navigating the demo site and you’ll see things like:

• Navigation feedback
• Navigation without page reloads
• Forms opening in modals
• Modals over modals
• Form submissions without page reloads
• Form validation feedback without page reloads

All this is implemented mostly with traditional Django class based views and templates in addition to a few unpoly attributes.

To understand how much of a difference this makes, after you have taken a peek at the “companies” functionality in the demo, take a look at the actual code that implementation (I’m only pasting the views, the other components are exactly the same as in a normal Django app):

class FormMixin:
    def form_valid(self, form):

        if form.is_valid() and not self.request.up.validate:
            if hasattr(self, "success_message"):
                messages.success(self.request, self.success_message)
            return super().form_valid(form)
        return self.render_to_response(self.get_context_data(form=form))

    def get_initial(self):
        initial = super().get_initial()

        initial.update(self.request.GET.dict())
        return initial


class CompanyListView(ListView):
    model = models.Company


class CompanyDetailView(DetailView):
    model = models.Company


class CompanyCreateView(FormMixin, CreateView):
    success_message = "Company created successfully"
    model = models.Company
    fields = ["name", "address"]


class CompanyUpdateView(FormMixin, UpdateView):
    model = models.Company
    success_message = "Company updated successfully"
    fields = ["name", "address"]


class CompanyDeleteView(DeleteView):
    model = models.Company

    def get_success_url(self):
        return reverse("company-list")

    def form_valid(self, form):
        self.request.up.layer.emit("company:destroyed", {})
        messages.success(self.request, "Company deleted successfully")
        return super().form_valid(form)

Experienced Django developers will immediately recognize that the above code has only two small diferences from what a traditional Django app would have:

• the check for self.request.up.validate on the form_valid of the FormMixin
• the self.request.up.layer.emit on the DeleteView form_valid

We’ll explain these later. However the thing to keep is that this is the same as a good-old Django app, without the need to implement special functionality like checks for ajax views, fragments, special form handling etc.

Integrating unpoly with Django

To integrate unpoly with Django you only need to include the unpoly JavaScript and CSS library to your project. This is a normal .js file that you can retrieve from the unpoly install page. Also, if you are using Bootstrap 3,4 or 5 I recommend to also download the corresponding unpoly-bootstrapX.js file.

Unpoly communicates with your backend through custom X-HTTP-UP headers. You could use the headers directly however it is also possible to install the python-unpoly library to make things easier. After installing that library you’ll add the unpoly.contrib.django.UnpolyMiddleware in your MIDDLEWARE list resulting in an extra up attribute to your request. You can then use this up attribute through the API for easier access to the unpoly headers.

To access up through your Django templates you can use request.up or add it to the default context using a context processor so you can access it directly.

To make sure that everything works, add the up-follow to one of your links, i.e change <a href='linkto'>link</a> to <a up-follow href='linkto'>link</a>. When you click on this link you should observe that instead of a full-page reload you’ll get the response immediately! What really happens is that unpoly will make an AJAX request to the server, retrieve the response and render it on the current page making the response seem much faster!

Unpoly configuration

The main way to use unpoly is to add up-x attributes to your html elements to enable unpoly behavior. However it is possible to use the unpoly js API (window.up or up) to set some global configuration. For example, you can use up.log.enable() and up.log.disable() to enable/disable the unpoly logging to your console. I recommend enabling it for your development environment because it will help you debug when things don’t seem to be working.

To use up to configure unpoly you only need to add it on a <script> element after loading the unpoly library, for example:

<script src="{% static 'unpoly/unpoly.min.js' %}"></script>
<script src="{% static 'unpoly/unpoly-bootstrap4.min.js' %}"></script>
<script src="{% static 'application.js' %}"></script>

And in application.js you can use up directly, for example to enable logging:

  up.log.enable()

We’ll see more up configuration directives later, however keep in mind that for a lot of up-x attributes it is possible to use the config to automatically add that attribute to multiple elements using a selector.

Navigation improvements

Using the up-follow directive you can start adding up-follow to all your links and you’ll get a much more responsive application. This is very simple and easy.

One interesting thing is that we didn’t need to change anything on the backend. The whole response will be retrieved by unpoly and will replace the body of the current page. Actually, it is possible to instruct unpoly to replace only a specific part of the page using a css selector (i.e replace only the #content div). To do this you can add the up-target attribute to the link, i.e <a up-target='#content' up-follow href='linkto'>link</a>. When unpoly retrieves the response, it will make sure that it has an #content element and put its contents to the original page #content element.

This technique is called linking to fragments in the unpoly docs. To see this in action, try going to the tasks in the demo and add a couple of new task. Then try to edit a that task. You’ll notice that the edit form of the task will replace the task show card! To do that, unpoly loads the edit task form and matches the .task element there with the current .task element and does the replacement (see here for rules on how this works).

Beyond the up-follow, you can also use two more directives to further improve the navigation:

• up-instant to follow the link on mousedown (without waiting for the user releasing the mouse button)
• up-preload to follow the link when the mouse hovers over the link

Using the up-main

To make things simpler, you can declare an element to be the default replacement target. This is done by adding the up-main attribute to an element. This way, all up-follow links will replace that particular element by default unless they have an up-target element themselves.

What I usually do is that I’ve got a base.html template looking something like this:

    {% include "partials/_nav.html" %}
    <div up-main class="container">
      {% include "partials/_messages.html" %}
      {% block content %}
      {% endblock %}
    </div>
    {% include "partials/_footer.html" %}

See the up-main on the .container? This way, all my up-follow links will replace the contents of the .container element by default. If I wanted to replace a specific part of the page, I could add the up-target attribute to the link.

If there’s no up-main element, unpoly will replace the whole body element.

Make all links followable

It is possible to make all links (or links that follow a selector) followable by default by using the up.link.config.followSelectors option. I would recommend to only do this on greenfield projects where you’ll test the functionality anyway. For existing projects I think it’s better to add the up-follow attribute explicitly to the links you want to make followable.

This is recommend it because there are cases where using unpoly will break some pages, especially if you have some JavaScript code that relies on the page being loaded. We’ll talk about this in the up.compiler section.

If you have made all the links followable but you want to skip some links and do a full page reload instead, add the up-follow=false attribute to the link or use the up.link.config.noFollowSelectors config to make multiple links non-followable.

You can also make all links instant or preload for example by using up.link.config.instantSelectors.push('a[href]') to make all followable links load on mousedown. This should be safe because it will only work on links that are already followable.

Navigation feedback

One very useful feature of unpoly is that it adds more or less free navigation feedback. This can be enabled by adding an [up-nav] element to the navigation section of your page. Unpoly then will add an up-current class to the links in that section that match the current URL. This works no matter if you are using up-follow or not. You can then style .up-current links as you want.

If you are using Bootstrap along with the unpoly-bootstrap integrations you’ll get all that without any extra work! The unpoly-bootstrap has the following configuration:

up.feedback.config.currentClasses.push('active');
up.feedback.config.navSelectors.push('.nav', '.navbar');

So it will automatically add the up-nav element on .nav and .navbar elements and will add the active class to the current link (in addition to the .up-current class). This is what happens in the demo, if you take a peek you’ll see that there are no up-nav elements (since these are marked by the unpoly-bootstrap integration) in the navigation bar and we style the .active nav links.

Aliases for navigation feedback

Unpoly also allows you to add aliases for the navigation feedback. For example, you may have /companies/ and /companies/new and you want the companies nav link to be active on both of them. To allow that you need to use the up-alias attribute on the link like

<a class='nav-item nav-link' up-follow href='{% url "company-list" %}' up-alias='{% url "company-list" %}new/'>Companies</a>

(notice that in my case the url of company-list is /companies/ that’s why I added {% url "company-list" %}new/ on the alias so the resulting alias path would be /companies/new/), or even add multiple links to the alias

<a class='nav-item nav-link' up-follow href='{% url "company-list" %}' up-alias='{% url "company-list" %}*'>Companies</a>

This will add the up-current class to the a element whenever the url starts with /companies/ (i.e /companies/, /companies/new, /companies/1/edit etc).

Please notice that it is recommended to have a proper url hierarchy for this to work better. For example, if you have /companies_list/ and /add_new_company/ you’ll need to add the aliases like up-alias='/companies_list/ /add_new_company/' (notice the space between the urls to add two aliases). Also, if you want to also handle URLS with query parameters i.e /companies/?name=foo then you’ll need to add ?* i.e /companies/?*. These urls aren’t aliased by default so /companies/ doesn’t match /companies/?name=foo unless you add an alias.

One final remark is that it is possible to do some trickery to automatically add up-alias to all your nav links. This is useful in case you have many nav elements and you don’t want to add aliases to each one of them, for example, using this code:

  up.compiler('nav a[href]', (link) => {
    if(!link.href.endsWith('#')) link.setAttribute('up-alias', link.href + '*')
  })

an up-alias attribute will be added to all links. The callback of the compiler will be called when the selector is matched and in this case add the up-alias attribute to the link. We’ll talk later about compilers more.

Handling forms

Unpoly can also be used to handle forms without page reloads, similar to following links. This is simple to do by adding an up-submit attribute to your form. Also similar to links you can make all your forms handled by unpoly but I recommend to be cautious before doing this on existing projects to make sure that stuff doesn’t break.

When you add an up-submit to a form unpoly will do an AJAX post to submit the form and replace the contents of the up-target element with the response (if you don’t specify an up-target element, it will use the up-main element in a similar way as links). This works fine with the default Django behavior, i.e when the form is valid Django will do a redirect to the success url, unpoly will follow that link and render the response of the redirect.

Integrating with messages

Django has the messages framework that can be used to add one-time flash messages after a form is successfully submitted. You need to make sure that these messages are actually rendered! For example, in the base.htm template I mentioned before, we’ve got the following:

    {% include "partials/_nav.html" %}
    <div up-main class="container">
      {% include "partials/_messages.html" %}
      {% block content %}
      {% endblock %}
    </div>
    {% include "partials/_footer.html" %}

please notice that we’ve got the partials/_messages.html template included in the up-main element (inside the container). This means that when unpoly replaces the contents of the up-main element with the response of the form submission, the messages will be rendered as well. So it will work fine in this case.

However, if you are using up-target to render only particular parts of the page the flash messages will be actually lost! This happens because unpoly will load the page with the flash messages normally, so these messages will be consumed; then it will match the .target and display only that part of the response.

To resolve that you can use the up-hungry attribute on your messages. For example, in the partials/_messages.html template we’ve got the following:

<div id='flash-messages' class="flash-messages" up-hungry>
    {% for message in messages %}
        <div class="alert fade show {% if message.tags %} alert-{% if 'error' in message.tags %}danger{% else %}{{ message.tags }}{% endif %}{% endif %}">
            {{ message }}
        </div>
    {% endfor %}
</div>

The up-hungry attribute will make unpoly refresh that particular part of the page on every page load even if it’s not on the target. For example notice how the message is displayed when you edit or mark as done an existing task in the demo.

However also notice that no messages are displayed if you create a new task! This happens because the actual response is “eaten” by the layer and the messages are discarded! We’ll see how to fix that later.

Immediate form validation

Another area in which unpoly helps with our forms is that if we add the up-validate attribute to our form, unpoly will do an AJAX post to the server whenever the input focus changes and will display the errors in the form without reloading the page. For this we need a little modification to our views to check if the unpoly wants to validate the form. I’m using the following form_valid on a form mixin:

def form_valid(self, form):

    if form.is_valid() and not self.request.up.validate:
        if hasattr(self, "success_message"):
            messages.success(self.request, self.success_message)
        return super().form_valid(form)
    return self.render_to_response(self.get_context_data(form=form))

So if the form is not valid or we get an unpoly validate request from unpoly we’ll render the response - this will render the form with or without errors. However if the form is actually valid and this is not an unpoly validate request we’ll do the usual form save and redirect to the success url. This is enough to handle all cases and is very simple and straightforward. It works fine without unpoly as well since the up.validate will be always False in this case.

One thing to keep in mind is that this works fine in most cases but may result to problematic behavior if you use components that rely on javascript onload events. The up-validate will behave more or less the same as with up-follow links.

Other form helpers

Beyond these, unpoly offers a bunch of form helpers to run callbacks or auto-submit a form when a field is changed. Most of this functionality can be replicated by other js libraries (i.e jquery) or even by vanilla.js and is geared towards the front-end so I won’t cover it more here.

Understanding layers

One of the most powerful features of unpoly is layers. To understand the terminology, a layer is any page that is stacked on top of another. The initial page is called the root layer, all other layers are called overlays. Layers can be arbitrary opened and stacked, there’s no limit on the number of layers that can be opened.

An overlay can be rendered like a modal / popup / drawer. The simplest way to use an overlay is to add an up-layer='new' attribute to a link. For example, in the demo app, the link to open a company is like this:

  <a
    up-layer='new'
    up-on-dismissed="up.reload('.table', { focus: ':main' })"
    up-dismiss-event='company:destroyed'
    href="{% url 'company-detail' company.id %}">{{ company.name }}</a>

(ignore the dismiss-related attributes for now). This opens a new modal dialog with the contents of the company detail. It will render the whole contents of the up-main element inside the modal since we don’t provide an up-target. If we added an up-target='.projects' attribute to this it would render only the .projects element inside the modal (but remember that it will retrieve the whole response since the /companies/detail/id is a normal django DetailView). So with up-layer='new' we open a page on a new overlay/modal. If we also add an up-target to it we’ll open only a particular part of that page.

You can use up-mode attribute to change the kind of overlay; the default is a modal. Also if you want to configure the ways this modal closes you can use the up-dismissable attribute, for example add up-dismissable='button' to allow closing only with the X button on the top right. Another useful thing is that there’s an up-size attribute for changing the size of the overlay. I recommend playing a bit with these options to have a feel on how they are working and what you can do with them.

Static overlay content

An overlay can also contain “static” content (i.e not follow a link but display some html) by using the up-content attribute. This is how the green dots are implemented, their html is similar to this:

<a href="#" class="tour-dot viewed" up-layer="new popup" up-content="<p>Navigation links have the <code>[up-follow]</code> attribute. 
    <p>
        <a href=&quot;#&quot; up-dismiss class=&quot;btn btn-success btn-sm&quot;>OK</a>
    </p>
    " up-position="right" up-align="top" up-class="tour-hint" up-size="medium">
</a>

Notice that the up-content contains a whole html snippet. This is implemented in Django using the following template tag:

@register.tag("tourdot")
def do_tourdot(parser, token):
    nodelist = parser.parse(("endtourdot",))
    parser.delete_first_token()
    return TourDotNode(nodelist)


class TourDotNode(template.Node):
    def __init__(self, nodelist):
        self.nodelist = nodelist

    def render(self, context):
        rendered = self.nodelist.render(context).strip()
        size = "medium"
        if len(strip_tags(rendered)) > 400:
            size = "large"
        if not rendered.startswith("<p"):
            rendered = "<p>{}</p>".format(rendered)

        rendered += """
        <p>
            <a href="#" up-dismiss class="btn btn-success btn-sm">OK</a>
        </p>
        """
        from django.utils.html import escape

        output = escape(rendered)
        return """
        <a 
            href="#" class="tour-dot" up-layer="new popup" 
            up-position="right" up-align="top" up-class="tour-hint"
            up-content="{}"
            up-size="{}"
            >
        </a>
        """.format(
            output, size
        )

So we can do something like this in our Django templates:

{% tourdot %}
  <p>Navigation links have the <code>[up-follow]</code> attribute. Clicking such links only updates a <b>page fragment</b>. The remaining DOM is not changed.</p>
{% endtourdot %}

Advanced layers/overlays

Opening overlays for popups or for modals to view links that don’t have interactivity is simple. However, when you open forms with overlays and need to handle these (i.e close the modals only when the form is submitted succesfully) the situation unfortunately starts to get more complex. I recommend to start by reading the subinteractions section of the unpoly documentation to understand how these things work. In the following subsections we’ll talk about specific cases and how to handle them with unpoly layers and Django.

Opening new layers over existing ones

How opening a new layer over an existing layer (i.e a modal inside a modal) would work? All links and forms that are handled in an existing layer will be handled in the same layer. So if we have opened a layer and there are up-follow links in the html of the layer, the user would be able to follow them normally inside that layer (of course if there are non up-follow links then a full page reload will be performed and the layer will disappear without a trace).

If we want to open a new layer we need to use the up-layer='new' attribute on that link; it doesn’t matter if this is inside an already opened layer, it will work as expected and open a layer-in-a-layer. If the parent layer is an overlay then it will open an overlay-in-an-overlay.

In the demo, if you click on an existing company to see its details you’ll get an overlay. If you try to edit that company the edit for will be opened in the same layer (notice that if you press the X button to close it you’ll go back to the company list without layers). Compare this with the behavior when adding a new project or viewing an existing one. You’ll get an overlay inside the parent overlay (both overlays should be visible). You need to close both overlays to go back to the company detail at the root layer.

Even more impressive: Go to the company detail layer, click an existing project to get to the project detail layer, click edit; this will be opened on the project detail layer. You can edit the project or even delete it, when you click that overlay the company overlay will be updated with the new data and work fine! All this also works fine from the project detail list without any modifications on the Django code.

The thing to remember here is that the layer behavior is very intuitive and is compatible with how a server side application works. Everything should work the same no matter if the link is opened in an overlay or in a new page or even an overlay over an overlay.

Closing overlays

There are three main ways to close an overlay (beyond of course using the (X) button or esc etc):

• Visiting a pre-defined link
• Explicitly closing the overlay from the server
• Emitting an unpoly event

Also, when an overlay is closed we can decide if the overlay did something (i.e the user saved the form) or not (i.e the user clicked the X button). This is called accepted or dismissed respectively. We can use this to do different things. All the methods of closing an overlay have a version for accepting or dismissing the overlay.

Closing the overlay when visiting a link

To close the overlay on visiting a link we’ll use the up-accept-location and up-dismiss-location respectively. For example, let’s take a peek on the new company link:

  <a
    class='btn btn-primary'
    up-layer='new'
    up-on-accepted="up.reload('.table', { focus: ':main' })"
    up-accept-location='/core/companies/detail/$id/'
    href='{% url "company-create" %}'>New company</a>

The important thing here is the up-accept-location. When Django creates a new object it redirects to the detail view of that object. In our case this detail view is '/core/companies/detail/$id/'; the $id is an unpoly thingie that will be replaced by the id of the new object and will be the result value of the overlay. This value (the id) can then be used on the up-on-accepted callback if we want.

Now, let’s suppose that we want to close the overlay when the user clicks on a cancel button that returns to the list of companies. We can do that by adding the up-dismiss-location attribute to that <a>

up-dismiss-location='{% url "company-list" %}'

The difference between these two is that the up-on-accepted event will only be called when the overlay is accepted and not on dismissed.

Handling hardcoded urls

One thing that Django developers may not like is that the url is hardcoded here. This is because using {% url "company-detail" "$id" %} will not work with our urls since we use have the following path for the company detail "companies/detail/<int:pk>/". We can change it to "companies/detail/<str:pk>/", to make it work but then it will allow strings in the url and it will throw 500 error instead of 404 when the user uses a string there (to improve that we have to override the get_object of the DetailView to handle the string case). Another way to improve that is to create a urlid template tag like this:

from django.urls import reverse

@register.simple_tag
def urlid(path, arg):
    if arg == "$id":
        arg = 999

    url = reverse(path, args=[arg])
    return url.replace("999", "$id")

And then using it like this on the up-accept-location:

up-accept-location='{% urlid "company-detail" "$id" %}'

Explicitly closing the layer

To close the layer from the server you can you use the X-Up-Accept-Layer or X-Up-Dismiss-Layer response header. When unpoly sees this header in a response it will close the overlay by accepting/dismissing it.

To do that from Django if you have integrated the unpoly middleware, call request.up.layer.accept() and request.up.layer.dismiss() respectively (passing an optional value if you want).

The same feature can be used to close the overlay from the client side. For example, if you want to close the overlay when the user clicks on a cancel button that returns to the list of companies you can do that by adding the up-accept or up-dismiss attribute, like:

<a href='{% urlid "company-detail" "$id" %}' up-dismiss>Return</a>

Please notice that the href here could be like href='#' since this is javascript only to close the overlay, however we added the correct href to make sure the return button will also work when we open the link in a new page (without any overlays).

Please notice that difference between this and up-accept-location or up-dismiss-location we mentioned before. In this case the up-accept/dismiss directive in placed in the a link that closes the overlay. In the former case the up-accept/dismiss-location directive is placed in the link that opens the overlay.

Closing the layer by emitting an unpoly event

The final way to close an overlay is by emitting an event. Unpoly can emit events both from the server, using the X-Up-Event response header or using request.up.emit(event_type, data) from the unpoly Django integration. Also events can be emitted from the client side using the up-emit attribute.

To close the overlay from an event we need to use up-accept-event and up-dismiss-event on the link that opens the overlay.

Let’s see what happens when we delete a company. We’ve got a form like this:

<form up-submit up-confirm='Really?' class="d-inline" method='POST' action='{% url "company-delete" company.id %}'>
  {% csrf_token %}
  <input type='submit' value='Delete' class='btn btn-danger mr-3' />
</form>

This form asks the user for confirmation (using the up-confirm directive) and then submits the form on the company delete view. The CompanyDeleteView is like this:

class CompanyDeleteView(DeleteView):
    model = models.Company

    def get_success_url(self):
        return reverse("company-list")

    def form_valid(self, form):
        self.request.up.layer.emit("company:destroyed", {})
        return super().form_valid(form)

So, it will emit the company:destroyed event and redirect to the list of companies (this is needed to make sure that delete works fine if we call it from a full page instead of an overlay). The company detail view overlay is opened from the following a link:

<a
  up-layer='new'
  up-on-dismissed="up.reload('.table', { focus: ':main' })"
  up-dismiss-event='company:destroyed'
  href="{% url 'company-detail' company.id %}">{{ company.name }}</a>

Notice that we have the up-dismiss-event here. If we didn’t have that then the overlay wouldn’t be closed when we deleted the company but we’d see the list of companies on the overlay because of the redirect on the Django side! Also, instead of the up-dismiss-event we could use the up-dismiss-location='{% url "company-list" %}' similar to how we discussed before. If we did it this way we wouldn’t even need to do anything unpoly related in our DeleteView, however using events for this is useful for educational reasons and we’ll see later how events will help us to dispaly a message when companies are deleted.

Doing stuff when a layer is closed

After a layer is closed (and depending if it was accepted or dismissed) unpoly allows us to use callbacks to do stuff. The most obvious things are to reload the list of results if a result is added/edited/deleted or to choose a result in a form if we used the overlay as an object picker.

The callbacks are up-on-accepted and up-on-dismissed.

Let’s see some examples from the demo.

On on the new company link we’ve got up-on-accepted="up.reload('.table', { focus: ':main' })". However on the show details company link we’ve got up-on-dismissed="up.reload('.table', { focus: ':main' })". This is a little strange (why up-on-accepted on the new vs up-on-dismissed on the detail) at first but we can explain it.

First of all, the up.reload method will do an HTTP request and reload that specific element from the server (in our case the .table element that contains the list of companies). The focus option that is passed instructs unopoly to move the focus to (that element)[https://unpoly.com/focus-option].

For the “Add new” company we reload the companies when the form is accepted (when the user clicks on the “Save” button). However for the show details we’ll reload every time the overlay is dismissed because when the user edits a company the layer will not be closed but will display the edit company data. Also when we delete the company the layer will be dismissed.

Notice that if the user clicks the company details and then presses the (X) button we’ll still do a reload even though it may not be needed because we can’t know if the user actually edited the company or not before closing the overlay. This is a little bit of a tradeoff but it’s not a big deal.

Actually, it is possible to know if the overlay was dismissed because the user clicked the (X) button (or pressed escape) or if the overlay was dismissed because the object was deleted. This is useful if we wanted to display a message to the user when the company was deleted since we’d need to differentiate between these cases. We’ll see how the section about overlays and messages.

On the company detail we’ve got up-on-accepted='up.reload(".projects")' for adding a new project but same as before we’ve got up-on-dismissed='up.reload(".projects")' for viewing the project detail. The .projects element is the projects holder inside the company detail. This is exactly the same behavior we explained before.

On the project form we’ve got up-on-accepted both on the suggest name and on the new company button. In the first case, we are opening the name suggestion overlay like this:

<a
    up-layer='new popup'
    up-align='left'
    up-size='large'
    up-accept-event='name:select'
    up-on-accepted="up.fragment.get('#id_name').value = value.name"
    href='{% url "project-suggest-name" %}'>Suggest name</a>

Notice that this overlay will be accepted when it receives the name:select event. This event passes it the selected name so it will put it on the #id_name input. The up.fragment.get is used to retrieve the input. To understand how this works we need to also see the name suggestion overlay. This is more or less similar to:

  {% for n in names %}
    <a up-emit="name:select"
       up-emit-props='{"name": "{{ n }}"}'
       class="btn btn-info text-light mb-2 mr-1"
       tabindex="0">
      {{ n }}
    </a>
  {% endfor %}

So we are using the up-emit directive here to emit the name:select event and we pass it some data which must be a json object. Then this data will be available as a javascript object named value on the up-on-accepted callback.

So the flow is:

1. When we click the suggest name link we open a new overlay and wait for the name:select event to be emitted. We don’t care if we are a full page or already inside an overlay
2. The suggest name overlay displays a link of <a> elements that emit the name:select event when clicked and also pass the selected name as data on the event
3. The overlay opener receives the name:select event and closes the overlay. It then uses the data to fill the #id_name input

The second case is similar but instead of filling an input it opens a new overlay to create and select a new company. This is the create company link from inside the project form:

  <a href='{% url "company-create" %}'
      up-layer='new'
      up-accept-location='{% urlid "company-detail" "$id" %}'
      up-on-accepted="up.validate('form', { params: { 'company': value.id } })"
  >
      New company
  </a>

Nothing extra is needed from the company form side! We use the up-accept-location to accept the overlay when the company is created (so the user will be redirect to the company-detail view). Then we call up.validate('form', { params: { 'company': value.id } }) after the overlay is accepted. First of all, please remember that when we use the up-accept-location the overlay result will be an object with the captured parts of the url. In this case we capture the new company id. Then, we call up.validate passing it the form and the company id we just retrieved (i.e the id of the newly created company).

It is important to understand that we do up.validate here instead of simply setting the value of the select to the newly created id (similar to what we did before with the name) because the newly created value is not in the options that this select contains so it can’t be picked at this time; when the validate returns though it will contain the newly created company to the options so it will be selected then.

If we wanted to select the newly created company without doing the validate instead we’d need to first add a new option to the select with the correct id and then set it to that value (which is a little bit more complex since we don’t know the name of the new company at this point). To properly implement that and to further understand how unpoly works, we’d need to emit a company:create event from our CreateCompanyView which would contain as data both the id and the name of the newly created company. Then we’d change our accept condition to up-accept-event='company:create'. Finally, our up-on-accepted would add a new option with the value.name and value.id it received from the event and select that option.

Overlays and messages

This probably is the most complex part of integrating unpoly with Django. The problem is that when we do an action the messages will be displayed on the page that our response redirects to. If we don’t display that page but we only use it as on-accept-location we’ll miss these messages. There are various solutions on how this can be fixed, and there also is a long discussion in the unpoly repo discussions about that.

We’ve already discussed about the up-hungry in your messages container element that will reread its contents from all responses. This will resolve all cases where the response is not discarded. For example, try to edit a project and you’ll see the edit message on the overlay (instead of the main page). This is because the overlay is contained in the up-main element so it will be rendered in the response in the overlay.

The problematic behavior is when creating a new project or deleting one. In both these cases we discard the response so the messages are lost. The simplest way to actually fix this is to ignore the server side message and render again the message from unpoly. This avoids changing anything on your server-side code. So, in order to implement this, we’ll use this function which we add on application.js (after a suggestion on the afforementioned discussion):

async function reloadWithFlash(selector, flash) {
  await up.reload(selector)
  up.element.affix(document.getElementById('flash-messages'), '.alert.fade.show.alert-success', { text: flash })
}

This function will call up.reload with a selection we pass to it (f.e up.reload('.table')) and wait until this function finished. Then it will add a new element on the flash-messages container with the flash text we pass to it. In order to use it, we’ll change the new company link to:

      <a
        class='btn btn-primary'
        up-layer='new'
        up-on-accepted="reloadWithFlash('.table', 'Company created!')"
        up-accept-location='{% urlid "company-detail" "$id" %}'
        href='{% url "company-create" %}'>New company</a>

(remember that up-on-accepted before was up-on-accepted="up.reload('.table', { focus: ':main' })"), let’s skip focus for now it’s not important). If we try it this way we’ll notice that we’ll get the Company created message after the overlay is closed! As I said before, the problem with this is that we ignore the server side message and duplicate the message both on server and on client side. The Django side message will be used when we open the /companies/new link on a new page (not an overlay) so the overlay functionality won’t be used and the message will be rendered properly on the response. When we use an overlay the client side message will be rendered instead.

Another solution would be to change our CompanyCreateView to redirect to the companies list page (instead of the newly created page). In this case, we can change the new company form like:

<form up-submit up-validate method="POST" up-layer='root'>
  ...
</form>

Adding the up-layer='root' will render the response in the root layer which will close the overlay and render everything on the up-main element. Since we redirect to the companies list, we’ll get the list of companies along with the server-side message. This solution is actually simpler but modifies our server-side app (instead of the usual behavior of redirecting to the new company detail well’ll redirect to the companies list).

Let’s now talk about delete. As we’ve already discussed above, the company detail overlay will be closed either when the user closes it explicitly by clicking the (X) button or because the company was deleted. In both cases we want to reload the companies but when the company is deleted we also want to display a message. So we need to know when the overlay was closed because the company was deleted vs when the overlay was closed explicitly by the user.

Right now we’ve got

<a
  up-layer='new'
  up-dismissable='button'
  up-on-dismissed="up.reload('.table', { focus: ':main' })"
  up-dismiss-event='company:destroyed'
  href="{% url 'company-detail' company.id %}">{{ company.name }}</a>

For starters, we’ll add the following function:

async function reloadWithFlashIfEvent(selector, flash, value) {
  await up.reload(selector, { focus: ':main' })
  if(value instanceof Event) {
    up.element.affix(document.getElementById('flash-messages'), '.alert.fade.show.alert-danger', { text: flash })
  }
}

and change up-on-dismissed to up-on-dismissed="reloadWithFlashIfEvent('.table', 'Company deleted!', value)" on the open overlay link.

The up-on-dismissed and up-on-accepted callbacks are passed these paremeters by unpoly: * this The link that originally opened the overlay * layer An up.Layer object for the dismissed overlay * value The overlay’s dismissal value * event An up:layer:dismissed event

If the event was dismissed because the user clicked the (X) button, the value would have a similar to :button (there are same string values for pressing escape or clicking outside the modal). However if it was dismissed because of the company:destroyed event, the value would be an Event object. So we pass the value to our reloadWithFlashIfEvent callback and check if the value is an Event object. If it is, we know that the overlay was dismissed because the company was deleted and we can display the flash message. If it’s not, we know that the overlay was dismissed because the user clicked the (X) button and we won’t display the flash message.

Another way we could implement this would be if we closed the company detail overlay when the company was deleted and returned the response (which is the company list view) to the root layer. Something like this:

<form up-submit up-confirm='Really?' class="d-inline" method='POST' action='{% url "company-delete" company.id %}' up-layer='root'>

(notice we added the up-layer='root' attribute). For this to work we need to not reload in the up-on-dismissed function because if we reload the companies list the contents of the flash-messages will be re-read (because it has the up-hungry attr) and be immediately cleared out! However in this case we need to reload because a company may be edited!

Improving delete

Right now, the delete button is a form, similar to this:

<form up-submit up-confirm='Really?' class="d-inline" method='POST' action='{% url "company-delete" company.id %}'>
  {% csrf_token %}
  <input type='submit' value='Delete' class='btn btn-danger mr-3'  />
</form>

So this is an unpoly-handled form and will display a Really? javascript prompt to make sure the user really wants to delete the company.

I have to confess that I don’t like javascript prompts because they can’t be styled and seem out of context from the app. However we can improve that behavior with unpoly. Here’s an improved version of the delete functionality:

<a class='btn btn-danger' up-layer="new" up-content='
      <h3>Delete company {{ company.name }}</h3>
      Do you want to delete the company? 
      <form up-submit up-target=".table" up-layer="root" class="d-inline" method="POST" action="{% url "company-delete" company.id %}">
        {% csrf_token %}
        <input type="submit" value="Yes" class="btn btn-danger mr-3"/>
        <a href="#" class="btn btn-secondary"  up-dismiss>No</a>
      </form>
      '>Delete</a>

We changed the delete button to open a new layer. Instead of having a special view for the delete confirmation, we’re using the up-content attribute to directly pass the static HTML for the confirmation, which actually includes the delete form like before. Notice that we also include an up-dismiss button that clears the overlay when the user presses No. The up-layer of the form is root so when the form is submitted it will close both the confirmation overlay and the company detail overlay! Now, we’ll change the reloadWithFlashIfEvent like this:

async function reloadWithFlashIfEvent(selector, flash, value) {
  await up.reload(selector, { focus: ':main' })
  if(value instanceof Event || value == ':peel') {
    up.element.affix(document.getElementById('flash-messages'), '.alert.fade.show.alert-danger', { text: flash })
  }
}

This checks if the value is an event or :peel; this is the value that is passed when the overlay is dismissed because we use the up-layer='root' from the delete form.

Improving interaction with Django packages

There are two very important packages that I use on almost all my projects: django-tables2 and django-filter. You can see these in action at the /core/tf/ path on the demo app. You’ll see that:

• Filtering is instant (when entering a character it will filter without the need to submit the form explicitly )
• The row detail links open in an overlay
• Sorting and pagination are handled by unpoly (so they don’t do full page reloads)

To have the instant filtering we’ve changed our filter form like this:

<form up-autosubmit up-delay='250' class='form-inline' method='GET' action='' up-target='.form-data'>
    {{ filter.form|crispy }}
    <input class='btn btn-info' type='submit' value='Filter'>
    <a up-follow href='{{ request.path }}' class='btn btn-secondary'>Reset</a>
</form>

Notice the up-autosubmit; this will submit the form when a field changes. Also the up-delay adds a small delay before the form is submitted so when the user writes foo it will do 1 query instead of 3 (if he writes fast enough of course). The up-target attribute is used to specify the element that will be updated with the response. In this case we’re using a form-data element that includes the whole table (using django-tables2 of course):

<div class='form-data'>
  {% render_table table %}
</div>

To open the links in a layer we only need to pass the correct parameters to the table field, for example in our case the table is like this:

class CompanyTable(tables.Table):
    id = tables.LinkColumn(
        "company-detail",
        args=[A("id")],
        attrs={
            "a": {
                "class": "btn btn-primary btn-sm",
                "up-on-dismissed": "up.reload('.table', { focus: ':main' })",
                "up-layer": "new",
            }
        },
    )

    class Meta:
        model = models.Company
        template_name = "django_tables2/bootstrap4.html"
        fields = ("id", "name", "address")

So we a pass the attributes directly to the link’s a element. Nothing really fancy is needed.

Furthermore, notice that we use the builtin bootstrap4 template. We don’t change the template at all. The original django-tables2 template does not have unpoly interation! So if we leave it like this the pagination and header links will start a full request/response. To fix that, we could override the template with our own however this is not the ideal solution for me.

Instead, we can use up.compiler:

up.compiler('.pagination .page-item a.page-link', (link) => {
  link.setAttribute('up-target', ".table-container")
})

up.compiler('th.orderable a[href]', (link) => {
  link.setAttribute('up-target', ".table-container")
})

The up.compiler function takes a CSS selector and a callback function. The callback function is called when a snippet matching the selector is added to the DOM. In this case we’re adding the up-target='.table-container' unpoly attributes to both the pagination and the table header order links. The .table-container is the element that contains the table (it is added by django-tables2).

This way, when unpoly sees these links it will add the up-target attribute (and functionality) to these without the need to override any templates.

Advanced concepts

We’ll discuss some more advanced concepts of unpoly now.

More about up.compiler

The up.compiler function is very powerful. We already used it to add functionality to all our nav links (see the navigation aliases before) to avoid forgetting it and to add the up-target to our table links to avoid overriding the django-table2 templates.

Beyond these, the most important functionality of up.compiler is to replace the javascript on load (or jquery $(function() {})) event. Most common javascript libraries will be initialized when the document is ready. Unfortunately, when a page is loaded through unpoly this event will not be trigger, so the javascript elements will not be initialized! Let’s suppose that we’ve got a bunch traditional jquery ui datepicker elements and all these have the .datepicker css class. Normally we’d initialize it like

$(function() {
  $('.datepicker').datepicker()
})

If we are to load a form with these elements through unpoly we won’t get the datepicker functionality. To fix this we can use up.compiler:

up.compiler('.datepicker', (element) => {
  $(element).datepicker()
})

So when unpoly sees a .datepicker element it will call that callback function and initialize it! This will work properly if you follow links through up-follow or open new overlays with up-layer='new'.

Passing context from unpoly to server

Unpoly has an up-context attribute that can be used to pass context to the server. This must be a json object and can then be used to change the response based on that context. If we are using the unpoly python package then the context will be available in the request.up.context dictionary.

Let’s see a particular example from the demo. When we create a new task we’ve got the following link:

<a
    class='btn btn-primary'
    up-layer='new'
    up-context='{"new_task": true}'
    up-accept-location='/core/tasks/detail/$id/'
    up-on-accepted="reloadWithFlash('.tasks', 'Task created!')"
    href='{% url "task-create" %}'>New task</a>
</div>

Compare this with the edit link:

<a up-target='.task' href='{% url "task-update" task.id %}' class='btn btn-sm btn-outline-secondary'>Edit</a>

Notice that the up-context is only included in the new link. Now, let’s see how the task form is implemented:

<form up-submit {% if not up.context.new_task %}up-target='.task'{% endif %} class='task card' method="POST">
    {% csrf_token %}
    <div class="card-body d-flex flex-column">
        <div class="form-group flex-grow-1 mb-0">
            {{ form|crispy }}
        </div>
        <div class="flex-grow-0">
            <input type='submit' class='btn btn-primary mt-2' value='Save'>
        </div>
    </div>
</form>

So, using Django we check to see if there’s new_task in the context and add an up-target='.task' if not. This way, we’ll get an up-target='.task' in the form only if we click the edit task button. Beyond this the form is the same for both the new and edit links.

This is needed because when we open the edit task it will be loaded in the same .task element we clicked the edit link from (remember that unpoly is smart enough to match closer elements). When the form is submitted we want the detail view of the task to also be rendered on the same .task element so we use the up-target='.task'. This isn’t needed in the create new since it will be rendered in a new layer and we want to reload the the tasks with the flash message when the new task is created.

Please notice that if we were to use up-target='.task' for both the new and edit form we’d get an error when the new task form was submitted because it wouldn’t be able to match the target .task element!

Listening to unpoly events

For most things happening in unpoly you’ll find out that there are events that you can listen to and add behavior. There are cases where handling these is useful.

For example, I’ve observed that if you’re using bootstrap dropdowns and click a link while the dropdowns are opened, the dropdowns will remain open when the fragment has been loaded! This is very annoying. One simple way to resolve that is include the navigation inside your up-main element so the dropdowns will be reloaded. However there’s a better way by using unpoly events like in the following snippet:

    up.on('up:link:follow', function(event, link) {
      // Hide visible dropdowns
      const dropdownElementList = document.querySelectorAll('.dropdown-toggle.show')
      const dropdownList = [...dropdownElementList].map(dropdownToggleEl => new bootstrap.Dropdown(dropdownToggleEl))
      dropdownList.forEach(dropdown => dropdown.hide())
    })

Please notice that this code is for bootstrap 5 (not 4 as the remaining code in the demo since it’s from a different project). So what happens is that whenever a link is followed from unpoly we’ll clear the open dropdowns (the code isn’t very important here).

Updating history

One thing to consider when using unpoly is when we actually need to update the browser history and url. By default, unpoly will update the url only if the up-target matches up-main (so if there’s no up-target the url will always be upgraded).

This can be configured through the up-history attribute. By default this has the value 'auto' and we can set to 'true' or 'false' if we want to configure it so that unpoly updates the history or not for a particlar link or form submission.

Let’s see a particular example from the demo. Because the up-target of the filter form on the is set to .form-data:

<form up-autosubmit up-delay='250' class='form-inline' method='GET' action='' up-target='.form-data'>

the url will not be updated when the filter is changed. This is contrary to the usual way these kind of filters work (i.e update the url with the filter parameters). So we can add the up-history attribute:

<form up-autosubmit up-history='true' up-delay='250' class='form-inline' method='GET' action='' up-target='.form-data'>

The same applies for the pagination and header ordering links. They update the .table-container element so we’ll need to add up-history=true also to them. Thus we’ll change the up.compiler for these elements like this:

up.compiler('.pagination .page-item a.page-link', (link) => {
  link.setAttribute('up-follow', link.href)
  link.setAttribute('up-target', ".table-container")
  link.setAttribute('up-history', "true")
})

up.compiler('th.orderable a[href]', (link) => {
  link.setAttribute('up-follow', link.href)
  link.setAttribute('up-target', ".table-container")
  link.setAttribute('up-history', "true")
})

This way, both the ordering links and the selected page will be reflected on the url history.

The final result is that this filter/table page will have the usual functionality of updating the url when the filter is changed or the table sorting/pagination links are used.

Troubleshooting

As I’ve already mentioned, the most common problem you are going to have with unpoly is when you use javascript on your page ready event. Unfortunately there’s a lot of functionality that relies on that event and pages will break when you use unpoly in these cases. That’s why I recommend to use up-follow and up-submit for your links and forms on a case-by-case basis on non greenfield projects so you’ve got more control on what works with unpoly and what is not working. Another thing that is very important to notice here is that I’ve stumbled upon libraries that not only rely on the load event but actually there’s no other way to initialize them! For example, there’ are js libraries that have code like

$(function() {
  let initElement = function(el) {
    ...
  }
  $('.selected-elements').each(function() {
    initElement(this)
  })
})

so the actual function that does the initialization (initElement) isn’t public and you can’t call it from the up.compiler. In these cases you’ll need to somehow make the initElement public so you can call it from the up.compiler or use a different library!

The other major case for headaches in unpoly overlays. Although they are very powerful I recommend to not abuse them and use them only when you feel that are really needed and would improve the UX of the user. For example, I’d recommend using them to add new options on a select list (similar to how the project form works for companies and of course similar to how django admin does it). Also you could use overlays to have a functionality similar to django-inlines (see how the projects are added to the company) however I’d probably prefer to do that using normal django inlines especially when the standalone child edit functionality isn’t needed.

Special care must be taken for the integration between layers and messages. I have tried to provide a solution in the previous sections by proposing flashing the message with javascript on the cases where the message will be “eaten” by a discarded response however I’m afraid that depending on how you’ve architectured your app you’ll may still get problems. The important thing is to understand how messages work (or not) and in which cases you may skip using messages at all since the feedback would be immediate and the users don’t really need messages.

Conclusion

In conclusion, using unpoly with your Django apps can enhance the UX of your users by reducing page reloads and providing a more responsive and intuitive interface with little work from the developer. I recommend everybody to start integrating unpoly in their projects and see how it can improve the UX of your users!

Have you ever needed to use data from a Microsoft Access database on a Django project? If so, you might have found that the data is contained in an .accdb file, which can make accessing it a challenge. Actually, access files are either .mdb (older versions) and .accdb (more current versions); although the .mdb files is easier to access from python, most Access database would be .accdb nowadays.

The naive/simple way to access this data is to bite the bullet, install Access on your computer and use it to export the tables one by one in a more “open” format (i.e xlsx files). After some research I found out that there are ways to connect to this Access database through python and querying it directly. Thus I decided to implement a more automatic method of exporting your data. In this article, I’ll walk you through the steps to accomplish this, specifically we’ll cover how to:

• Connect to an accdb database
• Export all the tables of that database to a json file
• Create a models.py based on the exported data
• Import the json data that was exported into the newly created models

For the first two steps we’ll only use python. For the last two we’ll also need some Django. By the end of this article, you’ll have a streamlined process for accessing your Microsoft Access data in your Django projects.

Connecting to a Microsoft Access database from python

To be able to connect to an Access database from python you can use the pypyodbc. This is a pure-python library that can connect to ODBC. To install it run pip install pypyodbc; since this is a pure-python the installation should be always successfull.

However, there is an important caveat when working with .accdb databases: You must use Windows and install Microsoft Access Database Engine 2010 Redistributable. This is necessary to ensure that the correct drivers are available, you can also take a look at the instructions from pyodbc here.

When installing the Access Redistributable, it’s crucial to remember that you need to install either the 32-bit or 64-bit of the Access Redistributable depending on your python (you can’t install both). To check if your python is 32bit or 64bit run python and check if it says 32 or 64 bit. Also please notice that you may be able to install only the 32bit or only the 64bit version if you have an MS Office installed (the Access Redistributable will match the MS Office bitness).

If that’s the case then my recommendation would be to generate a new virtualenv (similiar to the bitness of the Access Redistributable you’ve installed on your system). Then install pypyodbc on that virtualenv and you should be fine.

If you want to use a .mdb database you should be able to do it without installing anything on Windows and it also should be possible on Unix (I haven’t tested it though).

To ensure that you’ve got the correct drivers, run the following snippet on a python shell:

import pypyodbc
print('\n'.join(pypyodbc.drivers()))

If you have installed the correct Microsoft Access Database Engine 2010 Redistributable you should see .accdb somewhere in the output, like this:

Microsoft Access Driver (*.mdb)
Microsoft dBase Driver (*.dbf)
Microsoft Excel Driver (*.xls)
Microsoft ODBC for Oracle
Microsoft Paradox Driver (*.db )
Microsoft Text Driver (*.txt; *.csv)
SQL Server
Oracle in OraClient10g_home1
SQL Server Native Client 11.0
ODBC Driver 17 for SQL Server
Microsoft Access Driver (*.mdb, *.accdb)
Microsoft Excel Driver (*.xls, *.xlsx, *.xlsm, *.xlsb)
Microsoft Access dBASE Driver (*.dbf, *.ndx, *.mdx)
Microsoft Access Text Driver (*.txt, *.csv)

If on the other hand you can’t access .accdb files you’ll get much less options:

SQL Server
PostgreSQL ANSI(x64)
PostgreSQL Unicode(x64)
PostgreSQL ANSI
PostgreSQL Unicode
SQL Server Native Client 11.0
ODBC Driver 17 for SQL Server

In any case, after you’ve installed the correct drivers you can connect to the database (let’s suppose it’s named access_data.accdb on the parent directory) like this:

import pypyodbc

pypyodbc.lowercase = False
conn = pypyodbc.connect(
    r"Driver={Microsoft Access Driver (*.mdb, *.accdb)};"
    + r"Dbq=..\\access_data.accdb;"
)
cur = conn.cursor()

for row in cur.tables(tableType="TABLE"):
    print(row)

If everything’s ok the above will print all the tables that are contained in the database.

Exporting the data from the Access database to a json file

After you’re able to connect to the database you can export all the data to a json file. Actually we’ll export both the data of the database and a “description” of the data (the names of the tables along with their columns and types). The description of the data will be useful later.

The general idea is:

1. Connect the database
2. Get the names of the tables in a list
3. For each table
   • Export a description of its columns
   • Export all its data
4. Write the description and the data to two json files

This is done by running the following snippet:

import pypyodbc
import struct
import json
from datetime import datetime, date
import decimal

print("running as {0}-bit".format(struct.calcsize("P") * 8))

def normalize(s):
    """A simple function to normalize table names"""
    return s.lower().replace(" ", "_")


conn = pypyodbc.connect(
    r"Driver={Microsoft Access Driver (*.mdb, *.accdb)};"
    + r"Dbq=..\\access_data.accdb;"
)
cur = conn.cursor()
tables = []
for row in cur.tables(tableType="TABLE"):
    # Only get the table names
    tables.append(row[2])

# data will contain the data of all tables. It will have the following structure:
# {"table_1": [{"column_1": value, "column_2": value}, ...], "table_2": ...}
data = {}
# descriptions will have  a description of all the tables. It will have the following structure:
# [
#   {
#       "table_name": "table 1",
#       "fixed_table_name": "table_1",
#       "columns": [
#           {"name": "column_1", "fixed_name": "column_1","type": "str"},
#           {"name": "column_2", "fixed_name": "column_2","type": "int"},
# ]
descriptions = []

for table_name in tables:
    fixed_table_name = normalize(table_name)
    print(f"~~~~~~~~~~~~~{table_name} {fixed_table_name}~~~~~~~~~~~~~")
    q = f'SELECT * FROM "{table_name}"'
    description = {
        "table_name": table_name,
        "fixed_table_name": fixed_table_name,
        "columns": [],
    }
    descriptions.append(description)

    cur.execute(q)
    # Here we get the description of the columns of the table from the cursor; we'll use that to fill the description.columns list
    columns = cur.description
    for c in columns:
        description["columns"].append(
            {"name": c[0], "fixed_name": normalize(c[0]), "type": c[1].__name__}
        )

    print("")

    # And here we retrieve the data of the whole table
    # Notice we use some double for loop comprehension to 
    # create a json object with a column_name: value structure
    # for each row
    data[fixed_table_name] = [
        {normalize(columns[index][0]): column for index, column in enumerate(value)}
        for value in cur.fetchall()
    ]

cur.close()
conn.close()

# This is a function to serialize datetime and decimal objects 
# to json; without it the json.dump function will fail if the 
# results contain dates or decimals
def json_serial(obj):
    """JSON serializer for objects not serializable by default json code"""

    if isinstance(obj, (datetime, date)):
        return obj.isoformat()
    elif isinstance(obj, decimal.Decimal):
        return str(obj)
    raise TypeError("Type %s not serializable" % type(obj))


with open("..\\access_description.json", "w") as outfile:
    # Notice the default=json_serial 
    json.dump(descriptions, outfile, default=json_serial)

with open("..\\access_data.json", "w") as outfile:
    json.dump(data, outfile, default=json_serial)

If you run the above code and don’t see any errors you should have two json files in the parent directory: access_description.json and access_data.json. The dump of your access database is complete!

Creating a models.py based on the exported data

Now that we have the description of the data in our database it is possible to create a small script that would help us generate the models for importing that data into Django. This could by done by a snippet similar to this:

import json

def get_ctype(t):
    """Depending on the type of each column add a different field in the model"""
    if t == "str":
        return "TextField(blank=True, )"
    elif t == "int":
        return "IntegerField(blank=True, null=True)"
    elif t == "float":
        return "FloatField(blank=True, null=True)"
    elif t == "bool":
        return "BooleanField(blank=True, null=True)"
    elif t == "datetime":
        return "DateTimeField(blank=True, null=True)"
    elif t == "date":
        return "DateField(blank=True, null=True)"
    elif t == "Decimal":
        return "DecimalField(blank=True, null=True, max_digits=15, decimal_places=5)"
    else:
        print(t)
        raise NotImplementedError

# Load the descriptions we created in the previous step
descriptions = json.load(open("..\\access_description.json"))

# mlines will be an array of the lines of the models.py file
mlines = ["from django.db import models", "", ""]

for d in descriptions:
    # Create a model for each table
    mname = d["fixed_table_name"].capitalize()
    mlines.append(f"class {mname}(models.Model):")
    for c in d["columns"]:
        ctype = get_ctype(c["type"])
        mlines.append(f"    {c['fixed_name']} = models.{ctype}")
    mlines.append("")
    mlines.append("    class Meta:")
    mlines.append(f"        db_table = '{d['fixed_table_name']}'")
    mlines.append(f"        verbose_name = '{d['table_name']}'")

    mlines.append("")
    mlines.append("")


with open("..\\access_models.py", "w", encoding="utf-8") as outfile:
    outfile.write("\n".join(mlines))

This will generate a fine named access_models.py. You should edit this file a to add your primary and foreign keys. In an ideal world this would be done automatically, however I couldn’t find a way to extract the primary and foreign keys of the tables from the Access database. Also by default I’ve set all fields to allow blank and null values; please you should fix that according to your needs.

After you edit the file, create a new app in your Django project and copy the file to the models.py file of the new app. Add that app to your INSTALLED_APPS in the settings.py file and run python manage.py migrate to create the tables in your database.

Import the json data to Django

The final piece of the puzzle is to import the data we extracted before directly in Django. Because we know the names of all the models and fields this is trivial to do:

from django.core.management.base import BaseCommand
import json
from django.db import transaction
from access_data import models

# This is a list of all the fields that are foreign keys; these need special handling
FK_FIELDS = [
    # ...
]

# You need to add the table names from the access database here. This is required
# if you have relations in order to add first the tables without dependencies and last
# the tables that belong on these 
TABLE_NAMES = [
    # ...
]

def fix_fks(k):
    """Add _id to the end of the field name if it is a foreign key to pass the pk of the
    foreign key instead of the whole object"""
    if k in FK_FIELDS:
        return k + '_id'
    return k

def get_model_by_table(table):
    """Get the model by the table name"""
    return getattr(models, table.capitalize())

class Command(BaseCommand):

    @transaction.atomic
    def handle(self, *args, **options):

        with open("..\\access_data.json") as f:
            j = json.load(f)

        # Delete the existing data before importing. This is optional but I find it useful
        # Notice that we delete the tables in reverse order to avoid foreign key errors
        for table in reversed(TABLE_NAMES):
            get_model_by_table(table).objects.all().delete()

        for table in TABLE_NAMES:
            for row in j[table]:
                # Create a dictionary with the column name: column value;
                # notice the fix_fks to add the _id to the column
                row_ok = {fix_fks(k): v for k,v in row.items()}
                print(row_ok)
                # Create the object; we could add thse to an array and do a bulk_create instead
                get_model_by_table(table).objects.create(**row_ok)

The above code may error out if you have missing or bad data in your database. You should fix accordingly.

Conclusion

In conclusion, accessing and using data from Microsoft Access databases in Django may seem daunting at first, but with the right tools and techniques, it can be a straightforward process. By using the pypyodbc library and following the instructions outlined in this post, you can connect to your .mdb or .accdb database and export its tables and schema to JSON files. From there, it is trivial to create a models.py file for Django and a management command to import the data.

Although I’ve presented these steps as separate snippets, you could also combine them into a single management command within Django. The possibilities are endless, and with a little bit of creativity, you can tailor this approach to your specific needs and data.