How to add python serverless functions as Django background workers 🗡

I’ve spent 14 hours looking for the simplest way to add an async serverless function to Photon Designer.

This is the guide that I was missing. I’ll show you the simplest way to:

1. Develop serverless functions locally (including Python packages, without AWS Lambda layers, and without cumbersome Docker)

2. Call the serverless function async - both in your development app and in production (i.e., without blocking your server response)

3. Deploy the serverless function to production automatically in CI (using Github Actions)

See below for a list of all 10 providers I tried.

Here's what the async function, running locally and with a Github action to deploy a version to production, looks like:

🃏Here's a joke produced by the function we'll create:

Having done this 14 hour review, the best option I've found is DigitalOcean Functions. We'll use that. (See below for all 10 providers I tried)

I've also made an optional video guide (featuring me ) below that follows the steps in this guide:

Let's get started 🐎

Setup

• Create a DigitalOcean account (free)
• Install the Digital Ocean command line tool, including Serverless Functions support There are a few steps to; all are quick to do and explained in the link above.

Developing your function

We will create a simple function that calls a long-running functions. This will simulate a long-running task (i.e., >1 second), such as exporting a user's Photon Designer project to their computer.

Create a development namespace

Each function is deployed to a namespace. We'll create a namespace for development.

We're using the frankfurt region (fra) here because I'm in Hamburg 🇩🇪. Enter doctl serverless namespaces list-regions to see all.

doctl serverless namespaces create --label development --region fra1

You should see something like this:

t@tair ~ % doctl serverless namespaces create --label development --region fra1
Connected to functions namespace 'fn-7f1475fb-4a62-438a-a090-f8013810a856' on API host 'https://faas-fra1-afec6ce8.doserverless.co'

Create your function and folder

doctl serverless init simple-function-project --language=python 

This creates a folder called simple-function-project with a simple function called sample/hello.

Run your function locally

• Deploy your function to your development namespace

doctl serverless deploy simple-function-project

• Run it

doctl serverless functions invoke sample/hello -p name:Keith

Automatically deploy your function on save

Crucially, we want to update our function and then have it automatically deployed to our development namespace. This does that:

doctl serverless watch simple-function-project

You should see something like this:

Watching 'simple-function-project' [use Control-C to terminate]

Note: Make sure that you're iin the top-level directory of your project when you run this command. You need to be above the simple-function-project directory in order to watch it. You can't watch the directory from within itself.

Test it by replacing the code in simple-function-project/sample/hello/hello.py with the below:

def main():
      message = (
            "Ice is less dense than water, at approximately 0.9 g/cm3, due to the nature of the bonding between its molecules. "
            "The result of this is that ice floats on liquid water, which is an important feature in Earth's biosphere.\n"
            "It has been argued that without this property, natural bodies of water would freeze, in some cases permanently, "
            "from the bottom up, resulting in a loss of bottom-dependent animal and plant life in fresh and sea water.\n"
      )
      print(message)
      return {"body": message}

Get the url for your deployed function

Enter the below in the terminal to get the url for your deployed function:

doctl serverless functions get sample/hello --url

Click the url to see your function in the browser.

Update the function to simulate a long-running task

• Rename the file simple-function-project/sample/hello/hello.py to simple-function-project/sample/hello/__main__.py
• Update the code in simple-function-project/sample/hello/__main__.py with the below:

import requests
import time

def main():
    response = requests.get("https://v2.jokeapi.dev/joke/Spooky,Christmas")
    data = response.json()
    time.sleep(2) # Simulate a long-running function here.
    if data['type'] == 'single':
        joke = data['joke']
    else:
        joke = f"{data['setup']}\n{data['delivery']}"
    return {"body": f"{joke}"}

• Add a requirements.txt file at simple-function-project/requirements.txt with the below:

requests

• Add a build.sh file at simple-function-project/build.sh with the below:

#!/bin/bash

set -e

python -m pip install --upgrade pip
pip install virtualenv
virtualenv venv
source venv/bin/activate
pip install -r requirements.txt

• Make the build file executable:

chmod +x build.sh

• Deploy your function to your development namespace

doctl serverless deploy simple-function-project

Visit the url for your deployed function to see the result. It should show a jokes after 2 seconds. We'll call the function asynchronously in the next step.

Call your function asynchronously using Python

This is needed for web apps. This background execution avoids blocking your server (and making the app freeze for every user) while the function runs.

We'll now call the function asynchronously using Python. This means that the function will run in the background. The server will not wait for the function to finish before returning a response.

In a web app, e.g., Django, we can then set our serverless function to send a response to an endpoint once that the task is complete.

Docs for Digital Ocean functions are here

Get your command to call the function asynchronously

• Click on your function at Digital Ocean functions
• In the Access & Security section, under REST API, click the "Show Token" link and copy the entire curl command. This will look like this:

curl -X POST "https://faas-fra1-afec6c48.doserverless.co/api/v1/namespaces/fn-7f1475fb-4a62-438a-a090-f8013810a856/actions/sample/hello?blocking=true&result=true" \
  -H "Content-Type: application/json" \
  -H "Authorization: Basic MYlm3333344444444C00ZTdlLTlmZDEtt2VjYWQ5Y2QyMmJmOmdqMzU5RmlORkFMZHVWcWlLV25sVmJZNk0wRUFONkZUUkl4YTNWV043dHYweWJWcU9NMW5jNGhI3Uw3V4RzV3M="

This shows the:

1. The namespace url for your function
2. Your secret namespace key (after the 'Authorization: Basic' part)

We'll use both of these next.

Create a services.py file to call the serverless function asynchronously

We'll create a services.py file to call the function asynchronously.

We could use this in our imaginary web app, as I'm doing on the backend of Photon Designer with Django

1. Install the requests library to your project's environment:

pip install requests

1. Create a services.py file in the root of your project.
2. Add the below code to the services.py file,

import requests


def run():
    print(f'About to run function')
    function_url = '<your function url>'
    headers = {
        'Authorization': 'Basic <your_secret_namespace_key>',
        'Content-Type': 'application/json',
    }
    params = {
        'blocking': 'false',  # This makes the function run asynchronously.
    }
    response = requests.post(function_url, params=params, headers=headers)
    data = response.json()
    print(f'{data = }')
    return data


if __name__ == '__main__':
    run()

1. Update the code to use your namespace url and secret namespace key from the previous step. Make sure to remove the blocking=true and result=true parts from the url. These would make the function run synchronously, blocking our imaginary python app from responding until the function has finished running. We want to avoid this.

It should look like this (but you'll need to replace the namespace url and namespace secret key with your own):

import requests


def run():
    print(f'About to run function')
    function_url = 'https://faas-fra1-ttec6ce8.doserverless.co/api/v1/namespaces/fn-733335fb-4a62-438a-a090-f8013810a856/actions/sample/hello'  # Notice that we removed the blocking=true and result=true parts from the url.
    headers = {
        'Authorization': 'Basic 123321123',
        'Content-Type': 'application/json',
    }
    params = {
        'blocking': 'false',  # This makes the function run asynchronously.
    }
    response = requests.post(function_url, params=params, headers=headers)
    data = response.json()
    print(f'{data = }')
    return data


if __name__ == '__main__':
    run()

1. Call the function asynchronously

python services.py

The function should return near instantly, with no 2 second delay. You should see something like this:

About to run function
data = {'activationId': 'a9fb28e9a03f46f8bb28e9a03f86f8fe'}

To see a log of your serverless function running, go to the Digital Ocean UI and click on your 'development' namespace and function within in.

Sidenote: How I'm calling this in my Django app (Photon Designer)

I'm using this in the backend of Photon Designer with Django.

When a user exports their project, I call a serverless function to export the project to their computer. This is done asynchronously, so the export happens efficiently in the background.

Here's a sample of code I use in my Django app to call the serverless function asynchronously:

def send_export(data: dict) -> dict:
    if settings.ENVIRONMENT == 'production':
        namespace = 'fn-11111-2222-3333-4444'
    else:
        namespace = 'fn-22222-3333-4444-5555'

    headers = {
        'Content-Type': 'application/json',
        'Authorization': f'Basic {os.environ["DO_NAMESPACE_API_TOKEN"]}',  # I set this in my Django app's environment variables, depending on the environment.
    }
    params = {'blocking': 'false', 'result': 'false'}
    function_url = (
        'https://faas-nyc1-2ef2e6cc.doserverless.co/api/v1/'
        f'namespaces/{namespace}/actions/sample/export'
    )

    response = requests.post(
        function_url,
        params=params,
        headers=headers,
    )
    return response.json()

With sample output:

>>> send_export({'apple': 'dod'})
{'activationId': '6bc88119a76f4aal88811l333223333b'}

Deploy your serverless python function to production using Github Actions

Great. Our function is working locally. Now we want to deploy it to production using version control and Github Actions.

An overview of the development and deployment process is:

1. Develop the function locally in the development namespace

2. Once we want to deploy to production, we merge our development code to our master branch in our Github repo

3. During CI for this merge, we run a simple Github Action that connects our serverless function to a production namespace and deploys the function to the production namespace.

For this, you'll need a Github account and a Github repo with your code in it. I'll assume that you have this already (Github's docs).

Create a Digital Ocean production namespace

We'll create a production namespace for our function. This is where our function will run in production. Same process as before, but we'll call it 'production' instead of 'development'.

doctl serverless namespaces create --label production --region fra1

Then connect back to your development namespace to continue developing your function locally

doctl serverless connect development

Setup Github Action to deploy the serverless function to production

We'll use the DigitalOcean Github Action to deploy our function to production.

1. Create a DigitalOcean API token
2. Add it as a secret to your repository
3. Create a github workflow file at the path .github/workflows/deploy.yml with the below:

name: Deploy Serverless Function to Production

on:
  push:
    branches:
      - main
jobs:
  build:

    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.12"]

    steps:
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v4
        with:
          python-version: ${{ matrix.python-version }}

      - name: Install doctl
        uses: digitalocean/action-doctl@v2
        with:
          token: ${{ secrets.DIGITALOCEAN_ACCESS_TOKEN }}

      - name: Install doctl serverless
        run: doctl serverless install

      - name: Connect to production namespace
        run: doctl serverless connect production

      - name: Deploy function to production namespace
        run: doctl serverless deploy simple-function-project

      - name: Show function url in production
        run: doctl serverless functions get sample/hello --url

Now, push your branch to Github and create a pull request to merge it to master. Once you merge it, the Github Action will run and deploy your function to production.

You can see the result in the Digital Ocean UI by clicking on your 'production' namespace and function within in.

Complete ✅

P.S Interested in building your Django frontend even faster?

I want to release high-quality products as soon as possible. Probably like you, I want to make my Django product ideas become reality as soon as possible.

That's why I built Photon Designer - an entirely visual editor for building Django frontend at the speed that light hits your eyes.

Click here to get Photon Designer.