KevinSimper

Compute is everywhere and companies like AWS and Google Cloud are some of the most profitable companies in the world and they keep growing. They make it really easy and affordable to get compute.

But getting GPU compute is still new and unsolved, GPU compute is expensive and we are very much tied to a single provider called Nvidia if you want to do something.

The easiest way to get started is definitely your own computer, but most times if not all the time your computer is simply not powerful enough, so many things are not even possible to run locally in relation to machine learning.

What is also easy is using Google Colab, it is a easy web interface that allows you to run Python through what is called a notebook, concretely a Jupyter notebook. It is really good for proving a point and visualising the output of your machine learning or data processing.

Today I want to show how streaming an AI prompt response from a CloudFlare worker. We have been used to know that ChatGPT streams the response of the large language model, it is pretty mesmerizing and adds to the fealing of a thinking mashine that you can see the dialog as it is coming.

Simulating the response

We don't want to call an AI model during development just to simulate the LLM sending the events.

We can work with streams in Node.js by creating a TransformStream, that will allow us to both write and read the stream.

let { readable, writable } = new TransformStream();

When I started building a server for handling forward_auth requests, it was like stepping into uncharted territory. Googling this concept usually leads you to comprehensive but heavyweight solutions like Authelia or middleware examples in Traefik and Caddy. These are great for robust setups but overkill when you just need a simple, functional proof of concept.

Forward authentication is a mechanism where a reverse proxy delegates user authentication to an external server. The flow looks like this:

1. A request reaches the reverse proxy (e.g., Traefik, Caddy).
2. The proxy forwards the request to an authentication server for validation.
3. The authentication server responds with either:
   1. HTTP 2xx: User is authenticated, request proceeds to the backend.
   2. Otherwise: User should be redirected to the login page. All headers are taken from the auth service and returned

When developing software, decisions often come down to trade-offs between simplicity, readability, and maintainability. One recurring choice is whether to use a function or a class for a task. While both have their place, I often lean toward functions for many scenarios. Here’s why they can be a great option.

Why Choose Functions?

Fewer Side Effects

Functions tend to be straightforward: they take inputs, produce outputs, and avoid hidden states. This simplicity makes them easier to reason about and debug. If something breaks, you don’t have to untangle a web of interdependent attributes to locate the issue.

When testing, pure functions shine. They’re predictable: give them valid inputs, and they’ll always produce the same outputs. This predictability simplifies both writing and maintaining tests.

Coming from Node.js async code is the norm, and when async/await was introduced a big shift happened since it was competing with the callback approach where you could archive the same.

In Python it is not the same, Python is by default sync and webservers relies on WSGI (Web Server Gateway Interface) like gunicorn to handle multiple connections on a server.

So python tutorials does not normally talk or show async code examples, so learning it in python is defenitely harder.

The asyncio package

This package includes a lot of the functionality on how to control flow in your now async code. This blog shows really well the different methods like async.create_task, asyncio.wait, asyncio.gather and how to use them.