Running Python in WebAssembly
Adam Reese and Matt Butcher
python
webassembly
scripting
wagi
Python is the second most popular programming language. Thanks to a few different efforts, it is now possible to compile the Python interpreter to WebAssembly. It is now possible to run Python inside of a WebAssembly runtime.
This post illustrates how to write a server-side WebAssembly app in Python.
This article is somewhat outdated. Spin now supplies a full Python SDK and supports advanced WebAssembly Components. We recommend using the SDK instead of WAGI compatibility.
Python, Scripting Languages, and WebAssembly
WebAssembly is a binary executable format. There are a few ways to get source code to run in a WebAssembly runtime. One is to compile the source code directly to the WebAssembly. C, Rust, and AssemblyScript all take this approach.
A second way to run code in WebAssembly is to compile an interpreter to WebAssembly, and then pass the code to the interpreter. For scripting languages, this is often the best way. Ruby, JavaScript, and (you guessed it) Python all run this way.
When a Python script is executed on a native runtime (like the python binary on your system), the source code is loaded and executed at runtime. There is no compilation step necessary. In WebAssembly, the process is similar: Load the python.wasm module into the WebAssembly runtime and then point it toward the script you want to execute.
Therefore, the first step to running Python scripts in WebAssembly is to get a WebAssembly Python module. It is possible to build your own, as the CPython project supports this. We prefer SingleStore’s WASI-enabled Python WebAssembly build, which provides a minimal toolkit to bridge Python and WebAssembly. Building either of these from source requires a lot of configuration and can take quite a while. So for this example, we have prebuilt and pre-configured a Python environment that you can use for your own projects. (This repo also has a sample Wagi setup).
What is Wagi?
At Fermyon, we are less interested in writing WebAssembly for the browser, and more interested in cloud-side WebAssembly. We think WebAssembly provides the right foundation for rethinking microservices. In this post we will be writing a Wagi app.
Wagi is a specification for running WebAssembly modules as CGI 1.1 applications. When we first wrote Wagi, we coupled the specification and implementation. But soon there were multiple implementations of Wagi, like Wagi.NET for the .NET framework. And we at Fermyon added full support for Wagi into our powerful Spin engine. So the good news is that when you write Wagi applications, you can run them in different environments.
Wagi works in a very simple way: When an inbound HTTP request is received, the runtime (Wagi, Wagi.NET, Spin, or whatever) translates that request into environment variables and file objects. Why? Because files and environment variables are easy to use and nearly universally supported. This was good news when CGI was invented. And it’s good news for WebAssembly.
Writing a Simple Module
Let’s write a simple module that illustrates how Wagi works by printing the information it receives. We will write the script in a few steps so we don’t take on too much at once.
We will create a file called code/env.py and write the beginning of our code.
import sys
print('Content-Type: text/plain; charset=UTF-8')
print('Status: 200 OK')
print()
print('Hello from python on', sys.platform)
In the example above, we import the sys library so we can use sys.platform. Then we print the Wagi preamble followed by an empty line (print()). The preamble takes the form of HTTP headers. And at the very least, it must indicate the Content-Type of the content that will follow. For our example, we can use the content type text/plain because we are printing plain text. Were we to send back HTML, we’d use Content-Type: text/html. And if we were to return a PNG image, we’d send Content-Type: image/png. This line tells the Wagi engine (Wagi, Spin, Wagi.NET or whatever) how to handle the data. And, in fact, this information is sent all the way back to the browser.
While not necessary, we also included the Status: 200 OK line, which tells the Wagi engine that the result is good. If, during the course of running a Wagi program, we encountered an error, we might return Status: 500 Internal Server Error or Status: 404 Not Found.
Again, the print() line just prints an empty line, which tells the Wagi engine that we are done with the preamble. Everything after this goes straight to the browser.
The last line in our example prints Hello from python on and then the name of the platform. This is a regular Python script, so if we ran the same program with our system’s python3 command, this is what we would see:
$ python3 code/env.py
Content-Type: text/plain; charset=UTF-8
Status: 200
Hello from python on darwin
But when we run it as a Wagi app, it will look just slightly different because the preamble will be interpreted by Wagi. Let’s set up wagi (a server for Wagi applications) and execute the above.
Configuring Wagi
The easiest way to get a Wagi server is to download a prebuilt binary from the Wagi releases page.
Next, we need to create a modules.toml file for Wagi. (If you are running a preview release of Spin, you will want to write a spin.toml. Consult Spin’s documentation for examples.)
For us, the modules.toml is only five lines:
[[module]]
route = "/"
module = "opt/wasi-python/bin/python3.wasm"
volumes = { "/code" = "code", "/opt" = "opt" }
argv = "python /code/env.py ${ARGS}"
- The
route is set to /, which means this module will listen at http://localhost:3000/
- For Python, the
module line always points to the Python interpreter. In our case (from the GitHub repo), python3.wasm is located in opt/wasi-python/bin/python3.wasm.
- The
volumes directive tells Wagi to create two filesystems for our WebAssembly module. Remember, WebAssembly modules don’t get access to the filesystem unless we explicitly give it permission to do so. Here, we tell Wagi to take our code directory, and mount it to /code inside the module, and similarly opt is mounted to /opt.
- Finally,
argv is the command that python3.wasm will run. This is necessary for a scripting language. In our case, it is running python /code/env.py (the .py file we just wrote), and is passing it any arguments (${ARGS}) that the script received as query parameters. We’ll see this in action later.
Now we need to run Wagi. If you’re using the GitHub repo, you can use the make serve command. But let’s take a look at what it takes to run Wagi. At minimum, you will need this:
$ wagi \
-e 'PYTHONHOME=/opt/wasi-python/lib/python3.11' \
-e 'PYTHONPATH=/opt/wasi-python/lib/python3.11' \
-c modules.toml
Why the two -e lines? These set two environment variables (PYTHONHOME and PYTHONPATH) that tell python3.wasm where we put all of the Python libraries. Taking a look back at modules.toml, we added a volume for /opt. We need that because it has all of the important core Python files. So these two -e lines are almost always necessary for running Python code in Wagi.
Finally, the -c modules.toml merely tells Wagi to load its configuration from modules.toml.
Running the command will start a webserver on http://localhost:3000/. We can use the curl commandline HTTP client to test out our script above. (Of course, you can also just use your regular browser for this.)
$ curl http://localhost:3000/
Hello from python on wasi
Note that the preamble, the Content-Type and the Status, don’t print because Wagi converted those into HTTP headers. But if we were to use the -v flag for curl (or turn on developer tools in a web browser) we would see these in the HTTP headers.
We can see that the Python code print('Hello from python on', sys.platform) rendered Hello from python on wasi.
Printing Arguments
In our Wagi modules.toml, we set an argv directive that included ${ARGS}. This will send all of the HTTP query parameters as arguments to the Python script. To print them, let’s add a few more lines at the end of our Python script.
import sys
print('Content-Type: text/plain; charset=UTF-8')
print('Status: 200')
print()
print('Hello from python on', sys.platform)
print()
print('### Arguments ###')
print()
print(sys.argv)
If you haven’t stopped your Wagi server, you can just run curl http://localhost:3000 again. If you stopped it, just rerun that wagi -e... command from the previous section.
$ curl localhost:3000
Hello from python on wasi
### Arguments ###
['/code/env.py']
Note that we only see one argument here – the script name that we passed on our modules.toml’s argv line: argv = "python /code/env.py ${ARGS}". But let’s try something a little different. Let’s add a query string to the URL in curl:
$ curl localhost:3000\?arg1=value1\&arg2=value2
Hello from python on wasi
### Arguments ###
['/code/env.py', 'arg1=value1', 'arg2=value2']
Now the query string ?arg1=value1&arg2=value2 has been added to the arguments array as two separate values. This is an easy way to get query parameters into your program.
Now we’re ready to move on to the next chunk of Python code.
Printing Environment Variables
Our script is going to get a little longer. We will add code to print all of the environment variables received from Wagi.
import sys
import os # <-- THIS IS NEW! We need it to print env vars
print('Content-Type: text/plain; charset=UTF-8')
print('Status: 200')
print()
print('Hello from python on', sys.platform)
print()
print('### Arguments ###')
print()
print(sys.argv)
print()
print('### Env Vars ###')
print()
for k, v in sorted(os.environ.items()):
print(k+':', v)
The important part here is the last two lines:
for k, v in sorted(os.environ.items()):
print(k+':', v)
Here, we loop through all of the environment variables that Wagi gave us. We do this using the regular os.environ standard library object. Try running this script with your system’s python3 to see what it normally does. Then point curl or your web browser at our http://localhost:3000 URL and check out the results:
$ curl localhost:3000
Hello from python on wasi
### Arguments ###
['/code/env.py']
### Env Vars ###
AUTH_TYPE:
CONTENT_LENGTH: 0
CONTENT_TYPE:
GATEWAY_INTERFACE: CGI/1.1
HTTP_ACCEPT: */*
HTTP_HOST: localhost:3000
HTTP_USER_AGENT: curl/7.77.0
PATH_INFO:
PATH_TRANSLATED:
PYTHONHOME: /opt/wasi-python/lib/python3.11
PYTHONPATH: /opt/wasi-python/lib/python3.11
QUERY_STRING:
REMOTE_ADDR: 127.0.0.1
REMOTE_HOST: 127.0.0.1
REMOTE_USER:
REQUEST_METHOD: GET
SCRIPT_NAME: /
SERVER_NAME: localhost
SERVER_PORT: 3000
SERVER_PROTOCOL: HTTP/1.1
SERVER_SOFTWARE: WAGI/1
X_FULL_URL: http://localhost:3000/
X_MATCHED_ROUTE: /
X_RAW_PATH_INFO:
Suddenly we have a LOT of output. Some of the information there comes from the -e values we passed into Wagi:
PYTHONHOME: /opt/wasi-python/lib/python3.11
PYTHONPATH: /opt/wasi-python/lib/python3.11
Any environment variables that begin with HTTP_ are forwarded on from the HTTP client (curl or the web browser). For example, HTTP_USER_AGENT has the contents of the User-Agent HTTP header that web browsers send with every request.
The rest all comes from Wagi itself, which has translated all of the HTTP information into environment variables. While we won’t cover them here, you can read through the list of environment variables in the Wagi documentation.
Why does Wagi provide us all of this information? In theory, each piece of information provided here is useful information when building sophisticated web apps. For example, values like PATH_INFO can be used to construct URLs and HTTP_USER_AGENT can be used to determine what browsers are visiting your site.
Printing a Listing of Files
When we started Wagi, we mounted our code directory to the path /code. Now we can take a quick look at what that directory looks like inside of the WebAssembly environment. We can add a few lines to the end of our script:
print()
print('### Files ###')
print()
for f in os.listdir('/code'):
print(f)
Here, we use os.listdir to see all of the files in the directory /code. If we use curl to run the script now, we’ll see the directory listing:
$ curl localhost:3000
Hello from python on wasi
### Arguments ###
['/code/env.py']
### Env Vars ###
AUTH_TYPE:
CONTENT_LENGTH: 0
# ... More env vars...
### Files ###
env.py
Yup, there’s only one file in the code/ directory. If you wish, you can add a few more files and see how that changes the output.
Logging
As a very brief final topic, let’s look at logging. It is often useful to send messages to a log file so that you (as the developer) can troubleshoot failures. In Wagi, any message sent to the STDERR file handle is automatically logged. And the really cool thing about Python is that the core logging package does this for us.
So all we need to do is something like this:
import logging
logging.warn('Logging important warnings')
And now when we run the script from curl or the web browser, we will see this in Wagi’s error log:
WARNING:root:Logging important warnings
Conclusion (and a Few Warnings)
With a version of Python compiled to WebAssembly, it is easy to write Wagi apps. We’ve seen how to set up Python for WebAssembly, and also how to write Python scripts that use the appropriate formatting for Wagi.
There are a few things to keep in mind when writing Python in WebAssembly:
- Currently, there is no support for network sockets or threading, which means some libraries will not work. Support for these things is on the horizon, so the problem will be solved.
- Python that relies on external C libraries will not work. We think that this problem will be solved in the future, too.
- As an extension, modules installed with Pip may also not work because they use the unsupported features described above.
As frustrating as that may be right now, not only will these problems be resolved soon, but an awesome feature of WebAssembly will make that less important. Soon, it will be possible to import WebAssembly libraries written in other languages. So a library from Rust can be imported into Python and used without you ever having to know anything about Rust! We are optimistic that this feature will be absolutely game changing.
For now, though, what we can do with Python is just a little more limited than we would like. Still, there are many things you can do with Python in WebAssembly, and Wagi makes it easy to write web apps and microservices this way.