Building a social app with Spin (3/4): Post API and Spin SDK Bindings
Justin Pflueger
social
spin
wasm
go
wit
postgres
Hello Fermyon friends! Justin Pflueger here with part 3 of our blog series ‘Building a social app with Spin’. In this post we’re going to add a Spin component with a REST API for creating a social media post using our Go SDK. We’ll also take a look at how you can add support for a feature that may be present in other Spin SDKs but may not be available yet in your SDK of choice.
If you haven’t read the first two articles about setting up the project and performing OAuth2 token verification, you can find the previous articles here:
The team here at Fermyon is working diligently to improve support for more programming languages, JavaScript/TypeScript and Python are the two most recent additions. Things are moving really quickly in the world of WebAssembly and Spin, just since the last blog post the team has released Spin v0.9 and v0.10! When things are moving this quickly we are naturally going to find some of the rough edges. In my case, I chose to use the Go SDK for this component but later realized that the Go Spin SDK was missing support for outbound Postgres calls. Today we’ll be taking a look at how we can add support to the Go SDK for a feature that is already present on the host side (like Fermyon Cloud) but missing from the guest SDK. First let’s cover our bases and walk through setting up the Go component.
Setting up the Go Component
Official Go support is starting to pick up momentum based on GitHub issues but for now we’re using TinyGo. If you haven’t used the Go SDK yet, take a minute to follow our setup guide. The only difference is that I’m using TinyGo v0.27.0. No surprises here as we get things setup by adding our Go component:
$ spin add http-go post --output ./api/post
Description: Post API
HTTP path: /api/post/...
Similar to our Rest API component in Rust from the first blog post, I’ve chosen to organize this component by resource (i.e. a social media Post) and let the component handle the routing. This Post API component is scoped to handle CRUD operations for a user’s posts that they have authored. At a high level, our routes will look something like this:
POST /api/post # Create
GET /api/post/<id> # Read by ID
GET /api/post # List
PUT /api/post/<id> # Update
DELETE /api/post/<id> # Delete
To handle any internal routing, I found the go-chi library to be straightforward and well maintained. First we need to add the dependency:
$ cd api/post
$ go get -u github.com/go-chi/chi/v5
go: added github.com/go-chi/chi/v5 v5.0.8
One thing to be aware of here is that you must create the router inside of Spin’s HTTP handler and not just inside the scope of the init() function. Something for our team to look at in the future would be to add support for wizer so that our router is already initialized when our code is called. Now we can wire up our initial routes:
func init() {
spinhttp.Handle(func(res http.ResponseWriter, req *http.Request) {
// we need to setup the router inside spin handler
router := chi.NewRouter()
// mount our routes using the prefix
routePrefix := req.Header.Get("Spin-Component-Route")
router.Mount(routePrefix, Post())
// hand the request/response off to chi
router.ServeHTTP(res, req)
})
}
func PostRouter() chi.Router {
posts := chi.NewRouter()
posts.Post("/", createPost)
posts.Get("/", listPosts)
posts.Get("/{id:[0-9]+}", readPost)
posts.Put("/{id:[0-9]+}", updatePost)
posts.Delete("/{id:[0-9]+}", deletePost)
return posts
}
Because our component is assigned to the /api/post/... prefix, I needed to tell chi about that prefix. The easiest way I found was to get the prefix from the request’s headers and assign a child router to that prefix. Let’s run a quick test to make sure we’re on the right track:
$ curl -XGET http://127.0.0.1:3000/api/post/1
Read post not yet implemented
$ curl -XGET http://127.0.0.1:3000/api/post
List posts not yet implemented
$ curl -XPOST http://127.0.0.1:3000/api/post
Create post not yet implemented
$ curl -XPUT http://127.0.0.1:3000/api/post/1
Update post not yet implemented
$ curl -XDELETE http://127.0.0.1:3000/api/post/1
Delete post not yet implemented
Exactly as expected, the router is calling the correct handler functions and returning the prescribed response. The last piece of setup we need here is to define our Post resource. Several design considerations here. Because our social media app is focused on sharing snippets of code with some commentary, we want the post to be generic enough that we could support multiple types of posts. For instance, our MVP is a GitHub permalink but in the future we would like to support something like a GitHub gist. We also need to take visibility into account here so a user can control who sees their post. Keeping things simple, I ended up with this structure:
type Post struct {
ID int // auto-incremented by postgres
AuthorID string // foreign key to user's id
Content string // anything the poster wants to say about a piece of code they're sharing
Type string // post could be a permalink, pasted code, gist, etc.
Data string // actual permalink, code, gist link, etc.
Visibility string // basic visibility of public, friends, etc.
}
// enumerated values for type
var postTypes = []string{
"permalink-range",
"code",
}
// enumerated values for visibility
var postVisibilities = []string{
"public",
}
There are more Go-idiomatic ways to handle the enumeration but I didn’t want to return a meaningless integer, the response data should be obvious on first-glance and looking up an enumeration value is a poor experience in my opinion. I think that covers our initial setup. Let’s move on to parsing/serializing the object with JSON.
TinyGo and JSON (Un)Marshalling
This is where things start to get a little messy. TinyGo doesn’t support using the encoding/json package from the standard library. In fact, most libraries that depend on the reflect system package I have found to not work in TinyGo because most of those methods remain unimplemented by TinyGo. This definitely makes what we’re trying to do a challenge. Hopefully when official Go support lands this experience will improve. In the meantime, I found two projects that can do JSON (un)marshalling: tinyjson and fastjson. Looking at the code frequency in GitHub, neither seem to be seriously maintained anymore. I ended up choosing fastjson because it has a commit in the last 3 months and it also doesn’t rely on code generation so there aren’t additional build steps for me. Here’s how we parse the json for a Post object:
func ParseJsonPost(r io.ReadCloser) (Post, error) {
var post Post
// read the request bytes into []byte
b, err := io.ReadAll(r)
if err != nil {
return post, fmt.Errorf("Error reading the request: %v", err)
}
// parse the []byte array
var p fastjson.Parser
if val, err := p.ParseBytes(b); err != nil {
return post, fmt.Errorf("Error parsing json: %v", err)
} else {
post.ID = val.GetInt("id")
post.AuthorID = string(val.GetStringBytes("author_id"))
post.Content = string(val.GetStringBytes("content"))
post.Type = string(val.GetStringBytes("type"))
post.Data = string(val.GetStringBytes("data"))
post.Visibility = string(val.GetStringBytes("visibility"))
return post, nil
}
}
This isn’t ideal as I have to maintain the parsing any time I update the Post object but it does work. The fastjson library doesn’t support writing a struct as json. In the future I’ll improve this with a string build but for now, a simple string format will work:
func (post *Post) ToJson() string {
return fmt.Sprintf(`{
"id": %v,
"author_id": "%v",
"content": "%v",
"type": "%v",
"data": "%v",
"visibility": "%v"
}`,
post.ID,
post.AuthorID,
post.Content,
post.Type,
post.Data,
post.Visibility)
}
Now to actually parse the JSON from the request. I followed a Go idiom here and decided to use middleware to parse the post from the request’s body. I’m not entirely convinced about the usage of middleware and request’s context but this is a pattern I found in chi examples so I just ran with it:
func PostCtx(next http.Handler) http.Handler {
return http.HandlerFunc(func(res http.ResponseWriter, req *http.Request) {
var post Post
var err error
if req.ContentLength > 0 && req.Header.Get("Content-Type") == "application/json" {
if post, err = ParseJsonPost(req.Body); err != nil {
// parsing failed end the request here
msg := fmt.Sprintf("Failed to parse the post from request body: %v\n", err)
renderBadRequestResponse(res, msg)
return
}
if err = post.Validate(); err != nil {
msg := fmt.Sprintf("Request body failed validation: %v\n", err)
renderBadRequestResponse(res, msg)
return
}
}
ctx := context.WithValue(req.Context(), postCtxKey{}, post)
next.ServeHTTP(res, req.WithContext(ctx))
})
}
Our handlers can now look for the post or the identifier in the request’s context:
func createPost(res http.ResponseWriter, req *http.Request) {
post := req.Context().Value(postCtxKey{}).(Post)
//TODO: actually write the post to the database
post.RenderJsonResponse(res)
}
func readPost(res http.ResponseWriter, req *http.Request) {
var post Post
if id, err := getPostId(req); err != nil {
msg := fmt.Sprintf("Failed to parse URL param 'id': %v", id)
renderBadRequestResponse(res, msg)
return
} else {
// TODO: actually get the post from the database
post = Post{
ID: id,
}
}
post.RenderJsonResponse(res)
}
That should do it for handling JSON, now we need to actually persist the data to Postgres.
Adding Postgres support to Spin’s Go SDK
As I mentioned earlier, Spin’s Go SDK doesn’t currently support outbound Postgres calls. This is a blocker for us because that is the database that we chose. Instead of backtracking at this point and using another SDK that does support Postgres, I wanted to take a stab at adding support to the Spin SDK. We already know that spin hosts (like Fermyon Cloud) can handle outbound calls to Postgres because we’ve already used it in the Spin Rust SDK.
Looking at the Spin repository, we can see how the Host/Guest interface is defined by Wasm Interface Type (a.k.a. WIT) files like outbound-pg.wit, pg-types and rdbms-types. This is where I had to do a bit of a deep-dive on WIT to understand the semantics a bit. In summary, WIT is an interface definition language (a.k.a IDL) similar in concept to OpenAPI for Rest APIs or protobuf for gRPC. Next we need to use the WIT files to generate the code that marshals the calls from our component (i.e. guest program) to the host (i.e. Fermyon Cloud or local spin up). We can see how that is done for other SDK features by taking a look at the sdk/go/Makefile in the Spin repository. For example, this is how outbound redis support works today:
GENERATED_OUTBOUND_REDIS = redis/outbound-redis.c redis/outbound-redis.h
...
.PHONY: generate
generate: $(GENERATED_OUTBOUND_REDIS) $(GENERATED_SPIN_REDIS)
...
$(GENERATED_OUTBOUND_REDIS):
wit-bindgen c --import ../../wit/ephemeral/outbound-redis.wit --out-dir ./redis
Based on the above Makefile, we know that the Go SDK components use wit-bindgen to generate bindings in C and then wrap those bindings with Go functions and types. Important to note here that Spin uses version 0.2 of wit-bindgen because (as I understand it) newer versions require breaking changes to the WIT syntax. Let’s take the next step and install wit-bindgen:
$ cargo install --git https://github.com/bytecodealliance/wit-bindgen --rev cb871cfa1ee460b51eb1d144b175b9aab9c50aba wit-bindgen-cli
Now we have the tool and can actually generate the bindings. I followed the patterns of other Go SDK features and added similar lines to the Makefile to generate the bindings. The generated code is in C and like most generated code, it isn’t very readable so I won’t bore you with it here but if you’re interested you can check out the following commit.
This is where things get a little tricky and my lack of experience with Go or C shines through (maybe a little bit of laziness as well). I could start writing some Go code to wrap the generated C bindings. But I did find a newer addition to wit-bindgen that does support generating TinyGo bindings and I’d like to move on to actually persisting the data to the database. Worst case scenario, the TinyGo bindings don’t work and I’m back to writing the bindings myself. In order to use the newer wit-bindgen-go generator, we’ll need to install the v0.4.0 version of wit-bindgen and adjust our WIT file to match the new syntax that wit-bindgen is expecting. Here’s how I installed the new version and generated the TinyGo bindings:
$ cargo install --git https://github.com/bytecodealliance/wit-bindgen --rev 197d3be2d64e3d40c4e58ec42c255632f9e39486 wit-bindgen-cli
After installing the latest version of wit-bindgen with the TinyGo generator, I had to hack together a new WIT file that was compatible with the new syntax. This was surprisingly simple even though I’ve never actually used WIT before. To do this I created a temporary file and copied our existing WIT definitions into the new format, here’s a snippet of that:
interface rdbms-types {
// pasted in the contents of rdbms-types.wit, omitted for brevity
}
interface pg-types {
// pasted in the contents of pg-types.wit, omitted for brevity
}
default world outbound-pg {
// changed to 'use' statements from 'import' in the existing outbound-pg.wit file
use self.rdbms-types.{db-value,row,db-data-type,column,row-set,parameter-value}
use self.pg-types.{error}
// sourced from outbound-pg.wit
import query: func(address: string, statement: string, params: list<parameter-value>) -> result<row-set, error>
import execute: func(address: string, statement: string, params: list<parameter-value>) -> result<u64, error>
}
Now we can generate the bindings and see what we get as output:
$ wit-bindgen tiny-go ./outbound-pg.wit --out-dir ./postgres
Generating "./postgres/outbound-pg.go"
Generating "./postgres/outbound-pg_types.go"
Generating "./postgres/outbound_pg.c"
Generating "./postgres/outbound_pg.h"
Generating "./postgres/outbound_pg_component_type.o"
$ rm ./postgres/outbound_pg_component_type.o
$ rm ./postgres/outbound-pg_types.go
Essentially the same as the C generator, just with some additional go files and an object file for the component model (I assume). Taking a peek at the generated outbound-pg_types.go types, it seems to be generating types for Option and Result but I don’t particularly find those useful as the idioms in Go are just different so I removed that file as well. From here, I tried to pull out the generated types into their own files for my own understanding. I also removed the type prefixes from the generator because no one wants to type RdbmsTypesParameterValueKindBoolean. We’ll add this as feedback to the wit-bindgen maintainers and see if there is a better solution for later versions. If you’re still curious, the output after I cleaned it up a little bit can be found in the following commit:
At this point, I was able to add an example using the generated output and submit a pull request to the Spin repository. I thought that I would have to go find another SDK to use but (knock on wood) it just worked! You can find the pull request below:
Using the Postgres Bindings
Now that we have Postgres in the Spin Go SDK, we need to adjust our go dependency to use my local copy until the pull request is accepted and a new version of Spin is released. A quick update to the go.mod file will do that:
replace github.com/fermyon/spin/sdk/go => /Users/j12r/Development/fermyon/spin/sdk/go
This does mean that if you were to pull down my changes, you will have to clone my fork of Spin and adjust the path for your local machine. But hey, “it works on my machine” right? :D
To Be Continued
While I was able to generate the bindings for Postgres support in the Spin SDK, I’ve ran over my timebox for this blog post. So my changes for this post aren’t quite complete but I still wanted to get this out there. All we have left to do is write some Go code to actually perform the database operations. If you’re following along with the pull requests, this pull request will be marked as a draft until Postgres support is added to the Spin repository and I’ll continue adding code to perform the database operations. If time allows, I’ll write up a 3.5 article and cover how to do that using our new feature. Hopefully this helps you understand that if you’re using one of our SDKs, even someone with limited knowledge of WebAssembly and WIT can hack together support for the language of their choice and the engineering team at Fermyon will help you get those changes cleaned up and added to the Spin SDK.
PS, if you didn’t catch the livestream on YouTube from March 7th with David Flanagan, you can find it on our YouTube channel