Linking Fermyon Cloud Spin Apps to Custom Key Value Stores
MacKenzie Adam
spin
cloud
kv
Today we’re excited to announce support for linking Spin applications to custom key value stores on Fermyon Cloud. With spin cloud version 0.7.0 or newer, application developers can link their applications to their desired key value stores during deployment. Developers can later change which stores are linked to the application without the need to recompile or redeploy their source code.
If you’ve tried out SQLite on Fermyon Cloud before, you’re likely no stranger to the concept of linking applications to resources. However, if this is the first time you’ve heard of links, labels, or even Fermyon Cloud’s Key Value Store, fear not! Let’s build some context together on how to use these resources to build flexible and secure applications on Fermyon Cloud.
Spin apps are inherently stateless and ephemeral workloads, which means they rely on external data stores to persist state in between invocations. One popular state store used by Spin developers is Fermyon Cloud’s Key Value Store. With Fermyon Cloud’s Key Value Store, developers can store and retrieve non-relational data performantly without having to set up or manage any infrastructure themselves. In addition to the simplified ops workflow, developers also do not need to worry about compromising on portability as Fermyon Cloud’s Key Value Store is built on top of Spin’s key/value API. This means developers can run their apps locally or in Fermyon cloud without having to change anything about their application.
This portability inspired us to think creatively on how to empower users to flexibly connect their Spin applications to data stores in Fermyon Cloud while still maintaining a simple and ergonomic ops flow. Up until this point, Spin applications were constrained to a 1:1 relationship with a key-value store when running in Fermyon Cloud, and the two resource lifecycles were tied together. This design decision was convenient for a scratchpad environment, but introduced a unique set of challenges as developers graduated to more complex, production workflows. Uses cases include:
- Testing the same Spin app against a canary store and a production store.
- Sharing a KV store among applications
- A/B testing a Spin application’s behavior by seeding it with data from two unique stores.
When we launched Fermyon Cloud SQLite, we solved for these pain points by introducing links and labels for database management, and today we’re announcing link and label support for Fermyon Cloud’s Key Value store as well. Labels are strings provided by developers in the component manifest that describe a specific key value store or SQLite database it should have access to. If you’ve worked with Spin the past, odds are you’ve used the label default at least once in the key value or SQLite section of the component manifest before. With links and labels, we’re building on that functionality to allow developers to dynamically link their Spin applications to different stores without having to redeploy their application. The benefits to this approach include:
- Easy Sharing: Share your Fermyon Cloud resource across applications effortlessly.
- Resource Creation Control: You decide when to create and delete Fermyon Cloud resources.
- Decoupled Resource Lifecycle: Allow Fermyon Cloud resources to have lifecycles independent of your Spin application’s lifecycle.
- Seamless Cloud Integration: Cloud experience smoothly integrates with local Spin development, no runtime configuration changes required.
- Dynamic App Resource Configuration: Change the data resource your application points to without rebuilding it.
A/B Testing With Links and Labels
To make this more concrete, let’s use a specific example. Here we have a Spin application (written by Fermyon engineer Caleb Schoepp) that renders a UI which takes in user input and returns the sentiment analysis as to whether the feedback was positive, unsure, or negative using Fermyon’s Serverless AI inferencing feature. The results are then cached in a key-value store.
Testing Our Prompt Engineering
Now, my prompt engineering is a bit rusty. To see if I can make any improvements, I’m going to A/B test two different prompts to see if that increases the accuracy of the sentiment analysis as we review our users input over time.
Here we have our first prompt (Case A). This will be used to prompt llama-chat-2 when we feed it user input to analyze as positive, neutral, or negative. In this prompt, we are specifying that the model should behave as a chat bot:
let prompt1 = `
<<SYS>>
You're a bot specializing in emotion detection. Craft responses indicating sentiment as positive, negative, or neutral.
<</SYS>>
[INST]
Emulate the given examples:
User: Hey, how's it going?
neutral
User: I just aced my exam!
positive
User: The weather is ruining my day.
negative
[/INST]
User: {SENTENCE}`;
And with our second prompt (Case B), where we’re a bit less prescriptive and tell it that it’s role is to analyze sentiments and generate appropriate responses:
let prompt2 = `
<<SYS>>
Your role is to analyze sentiments and generate appropriate responses. Choose between positive, negative, or neutral.
<</SYS>>
[INST]
Mirror the examples provided:
User: What a beautiful sunrise!
positive
User: My car broke down again.
negative
User: Just finished my chores.
neutral
[/INST]
User: {SENTENCE}`;
Creating Two Key Value Stores on Fermyon Cloud
Now, to organize my results, I will create two separate key-value stores on Fermyon Cloud, and pre-seed each one with a key-value pair of {”prompt”: {prompt}}:
$ spin cloud kv create case-a
Key value store "case-a" created
$ spin cloud kv create case-b
Key value store "case-b" created
If we run a spin cloud kv list we can quickly verify we have not connected these key-value stores to a Spin application yet:
$ spin cloud kv list
Key value stores not linked to any app
+-----------------+
| Key Value Store |
+=================+
| case-a |
|-----------------|
| case-b |
+-----------------+
Now it’s time to pre-seed each key-value store with our prompt:
$ spin cloud kv set --store case-a prompt="
<<SYS>>
You're a bot specializing in emotion detection. Craft responses indicating sentiment as positive, negative, or neutral.
<</SYS>>
[INST]
Emulate the given examples:
User: Hey, how's it going?
neutral
User: I just aced my exam!
positive
User: The weather is ruining my day.
negative
[/INST]
User: {SENTENCE}"
$ spin cloud set --store case-b prompt="
<<SYS>>
Your role is to analyze sentiments and generate appropriate responses. Choose between positive, negative, or neutral.
<</SYS>>
[INST]
Mirror the examples provided:
User: What a beautiful sunrise!
positive
User: My car broke down again.
negative
User: Just finished my chores.
neutral
[/INST]
User: {SENTENCE}"
Great, now we have two key-value stores pre-seeded with the two prompts we want to test.
Building Spin Application to Link to Key Value Stores
Now, let’s look at how our application will handle the A/B test. In our index.ts file, we will connect to a key-value store with the label prompt_store. After, we will check to see if the key “prompt” exists in the store. Otherwise, we have a safety default prompt:
async function performSentimentAnalysis(request: HttpRequest) {
// Parse sentence out of request
let data = request.json() as SentimentAnalysisRequest;
let sentence = data.sentence.trim();
console.log("Performing sentiment analysis on: " + sentence);
// Prepare the KV store
let kv = Kv.open("prompt_store");
// Set fallback prompt in case it cannot be retreived
let defaultPrompt = `
<<SYS>>
You are a bot that generates sentiment analysis responses. Respond with a single positive, negative, or neutral.
<</SYS>>
[INST]
Follow the pattern of the following examples:
User: Hi, my name is Bob
neutral
User: I am so happy today
positive
User: I am so sad today
negative
[/INST]
User: {SENTENCE}`;
// Grab prompt from KV Store if it exits, otherwise set to this default
let prompt = kv.exists("prompt") ? decoder.decode(kv.get("prompt") || new Uint8Array) : defaultPrompt;
console.log("Prompt is:" + prompt);
<...>
From there, the application’s logic will check if the sentiment is cached for the user input, otherwise it will run an inferencing operation:
// Checks if user sentence is in KV cache
let cachedSentiment = kv.get(sentence);
if (cachedSentiment !== null) {
console.log("Found sentence in KV store returning cached sentiment");
return {
status: 200,
body: JSON.stringify({
sentiment: decoder.decode(cachedSentiment),
} as SentimentAnalysisResponse),
};
}
console.log("Sentence not found in KV store");
// Otherwise, perform sentiment analysis
console.log("Running inference");
let options: InferencingOptions = { maxTokens: 6 };
let inferenceResult = Llm.infer(
InferencingModels.Llama2Chat,
prompt.replace("{SENTENCE}", sentence),
options
);
<...>
For a more indepth review of the sentiment analysis application, please check out Caleb’s blog post, How I Built an AI Inferencing API with Lllama2 on Spin.
Composing Application Manifest and Runtime Configuration File for Local Testing
Now we must configure the application manifest so that our components can access critical resources such as the key-value store and Large Language Model (LLM). This Spin application has three components:
sentiment-analysis - the API that receives user input, checks the cache, sends the sentence to the LLM for sentiment analysis, and updates the cache appropriately.kv-explorer - an observability component allowing us to view the key-value pairs inside a given key-value store. Note that it’s mandatory to grant access to the key value stores you’d like to view.ui - responsible for rendering the website.
We’ve given both sentiment-analysis and kv-explorer access to a key value store with the label promptstore. This will allow our API to send the correct prompt to our inferencing engine and cache the subsequent results.
spin_manifest_version = 2
[application]
name = "sentiment-analysis-test"
version = "0.1.0"
[[trigger.http]]
route = "/api/..."
component = "sentiment-analysis"
[component.sentiment-analysis]
source = "target/spin-http-js.wasm"
allowed_outbound_hosts = []
exclude_files = ["**/node_modules"]
key_value_stores = ["promptstore"]
[component.sentiment-analysis.build]
command = "npm run build"
watch = ["src/**/*", "package.json", "package-lock.json"]
[[trigger.http]]
route = "/..."
component = "ui"
[component.ui]
source = { url = "https://github.com/fermyon/spin-fileserver/releases/download/v0.0.1/spin_static_fs.wasm", digest = "sha256:650376c33a0756b1a52cad7ca670f1126391b79050df0321407da9c741d32375" }
allowed_outbound_hosts = []
files = [{ source = "../sentiment-analysis-assets", destination = "/" }]
[[trigger.http]]
component = "kv-explorer"
route = "/internal/kv-explorer/..."
[component.kv-explorer]
source = { url = "https://github.com/fermyon/spin-kv-explorer/releases/download/v0.10.0/spin-kv-explorer.wasm", digest = "sha256:65bc286f8315746d1beecd2430e178f539fa487ebf6520099daae09a35dbce1d" }
key_value_stores = ["promptstore"]
[component.kv-explorer.variables]
kv_credentials = "{{ kv_explorer_user }}:{{ kv_explorer_password }}"
[variables]
kv_explorer_user = { required = true }
kv_explorer_password = { required = true }
To test this locally, I’ve added a connect to Fermyon Cloud’s GPUs using the spin cloud plugin. This allows for quick local testing, without the need to download the models required for my inferencing operation
$ spin cloud-gpu init
I also created a modifed a runtime-config.toml since we are no longer using the default label. This simply tells Spin to create a store via a different label, in our case promptstore.
[llm_compute]
type = "remote_http"
url = "https://fermyon-cloud-gpu-tqbjde2w.fermyon.app/"
auth_token = "------------------------------"
[key_value_store.promptstore]
type = "spin"
path = ".spin/prompt_store.db"
The following command will deploy my Spin application locally, while using GPUs in Fermyon Cloud:
$ spin build --up --runtime-config-file runtime-config.toml
Building component sentiment-analysis with `npm run build`
> sentiment-analysis@1.0.0 build
> npx webpack --mode=production && mkdir -p target && spin js2wasm -o target/spin-http-js.wasm dist/spin.js
asset spin.js 15.7 KiB [compared for emit] (name: main)
orphan modules 5.79 KiB [orphan] 11 modules
runtime modules 937 bytes 4 modules
cacheable modules 12.7 KiB
./src/index.ts + 5 modules 12 KiB [built] [code generated]
./node_modules/typedarray-to-buffer/index.js 646 bytes [built] [code generated]
webpack 5.88.2 compiled successfully in 597 ms
Since we aren’t using either of our pre-seeded key-value stores, we’d expect the application to fall back on the default. Looking at our logs, we can quickly validate that this is the case:
Prompt is:
<<SYS>>
You are a bot that generates sentiment analysis responses. Respond with a single positive, negative, or neutral.
<</SYS>>
[INST]
Follow the pattern of the following examples:
User: Hi, my name is Bob
neutral
User: I am so happy today
positive
User: I am so sad today
negative
[/INST]
User: {SENTENCE}
Linking Spin Application to Fermyon Cloud Key Value Stores
Finally we’re ready to deploy our application to Fermyon Cloud. No further modifications are required, we’ll simply run a spin cloud deploy as shown below:
$ spin cloud deploy
Uploading sentiment-analysis-ab-test version 0.1.0 to Fermyon Cloud...
Deploying...
App "sentiment-analysis-ab-test" accesses a key value store labeled "prompt_store"
Would you like to link an existing key value store or create a new key value store?:
Use an existing key value store and link app to it
> Create a new key value store and link the app to it
We’re then prompted to link our Spin application to a key-value store. Now, we can link our Spin application to one of our provisioned key-value stores. For those who might not have completed that step or simply want an empty key-value store, there is the first option of having the spin cloud plugin create one for you.
We’ll proceed by using an existing key value store and link our app to key value store case-a with our label “promptstore”:
...
Which key value store would you like to link to sentiment-analysis-ab-test using the label "prompt_store":
> case-a
case-b
Waiting for application to become ready........ ready
View application: https://sentiment-analysis-ab-test-mbnuy7zl.fermyon.app/
Routes:
- sentiment-analysis: https://sentiment-analysis-ab-test-mbnuy7zl.fermyon.app/api (wildcard)
- ui: https://sentiment-analysis-ab-test-mbnuy7zl.fermyon.app (wildcard)
- kv-explorer: https://sentiment-analysis-ab-test-mbnuy7zl.fermyon.app/internal/kv-explorer (wildcard)
Manage application: https://cloud.fermyon.com/app/sentiment-analysis-ab-test
Visiting the custom domain that links to the UI and inputting a prompt, we can quickly check that the sentiment analysis is behaving as expected.
Visiting the key-value store explorer, we can see that the correct prompt and prompt value pair are being used for Case A and that the user input and sentiment analysis pairs are being stored as expected.
Using spin cloud kv To Manage Stores
Running spin cloud kv list will now show that our sentiment analysis application is linked to Fermyon Cloud Key Value Store Case A via the label promptstore:
$ spin cloud kv list
+-----------------------------------------------------------------------------------------+
| App Label Key Value Store |
+=========================================================================================+
| sentiment-analysis-ab-test promptstore case-a |
+-----------------------------------------------------------------------------------------+
Key value stores not linked to any app
+-----------------+
| Key Value Store |
+=================+
| case-b |
+-----------------+
Let’s switch out our stores without having to touch or modify our Spin application. First step will be to unlink our sentiment analysis app from case-a with the spin cloud unlink command:
$ spin cloud unlink key-value -a sentiment-analysis-ab-test promptstore
Key value store 'case-a' no longer linked to app sentiment-analysis-ab-test
Now we can relink our Spin application to case-b with the spin cloud link command:
$ spin cloud link key-value --app sentiment-analysis-ab-test --store case-b promptstore
Key value store "case-b" is now linked to app "sentiment-analysis-ab-test" with the label "promptstore"
A quick check with spin cloud kv list validates everything is set up as expected:
$ spin cloud kv list
+-----------------------------------------------------------------------------------------+
| App Label Key Value Store |
+=========================================================================================+
| sentiment-analysis-ab-test promptstore case-b |
+-----------------------------------------------------------------------------------------+
Key value stores not linked to any app
+-----------------+
| Key Value Store |
+=================+
| case-a |
+-----------------+
We can run a quick user experience test by visiting the UI again and logging piece of feedback. This time we would expect the feedback to be evaluating with the Case B prompt since we’ve switched out our backing key value stores.
Visiting the key value store explorer, we can validate that the Case B prompt is being evaluated and that the user input and sentiment analysis pairs are being stored as expected.
We can run our website for the desired period of time using Case B prompt until we’ve reached a desired sample size and switch back to Case A.
Linking Both Key Value Stores To Our Spin App
To minimize confounding variables such as time of day Store A or Store B is linked to our application, let’s link both key value stores simultaneously to our application and have our source code randomly choose which store to use. Now that Fermyon Cloud supports links for key value stores, you can assign multiple key value stores to your Spin application.
In our application manifest, we’ll add a label for the store associated with Case A (promptstore1) and another label (promptstore2) for the store associated with Case B:
<...>
[component.sentiment-analysis]
source = "target/spin-http-js.wasm"
allowed_outbound_hosts = []
exclude_files = ["**/node_modules"]
key_value_stores = ["promptstore1", "promptstore2"]
We’ll also add a few additional lines of code to handle the store randomization in our index.ts:
// Randomly decide to use Store A or Store B
let seed = Math.random()
// Open the correct store based on random number generator
let kv = seed === 0 ? Kv.open("promptstore1") : Kv.open("promptstore2");
After recompiling our Spin application, we will deploy it again to Fermyon Cloud. The recompilation step is necessary here as we have made modifications to our source code. In the deployment step, we’ll be prompted twice to link our Spin application to a KV store as we’ve added two labels in our application manifest:
Let’s take a look at the spin cloud plugin linking our Spin app to the two seperate key value stores in action:
Now if we run spin cloud kv list we’ll see our sentiment analysis app is concurrently connected to both Spin KV Stores:
$ spin cloud kv list
+-----------------------------------------------------------------------------------------+
| App Label Key Value Store |
+=========================================================================================+
| sentiment-analysis-ab-test promptstore1 case-a
| sentiment-analysis-ab-test promptstore2 case-b |
+-----------------------------------------------------------------------------------------+
After letting the application collect user input for a while, we can visit the results in our key value explorer and see that both Case A store and Case B store should be populated with roughly the same volume of responses.
Here we see 5 responses in Case A, by inputting our label promptstore1:
And another 5 responses in Case B, by inputting our label promptstore2:
When exploring the key value pairs in case-a, we noticed that the prompt is often erroneoulsy clearling labeling sentiment such as “Dang, this is difficult!” as neutral. Therefore, in the future we’d likely proceed with case-a given it yielded higher accuracy.
Conclusion
Now you can use links and labels to manage SQLite and key value stores running on Fermyon Cloud dynamically during application runtime. To try it out yourself, install the latest version of Spin Cloud plugin (v0.7.0). We’d love to hear your thoughts on the #Cloud channel on Discord. In the meantime, here are a few resources to get you started: