PortKey Teardown

Dealing with Errors in a Prompt Manager

Today I am reviewing Portkey, a prompt management tool, to see how well it handles errors that crop up during the course of building an AI based application.

This is Portkey’s logger. Rather showing logs on the dashboard for each individual prompt they went with a universal logging panel for all prompts together. If there is an error with you prompts, this is where you will be investigating.

Changing the shape of the payload ❌

I switched the shape of the payload to a prompt from an object where each key corresponded to a form field in my simple app to an array with the same values.

To be clear, I’m not expecting this to work. What I want to understand is how it handles these kind of catastrophic changes.

In Portkey, if you click on any of the logs it brings up a side panel with a preview of the request and the response.

Prompt Manager - No errors

The prompt ran normally. There was no indication of any errors. Previewing the request, it look like the prompt just ran with blank input variable values and then returned a made-up response.

Codebase - No errors

There were no errors in the code base. The prompt ran as normal and just returned a made up response about a fitness app, rather than creating content related to the information I was passing in.

Sending too much information 🙈

Next I tried adding an extra key value to the object I was passing into my prompt. So instead of sending a payload object with data from 5 form fields, I was sending an object with 6 bits of information.

Prompt Manager - The prompt worked as normal and just ignored the extra information.

Codebase - Same, no errors or warnings, the redundant data was just ignored.

As a developer if I was sending over the necessary information to a prompt manager and it was not being used, i’d want to know about that. Especially if I’m trying to help the prompt team debug a problem where the prompts are not working as expected. At the very least I would expect the prompt manager on their end to let them know they are receiving extra information that is not being used.

Not sending enough information 🤐

I added a new input variable to the prompt in the prompt manager to see if calling a prompt with insufficient data would throw any kind of warning.

Prompt Manager - The prompt worked as normal and just ignored the missing information. I could not find any way to mark a input variable as required. they all just seem to be optional by default.

Codebase - Same, I wrapped the create method of the portkey.prompts.completions API in a try catch block and there were no errors or warnings in my console.

When I added the extra input variable in the prompt manager it did ask me to make a note of the changes made before updating the prompt. In a team scenario, I can see who made this note to track down who made the breaking change in a future review.

Great Rollbackability ✅

PortKey made it easy to rollback to a previous version of prompt when I tried to undo the mess I had made. On the dashboard for each individual prompt there is a side panel with all the version and there’s dropdown that lets you publish whichever version you want.

Returns Incomplete JSON ⚠️

I was happy to see that the advanced parameters in PortKey let you set the response format to JSON. So I set the response format to JSON and then reduced to max tokens down so that it could not complete the JSON output I was expecting.

The response was incomplete JSON.

On the developer end I do get a response object that has the headers. So I would be able to debug this and see that the reason it finished because of a length limitation ("finish_reason": "length")

However I would have liked to also see this in the prompt manager since this is clearly an avoidable and easily detectable error but I was not alerted to it in any way.

As a developer working with a client or a product team via a prompt manager I would expect more proactive error detection and clearer communication of issues so that I can be relatively confident that the prompt team is making obvious error when they update prompts.

This teardown was conducted with respect for PortKey's team and product, I hope this was useful to them, and I welcome any feedback or updates to the post.