Why am I getting a configuration error in OpenAI for exceeding token limit despite my input being under 8192 tokens?

Help
1

This topic was automatically generated from Slack. You can find the original thread here.

Hi there - having a slight trouble with a create embeddings step with OpenAI - I’m using an input that is significantly below 8192 tokens, but I keep getting this error:

“Configuration error
Element #0 is more than 8192 tokens in length. Each input must not exceed 8192 tokens in length.” Per the OpenAI tokenizer, i’m seeing 1607 tokens & 9034 characters - is the pipedream validation kicking off of the character count vs. tokens since they upped ada’s max token capabilities? cc

2

Just a bad bug! I’ll correct this to actually count tokens and ship a fix.

Are you familiar with Node.js / Python? Could you use the OpenAI API clients in a code step to create the embeddings for now?

3

amazing, thank you so much! We have used the Node.js and Python steps, but the way we’re doing it is quite complex and using variations of both a code step and the integration one - all good will hold for a fix for now :slightly_smiling_face: thank you again!