How can I increase the context length for a workflow to process longer audio files without encountering a context_length_exceeded error?

Help
1

This topic was automatically generated from Slack. You can find the original thread here.

Hello. How do I increase “the context length” for a workflow? Seems like the audio that I’d like to process takes more that what I have set but I can seem to see where to edit that. Here is the message that I received: Error - Request failed with status code 400 — {“error”:{“message”:“This model’s maximum context length is 4097 tokens. However, your messages resulted in 4920 tokens. Please reduce the length of the messages.“,”type”:“invalid_request_error”,“param”:“messages”,“code”:“context_length_exceeded”}}

2

Hi Peter, that error message is not coming from Pipedream, but from the service you’re integrating with. I suspect it’s OpenAI’s API right?

3

Yes. So that increase needs to be adjusted there?

4

At this time I don’t believe it’s possible to increase the limit on OpenAPI, it’s a limitation of the service

5

Here’s an example of an Node.js code step that will reduce tokens for a conversation’s history, you can modify this to accept a single argument:

import { get_encoding, encoding_for_model } from "@dqbd/tiktoken";

// To use previous step data, pass the `steps` object to the run() function
export default defineComponent({
    key: "openai-reduce-chat-history",
    name: 'Reduce Chat History',
    type: "action",
    version: '0.0.1',
  props: {
    messages: {
      type: 'string[]',
      label: "Chat History",
      description: "An array of messages in chronological order"
    },
    max_tokens: {
      type: "integer",
      label: "Maximum Tokens",
      description: "The maximum number of tokens to include in the chat history",
      default: 7000,
    }
  },
  async run({ steps, $ }) {
    // compute the tokens per message
    const enc = encoding_for_model("gpt-3.5-turbo");

    const messages = this.messages.map(msg => {
      return {
        text: msg,
        tokens: Object.keys(enc.encode(msg)).length // calculate token usage
      }
    })
    
    const reducedMessages = messages.reduceRight((sum, curr) => {
      if(sum.totalTokens + curr.tokens <= this.max_tokens) {
        const currChat = {
          content: curr.text,
          role: 'user' // hardcoding all messages as user messages for now
        }

        return { messages: [...sum.messages, currChat ], totalTokens: sum.totalTokens + curr.tokens }
      }

      return sum;

    }, { messages: [], totalTokens: 0 })

    enc.free();


    $.export('thread_history', messages);
    $.export('reduced_history', reducedMessages)
  },
})
6

So since i’m trying to process audio and transcribe it with OpenAi…at this time, there are limitations on the file size to process. Ok. Thanks for the update.

7

Right, you’ll have to reduce the token usage after the audio file has been transcribed

8

Best of luck

9

Newb to this, but under event log it mentioned I was timing out.

I increased time out to 3 mins = 180 seconds

Then I got this error, and the event log said I was out of memory, so I increase memory up from 256mb to 1028mb, seemingly made the transcription. perhaps give that a whirl?