Eliminating 429 Errors with pd.flow.rerun

My workflow looks like this:

  1. Trigger: TradingView signal
  2. API call to Kucoin Futures to execute trades

I’m encountering 429 errors even though I’ve throttled workflow executions to 29 requests per 3 seconds (I believe their limit is 30/3s but please correct me if I’m wrong).

I’ve found this article which seems to make sense. Do I add a new step (“Python”) after my 2nd step in the workflow (“Generic API Call”) and paste this code exactly as is? I’m very inexperienced with coding so please ELI5. Thanks!!

import requests

def handler(pd: 'pipedream'):
  MAX_RETRIES = 3
  # 10 seconds
  DELAY = 1000 * 10
  run = pd.context['run']
  print(pd.context)
  # pd.context.run.runs starts at 1 and increments when the step is rerun
  if run['runs'] == 1:
    # pd.flow.rerun(delay, context (discussed below), max retries)
    pd.flow.rerun(DELAY, None, MAX_RETRIES)
  
  elif run['runs'] == MAX_RETRIES + 1:
    raise Exception("Max retries exceeded")
  
  else:
    # Poll external API for status
    response = requests.get("https://example.com/status")
    # If we're done, continue with the rest of the workflow
    if response.json().status == "DONE":
      return response.json()

    # Else retry later
    pd.flow.rerun(DELAY, None, MAX_RETRIES)

Hello @snelson1016,

I think throttling is quite a complex task to complete with an inexperienced programmer. I think to best save your time, you might need dedicated support for your usecase. For this you can subscribe to Pipedream team plan to be invited to a dedicated support channel, or you can hire a Pipedream expert using this link: Connect with a Pipedream Partner

Thank you, but prior Pipedream consultations were outrageously expensive.

Kucoin’s support confirmed that their server was overloaded. I need a simple solution to to automatically rerun failed requests within Pipedream.

@snelson1016 configuring specific steps to auto-retry is coming soon! Subscribe to this issue to get notified when we ship it.

Will this work? “{{steps.trigger.context.id}}” is configured as a parameter in step 2 (Generic API Call) to Kucoin Futures that identifies “clientOid” on every trade execution.

import requests

def handler(pd: 'pipedream'):
  MAX_RETRIES = 5
  # 10 seconds
  DELAY = 500 * 10
  run = pd.context['run']
  print(pd.context)
  # pd.context.run.runs starts at 1 and increments when the step is rerun
  if run['runs'] == 1:
    # pd.flow.rerun(delay, context (discussed below), max retries)
    pd.flow.rerun(DELAY, None, MAX_RETRIES)
  
  elif run['runs'] == MAX_RETRIES + 1:
    raise Exception("Max retries exceeded")
  
  else:
    # Poll external API for status
    response = requests.get("https://api.kucoin.com/api/v1/orders/{{steps.trigger.context.id}}")
    # If we're done, continue with the rest of the workflow
    if response.json().status == "isActive":
      return response.json()

    # Else retry later
    pd.flow.rerun(DELAY, None, MAX_RETRIES)

Hello, just discovered your message here. Please take a moment to review the message I left.
Thank you.

It says your post was deleted, please advise.

It’s the same msg I left you on fiverr. I don’t think the current approach(as per the code you have shared) will work.