Uploading large files using twitter api

Hello

I’m trying to upload a video file that is larger than 10mb using twitter api.

signConfig={
  token: {
    key: auths.twitter.oauth_access_token,
    secret: auths.twitter.oauth_refresh_token
  },
  oauthSignerUri: auths.twitter.oauth_signer_uri
}

image=await require('@pipedreamhq/platform').axios(this, {
  method: 'GET',
  url: params.imageurl,
  responseType: 'arraybuffer'
})

media_id=(await require('@pipedreamhq/platform').axios(this, {
  method: 'POST',
  url: 'https://upload.twitter.com/1.1/media/upload.json',
  params: {
    command: 'INIT',
    total_bytes: image.length,
    media_type: 'video/mp4'
  }
}, signConfig)).media_id_string

for ([segment_index, media_data] of Buffer.from(image, 'binary').toString('base64').match(/.{1,15000}/g).entries()){
  await require('@pipedreamhq/platform').axios(this, {
    method: 'POST',
    url: 'https://upload.twitter.com/1.1/media/upload.json',
    params: {
      command: 'APPEND',
      media_id,
      media_data,
      segment_index
    }
  }, signConfig)
}

await require('@pipedreamhq/platform').axios(this, {
  method: 'POST',
  url: 'https://upload.twitter.com/1.1/media/upload.json',
  params: {
    command: 'FINALIZE',
    media_id
  }
}, signConfig)

I’m using this code to make append commands but since my video file is larger than 10mb and since twitter api limits segment_index value to be between 0-999 inclusive, I fail to upload the file when using this code.

I tried increasing the size of each chunks by changing this line
.match(/.{1,10000}/g)
to
.match(/.{1,15000}/g)
but then it gave me the following error


How should I fix this?

Also I want to upload the files using raw binary rather than base64 to decrease file sizes.
Is there a way to do this?

Thanks

@ehdus6202 Thanks for reaching out. I’d actually recommend reaching out on the Twitter developer forums to see how they’d solve this. If they can give you some generic advice (or point you to a Node.js library that handles this case), we should be able to help you implement on Pipedream.

Thanks for your response! I actually searched on google a bit and it seems like 431 error is coming from node js. The suggested workaround was using --max-http-header-size on the node command-line to accept larger headers but since I’m using pipedream to run node.js codes I’m not sure how I can do that. Do you have any suggestions?

node --max-http-header-size=20000 server.js

@ehdus6202 The 431 error is actually being returned from Twitter. The --max-http-header-size flag specifies the max header size Node HTTP servers are able to accept. It doesn’t specify the max size of headers send by Pipedream / the client.

I do think it might be best to reach out to the Twitter community to see how they’d suggest solving this, or what their preferred method is for uploading large files in Node.js. I’d recommend taking a look at packages like twitter-lite or other community-maintained npm packages that wrap the media upload logic.