Uploading large files using twitter api

Help
,
1

Hello

I’m trying to upload a video file that is larger than 10mb using twitter api.

signConfig={
  token: {
    key: auths.twitter.oauth_access_token,
    secret: auths.twitter.oauth_refresh_token
  },
  oauthSignerUri: auths.twitter.oauth_signer_uri
}

image=await require('@pipedreamhq/platform').axios(this, {
  method: 'GET',
  url: params.imageurl,
  responseType: 'arraybuffer'
})

media_id=(await require('@pipedreamhq/platform').axios(this, {
  method: 'POST',
  url: 'https://upload.twitter.com/1.1/media/upload.json',
  params: {
    command: 'INIT',
    total_bytes: image.length,
    media_type: 'video/mp4'
  }
}, signConfig)).media_id_string

for ([segment_index, media_data] of Buffer.from(image, 'binary').toString('base64').match(/.{1,15000}/g).entries()){
  await require('@pipedreamhq/platform').axios(this, {
    method: 'POST',
    url: 'https://upload.twitter.com/1.1/media/upload.json',
    params: {
      command: 'APPEND',
      media_id,
      media_data,
      segment_index
    }
  }, signConfig)
}

await require('@pipedreamhq/platform').axios(this, {
  method: 'POST',
  url: 'https://upload.twitter.com/1.1/media/upload.json',
  params: {
    command: 'FINALIZE',
    media_id
  }
}, signConfig)

I’m using this code to make append commands but since my video file is larger than 10mb and since twitter api limits segment_index value to be between 0-999 inclusive, I fail to upload the file when using this code.

I tried increasing the size of each chunks by changing this line
.match(/.{1,10000}/g)
to
.match(/.{1,15000}/g)
but then it gave me the following error

image
image1752×387 36.9 KB

How should I fix this?

Also I want to upload the files using raw binary rather than base64 to decrease file sizes.
Is there a way to do this?

Thanks

2

@anon72461701 Thanks for reaching out. I’d actually recommend reaching out on the Twitter developer forums to see how they’d solve this. If they can give you some generic advice (or point you to a Node.js library that handles this case), we should be able to help you implement on Pipedream.

4

@anon72461701 The 431 error is actually being returned from Twitter. The --max-http-header-size flag specifies the max header size Node HTTP servers are able to accept. It doesn’t specify the max size of headers send by Pipedream / the client.

I do think it might be best to reach out to the Twitter community to see how they’d suggest solving this, or what their preferred method is for uploading large files in Node.js. I’d recommend taking a look at packages like twitter-lite or other community-maintained npm packages that wrap the media upload logic.