Out of memory - How can I tell how much memory my scenario requires?

Hi, I have a scenario of updating records and deleting records in airtable. that works on a daily basis, for a chatbot I built on airtable. (so as not to exceed the limit of 50,000 records in the table)

The scenario works, but I get the error “The default is out of memory for this workflow”.

I understand that I need to increase the default memory,

I can limit the number of records to update each day, but I don’t know how much memory each operation requires.

In testing the scenario I ran I got an error about updating 144 records, and the memory setting was 256MB.

Is there a way to track how much memory the scenario requires?

Is there a way to reduce the amount of memory the scenario requires?

I would appreciate any help

Hi @rrbbtalnorth,

I think this is more like the engineering problem than the Pipedream problem. Your memory usage increasing per your records means at some point you loads all the records in your workflow (which stores those records in memory).

Is there a way to track how much memory the scenario requires?

This is entirely depends on your workflow logic, so you can try to run with 1000, 10000, or 50000 records to measure how your workflow memory increased

Is there a way to reduce the amount of memory the scenario requires?

I think there’re 2 options to solve this:

  1. Increase your workflow memory limit to the maximum 10GB, it is quick, but it will cost you a lot more if you have many records. Refer to this doc to understand how does workflow memory affect your billing
  2. If Airtable API supports Batch update operation, you can change your code to use that, or you might need to migrate your Airtable to a platform that support Batch update operation (a SQL database on Supabase for example). This requires more development effort and time, but it will scale.

Could you direct me what could be the focus of the problem in the code?

module.exports = defineComponent({
props: {
airtable: {
type: “app”,
app: “airtable”,
},
baseId: {
type: “$.airtable.baseId”,
appProp: “airtable”,
},
tableId: {
type: “$.airtable.tableId”,
baseIdProp: “baseId”,
},
records_Update: {
type: “object”,
label: “records to update”,
},
},
async run({steps, $}) {
const Airtable = require(‘airtable’);
const base = new Airtable({apiKey: this.airtable.$auth.api_key}).base(this.baseId);

let data = []

for (let i = 0; i < this.records_Update.length; i++) {
  let rec = this.records_Update[i];
  console.log(rec);

  data.push({
    id: rec.id,
    fields: {
      BackupMessageGroup: rec.fields.MessageGroup,
      BackupTriggersGroup: rec.fields.TriggersGroup,
      BackupCategoryGroup: rec.fields.CategoryGroup,
      BackupMainCategories: rec.fields.MainCategories,
      BackupSecondaryCategories: rec.fields.SecondaryCategories,
      BackupAction: rec.fields.Action,
      BackupProfileCounterGroup: rec.fields.ProfileCounterGroup,
      DateUpdate: rec.fields.DateNOW,
    }
  })

  if (i > 0 && ((i + 1) % 10 === 0 || i === this.records_Update.length - 1)) {
    await base(this.tableId).update(data);
    data = []
  }
}

},
})

@rrbbtalnorth,

I think your code is not wrong and it will works well when you have a small amount of records. Because it loads all the records into the memory (and also pass then between steps).

The problem your code will face is it will use memory increasingly as your records grow. So I would suggest you not to loading all of the records into your workflow, but use the batch update operation instead.

An example for batch update operation in SQL is something like this:

UPDATE Messages SET BackupMessageGroup = 'Some value' WHERE createdAt > '2023-02-08'

Using Batch update operation, instead of loading all records into your workflow, you send an update command to the platform for it to update the records for you

1 Like

Ok thank you.
I will try to change to “batch update operation”.

But I would still like to know if there is a way to track the amount of memory my scenario requires?
I saw the documentation you sent me, but it doesn’t give an answer.

@rrbbtalnorth

But I would still like to know if there is a way to track the amount of memory my scenario requires?

This is entirely depends on your workflow logic, so you can try to run with 1000, 10000, or 50000 records with increasing memory limits. In all case, your workflow memory usage will be converted to invocations, so you can access Billing & Usage to track how much invocations your workflow cost

Is it possible to change the memory limit to automatic according to the need that my scenario requires?

I see that I only have the option to manually select the amount of memory I set for the scenario.

If there was an option to let pipedream choose for me the amount of memory my scenario needs, that would be great.

Unfortunately this is not possible for now due to the nature of Pipedream running on AWS Lambda. I think what we can do for now is to manually increase it to the number that enough to handle all the records in your Airtable

Fine,

But I still didn’t understand but can I know how much memory is needed for example to update 100 records?

Just check manually?
Run an update of 100 records in 256MB and see if there is an error?

Isn’t there a more elegant way to check this?

Yes this is the method I’m thinking right now. Maybe @pierce will have more insights and advice another way :pray:

1 Like

Okay thank you

Well, I see there is a problem here.

Even in the test I did to update only 30 records, it sends me the error - “You exceeded the default memory for this workflow”

The scenario works fine, only the problem is with the errors that pipedream writes to me

The pipedream page just hangs on the same run. Until I refresh the page

Hello @rrbbtalnorth,

Could you share the step that fetch the data from Airtable? May I confirm that you have added the limit to fetch only 30 records from airtable?

Hi @rrbbtalnorth

Memory usage is highly dependent on your code, code dependencies, and incoming data from 3rd party sources.

For those reasons, we cannot provide specific guidelines on each of those parameters, because they’re specific to your use.

In general we recommend using the default memory allotment where you can (256 megabytes), and if you start to exceed this then consider:

  1. Increase the memory size to your workflow.
  2. Split your current workflow into different workflows using subscriptions or HTTP webhooks between them.
  3. Make sure that you don’t have a memory leak or have inefficient memory usage in your code.
1 Like

Yes,

I’m only looking for 30 records,
This is the search set here

Well, I sent the script to the programmer,

And really the problem was not in the existing code, but in the built-in record search, which brought me all the information I have in the table, which is very burdensome,

That’s why the programmer wrote me a code that searches for records and does not upload irrelevant data.

1 Like