This topic was automatically generated from Slack. You can find the original thread here.
when workflow is triggered, getting this error:
Out of Memory
You’ve exceeded the default memory for this workflow. You can raise the memory in your workflow’s Settings.
It seems like your workflow is running out of the allocated memory, which is causing the “Out of Memory” error. To resolve this, you can increase the memory limit for your workflow in the workflow settings. Here’s how you can do that: 1. Go to your workflow’s page on Pipedream. 2. Click on the Settings tab for the workflow. 3. Look for the Memory section. 4. Adjust the memory limit slider to increase the amount of memory available to your workflow. Please note that increasing the memory limit may affect the number of credits you’re charged per execution. For more information on how workflow memory affects credits, you can visit Pipedream’s pricing page. If you continue to experience issues or need further assistance, please visit Pipedream’s support page.
how will I get to know how much memory workflow has
By default, workflows on Pipedream run with 256MB of memory. You can check and modify a workflow’s memory configuration in your workflow’s settings. To adjust the memory for your workflow, follow these steps: 1. Open your workflow in the Pipedream UI. 2. Click on the Settings tab at the top of the workflow page. 3. Look for the Memory setting to view the current memory allocation. 4. To change the memory setting, select a new value from the dropdown menu and save your changes. Remember that increasing your workflow’s memory gives you a proportional increase in CPU, which can make your workflow more performant. However, Pipedream charges credits proportional to your memory configuration. For more details on how memory affects credits, please visit Pipedream’s pricing documentation.
this code I’ve written for Cron job, but it’s giving Timeout error. Can you help whether there is something wrong in my code.
import { axios } from "@pipedream/platform";
export default defineComponent({
props: {
data: { type: "data_store" },
},
async run({ steps, $ }) {
// Code to read your Excel file and extract data goes here
// For example, you might use an API to read the Excel file from a service like Google Sheets
// You would process rows from `lastRowIndex` to `lastRowIndex + numberOfRowsToProcess`
const data = steps.get_values.$return_value;
const data1 = steps.get_values_1.$return_value;
// Retrieve the index of the last processed row, or start at 0 if it's the first run
let lastRowIndex = (await this.data.get("lastRowIndex")) || 0;
const numberOfRowsToProcess = 15;
const dataObject = {
data: [],
};
const fetchImagesData = async () => {
try {
const response = await fetch("https://mm-strapi-v2.azurewebsites.net/api/upload/files/");
let data = await response.json();
return data;
} catch (error) {
console.error(`Error while fetching the data`);
}
};
const dataJsonNew = await fetchImagesData();
function fetchIdByName(dataJsonNew, name) {
for (let i = 0; i < dataJsonNew.length; i++) {
if (dataJsonNew[i].name == name) {
return dataJsonNew[i].id;
}
}
return null; // Return null if no match is found
}
function convertDataToJSON(data) {
const headers = data[0];
const convertedData = [];
for (let i = 1; i < data.length; i++) {
const row = data[i];
const rowData = {};
for (let j = 0; j < headers.length; j++) {
rowData[headers[j]] = row[j];
}
convertedData.push(rowData);
}
return convertedData;
}
const categoryData = convertDataToJSON(data);
const appsData = convertDataToJSON(data1);
function convertToTitleCase(text) {
const words = text.split("-");
const titleCaseText = words
.map((word) => {
const firstLetter = word.charAt(0).toUpperCase();
const restOfWord = word.slice(1).toLowerCase();
return firstLetter + restOfWord;
})
.join(" ");
return titleCaseText;
}
// console.log(categoryData.slice(lastRowIndex, lastRowIndex + numberOfRowsToProcess))
const combinedData = categoryData.slice(lastRowIndex, lastRowIndex + numberOfRowsToProcess).map((item, index) => {
const Keyword = item?.Slug ?? "";
const seoTitle = item["SEO Title"];
const seoDescription = item["SEO Description"];
const KW = convertToTitleCase(Keyword);
const FAQ1Question = "What is " + KW + "?";
const FAQ1Answer = item["What is <KW>?"] ?? null;
const FAQ2Question = "How " + KW + " work or function?";
const FAQ2Answer = item["How <KW> work or function?"] ?? null;
const FAQ3Question = "Benefits of " + KW + "?";
const FAQ3Answer = item["Benefits of <KW> "] ?? null;
const dynamicVariables = {};
for (let i = 1; i <= 5; i++) {
dynamicVariables[`FAQS${i}Question`] = item[`FAQ${i}`] ?? null;
dynamicVariables[`FAQS${i}Answer`] = item[`FAQA${i}`] ?? null;
}
const content = `## ${FAQ1Question}\n${FAQ1Answer}\n### ${FAQ2Question}\n${FAQ2Answer}\n### ${FAQ3Question}\n${FAQ3Answer}`;
const FAQs = [
{
question: dynamicVariables.FAQS1Question,
answer: dynamicVariables.FAQS1Answer,
},
{
question: dynamicVariables.FAQS2Question,
answer: dynamicVariables.FAQS2Answer,
},
{
question: dynamicVariables.FAQS3Question,
answer: dynamicVariables.FAQS3Answer,
},
{
question: dynamicVariables.FAQS4Question,
answer: dynamicVariables.FAQS4Answer,
},
{
question: dynamicVariables.FAQS5Question,
answer: dynamicVariables.FAQS5Answer,
},
];
const filteredRows = appsData.filter((row) => row.Slug === Keyword);
const numbers = filteredRows.map((row) => row["Logo URL "]);
const product = filteredRows.map((row) => row["Product - App Name"]);
const websiteLink = filteredRows.map((row) => row["Website URL"]);
const shopify = filteredRows.map((row) => row["Shopify URL"]);
const productDescription = filteredRows.map(
(row) => row["Product Description "]
);
const pros = filteredRows.map((row) => row["Pros "]);
const cons = filteredRows.map((row) => row["Cons"]);
const ratingvalue = filteredRows.map((row) => row["Final Rating (Text)"]);
const developer = filteredRows.map((row) => row["Developers"]);
const pricingData = filteredRows.map((row) => row["Pricing plan"]);
//console.log('ratingvalue :>> ', ratingvalue);
const pricingArray = [];
for (const pricingInfo of pricingData) {
const planCostArray = pricingInfo.split("\n");
const planDetails = planCostArray.map((planCost) => {
const [plan, cost] = planCost.split(":");
return {
planName: plan.trim(),
planPrice: cost !== " " ? Number(cost?.trim()) : null,
};
});
pricingArray.push(planDetails); //array of arrays
}
//console.log("pricingArray :>> ", pricingArray);
let count = numbers.length;
const productArray = numbers.map((number, i) => {
count++;
const flattenedPricingArray = pricingArray[i].map((plan) => ({
planName: plan.planName,
planPrice:
isNaN(plan.planPrice) || plan.planPrice === null
? null
: Number(plan.planPrice),
})); //array of objects
const name = [];
const extension = [];
if (numbers[i] && typeof numbers[i] === "string") {
name[i] = numbers[i]
.split("/")
.pop()
.split(".")
.slice(0, -1)
.join(".");
extension[i] = numbers[i].split("/").pop().split(".").slice(-1)[0];
}
let imageVariable = fetchIdByName(
dataJsonNew,
name[i] + `_${count}.` + extension[i]
);
// console.log(
// imageVariable,
// Keyword,
// name[i] + `_${count}.` + extension[i]
// );
return {
productName: product[i] ?? null,
websiteLink: websiteLink[i] ?? null,
shopifyUrl: shopify[i] ?? null,
productDescription: productDescription[i] ?? null,
pros: pros[i] ?? null,
cons: cons[i] ?? null,
rating: {
totalSum: ratingvalue[i] ?? 0,
totalCount: ratingvalue[i] ? 1 : 0,
},
productPricing: flattenedPricingArray ?? null,
logo: imageVariable ? [imageVariable] : null,
developer: developer[i].toString() ?? null,
};
});
const rowObject = {
slug: Keyword,
products: productArray,
faqs: { faqs: FAQs },
seo: { title: seoTitle, description: seoDescription },
content: content,
};
return rowObject;
});
//console.log(combinedData)
// After processing, update the index of the last processed row
lastRowIndex += numberOfRowsToProcess;
await this.data.set("lastRowIndex", lastRowIndex);
// const apiBody = {
// data: {
// structuredData: combinedData,
// },
// };
dataObject.data.push(...combinedData);
const strapiEndpoint = "https://mm-strapi-v2.azurewebsites.net/api/app-stashes"; // Replace with your Strapi endpoint
const strapiToken = "1ee97ab83b4effa6432b2ab40214bc60b027acf8df5e3eb87f27b458a339634eced73f19b86900905b05ea599a618f170ea6c2689de35ab5c6f2699d39dbfd59e0fb3991ecc7ab366837a19d64250c4d79177f4d10a92e7102cdc79cde17e520b4c68c8d1c469542ff1ba3252a19716c26ea60a20093b77a51dbafbf7d2593af"; // Replace with your Strapi API token
for (const data of dataObject.data) {
//console.log(data)
try {
const response = await axios($, {
method: "POST",
url: strapiEndpoint,
headers: {
Authorization: `Bearer ${strapiToken}`,
"Content-Type": "application/json",
},
data: {
data: data || [], // Wrap your dataObject inside a "data" property
},
});
} catch (error) {
console.error("Error while making the API request:", error);
console.error("Error details:", JSON.stringify(error, null, 2));
}
}
}
// Return some result or export data if needed
});
Hi , you can increase your workflow memory and timeout in your Workflow settings. In this case I would recommend you to increase the timeout
what I recommend is to try to port your code to your local machine and develop it there. Develop the code on local will make sure that you won’t faced platform error.
After the code is working on your local, you can port it back to Pipedream action.