It still generates the same error. How can I ensure that the pdfjs-dist package is available in my Pipedream environment.
I apologize for the confusion. Currently, Pipedream does not support importing the pdfjs-dist package directly. However, you can use alternative methods to process PDF files, such as using an external API or another supported package. If you could provide more details about your use case and what you’re trying to achieve, I’d be happy to help you find an alternative solution.
I am trying to load the text content from a PDF file in /tmp. chunk it into 1000 character blocks with 200 character overlap suitable to pass to the next Step open_create_embeddings
Since Pipedream doesn’t support the pdfjs-dist package, you can use an alternative package like pdf-parse to extract text from a PDF file. Here’s a code snippet that reads a PDF file from the /tmp directory, extracts the text, and chunks it into 1000 character blocks with a 200 character overlap: import fs from "fs"; import pdfParse from "pdf-parse"; export default defineComponent({ props: { pdfFilePath: { type: "string", label: "PDF File Path", default: "/tmp/your_pdf_file.pdf", }, }, async run({ $ }) { const dataBuffer = fs.readFileSync(this.pdfFilePath); const pdfData = await pdfParse(dataBuffer); const chunkSize = 1000; const overlap = 200; const text = pdfData.text; const chunks = []; for (let i = 0; i < text.length; i += chunkSize - overlap) { const chunk = text.slice(i, i + chunkSize); chunks.push(chunk); } return chunks; }, }); Replace your_pdf_file.pdf with the name of your PDF file in the /tmp directory. This code will return an array of text chunks that you can pass to the next step open_create_embeddings.
now we are back to the original issue of the phantom file directory Code Error
ENOENT: no such file or directory, open ‘./test/data/05-versions-space.pdf’ which does not appear anywhere in the code:
import { axios } from “@pipedream/platform”;
import fs from “fs”;
import pdfParse from “pdf-parse”;
//import mammoth from “mammoth”;
export default defineComponent({
props: {
filePath: {
type: “string”,
label: “File Path”,
default: steps.download_file_to_tmp.$return_value.tmpPath,
},
},
async run({ steps, $ }) {
const extension = filePath.split(‘.’).pop();
let text;
**switch**(extension) {
**case** 'pdf':
**const** dataBuffer = fs.readFileSync(**this**.pdfFilePath);
**const** pdfData = **await** pdfParse(dataBuffer);
text = pdfData.text;
**break**;
// case ‘doc’:
// case ‘docx’:
// const wordData = await axios($, { url: filePath, responseType: “arraybuffer” });
// const wordBuffer = Buffer.from(wordData.data);
// text = (await mammoth.extractRawText({ buffer: wordBuffer })).value;
// break;
case ‘txt’:
const textData = await axios($, { url: filePath });
text = textData.data;
break;
default:
throw new Error(Unsupported file type: ${extension});
}
**const** textSplitter = (text, maxLength, overlap) => {
**const** chunks = [];
**for** (**let** i = 0; i < text.length; i += maxLength - overlap) {
chunks.push(text.slice(i, i + maxLength));
}
**return** chunks;
};
**const** chunks = textSplitter(text, 1000, 200);
**return** chunks;
},
});
I apologize for the confusion. It seems there’s a small mistake in the code. You should use this.filePath instead of this.pdfFilePath when reading the file. Here’s the corrected code: import { axios } from "@pipedream/platform"; import fs from "fs"; import pdfParse from "pdf-parse"; export default defineComponent({ props: { filePath: { type: "string", label: "File Path", default: steps.download_file_to_tmp.$return_value.tmpPath, }, }, async run({ steps, $ }) { const extension = this.filePath.split('.').pop(); let text; switch (extension) { case 'pdf': const dataBuffer = fs.readFileSync(this.filePath); const pdfData = await pdfParse(dataBuffer); text = pdfData.text; break; case 'txt': const textData = await axios($, { url: this.filePath }); text = textData.data; break; default: throw new Error(`Unsupported file type: ${extension}`); } const textSplitter = (text, maxLength, overlap) => { const chunks = []; for (let i = 0; i < text.length; i += maxLength - overlap) { chunks.push(text.slice(i, i + maxLength)); } return chunks; }; const chunks = textSplitter(text, 1000, 200); return chunks; }, }); Make sure to replace steps.download_file_to_tmp.$return_value.tmpPath with the actual path to your PDF file in the /tmp directory if you haven’t done that already.
updated the code to harcode the filePath, still the same error:
Code Error
ENOENT: no such file or directory, open ‘./test/data/05-versions-space.pdf’
DETAILS
Error: ENOENT: no such file or directory, open ‘./test/data/05-versions-space.pdf’
at Object.openSync (node:fs:603:3)
at Object.readFileSync (node:fs:471:35)
at Object. (/tmp/pdg/dist/code/f11c96b51496116a6f364b70f2bdd0b36daa6a584470b8e564c265349ede5424/node_modules/.pnpm/pdf-parse@1.1.1/node_modules/pdf-parse/index.js:15:25)
at Module._compile (node:internal/modules/cjs/loader:1256:14)
at Module._extensions…js (node:internal/modules/cjs/loader:1310:10)
at Module.load (node:internal/modules/cjs/loader:1119:32)
at Module._load (node:internal/modules/cjs/loader:960:12)
at ModuleWrap. (node:internal/modules/esm/translators:169:29)
at ModuleJob.run (node:internal/modules/esm/module_job:194:25)
Updated code:
import { axios } from “@pipedream/platform”;
import fs from “fs”;
import pdfParse from “pdf-parse”;
//import mammoth from “mammoth”;
export default defineComponent({
props: {
filePath: {
type: “string”,
label: “File Path”,
default: “/tmp/C7335 Supplemental Response Detail - unitiFM_v4-9t.pdf”,
},
},
async run({ steps, $ }) {
const extension = filePath.split(‘.’).pop();
let text;
**switch**(extension) {
**case** 'pdf':
**const** dataBuffer = fs.readFileSync(**this**.FilePath);
**const** pdfData = **await** pdfParse(dataBuffer);
text = pdfData.text;
**break**;
// case ‘doc’:
// case ‘docx’:
// const wordData = await axios($, { url: filePath, responseType: “arraybuffer” });
// const wordBuffer = Buffer.from(wordData.data);
// text = (await mammoth.extractRawText({ buffer: wordBuffer })).value;
// break;
case ‘txt’:
const textData = await axios($, { url: filePath });
text = textData.data;
break;
default:
throw new Error(Unsupported file type: ${extension});
}
**const** textSplitter = (text, maxLength, overlap) => {
**const** chunks = [];
**for** (**let** i = 0; i < text.length; i += maxLength - overlap) {
chunks.push(text.slice(i, i + maxLength));
}
**return** chunks;
};
**const** chunks = textSplitter(text, 1000, 200);
**return** chunks;
},
});
It looks like the issue is related to the file path. In Pipedream, you should read files from the /tmp directory. Update the filePath default value to the correct path in the /tmp directory. Also, there’s a typo in the fs.readFileSync line, it should be this.filePath instead of this.FilePath. Here’s the updated code: import { axios } from "@pipedream/platform"; import fs from "fs"; import pdfParse from "pdf-parse"; export default defineComponent({ props: { filePath: { type: "string", label: "File Path", default: "/tmp/C7335_Supplemental_Response_Detail_unitiFM_v4-9t.pdf", }, }, async run({ steps, $ }) { const extension = this.filePath.split('.').pop(); let text; switch (extension) { case 'pdf': const dataBuffer = fs.readFileSync(this.filePath); const pdfData = await pdfParse(dataBuffer); text = pdfData.text; break; case 'txt': const textData = await axios($, { url: this.filePath }); text = textData.data; break; default: throw new Error(`Unsupported file type: ${extension}`); } const textSplitter = (text, maxLength, overlap) => { const chunks = []; for (let i = 0; i < text.length; i += maxLength - overlap) { chunks.push(text.slice(i, i + maxLength)); } return chunks; }; const chunks = textSplitter(text, 1000, 200); return chunks; }, }); Make sure the file you’re trying to read is located in the /tmp directory with the correct name.
updated the code, still the same error
I apologize for the inconvenience. To help you further, I need more information about the error you’re encountering. Can you please provide the error message and any additional details that might help me understand the issue better?