How to Resolve Pinecone Upsert Vector App Error in Pipedream Create_Embeddings App?

Help
1

This topic was automatically generated from Slack. You can find the original thread here.

Need Help :pray: **- Pinecone Upsert Vector Pipedream APP issue:

  1. :one:I am using the Pipedream Create_Embeddings app and using the direct export as the input for the Pinecone Upsert Vector App.
  2. :two:The data structure looks like this (edited down to small sample): [ { “object”: “embedding”, “index”: 0, “embedding”: [ -0.018727975, -0.023514321, -0.031027773, -0.03431143, -0.010769281, 0.030582532, -0.027883254, 0.00019577151, -0.052315887 ] } ]
  3. :three:The error I am getting: Error - Request failed with status code 400
    “vectors[0].values: invalid value Starting an object on a scalar field for type values”
    DETAILS
    at null.createError (/tmp/pdg/dist/code/2b47e20b605683bdd85cdf378fc6db2cc896977b3d7fb66566c1c685058f6511/node_modules/.pnpm/axios@0.21.4/node_modules/axios/lib/core/createError.js:16:15)
    at null.settle (/tmp/pdg/dist/code/2b47e20b605683bdd85cdf378fc6db2cc896977b3d7fb66566c1c685058f6511/node_modules/.pnpm/axios@0.21.4/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/tmp/pdg/dist/code/2b47e20b605683bdd85cdf378fc6db2cc896977b3d7fb66566c1c685058f6511/node_modules/.pnpm/axios@0.21.4/node_modules/axios/lib/adapters/http.js:269:11)
    at IncomingMessage.emit (events.js:412:35)
    at null.endReadableNT (internal/streams/readable.js:1333:12)
    at process.processTicksAndRejections (internal/process/task_queues.js:82:21)
    ExportsInputsLogsDetails

*Here is a short loom video to give full context:

2

, you’re right that you need to convert the result from Open AI into Pinecone Vector Values

As Pinecone Vector Values accept an array, you might want to merge all embedding array in your Open AI object. For this, you can ask Pi - the Pipedream bot for how to add the code for it

3

I have the $return_value from the action get_embedings as follow:

{
   "object": "list",
   "data": [
     {"object": "embedding", index: 0, embedding: [0, 1, 2, 3]}
    ]
}

How can I merge all the embedding array into one array using Node.js code

4

To merge all the embedding arrays into one array, you can use the following Node.js code: const embeddingsData = { "object": "list", "data": [ {"object": "embedding", index: 0, embedding: [0, 1, 2, 3]} ] }; const mergedEmbeddings = embeddingsData.data.flatMap(item => item.embedding); In this example, mergedEmbeddings will be a single array containing all the values from the embedding arrays.

5

I have been missing the proper verbiage to ask PI. I have been asking Pi to convert to an array or to a dictionary. I will try what you are suggesting. Thank you my friend!

6

rewrite this to include a prop field for the input data to be converted:

const embeddingsData = {
  "object": "list",
  "data": [
    {"object": "embedding", index: 0, embedding: [0, 1, 2, 3]}
  ]
};

const mergedEmbeddings = embeddingsData.data.flatMap(item => item.embedding);
7

To include a prop field for the input data to be converted, you can modify the Pipedream component like this: export default defineComponent({ props: { inputData: { type: "object", label: "Input Data", }, }, async run({ steps, $ }) { const data = this.inputData || steps.Embeddings_Conversion_code_not_working_.$return_value[0]; // Add your conversion logic here }, }); In this example, the inputData prop is added to the component, allowing you to pass the input data directly. If inputData is not provided, it will use the value from steps.Embeddings_Conversion_code_not_working_.$return_value[0].

8

To include a prop field for the input data to be converted, you can modify the code like this: export default defineComponent({ props: { embeddingsData: { type: "object", label: "Embeddings Data", default: { "object": "list", "data": [ {"object": "embedding", index: 0, embedding: [0, 1, 2, 3]} ] } }, }, async run({ steps, $ }) { const mergedEmbeddings = this.embeddingsData.data.flatMap(item => item.embedding); return mergedEmbeddings; }, }); In this example, the embeddingsData prop is added to the component, allowing you to pass the input data directly. If embeddingsData is not provided, it will use the default value specified in the default property.

9

when I used that code, it gave me this error: TypeError
Cannot read property ‘flatMap’ of undefined
DETAILS
at Object.run (file:///tmp/pdg/dist/code/b3e0cbf8e2fdad0c3b1d847cc721a8006ef97e75906580cbef0f8fcbe1e10fb0/component.mjs:15:55)
at global.executeComponent (/var/task/launch_worker.js:225:53)
at MessagePort.messageHandler (/var/task/launch_worker.js:722:28)

10

That code worked!! Now I have to correct the dimension size, as it output 153600 instead of the index 1536.

11

when I use the upsert_vector pipedream app, I am getting this error: Error - Request failed with status code 400
{“code”:3,“message”:“Vector dimension 153600 does not match the dimension of the index 1536”,“details”:[]}
DETAILS
at null.createError (/tmp/pdg/dist/code/2b47e20b605683bdd85cdf378fc6db2cc896977b3d7fb66566c1c685058f6511/node_modules/.pnpm/axios@0.21.4/node_modules/axios/lib/core/createError.js:16:15)
at null.settle (/tmp/pdg/dist/code/2b47e20b605683bdd85cdf378fc6db2cc896977b3d7fb66566c1c685058f6511/node_modules/.pnpm/axios@0.21.4/node_modules/axios/lib/core/settle.js:17:12)
at IncomingMessage.handleStreamEnd (/tmp/pdg/dist/code/2b47e20b605683bdd85cdf378fc6db2cc896977b3d7fb66566c1c685058f6511/node_modules/.pnpm/axios@0.21.4/node_modules/axios/lib/adapters/http.js:269:11)
at IncomingMessage.emit (events.js:412:35)
at null.endReadableNT (internal/streams/readable.js:1333:12)
at process.processTicksAndRejections (internal/process/task_queues.js:82:21)

12

I understand that you’re facing an error related to vector dimensions. It seems like the input vector has a dimension of 153600, while the expected dimension is 1536. To help you further, I would need more information about the code you’re using for the upsert_vector operation and how you’re processing the input data. Please provide the relevant code snippet or more details about your implementation, so I can assist you better.

13

rewrite the provided code to trim the index of vectors to a maximum of 1536, leaving the most recent vectors and removing the oldest (vectors at the beginning are oldest and vectors at the end of the vectors are the most recent:
export default defineComponent({
props: {
embeddingsData: {
type: “object”,
label: “Embeddings Data”,
default: {
“object”: “list”,
“data”: [
{“object”: “embedding”, index: 0, embedding: [0, 1, 2, 3]}
]
}
},
},
async run({ steps, $ }) {
const mergedEmbeddings = this.embeddingsData.data.flatMap(item => item.embedding);
return mergedEmbeddings;
},
});