This topic was automatically generated from Slack. You can find the original thread here.
Nat : I think I already know the answer to this, but will ask: has anyone used a more streamlined way to grab not just, for example, a New Record in Airtable, but also all it’s associated data living in linked tables – all in one go (like a GraphQL query allows for)? Or is the only option to do multiple API calls via multiple Pipedream workflow actions for each linked table to grab that linked data?
This:
• Trigger: new record added in Table X. Get Table X data plus all linked data from linked Tables Y and Z.
Instead of:
• Trigger: new record added in Table X. Get Table X data.
• Action: Get linked data in Table Y.
• Action: Get linked data in Table Z…
Thanks.
Dylan Sather (Pipedream) : based on your experience with the Airtable API, do you know if this is possible? Could we modify the event source to optionally retrieve linked records and emit those as a part of the event?
Pravin Savkar : Hi - thanks for raising this! i’m happy to help provide some options.
First, is there any more information you can share about your use case? Is it challenging to use the trigger with multiple actions? Or is this a pattern that you find yourself repeating across across multiple workflows?
In terms of options, are you comfortable writing Node.js code? One of the great things about Pipedream is that it’s fully programmable. You can directly modify the code for your source in the configuration tab (or you can develop locally and use the CLI). If you develop a source to do this, then you can also share it with the community via your own Github repo (we’ll have better ways to surface your own sources in the future too). You can check out our quickstart for source development as well as the component API reference.
If you’re not comfortable editing the code, you can also create a feature request in Github. If this is a common need we can definitely explore updating the core Airtable triggers.
Use case: syncing multi-table records in Airtable with a data warehouse (Snowflake).
Pain: To retrieve 1 multi-linked-table record in Airtable that is created or updated would require multiple API calls and multi Pipedream actions. And multiply that for every type of object to sync.
And wodering if anyone has dealt with this before on Pipedream and how they handled it.
Yes, comfortable with nodejs.
Yeah, maybe this is just a limitation with Airtable - and other - rest apis that you want to get more complicated info from… and would be up to us to figure out something like a GraphlQL API as a middleman…
As Pipedream is positioned as the easy-to-spool-up Data and Logic Middleman between different sources and destinations, it makes me wonder if there are opportunities for Pipedream to offer that GraphQL middleman role?: Lots of Sources Into Pipedream. Lots of sources out. Easy to manage via GraphQL.
Pravin Savkar : Hi Nat - thanks for the feedback - I’ll share it with the team.
I searched Google and ran a few tests with Airtable and I see your point. Airtable doesn’t appear to support joining data across tables when you query it.
Without reviewing your workflow it’s hard to offer specific advice, but here’s one option to consider:
If you find that you’re repeating the same enrichment pattern multiple times, you might try creating separate “enrichment workflows” using the HTTP API trigger and $respond(). That would allow you centralize and reuse the enrichment logic across multiple workflows (i.e., call them from any workflow and return the enriched data).
Do you think that would help? I’ll also send you a DM with some follow ups.
Hi, I had the same problem. I solved it in Airtable by creating a new fx column where its formula simply specifies the name of the column having the link to the other table. Then in pipedream step retrieve this field instead, and you’ll get the content from the linked table (and not just its id).