Stream JSON to Snowflake
•@dylburger•
•code:
•data:private•last updated:3 years ago
today
Build integrations remarkably fast!
You're viewing a public workflow template.
Sign up to customize, add steps, modify code and more.
Join 800,000+ developers using the Pipedream platform
steps.
trigger
HTTP API
Deploy to generate unique URL
This workflow runs on Pipedream's servers and is triggered by HTTP / Webhook requests.
steps.
format_payload
auth
to use OAuth tokens and API keys in code via the
auths objectcode
Write any Node.jscodeand use anynpm package. You can alsoexport datafor use in later steps via
return or this.key = 'value', pass input data to your code viaparams, and maintain state across executions with$checkpoint.async (event, steps) => {1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
}20
// We want to extract browser, OS data from the User-Agent header, so we use the user-agent module.
// Just require() npm modules that you want to use here, no need to npm install
const useragent = require("useragent")
// Our payload starts as a copy of event.body, the HTTP payload in the request
this.payload = event.body
if (event.headers && 'user-agent' in event.headers) {
this.payload.ua = useragent.parse(event.headers['user-agent'])
}
// Then, we decorate our incoming HTTP request with the Pipedream event ID
// and the timestamp we processed it at, both accessible in the global variable $context
this.payload.ts = steps.trigger.context.ts
this.payload.pd_event_id = steps.trigger.context.id
// There's a lot more you can do with Node.js code cells. Check out the docs for more info!
// https://docs.pipedream.com/workflows/steps/code/steps.
send_json_to_snowflake
Send JSON to Snowflake (each JSON object gets written as a new record) using the Pipedream Snowflake integration
auth
(auths.snowflake)
params
Payload
A JavaScript object representing the payload to send to Snowflake
string ·params.payload
code
async (params, auths) => {1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
}25
const {
user,
private_key,
database,
schema,
stage_name,
pipe_name,
account,
host,
} = auths.snowflake
// See https://docs.pipedream.com/destinations/snowflake/
$send.snowflake({
user,
private_key,
database,
schema,
stage_name,
pipe_name,
account,
host: `${account}.snowflakecomputing.com`,
payload: params.payload
})