This topic was automatically generated from Slack. You can find the original thread here.
Liran Galizyan : Trying to add new raw to Big Query using Python with google-cloud library but getting this error: No module named ‘google.cloud’
(I’ve added my code in a response to this thread)
Liran Galizyan : def handler(pd: “pipedream”):
from google.cloud import bigquery
import json
from datetime import datetime, timedelta
content = json.loads(pd.steps[“trigger”][“event”][“body”][“content”])[0][“body”][0][“results”]
campaign_id = content[0][“campaign”][“resourceName”].split(“/”, -1)[3]
leados_id = content[0][“campaign”][“resourceName”]
budget = content[0][“campaign”][“resourceName”]
bid_raw = content[0][“campaign”][“maximizeConversions”][“targetCpa”]
bid = bid_raw[:len(bid_raw) - 6]
account_id = content[0][“campaign”][“resourceName”].split(“/”, -1)[1]
current_date = datetime.now() + timedelta(days=-1)
yesterday_date = current_date.strftime(“%d/%m/%Y”)
start_end = ‘“”"’
body1 = “INSERT INTO lead-os.c_tmr_42069.bid_budget_test(campaign_id,leados_id,budget,bid) VALUES”
data = f”(campaign_id,leados_id,budget,bid)”
query_raw = start_end + body1 + data + start_end
print(query_raw)
GOOGLE_APPLICATION_CREDENTIALS={
“type”: “service_account”,
“project_id”: “lead-os”,
“private_key_id”: “b22adf2fe22f698f444887a5453d60aaf3176456",
“private_key”: “-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBge+eCVTH+pwUee3xIAIuexKUbhTl7FZ\n6Ii0oD7TFCDzFHaMVl/M5xg=\n-----END PRIVATE KEY-----\n”,
“client_email”: “lead-os-pipedream-service-acco@lead-os.iam.gserviceaccount.com”,
“client_id”: “105192752132652398886",
“auth_uri”: “https://accounts.google.com/o/oauth2/auth”,
“token_uri”: “https://oauth2.googleapis.com/token”,
“auth_provider_x509_cert_url”: “https://www.googleapis.com/oauth2/v1/certs”,
“client_x509_cert_url”: “https://www.googleapis.com/robot/v1/metadata/x509/lead-os-pipedream-service-acco%40lead-os.iam.gserviceaccount.com”
}
# Construct a BigQuery client object.
client = bigquery.Client()
query = query_raw
query_job = client.query(query) # Make an API request.
Dylan Pierce : Hi Liran, have you tried using the special comment import system for Python?
Dylan Pierce : Use PyPI packages with differing import names
Dylan Pierce : Also, we recommend placing your import statements at the top of the code step, just like a normal Python script
Liran Galizyan : Thanks, pierce!
Liran Galizyan : Now I get this:
Liran Galizyan : Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see S'authentifier auprès de services cloud à l'aide de bibliothèques clientes | Authentification | Google Cloud
DETAILS
No inputs.
Liran Galizyan : I am using the connected apps
Dylan Pierce : You will need to include your Google Application Credentials when you instantiate the client. I would advise placing them within an Environment Variable in your settings:
Dylan Pierce : Then you’ll be able to pass your Google credentials to the BigQuery client
Liran Galizyan : I also tried with environment variable, cant get it to work.
Liran Galizyan : This is the code
Liran Galizyan : This is the result
Liran Galizyan :
Dylan Pierce : I think the credentials need to be explicitly passed to the BigQuery constructor as configuration options.
Dylan Pierce : The documentation shows how you can pass specific configuration parameters to the constructor here:
https://googleapis.dev/python/bigquery/latest/usage/client.html
Liran Galizyan : Thanks Pierce, appriciate it. But still doesnt work, same results. Here’s the code:
Dylan Sather (Pipedream) : It looks like you’re passing the project ID, but not the credentials. You’ll see that Google requires that to also be passed in the Client constructor, here: google.cloud.bigquery.client.Client — google-cloud-bigquery documentation
If you’re still not able to get it to work, this would be a great question for the Google team. I believe they do support for Google APIs on Stack Overflow. I’d recommend reaching out there, and they should be able to tell you exactly how to pass the credentials!