I’m tring to migrate a scala spark job from hadoop cluster to GCP, I have this snippest of code that read a file and create an ArrayBuffer[String] This code runs in the cluster and gives me 3025000 chars, I tried to run this code in dataproc: it gives 3175025 chars, I think there is whitespaces added to file contents or
Tag: google-cloud-platform
Issue with Spark Big Query Connector with Java
Getting Below issue with the Spark Big Query connector in Dataproc cluster with below configuraton. Image: 1.5.21-debian10 Spark Version: 2.4.7 Scala Version: 2.12.10 This is working fine from local but failing when I deploy this in dataproc cluster.Can someone suggest some pointers for this issue? pom.xml: Here is the sample Code: Answer Can you please replace the Spark BigQuery connector
Google Cloud Java SDK with Workload Identity?
Trying to figure out how to authenticate with the storage API from within a GKE cluster. Code: getApplicationDefault() is documented to use these means to authenticate with the API: Credentials file pointed to by the {@code GOOGLE_APPLICATION_CREDENTIALS} environment variable Credentials provided by the Google Cloud SDK {@code gcloud auth application-default login} command Google App Engine built-in credentials Google Cloud Shell
Insert data into BigQuery using native Insert query using Java
I Insert rows into BigQuery with the InsertAll method using JAVA. It is working always fine. But when we try to update the same row from the JAVA code am getting the below error, com.google.cloud.bigquery.BigQueryException UPDATE or DELETE DML statements over table project123:mydataset.test would affect rows in the streaming buffer, which is not supported So I tried from BigQueryConsole. I
How to read a file from maven resources folder in GCP Cloud Function?
My project uses Maven with the default folder structure and when my Google Cloud Function is trying to read a JSON file from the resources directory (src/main/resources), it fails with: File Not Found Exception Below is my code which is standard to read from the classpath resources folder. Any hints what could be wrong? Answer I use a FileReader with
Unable to collect data from metric query language MQL – GCP
I want to execute MQL (metric query language) using below library. Here is my code snippet. which will create monitoring client and will try to collect data from GCP monitoring. Above code is working fine but its not returning data of given startTime and endTime , It always returns latest datapoint available. is there any problem with my code ?
How to trigger Cloud Dataflow pipeline job from Cloud Function in Java?
I have a requirement to trigger the Cloud Dataflow pipeline from Cloud Functions. But the Cloud function must be written in Java. So the Trigger for Cloud Function is Google Cloud Storage’s Finalise/Create Event, i.e., when a file is uploaded in a GCS bucket, the Cloud Function must trigger the Cloud dataflow. When I create a dataflow pipeline (batch) and
Why is my java docker image created successfully on laptop but when uploading to google cloud run it fails?
dockerfile: In local build is successful using the command: But when trying to upload image to google cloud run using command: It gives error: What might be wrong here? Answer Couldn’t find an answer so posting the solution that I found on my own. If you can build docker images on local machine but google cloud run is failing for
Setting GCP environment variable in Spring boot with vs code
I’m trying to set up GOOGLE_APPLICATION_CREDENTIALS but keep getting this error java.io.IOException: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credential I have defined in my application properties: Added the dependency in the pom file: Tried to do
How to deploy Spring Boot application in Google Cloud Compute Engine?
I am a new to Google Cloud platform. I want to deploy my Spring Boot project in Compute Engine because deploying in App Engine costs more than the Compute Engine. There are a lot of videos/articles are available in YouTube/Websites for deploying in App Engine but I did not find any tutorial on deploying Spring Boot app in Compute Engine.