Google Cloud Debugger can’t find Spring Boot web app deployed as Cloud Run service

I followed the guide (https://cloud.google.com/debugger/docs/setup/java#cloud-run) to setup the cloud debugger for my cloud run services. Everything looks like it should work. However the debugger UI tells me it can not find any application.

However I did everything that was suugested in the documentation

Source code is in Source Repositories
Cloud Build triggers on master push
I download the cdbg-java-agent.so
I run the java application with the -agentpath option
Cloud Run service is being deployed via cloudbuild.yaml
Cloud Run service starts healthy and works

However as you can see in the screenshot my application can not be found after successful deployment.

Here are some details of my configuration:

Spring Boot v2.4.2
Kotlin 1.4.21

This is my Dockerfile

FROM gradle:6.7.0-jdk14 as build
USER root
WORKDIR /dist
ADD . /dist
RUN gradle assemble

FROM alpine as agent
USER root
WORKDIR /agent
RUN wget -qO- https://storage.googleapis.com/cloud-debugger/compute-java/debian-wheezy/cdbg_java_agent_gce.tar.gz | tar xvz

FROM openjdk:14-slim
WORKDIR /app
COPY --from=agent /agent/cdbg_java_agent.so agent.so
COPY --from=build /dist/build/libs/user-service.jar app.jar
EXPOSE 8080
CMD ["java", "-agentpath:/app/agent.so", "-Dcom.google.cdbg.breakpoints.enable_canary=false", "-jar", "app.jar"]

JavaScript
​x
 
FROM gradle:6.7.0-jdk14 as buildUSER rootWORKDIR /distADD . /distRUN gradle assemble​FROM alpine as agentUSER rootWORKDIR /agentRUN wget -qO- https://storage.googleapis.com/cloud-debugger/compute-java/debian-wheezy/cdbg_java_agent_gce.tar.gz | tar xvz​FROM openjdk:14-slimWORKDIR /appCOPY --from=agent /agent/cdbg_java_agent.so agent.soCOPY --from=build /dist/build/libs/user-service.jar app.jarEXPOSE 8080CMD ["java", "-agentpath:/app/agent.so", "-Dcom.google.cdbg.breakpoints.enable_canary=false", "-jar", "app.jar"]​

Here is the docker build output

Deploying '<unknown> Dockerfile: Dockerfile'...
Building image...
Preparing build context archive...
[==================================================>]1855/1855 files
Done

Sending build context to Docker daemon...
[==================================================>] 76.19MB
Done

Step 1/15 : FROM gradle:6.7.0-jdk14 as build
 ---> 5af4d25725b2
Step 2/15 : USER root
 ---> Using cache
 ---> c97e9145e3e1
Step 3/15 : WORKDIR /dist
 ---> Using cache
 ---> 468fff36a2e7
Step 4/15 : ADD . /dist
 ---> c435bd2ffde9
Step 5/15 : RUN gradle assemble
 ---> Running in 70f5c69b7dde

Welcome to Gradle 6.7!

Here are the highlights of this release:
 - File system watching is ready for production use
 - Declare the version of Java your build requires
 - Java 15 support

For more details see https://docs.gradle.org/6.7/release-notes.html

Starting a Gradle Daemon (subsequent builds will be faster)
> Task :compileKotlin
> Task :compileJava NO-SOURCE
> Task :processResources
> Task :classes
> Task :bootJarMainClassName
> Task :bootJar
> Task :inspectClassesForKotlinIC
> Task :jar SKIPPED
> Task :assemble

Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.7/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 1m 48s
5 actionable tasks: 5 executed
Removing intermediate container 70f5c69b7dde
 ---> 5bbd9485f873

Step 6/15 : FROM alpine as agent
 ---> 28f6e2705743
Step 7/15 : USER root
 ---> Using cache
 ---> 4a0572e8ff4a
Step 8/15 : WORKDIR /agent
 ---> Using cache
 ---> 0374735af05d
Step 9/15 : RUN wget -qO- https://storage.googleapis.com/cloud-debugger/compute-java/debian-wheezy/cdbg_java_agent_gce.tar.gz | tar xvz
 ---> Using cache
 ---> ef141a7d5acc

Step 10/15 : FROM openjdk:14-slim
 ---> 38fca30874ab
Step 11/15 : WORKDIR /app
 ---> Using cache
 ---> 4dd576d38d65
Step 12/15 : COPY --from=agent /agent/cdbg_java_agent.so agent.so
 ---> Using cache
 ---> ff5fc93d2d5e
Step 13/15 : COPY --from=build /dist/build/libs/user-service.jar app.jar
 ---> 4b009124345f
Step 14/15 : EXPOSE 8080
 ---> Running in 99f71fb86534
Removing intermediate container 99f71fb86534
 ---> 38fcd9d00999
Step 15/15 : CMD ["java", "-agentpath:/app/agent.so", "-Dcom.google.cdbg.breakpoints.enable_canary=false", "-jar", "app.jar"]
 ---> Running in 309f35aed48f
Removing intermediate container 309f35aed48f
 ---> 43757d33b2ee

Successfully built 43757d33b2ee
Existing container found: 94d3fab4505461e8e694b61909ba1f7d827d8f7078b4dd8d1d36d4c4e0a0ac08, removing...
Creating container...
Container Id: b99675b5f6c00a0e5bec76999db04abf92eab29d257ee887bcea123a02789850
Container name: '/quirky_stonebraker'
Attaching to container '/quirky_stonebraker'...
Starting container '/quirky_stonebraker'
'<unknown> Dockerfile: Dockerfile' has been deployed successfully.

JavaScript
 
Deploying '<unknown> Dockerfile: Dockerfile'...Building image...Preparing build context archive...[==================================================>]1855/1855 filesDone​Sending build context to Docker daemon...[==================================================>] 76.19MBDone​Step 1/15 : FROM gradle:6.7.0-jdk14 as build ---> 5af4d25725b2Step 2/15 : USER root ---> Using cache ---> c97e9145e3e1Step 3/15 : WORKDIR /dist ---> Using cache ---> 468fff36a2e7Step 4/15 : ADD . /dist ---> c435bd2ffde9Step 5/15 : RUN gradle assemble ---> Running in 70f5c69b7dde​Welcome to Gradle 6.7!​Here are the highlights of this release: - File system watching is ready for production use - Declare the version of Java your build requires - Java 15 support​For more details see https://docs.gradle.org/6.7/release-notes.html​Starting a Gradle Daemon (subsequent builds will be faster)> Task :compileKotlin> Task :compileJava NO-SOURCE> Task :processResources> Task :classes> Task :bootJarMainClassName> Task :bootJar> Task :inspectClassesForKotlinIC> Task :jar SKIPPED> Task :assemble​Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.Use '--warning-mode all' to show the individual deprecation warnings.See https://docs.gradle.org/6.7/userguide/command_line_interface.html#sec:command_line_warnings​BUILD SUCCESSFUL in 1m 48s5 actionable tasks: 5 executedRemoving intermediate container 70f5c69b7dde ---> 5bbd9485f873​Step 6/15 : FROM alpine as agent ---> 28f6e2705743Step 7/15 : USER root ---> Using cache ---> 4a0572e8ff4aStep 8/15 : WORKDIR /agent ---> Using cache ---> 0374735af05dStep 9/15 : RUN wget -qO- https://storage.googleapis.com/cloud-debugger/compute-java/debian-wheezy/cdbg_java_agent_gce.tar.gz | tar xvz ---> Using cache ---> ef141a7d5acc​Step 10/15 : FROM openjdk:14-slim ---> 38fca30874abStep 11/15 : WORKDIR /app ---> Using cache ---> 4dd576d38d65Step 12/15 : COPY --from=agent /agent/cdbg_java_agent.so agent.so ---> Using cache ---> ff5fc93d2d5eStep 13/15 : COPY --from=build /dist/build/libs/user-service.jar app.jar ---> 4b009124345fStep 14/15 : EXPOSE 8080 ---> Running in 99f71fb86534Removing intermediate container 99f71fb86534 ---> 38fcd9d00999Step 15/15 : CMD ["java", "-agentpath:/app/agent.so", "-Dcom.google.cdbg.breakpoints.enable_canary=false", "-jar", "app.jar"] ---> Running in 309f35aed48fRemoving intermediate container 309f35aed48f ---> 43757d33b2ee​Successfully built 43757d33b2eeExisting container found: 94d3fab4505461e8e694b61909ba1f7d827d8f7078b4dd8d1d36d4c4e0a0ac08, removing...Creating container...Container Id: b99675b5f6c00a0e5bec76999db04abf92eab29d257ee887bcea123a02789850Container name: '/quirky_stonebraker'Attaching to container '/quirky_stonebraker'...Starting container '/quirky_stonebraker''<unknown> Dockerfile: Dockerfile' has been deployed successfully.​​

The container runs fine as already stated with the mentioned deployed cloud run service. So I am out of ideas on what to do to make this work. I hope anybody can help me out here

I also add my cloudbuild.yaml in case that is relevant

steps:
  - name: 'gradle:6.8.3-jdk11'
    entrypoint: 'gradle'
    args: [ 'check', '-x', 'test']
  - name: 'gradle:6.8.3-jdk11'
    entrypoint: 'gradle'
    args: [ 'test']
  - name: 'gradle:6.8.3-jdk11'
    entrypoint: 'gradle'
    args: [ 'integration-test']
  - name: 'gcr.io/cloud-builders/docker'
    args: [ 'build', '-t', 'eu.gcr.io/$PROJECT_ID/user-service:$BUILD_ID', '.' ]
  - name: 'gcr.io/cloud-builders/docker'
    args: [ 'push', 'eu.gcr.io/$PROJECT_ID/user-service:$BUILD_ID' ]
  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
    entrypoint: gcloud
    args:
      - 'alpha'
      - 'run'
      - 'deploy'
      - 'user-service'
      - '--image=eu.gcr.io/$PROJECT_ID/user-service:$BUILD_ID'
      - '--concurrency=10'
      - '--cpu=1'
      - '--memory=512Mi'
      - '--region=europe-west3'
      - '--max-instances=2'
      - '--platform=managed'
      - '--port=8080'
      - '--timeout=3000'
      - '--set-env-vars=SQL_CONNECTION=10.28.96.3, SQL_USER=test, SQL_PASSWORD=test'
      - '--set-env-vars=AUTH0_DOMAIN=prototype.eu.auth0.com, AUTH0_CLIENT_ID=123, AUTH0_CLIENT_SECRET=123'
      - '--set-env-vars=^#^SPRING_PROFILES_ACTIVE=prod'
      - '--allow-unauthenticated'
      - '--ingress=internal'
      - '--vpc-connector=cloud-run'
      - '--vpc-egress=private-ranges-only'
      - '--set-cloudsql-instances=$PROJECT_ID:europe-west1:prototype'

images:
  - 'eu.gcr.io/$PROJECT_ID/user-service:$BUILD_ID'
timeout: 3000s

JavaScript
 
steps:  - name: 'gradle:6.8.3-jdk11'    entrypoint: 'gradle'    args: [ 'check', '-x', 'test']  - name: 'gradle:6.8.3-jdk11'    entrypoint: 'gradle'    args: [ 'test']  - name: 'gradle:6.8.3-jdk11'    entrypoint: 'gradle'    args: [ 'integration-test']  - name: 'gcr.io/cloud-builders/docker'    args: [ 'build', '-t', 'eu.gcr.io/$PROJECT_ID/user-service:$BUILD_ID', '.' ]  - name: 'gcr.io/cloud-builders/docker'    args: [ 'push', 'eu.gcr.io/$PROJECT_ID/user-service:$BUILD_ID' ]  - name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'    entrypoint: gcloud    args:      - 'alpha'      - 'run'      - 'deploy'      - 'user-service'      - '--image=eu.gcr.io/$PROJECT_ID/user-service:$BUILD_ID'      - '--concurrency=10'      - '--cpu=1'      - '--memory=512Mi'      - '--region=europe-west3'      - '--max-instances=2'      - '--platform=managed'      - '--port=8080'      - '--timeout=3000'      - '--set-env-vars=SQL_CONNECTION=10.28.96.3, SQL_USER=test, SQL_PASSWORD=test'      - '--set-env-vars=AUTH0_DOMAIN=prototype.eu.auth0.com, AUTH0_CLIENT_ID=123, AUTH0_CLIENT_SECRET=123'      - '--set-env-vars=^#^SPRING_PROFILES_ACTIVE=prod'      - '--allow-unauthenticated'      - '--ingress=internal'      - '--vpc-connector=cloud-run'      - '--vpc-egress=private-ranges-only'      - '--set-cloudsql-instances=$PROJECT_ID:europe-west1:prototype'​images:  - 'eu.gcr.io/$PROJECT_ID/user-service:$BUILD_ID'timeout: 3000s​

Answer

If you look what’s inside the zipped agent file, you’ll notice that there’s a few files in it called:

cdbg_java_agent.so
cdbg_java_agent_internals.jar
version.txt

You’re getting this problem because you missed to copy the internals.jar file to /app dir. The shared object file has a dependency and that can explain why users are instructed to create a separate directory for the Debugger.

To fix it on this case, add the missing file on to your Dockerfile like this:

COPY --from=agent /agent/cdbg_java_agent.so agent.so
COPY --from=agent /agent/cdbg_java_agent_internals.jar cdbg_java_agent_internals.jar
COPY --from=build /dist/build/libs/user-service.jar app.jar

JavaScript
 
COPY --from=agent /agent/cdbg_java_agent.so agent.soCOPY --from=agent /agent/cdbg_java_agent_internals.jar cdbg_java_agent_internals.jarCOPY --from=build /dist/build/libs/user-service.jar app.jar​

Then wait for approx. ~5 minutes until your service appears on the Debugger. Note that the source code will not appear automatically unless it’s located on a Git repo. For that you have to select the source code manually.

Advertisement

Answer