2

I installed openjdk-devel and openjdk-devel-debuginfo of the same major/minor version for an architecture in RedHat Linux Server 8+. I would like to make sure that the OpenJDK runtime has symbols for debugging. After installation, I ran the followings:

[root@localhost bin]# objdump --syms /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el8_1.x86_64/jre/bin/java

/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el8_1.x86_64/jre/bin/java:     file format elf64-x86-64

SYMBOL TABLE:
0000000000000270 l    d  .interp    0000000000000000              .interp
0000000000000290 l    d  .note.gnu.property 0000000000000000              .note.gnu.property
00000000000002b0 l    d  .note.ABI-tag  0000000000000000              .note.ABI-tag
00000000000002d0 l    d  .note.gnu.build-id 0000000000000000              .note.gnu.build-id
00000000000002f8 l    d  .hash  0000000000000000              .hash
0000000000000348 l    d  .gnu.hash  0000000000000000              .gnu.hash
0000000000000370 l    d  .dynsym    0000000000000000              .dynsym
....
....
....

[root@localhost etc]# file /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el8_1.x86_64/jre/bin/java
/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el8_1.x86_64/jre/bin/java: 
ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=613871d1514ba05fa2914c22c10f1dfe01d3d2e8, not stripped


[root@localhost bin]# objdump --syms /usr/lib/debug/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el8_1.x86_64/bin/java-1.8.0.242.b08-0.el8_1.x86_64.debug

/usr/lib/debug/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el8_1.x86_64/bin/java-1.8.0.242.b08-0.el8_1.x86_64.debug:     file format elf64-x86-64

SYMBOL TABLE:
no symbols

[root@localhost bin]# file /usr/lib/debug/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el8_1.x86_64/bin/java-1.8.0.242.b08-0.el8_1.x86_64.debug
/usr/lib/debug/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el8_1.x86_64/bin/java-1.8.0.242.b08-0.el8_1.x86_64.debug: 
ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter \004, for GNU/Linux 3.2.0, BuildID[sha1]=613871d1514ba05fa2914c22c10f1dfe01d3d2e8, with debug_info, not stripped

As the above suggests, I see the objdump for java prints out some sort of symbol table, but I had read that one should also look for .debug* in the output, which I'm not seeing in the remaining of SYMBOL TABLE section (a few dozens of lines omitted from above output for brevity).

I see the file for /usr/lib/debug/..../java...debug says with debug_info, but I need confirmation that the Java installation does have symbols.

NuCradle
  • 665
  • 1
  • 9
  • 31
  • openjdk does not provide binary builds. Those are (presumably) RedHat builds of openjdk. Java debugging does not usually rely on native debugging, so I'm confused as to what you are trying to confirm. – Elliott Frisch May 06 '20 at 02:39
  • @ElliottFrisch Thanks for the comment. Yes, these are RedHat builds, and I'm assuming `openjdk-debuginfo` is the OpenJDK build with debugging-enable option. My ultimate goal is to load a malloc profiler alongside my server and try to trace the native memory allocations. In this case, if the calls are traced back to JVM, I need to know which method is getting called. – NuCradle May 06 '20 at 02:48
  • I was worried you were going to say something like that. I think you're going to find this an exercise in frustration. Tools like flightrecorder make it look easy, but you'll be tracing an interpreter that includes its' own native code compilation. I believe the answer to your question is yes. – Elliott Frisch May 06 '20 at 02:54
  • It's a native off-heap memory or else Flight Recorder or heap dump with MAT would be a starting point. But thanks for the confirmation. – NuCradle May 06 '20 at 03:14

1 Answers1

2

java executable is just a simple launcher. You won't find JVM symbols there.

To see if JVM has debug symbols, check libjvm.so instead:

nm /usr/lib/jvm/jre/lib/amd64/server/libjvm.so

My ultimate goal is to load a malloc profiler alongside my server and try to trace the native memory allocations. In this case, if the calls are traced back to JVM, I need to know which method is getting called.

Well, if you'd started with this question, you wouldn't have fallen into XY problem trap.

Even with JVM debug symbols, native memory profilers (e.g. jemalloc) cannot show Java methods. They simply don't know how to unwind Java stack, so the traces will likely break at some random hex addresses, like in this question.

I'd suggest to try async-profiler to profile malloc, mprotect and mmap calls. This tool can show mixed Java+native stack traces. Here is an example of using async-profiler to profile native allocations. This video also demonstrates how async-profiler can help in finding native memory leaks.

apangin
  • 92,924
  • 10
  • 193
  • 247
  • Thank you for sharing this. Unfortunately, this new profiler needs to be approved in our env. which is going to be a lengthy process (nonetheless, after trying it, I think I'd push hard to use it). Do we have references as to what other projects have utilized async-profiler? I will try to watch/read some of the related resources to get a better hang of its capabilities but this looks very powerful, so thank you for sharing this. – NuCradle May 07 '20 at 02:43
  • As for your comment regarding Java methods, you are correct. I was hoping we can see where the native calls were being made to identify the culprit. Once that native method is identified, we might be able to use it to repeatedly dump Java stack with `jstack` as demonstrated [in this article] (https://github.com/jeffgriffith/native-jvm-leaks) using `jstack ${PID} | tee stack-${i}.txt | grep "${SEARCH_FOR}"`. But I think your async-profile does it intrusively after all. – NuCradle May 07 '20 at 02:47
  • If the debug info is in /usr/lib/debug, this nm command is not relevant. I have not used openjdk-devel-debuginfo, but openjdk-8-dbg installs to /usr/lib/debug: `$ dpkg -L openjdk-8-dbg` outputs entries like `/usr/lib/debug/.build-id/03/17f517a518dee47febb310f62e609737e1dc63.debug` – Deanna Gelbart Apr 16 '21 at 21:05
  • 1
    @DeannaGelbart Right. I usually check the presense of symbols with `gdb` - it works both with embedded symtab and with external debuginfo. – apangin Apr 16 '21 at 22:11