Chris Ball: Tracing internal function calls in a binary

Tracing internal function calls in a binary

Chris Ball — Fri, 30 Nov 2007 20:27:00 +0000

Dear everyone who likes Unix,

I have a binary (which uses glib and was compiled from C) and I'd like to get output with the function name each time any function in that binary is called. So, I'd like the output of ltrace(1), but for function calls rather than dynamic library calls. I am bored of adding g_debug("%s", G_STRFUNC); to the top of all my functions.

You'd think this would be easy, given that incredibly similar tools have existed for twenty years, but so far the shortest answer I've heard starts "well, you could write a gcc profile function stub that..". It would be nice not have to recompile, since gdb certainly doesn't have to, but I'd welcome the way to achieve this with a recompile as well.

Any ideas? Thanks!

Update: jmbr wins, with the only solution that doesn't require anything more than gdb, and no recompile. Here's his script: http://superadditive.com/software/callgraph. I'd like to work on it to add support for modules loaded with dlopen().

"Tracing internal function calls in a binary" by TomTromey

Wed, 05 Dec 2007 01:10:37 +0000

Like Mark said, ftrace -- part of frysk -- can do this. frysk has a lot of potential for fun stuff like this; last year I was playing around writing jython scripts to debug things. Back then frysk wasn't mature enough to do much interesting, but this has changed.

Also ltrace can do it for things loaded from shared libraries. Unfortunately this doesn't work with the main executable. You could hack around this, though, by having a small main and then putting the rest of your program into a .so.

Pace Federico, but systemtap can't do userspace yet. However, that is also coming.

Finally, I wonder if a valgrind skin exists to do this.

"Tracing internal function calls in a binary" by John Levon

Mon, 03 Dec 2007 01:51:03 +0000

It's easy with DTrace:

dtrace -n 'pid:::entry, pid:::return {}' -c /bin/mycommand

"Tracing internal function calls in a binary" by Rob

Sun, 02 Dec 2007 13:43:39 +0000

LTTng can apparently do this, but I have to admit I haven't got it working yet!

"Tracing internal function calls in a binary" by Ross

Sun, 02 Dec 2007 08:32:29 +0000

oprofile can't do this, because it's a statistical profiler. If you have a function which is called frequently but very short, it may never notice it being called.

"Tracing internal function calls in a binary" by Stoffe

Sat, 01 Dec 2007 09:26:41 +0000

I thought that oprofile could do this.

So, are you surprised to learn that it can't, or are you trying to be helpful but in a smug oh-look-I-know-more-than-someone-else-on-the-internet way?

If you are trying to be helpful, the grown up way to do that is saying "oprofile will do want you want".

Thanks.

"Tracing internal function calls in a binary" by Davyd

Sat, 01 Dec 2007 01:30:18 +0000

I thought that oprofile could do this.

"Tracing internal function calls in a binary" by Chris

Fri, 30 Nov 2007 23:56:10 +0000

Hi Toby!

Yes, your example is what one of my coworkers recommended (but he didn't have code handy, so thanks for that).

It's clear that a gdb solution is preferable, though. I wonder if a patch to accomplish jmbr's hack through a "trace *"-like syntax would be accepted upstream.

"Tracing internal function calls in a binary" by Chris

Fri, 30 Nov 2007 23:37:19 +0000

Ian, many thanks to you too -- I also like the look of etrace.

"Tracing internal function calls in a binary" by Chris

Fri, 30 Nov 2007 23:31:41 +0000

jmbr, thanks so much! This looks perfect, and works.

My process opens some shared libraries with the glib module_open() call, though, and those obviously don't get breakpoints set ahead of time.

Those libraries have individual debuginfo files in /usr/lib/debug; I'm going to try doing nm /usr/lib/debug/foo/*.debug and massaging the output so that it fits into the trace.gdb style, then I'll hope that gdb can resolve the functions once they're dynamically loaded. Can anyone think of a reason this won't work, or a better way? :)

Thanks!

"Tracing internal function calls in a binary" by Mark Wielaard

Fri, 30 Nov 2007 23:28:18 +0000

You might want to check out frysk http://www.sourceware.org/frysk/

It very recently (read today) got various function and syscall tracing options merged in by Petr Machata: http://sourceware.org/ml/frysk/2007-q4/msg00164.html

man page explaining the different options of call tracing: http://sourceware.org/git/?p=frysk.git;a=blob;f=frysk-core/frysk/bindir/ftrace.xml;hb=HEAD

I am afraid you will have to build it from a current git checkout for now though: http://sourceware.org/frysk/build/

And you might find some bugs, but we welcome reports! http://sourceware.org/bugzilla/

"Tracing internal function calls in a binary" by Toby Jaffey

Fri, 30 Nov 2007 23:06:05 +0000

I'd do it with instrumentation as you can recompile, but etrace looks a bit heavyweight.

Here's a contrived example showing how to add the hooks, compile the code and get a live trace.

http://the.earth.li/~toby/hello.c

"Tracing internal function calls in a binary" by Ian McKellar

Fri, 30 Nov 2007 22:44:30 +0000

If you can recompile you can play with GCC's function instrumentation feature. Google found this: http://ndevilla.free.fr/etrace/

"Tracing internal function calls in a binary" by Federico Mena Quintero

Fri, 30 Nov 2007 22:06:06 +0000

Dtrace can do this.

I'm not sure if Systemtap can dive into userspace these days, or if they have a related tool for that.

"Tracing internal function calls in a binary" by jmbr

Fri, 30 Nov 2007 21:54:18 +0000

Hi Chris,

My hack to achieve is to compile the program using gcc's -g option and take control of gdb. The technique is illustrated in the following script:

http://superadditive.com/software/callgraph

Hope it helps

"Tracing internal function calls in a binary" by Andrew Sutherland

Fri, 30 Nov 2007 21:22:06 +0000

Have you looked at chronicle recorder (http://code.google.com/p/chronicle-recorder/)? It's arguably a bit heavy-weight; you are running your program under valgrind, resulting in a complete execution trace. But it can tell you all the functions that were called, what their arguments were, and other exciting debugging functionality from the year 2020. (But the run will be slow...)