Using OPerf to profile custom kernel modules

Discussion:

Bow-Nan Cheng

2013-04-05 17:22:14 UTC

Hello,

I'm having a bit of difficulty getting oprofile to correctly profile my
custom kernel module (or even profile the current kernel in general). I'm
running CentOS 6.3 with a custom 3.2 kernel. Here's the process i used to
install oprofile-0.9.8:

./configure --with-kernel=/home/username/linux-3.2.32
make
sudo make install

Then i run operf to profile the entire system:

sudo /usr/local/bin/operf -s

However, when i check the oprofile_data/current folder, it's empty and the
"opreport" command throws a "no samples found" error

Any suggestions? I'm sure its just something small..

Maynard Johnson

2013-04-05 17:48:36 UTC

Permalink

Post by Bow-Nan Cheng
Hello,
./configure --with-kernel=/home/username/linux-3.2.32
make
sudo make install
sudo /usr/local/bin/operf -s
However, when i check the oprofile_data/current folder, it's empty and the "opreport" command throws a "no samples found" error
Any suggestions? I'm sure its just something small..

The first thing I would suggest is to boot back to your stock CentOS 6.3 kernel and build oprofile without the --with-kernel configure option. I'm quite certain that 6.3 uses a kernel version that supports operf, so as long as you have the CentOS kernel headers package installed, oprofile (with operf) will build just fine. So try that, and see how operf runs. Try operf in both single app mode and system-wide mode (see operf man page -- and since you're building it yourself, you'll need to do: 'export MANPATH=<oprof-install-dir>/share/man:$MANPATH').

If operf built with the stock CentOS kernel works OK, then you may be running into the bug with the '--with-kernel' option that another user recently encountered. If so, apply my patch that I posted to the list on Wednesday of this week -- subject "[PATCH] Fix broken --with-kernel configure option".

Good luck!

-Maynard

Post by Bow-Nan Cheng
------------------------------------------------------------------------------
Minimize network downtime and maximize team effectiveness.
Reduce network management and security costs.Learn how to hire
the most talented Cisco Certified professionals. Visit the
Employer Resources Portal
http://www.cisco.com/web/learning/employer_resources/index.html
_______________________________________________
oprofile-list mailing list
https://lists.sourceforge.net/lists/listinfo/oprofile-list

Bow-Nan Cheng

2013-04-05 18:27:05 UTC

Permalink

Hello,

Thanks for your quick response!

I rebooted into the latest stock kernel -- 2.6.32-358.2.1.el6.x86_64

Recompiled oprofile:
make clean
./configure
make
sudo make install

Then tried to re-run operf, and it gave me this error:
Your kernel's Performance Events Subsystem does not support your processor
type. Please use the opcontrol command instead of operf.

cat /proc/cpuinfo says my processor is:
model : 23
model name : Intel(R) Core(TM)2 Duo CPU T9900 @ 3.06GHz

Which.. according to this site:
http://web.eece.maine.edu/~vweaver/projects/perf_events/support.html should
support the perf_event.

When i run:

/usr/local/bin/opcontrol --start
It seems to update the /var/lib/oprofile/samples folder once, but then it
doesn't touch it again. The result is that "opreport -l" gives the same
error about being too strict. Perhaps i'm doing something wrong in my
compile or setup?

Thanks for your help again!

Post by Bow-Nan Cheng

Post by Bow-Nan Cheng
Hello,
I'm having a bit of difficulty getting oprofile to correctly profile my

custom kernel module (or even profile the current kernel in general). I'm
running CentOS 6.3 with a custom 3.2 kernel. Here's the process i used to

Post by Bow-Nan Cheng
./configure --with-kernel=/home/username/linux-3.2.32
make
sudo make install
sudo /usr/local/bin/operf -s
However, when i check the oprofile_data/current folder, it's empty and

the "opreport" command throws a "no samples found" error

Post by Bow-Nan Cheng
Any suggestions? I'm sure its just something small..

The first thing I would suggest is to boot back to your stock CentOS 6.3
kernel and build oprofile without the --with-kernel configure option. I'm
quite certain that 6.3 uses a kernel version that supports operf, so as
long as you have the CentOS kernel headers package installed, oprofile
(with operf) will build just fine. So try that, and see how operf runs.
Try operf in both single app mode and system-wide mode (see operf man page
-- and since you're building it yourself, you'll need to do: 'export
MANPATH=<oprof-install-dir>/share/man:$MANPATH').
If operf built with the stock CentOS kernel works OK, then you may be
running into the bug with the '--with-kernel' option that another user
recently encountered. If so, apply my patch that I posted to the list on
Wednesday of this week -- subject "[PATCH] Fix broken --with-kernel
configure option".
Good luck!
-Maynard
------------------------------------------------------------------------------

Post by Bow-Nan Cheng
Minimize network downtime and maximize team effectiveness.
Reduce network management and security costs.Learn how to hire
the most talented Cisco Certified professionals. Visit the
Employer Resources Portal
http://www.cisco.com/web/learning/employer_resources/index.html
_______________________________________________
oprofile-list mailing list
https://lists.sourceforge.net/lists/listinfo/oprofile-list

Maynard Johnson

2013-04-05 19:04:59 UTC

Permalink

Post by Bow-Nan Cheng
Hello,
Thanks for your quick response!
I rebooted into the latest stock kernel -- 2.6.32-358.2.1.el6.x86_64
make clean
./configure
make
sudo make install
Your kernel's Performance Events Subsystem does not support your processor type. Please use the opcontrol command instead of operf.
model: 23
Which.. according to this site: http://web.eece.maine.edu/~vweaver/projects/perf_events/support.html should support the perf_event.
/usr/local/bin/opcontrol --start
It seems to update the /var/lib/oprofile/samples folder once, but then it doesn't touch it again. The result is that "opreport -l" gives the same error about being too strict. Perhaps i'm doing something wrong in my compile or setup?
Thanks for your help again!

Do you have the CentOS 6.3 kernel headers package installed -- i.e., with version that matches your running kernel?

Try 'strace operf --help > out 2>&1' and search for a syscall. You should see a number associated with that syscall entry in the strace output which should match the number for the '#define __NR_perf_event_open' in unistd.h (will need to grep /usr/include for '__NR_perf_event_open' to find the actual unistd.h file that has this #define).

It may also be instructive to cd into oprofile/pe_profiling and do the following:
'touch operf.cpp'
'make'
Copy the g++ command onto the command line, add '-save-temps', then run 'make' again.
Look at the operf.ii file that is generated by -save-temps option and look for "asm/unistd". What's the pathname?

-Maynard

Post by Bow-Nan Cheng

The first thing I would suggest is to boot back to your stock CentOS 6.3 kernel and build oprofile without the --with-kernel configure option. I'm quite certain that 6.3 uses a kernel version that supports operf, so as long as you have the CentOS kernel headers package installed, oprofile (with operf) will build just fine. So try that, and see how operf runs. Try operf in both single app mode and system-wide mode (see operf man page -- and since you're building it yourself, you'll need to do: 'export MANPATH=<oprof-install-dir>/share/man:$MANPATH').
If operf built with the stock CentOS kernel works OK, then you may be running into the bug with the '--with-kernel' option that another user recently encountered. If so, apply my patch that I posted to the list on Wednesday of this week -- subject "[PATCH] Fix broken --with-kernel configure option".
Good luck!
-Maynard

Bow-Nan Cheng

2013-04-05 19:47:06 UTC

Permalink

Hello Maynard,

Thank you again for all your help, to answer your questions below:

1. Kernel headers -- i have kernel headers installed for the kernel version
i'm running:
yum install kernel-headers
uname -a yields 2.6.32-358.2.1.el6.x86_64
and /usr/src/kernels/2.6.32-358.2.1.el6.x86_64/ has the headers

2. strace -- i ran the command you suggested and the number after syscall
(298) matches what's in the one in the kernel headers:

Bow-Nan Cheng

2013-04-05 20:01:58 UTC

Permalink

Actually, i did a quick grep for __NR_perf_event_open in /usr/include/ and:

asm/unistd_64.h:#define __NR_perf_event_open 298

so it seems like the same #

Post by Bow-Nan Cheng
Hello Maynard,
1. Kernel headers -- i have kernel headers installed for the kernel
yum install kernel-headers
uname -a yields 2.6.32-358.2.1.el6.x86_64
and /usr/src/kernels/2.6.32-358.2.1.el6.x86_64/ has the headers
2. strace -- i ran the command you suggested and the number after syscall

Maynard Johnson

2013-04-05 20:54:34 UTC

Permalink

Post by Bow-Nan Cheng
Hello Maynard,
yum install kernel-headers
uname -a yields 2.6.32-358.2.1.el6.x86_64 and /usr/src/kernels/2.6.32-358.2.1.el6.x86_64/ has the headers

Bow-Nan Cheng

2013-04-05 21:25:56 UTC

Permalink

Thanks for your help again Maynard,

You were right -- i actually didn't install the headers, but rather the
kernel-devel package. I can try again to see if the operf works now with
the base kernel, but in the meantime, i pulled the latest from GIT and
applied the patch. Everything seems to be in order:

./configure --with-kernel=/home/username/linux-3.2.32/usr
make
sudo make install

Then i run:

operf -s
--vmlinux=/home/username/linux-3.2.32/arch/x86/boot/compressed/vmlinux

When i run it for 10 minutes or so, i get a "stats" folder created in:

oprofile_data/samples/current

The stats folder is completely empty. In the operf.log file, here's what it
says:

-- OProfile/operf Statistics --
Nr. non-backtrace samples: 0
Nr. kernel samples: 0
Nr. user space samples: 0
Nr. samples lost due to sample address not in expected range for domain: 0
Nr. lost kernel samples: 0
Nr. samples lost due to sample file open failure: 0
Nr. samples lost due to no permanent mapping: 0
Nr. user context kernel samples lost due to no app info available: 0
Nr. user samples lost due to no app info available: 0
Nr. backtraces skipped due to no file mapping: 0
Nr. hypervisor samples dropped due to address out-of-range: 0
Nr. samples lost reported by perf_events kernel: 0

Prior to doing a manual compile on oprofile, i did a yum install oprofile
and it installed a 0.9.7 version. When i ran the opcontrol and opreport
commands, even if i only ran opcontrol for only a few minutes, i would get
tons and tons of records. Perhaps i'm not configuring it correctly to
record the right things? Thanks again for your help..

-Bow-Nan

Post by Bow-Nan Cheng

Post by Bow-Nan Cheng
Hello Maynard,
1. Kernel headers -- i have kernel headers installed for the kernel
yum install kernel-headers
uname -a yields 2.6.32-358.2.1.el6.x86_64 and

/usr/src/kernels/2.6.32-358.2.1.el6.x86_64/ has the headers

Post by Bow-Nan Cheng
2. strace -- i ran the command you suggested and the number after

Maynard Johnson

2013-04-05 21:36:43 UTC

Permalink

Post by Bow-Nan Cheng
Thanks for your help again Maynard,
./configure --with-kernel=/home/username/linux-3.2.32/usr
make
sudo make install
operf -s --vmlinux=/home/username/linux-3.2.32/arch/x86/boot/compressed/vmlinux
oprofile_data/samples/current
-- OProfile/operf Statistics --
Nr. non-backtrace samples: 0
Nr. kernel samples: 0
Nr. user space samples: 0
Nr. samples lost due to sample address not in expected range for domain: 0
Nr. lost kernel samples: 0
Nr. samples lost due to sample file open failure: 0
Nr. samples lost due to no permanent mapping: 0
Nr. user context kernel samples lost due to no app info available: 0
Nr. user samples lost due to no app info available: 0
Nr. backtraces skipped due to no file mapping: 0
Nr. hypervisor samples dropped due to address out-of-range: 0
Nr. samples lost reported by perf_events kernel: 0
Prior to doing a manual compile on oprofile, i did a yum install oprofile and it installed a 0.9.7 version. When i ran the opcontrol and opreport commands, even if i only ran opcontrol for only a few minutes, i would get tons and tons of records. Perhaps i'm not configuring it correctly to record the right things? Thanks again for your help..

Well, that is very strange. At least with the 3.2 kernel, you don't get the ENOENT error. Let's collect a complete debug output of something simple (not a system-wide profile). First, just try 'operf ls', and then run opreport. I'm assuming opreport will show no samples. If so, then run the following:
operf --verbose=all ls > operf-debug.out 2>&1
Please send the operf-debug.out file back here, and I'll take a look.

Thanks.
-Maynard

Post by Bow-Nan Cheng
-Bow-Nan

Post by Bow-Nan Cheng
Hello Maynard,
yum install kernel-headers
uname -a yields 2.6.32-358.2.1.el6.x86_64 and /usr/src/kernels/2.6.32-358.2.1.el6.x86_64/ has the headers

Bow-Nan Cheng

2013-04-05 21:49:11 UTC

Permalink

Thanks for taking a look into this. I've run the commands as you requested
and have attached the output. Please let me know if there's anything else
that you need. Thanks!

opreport returns:

Using /home/bcheng/Documents/oprofile/oprofile_data/samples/ for samples
directory.
opreport error: No sample file found: If using opcontrol for profiling,
try running 'opcontrol --dump'; otherwise, specify a session containing
sample files.

-Bow-Nan

Post by Bow-Nan Cheng

Post by Bow-Nan Cheng
Thanks for your help again Maynard,
You were right -- i actually didn't install the headers, but rather the

kernel-devel package. I can try again to see if the operf works now with
the base kernel, but in the meantime, i pulled the latest from GIT and

Post by Bow-Nan Cheng
./configure --with-kernel=/home/username/linux-3.2.32/usr
make
sudo make install
operf -s

--vmlinux=/home/username/linux-3.2.32/arch/x86/boot/compressed/vmlinux

Post by Bow-Nan Cheng
oprofile_data/samples/current
The stats folder is completely empty. In the operf.log file, here's what
-- OProfile/operf Statistics --
Nr. non-backtrace samples: 0
Nr. kernel samples: 0
Nr. user space samples: 0

Post by Bow-Nan Cheng
Nr. lost kernel samples: 0
Nr. samples lost due to sample file open failure: 0
Nr. samples lost due to no permanent mapping: 0
Nr. user context kernel samples lost due to no app info available: 0
Nr. user samples lost due to no app info available: 0
Nr. backtraces skipped due to no file mapping: 0
Nr. hypervisor samples dropped due to address out-of-range: 0
Nr. samples lost reported by perf_events kernel: 0
Prior to doing a manual compile on oprofile, i did a yum install

oprofile and it installed a 0.9.7 version. When i ran the opcontrol and
opreport commands, even if i only ran opcontrol for only a few minutes, i
would get tons and tons of records. Perhaps i'm not configuring it
correctly to record the right things? Thanks again for your help..
Well, that is very strange. At least with the 3.2 kernel, you don't get
the ENOENT error. Let's collect a complete debug output of something simple
(not a system-wide profile). First, just try 'operf ls', and then run
opreport. I'm assuming opreport will show no samples. If so, then run the
operf --verbose=all ls > operf-debug.out 2>&1
Please send the operf-debug.out file back here, and I'll take a look.
Thanks.
-Maynard

Post by Bow-Nan Cheng
-Bow-Nan

Post by Bow-Nan Cheng
Hello Maynard,
1. Kernel headers -- i have kernel headers installed for the
yum install kernel-headers
uname -a yields 2.6.32-358.2.1.el6.x86_64 and

/usr/src/kernels/2.6.32-358.2.1.el6.x86_64/ has the headers

Post by Bow-Nan Cheng

Post by Bow-Nan Cheng
2. strace -- i ran the command you suggested and the number after

Maynard Johnson

2013-04-06 16:03:05 UTC

Permalink

Thanks for taking a look into this. I've run the commands as you requested and have attached the output. Please let me know if there's anything else that you need. Thanks!
Using /home/bcheng/Documents/oprofile/oprofile_data/samples/ for samples directory.
opreport error: No sample file found: If using opcontrol for profiling,
try running 'opcontrol --dump'; otherwise, specify a session containing
sample files.

Strange . . . Everything looks normal in the debug output *except* no samples are arriving from the kernel. We're getting other information from the kernel, such as PERF_RECORD_MMAP and PERF_RECORD_COMM events, but no samples. Can you try building and installing the 3.2 perf tool and see if 'perf record' works.

-Maynard

-Bow-Nan

operf --verbose=all ls > operf-debug.out 2>&1
Please send the operf-debug.out file back here, and I'll take a look.
Thanks.
-Maynard

Post by Bow-Nan Cheng
-Bow-Nan

Post by Bow-Nan Cheng
Hello Maynard,
yum install kernel-headers
uname -a yields 2.6.32-358.2.1.el6.x86_64 and /usr/src/kernels/2.6.32-358.2.1.el6.x86_64/ has the headers