Sunday, 23 December 2012

Linux Tip:a process uses 100% cpu core how to improve performance in linux

1. Is there any way to measure a specific process CPU usage by cores?

I know top is good for measuring the whole system's CPU usage by cores and taskset can provide information about which CPU core is allowed for the process to run on. But how to measure a specific process' CPU usage by CPU cores?

ou can use:

mpstat -P ALL 1

It shows how much each core is busy and it updates automatically each second. The output would be something like this (on a quad-core processor):

10:54:41 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
10:54:42 PM  all    8.20    0.12    0.75    0.00    0.00    0.00    0.00    0.00   90.93
10:54:42 PM    0   24.00    0.00    2.00    0.00    0.00    0.00    0.00    0.00   74.00
10:54:42 PM    1   22.00    0.00    2.00    0.00    0.00    0.00    0.00    0.00   76.00
10:54:42 PM    2    2.02    1.01    0.00    0.00    0.00    0.00    0.00    0.00   96.97
10:54:42 PM    3    2.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   98.00
10:54:42 PM    4   14.15    0.00    1.89    0.00    0.00    0.00    0.00    0.00   83.96
10:54:42 PM    5    1.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   99.00
10:54:42 PM    6    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
10:54:42 PM    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  

You can still do this in top


Type 1 - it shows each cpu

Limit the processes shown by having that specific process run under a specific user account and use Type 'u' to limit to that user


I have a piece of java code running on two different machines, but on one of the linux machines, the code uses a lot of CPU (close to 100% cpu usage). On the other machine the same code uses less cpu (under 3 to 4%). The machine where cpu usage is high, is a more powerful machine, more CPU and and more memory. This has started happening recently and performance on the machine with high cpu usage has degraded significantly. I am wondering if anyone has any ideas why something like this could happen, possible causes behind this etc. any guesses? No recent changes in hardware were made, no recent code updates.

If you suspect hardware failure, check dmesg for any output. Depending on the hardware vendor, there may be some kind of IPMI implementation (like Dell's DRAC) with a web interface that will show you failed hardware.

To be honest, it's probably a bug in your application. "Add timer hooks, find the bottleneck."

3Q and Ans:

I'm experiencing some problems with my LAMP server. Recently, everything became very slow, even though visitor count on my websites didn't change to much. When I run top command, it sais that mysql process has taken over 150-200% of CPU. How's that possible, I always thought that 100% is a maximum?
I'm running Ubuntu 9.04 server edition with 1,5 GB RAM
my.cnf settings:
key_buffer      = 64M
max_allowed_packet  = 16M
thread_stack        = 192K
thread_cache_size       = 8

myisam-recover         = BACKUP
max_connections        = 200
table_cache            = 512
table_definition_cache = 512
thread_concurrency     = 2

read_buffer_size = 1M
sort_buffer_size = 4M
join_buffer_size = 1M

query_cache_limit   = 1M  # the maximum size of individual query results
query_cache_size    = 128M

  1. Increase Key Buffer ( yours is currently 64MB, but total indexes are 116M, so put at least 128MB). Should help immediately.
  2. Run mysqloptimize and mysqlrepair on your tables
  3. Increase table cache/ decrease total number of tables to increase the table cache hit rate. Maybe you've got some unused or old tables which could be deleted.
Other recommended confugration options:
  • log_slow_queries = /var/log/mysql/mysql-slow.log
  • long_query_time = 4
  • log-queries-not-using-indexes

I've noticed a problem, not related to CPU. If you're using apache and MySQL on the same server you can reach bad conditions (RAM) when your apache activity increase.

MySQLTunner tells you that using the 200 available connections (your max connection setting) you will fill the RAM. Let's say you have limited apache to 150 process you'll certainly won't have enough RAM when MySQL and apache will try to use 150 connexions (as Apache as well is a good RAM eater).

So this is about RAM and you're maybe not hit yet :-) The top commands show only 15 apache process (but you're in load average 3/6/16 so that means the storm was 15 minute ago and is now in leaving).

About the CPU problem, to complement the good response of shakalandy, this may be because of one single query. It can be on a huge table, or doing a lot of re-index tasks, or using a lot of temporary file, an index missing (removed?), etc. The only way to detect it is activating the slow query log (maybe with a high thresold, like 8s). Then use the mysqlsla tool to analyse this slow query log, and run some explain on the identified queries.

1 comment: