We upgraded our appliance to 5.5 Friday and immediately started having issues. Using the System Admin Tools -> Top we can see our CPU usage on mysql running between 300% to 400%. Machines will not checking and system becomes unusable because of slowness.

I have opened a ticket with kace and installed 6 or 7 hotfixes with no help. They are asking use to rewrite about 300+ smart labels to see if this will help. These smart labels are nothing special and were all built using the “create smart label” option.

 

Anyone else having this after upgrading to 5.5?

0 Comments   [ + ] Show Comments

Comments

Please log in to comment

Answers

0

We had the same issue after upgrading to 5.5.  After working with support they had us basically do the same fixes that you listed above.  We installed 7-8 patches and they went in and updated the 15 smart labels we have setup.  In addition to these fixes, the one thing that was done on our support call that seemed to have corrected the problem was turning off classic metering.  5.5 has a new option for metering and the classic way was causing our agent tasks to become unresponsive. Here is the link that I was sent in order to setup metering in 5.5: https://www.kace.com/support/resources/kb/solutiondetail?sol=SOL114739

Hope this helps you resolve the issue. 

Answered 10/14/2013 by: Gary.Cooper
Senior White Belt

  • Thank you for you update We have about 75 classic metering jobs active. I will disable them and give that a shot.

    I have modified the sql on about 80 machine smart labels so far. I still have about 300ish more to look at for machine and patching.

    Just curious did you talk to a kace engineer for assistance?
    • I did not talk to a KACE engineer, it was first line support that assisted me. One thing I want to mention, once you disable the classic metering, restart the AMP Server from the Agent Messaging Protocol under settings.
Please log in to comment
0

Have you upgraded the clients as well to 5.5? I found that it major releases it can cause httpstorming and that causes the appliance to bog down. A simple fix would be put the appliance on a different ip address or to restart the amp server. Just some ideas.

Answered 10/07/2013 by: ms01ak
Tenth Degree Black Belt

  • I would have to lean toward this as a known issue.

    Make sure all of your KACE clients are up to date on 5.5.
Please log in to comment
0

Can you run the command "database" from the System admin tools and see what mysql queries are running when mysql usage is 300% ?

Answered 10/07/2013 by: AbhayR
Fourth Degree Black Belt

  • | 51569784 | B1 | localhost | ORG1 | Query | 10 | Sending data | select HD_TICKET.*,
    HD_STATUS.NAME AS STATUS_NAME,
    H |
    |
    | 51575231 | B1 | localhost | ORG1 | Query | 6 | Waiting for table level lock | select HD_TICKET.*,
    HD_STATUS.NAME AS STATUS_NAME,
    H |
    |
    | 51575510 | B1 | localhost | ORG1 | Query | 6 | Waiting for table level lock | select HD_TICKET.*,
    HD_STATUS.NAME AS STATUS_NAME,
    H |
    |
    | 51584654 | B1 | localhost | ORG1 | Query | 8 | Waiting for table level lock | UPDATE HD_TICKET SET TITLE='Network Infrastructure Repairs - Baseball Stadium',HD_PRIORITY_ID='1',HD |
    | 51584655 | B1 | localhost | ORG1 | Query | 8 | Waiting for table level lock | select HD_TICKET.*, STATE
    from HD_TICKET, HD_STATUS
    |
    | 51584658 | B1 | localhost | ORG1 | Query | 8 | Waiting for table level lock | select OWNER_ID, count(*)
    from HD_TICKET, HD_STATUS
    |
    | 51584666 | B1 | localhost | ORG1 | Sleep | 6 | | NULL |
    | 51584695 | B1 | localhost | ORG1 | Query | 6 | Waiting for table level lock | SELECT COUNT(*) FROM (HD_TICKET, HD_STATUS, HD_IMPACT, HD_CATEGORY)





    It has a long list but i deleted out what showed nothing.

    Looks like where the service desk is trying to update tickets.
    • Can you check from that list, which query is running for the longest time?

      It should be in the 6th column and most probably an INSERT/UPDATE/DELETE query
      • ADODB_EXTENSION: 5.04
        Warning: Using a password on the command line interface can be insecure.
        +----------+----------------------+--------------------+-------+---------+------+------------------------------+------------------------------------------------------------------------------------------------------+
        | Id | User | Host | db | Command | Time | State | Info |
        +----------+----------------------+--------------------+-------+---------+------+------------------------------+------------------------------------------------------------------------------------------------------+
        | 139615 | R1 | 55.555.55.55:62537 | ORG1 | Sleep | 112 | | NULL |
        | 8065921 | R1 | 55.555.55.55:53102 | ORG1 | Sleep | 85 | | NULL |
        | 35169717 | root | localhost | KBSYS | Sleep | 8 | | NULL |
        | 40409378 | R2 | 55.555.55.55:51223 | NULL | Sleep | 85 | | NULL |
        | 40409383 | R2 | 55.555.55.55:51224 | ORG2 | Sleep | 438 | | NULL |
        | 48392047 | B1 | localhost | ORG1 | Sleep | 7210 | | NULL |
        | 48392177 | B1 | localhost | ORG1 | Sleep | 7209 | | NULL |
        | 48392243 | B1 | localhost | ORG1 | Sleep | 7209 | | NULL |
        | 48392379 | B1 | localhost | ORG1 | Sleep | 7209 | | NULL |
        | 48392496 | B1 | localhost | ORG1 | Sleep | 7209 | | NULL |
        | 48392758 | B1 | localhost | ORG1 | Sleep | 7209 | | NULL |
        | 48393104 | B1 | localhost | ORG1 | Sleep | 7208 | | NULL |
        | 48393208 | B1 | localhost | ORG1 | Sleep | 7208 | | NULL |
        | 48393534 | B1 | localhost | ORG1 | Sleep | 7207 | | NULL |
        | 48393793 | B1 | localhost | ORG1 | Sleep | 7206 | | NULL |
        | 49063703 | R7 | 55.555.55.55:56863 | NULL | Sleep | 85 | | NULL |
        | 49063723 | R7 | 55.555.55.55:56864 | ORG7 | Sleep | 539 | | NULL |
        | 49168498 | root | localhost | KBSYS | Sleep | 5693 | | NULL |
        | 49899033 | root | localhost | KBSYS | Sleep | 4052 | | NULL |
        | 50117217 | root | localhost | KBSYS | Sleep | 2954 | | NULL |
        | 51873597 | root | localhost | KBSYS | Sleep | 24 | | NULL |
        | 51875543 | B1 | localhost | ORG1 | Sleep | 21 | | NULL |
        | 51875575 | root | localhost | KBSYS | Sleep | 22 | | NULL |
        | 51875612 | root | localhost | KBSYS | Sleep | 21 | | NULL



        This is pretty much what is looks like the first few entries. then it gets down to where I see the ticket rules. they have low times on them around 20 or so. I took the ip addresses out.
Please log in to comment
0

Few questions:

How many ticket rules are defined approximately?

How many tickets are there overall?

Also, are you seeing mysql to eat up 300% CPU always or intermittently?

Are you on a physical appliance or a virtual ?

 

Answered 10/07/2013 by: AbhayR
Fourth Degree Black Belt

  • We are on a Virtual K1000 we have 7 Orgs
    19 active ticket rules in Org 1, I would have to check the other orgs
    Org 1 we have over 111k tickets in org 1, it is our main org. Archiving is turned on for tickets older than a year.

    I have not noticed the CPU issues until 5.5 upgrade on Friday.
    • Ok. What kind of issues are you having exactly? slow ui on some pages? or something else?

      Also, when you run the "top" command, what are the load averages?
      • Out of 9K+ machines we have 1 machine checking in every hour or so.

        We are unable to patch, unable to deploy software, service desk hangs up when opening pages, saving tickets etc.

        last pid: 13358; load averages: 5.10, 4.88, 4.93 up 4+22:46:04 15:36:18
        201 processes: 8 running, 193 sleeping

        Mem: 4616M Active, 7799M Inact, 473M Wired, 705M Cache, 214M Buf, 2226M Free
        Swap: 512M Total, 3180K Used, 509M Free


        PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
        1228 mysql 75 51 0 3188M 3125M select 3 0:01 394.19% mysqld
        9090 www 1 -8 0 331M 44288K biowr 6 1:42 36.47% httpd
        12068 www 1 66 0 331M 44408K select 4 0:07 14.60% httpd
        3339 www 1 20 0 331M 45940K lockf 1 0:43 12.50% httpd
        13259 www 1 20 0 331M 44136K lockf 4 0:02 11.96% httpd
        11917 www 1 4 0 331M 44348K kqread 6 0:07 10.99% httpd
        7372 www 1 53 0 330M 44148K select 6 0:30 10.69% httpd
        13223 www 1 20 0 331M 44028K lockf 3 0:02 8.89% httpd
        11043 www 1 52 0 337M 49172K CPU6 6 0:10 6.88% httpd
        12946 www 1 59 0 331M 45156K select 1 0:03 6.15% httpd
        12896 www 1 60 0 331M 44220K select 1 0:02 6.15% httpd
        13272 www 1 58 0 330M 37944K select 7 0:00 2.29% httpd
        11146 www 1 -8 0 332M 44340K piperd 7 0:07 2.10% httpd
        79584 root 1 4 0 487M 472M kqread 7 263:55 1.86% kmsgr.7.0-amd64
        1587 root 1 8 0 157M 33560K nanslp 5 37:52 1.86% php
        12066 www 1 54 0 331M 45012K select 0 0:06 1.86% httpd
        12221 www 1 48 0 331M 45516K CPU5 5 0:05 1.56% httpd
        11916 www 1 -8 0 331M 45204K piperd 7 0:06 1.27% httpd
Please log in to comment
0

Can you check the "load average" from the "top" command? 

Answered 10/07/2013 by: AbhayR
Fourth Degree Black Belt

  • last pid: 13358; load averages: 5.10, 4.88, 4.93 up 4+22:46:04 15:36:18 201 processes: 8 running, 193 sleeping - See more at: http://www.itninja.com/question/upgrading-to-5-5-problems#sthash.mw4kAQzs.dpuf

    I replied to other question also shows more info
    • Can you try disabling all the ticket rules for a few hours and see if that eases the mysql cpu ?
Please log in to comment
This content is currently hidden from public view.
Reason: Removed by member request
For more information, visit our FAQ's.

Answer this question or Comment on this question for clarity