Обсуждение: "soft lockup" in kernel

Dear community

Twice today our PG 9.1 server has caused a "soft lockup", with a kernel
message like this:

[1813775.496127] BUG: soft lockup - CPU#3 stuck for 73s! [postgres:18723]

Full dmesg output - http://pastebin.com/YdWSmNUp

The incidents were approximately two hours apart and the server was
momentarily unavailable before coming back again, with no restore actions
required - it just carried on. It's never done this before, I've checked
in the logs. The server is about 3 weeks old and runs Debian 7 under
VMWare.

Does anybody know what could cause this, and, if so, is it something to be
concerned about and what can be done to stop it?

Any help greatly appreciated.

Stuart


This email and any attachments contain confidential and proprietary information of Glide Utilities Limited intended
onlyfor the use of the person to whom it is addressed. Unauthorised disclosure, copying or distribution of the email or
itscontent may result in legal liability. If you have received the email in error, please immediately notify us by
telephoneon +44 333 666 5555 or email glide@glide.uk.com 

The sender does not accept liability for any loss or damage from receipt or use thereof which arises as a result of
internettransmission. Any views/opinions expressed within this email and any attachments are that of the individual and
notnecessarily that of Glide Utilities Limited. 

Glide is a registered trademark of Glide Utilities Limited. Registered Office: Alpha Tower, Suffolk Street Queensway,
Birmingham,B1 1TT. Registered in England & Wales. Registered Company No. 06194523. 

On Fri, Jul 5, 2013 at 7:00 AM, Dennis Jenkins wrpte


> Before I looked at your pastebin, I was going to ask "What kind of
>storage
> are the VMDKs on?  If they are on NFS, iSCSI or FC, could the NAS/SAN be
> experiencing a problem?" But I see in the stack trace that the kernel
>thread
> hung in "vmxnet3_xmit_frame" (sending an Ethernet frame on your virtual
>NIC).

I did not spot that.

> Describe your vmware network topology.
> Do you share the same VLAN for guest traffic with NFS or iSCSI used by
>vmware
> for storing VMDKs?

No. iSCSI traffic between the VMWare hosts and the SAN uses completely
separate NICs and different switches to the "production" LAN.

> Are there any errors recorded on the Ethernet switch connected to your
> VMWare servers?

No, there is nothing in the logs on the switch.

> What path does a packet take to get from a postgresql server process in
>your
> VM to a client?

Postgres is the backend database to a web application. Only the web
application uses it. It sits on another VM in the same cluster (but not on
the same physical host, so it definitely goes through a physical switch).
Is that what you mean?

> What version of VMWare are you running?

ESXi 5.0.0.

> If you are managing it with vCenter, are there any alarms of events in
>the
> VMWare logs?

I've had a look at the task activity in VCEnter and found these two events
at almost the same time as the kernel messages. In both cases the start
time (the first time below) is 5-6 seconds after the kernel message, and
I've seen that the clock on the Postgres VM and the VCenter server, at
least, are in sync (it may not, of course, be the VCenter server's clock
that these logs get the time from).

Remove snapshot
GLIBMPDB001_replica
Completed
GLIDE\Svc_vcenter
05/07/2013 11:58:41
05/07/2013 11:58:41
05/07/2013 11:59:03

Remove snapshot
GLIBMPDB001_replica
Completed
GLIDE\Svc_vcenter
05/07/2013 10:11:10
05/07/2013 10:11:10
05/07/2013 10:11:23

For clarity, here are the kernel messages:


syslog:Jul  5 10:11:05 glibmpdb001 kernel: [1807328.506991] BUG: soft
lockup - CPU#0 stuck for 25s! [postgres:5937]
syslog:Jul  5 11:58:35 glibmpdb001 kernel: [1813775.496127] BUG: soft
lockup - CPU#3 stuck for 73s! [postgres:18723]


This is done by Veeam and not a person. There has not been a newer version
of this event since (at the time of writing this e-mail).

Regardless of this, it's not like we've only just introduced Veeam this
morning, it's been running for a few weeks, and these lockups have only
happened today.


Do you think this is worth further investigation or could there be another
explanation?

Stuart


This email and any attachments contain confidential and proprietary information of Glide Utilities Limited intended
onlyfor the use of the person to whom it is addressed. Unauthorised disclosure, copying or distribution of the email or
itscontent may result in legal liability. If you have received the email in error, please immediately notify us by
telephoneon +44 333 666 5555 or email glide@glide.uk.com 

The sender does not accept liability for any loss or damage from receipt or use thereof which arises as a result of
internettransmission. Any views/opinions expressed within this email and any attachments are that of the individual and
notnecessarily that of Glide Utilities Limited. 

Glide is a registered trademark of Glide Utilities Limited. Registered Office: Alpha Tower, Suffolk Street Queensway,
Birmingham,B1 1TT. Registered in England & Wales. Registered Company No. 06194523.