Обсуждение: High CPU usage

Hello,

I am using PostgreSQL v14.5 on Linux Debian 11.5. I recently observe very
high CPU usage on my Linux system as below

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
COMMAND
2357756 postgres  20   0 2441032   2,3g      4 S 298,7  67,9   2114:58
Tspjzj2Z

I could not find any file named Tspjzj2Z on the file system. I could not
find PID number using below SQL

SELECT * FROM pg_stat_activity

I also see another suggestion like below to identify long running queries

SELECT max(now() - xact_start) FROM pg_stat_activity
                               WHERE state IN ('idle in transaction',
'active');

I get no long running query at all.

There is no replication of any kind. This is a single instance server which
alows certification login only.

I appreciate any help to figure this out.

Thanks & Regards,
Ertan

On 2022-10-20 15:59, ertan.kucukoglu@1nar.com.tr wrote:
> Hello,
>
> I am using PostgreSQL v14.5 on Linux Debian 11.5. I recently observe very
> high CPU usage on my Linux system as below
>
>      PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
> COMMAND
> 2357756 postgres  20   0 2441032   2,3g      4 S 298,7  67,9   2114:58
> Tspjzj2Z
>
> I could not find any file named Tspjzj2Z on the file system. I could not
> find PID number using below SQL

Hi,

I'm not an expert in PostgreSQL but that looks like a rogue app, if 
you're lucky just a miner running as the prostgres user, likely the 
result of a postgres RCE exploited successfully... The more worring case 
would be a program exfiltrating and/or encrypting the database in a 
ransomware attack.

The executable has most likely been removed to hide traces, or cleaned 
up automatically from ex. /tmp, however if the process is still running 
you should be able to cat the executable, and any other open files, 
directly from /proc/<pid>/ (look for exe and fd/*).

I strongly recommend you check other postgres servers you have, make a 
copy of any process file found (for later investigation), then isolate 
or shutdown these servers and proceed with a proper investigation from a 
livecd or revovery OS.

> There is no replication of any kind. This is a single instance server which
> alows certification login only.

Is is even available from the outside world? Else you should likely 
audit any internal hosts that could have accessed your postgresql 
server. If you have firewall logs looks for unusual connection attempts, 
any evidence of scanning, etc.

Hackers will often spend quite some time once inside to gather as much 
information as possible before doing any real damage, although if this 
is effectively a miner it would be less likely to be that kind of attack 
as they would probably not risk getting discovered with something that 
will at best make them pennies...

If you see the process having any open database files, it's possible 
it's either compressing them to exfiltrate the data or encrypting them, 
or both....

Hope this helps...

--
Thomas

On 10/20/22 12:59, ertan.kucukoglu@1nar.com.tr wrote:
> Hello,
> 
> I am using PostgreSQL v14.5 on Linux Debian 11.5. I recently observe very
> high CPU usage on my Linux system as below
> 
>      PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
> COMMAND
> 2357756 postgres  20   0 2441032   2,3g      4 S 298,7  67,9   2114:58
> Tspjzj2Z

The USER is the system user postgres which could be running any program 
not necessarily the Postgres server. In this case that would seem to be 
the program Tspjzj2Z.

Try running:

top -U postgres -c

to get the full command line under COMMAND.

> 
> I could not find any file named Tspjzj2Z on the file system. I could not
> find PID number using below SQL

I doubt this has anything to do with the Postgres server itself.

Is this machine accessible to the Internet?

There is the possibility you have had an intrusion.

> 
> SELECT * FROM pg_stat_activity
> 
> I also see another suggestion like below to identify long running queries
> 
> SELECT max(now() - xact_start) FROM pg_stat_activity
>                                 WHERE state IN ('idle in transaction',
> 'active');
> 
> I get no long running query at all.
> 
> There is no replication of any kind. This is a single instance server which
> alows certification login only.
> 
> I appreciate any help to figure this out.
> 
> Thanks & Regards,
> Ertan
> 
> 
> 
> 

-- 
Adrian Klaver
adrian.klaver@aklaver.com

ertan.kucukoglu@1nar.com.tr schrieb am 20.10.2022 um 21:59:
> Hello,
>
> I am using PostgreSQL v14.5 on Linux Debian 11.5. I recently observe very
> high CPU usage on my Linux system as below
>
>      PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
> COMMAND
> 2357756 postgres  20   0 2441032   2,3g      4 S 298,7  67,9   2114:58
> Tspjzj2Z
>

The program name looks suspiciously as if you have been hacked.

See e.g.

https://stackoverflow.com/questions/50149034
https://dba.stackexchange.com/questions/191282/

https://www.imperva.com/blog/deep-dive-database-attacks-scarlett-johanssons-picture-used-for-crypto-mining-on-postgre-database/