Обсуждение: posgresql.log

On 05/21/2018 02:40 PM, Bartosz Dmytrak wrote:
> Hi Gurus,
> 
> Looking into my postgresql.log on one of my test servers I found scary 
> entry:

Is there a Web app running on this server?

The log entries below are from the Postgres logs in?:

/var/log/postgresql/

> 
> --2018-05-19 05:28:21--  http://207.148.79.161/post0514/post
> 
> Connecting to 207.148.79.161:80... connected.
> 
> HTTP request sent, awaiting response... 200 OK
> 
> Length: 1606648 (1.5M) [application/octet-stream]

Hmm, the below says it downloaded 12.5M.

> 
> Saving to: ‘/var/lib/postgresql/10/main/postgresq1’

The postgresq1 file is actually there?

If so have you looked at the file:

file postgresq1

to get an idea of what it is?

> 
> 0K .......... .......... .......... .......... ..........  3% 71.0K 21s
> 
>      50K .......... .......... .......... .......... ..........  6%  
> 106K 17s
> 
>     100K .......... .......... .......... .......... ..........  9%  
> 213K 13s
> 
>     150K .......... .......... .......... .......... .......... 12%  
> 213K 11s
> 
>     200K .......... .......... .......... .......... .......... 15% 16.3M 9s
> 
>     250K .......... .......... .......... .......... .......... 19%  215K 8s
> 
>     300K .......... .......... .......... .......... .......... 22% 15.6M 7s
> 
>     350K .......... .......... .......... .......... .......... 25% 11.7M 6s
> 
>     400K .......... .......... .......... .......... .......... 28%  219K 5s
> 
>     450K .......... .......... .......... .......... .......... 31% 12.1M 5s
> 
>     500K .......... .......... .......... .......... .......... 35% 11.7M 4s
> 
>     550K .......... .......... .......... .......... .......... 38% 12.2M 3s
> 
>     600K .......... .......... .......... .......... .......... 41% 12.1M 3s
> 
>     650K .......... .......... .......... .......... .......... 44%  228K 3s
> 
>     700K .......... .......... .......... .......... .......... 47% 12.2M 3s
> 
>     750K .......... .......... .......... .......... .......... 50% 12.1M 2s
> 
>     800K .......... .......... .......... .......... .......... 54% 11.7M 2s
> 
>     850K .......... .......... .......... .......... .......... 57% 12.1M 2s
> 
>     900K .......... .......... .......... .......... .......... 60% 11.8M 2s
> 
>     950K .......... .......... .......... .......... .......... 63% 12.1M 1s
> 
>    1000K .......... .......... .......... .......... .......... 66% 12.0M 1s
> 
>    1050K .......... .......... .......... .......... .......... 70%  243K 1s
> 
>    1100K .......... .......... .......... .......... .......... 73% 12.1M 1s
> 
>    1150K .......... .......... .......... .......... .......... 76% 12.1M 1s
> 
>    1200K .......... .......... .......... .......... .......... 79% 11.7M 1s
> 
>    1250K .......... .......... .......... .......... .......... 82% 12.1M 1s
> 
>    1300K .......... .......... .......... .......... .......... 86% 12.1M 0s
> 
>    1350K .......... .......... .......... .......... .......... 89% 11.8M 0s
> 
>    1400K .......... .......... .......... .......... .......... 92% 12.1M 0s
> 
>    1450K .......... .......... .......... .......... .......... 95% 12.1M 0s
> 
>    1500K .......... .......... .......... .......... .......... 98% 11.8M 0s
> 
>    1550K .......... ........ 100% 12.5M=2.6s
> 
> 2018-05-19 05:28:25 (598 KB/s) - 
> ‘/var/lib/postgresql/10/main/postgresq1’ saved [1606648/1606648]
> 
> Downloaded file is not posgresql but postgresq1(one).
> 
> It was pure pg instalation without any contrib modules addons etc, 
> istalled on ubuntu box by apt manager using repos:
> 
> http://apt.postgresql.org/pub/repos/apt xenial-pgdg/main
> 
> http://apt.postgresql.org/pub/repos/apt xenial-pgdg
> 
> I have never seen such entry on other my other servers…
> 
> Could you be so kind and explain me what is it? I am afraid my postgres 
> has been hacekd.
> 
> Best Regards
> 
> */Bartosz Dmytrak/*
> 


-- 
Adrian Klaver
adrian.klaver@aklaver.com

> On May 21, 2018, at 3:21 PM, Steve Crawford <scrawford@pinpointresearch.com> wrote:
>
>
>
> If this is a test server and you can take it offline for forensics I would do so, especially if it could provide a
pathto other internal or critical resources. If you can image it for safekeeping and forensics, even better. 

+1

It's compromised. Image it if possible; save the compromise payload you know about if not.

Treat it as compromised and unsafe to attach to a network until you completely wipe and reinstall it.

>
> That appears to be output from wget but the intrusion, if any, could be through any number of vectors (web, ssh,
localattack, etc.) not directly related to PostgreSQL. Check in your other logs starting with a search for anything
relatedto that IP address. 

It's probably a compromise via postgresql open to the network with insecure settings. I've seen several of those
reportedrecently, and this one is saving it's payload to the postgresql data directory - somewhere no other user or app
willhave access to, but which a compromised postgresql can easily write to. 

Check the pg_hba.conf and packet filter / firewall settings and see what the issue may be. Do the same checks on all
yourother postgresql servers, test and production. If there's a configuration mistake that let one server be
compromisedit's may well be there on others too. 

>
> Verify the usual. Patches up to date, ports appropriately firewalled off, no default passwords, etc.
>
> IP comes back to vultr.com which is a cloud company (i.e. could be anyone) but if it is an attack perhaps contact
theirabuse department. 

The C&C server there is already down; It can't hurt to notify them, but I doubt Choopa would be particularly interested
beyondthat point unless a subpoena or search warrant were involved. 

> Unless you are positive the server was not attacked, don't trust it unless you can be absolutely certain it is clean.
Bestbet is to backup any critical data (and check it for trustworthiness), wipe and rebuild. 

+1.

>
> Only you (well, OK, maybe them, now) know what data was on this server but depending on its importance, internal
policies,legal requirements and agreements with third-parties you may have notification requirements and could need to
engageforensics experts. 

Cheers,
  Steve

On 05/21/2018 04:40 PM, Bartosz Dmytrak wrote:

Hi Gurus,

Looking into my postgresql.log on one of my test servers I found scary entry:

 

--2018-05-19 05:28:21--  http://207.148.79.161/post0514/post

Connecting to 207.148.79.161:80... connected.

HTTP request sent, awaiting response... 200 OK

Length: 1606648 (1.5M) [application/octet-stream]

Saving to: ‘/var/lib/postgresql/10/main/postgresq1’

 

     0K .......... .......... .......... .......... ..........  3% 71.0K 21s

    50K .......... .......... .......... .......... ..........  6%  106K 17s

   100K .......... .......... .......... .......... ..........  9%  213K 13s

   150K .......... .......... .......... .......... .......... 12%  213K 11s

[snip]

  1500K .......... .......... .......... .......... .......... 98% 11.8M 0s

  1550K .......... ........                                   100% 12.5M=2.6s

 

2018-05-19 05:28:25 (598 KB/s) - ‘/var/lib/postgresql/10/main/postgresq1’ saved [1606648/1606648]

 

Downloaded file is not posgresql but postgresq1(one).

 

It was pure pg instalation without any contrib modules addons etc, istalled on ubuntu box by apt manager using repos:

http://apt.postgresql.org/pub/repos/apt xenial-pgdg/main

http://apt.postgresql.org/pub/repos/apt xenial-pgdg

 

I have never seen such entry on other my other servers…

Could you be so kind and explain me what is it? I am afraid my postgres has been hacekd.

This looks like what happens when the adobe flash player package downloads the closed-source binary installer.  Thus, I wouldn't be surprised if the repository package isn't downloading the installation binaries from http://207.148.79.161/post0514/post.

-----Original Message-----
From: Adrian Klaver [mailto:adrian.klaver@aklaver.com]
Sent: Tuesday, May 22, 2018 12:03 AM
To: Bartosz Dmytrak <bdmytrak@gmail.com>; pgsql-general@postgresql.org
Subject: Re: posgresql.log

On 05/21/2018 02:40 PM, Bartosz Dmytrak wrote:
> Hi Gurus,
>
> Looking into my postgresql.log on one of my test servers I found scary
> entry:

Is there a Web app running on this server?

The log entries below are from the Postgres logs in?:

/var/log/postgresql/

>
> --2018-05-19 05:28:21--  http://207.148.79.161/post0514/post
>
> Connecting to 207.148.79.161:80... connected.
>
> HTTP request sent, awaiting response... 200 OK
>
> Length: 1606648 (1.5M) [application/octet-stream]

Hmm, the below says it downloaded 12.5M.

>
> Saving to: ‘/var/lib/postgresql/10/main/postgresq1’

The postgresq1 file is actually there?

If so have you looked at the file:

file postgresq1

to get an idea of what it is?

>
> 0K .......... .......... .......... .......... ..........  3% 71.0K
> 21s
>
>      50K .......... .......... .......... .......... ..........  6%
> 106K 17s
>
>     100K .......... .......... .......... .......... ..........  9%
> 213K 13s
>
>     150K .......... .......... .......... .......... .......... 12%
> 213K 11s
>
>     200K .......... .......... .......... .......... .......... 15%
> 16.3M 9s
>
>     250K .......... .......... .......... .......... .......... 19%
> 215K 8s
>
>     300K .......... .......... .......... .......... .......... 22%
> 15.6M 7s
>
>     350K .......... .......... .......... .......... .......... 25%
> 11.7M 6s
>
>     400K .......... .......... .......... .......... .......... 28%
> 219K 5s
>
>     450K .......... .......... .......... .......... .......... 31%
> 12.1M 5s
>
>     500K .......... .......... .......... .......... .......... 35%
> 11.7M 4s
>
>     550K .......... .......... .......... .......... .......... 38%
> 12.2M 3s
>
>     600K .......... .......... .......... .......... .......... 41%
> 12.1M 3s
>
>     650K .......... .......... .......... .......... .......... 44%
> 228K 3s
>
>     700K .......... .......... .......... .......... .......... 47%
> 12.2M 3s
>
>     750K .......... .......... .......... .......... .......... 50%
> 12.1M 2s
>
>     800K .......... .......... .......... .......... .......... 54%
> 11.7M 2s
>
>     850K .......... .......... .......... .......... .......... 57%
> 12.1M 2s
>
>     900K .......... .......... .......... .......... .......... 60%
> 11.8M 2s
>
>     950K .......... .......... .......... .......... .......... 63%
> 12.1M 1s
>
>    1000K .......... .......... .......... .......... .......... 66%
> 12.0M 1s
>
>    1050K .......... .......... .......... .......... .......... 70%
> 243K 1s
>
>    1100K .......... .......... .......... .......... .......... 73%
> 12.1M 1s
>
>    1150K .......... .......... .......... .......... .......... 76%
> 12.1M 1s
>
>    1200K .......... .......... .......... .......... .......... 79%
> 11.7M 1s
>
>    1250K .......... .......... .......... .......... .......... 82%
> 12.1M 1s
>
>    1300K .......... .......... .......... .......... .......... 86%
> 12.1M 0s
>
>    1350K .......... .......... .......... .......... .......... 89%
> 11.8M 0s
>
>    1400K .......... .......... .......... .......... .......... 92%
> 12.1M 0s
>
>    1450K .......... .......... .......... .......... .......... 95%
> 12.1M 0s
>
>    1500K .......... .......... .......... .......... .......... 98%
> 11.8M 0s
>
>    1550K .......... ........ 100% 12.5M=2.6s
>
> 2018-05-19 05:28:25 (598 KB/s) -
> ‘/var/lib/postgresql/10/main/postgresq1’ saved [1606648/1606648]
>
> Downloaded file is not posgresql but postgresq1(one).
>
> It was pure pg instalation without any contrib modules addons etc,
> istalled on ubuntu box by apt manager using repos:
>
> http://apt.postgresql.org/pub/repos/apt xenial-pgdg/main
>
> http://apt.postgresql.org/pub/repos/apt xenial-pgdg
>
> I have never seen such entry on other my other servers…
>
> Could you be so kind and explain me what is it? I am afraid my
> postgres has been hacekd.
>
> Best Regards
>
> */Bartosz Dmytrak/*
>


--
Adrian Klaver
adrian.klaver@aklaver.com



HI, thanks for response,
Yes - there is also webapp running on the server, but still it's rather odd to find it's logs in postgresql.log file
(locatedin /var/log/postgresql, where my log exists). postgresq1 file exists in /var/lib/postgresql/10/main and it's
binaryfile, I've also noticed there is a n596tx.so which is not a part of standard installation. 
Fortunately there is no important data on this server so, a according to other advice, I'll rebuilt it with more
aggressivesecurity settings and I'll apply them to other servers too.  

Best regards,
Bartek

-----Original Message-----
From: Steve Atkins [mailto:steve@blighty.com]
Sent: Tuesday, May 22, 2018 12:44 AM
To: PG-General Mailing List <pgsql-general@postgresql.org>
Cc: bdmytrak@gmail.com
Subject: Re: posgresql.log


> On May 21, 2018, at 3:21 PM, Steve Crawford <scrawford@pinpointresearch.com> wrote:
>
>
>
> If this is a test server and you can take it offline for forensics I would do so, especially if it could provide a
pathto other internal or critical resources. If you can image it for safekeeping and forensics, even better. 

+1

It's compromised. Image it if possible; save the compromise payload you know about if not.

Treat it as compromised and unsafe to attach to a network until you completely wipe and reinstall it.

>
> That appears to be output from wget but the intrusion, if any, could be through any number of vectors (web, ssh,
localattack, etc.) not directly related to PostgreSQL. Check in your other logs starting with a search for anything
relatedto that IP address. 

It's probably a compromise via postgresql open to the network with insecure settings. I've seen several of those
reportedrecently, and this one is saving it's payload to the postgresql data directory - somewhere no other user or app
willhave access to, but which a compromised postgresql can easily write to. 

Check the pg_hba.conf and packet filter / firewall settings and see what the issue may be. Do the same checks on all
yourother postgresql servers, test and production. If there's a configuration mistake that let one server be
compromisedit's may well be there on others too. 

>
> Verify the usual. Patches up to date, ports appropriately firewalled off, no default passwords, etc.
>
> IP comes back to vultr.com which is a cloud company (i.e. could be anyone) but if it is an attack perhaps contact
theirabuse department. 

The C&C server there is already down; It can't hurt to notify them, but I doubt Choopa would be particularly interested
beyondthat point unless a subpoena or search warrant were involved. 

> Unless you are positive the server was not attacked, don't trust it unless you can be absolutely certain it is clean.
Bestbet is to backup any critical data (and check it for trustworthiness), wipe and rebuild. 

+1.

>
> Only you (well, OK, maybe them, now) know what data was on this server but depending on its importance, internal
policies,legal requirements and agreements with third-parties you may have notification requirements and could need to
engageforensics experts. 

Cheers,
  Steve


Hi Steve,
Thanks a lot. That’s virtual server, so I'll do the backup. I suppose it was hacked, but fortunately there are no
importantdata and I can wipe it out with no hurt. According to your advice I'll check firewall rules and block new IP
ranges(many of them are blocked now). There are no default passwords, but will work on more secure approach. This
serverhas to be accessible from internet, but not necessary for every IP. 

Thanks again for your help and advice. Hope this case will let other keep their servers more secure.

Regards,
Bartek

Hi Bartek

It is quite significant that your postgres log file has these entries.
Normally if a web application gets compromised allowing remote code
execution, the attacker will be able to run scripts (often via cron
and at) as the user running the web application.  Typically www-data,
tomcat8 etc.  If the attacker has managed to execute something as the
postgres user you need to check:
(i) that the web application server (eg tomcat) is not running as
root.  I see this often enough. Then when the web app gets attacked,
root is gone on the machine;
(ii) the web application has minimally configured access vi
pg_hba.conf.  For example not as the postgres user.
Just some thoughts to consider as you put your stuff back together on
the new machine.

Regards
Bob

On 22 May 2018 at 07:47, Bartosz Dmytrak <bdmytrak@gmail.com> wrote:
>
>
> -----Original Message-----
> From: Steve Atkins [mailto:steve@blighty.com]
> Sent: Tuesday, May 22, 2018 12:44 AM
> To: PG-General Mailing List <pgsql-general@postgresql.org>
> Cc: bdmytrak@gmail.com
> Subject: Re: posgresql.log
>
>
>> On May 21, 2018, at 3:21 PM, Steve Crawford <scrawford@pinpointresearch.com> wrote:
>>
>>
>>
>> If this is a test server and you can take it offline for forensics I would do so, especially if it could provide a
pathto other internal or critical resources. If you can image it for safekeeping and forensics, even better. 
>
> +1
>
> It's compromised. Image it if possible; save the compromise payload you know about if not.
>
> Treat it as compromised and unsafe to attach to a network until you completely wipe and reinstall it.
>
>>
>> That appears to be output from wget but the intrusion, if any, could be through any number of vectors (web, ssh,
localattack, etc.) not directly related to PostgreSQL. Check in your other logs starting with a search for anything
relatedto that IP address. 
>
> It's probably a compromise via postgresql open to the network with insecure settings. I've seen several of those
reportedrecently, and this one is saving it's payload to the postgresql data directory - somewhere no other user or app
willhave access to, but which a compromised postgresql can easily write to. 
>
> Check the pg_hba.conf and packet filter / firewall settings and see what the issue may be. Do the same checks on all
yourother postgresql servers, test and production. If there's a configuration mistake that let one server be
compromisedit's may well be there on others too. 
>
>>
>> Verify the usual. Patches up to date, ports appropriately firewalled off, no default passwords, etc.
>>
>> IP comes back to vultr.com which is a cloud company (i.e. could be anyone) but if it is an attack perhaps contact
theirabuse department. 
>
> The C&C server there is already down; It can't hurt to notify them, but I doubt Choopa would be particularly
interestedbeyond that point unless a subpoena or search warrant were involved. 
>
>> Unless you are positive the server was not attacked, don't trust it unless you can be absolutely certain it is
clean.Best bet is to backup any critical data (and check it for trustworthiness), wipe and rebuild. 
>
> +1.
>
>>
>> Only you (well, OK, maybe them, now) know what data was on this server but depending on its importance, internal
policies,legal requirements and agreements with third-parties you may have notification requirements and could need to
engageforensics experts. 
>
> Cheers,
>   Steve
>
>
> Hi Steve,
> Thanks a lot. That’s virtual server, so I'll do the backup. I suppose it was hacked, but fortunately there are no
importantdata and I can wipe it out with no hurt. According to your advice I'll check firewall rules and block new IP
ranges(many of them are blocked now). There are no default passwords, but will work on more secure approach. This
serverhas to be accessible from internet, but not necessary for every IP. 
>
> Thanks again for your help and advice. Hope this case will let other keep their servers more secure.
>
> Regards,
> Bartek
>
>