Обсуждение: Pgsql troubleshooting & Iscsi

Hi everybody,

I installed a postgresql database on Redhat 7.1 and I decided to move the database on an ISCSI device (LUN) inside a
logicalvolume, mounted at starting of the machine (xfs formatted). The mounting point is /var/lib/pgsql  

At the boot of the server, postgresql.service is in failed status.

In messages.log :
systemd: mounting /var/lib/pgsql
 starting PostgreSQL database server
 kernel sdv: unknown partition table
 sd 2:0:0:0: [sdb] attached SCSI disk
 xfs (dm-4): Mounting V4 Filesystem
 postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty
 postgresql.service: control process exited, code=exited status=1
 Failed to start PostgreSQL database server.


When I'm logged on the server, if it try to start manually the database : systemctl start postgresql --> OK (and I
don'tlose any data, database is available) 

I think it's a problem of order in the boot process : network service must be started, then iscsi, then lvm etc... So I
triedto force dependencies on the /usr/lib/systemd/system/postgresql.service adding "After=lvm-pgscan.service
iscsi.service"etc... but the result is the same : failure in starting postgresql 



systemd: Starting Remote File Systems (Pre)
systemd: Reached Remote File Systems (Pre)
systemd: mounting /var/lib/pgsql
systemd:  starting PostgreSQL database server
kernel sdb: unknown partition table
postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty
kernel: xfs (dm-4): Mounting V4 Filesystem
postgresql.service: control process exited, code=exited status=1
Failed to start PostgreSQL database server.
Unit postgresql.service entered failed state
sd 2:0:0:0: [sdb] attached SCSI disk
starting LVM2 PV scan on device 8:16
kernel: xfs (dm-3): Ending clean mount
systemd: Mounted /var/lib/pgsql
Starting Remote File Systems
Reached target Remote File Systems

Any ideas ?

regards,
Magique

On 02/17/2016 09:07 AM, proj@free.fr wrote:
> Hi everybody,
>
> I installed a postgresql database on Redhat 7.1 and I decided to move the database on an ISCSI device (LUN) inside a
logicalvolume, mounted at starting of the machine (xfs formatted). The mounting point is /var/lib/pgsql 
>
> At the boot of the server, postgresql.service is in failed status.
>
> In messages.log :
> systemd: mounting /var/lib/pgsql
>   starting PostgreSQL database server
>   kernel sdv: unknown partition table
>   sd 2:0:0:0: [sdb] attached SCSI disk
>   xfs (dm-4): Mounting V4 Filesystem
>   postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty
>   postgresql.service: control process exited, code=exited status=1
>   Failed to start PostgreSQL database server.
>
>
> When I'm logged on the server, if it try to start manually the database : systemctl start postgresql --> OK (and I
don'tlose any data, database is available) 
>
> I think it's a problem of order in the boot process : network service must be started, then iscsi, then lvm etc... So
Itried to force dependencies on the /usr/lib/systemd/system/postgresql.service adding "After=lvm-pgscan.service
iscsi.service"etc... but the result is the same : failure in starting postgresql 
>
>
>
> systemd: Starting Remote File Systems (Pre)
> systemd: Reached Remote File Systems (Pre)
> systemd: mounting /var/lib/pgsql

^ File system not mounted yet.

> systemd:  starting PostgreSQL database server
> kernel sdb: unknown partition table
> postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty

^ Cannot find mount point

> kernel: xfs (dm-4): Mounting V4 Filesystem
> postgresql.service: control process exited, code=exited status=1
> Failed to start PostgreSQL database server.
> Unit postgresql.service entered failed state
> sd 2:0:0:0: [sdb] attached SCSI disk
> starting LVM2 PV scan on device 8:16
> kernel: xfs (dm-3): Ending clean mount
> systemd: Mounted /var/lib/pgsql

^ Now file system is mounted.

> Starting Remote File Systems
> Reached target Remote File Systems
>
> Any ideas ?

Find where file system mounting is completed and start Postgres after
that. I am still figuring out systemd, so I can not offer anything more
concrete.

>
> regards,
> Magique
>
>


--
Adrian Klaver
adrian.klaver@aklaver.com

On 02/17/2016 06:07 PM, proj@free.fr wrote:
> Hi everybody,
>
> I installed a postgresql database on Redhat 7.1 and I decided to move the database on an ISCSI device (LUN) inside a
logicalvolume, mounted at starting of the machine (xfs formatted). The mounting point is /var/lib/pgsql 
>
> At the boot of the server, postgresql.service is in failed status.
>
> In messages.log :
> systemd: mounting /var/lib/pgsql
>   starting PostgreSQL database server
>   kernel sdv: unknown partition table
>   sd 2:0:0:0: [sdb] attached SCSI disk
>   xfs (dm-4): Mounting V4 Filesystem
>   postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty
>   postgresql.service: control process exited, code=exited status=1
>   Failed to start PostgreSQL database server.
>
>
> When I'm logged on the server, if it try to start manually the database : systemctl start postgresql --> OK (and I
don'tlose any data, database is available) 
>
> I think it's a problem of order in the boot process : network service must be started, then iscsi, then lvm etc... So
Itried to force dependencies on the /usr/lib/systemd/system/postgresql.service adding "After=lvm-pgscan.service
iscsi.service"etc... but the result is the same : failure in starting postgresql 
>
>
>
> systemd: Starting Remote File Systems (Pre)
> systemd: Reached Remote File Systems (Pre)
> systemd: mounting /var/lib/pgsql
> systemd:  starting PostgreSQL database server
> kernel sdb: unknown partition table
> postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty
> kernel: xfs (dm-4): Mounting V4 Filesystem
> postgresql.service: control process exited, code=exited status=1
> Failed to start PostgreSQL database server.
> Unit postgresql.service entered failed state
> sd 2:0:0:0: [sdb] attached SCSI disk
> starting LVM2 PV scan on device 8:16
> kernel: xfs (dm-3): Ending clean mount
> systemd: Mounted /var/lib/pgsql
> Starting Remote File Systems
> Reached target Remote File Systems
>
> Any ideas ?
>
> regards,
> Magique
>
>
Put  _netdev in the option field in fstab. This indicates that the
device must be mounted after network is available.

I have for example :

/dev/disk/by-path/ip-10.48.4.1:3260-iscsi-iqn.2004-04.com.qnap:ts-851:iscsi.lun0.e6ae62-lun-0-part1
/data3 btrfs _netdev 0 0

Regards
Jeroen

I already try to change the dependencies with postgresql.service (adding as a prerequesite lvm scan, iscsi) .
The mounting began before postgresql but the mount was successfull after the failure.
----- Mail d'origine -----
De: Adrian Klaver <adrian.klaver@aklaver.com>
À: proj@free.fr, pgsql-general@postgresql.org
Envoyé: Wed, 17 Feb 2016 18:35:15 +0100 (CET)
Objet: Re: [GENERAL] Pgsql troubleshooting & Iscsi

On 02/17/2016 09:07 AM, proj@free.fr wrote:
> Hi everybody,
>
> I installed a postgresql database on Redhat 7.1 and I decided to move the database on an ISCSI device (LUN) inside a
logicalvolume, mounted at starting of the machine (xfs formatted). The mounting point is /var/lib/pgsql 
>
> At the boot of the server, postgresql.service is in failed status.
>
> In messages.log :
> systemd: mounting /var/lib/pgsql
> starting PostgreSQL database server
> kernel sdv: unknown partition table
> sd 2:0:0:0: [sdb] attached SCSI disk
> xfs (dm-4): Mounting V4 Filesystem
> postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty
> postgresql.service: control process exited, code=exited status=1
> Failed to start PostgreSQL database server.
>
>
> When I'm logged on the server, if it try to start manually the database : systemctl start postgresql --> OK (and I
don'tlose any data, database is available) 
>
> I think it's a problem of order in the boot process : network service must be started, then iscsi, then lvm etc... So
Itried to force dependencies on the /usr/lib/systemd/system/postgresql.service adding "After=lvm-pgscan.service
iscsi.service"etc... but the result is the same : failure in starting postgresql 
>
>
>
> systemd: Starting Remote File Systems (Pre)
> systemd: Reached Remote File Systems (Pre)
> systemd: mounting /var/lib/pgsql

^ File system not mounted yet.

> systemd: starting PostgreSQL database server
> kernel sdb: unknown partition table
> postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty

^ Cannot find mount point

> kernel: xfs (dm-4): Mounting V4 Filesystem
> postgresql.service: control process exited, code=exited status=1
> Failed to start PostgreSQL database server.
> Unit postgresql.service entered failed state
> sd 2:0:0:0: [sdb] attached SCSI disk
> starting LVM2 PV scan on device 8:16
> kernel: xfs (dm-3): Ending clean mount
> systemd: Mounted /var/lib/pgsql

^ Now file system is mounted.

> Starting Remote File Systems
> Reached target Remote File Systems
>
> Any ideas ?

Find where file system mounting is completed and start Postgres after
that. I am still figuring out systemd, so I can not offer anything more
concrete.

>
> regards,
> Magique
>
>


--
Adrian Klaver
adrian.klaver@aklaver.com

On 02/17/2016 10:30 AM, proj@free.fr wrote:
> I already try to change the dependencies with postgresql.service (adding as a prerequesite lvm scan, iscsi) .
> The mounting began before postgresql but the mount was successfull after the failure.

Exactly, the drive was not mounted when the Postgres service started, so
Postgres could not find its data directory. From your log:

postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty


> ----- Mail d'origine -----
> De: Adrian Klaver <adrian.klaver@aklaver.com>
> À: proj@free.fr, pgsql-general@postgresql.org
> Envoyé: Wed, 17 Feb 2016 18:35:15 +0100 (CET)
> Objet: Re: [GENERAL] Pgsql troubleshooting & Iscsi
>
> On 02/17/2016 09:07 AM, proj@free.fr wrote:
>> Hi everybody,
>>
>> I installed a postgresql database on Redhat 7.1 and I decided to move the database on an ISCSI device (LUN) inside a
logicalvolume, mounted at starting of the machine (xfs formatted). The mounting point is /var/lib/pgsql 
>>
>> At the boot of the server, postgresql.service is in failed status.
>>
>> In messages.log :
>> systemd: mounting /var/lib/pgsql
>> starting PostgreSQL database server
>> kernel sdv: unknown partition table
>> sd 2:0:0:0: [sdb] attached SCSI disk
>> xfs (dm-4): Mounting V4 Filesystem
>> postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty
>> postgresql.service: control process exited, code=exited status=1
>> Failed to start PostgreSQL database server.
>>
>>
>> When I'm logged on the server, if it try to start manually the database : systemctl start postgresql --> OK (and I
don'tlose any data, database is available) 
>>
>> I think it's a problem of order in the boot process : network service must be started, then iscsi, then lvm etc...
SoI tried to force dependencies on the /usr/lib/systemd/system/postgresql.service adding "After=lvm-pgscan.service
iscsi.service"etc... but the result is the same : failure in starting postgresql 
>>
>>
>>
>> systemd: Starting Remote File Systems (Pre)
>> systemd: Reached Remote File Systems (Pre)
>> systemd: mounting /var/lib/pgsql
>
> ^ File system not mounted yet.
>
>> systemd: starting PostgreSQL database server
>> kernel sdb: unknown partition table
>> postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty
>
> ^ Cannot find mount point
>
>> kernel: xfs (dm-4): Mounting V4 Filesystem
>> postgresql.service: control process exited, code=exited status=1
>> Failed to start PostgreSQL database server.
>> Unit postgresql.service entered failed state
>> sd 2:0:0:0: [sdb] attached SCSI disk
>> starting LVM2 PV scan on device 8:16
>> kernel: xfs (dm-3): Ending clean mount
>> systemd: Mounted /var/lib/pgsql
>
> ^ Now file system is mounted.
>
>> Starting Remote File Systems
>> Reached target Remote File Systems
>>
>> Any ideas ?
>
> Find where file system mounting is completed and start Postgres after
> that. I am still figuring out systemd, so I can not offer anything more
> concrete.
>
>>
>> regards,
>> Magique
>>
>>
>
>


--
Adrian Klaver
adrian.klaver@aklaver.com

I already mount with the netdev option.
The network service is started before postgresql but not lvm for.
I tried to start lvm service as dependency but no changes.

----- Mail d'origine -----
De: Jeroen van Iddekinge <iddekingej@lycos.com>
À: proj@free.fr, pgsql-general@postgresql.org
Envoyé: Wed, 17 Feb 2016 18:51:42 +0100 (CET)
Objet: Re: [GENERAL] Pgsql troubleshooting & Iscsi

On 02/17/2016 06:07 PM, proj@free.fr wrote:
> Hi everybody,
>
> I installed a postgresql database on Redhat 7.1 and I decided to move the database on an ISCSI device (LUN) inside a
logicalvolume, mounted at starting of the machine (xfs formatted). The mounting point is /var/lib/pgsql 
>
> At the boot of the server, postgresql.service is in failed status.
>
> In messages.log :
> systemd: mounting /var/lib/pgsql
> starting PostgreSQL database server
> kernel sdv: unknown partition table
> sd 2:0:0:0: [sdb] attached SCSI disk
> xfs (dm-4): Mounting V4 Filesystem
> postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty
> postgresql.service: control process exited, code=exited status=1
> Failed to start PostgreSQL database server.
>
>
> When I'm logged on the server, if it try to start manually the database : systemctl start postgresql --> OK (and I
don'tlose any data, database is available) 
>
> I think it's a problem of order in the boot process : network service must be started, then iscsi, then lvm etc... So
Itried to force dependencies on the /usr/lib/systemd/system/postgresql.service adding "After=lvm-pgscan.service
iscsi.service"etc... but the result is the same : failure in starting postgresql 
>
>
>
> systemd: Starting Remote File Systems (Pre)
> systemd: Reached Remote File Systems (Pre)
> systemd: mounting /var/lib/pgsql
> systemd: starting PostgreSQL database server
> kernel sdb: unknown partition table
> postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty
> kernel: xfs (dm-4): Mounting V4 Filesystem
> postgresql.service: control process exited, code=exited status=1
> Failed to start PostgreSQL database server.
> Unit postgresql.service entered failed state
> sd 2:0:0:0: [sdb] attached SCSI disk
> starting LVM2 PV scan on device 8:16
> kernel: xfs (dm-3): Ending clean mount
> systemd: Mounted /var/lib/pgsql
> Starting Remote File Systems
> Reached target Remote File Systems
>
> Any ideas ?
>
> regards,
> Magique
>
>
Put _netdev in the option field in fstab. This indicates that the
device must be mounted after network is available.

I have for example :

/dev/disk/by-path/ip-10.48.4.1:3260-iscsi-iqn.2004-04.com.qnap:ts-851:iscsi.lun0.e6ae62-lun-0-part1
/data3 btrfs _netdev 0 0

Regards
Jeroen

On Wed, Feb 17, 2016 at 06:07:29PM +0100, proj@free.fr wrote:
> Hi everybody,
>
> I installed a postgresql database on Redhat 7.1 and I decided to move the
> database on an ISCSI device (LUN) inside a logical volume, mounted at
> starting of the machine (xfs formatted). The mounting point is /var/lib/pgsql
>
> At the boot of the server, postgresql.service is in failed status.
>
> In messages.log :
> systemd: mounting /var/lib/pgsql
>  starting PostgreSQL database server
>  kernel sdv: unknown partition table
>  sd 2:0:0:0: [sdb] attached SCSI disk
>  xfs (dm-4): Mounting V4 Filesystem
>  postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty
>  postgresql.service: control process exited, code=exited status=1
>  Failed to start PostgreSQL database server.
>
>
> When I'm logged on the server, if it try to start manually the database :
> systemctl start postgresql --> OK (and I don't lose any data, database is
> available)
>
> I think it's a problem of order in the boot process : network service must
> be started, then iscsi, then lvm etc... So I tried to force dependencies
> on the /usr/lib/systemd/system/postgresql.service adding
> "After=lvm-pgscan.service iscsi.service" etc... but the result is
> the same : failure in starting postgresql
>
> Any ideas ?

Take a look at this thread:
http://www.gossamer-threads.com/lists/linux/kernel/1332888

I'm not an XFS guru, but it seems that XFS does a bunch of checks just
on/after mount, so perhaps PgSQL cannot access /var/lib/pgsql/data
immediately.

You could igive it a try forcing some delay in systemd conf file, or
tweaking the PgSQL startup script.

--
Alberto Cabello Sánchez
Universidad de Extremadura