Cannot run rsyslogd, so no current data in /var/log/messages.

Discussion:

(too old to reply)

Jean-David Beyer

2015-10-31 05:18:38 UTC

I am running Red Hat Enterprise Linux Server release 6.7 (Santiago) and
about 9:39 my machine started screwing up. It was very busy, but not
doing anything much.

After working on some things without success, I tried rebooting the
machine, but I could not get it to restart or shutdown. So I powered it
down and then restarted it. It sort-of runs, but no sound (though
Windows 7 Professional runs the sound just fine). I notice that
/var/log/messages never gets any larger.

I tried noticed rsyslogd is not running and I tried to start it. I get
this message:

root[/etc/rc.d/init.d]# ./rsyslog start
Starting system logger: /sbin/rsyslogd: error while loading shared
libraries: libgcc_s.so.1: cannot open shared object file: No such file
or directory [FAILED]

root[/etc/rc.d/init.d]# ldd /sbin/rsyslogd
linux-vdso.so.1 => (0x00007ffe593ee000)
libz.so.1 => /lib64/libz.so.1 (0x00007f94dea62000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f94de845000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f94de640000)
librt.so.1 => /lib64/librt.so.1 (0x00007f94de438000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f94de222000)
libc.so.6 => /lib64/libc.so.6 (0x00007f94dde8d000)
/lib64/ld-linux-x86-64.so.2 (0x000000328c200000)

Now /lib64/libgcc_s.so.1 is there.

jeandavid8[/lib64]$ ls -l libgcc_s*
-rwxr-xr-x. 1 root root 93320 Oct 8 03:42 libgcc_s-4.4.7-20120601.so.1
lrwxrwxrwx. 1 root root 28 Jul 22 12:36 libgcc_s.so.1 ->
libgcc_s-4.4.7-20120601.so.1

And the link points to where I suppose it should.

jeandavid8[/lib64]$ file libgcc_s*
libgcc_s-4.4.7-20120601.so.1: ELF 64-bit LSB shared object, x86-64,
version 1 (SYSV), dynamically linked, stripped
libgcc_s.so.1: symbolic link to
`libgcc_s-4.4.7-20120601.so.1'

Anyone have any ideas how to debug this?

--
.~. Jean-David Beyer Registered Linux User 85642.
/V\ PGP-Key:166D840A 0C610C8B Registered Machine 1935521.
/( )\ Shrewsbury, New Jersey http://linuxcounter.net
^^-^^ 01:00:01 up 1:00, 2 users, load average: 4.14, 4.25, 4.25

Scott Lurndal

2015-11-02 14:01:50 UTC

Permalink

Post by Jean-David Beyer
I am running Red Hat Enterprise Linux Server release 6.7 (Santiago) and
about 9:39 my machine started screwing up. It was very busy, but not
doing anything much.
After working on some things without success, I tried rebooting the
machine, but I could not get it to restart or shutdown. So I powered it
down and then restarted it. It sort-of runs, but no sound (though
Windows 7 Professional runs the sound just fine). I notice that
/var/log/messages never gets any larger.
I tried noticed rsyslogd is not running and I tried to start it. I get
root[/etc/rc.d/init.d]# ./rsyslog start
Starting system logger: /sbin/rsyslogd: error while loading shared
libraries: libgcc_s.so.1: cannot open shared object file: No such file
or directory [FAILED]
root[/etc/rc.d/init.d]# ldd /sbin/rsyslogd
linux-vdso.so.1 => (0x00007ffe593ee000)
libz.so.1 => /lib64/libz.so.1 (0x00007f94dea62000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f94de845000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f94de640000)
librt.so.1 => /lib64/librt.so.1 (0x00007f94de438000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f94de222000)
libc.so.6 => /lib64/libc.so.6 (0x00007f94dde8d000)
/lib64/ld-linux-x86-64.so.2 (0x000000328c200000)
Now /lib64/libgcc_s.so.1 is there.
jeandavid8[/lib64]$ ls -l libgcc_s*
-rwxr-xr-x. 1 root root 93320 Oct 8 03:42 libgcc_s-4.4.7-20120601.so.1
lrwxrwxrwx. 1 root root 28 Jul 22 12:36 libgcc_s.so.1 ->
libgcc_s-4.4.7-20120601.so.1
And the link points to where I suppose it should.
jeandavid8[/lib64]$ file libgcc_s*
libgcc_s-4.4.7-20120601.so.1: ELF 64-bit LSB shared object, x86-64,
version 1 (SYSV), dynamically linked, stripped
libgcc_s.so.1: symbolic link to
`libgcc_s-4.4.7-20120601.so.1'
Anyone have any ideas how to debug this?

Have you run an fsck on the affected filesystem?

Does 'dmesg' show anything unusual?

Jean-David Beyer

2015-11-02 14:19:23 UTC

Permalink

Post by Scott Lurndal

Have you run an fsck on the affected filesystem?

No.

Post by Scott Lurndal
Does 'dmesg' show anything unusual?

No.

Intuition to the rescue.

I set SELinux to permissive mode and SELinux complained I was trying to
access that library and did not have permission (even though I was super
user). So I did

# touch /.autorelabel; reboot

And a reboot occurred (because I was in permissive mode).
The reboot was very slow because it checked hundreds of thousands of
files and directories.

But after that the reboot restarted (removing /.autorelabel and setting
SELinux back to enforcing mode), everything works.

It is still curious because I could read that library (with od) OK.

How a few items got the wrong labels I do not know.

--
.~. Jean-David Beyer Registered Linux User 85642.
/V\ PGP-Key:166D840A 0C610C8B Registered Machine 1935521.
/( )\ Shrewsbury, New Jersey http://linuxcounter.net
^^-^^ 09:15:01 up 1 day, 21:59, 4 users, load average: 4.43, 4.47, 4.89