Some Useful Technical Information and issues.

At times, you may need to find out which Network switch and switch port are connected to which NIC of the server.

In these scenarios, you can use "tcpdump" command in your Linux/UNIX shell to find out network switch and switch port which is connected to a NIC.

Note: The server should have tcpdump installed to use this.

Here is the Syntax of the command:

tcpdump -nn -v -i -s 1500 -c 1 'ether[20:2] == 0x2000'

Example:

testserver:~ # tcpdump -nn -v -i eth3 -s 1500 -c 1 'ether[20:2] == 0x2000'

tcpdump: listening on eth3, link-type EN10MB (Ethernet), capture size 1500 bytes

03:25:22.146564 CDPv2, ttl: 180s, checksum: 692 (unverified), length 370

Device-ID (0x01), length: 11 bytes: 'ch-bx48-sw13'

Address (0x02), length: 13 bytes: IPv4 (1) 192.168.1.15

Port-ID (0x03), length: 15 bytes: 'FastEthernet0/7'

Capability (0x04), length: 4 bytes: (0x00000028): L2 Switch, IGMP snooping

Version String (0x05), length: 220 bytes:

Cisco Internetwork Operating System Software

IOS (tm) C2950 Software (C2950-I6Q4L2-M), Version 12.1(14)EA1a, RELEASE SOFTWARE

(fc1)

Copyright (c) 1986-2003 by cisco Systems, Inc.

Compiled Tue 02-Sep-03 03:33 by antonino

Platform (0x06), length: 18 bytes: 'cisco WS-C2950T-24'

Protocol-Hello option (0x08), length: 32 bytes:

VTP Management Domain (0x09), length: 6 bytes: 'ecomrd'

Duplex (0x0b), length: 1 byte: full

AVVID trust bitmap (0x12), length: 1 byte: 0x00

AVVID untrusted ports CoS (0x13), length: 1 byte: 0x00

1 packets captured

2 packets received by filter

0 packets dropped by kernel

testserver:~ #

In the above example, the network switch name and Port connected are highlighted.

##############################################

SSH time-lock tricks

You can also use different iptables parameters to limit connections to the SSH service for specific time periods.

You can use the /second, /minute, /hour, or /day switch in any of the following examples.

In the first example, if a user enters the wrong password, access to the SSH service is blocked for one minute, and the user gets only one login try per minute from that moment on:

~# iptables -A INPUT -p tcp -m state --syn --state NEW --dport 22 -m limit --limit 1/minute --limit-burst 1 -j ACCEPT
~# iptables -A INPUT -p tcp -m state --syn --state NEW --dport 22 -j DROP

In a second example, iptables are set to allow only host 192.168.10.25 to connect to the SSH service. After three failed login tries, iptables allows the host only one login try per minute:

~# iptables -A INPUT -p tcp -s 192.168.10.25 -m state --syn --state NEW --dport 22 -m limit --limit 1/minute --limit-burst 3 -j ACCEPT
~# iptables -A INPUT -p tcp -s 192.168.10.25 -m state --syn --state NEW --dport 22 -j DROP

############################################################################

Cron processing fails with user authentication failure in PAM.

Following message is logged in /var/log/cron.

crond[2393]: (root) FAILED to authorize user with PAM (The return value should be

ignored by PAM dispatch)

It seems that this message is logged every time sar processing which is executed by cron

starts and that the processing fails.

It was found that this message is logged when following configuration is

added in /etc/pam.d/password-auth (RHEL 6) and the same issue could be reproduced.

auth required pam_env.so

auth [default=1 success=ignore] pam_succeed_if.so gid = 511 <== Add

auth required pam_tally2.so deny=5 <== Add

auth sufficient pam_unix.so nullok try_first_pass

auth requisite pam_succeed_if.so uid >= 500 quiet

auth required pam_deny.so

The above configuration was added for security control of specific users. The processing

except cron works without problems. Only cron is affected by this configuration.

Also, if above configuration is changed like following then it works as expected without the

message in question.

Before change

auth [default=1 success=ignore] pam_succeed_if.so gid = 511

After change

auth [default=ignore success=1] pam_succeed_if.so gid != 511

Note - This solution is part of Red Hat’s fast-track publication program

############################################################

postfix doesn't start on boot with error

postfix: fatal: parameter inet_interfaces: no local interface found

postfix attempts to start on boot, but shuts down with the following error:

postfix[1516]: fatal: parameter inet_interfaces: no local interface found for  (insert ip address)

Root Cause: The network is not ready when postfix starts

Work around:

In /lib/systemd/system/postfix.service change “After=”

from:

After=syslog.target network.target

to

After=syslog.target network-online.target network.target NetworkManager-wait-online.service

Enable the new settings:

# systemctl enable NetworkManager-wait-online.service

########################################################

How to save command to history immediately after you type it in bash?

·         By default, if a user log in a system via bash, all the command history will be saved after exit.

·         How to save every command into ~/.bash_history immediately right after the user type it?

·         How to save all commands typed in a session manually before exit bash?

To save all commands typed in a session manually before exit bash, issue this command:

$ history -a

To save all typed commands immediately after you type it, add these two lines to ~/.bashrc file:

shopt -s histappend

PROMPT_COMMAND='history -a'

After making above changes, exit and log back in again.

To make it effective for all user in the system, append the following two lines into /etc/bashrc file:

shopt -s histappend

PROMPT_COMMAND='history -a'

############################################

How to clear ARP cache

The arp table or arp cache keeps track of all devices on your network that your computer is capable of communicating with. It stores the Layer 2 data (MAC addresses) as well as the interface in which the device is connected through (i.e. which interface the traffic came in on). This table can be viewed, modified, flushed using the arp command in Linux.

View Arp Cache Entries

arp -n

The output will something like the following.

Address                HWtype  HWaddress           Flags Mask            Iface

192.168.1.100          ether   00:21:a0:63:38:3f   C             eth0

To add an arp entry we simply take advantage of the options that the arp command provides. Let’s add an arbitrary entry.

arp -i eth0 -s 10.11.12.13 de:ad:be:ef:fe:ed

To remove an entry

arp -i -d 

Some example usages are given below

arp -i eth0 -d 10.11.12.13

arp -d 192.168.1.100

Clearing arp cache with verbose

ip -s -s neigh flush all

################################

Postfix - Flush the Mail Queue

Usually, you use the "sendmail -q" command to flush mail queue under Sendmail MTA.

Under Postfix MTA, just enter the following command to flush the mail queue:

# postfix flush

OR

# postfix -f

To see mail queue, enter:

# mailq

To remove all mail from the queue, enter:

# postsuper -d ALL

To remove all mails in the deferred queue, enter:

# postsuper -d ALL deferred

################################################################

File listing is not in sync for NFS Server/Client in RHEL

 

Þ      From time to time the file listing in directory on NFS server and on client are not in sync. In detail, there are files visible on the nfs-server, which doesn't exist anymore. Additionally, different files on nfs-client (in shared directory!), are seen which are not visible on the server and also does not exist anymore.

Þ      When touching this directory, by adding an empty file i.e. the file listing instantly gets in sync and this issue is gone for a while.

Cause:

In this case, noac option was tried for the mount. The noac option only controls the attribute cache and it does not have any effect on the lookup cache. This has been confirmed by doing a touch in the directory, the files are synced. In this case, NFS client checks the mtime of the directory. The reason why the “out of sync” issue happens is when client is doing a revalidation of cache and at that instant, the server side changes are also happening. Hence, if the modification happens after the client validates and at the same instant, another change happens, the NFS client may not take it up. The NFS client has a "lookup cache" which does positive and negative lookup of the directories. The lookup cache will be revalidated if the mtime changes which happens when a file is touched within the directory. By default, lookup cache includes both positive and negative lookups.

Solution:

Use the option lookupcache=pos with NFS mount. When using this option, it is recommended to remove the noac option and instead tune the actime option, else performance will come down

##############################################################

What is $* and $# in Linux?

$#           Stores the number of command‐line arguments that were passed to the shell program.

$?           Stores the exit value of the last command that was executed.

$0           Stores the first word of the entered command (the name of the shell program).

$*            Stores all the arguments that were entered on the command line ($1 $2 ...).

$@          Stores all the arguments that were entered on the command line, individually quoted ("$1" "$2" ...).

So basically, $# is a number of arguments given when your script was executed. $* is a string containing all arguments. For example, $1 is the first argument and so on…

This is useful, if you want to access a specific argument in your script.

Here is a simple example. If you run following command:

./command ‐yes ‐no /home/username

$# = 3

$* = ‐yes ‐no /home/username

$@ = array: {"‐yes", "‐no", "/home/username"}

$0 = ./command, $1 = ‐yes etc.

These are part of POSIX standard, and should be supported by all compliant shells.

For the reference, below is POSIX standard definitions for each special parameter.

Do note there's three additional variables: $-, $$ and $!

###################################################################

crontab was not working with dynamic date filename

To set a simple cronjob which will redirect the output to the log file with timestamp,

Hourly running cron entry mentioned below

0 * * * * ksh /root/test.sh > test_output_`date "+%Y-%m-%d_%H-%M"`.log

While running the same command on shell prompt, it was working fine. Execute permissions and path were proper.

But it kept on failing to create the proper log when scheduled through cron.

It worked, after the crontab entry was changed as follows, added \ before %

0 * * * * ksh /root/test.sh > test_output_`date "+\%Y-\%m-\%d_\%H-\%M"`.log

Good to note some of the date changers in Linux for finding yesterdays and tomorrows and so on:

[root@mylab ~]# date --date="1 Days Ago"

Wed Nov 18 07:50:24 UTC 2015

[root@mylab ~]# date --date=”yesterday”

Thu Nov 19 00:00:00 UTC 2015

[root@mylab ~]# date --date="tomorrow"

Fri Nov 20 07:54:26 UTC 2015

[root@mylab ~]# date --date='1 Day'

Fri Nov 20 07:55:03 UTC 2015

[root@mylab ~]# date --date='10 Day'

Sun Nov 29 07:55:22 UTC 2015

[root@mylab ~]# date --date='next Day'

Fri Nov 20 07:55:35 UTC 2015

[root@mylab ~]# date --date='-1 Day'

Wed Nov 18 07:55:56 UTC 2015

[root@mylab ~]# date --date='1 Week'

Thu Nov 26 07:56:22 UTC 2015

[root@mylab ~]# date --date='10 Week'

Thu Jan 28 07:56:27 UTC 2016

[root@mylab ~]# date --date='1 Month'

Sat Dec 19 07:56:40 UTC 2015

[root@mylab ~]# date --date='1 Year'

Sat Nov 19 07:56:55 UTC 2016

[root@mylab ~]#

#############################################################

Disaster Recovery tool in RHEL 7

In RHEL7.2GA and later: Red Hat contains REAR which can be used for image based backups.

Details -> https://rhn.redhat.com/errata/RHEA-2015-2244.html

In RHEL6 and earlier, Red Hat does not ship any network applications as such which can be used to do a complete OS disaster recovery. Third party applications and backup solutions compatible with Red Hat Enterprise Linux (RHEL) are available.

While Red Hat does not offer a direct bare-metal recovery solution you can use Red Hat Network Satellite to perform a kickstart and then use a custom channel with your own application specific RPMs to install the required applications and configuration files. You can further automate this by creating server specific kickstart profiles that include all the applications you require. Once you have set this up, the bare metal backup recovery can be done by kick starting a server profile and putting the application files back from backup. Alternatively you can keep your application data on NFS so you don't need to restore from backup at all.

###########################################################################################

Not able to detect SAN LUNs on the server.

Reboot too does not detect the LUNs from external storage.

Cause:

The root cause could be one of the following.

Þ      The LUNs are not visible in the BIOS due to improper zoning configuration.

Þ      HBA port status was seen as 'Link down' for all the fc ports on the HBA.

Þ      Re-enabling the corresponding fc ports on switch changed the HBA ports status to 'Online' after which all the LUNs from the external storage were detected.

 Tested Solution:

Þ      Check whether the LUN is visible in the BIOS. If not visible then check the zoning configuration.

Þ      Re-enable fibre channel ports that are used by this server on the fc switch.

##############################################

RHEL7: Firewalld – ZONE MANAGEMENT

Firewalld is the new user land interface in RHEL 7. It replaces the iptables interface and connects to the netfilter kernel code. It mainly improves the security rules management by allowing configuration changes without stopping the current connections.

To know if Firewalld is running, type:

# systemctl status firewalld

firewalld.service - firewalld - dynamic firewall daemon

   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)

   Active: active (running) since Tue 2014-06-17 11:14:49 CEST; 5 days ago

...

or alternatively:

# firewall-cmd --state

running

Note: If Firewalld is not running, the command displays not running.

If you’ve got several network interfaces in IPv4, you will have to activate ip forwarding.

To do that, paste the following line into the /etc/sysctl.conf file:

net.ipv4.ip_forward=1

Then, activate the configuration:

# sysctl -p

Although Firewalld is the RHEL 7 way to deal with firewalls and provides many improvements, iptables can still be used (but both shouldn’t run at the same time).

You can also look at the iptables rules created by Firewalld with the iptables-save command.

Zone Management

Also, a new concept of zone appears: all network interfaces can be located in the same default zone or divided into different ones according to the levels of trust defined. In the latter case, this allows to restrict traffic based on origin zone.

Note: Without any configuration, everything is done by default in the public zone. If you’ve got more than one network interface or use sources, you will be able to restrict traffic between zones.

To get the default zone, type:

# firewall-cmd --get-default-zone

public

To get the list of zones where you’ve got network interfaces or sources assigned to, type:

# firewall-cmd --get-active-zones

public

interfaces: eth0

To get the list of all the available zones, type:

# firewall-cmd --get-zones

block dmz drop external home internal public trusted work

To change the default zone to home permanently, type:

# firewall-cmd --set-default-zone=home

success

Note: This information is stored in the /etc/firewalld/firewalld.conf file.

Network interfaces can be assigned to a zone in a temporary (until the next reboot or reload) or in a permanent way. Either way, you don’t need to reload the firewall configuration.

To assign the eth0 network interface temporarily to the internal zone, type:

# firewall-cmd --zone=internal --change-interface=eth0

success

To assign the eth0 network interface permanently to the internal zone (a file called internal.xml is created in the /etc/firewalld/zones directory), type:

# firewall-cmd --permanent --zone=internal --change-interface=eth0

success

Note: This operation can also be done with the nmcli con mod command

To know which zone is associated with the eth0 interface, type:

# firewall-cmd --get-zone-of-interface=eth0

internal

To get the current configuration of the public zone, type:

# firewall-cmd --zone=public --list-all

 public (default, active)

 interfaces: eth0

 sources:

 services: dhcpv6-client ssh

ports:

 masquerade: no

 forward-ports:

 icmp-blocks:

 rich rules:

Note: The previous command displays the current configuration, ie the permanent settings and the temporary ones. To only get the permanent settings, use the –permanent option.

It is also possible to create new zones. To create a new zone (here test), type:

# firewall-cmd --permanent --new-zone=test

success

# firewall-cmd --reload

success

Note: Only permanent zones can be created.

#########################################################################

RHEL 7 - "WRITE SAME failed. Manually zeroing"

Each time when the server is rebooted, the following messages are recorded  in /var/log/messages.

Mar 19 07:53:55 serverX kernel: dm-4: WRITE SAME failed. Manually zeroing.

Mar 19 09:58:36 serverX kernel: dm-4: WRITE SAME failed. Manually zeroing.

Mar 19 10:00:52 serverX kernel: dm-4: WRITE SAME failed. Manually zeroing.

The WRITE SAME SCSI command implements a feature that allows to write multiple blocks with the same content. Red Hat Enterprise Linux use it mostly for ensuring blocks contain all zero’s.

The functionality defaults to use 0xFFFF (65535) blocks for WRITE SAME, except for certain classes of disks which are known not to implement the feature.

If the device supports WRITE SAME, use that to optimize zeroing of blocks. If the device does not support WRITE SAME or if the operation fails, fall back to writing zeroes the old-fashioned way.

To disable WRITE SAME on devices that do not support WRITE SAME and to get rid of the message, you have to set

max_write_same_block for the specific device in /sys to 0

As these settings will vanish during reboot, implementing tmpfiles.d(5) rules will help to make the change persistent during system reboots

$ cat /etc/tmpfiles.d/write_same.conf

w /sys/devices/pci0000:00/0000:00:02.0/0000:03:00.0/host0/target0:2:0/0:2:0:0/scsi_disk/0:2:0:0/max_write_same_blocks  -   -   - 0

###########################################################################

"tur checker reports path is down"

May 16 21:17:36 hostA multipathd: sde: tur checker reports path is down

May 16 21:17:36 hostA multipathd: sdf: tur checker reports path is down

 May 16 21:17:36 hostA multipathd: sdg: tur checker reports path is down

May 16 21:17:36 hostA multipathd: sdh: tur checker reports path is down

This is due to temporary issues, path from the storage was seen failed/faulty. When the issue was resolved it seems that the working path was not taken into consideration and still the wrong status of the path was seen.

You have to remove the device path which is down and these messages will stop.

For Example:

1. Check which device is down by multipath command

# multipath -ll

sda: checker msg is "tur checker reports path is down"

mpath0 (16465616462656166313a3100000000000000000000000000) dm-2 IET,VIRTUAL-DISK

[size=1020M][features=0][hwhandler=0]

\_ round-robin 0 [prio=0][enabled]

\_ 1:0:0:1 sda 8:0   [failed][faulty]    <===== (1)

\_ round-robin 0 [prio=0][enabled]

\_ 2:0:0:1 sdb 8:16  [active][ready]

2. Remove the device which is down

# echo "scsi remove-single-device 1 0 0 1" > /proc/scsi/scsi

3. Verify the device is removed

# multipath -ll

mpath0 (16465616462656166313a3100000000000000000000000000) dm-2 IET,VIRTUAL-DISK

[size=1020M][features=0][hwhandler=0]

\_ round-robin 0 [prio=0][active]

\_ 2:0:0:1 sdb 8:16  [active][ready]

If the path's "failed | faulty" status is temporary, rescanning the scsi bus with rescan-scsi-bus.sh command should bring the path / paths back in working condition.

Note - In the above case with two paths to the storage, the storage is active / active storage. Hence there would be no impact of the LUN as the other active path / paths will be available.

Additionally, you can check with fdisk –l ; if the device is present, if it's not then blacklist it in multipath.conf

############################################################################

Red Hat Enterprise Linux reports lun has a LUN larger than allowed by the host adapter

Errors

1 The following scsi errors are reported; where is greater than lpfc module parameter lpfc_max_luns.

kernel: scsi: host X channel Y id Z lun has a LUN larger than allowed by the host adapter

This issue can be also seen if you have a LUN ID that is greater than 255 even though you have a much fewer number of LUNs than 255.

2 After HBA rescan on physical server we are unable to add new LUN. We see this error in the system logs:

scsi: host 0 channel 0 id 0 lun557 has a LUN larger than allowed by the host adapter

scsi: host 1 channel 0 id 0 lun557 has a LUN larger than allowed by the host adapter

3 Unable to install RHEL 6.1 on LUNs exported from SAN when LUN id value is less than FC driver's supported maximum lun id value.

ERR kernel:lpfc 0000:0f:00.0: 1:1303 Link Up Event x1 received Data: x1 xf7 x20 x0 x0 x0 0

NOTICE kernel:scsi 3:0:0:0: RAID              EMC      SYMMETRIX        5874 PQ: 0 ANSI: 4

WARNING kernel:scsi: host 3 channel 0 id 0 lun16384 has a LUN larger than allowed by the host adapter

Probable Cause:

Storage has returned a LUN id value that exceeds the current maximum LUN id value supported by the driver. Either

the driver's maximum supported LUN id value needs to be increased, as in the case of lpfc, or

the LUN id value provided by storage must be changed via reconfiguration within the storage box.

$ cat /sys/class/scsi_host/hostX/lpfc_max_luns 255

The default maximum LUNs for lpfc is 256 (starting at 0) LUNs. Having a LUN number larger than default (255) or lpfc_max_luns will generate error.

For Qlogic HBA cards, you can find the value  in /var/log/dmesg

##############################################################

RHEL7: How to configure I/O schedulers.

I/O schedulers are used to optimize reads/writes on disk.

There are three types of I/O schedulers (also called I/O elevators) in RHEL 7:

·         CFQ (Completely Fair Queuing) promotes I/O coming from real time processes and uses historical data to anticipate whether an application will issue more I/O requests in the near future (causing a slight tendency to idle).

·         Deadline attempts to provide a guaranteed latency for requests and is particularly suitable when read operations occur more often than write operations (one queue for reads and one for writes, I/O’s are dispatched based on time spent in queue).

·         Noop implements a simple FIFO (first-in first-out) scheduling algorithm with minimal CPU cost.

With RHEL 7, the default I/O Scheduler is now CFQ for SATA drives and Deadline for everything else.

This is because Deadline outperforms CFQ for faster storage than SATA drives.

Configuration at boot

To define a global I/O scheduler (here cfq) at boot, type:

# grubby --update-kernel=ALL --args="elevator=cfq"

Configuration for a particular disk

To get the current configuration of a disk (here /dev/sda), type:

# more /sys/block/sda/queue/scheduler

 noop deadline [cfq]

To assign an I/O scheduler (here deadline) to a particular disk (here /dev/sda), type:

# echo deadline > /sys/block/sda/queue/scheduler

Note: This could be set permanently through the rc-local service.

To check the new configuration, type:

# more /sys/block/sda/queue/scheduler

   noop [deadline] cfq

#######################################################################

NTP: No association ID's returned

[root@station1 ~]# ntpq -p

No association ID's returned

Probable cause:

SELinux was preventing access to /etc/ntp.conf.

Solution:

Disable SELinux.

(or)

Restore selinux context for the /etc/ntp.conf file :-

[root@station1 ~]# ls -lZ /etc/ntp.conf

-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 /etc/ntp.conf

was changed to

[root@station1 ~]# restorecon -v /etc/ntp.conf

restorecon reset /etc/ntp.conf context unconfined_u:object_r:admin_home_t:s0->unconfined_u:object_r:net_conf_t:s0~~~

##############################################################

Remove virbr0 interface in Linux

virbr0 is an Ethernet bridge.

The virbr0 bridge interface is created by libvirtd's default network configuration. libvirtd is the service which provides a basis for the host to act as a hypervisor.

root@labserver:~ # ifconfig virbr0

virbr0 Link encap:Ethernet HWaddr 00:00:00:00:00:00

inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0

inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:0 errors:0 dropped:0 overruns:0 frame:0

TX packets:41 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:0 (0.0 b) TX bytes:7630 (7.4 KiB)

You use brctl to manage the interface.

brctl is used to set up, maintain, and inspect the Ethernet bridge configuration in the Linux kernel

root@labserver:~ # brctl show

bridge name bridge id STP enabled interfaces

virbr0 8000.000000000000 yes

Verify if libvirtd is running

root@labserver:~ # service libvirtd status

libvirtd (pid 12529) is running…

Verify if there is any host running. In my case there isn’t

root@labserver:~ # virsh list

Id Name State

———————————-

This is the default network set-up for the virtual machines

root@labserver:~ # virsh net-list

Name State Autostart

—————————————–

default active yes

One could prevent libvirtd's default network from being activated on boot, or you could prevent libvirtd itself from activating on boot. The former will prevent any VM guest attached to libvirtd's default network from having network connectivity and the latter would prevent VMs from running at all.

To immediately stop libvirtd's default network (this will not persist across reboots):

           # virsh net-destroy default

To permanently disable the libvirtd default network from being created at boot:

           # virsh net-autostart default --disable

To permanently remove the libvirtd default network:

           # virsh net-undefine default

To permanently disable the libvirtd service from starting at boot on RHEL5 and RHEL6:

# chkconfig libvirtd off

To permanently disable the libvirtd service from starting at boot on RHEL7:

           # systemctl disable libvirtd.service

###########################################################

SSH connections are closed suddenly in Red Hat Enterprise Linux 6

·         SSH connections are refusing with error "fatal: mm_request_send: write: Broken pipe"

·         SSH Client exits with the message "Connection closed by XXX.XXX.XXX.XXX" as soon as it starts.

Probable Cause:

·         It's possible that the sshd server is exposed by malware.

·         We found that malware udev-fall was killing sshd child processes.

·         Check if the process, which name is "udev-fall", is running on the sshd server.

$ cat ps |grep udev-fall

root     14289  0.7  0.0   1664  1192 ?        S    Nov20 143:58 /usr/bin/udev-fall -d

For Solution:

1.    Stop the udev-fall service.

2.    Disable the service with chkconfig

3.    Remove /usr/bin/udev-fall and the initscript, /etc/init.d/udev-fall.

Red Hat never shipped this init script in any product. So, it's needed to investigate how it's introduced in each customer's environment.

#################################################################

tail: cannot watch `/var/log/messages': No space left on device

When performing 'tail -f filename', error shows as follows

# tail -f /var/log/messages

tail: cannot watch `/var/log/messages': No space left on device

Increase /proc/sys/fs/inotify/max_user_watches (default: 8192) with doubled value.

# echo "fs.inotify.max_user_watches = 16354" >> /etc/sysctl.conf

# sysctl –p

#################################################################

XFS is now the RHEL7 default file system.

XFS brings the following benefits:

·         ability to manage up to 500TB file systems with files up to 50TB in size,

·         best performance for most workloads (especially with high speed storage and larger number of cores),

·         less CPU intensive than most of the other file systems (better optimizations around lock contention, etc),

·         the most robust at large scale (has been run at hundred plus TB sizes for many years),

·         the most common file system in multiple key upstream communities: most common base for ceph, gluster and openstack more broadly,

·         pioneered most of the techniques now in Ext4 for performance (like delayed allocation),

·         no file system check at boot time,

·         CRC checksum on all metadata blocks.

·         XFS was already fully supported in RHEL6 but, as it was not the default file system, perhaps are you not used to it. It’s time to rectify this.

XFS Basic Management

To create a new logical volume called lv_vol with a size of 100MB in the vg volume group, type:

# lvcreate --size 100M --name lv_vol /dev/vg

To create a new XFS file system, type:

# mkfs.xfs /dev/vg/lv_vol

To mount the new file system under /mnt, type:

# mount /dev/vg/lv_vol /mnt

To increase the file system size by 50MB, type:

# lvextend --size +50M /dev/vg/lv_vol

# xfs_growfs /mnt

Note 1: This is only possible for a mounted file system.

Note 2: You can’t reduce a XFS file system even though you unmount it. You have to back it up, drop it and recreate it.

XFS Advanced Management

If a problem occurs and you want to repair the file system, according to the man pages you are supposed to mount and unmount it before typing:

# xfs_repair /dev/vg/lv_vol

Note : Try the “-L” option (“force log zeroing“) to clear the log if nothing else works.

To assign a label (up to 12 characters) to the file system, type:

# umount /dev/vg/lv_vol

# xfs_admin -L "Label" /dev/vg/lv_vol

To read the label (the file system can be mounted or not but you need to specify the partition name), type:

# xfs_admin -l /dev/vg/lv_vol

XFS Backup Management

To do a full backup of the file system and put it into the /root/dump.xfs file without specifying any dump label, type:

# xfsdump -F -f /root/dump.xfs /mnt

Note 1: You can specify the mounting point or the partition name.

Note 2: It is only possible for a mounted file system.

Note 3: You can run an incremental dump by using the “-l” option with a number between 0 and 9 (0=full dump).

If you want to specify a session label and a media label, type:

# xfsdump -L session_label -M media_label -f /root/dump.xfs /dev/vg/lv_vol

To restore the file system, type:

# xfsrestore -f /root/dump.xfs /mnt

To get the list of all the available dumps, type:

# xfsrestore -I

To defragment the file system (operation normally not needed), type:

# xfs_fsr /dev/vg/lv_vol

Note : Both mount point and partition name are valid.

To freeze the file system before taking a snapshot, type:

# xfs_freeze -f /mnt

Note : You have to specify the mount point, the partition name is not allowed.

To unfreeze the file system, type:

# xfs_freeze -u /mnt

To copy the contents of a file system (here mounted under /mnt) to another directory, type:

# xfsdump -J - /mnt | xfsrestore -J - /new

Note : The “-J” option avoids any write into the inventory

###################################

RHEL7: Check if a system is vulnerable to a CVE.

CVE stands for Common Vulnerabilities and Exposure. It’s a dictionary of publicly known information security vulnerabilities and exposures.

CVE’s common identifiers enable data exchange between security products and provide a baseline index point for evaluating coverage of tools and services.

To check whether a RHEL 7 system is vulnerable or not to a CVE, first install the following yum plugin:

# yum install yum-plugin-security

Then, check whether the vulnerability is present (here openssl security update):

# yum updateinfo info --cve CVE-2014-0224

==============================================

Important: openssl security update

===============================================

Update ID : RHSA-2014:0679

Release :

 Type : security

Status : final

Issued : 2014-06-10 00:00:00

Bugs : 1087195 - CVE-2010-5298 openssl: freelist misuse causing

        a possible use-after-free

: 1093837 - CVE-2014-0198 openssl: SSL_MODE_RELEASE_BUFFERS NULL

   pointer dereference in do_ssl3_write()

: 1103586 - CVE-2014-0224 openssl: SSL/TLS MITM vulnerability

: 1103593 - CVE-2014-0221 openssl: DoS when sending invalid DTLS

   handshake

: 1103598 - CVE-2014-0195 openssl: Buffer overflow via DTLS

   invalid fragment

: 1103600 - CVE-2014-3470 openssl: client-side denial of service

   when using anonymous ECDH

CVEs : CVE-2014-0224

: CVE-2014-0221

: CVE-2014-0198

: CVE-2014-0195

: CVE-2010-5298

: CVE-2014-3470

Description : OpenSSL is a toolkit that implements the Secure

Sockets Layer

Note: In the case of a non-vulnerable system, nothing is displayed.

At any time, you can check a particular CVE to get more information:

https://access.redhat.com/security/security-updates/#/cve

All CVEs are available at the Red Hat CVE page.

##############################################################

AD users able to ssh in but AD user not able to sudo

=============var/log/sssd/sssd_pam.log================

Still see the same errors :

[sssd[pam]] [sbus_remove_timeout] (0x2000): 0x8cdd30

[sssd[pam]] [sbus_dispatch] (0x4000): dbus conn: 0x8d1e90

[sssd[pam]] [sbus_dispatch] (0x4000): Dispatching.

[sssd[pam]] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 1 errno: 11 error message: Offline

[sssd[pam]] [pam_check_user_dp_callback] (0x0040): Unable to get information from Data Provider Error: 1, 11, Offline

==============var/log/sssd/sssd_example.com.log===========

[acctinfo_callback] (0x0100): Request processed. Returned 1,11,Offline

[sbus_dispatch] (0x4000): dbus conn: 0x20655c0

[sbus_dispatch] (0x4000): Dispatching.

To resolve this error, set the DNS discovery domain in the sssd.conf

[domain/example.com] section

dns_discovery_domain = example.com

Restart sssd service afterwards.

service sssd restart

#########################################################

How to exclude Data Protector (omni) info messages from /var/log/messages?

/var/log/messages file is flooded with below Data Protector (omni) messages.

Oct 17 07:14:34 hostname1 xinetd[5508]: EXIT: omni status=0 pid=1362 duration=869(sec)

Oct 17 07:15:07 hostname1 xinetd[5508]: START: omni pid=1631 from=::ffff:172.26.152.220

Oct 17 03:58:59 hostname1 xinetd[5508]: EXIT: omni status=1 pid=21514 duration=1732(sec)

Oct 17 04:00:09 hostname1 xinetd[5508]: START: omni pid=22009 from=::ffff:172.26.162.103

Comment out the "log_on_failure" and "log_on_success" lines in the /etc/xined.d/ and

restart the xinetd service and check whether still the issue persists.

log_on_success — Configures xinetd to log if the connection is successful. By

default, the remote host's IP address and the process ID of server processing

the request are recorded.

log_on_failure — Configures xinetd to log if there is a connection failure or

if the connection is not allowed.