« Posts tagged rhel6

Restoring /usr/bin with yum after accidental deletion

I was recently writing a Makefile for cramfs, specifically the distclean and install sections. The installation would copy the program binaries to /usr/bin while the cleanup would remove them… simple enough right?

I wrote a for loop to go through $(PROGS) and remove them from $(INSTLOC):

INSTLOC = /usr/bin
PROGS = mkcramfs cramfsck

all: $(PROGS)

distclean clean:
    for p in $(PROGS);\
    do\
        rm -rf $$p $(INSTLOC)/$$p;\
    done

install:
    cp $(PROGS) $(INSTLOC)

The problem was that I ran this as root (tsk tsk), and since Makefile requires that for loop variables be escaped (line 9: $$p not $p), the rm command translated to this:

rm -rf  /usr/bin/

Great! So now I had no binaries in /usr/bin, which includes: yum, bash, crontab, python, perl… (800+ in total on a minimal install).

Since I only deleted the binaries, the programs were still listed as installed in the RPM database. The first thing I had to do was reinstall yum and it’s “usrbin” dependency python:

[root@demon ~]# mount /dev/sr0 /media/cdrom
[root@demon ~]# cd /media/cdrom/Packages
[root@demon ~]# rpm -Uvh --force python-2.6.5-3.el6.i686.rpm
[root@demon ~]# rpm -Uvh --force yum-3.2.27-14.el6.noarch.rpm

The next step was to figure out which packages had binaries in /usr/bin so I can reinstall them:

[root@demon ~]# rpm -qf $(rpm -qla|grep ^/usr/bin)|uniq|sort

… and finally send those to yum to do a reinstall and get the binaries back:

[root@demon ~]# yum reinstall $(rpm -qf $(rpm -qla|grep ^/usr/bin)|uniq|sort)
[root@demon ~]# ls -la /usr/bin|wc -l
848
[root@demon ~]#

… crisis averted! Snapshot time.

One last note: If you manually installed third party RPMs (not listed in the /etc/yum.repos.d repositories), they will not be reinstalled. You can perform reinstall these one by one using the rpm -Uvh command above. Keep in mind that if these RPMs have not undergone proper QA they may overwrite your current configuration files

You can run these RPMs through rpmlint to see if they produce any warnings or errors that may cause a problem when reinstalling:

[root@demon ~]# rpmlint -iv iplog-2.2.1-1_RH7.i386.rpm
...
iplog.i386: W: conffile-without-noreplace-flag /etc/iplog.conf
A configuration file is stored in your package without the noreplace flag. A
way to resolve this is to put the following in your SPEC file:
%config(noreplace) /etc/your_config_file_here
...
[root@demon ~]#

Goodluck!

Identify and block malicious HTTP traffic with IPtables

So I was looking through my webservers’ access_log files and this popped up every couple of days:

93.157.0.142 - - [14/Dec/2010:16:01:19 -0500] "GET /w00tw00t.at.blackhats.romanian.anti-sec:) HTTP/1.1" 404 13
72.167.164.72 - - [17/Dec/2010:02:02:54 -0500] "GET /w00tw00t.at.blackhats.romanian.anti-sec:) HTTP/1.1" 404 13
74.55.205.98 - - [18/Dec/2010:03:06:49 -0500] "GET /w00tw00t.at.blackhats.romanian.anti-sec:) HTTP/1.1" 404 13
150.217.19.5 - - [19/Dec/2010:14:36:52 -0500] "GET /w00tw00t.at.blackhats.romanian.anti-sec:) HTTP/1.1" 404 13
173.201.39.105 - - [21/Dec/2010:08:16:35 -0500] "GET /w00tw00t.at.blackhats.romanian.anti-sec:) HTTP/1.1" 404 13
74.55.205.98 - - [24/Dec/2010:14:43:28 -0500] "GET /w00tw00t.at.blackhats.romanian.anti-sec:) HTTP/1.1" 404 13

This is a truncated list, but each one of these “romanian blackhats” would attempt a few other directories as well. These are not really critical intrusion attempts but they do indicate drones that scan the Internet for potential security holes in webservers (read Phil’s Getting A Little Sick of ZmEu). I don’t want these hosts to access my server in any way since, well, they don’t really need to. I could’ve blocked each one of those IPs by hand but I decided to script it and crontab it.

The first thing I needed is a chain that would handle all of these bad IP addresses:

[root@demon ~]# iptables -N bad_traffic
[root@demon ~]# iptables -A INPUT -j bad_traffic
[root@demon ~]# iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT

The two rules should be applied in the order specified above. You want to DROP bad traffic before you ACCEPT any web connection.

This script will add a rule for each IP with the DROP target in the bad_traffic chain, if it is not already in the chain:

#!/usr/bin/env perl
# badht - Bad HTTP Traffic blocker
#
# Scans an Apache access log file for bad
# requests and blocks the IP responsible
#
# Usage: badht <access_log> [iptables_chain]
#
# ./badht /var/log/httpd/access_log bad_traffic
#
# badht will use the chain 'bad_traffic' unless
# otherwise specified

use strict;
use warnings;
use POSIX qw(strftime);

die("Usage: $0 </var/log/httpd/access_log> [iptables_chain]") if !$ARGV[0];
my $log = $ARGV[0];

my $chain = ($ARGV[1] ? $ARGV[1] : "bad_traffic");

my @bad = `grep w00tw00t $log|cut -f1 -d" "|sort -u`;
my @ablk = `/sbin/iptables -S $chain|grep DROP|awk '{print \$4}'|cut -d"/" -f1`;

foreach my $ip (@bad) {
    if (!grep $_ eq $ip, @ablk) {
        chomp $ip;
        `/sbin/iptables -A $chain -s $ip -j DROP`;
        print strftime("%b %d %T",localtime(time))." badht: blocked bad HTTP traffic from: $ip\n";
    }
}

By the way, it’s a good idea to block ALL incoming traffic (line 29) coming from these IP addresses because chances are they have already attempted to brute-force your SSH service:

[root@demon admin]# grep -E "sshd.*Failed password for.*from ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)" /var/log/secure|wc -l
103
[root@demon admin]#

… within just 7 days of bringing demon.* online! These packets are just wasted CPU cycles from compromised hosts and they should be dropped before they get to any of my services.

Anyway… when I execute badht I get this output:

[root@demon admin]# ./badht /var/log/httpd/access_log bad_traffic
Dec 25 15:56:44 badht: blocked bad HTTP traffic from: 150.217.19.5
Dec 25 15:56:44 badht: blocked bad HTTP traffic from: 173.201.39.105
Dec 25 15:56:44 badht: blocked bad HTTP traffic from: 72.167.164.72
Dec 25 15:56:44 badht: blocked bad HTTP traffic from: 74.55.205.98
Dec 25 15:56:44 badht: blocked bad HTTP traffic from: 93.157.0.142
[root@demon admin]# ./badht /var/log/httpd/access_log bad_traffic
[root@demon admin]# iptables -L bad_traffic -n
Chain bad_traffic (1 references)
target     prot opt source               destination
DROP       all  --  150.217.19.5         0.0.0.0/0
DROP       all  --  173.201.39.105       0.0.0.0/0
DROP       all  --  72.167.164.72        0.0.0.0/0
DROP       all  --  74.55.205.98         0.0.0.0/0
DROP       all  --  93.157.0.142         0.0.0.0/0
[root@demon admin]#

As you can see the second time I ran the script it skipped the already-blocked IPs and said nothing.

I don’t want to run this manually, so I’ll let crontab handle it:

[root@demon ~]# crontab -lu root
*/30 * * * * ~/admin/badht /var/log/httpd/access_log bad_traffic >> /var/log/bad_traffic 2>&1
[root@demon ~]#

… this will run twice an hour and send all output to /var/log/bad_traffic. You can increase the frequency but you should keep in mind that this may needlessly slow the system down on large access_log files.

Note: The rules created by badht are temporary and will be lost on system reboot or when the iptables ‘service’ is restarted. Remember to periodically save the iptables rules, or at least the ‘bad_traffic’ chain. Since the crontab is persistant, badht will recreate all the rules the next time it runs.

SETroubleshoot mail notification on SELinux denial

I’ve recently installed setroubleshoot-server on my RHEL6 server to help diagnose various SELinux denials as I attempt to secure the box.

SETroubleshoot also has an email notification system that is really easy to implement. There are a couple of things that you should consider before going forward.

Add the recipient email addresses to /var/lib/setroubleshoot/email_alert_recipients:

admin@example.com       filter_type=after_first

Note: the ‘after_first’ filter will prevent setroubleshoot from flooding your inbox with the same alert. There are other filter types, see the man page.

…and finally modify the [email] section in /etc/setroubleshoot/setroubleshoot.cfg:

[email]
# recipients_filepath: Path name of file with email recipients. One address
# per line, optionally followed by enable flag. Comment character is #.
recipients_filepath = /var/lib/setroubleshoot/email_alert_recipients

# smtp_port: The SMTP server port
smtp_port = 2525

# smtp_host: The SMTP server address
smtp_host = mail.example.com

# from_address: The From: email header
from_address = security@demon.local

In my case, my MTA is listening on port 2525 as well as port 25, due to most ISPs blocking 25. The RHEL6 server is behind such an ISP and I had to use this as the mail port.

By default, SELinux allows only a short list of ports to be used by the SMTP protocol, and when setroubleshoot tried to send the alert, I saw this in /var/log/messages:

Dec 14 16:41:58 demon setroubleshoot: [avc.ERROR] Plugin Exception httpd_bad_labels #012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/setroubleshoot/analyze.py", line 156, in analyze_avc#012    report_receiver.report_problem(report)#012  File "/usr/lib/python2.6/site-packages/setroubleshoot/server.py", line 195, in report_problem#012    email_alert(siginfo, to_addrs)#012  File "/usr/lib/python2.6/site-packages/setroubleshoot/email_alert.py", line 77, in email_alert#012    smtp = smtplib.SMTP(smtp_host, smtp_port)#012  File "/usr/lib/python2.6/smtplib.py", line 239, in __init__#012    (code, msg) = self.connect(host, port)#012  File "/usr/lib/python2.6/smtplib.py", line 295, in connect#012    self.sock = self._get_socket(host, port, self.timeout)#012  File "/usr/lib/python2.6/smtplib.py", line 273, in _get_socket#012    return socket.create_connection((port, host), timeout)#012  File "/usr/lib/python2.6/socket.py", line 514, in create_connection#012    raise error, msg#012error: [Errno 13] Permission denied

…which basically means that the email_alert.py setroubleshoot script could not create an SMTP connection to my mail server on the port specified.

On RHEL6, these are the allowed SMTP ports:

[root@demon ~]# semanage port -l|grep smtp
smtp_port_t                    tcp      25, 465, 587

In order to allow demon.* to send mail to the remote MTA, I had to:

[root@demon ~]# semanage port -a -t smtp_port_t -p tcp 2525
[root@demon ~]# semanage port -l|grep smtp
smtp_port_t                    tcp      2525, 25, 465, 587

And that’s it! You can quickly test by generating an SELinux denial, and see if you get an email.

In my case, the remote MTA (running Exim) was dropping the messages and setroubleshoot would throw this in /var/log/messages:

Dec 17 09:53:19 demon setroubleshoot: [email.ERROR] email failed: {'admin@example.com': (550, 'Verification failed for <security@demon.local>\nThe mail server could not deliver mail to security@demon.local.  The account or domain may not exist, they may be blacklisted, or missing the proper dns entries.\nSender verify failed')}

This was due to Exim+SpamAssassin performing callbacks or callouts to ensure that the From: email address is valid on the mail server it comes from.

I got around this by adding the RHEL6 server’s IP block as a trusted ‘mail provider’. In /etc/mailproviders/ on the Exim server, I created the following tree:

root@exim [/etc/mailproviders]# tree
|-- rim
|   `-- ips
`-- demon
    `-- ips

2 directories, 2 files
root@exim [/etc/mailproviders]# cat demon/ips
172.16.1.0/24
root@exim [/etc/mailproviders]#

The ips files contain a list of IP blocks for Exim to trust as ‘mail providers’ and add to the whitelist. This is probably not the safest solution but it is the quickest.

Warning: if you don’t trust the entire IP block you can open your MTA to unchallenged spam. Use this method with caution.

SELinux’s setroubleshoot install on a RHEL6 server

I am planning on using RHEL6 as a web server, primarily for my Mercurial/GIT repositories. This was to replace my current Fedora13 instance. After the initial minimal install, there were a couple of things I’ve wanted but were not setup. Mainly setroubleshoot and mail notification on AVC denial.

During my F13 repository setup, I had to turn on a few SELinux booleans in order for HG to successfully serve my repositories. Apache was spitting out forbidden errors, and I suspected SELinux as the culprit. This was to be expected, however, unlike the F13 box there were no setroubleshoot messages in /var/log/messages. You know.. the ones with the friendly ‘sealert -l [hash]‘ and whatnot.

Everything was going to /var/log/audit/audit.log and written in a slightly less readable format. After going through Dan Walsh‘s blog, I’ve noticed I was missing the setroubleshoot-* packages. In a server environment (that is, no desktop) I only need to install setroubleshoot-server (and its deps) in order to get the cool descriptive SELinux audit messages.

Sample /var/log/audit/audit.log AVC denials:

[root@demon ~]# grep AVC /var/log/audit/audit.log
...
/var/log/audit/audit.log:type=AVC msg=audit(1292588343.092:3941): avc:  denied  { getattr } for  pid=2295 comm="httpd" path="/home/hg" dev=dm-3 ino=130823 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
/var/log/audit/audit.log:type=AVC msg=audit(1292588361.410:3942): avc:  denied  { search } for  pid=4945 comm="httpd" name="hg" dev=dm-3 ino=130823 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
/var/log/audit/audit.log:type=AVC msg=audit(1292588361.410:3943): avc:  denied  { getattr } for  pid=4945 comm="httpd" path="/home/hg" dev=dm-3 ino=130823 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
...

So I went ahead and installed the setroubleshoot-server RPMs, but I was still not getting anything friendly in /var/log/messages.

By the way, if you want to generate some SELinux denials you can try this command:

[root@demon ~]# sandbox /usr/bin/perl -e '`cat /dev/urandom`'
cat: /dev/urandom: Permission denied

The sandbox tool will run a binary in a paranoid domain, restricting it from accessing most objects in the system. Sandboxing is very cool and you should read up on it, especially if you’re running a web applications (hgweb in my case).

Starting with the Fedora 11 release date, Dan Walsh made a few changes to setroubleshoot to make it less of a memory hog. This meant that setroubleshootd was obsolete and replaced by sedispatch. This new binary was to be called by /sbin/audispd, which is called by auditd as the dispatcher for AVC messages. SEDispatch would only start setroubleshootd if it was needed. In fact, if you try to run setroubleshootd manually, it will start, wait for about 10 seconds and exit with code 0.

To make sure sedispatch is functional, you can do something like this:

[root@demon ~]# grep AVC /var/log/audit/audit.log | sedispatch
...
Got Reply: AVC
Got Reply: AVC
...
[root@demon ~]#

You should now see the setroubleshoot messages in /var/log/messages.

It turns out all I had to do to get setroubleshoot to work was to restart the auditd service to make sure it picked up the newly installed /etc/audisp/plugins.d/sedispatch.conf plugin.

Besides the newbie-friendly sealert database, setroubleshoot can also send email notifications when denials happen. This is a fairly straightforward process, however I did run into a couple of issues. Dan Walsh and the guys in #selinux@freenode were nice enough to help me get it working.