Admin Life Errors, Fixes, and Encounters

14Dec/101

Rate limit e-mail providers

If you're like me and need to manage large amounts of emails, one thing that must be done is limiting the speed email providers like yahoo.com, hotmail.com, live.com, gmail.com, aol.com get your email.

I don't go into much detail here but this is the setup I've researched and currently use.

First we edit main.cf and go to line 561 (i just like to keep things together)

vi /etc/postfix/main.cf

Add the following lines into main.cf

transport_maps = hash:/etc/postfix/transport

smtpslow_destination_concurrency_limit = 2
smtpslow_destination_recipient_limit = 15
smtpslow_destination_rate_delay = 2s
smtpslow_destination_concurrency_failed_cohort_limit = 2

Edit the transport file and add the domains we'd like to slow sending with.

vi /etc/postfix/transport

yahoo.com smtpslow:
gmail.com smtpslow:
hotmail.com smtpslow:
aol.com smtpslow:
comcast.com smtpslow:
live.com smtpslow:
msn.com smtpslow:
sbcglobal.net smtpslow:
verizon.net smtpslow:
bellsouth.net smtpslow:
yahoo.ca smtpslow:
cox.net smtpslow:
ymail.com smtpslow:

Go to the postfix directory and postmap(hash) the transport file

cd /etc/postfix && /usr/local/sbin/postmap transport

Now we need to change the following in master.cf.

vi /etc/postfix/master.cf

Make sure you have space or tabs before -o as it will not work if you don't.

smtpslow unix - - n - - smtp
-o smtp_helo_timeout=5
-o smtp_connect_timeout=5

Restart postfix and we're done.

/usr/local/sbin/postfix reload

19Oct/101

setup jails on freebsd

we'll be creating 4 jails + main host (5 machines in total)

First, lets create some folders where the jails will be stored.

mkdir /usr/jails && mkdir /usr/jails/mail1 && mkdir /usr/jails/mail2 && mkdir /usr/jails/mail3 && mkdir /usr/jails/mail4

then we add some settings to rc.conf

# vi /etc/rc.conf

Add these line right after (remember, every machine has a differnt network card reference (bge0)). Change it to your machines network card.

ifconfig_bge0="inet 192.168.1.157 netmask 255.255.255.0"

ifconfig_bge0_alias0="inet 192.168.1.211 netmask 0xfffffff8"
ifconfig_bge0_alias1="inet 192.168.1.212 netmask 0xfffffff8"
ifconfig_bge0_alias2="inet 192.168.1.213 netmask 0xfffffff8"
ifconfig_bge0_alias3="inet 192.168.1.214 netmask 0xfffffff8"

if you don't want to reboot, you'll also have to setup the alias now so you can ssh in later...
ifconfig bge0 inet alias 192.168.1.211 netmask 255.255.255.255
ifconfig bge0 inet alias 192.168.1.212 netmask 255.255.255.255
ifconfig bge0 inet alias 192.168.1.213 netmask 255.255.255.255
ifconfig bge0 inet alias 192.168.1.214 netmask 255.255.255.255

near the bottom of the file, add following and makre sure you change the IP's & paths if different than above.

#only have 5 IP's. 1 for the main server and 4 for the jails
#jail_list="mail1 mail2 mail3 mail4"

jail_enable="YES"
jail_list="mail1 mail2 mail3 mail4"
jail_set_hostname_allow="YES"

jail_mail1_rootdir="/usr/jails/mail1"
jail_mail1_hostname="mail1.example.com"
jail_mail1_ip="192.168.1.231"
jail_mail1_start="/bin/sh /etc/rc"
jail_mail1_devfs_enable="YES"
jail_mail1_devfs_ruleset="devfsrules_jail"

jail_mail2_rootdir="/usr/jails/mail2"
jail_mail2_hostname="mail2.example.com"
jail_mail2_ip="192.168.1.232"
jail_mail2_start="/bin/sh /etc/rc"
jail_mail2_devfs_enable="YES"
jail_mail2_devfs_ruleset="devfsrules_jail"

jail_mail3_rootdir="/usr/jails/mail3"
jail_mail3_hostname="mail3.example.com"
jail_mail3_ip="192.168.1.233"
jail_mail3_start="/bin/sh /etc/rc"
jail_mail3_devfs_enable="YES"
jail_mail3_devfs_ruleset="devfsrules_jail"

jail_mail4_rootdir="/usr/jails/mail4"
jail_mail4_hostname="mail4.example.com"
jail_mail4_ip="192.168.1.234"
jail_mail4_start="/bin/sh /etc/rc"
jail_mail4_devfs_enable="YES"
jail_mail4_devfs_ruleset="devfsrules_jail"

------------------------------------------

Make the virtual system

cd /usr/src
make world DESTDIR=/usr/jails/mail1

if you get this error "make: don't know how to make world. Stop"
it's because you don't have to source tree installed...
look at cvs-supfile blog post to know how to do it...

cd etc
make distribution DESTDIR=/usr/jails/mail1
cd /usr/jails/mail1
ln -sf /dev/null kernel

touch /usr/jails/mail1/etc/fstab
cat /etc/resolv.conf > /usr/jails/mail1/etc/resolv.conf
mount -t procfs proc /usr/jails/mail1/proc
mount -t devfs dev /usr/jails/mail1/dev
mkdir /usr/jails/mail1/usr/ports
mount_nullfs /usr/ports /usr/jails/mail1/usr/ports

Once the first jail is created, we can just copy jails and not have to redo "make world" which takes awhile.

----------------------------------------------

Configuration of the jail
Start the jail for the first time

# /etc/rc.d/jail start mail1

If you try to SSH into the jail, you will end up in the main machine and not in the jail.
To fix this add the following line to /etc/rc.conf

rpcbind_enable="NO"

vi /etc/ssh/sshd_config

Port 22
Protocol 2
ListenAddress 192.168.1.157

Now we jump into the jail for some 1st step configurations

# jail /usr/jails/mail1 mail1.example.com 192.168.1.231 /bin/sh

now you are inside the jail:

change the root password
# passwd

add a new user and invite the user to the wheel group (so you can su to root)
# adduser

So we can ssh into the jail

# echo 'sshd_enable="YES"' >> /etc/rc.conf

install bash, change the users shell to bash
if you are in a jail and ports doesn't exit you have to remount the ports Filesystem
# cd /usr/ports/shells/bash
# make clean; make install clean
# chsh
change the shell to: /usr/local/bin/bash

Host environment syslogd
This entry in /etc/rc.conf tells syslogd to not listen on any IP address.

syslogd_flags="-ss"

That allows syslogd to run in both the host and the jail environments.

Host environment inetd

This entry in /etc/rc.conf tells inetd to listen on a specific IP address. This address is that of the host environment:

inetd_flags="-wW -C 60 -a 192.168.0.100"

You should note that the first part of the above flags is from /etc/defaults/rc.conf:

inetd_flags="-wW -C 60" # Optional flags to inetd

jail_enable="YES"
jail_list="mail_1"
jail_set_hostname_allow="YES"

jail_mail_1_rootdir="/usr/jails/mail_1"
jail_mail_1_hostname="mail_1.example.com"
jail_mail_1_ip="192.168.1.202" jail_mail_1_start="/bin/sh /etc/rc"
jail_mail_1_devfs_enable="YES"
jail_mail_1_devfs_ruleset="devfsrules_jail"

if you forget root password
jail /dumpdrive500/jails/devbox2 devbox2 192.168.1.152 /usr/bin/passwd root

------------------------------------------------

Clone the Jails

mkdir /usr/jails/mail2
cd /usr/jails/mail1
tar -cpf - . | tar -C /usr/jails/mail2 -xpf -

Mount all the needed directories for the cloned jails

mount -t procfs proc /usr/jails/mail1/proc
mount -t devfs dev //usr/jails/mail1/dev
mount_nullfs /usr/ports /usr/jails/mail1/usr/ports

vi /etc/rc.conf
added in -------------
jail_devbox5_rootdir="/usr/jails/mail1"
jail_devbox5_hostname="devbox5.example.com"
jail_devbox5_ip="192.168.1.155"
jail_devbox5_start="/bin/sh /etc/rc"
jail_devbox5_devfs_enable="YES"
jail_devbox5_devfs_ruleset="devfsrules_jail"

/etc/rc.d/jail start devbox5
ref: http://www.freebsd.org/doc/en/books/handbook/jails-build.html [[BR]]
ref: http://www.freebsddiary.org/jail-6.php

19Oct/100

freebsd cyrus sasl2 saslauthd

Here is a step by step setup of what I did to get cyrus sasl2 saslauthd working on Freebsd

You'll notice that i reinstall postfix due to the fact that I needed it compiled with sasl2 which I didn't have.

# cd /usr/ports/security/cyrus-sasl2-saslauthd
# make install clean
# cd /usr/ports/mail/postfix
# make config (make sure you check on cyrus-sasl2-saslauthd)
# make
# make FORCE_PKG_REGISTER=yes install clean

now you edit /etc/rc.conf and add the following lines near the top of the file.

postfix_enable="YES"
saslauthd_enable="YES"

#If you not need sendmail anymore, please add in your rc.conf:
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

# And you can disable some sendmail specific daily maintenance routines in your system
# If /etc/periodic.conf does not exist please create it and add those values.

/etc/periodic.conf file:

daily_clean_hoststat_enable="NO"
daily_status_mail_rejects_enable="NO"
daily_status_include_submit_mailq="NO"
daily_submit_queuerun="NO"

After installing saslauthd

/usr/local/etc/rc.d/saslauthd start
vi /etc/postfix/main.cf

Add this to the end...

# sasl config
broken_sasl_auth_clients = yes
smtpd_sasl_auth_enable = yes
smtpd_sasl_local_domain =

smtpd_sender_restrictions = permit_mynetworks, permit_sasl_authenticated
smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination

/usr/local/sbin/postfix stop && /usr/local/sbin/postfix start

/usr/local/sbin/saslpasswd2 -c -u domain.com -a smtpauth usernameeee

/usr/local/etc/rc.d/saslauthd restart

Filed under: FreeBSD, Unix No Comments
2Jun/102

how to add a new munin node with freebsd

Here is what I did to add a new munin node in freebsd to an existing munin server.

cd /usr/ports/sysutils/munin-node/
make install clean

When asked about the logs, answer yes. We want munin to rotate the logs.

Would you like me to set up log rotation [y]?

After the munin install this is what we see and need to do.

********************************************************************
Unless this file already existed, a sample configuration file
has been placed in /usr/local/etc/munin/munin-node.conf.

Please edit it according to your needs.

********************************************************************

The Munin client will *not* be started automatically. To allow it
to start, put this line in /etc/rc.conf:

munin_node_enable="YES"

Then, it will be started on the next boot. If this line is already
present, the client will be started now. Otherwise, edit
/etc/rc.conf and execute this command:

/usr/local/etc/rc.d/munin-node start
********************************************************************

Lets do the easy stuff first for munin.

vi /etc/rc.conf

add munin_node_enable="YES" somewhere in the file. I like to keep all the enable options together

munin_node_enable="YES"

The plugins directory in freebsd is empty, so for munin node to work we need to add symbolic links.

cd /usr/local/etc/munin/plugins

ln -s /usr/local/share/munin/plugins/cpu cpu
ln -s /usr/local/share/munin/plugins/df df
ln -s /usr/local/share/munin/plugins/df_inode df_inode
ln -s /usr/local/share/munin/plugins/if_errcoll_ if_errcoll_fxp0
ln -s /usr/local/share/munin/plugins/if_ if_fxp0
ln -s /usr/local/share/munin/plugins/load load
ln -s /usr/local/share/munin/plugins/memory memory
ln -s /usr/local/share/munin/plugins/netstat netstat
ln -s /usr/local/share/munin/plugins/ntp_offset ntp_offset
ln -s /usr/local/share/munin/plugins/open_files open_files
ln -s /usr/local/share/munin/plugins/processes processes
ln -s /usr/local/share/munin/plugins/sendmail_mailqueue sendmail_mailqueue
ln -s /usr/local/share/munin/plugins/sendmail_mailstats sendmail_mailstats
ln -s /usr/local/share/munin/plugins/sendmail_mailtraffic sendmail_mailtraffic
ln -s /usr/local/share/munin/plugins/swap swap
ln -s /usr/local/share/munin/plugins/vmstat vmstat

Next, lets look at the conf file

vi /usr/local/etc/munin/munin-node.conf

update the allow line to the IP address of the munin server

allow ^192\.168\.1\.105$

everything else is good, save and exit

for the munin server
add the IP address of that machine

vi /usr/local/etc/munin/munin.conf

Add this to the end of munin.conf

[server1.domain.com]
address 192.168.1.203
use_node_name yes
notify_alias server1

load.notify_alias load

df.notify_alias df

# /
df._dev_mirror_gm0s1a.warning :85
df._dev_mirror_gm0s1a.critical :90

# /tmp
df._dev_mirror_gm0s1e.warning :80
df._dev_mirror_gm0s1e.critical :90

# /usr
df._dev_mirror_gm0s1f.warning :80
df._dev_mirror_gm0s1f.critical :90

# /var
df._dev_mirror_gm0s1d.warning :80
df._dev_mirror_gm0s1d.critical :90

If you haven't done so you can start munin-node from

cd /usr/local/etc/rc.d/
./munin-node start

If munin node doesn't update after 5..10 minutes look at the munin main log to find any errors that you may have.

cd /var/log/munin-main
vi munin-update.log

17Mar/101

mod_wsgi apache freebsd python

The version of Apache on older versions of FreeBSD is not compiled with support for multithreading but Python itself is. Normally this would be where Apache 1.3 is being used. The result is that when starting up Apache with mod_wsgi, an error similar to the following can occur:

Syntax error on line 37 of /usr/local/apache2/conf/httpd.conf: \
Cannot load /usr/local/apache2/modules/mod_wsgi.so into server: \
/usr/local/apache2/modules/mod_wsgi.so: Undefined symbol \
"pthread_attr_init"

You may also get this Error with apache 1.3

Syntax error on line 244 of /usr/local/etc/apache/httpd.conf:
Cannot load /usr/local/libexec/apache/mod_wsgi.so into server: \
/usr/local/lib/libpython2.4.so: Undefined symbol "pthread_attr_destroy"

The solution is to force Apache to preload the reentrant version of the C runtime library which contains the POSIX thread library functions. This can be done by modifying the 'envvars' script, if it exists and is used, contained in the same directory as where the Apache 'httpd' binary is installed, or the Apache startup script, and add the lines:

LD_PRELOAD=/usr/lib/libc_r.so
export LD_PRELOAD

Step 1.

Set the library to system environment.

# LD_PRELOAD=/usr/lib/libc_r.so
# export LD_PRELOAD

Step 2.

Stop apache then start it up again

root@host rc.d # ./apache stop
Stopping apache.
Waiting for PIDS: 35117.

root@host rc.d # ./apache start
Starting apache.

root@host rc.d #

15Mar/100

sqlite error trac install

While trying to install trac on freebsd 7.0 I had an error with sqlite.
The problem was that I had an old version already installed, but the system didn't tell me that!

# cd /usr/ports/www/trac
# make install clean

Installed /usr/local/lib/python2.5/site-packages/pytz-2009r-py2.5.egg
===> Registering installation for py25-pytz-2009r
===> Returning to build of trac-0.11.5
===> trac-0.11.5 depends on file: /usr/local/lib/python2.5/site-packages/_sqlite3.so - not found
===> Verifying install for /usr/local/lib/python2.5/site-packages/_sqlite3.so in /usr/ports/databases/py-sqlite3
===> Vulnerability check disabled, database not found
===> Extracting for py25-sqlite3-2.5.2_1
=> No MD5 checksum recorded for python/Python-2.5.2.tgz.
=> No SHA256 checksum recorded for python/Python-2.5.2.tgz.
=> No suitable checksum found for python/Python-2.5.2.tgz.
*** Error code 1

Stop in /usr/ports/databases/py-sqlite3.
*** Error code 1

Stop in /usr/ports/www/trac.
*** Error code 1

Stop in /usr/ports/www/trac.

I thought to myself, maybe I need to update the system.

# cvsup -g -L 2 /root/cvs-supfile

# make install clean

That did nothing, I get this same error. Maybe if I install it manually

cd /usr/ports/databases/py-sqlite3

root@host py-sqlite3 # make install clean
===> Vulnerability check disabled, database not found
===> Extracting for py25-sqlite3-2.5.2_1
=> No MD5 checksum recorded for python/Python-2.5.2.tgz.
=> No SHA256 checksum recorded for python/Python-2.5.2.tgz.
=> No suitable checksum found for python/Python-2.5.2.tgz.
*** Error code 1

Stop in /usr/ports/databases/py-sqlite3.

that didn't work either, so I tried

# pkg_add -R sqlite3

But the ftp server was wrong to get the correct port...

for sh you type in
setenv PACKAGESITE ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-7-stable/Latest/

for bash you update the ftp location by typing
export PACKAGESITE=ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-7-stable/Latest/
export PACKAGESITE=ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-6-stable/Latest/

# pkg_add -r trac

That worked but this is where the Error shows me I have an old version of sqlite already installed!

...
Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-7-stable/All/py26-sqlite3-2.6.4_1.tbz... Done.
pkg_add: warning: package 'py26-sqlite3-2.6.4_1' requires 'sqlite3-3.6.19', but 'sqlite3-3.5.6' is installed
...

so I upgrade my port that I do have.
# /usr/local/sbin/portupgrade -R sqlite3

---------
root@draco sqlite3 # cd /usr/ports/www/trac
root@draco trac # make install clean

Traceback (most recent call last):
File "setup.py", line 15, in
from setuptools import setup, find_packages
ImportError: No module named setuptools
*** Error code 1

Stop in /usr/ports/www/trac.
*** Error code 1

Stop in /usr/ports/www/trac.

3Mar/100

mysql replication error

MySQL replication broke after a server crash. When trying to start it backup this error shows up.

100303 10:09:54 [ERROR] Slave: Query caused different errors on master and slave. Error on master: 'Invalid error code' (126), Error on slave: 'no error' (0). Default database: 'db_name'. Query: 'INSERT DELAYED INTO stats (clientID, date_accessed, current_url, referred_by, unique_id)
VALUES (0, now(), '/index.php', '', '076c47a32066f611d60e107b3c6be475')', Error_code: 0
100303 10:09:54 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'mysql-bin.000463' position 207592

After a little searching, in order to fix the problem I just have to repair the table with the following command.

mysql> repair table stats;

In my case the query is fine, the reason why it is failing may be
index file is crashed or file format is wrong. Here is a list of error codes that may be helpful to other people.
126 = Index file is crashed / Wrong file format
127 = Record-file is crashed
132 = Old database file
134 = Record was already deleted (or record file crashed)
135 = No more room in record file
136 = No more room in index file
141 = Duplicate unique key or constraint on write or update
144 = Table is crashed and last repair failed
145 = Table was marked as crashed and should be repaired
28 = You ran out of disk space. Delete unnecesary files, or move the MySQL data files to another partition.
1045: Access denied for user: = You forgot to specify a password, or you used the wrong one. Use the -u and -p options when starting the client. (use mysql --help to see how)
1046: No Database Selected = You must use the command USE database_name or the mysql_select_db() function to select a database to use.
1062: Duplicate entry 'x' for key n = You have attempted to insert or update data that would break the integrity of a UNIQUE index (most likely the primary key). You must redefine either the data you're trying to insert, or the indexes which you have imposed on the table.
1064: Syntax error = You wrote something that doesn't match the syntax definitions in the manual. When you have checked everything else, here's something to consider: A common error is to put a space between a function name and its opening parenthesis '('. Another common error is to use a reserved word out of its normal context.
2000: parse error near ... = see explanation for 1064: Syntax error
2002: Can't connect to local MySQL server = The MySQL socket file (usually, /tmp/mysql.sock) is missing. See the notes on connecting to the server.

Filed under: Errors, MySQL, Unix No Comments
10Feb/100

vi search and replace ^M

To easily search and replace the ^M (new line) character in vi use the following command. Make sure your in command mode and not input mode when you do it.

:%s/^M$//g

to get the proper ^M in vi you'll need to do CTRL-v CTRL-M and that should display ^M

Filed under: commands, Errors, Unix No Comments
5Feb/1010

Installing Apache Tomcat 6 on FreeBSD

So you want to install Tomcat 6 on FreeBSD, this is how I did it.

cd /usr/ports/www/tomcat6
make install clean

use the default Options for diablo-jdk

Then you get some nice "licensing restrictions"

Because of licensing restrictions, you must fetch the distribution
manually.

Please access

http://www.FreeBSDFoundation.org/cgi-bin/download?download=diablo-caffe-freebsd6-i386-1.6.0_07-b02.tar.bz2

with a web browser and "Accept" the End User License Agreement for
"Caffe Diablo 1.6.0".

Please open http://java.sun.com/javase/downloads/index.jsp
in a web browser and follow the "Download" link for
"JDK US DST Timezone Update Tool - 1_3_21" to obtain the
time zone update file, tzupdater-1_3_21-2009p.zip.

Please place the downloaded file(s) in /usr/ports/distfiles.

*** Error code 1

Stop in /usr/ports/java/diablo-jdk16.
*** Error code 1

Stop in /usr/ports/java/diablo-jdk16.
*** Error code 1

Stop in /usr/ports/www/tomcat6.

So we go to the freebsd foundation website and download diablo caffe.
Then we go to the sun websute to download the Timezone update and what do you know, Sun has version 1.3.25 and not 1.3.21.

So we do an "cvsup -g -L 2 /root/cvs-supfile" on our system to update the ports collection.
... wait 10..20 minutes as the system wasn't up to date ...

While that's going, we copy (sftp) the file over to the server "diablo-caffe-freebsd6-i386-1.6.0_07-b02.tar.bz2"

cp /home/user_name/diablo-caffe-freebsd6-i386-1.6.0_07-b02.tar.bz2 /usr/ports/distfiles

The ports collection was updated but it didn't do anything, still need 1.3.21... Maybe because it's an old version of FreeBSD (6.2) o_0
Luckily for me, I had tzupdater-1_3_21-2009p.zip from a pervious install on another server (email me if you need the file)
I rsync the file accross

rsync -ave ssh /usr/ports/distfiles/tzupdater-1_3_21-2009p.zip user_name@192.168.1.101:/home/user_name/tzupdater-1_3_21-2009p.zip

cp /home/user_name/tzupdater-1_3_21-2009p.zip /usr/ports/distfiles
cd /usr/ports/www/tomcat6
make install clean

The installation continues. Then it finishes without issue.

cd /usr/local/apache-tomcat-6.0
vi RUNNING.txt

To run it, we need to set up the JAVA_HOME environment variable... I'm on the bash shell (GO BASH) and we installed the JDK not JRE so I do

JAVA_HOME=/usr/local/diablo-jdk1.6.0
cd /usr/local/etc/rc.d/
./tomcat6 start

and nothing happened... Probably because it's not in /etc/rc.conf

cd /usr/local/apache-tomcat-6.0/logs/
ls -l

no logs in the log folder...

cd /usr/local/apache-tomcat-6.0/bin/

./startup.sh

Now we have logs

tail catalina.out

The following line shows up...

Feb 5, 2010 10:17:01 AM org.apache.coyote.http11.Http11Protocol start
INFO: Starting Coyote HTTP/1.1 on http-8180

so we go to http://192.168.1.101:8180 and just like that it comes up..

If you want to change the port, edit /usr/local/apache-tomcat-6.0/conf/server.xml

Now that was a long install, hope it helps someone.

4Feb/102

postfix bcc on all emails

Track the email messages that make their way across your system.
What are those reasons? You may have noticed an increase in email abuse, either internal or external, Where forwarding a copy of each message to an administrator would be desirable.

If your looking just to get bcc'ed on a users email, please look at my previous post.

# cd /etc/postfix
# vi main.cf

//add this line to the bottom of your main.cf
always_bcc=admin@domain.com

save and quit then restart postfix.

# cd /usr/local/sbin
# ./postfix reload

now every email that gets sent through that machine, admin@domain.com will get bcc'ed on it.