Thursday, February 23, 2012

DNS kerberos-client host server ldap Nagios


tsu - a new social networking platform which pays users to post!

https://www.tsu.co/ashoklucky


https://nsrc.org/workshops/ws-files/2011/sanog17/exercises/

Source from: nsrc.org

Exercise 1: Set up a kerberos client

A Kerberos client is easy to set up to work with existing Kerberos infrastructure.

Setup

Install the Kerberos client packages:
# apt-get install krb5-user
To show how little configuration is really needed, we will move the auto-generated config file out of the way (which contains a load of junk anyway) and create a new minimal one with just 4 lines:
# mv /etc/krb5.conf /etc/krb5.conf.example
# editor /etc/krb5.conf

[libdefaults]
default_realm = WS.NSRC.ORG
dns_lookup_realm = true
dns_lookup_kdc = true
Now check that the ssh client program will try Kerberos authentication. It's enabled by default in Ubuntu, but in other operating systems it might not be.
# editor /etc/ssh/ssh_config      (on MacOSX it's /etc/ssh_config)

...
GSSAPIAuthentication yes    # check this line present (near end)
GSSAPIKeyExchange yes       # add this line too
...

Testing ssh client

The class Kerberos setup has an account "testuser" which you can use.
$ kinit testuser
... enter password when prompted
(password is "nsrc2020" unless you've been told otherwise)
Now you should now be able to login to servers in your Kerberos realm without re-entering your password:
$ ssh testuser@noc.ws.nsrg.org
-- logout, then login somewhere else

$ ssh testuser@noc2.ws.nsrc.org
-- logout
Have a look at the tickets you've picked up:
$ klist
You should see your own TGT plus tickets for the servers you've connected to.
To get rid of them, use kdestroy; check that you can no longer login (the server will give you a password prompt instead)

Reference material [not part of the exercise]

We've actually enabled two different authentication mechanisms. GSSAPIAuthentication is the standard one (gssapi-with-mic), and just authenticates the user. GSSAPIKeyExchange is a new one (gssapi-keyex) which also validates the authenticity of the host. It avoids the need to accept host keys into your known_hosts file, because Kerberos already provides mutual authentication.
GSSAPIKeyExchange is available as a patch to ssh, and is included by default in recent Debian/Ubuntu, RHEL6, and Fedora 13+. But it probably won't work when connecting to older Unix boxes or to BSD boxes, in which case you'll fall back to gssapi-with-mic.
More info at http://www.sxw.org.uk/computing/patches/openssh.html

Enabling Kerberos in HTTP clients

For curl, you must supply an empty username and password in option -u
$ curl --negotiate -u: http://noc.ws.nsrc.org/secure/
For Firefox:
  • Go to about:config
  • Filter on "negotiate"
  • network.negotiate-auth.trusted-uris ws.nsrc.org
For Google Chrome: apply the option when starting it up:
/opt/google/chrome/google-chrome \
  --auth-server-whitelist=*.ws.nsrc.org

Enabling Kerberos in LDAP client

Under Ubuntu you have to have the appropriate SASL-GSSAPI module installed.
# apt-get install ldap-utils libsasl2-modules-gssapi-mit

$ ldapsearch -Y GSSAPI -h ldap.ws.nsrc.org \
    -b "dc=ws,dc=nsrc,dc=org" "(cn=*candler*)"

---------------------------------------------------------------------------------------

Exercise2 : enable Kerberos logins to a host

This is to connect a server to an existing Kerberos/LDAP infrastructure. In part 1 we'll just enable Kerberos authentication, and in part 2 we'll enable LDAP authorization for uid/gid mapping.

Part 1: Kerberos authentication

Your machine must already be working as a Kerberos client (i.e. /etc/krb5.conf is setup correctly)

Sync your clock

Kerberos only works if your clocks are in close sync (certainly within 5 minutes). You can do a one-off clock sync like this:
# ntpdate ntp.ubuntu.com
But it's better to install the ntp daemon, which will continuously keep your clock in accurate sync.
# apt-get install ntp

Enable kerberos for sshd

In the sshd_config file, uncomment the GSSAPIAuthentication line and set it to yes; also add GSSAPIKeyExchange yes. Then restart sshd.
# editor /etc/ssh/sshd_config
...
# GSSAPI options
GSSAPIAuthentication yes
GSSAPIKeyExchange yes
....

# service ssh restart
You're also going to add some extra lines to the end of krb5.conf, because of a current limitation in the kadmin utility.
# editor /etc/krb5.conf
...

[realms]
WS.NSRC.ORG = {
    admin_server = kdc1.ws.nsrc.org
}
Now you need to set up a shared key between the KDC and the host's keytab. Call over an instructor to do this for you (because we're not letting you have admin access to the workshop Kerberos server :-) Here's what they will do:
# kadmin -p nsrc/admin
... enter secret instructor password
addprinc -randkey host/pcN.ws.nsrc.org
ktadd host/pcN.ws.nsrc.org
^D
This has (1) created the principal on the KDC, and (2) has copied the key into a local keytab file.
You can examine your own keytab using either klist -k for a summary, or ktutil for more detailled information. Inside ktutil use '?' for help and ^D (ctrl-D) to exit.
# ktutil
rkt /etc/krb5.keytab
list -e
^D
(The key will be listed multiple times, for use with different encryption algorithms)
Now you're going to allow ssh logins for 'testuser'. Because we don't have LDAP configured yet, we'll have to create a local account - but there's no need to set a password.
# useradd -m -s /bin/bash testuser
Now you're ready to test it. You can try logging in from your own machine to yourself, and you can get someone else to login to your machine.
$ kinit testuser         # use password from exercise 1
$ ssh testuser@pcN.ws.nsrc.org
If instead of logging straight in you see a password prompt from ssh, then there's a problem which needs investigating.

Debugging

The first things to check are:
  • Is your system clock at the correct time?
  • Does your host have working forward and reverse DNS?
  • Did you get a ticket? (klist)
  • Did you restart sshd after setting GSSAPIAuthentication yes?
If those are correct, then in another window you can start a new instance of sshd in debug mode on a different port:
# /usr/sbin/sshd -d -p99
Then try to connect to it:
$ ssh -v -p99 testuser@pcN.ws.nsrc.org
You will see logs at both the client and server side.

Part 2: LDAP authorization

This centralises your user/uid/gid/homedir management, and removes the need to add local users.
We're going to configure it manually. In practice you'd probably just make a tarball containing all the preconfigured files and drop them onto each server which needed it.

Install LDAP utils

# apt-get install ldap-utils libsasl2-modules-gssapi-mit

# editor /etc/ldap/ldap.conf
BASE    dc=ws,dc=nsrc,dc=org
URI     ldap://ldap.ws.nsrc.org
Now this is configured, you should be able to query the LDAP directory - but the server is set up only to allow read access if you have a Kerberos ticket.
$ kinit testuser
$ ldapsearch
... should list everything in ldap
$ ldapsearch "(cn=*test*)"
... should list only entries matching the search query

Automated Kerberos ticket

For security, the LDAP server only permits connections which are Kerberos authenticated and encrypted. This means that the host itself needs a Kerberos ticket when doing LDAP lookups. (We could instead have used TLS security, but that would involve setting up a Certificate Authority, and we may as well leverage our Kerberos setup)
We will run a cronjob every hour, using our existing keytab entry to obtain a ticket for host/pcN.ws.nsrc.org
# editor /etc/cron.hourly/kerberos
#!/bin/sh
/usr/bin/kinit -k host/`hostname` -c /tmp/krb5cc_host
Make it executable, and then run it once manually:
# chmod +x /etc/cron.hourly/kerberos
# /etc/cron.hourly/kerberos
You can use this keytab to be able to do 'ldapsearch' when logged in as root:
# KRB5CCNAME=/tmp/krb5cc_host ldapsearch
Note: our LDAP queries will come from the caching daemon (nscd) which runs as root, and so this is all we need. If you wanted to run nscd as a different user, you would have to extend the cron script to make the ticket readable to that user. e.g.
#!/bin/sh
/usr/bin/kinit -k host/`hostname` -c /tmp/krb5cc_host

umask 077
cp /tmp/krb5cc_host /tmp/krb5cc_nscd
chown nscd /tmp/krb5cc_nscd

Install nss_ldap

This is the library which intercepts password/group lookups and redirects them to LDAP.
# apt-get install libnss-ldap nscd
(Beware: there is an alternative package libnss-ldapd with a trailing "d". These exercises have been tested using libnss-ldap)
Answer prompted questions as:
LDAP server                            [ldap://ldap.ws.nsrc.org]
Distinguished name of search base      [dc=ws,dc=nsrc,dc=org]
LDAP version to use                    [3]
Make local root database admin         [No]
Does the LDAP database require login?  [No]
NOTE: The LDAP server starts with ldap:// not ldapi:// !
The configuration file is /etc/ldap.conf. We're going to replace it with a simple one which uses Kerberos authentication:
# mv /etc/ldap.conf /etc/ldap.conf.example
# editor /etc/ldap.conf
krb5_ccname /tmp/krb5cc_host
use_sasl on
rootuse_sasl on
base dc=ws,dc=nsrc,dc=org
uri ldap://ldap.ws.nsrc.org
ldap_version 3
sasl_secprops minssf=56
nss_initgroups_ignoreusers backup,bin,bind,daemon,games,gnats,irc,libuuid,list,lp,mail,man,news,nslcd,ntp,openldap,proxy,root,sshd,sync,sys,syslog,uucp,www-data
We are also going to tweak nscd.conf to reduce caching of group information from 3600 to 600 seconds.
# editor /etc/nscd.conf
...
positive-time-to-live   group           600
...

Configure nsswitch

This is the point where we actually start to use LDAP for lookups.
# editor /etc/nsswitch.conf
...
passwd:         compat ldap
group:          compat ldap
shadow:         compat ldap
...

# service nscd restart
At this point you're ready to test: you should be able to map a username to a uid over LDAP, even though it doesn't exist in your local /etc/passwd file
# id ldapuser
uid=10004(ldapuser) gid=100(users) groups=100(users)
If it fails, places to look are:
  • /var/log/auth.log
  • has root got a kerberos ticket? Use the following commands as root:
    # KRB5CCNAME=/tmp/krb5cc_host klist
    # KRB5CCNAME=/tmp/krb5cc_host ldapsearch
    
  • after any changes which might fix the problem, remember to restart nscd because it has both positive and negative caching
  • try stopping nscd, then try id ldapuser as root - there will be no caching, so it will make a separate LDAP query for each request. You can look at tcpdump, and logs on the LDAP server side.
  • if you control the LDAP server, run it in debug mode:
    # /etc/init.d/slapd stop
    # KRB5_KTNAME=/etc/ldap/krb5/krb5.keytab slapd -d 255 -u openldap -g openldap
    

Home directory creation

When a user first logs into a machine, you can create their home directory automatically via PAM (Pluggable Authentication Modules).
# editor /etc/pam.d/common-session
...
session required        pam_mkhomedir.so umask=0022 skel=/etc/skel/
...

Miscellaneous Notes

Using kadmin to extract the key on the target host itself is almost always the best way to set up the shared host/service key.
If joining a Microsoft Active Directory domain: you will need a third-party tool such as css_adkadmin (or it may be possible to use bits from samba). This may be a starting point:
kinit Administrator
css_adkadmin -r AD.NSRC.ORG -s dc1.ad.nsrc.org -q "ank -k -attr description=$HOSTNAME +randkey host/$HOSTNAME"
css_adkadmin -r AD.NSRC.ORG -s dc1.ad.nsrc.org -q "xst +randkey host/$HOSTNAME"
The alternative is to set a manual password on a host principal and then use ktutil to create a keytab with the same password. This is awkward, because (1) you need to use the right kvno, and (2) you need to repeat it for each encryption type which might be used on your network.
[KDC side]
kadmin
addprinc host/pcX.ws.nsrc.org
... you are prompted for password
getprinc host/pcX.ws.nsrc.org
... make a note of the kvno

[host side]
# ktutil
ktutil:  addent -password -p host/pcX.ws.nsrc.org -k 4 -e aes256-cts
Password for host/pcX.ws.nsrc.org@WS.NSRC.ORG: 
ktutil:  addent -password -p host/pcX.ws.nsrc.org -k 4 -e arcfour-hmac
Password for host/pcX.ws.nsrc.org@WS.NSRC.ORG: 
ktutil:  addent -password -p host/pcX.ws.nsrc.org -k 4 -e des3-hmac-sha1
Password for host/pcX.ws.nsrc.org@WS.NSRC.ORG: 
ktutil:  list
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
   1    4         host/pcX.ws.nsrc.org@WS.NSRC.ORG
   2    4         host/pcX.ws.nsrc.org@WS.NSRC.ORG
   3    4         host/pcX.ws.nsrc.org@WS.NSRC.ORG
ktutil:  wkt /etc/krb5.keytab

****************************************************************************

Exercise 3: Build a KDC

These exercises assume your host is pc1.ws.nsrc.org and you will be building a realm REALM1.WS.NSRC.ORG - change 1 to the appropriate value for your machine.

1. Install ntp

Because tickets are timestamped, it's important that your machine always has an accurately synchronized clock. Double-check it's installed:
# apt-get install ntp
This will run a daemon (ntpd) which keeps your clock synchronized.
Note: if you have a large network then it makes sense to have two local timeservers and all other machines sync to those. You would then change /etc/ntpd.conf to point to your time server(s).

2. Install kdc packages

# apt-get install krb5-kdc krb5-admin-server
If you get a dialog box about setting up a kerberos realm, just accept [Ok]

3. Configure krb5 library

Now we are going to configure the krb5 client library, manually associating your own machine with your own realm. (We could instead change the DNS, but doing it here lets you control your own settings)
# editor /etc/krb5.conf
[libdefaults]
default_realm = REALM1.WS.NSRC.ORG
dns_lookup_realm = true
dns_lookup_kdc = true

[realms]
REALM1.WS.NSRC.ORG = {
        kdc = pc1.ws.nsrc.org
        admin_server = pc1.ws.nsrc.org
}

[domain_realm]
pc1.ws.nsrc.org = REALM1.WS.NSRC.ORG

4. Create and initialize your database

Create Kerberos database:
kdb5_util create -r REALM1.WS.NSRC.ORG -s
# This can pause for several minutes. Eventually you will be asked to
# choose a database master password. Use "abcd" for this exercise,
# but normally you'd choose something much stronger.
Create the ACL file and grant admin rights to all */admin principals:
# editor /etc/krb5kdc/kadm5.acl
*/admin@REALM1.WS.NSRC.ORG    *
Now we use kadmin.local (running as root) to create the first few principals: a host principal for the host itself (putting the randomly-chosen key into its own keytab file); a regular principal "student" and a KDC admin principal "student/admin"
# kadmin.local
addprinc -randkey host/pc1.ws.nsrc.org
ktadd host/pc1.ws.nsrc.org
addprinc student
-- you'll be prompted to choose a password
addprinc student/admin
-- you'll be prompted to choose a password
^D
Now start the daemons:
# /etc/init.d/krb5-kdc start
# /etc/init.d/krb5-admin-server start
At this point, you should be able to get a ticket from your own realm:
$ kinit student
Password for student@REALM1.WS.NSRC.ORG: 
$ klist
(Note that you've obtained a ticket from your own realm, using the username and password you created in your own KDC)
If this doesn't work, then debug as follows:
  • Check your clock is synchronized (type date)
  • Check your krb5.conf
  • Check kdc is running: ps auxwww | grep krb5kdc
  • Look at KDC logs in /var/log/auth.log

5. Managing principals

From now on, you don't need kadmin.local as root to administer the Kerberos database; you can use kadmin instead, as a normal user. This runs over a TCP socket (so it can be run from a remote workstation, not necessarily the KDC). The traffic is authenticated and encrypted using Kerberos.
$ kadmin -p student/admin
... enter password for 'student/admin' which you chose earlier
Type '?' for a list of commands. The important ones are:
listprincs              -- list principals
addprinc     -- add principal
cpw          -- change password
delprinc     -- delete principal
Use ^D (ctrl-D) to exit.
Try the following:
  • Change the password for the student principal. In another screen, check that you can kinit student with the new password.
  • Create a new user principal of your choice, and check you can get a ticket for it with kinit.

6. Kerberos ssh

Since we have modified krb5.conf earlier, restart sshd:
# service ssh restart
Create a local student user:
# useradd -m -s /bin/bash student
Note that we have not set any password. However you should be able to login using a kerberos ticket:
$ kinit student
$ ssh student@localhost

7. Extra exercises

If you have spare time, you and your neighbour can add [realms] and [domain_realms] sections for each other, and you can try to get a ticket directly from the other realm and use it to login to the other machine:
kinit student@REALM2.WS.NSRC.ORG
ssh student@pc2.ws.nsrc.org
You could also try setting up cross-realm authentication. Each KDC has to share a secret with its neighbour. On both KDCs create the following principals; for each principal enter the same (preferably long and complex) passphrase on both KDCs.
# kadmin -p student/admin
addprinc krbtgt/REALM2.WS.NSRC.ORG@REALM1.WS.NSRC.ORG
addprinc krbtgt/REALM1.WS.NSRC.ORG@REALM2.WS.NSRC.ORG
(The first sets up realm2 to trust realm1; the second sets up realm1 to trust realm2)
At this point, Kerberos authentication will work, but ssh on pc1 will reject a user who has authenticated as someone@REALM2 because pc1 is part of REALM1.
-- on pc1, logging in to pc2
$ kinit student
$ ssh -v student@pc2.ws.nsrc.org
-- should be rejected
To permit it, create file ~/.k5login containing the authorized principal(s), one per line.
-- on pc2
# editor ~student/.k5login
student@REALM1.WS.NSRC.ORG

Notes

In a real environment, you would add at least one slave KDC for resilience. There is information how to do this at http://web.mit.edu/kerberos/krb5-1.8/krb5-1.8.3/doc/krb5-install.html#Install%20the%20Slave%20KDCs

***********************************************************************

Exercise 4: build an LDAP server

This is by far the hardest part of the operation, due to the cryptic way OpenLDAP 2.4 now stores its configs within LDAP instead of in a config file. See man slapd-config
We are going to build an LDAP server with Base DN "dc=realm1,dc=ws,dc=nsrc,dc=org". Change this to match your own realm.
Install the slapd server and ensure everything else needed is present:
# apt-get install slapd ldap-utils libsasl2-modules-gssapi-mit
Set up the service principal with keytab readable by slapd:
# mkdir /etc/ldap/krb5
# kadmin -p student/admin
addprinc -randkey ldap/pc1.ws.nsrc.org
ktadd -k /etc/ldap/krb5/krb5.keytab ldap/pc1.ws.nsrc.org
^D
# chown -R openldap:openldap /etc/ldap/krb5
# chmod 550 /etc/ldap/krb5
# chmod 440 /etc/ldap/krb5/krb5.keytab
# editor /etc/default/slapd
...
export KRB5_KTNAME=/etc/ldap/krb5/krb5.keytab

# service slapd restart
Install the schemas we need:
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/cosine.ldif
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/nis.ldif
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/inetorgperson.ldif
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/misc.ldif
NOTE: For a real production LDAP server, read the files in /usr/share/doc/slapd carefully, especially README.Debian.gz and README.DB_CONFIG.gz
Now we need to run some scripts - either paste directly to a root shell, or create in temporary files and then run them.
BEWARE: in LDIF files, superfluous spaces at the end of lines may cause attributes to be rejected as invalid!
First is create_database.sh
ldapadd -Y EXTERNAL -H ldapi:/// <
(Note: we have no olcRootDN or olcRootPW. This is a pure Kerberos config)
Next is init_database.sh

ldapadd -Y EXTERNAL -H ldapi:/// <
And finally config.sh

# Because these are 'replace' operations it's safe to modify this script
# and re-run it
ldapmodify -Y EXTERNAL -H ldapi:/// <
The simple ACL says that full access is permitted to */admin principals in our realm, and otherwise read access is granted to any valid user (i.e. anyone with a kerberos ticket that we recognize). See OpenLDAP ITS#6757 for a note on the format of the auth DN.
We also require SASL, which disables anonymous and simple binds.
Note: minssf enforces encryption. GSSAPI always returns 56 for ssf, regardless of the strength of the actual crypto mechanism in use. See http://lists.andrew.cmu.edu/pipermail/cyrus-sasl/2006-September/000628.html
The following tweak disables unused sasl mechanisms, and means that -Y GSSAPI can be omitted form the ldapsearch command line.
# editor /etc/ldap/sasl2/slapd.conf
mech_list: gssapi external

Test LDAP

Update /etc/ldap/ldap.conf to point to your own LDAP server and base DN. Use ldapsearch to check that you can query it.
$ kinit testuser
$ ldapsearch [-Y GSSAPI] [-b "dc=realm1,dc=realm1,dc=ws,dc=nsrc,dc=org"]
This should dump back the (mostly empty) LDAP database, as long as you have a Kerberos ticket

Create user

We are going to create a user called "newuser"
Firstly in Kerberos:
$ kadmin -p student/admin
addprinc newuser
... choose a password
^D
And now in LDAP:
$ kinit student/admin
$ ldapadd <

Test NSS

Now update your LDAP nscd/nsswitch configuration (part 2 of exercise 2). You will need to edit /etc/ldap.conf with your own PC as LDAP server and your own Base DN. Restart nscd after doing this.
To verify all is working:
$ id newuser
You should see a result with the uid and gid. Repeated operations should be quick as nscd will be caching them.
Troubleshooting:
  • Make sure clock is synced
  • Make sure you have a kerberos ticket for root. Re-run /etc/cron.hourly/kerberos if necessary.
  • Restart nscd
  • If you stop nscd, then 'id ' will make an LDAP query as the current user. Make sure that the current user has a kerberos ticket.

ldapscripts

There is a package called ldapscripts with simple tools for adding and managing LDAP users. Unfortunately it doesn't know about GSSAPI authentication, so you need to patch it. See ldapscripts-sasl.diff
Then configure it:
# editor /etc/ldapscripts/ldapscripts.conf
SASLAUTH="GSSAPI"
...
SERVER="ldap://pc1.ws.nsrc.org"
...
SUFFIX="dc=realm1,dc=ws,dc=nsrc,dc=org"
Then adding a new user in LDAP is as simple as:
$ kinit student/admin
$ ldapadduser someuser users
$ kdestroy
Adding groups and users into groups:
$ ldapaddgroup noc
$ ldapaddusertogroup someuser noc
You may need to restart nscd before 'id' shows this information.

Configure backup LDAP

OpenLDAP replication is left as an exercise to the reader.

*******************************************************************

Preparation for the Kerberos lab

To get the "wow" factor for the initial exercises, showing how easy it is to add clients and servers to a Kerberised network, we need to have a Kerberos/LDAP environment already up and running.

Prepare the DNS

Build the NOC machine and ensure forward and reverse DNS is fully working, if not already done.
Create kdc1 as an alias to whatever machine will be running the primary KDC (probably also the NOC), and add the Kerberos service records.
# apt-get install bind9 bind9-utils
# editor /etc/bind/named.conf.local
zone "ws.nsrc.org"      { type master; file "/etc/bind/ws.nsrc.org"; };
zone "10.in-addr.arpa"  { type master; file "/etc/bind/10.in-addr.arpa"; };

# editor /etc/bind/ws.nsrc.org
$TTL    600
@       IN      SOA     noc.ws.nsrc.org. root.noc.ws.nsrc.org. (
                              1         ; Serial
                         604800         ; Refresh
                            600         ; Retry
                        2419200         ; Expire
                            600 )       ; Negative Cache TTL
@       IN      NS      noc.ws.nsrc.org.

_kerberos               IN      TXT     "WS.NSRC.ORG"
_kerberos-master._udp   IN      SRV     0 0 88 kdc1
_kerberos-adm._tcp      IN      SRV     0 0 749 kdc1
_kpasswd._udp           IN      SRV     0 0 464 kdc1
; List all active Kerberos servers here
_kerberos._udp          IN      SRV     0 0 88 kdc1
ldap                    IN      A       10.10.254.250

noc     IN      A       10.10.254.250
kdc1    IN      A       10.10.254.250
$GENERATE 1-20 $ IN PTR pc$.ws.nsrc.org.

# editor /etc/bind/10.in-addr.arpa
$TTL    600
@       IN      SOA     noc.ws.nsrc.org. root.noc.ws.nsrc.org. (
                              1         ; Serial
                         604800         ; Refresh
                            600         ; Retry
                        2419200         ; Expire
                            600 )       ; Negative Cache TTL
@       IN      NS      noc.ws.nsrc.org.

$GENERATE 1-20 $.1.10 IN PTR pc$.ws.nsrc.org.
250.254.10      IN      PTR     noc.ws.nsrc.org.

# editor /etc/resolv.conf
search ws.nsrc.org
nameserver 127.0.0.1
Test it:
# dig pc1
# dig -x 10.10.1.1
# dig _kerberos._udp.ws.nsrc.org. srv
If the NOC machine also has an entry for itself in /etc/hosts, make sure its FQDN is the first one listed.
# editor /etc/hosts
...
10.10.254.250  noc.ws.nsrc.org noc

Build slave DNS if desired

You like resilience don't you? :-) And it sets a good example!

Build primary KDC

See also the MIT Kerberos docs (esp. install guide) at http://web.mit.edu/kerberos/krb5-1.8/#documentation
Install ntp:
# apt-get install ntp
Install kdc packages:
# apt-get install krb5-kdc krb5-admin-server
Answer the annoying debconf questions:
Default Kerberos 5 realm:         [WS.NSRC.ORG]
Kerberos servers for your realm:  [kdc1.ws.nsrc.org]
Administrative server:            [kdc1.ws.nsrc.org]
Create the client library config:
# mv /etc/krb5.conf /etc/krb5.conf.example
# editor /etc/krb5.conf
[libdefaults]
default_realm = WS.NSRC.ORG
dns_lookup_realm = true
dns_lookup_kdc = true

[realms]
WS.NSRC.ORG = {
        admin_server = kdc1.ws.nsrc.org
}
(note that kadmin doesn't yet support looking up the admin server via DNS; hence we have to list it in krb5.conf)
Create Kerberos database:
kdb5_util create -r WS.NSRC.ORG -s
# You will be asked to choose a database master password. We use "abcd"
Create the ACL file and grant admin rights to all */admin principals:
# editor /etc/krb5kdc/kadm5.acl
*/admin@WS.NSRC.ORG    *
Now create some principals: a host principal for the host itself, putting the random key into its own keytab file; an instructor principal ("nsrc"); and an instructor KDC admin principal ("nsrc/admin")
# kadmin.local
addprinc -randkey host/noc.ws.nsrc.org
ktadd host/noc.ws.nsrc.org
addprinc nsrc
-- you'll be prompted to choose a password
addprinc nsrc/admin
-- you'll be prompted to choose a password
^D
Now start the daemons:
# /etc/init.d/krb5-kdc start
# /etc/init.d/krb5-admin-server start
At this point, you should be able to kinit nsrc and get a ticket. If you get "Cannot resolve network address for KDC" then check the DNS.
Note: new principals can now be added using kadmin instead of kadmin.local, and you can use it from any remote machine where you can kinit. (So you never need to do a local login to the KDC)

Build slave KDC if desired

If you want to build a slave KDC, see the instructions at http://web.mit.edu/kerberos/krb5-1.8/krb5-1.8.3/doc/krb5-install.html#Install%20the%20Slave%20KDCs
Add another _kerberos._udp srv entry in the DNS pointing to kdc2 so that clients know about both of them.

Enable Kerberized ssh

The first exercise requires students to make a single-signon ssh login.
Create a 'testuser' kerberos principal in the KDC, which the students will login as.
$ kadmin -p nsrc/admin
addprinc testuser
... choose a password which will be given to the students, e.g. nsrc2020
^D
On the machine(s) where the students will be allowed to login via kerberos, create a 'testuser' account (but don't set any local password) and enable Kerberos authentication for sshd. Note we're not doing LDAP yet.
# useradd -m -s /bin/bash testuser
# editor /etc/ssh/sshd_config
...
GSSAPIAuthentication yes
GSSAPIKeyExchange yes
...
# service ssh restart
Now we'll test it. You can either do this from the noc to itself, or you can use a separate throw-away client VM, or your own laptop. If using a separate machine you'll first have to make it a Kerberos client like this:
# apt-get install krb5-user
# mv /etc/krb5.conf /etc/krb5.conf.example
# editor /etc/krb5.conf
[libdefaults]
default_realm = WS.NSRC.ORG
dns_lookup_realm = true
dns_lookup_kdc = true
Check that /etc/ssh/ssh_config has GSSAPIAuthentication yes and if supported GSSAPIKeyExchange yes (On Mac OSX: /etc/ssh_config)
Now it should work:
$ kinit testuser
$ ssh testuser@noc
If it still doesn't work after this, Kerberos problems can be tricky to debug. First check both machines have accurately synced clocks, and that forward and reverse DNS is working. Then you can try starting another sshd in debug mode on another port:
# /usr/sbin/sshd -d -p99
Then on the client:
$ ssh -v -p99 nsrc@noc

Test with Kerberized Apache

If Apache isn't already installed on the noc, install it, and also curl for testing.
# apt-get install apache2 curl
Create a test directory and test page:
# mkdir /var/www/secure
# editor /var/www/secure/index.html
... whatever you like

-- check it works and is not yet protected
# curl http://noc.ws.nsrc.org/secure/
Now we're going to kerberize it.
# apt-get install libapache2-mod-auth-kerb
# editor /etc/apache2/conf.d/topsecret

  AuthName "Hello Kerberos World"
  AuthType Kerberos
  # Allow fallback to Basic Auth?
  KrbMethodK5Passwd Off
  KrbAuthRealms WS.NSRC.ORG
  Krb5Keytab /etc/apache2/krb5/krb5.keytab
  # require user nsrc@WS.NSRC.ORG
  require valid-user


# service apache2 restart
Check that the page is no longer visible (curl gives a 403 error)
We now have to create a service principal, and extract its key into the keytab specified in the config above, readable to the apache daemon.
# mkdir /etc/apache2/krb5
# kadmin -p nsrc/admin
addprinc -randkey HTTP/noc.ws.nsrc.org
ktadd -k /etc/apache2/krb5/krb5.keytab HTTP/noc.ws.nsrc.org 
^D
# chown -R www-data:www-data /etc/apache2/krb5
# chmod 550 /etc/apache2/krb5
# chmod 440 /etc/apache2/krb5/krb5.keytab
Check that the page is visible while you have a ticket with a client which supports HTTP Negotiate (and also check it isn't after kdestroy)
# curl --negotiate -u: http://noc.ws.nsrc.org/secure/
If you want to test from Firefox or Chrome, see the presentation.
NOTE: testing suggests that modauthnzldap only does anonymous binds or fixed simple binds (i.e. I couldn't get it to use SASL, even if the www-data user has a ticket). We can gloss over this for now.

Configure master LDAP server

This is by far the hardest part of the operation, due to the cryptic way OpenLDAP 2.4 now stores its configs within LDAP instead of in a config file. See man slapd-config
There is a good series of articles here: http://www.opinsys.fi/en/setting-up-openldap-on-ubuntu-10-04-alpha2 But there are many others which user openldap <2.4 and slapd.conf
Install the slapd server and Kerberos bits:
# apt-get install slapd ldap-utils libsasl2-modules-gssapi-mit
Set up the service principal with keytab readable by slapd:
# mkdir /etc/ldap/krb5
# kadmin -p nsrc/admin
addprinc -randkey ldap/noc.ws.nsrc.org
ktadd -k /etc/ldap/krb5/krb5.keytab ldap/noc.ws.nsrc.org
^D
# chown -R openldap:openldap /etc/ldap/krb5
# chmod 550 /etc/ldap/krb5
# chmod 440 /etc/ldap/krb5/krb5.keytab
# editor /etc/default/slapd
...
export KRB5_KTNAME=/etc/ldap/krb5/krb5.keytab

# service slapd restart
Install the schemas we need:
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/cosine.ldif
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/nis.ldif
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/inetorgperson.ldif
# ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/ldap/schema/misc.ldif
NOTE: For a real production LDAP server, read the files in /usr/share/doc/slapd carefully, especially README.Debian.gz and README.DB_CONFIG.gz
Now we need to run some scripts. First is create_database.sh
ldapadd -Y EXTERNAL -H ldapi:/// <
(Note: we have no olcRootDN or olcRootPW. This is a pure Kerberos config)
Next is init_database.sh
ldapadd -Y EXTERNAL -H ldapi:/// <
And finally config.sh
# Because these are 'replace' operations it's safe to modify this script
# and re-run it
ldapmodify -Y EXTERNAL -H ldapi:/// <
The following tweak disables unused sasl mechanisms, and means that -Y GSSAPI can be omitted form the ldapsearch command line.
# editor /etc/ldap/sasl2/slapd.conf
mech_list: gssapi external

Test LDAP

(On the same machine, or on a separate client)
# apt-get install ldap-utils libsasl2-modules-gssapi-mit
# editor /etc/ldap/ldap.conf
BASE    dc=ws,dc=nsrc,dc=org
URI     ldap://ldap.ws.nsrc.org

$ kinit nsrc
$ ldapsearch [-Y GSSAPI] [-b "dc=ws,dc=nsrc,dc=org"]
This should dump back the (mostly empty) LDAP database, as long as you have a Kerberos ticket

Create user

We are going to create a central user called "ldapuser"
Firstly in Kerberos:
$ kadmin -p nsrc/admin
addprinc ldapuser
... password: foo123bar
^D
And now in LDAP:
$ kinit nsrc/admin
$ ldapadd <

Test NSS

Now configure LDAP nscd/nsswitch, as per part 2 of exercise 2.
To verify all is working:
$ id ldapuser
You should see a result with the uid and gid. Repeated operations should be quick as nscd will be caching them.
Troubleshooting:
  • Make sure clock is synced
  • Make sure you have a kerberos ticket for root. Re-run /etc/cron.hourly/kerberos if necessary.
  • Restart nscd
  • If you stop nscd, then 'id ' will make an LDAP query as the current user. Make sure that the current user has a kerberos ticket.
For easier management you can install the ldapscripts package, but see the patches at the end of exercise 4.

Configure backup LDAP

OpenLDAP replication is left as an exercise to the reader. You can configure round-robin DNS for ldap.ws.nsrc.org and it should work.

***************************************************************************


Nagios Installation and Configuration

Notes:
------
* Commands preceded with "$" imply that you should execute the command as
  a general user - not as root.
* Commands preceded with "#" imply that you should be working as root.
* Commands with more specific command lines (e.g. "RTR-GW>" or "mysql>") 
  imply that you are executing commands on remote equipment, or within 
  another program.

Exercises
---------

Exercises Part I
----------------

0. Log in to your PC or open a terminal window as the sysadm user.

1. You may need to install Nagios version 3. You would do this as root or as the sysadmin
   user and use the "sudo" command. As sysadm:

   $ sudo apt-get install nagios3

   Unless you already have an MTA installed, nagios3 will install
   postfix as a dependency. Select "Internet Site" option. (If you had wanted
   to use a different MTA likely you'd install it before nagios3)

   You will be prompted for nagiosadmin password. Give it the normal
   workshop password.

   To get the documentation in /usr/share/doc/nagios3-doc/html/ (which
   can also be read via the nagios web interface), do:

    $ sudo apt-get install nagios3-doc

   Now lets install the Apaceh web server so that you'll be running some additional
   services for us to monitor.

    $ sudo apt-get install apache2


2. Look at the file which contains the password. It's hashed (encrypted)

    $ cat /etc/nagios3/htpasswd.users


3. You should already have a working Nagios!

    - Open a browser, and go to

    http://pcN.ws.nsrc.org/nagios3/

 Check with the instructor or your neighbor if you are in doubt.

    - At the login prompt, login as:

        user: nagiosadmin
        pass: 

    Browse to the "Host Detail" page to see what's already configured.


4. Let's look at the configuration layout... But, first, let's become the root
   user on your machine:

    $ sudo bash

    # cd /etc/nagios3
    # ls -l 

    -rw-r--r-- 1 root root    1882 2008-12-18 13:42 apache2.conf
    -rw-r--r-- 1 root root   10524 2008-12-18 13:44 cgi.cfg
    -rw-r--r-- 1 root root    2429 2008-12-18 13:44 commands.cfg
    drwxr-xr-x 2 root root    4096 2009-02-14 12:33 conf.d
    -rw-r--r-- 1 root root      26 2009-02-14 12:36 htpasswd.users
    -rw-r--r-- 1 root root   42539 2008-12-18 13:44 nagios.cfg
    -rw-r----- 1 root nagios  1293 2008-12-18 13:42 resource.cfg
    drwxr-xr-x 2 root root    4096 2009-02-14 12:32 stylesheets

    # cd conf.d
    # ls -l    

    -rw-r--r-- 1 root root 1695 2008-12-18 13:42 contacts_nagios2.cfg
    -rw-r--r-- 1 root root  418 2008-12-18 13:42 extinfo_nagios2.cfg
    -rw-r--r-- 1 root root 1152 2008-12-18 13:42 generic-host_nagios2.cfg
    -rw-r--r-- 1 root root 1803 2008-12-18 13:42 generic-service_nagios2.cfg
    -rw-r--r-- 1 root root  210 2009-02-14 12:33 host-gateway_nagios3.cfg
    -rw-r--r-- 1 root root  976 2008-12-18 13:42 hostgroups_nagios2.cfg
    -rw-r--r-- 1 root root 2167 2008-12-18 13:42 localhost_nagios2.cfg
    -rw-r--r-- 1 root root 1005 2008-12-18 13:42 services_nagios2.cfg
    -rw-r--r-- 1 root root 1609 2008-12-18 13:42 timeperiods_nagios2.cfg

    Notice that the package installs files with "nagios2" in their name.
    This is because they are the same files as were used for the Nagios
    version 2 Debian package. However there was a change made to the
    host-gateway configuration file, so this has a new name.


5. You have a config which is already monitoring your own system
(localhost_nagios2.cfg) and your upstream default gateway
(host-gateway_nagios3.cfg).

Have a look at the config file for the default gateway: it's very simple.
(Note: tab completion is useful here. Type cat host-g then hit tab; the
filename will be filled in for you)

    # cat host-gateway_nagios3.cfg

    # a host definition for the gateway of the default route
    define host {
            host_name   gateway
            alias       Default Gateway
            address     10.10.0.254
            use         generic-host
            }



PART II
Configuring Equipment
-----------------------------------------------------------------------------

0. Order of configuration

Conceptually we will build our configuration files from the "nearest" device
then the further away ones.

By going in this order you will have defined the devices that act as parents
for other devices.

The classroom GW router is already defined (10.10.0.254).

1. First we need to tell Nagios to monitor the gateway for the router instances,
   which is 10.10.254.254 or gw-254.ws.nsrc.org.

   # cd /etc/nagios3/conf.d/

Create the routers gateway like this:

   # editor routers.cfg

define host {
    use         generic-host
    host_name   gw-254
    alias       Routers Gateway
    address     10.10.254.254
    parents     gateway
}

Exit and save this file.

*NOTE* - "gateway" is the same machine as gw.ws.nsrc.org. Nagios has simply given
this machine the name "gateway". While it's nice to have the host_name mactch the 
name in DNS it is not strictly necessary. This will be our only exception.


2. the final parent we have in our network is our backbone switch. Create
   a file called switches.cfg and add an entry for this item:

   # editor switches.cfg

define host {
    use         generic-host
    host_name   sw
    alias       Backbone Switch
    address     10.10.0.253
    parents     gateway
}

At this point Nagios is configured to monitor whether our core hosts (the parents) 
are up on our classroom network. Your next steps are to add in the individual hosts
such as the classroom virtual PC images (pc1 to pc26), the Wireless Access Points
(ap1 and ap2), the virtual router images (r1 through r26) and the classroom noc
host.

Be sure you add in a proper "parents" entry for each host.

To understand the parent relationship in our network review the logical 
network diagram located here:

 http://nocws.nsrc.org/wiki/wiki/NetworkDiagram

Note the Nagios parent bullet points:

Nagios Parent Relationships 

Parents are "gw", "sw" and "gw-254". The parent relations are:

    * gw is the parent of sw and gw-254
    * gw-254 is the parent of r1 through r26
    * sw is the parent of s0, s1, s2, ap1, ap2, noc and pc1 through pc26 



STEPS 2a - 2c SHOULD BE REPEATED WHENEVER YOU UPDATE THE CONFIGURATION!
    

2a. Verify that your configuration files are OK:

    # nagios3 -v /etc/nagios3/nagios.cfg 

    ... You should get :
Warning: Host 'bb-sw' has no services associated with it!
Warning: Host 'bb-gw' has no services associated with it!
...
Total Warnings: 2
Total Errors:   0

Things look okay - No serious problems were detected during the check.
Nagios is saying that it's unusual to monitor a device just for its
existence on the network, without also monitoring some service.


2b. Reload/Restart Nagios

    # /etc/init.d/nagios3 restart

Not always 100% reliable to use the "restart" option due to a bug in the Nagios init script.
To be sure you may want to get used to doing:

    # /etc/init.d/nagios3 stop
    # /etc/init.d/nagios3 start


2c. Go to the web interface (http://pcN.ws.nsrc.org/nagios3) and check that the hosts 
   you just added are now visible in the interface. Click on the "Host Detail" item
   on the left of the Nagios screen to see this. You may see it in "PENDING"
   status until the check is carried out.


HINT: You will be doing this a lot. If you do it all on one line, like this,
then you can hit cursor-up and rerun all in one go:

    nagios3 -v /etc/nagios3/nagios.cfg && /etc/init.d/nagios3 restart

The '&&' ensures that the restart only happens if the config is valid.


3. Create entries for ther routers and PCs in the classroom

Now that we have our routers and switches defined it is quite easy to create
entries for all our PCs.  Think about the parent relationships:

Remember, if you do not understand the parent relationship refer back to the
classroom network diagram here:

 http://noc.ws.nsrc.org/wiki/wiki/NetworkDiagram

Below are three sample entries. One for the NOC, one for pc1 and one for
pc26.  You should be able to use this example to create entries for all
classroom PCs plus the NOC.

We could put these entries in to separate files, but as our network is small
we'll use a single file called pcs.cfg.

NOTE! You do not add in an entry for your own PC or router. This has already
been defined in the file /etc/nagios3/conf.d/localhost_nagios2.cfg.  This
definition is what defines the Nagios network viewpoint. So, when you come to
the spot where you might add an entry for your PC you should skip this and go
on to the next PC in the list.

 # editor pcs.cfg
 
# Our classroom NOC

define host {
    use         generic-host
    host_name   noc
    alias       Workshop NOC machine
    address     10.10.0.250
    parents     sw
}

# PCs

define host {
    use         generic-host
    host_name   pc1
    alias       pc1
    address     10.10.0.1
    parents     sw
}

define host {
    use         generic-host
    host_name   pc26
    alias       pc26
    address     10.10.0.26
    parents     sw
}

Take the three entries above and now expand this to create the remaining
entries for all active PCs. That is, fill in for PCs 2 through 25 (rememember to 
skip your PC).


Exit and save the file pcs.cfg

As before, repeat steps 2a-2c to verify your configuration, correct any
errors, and activate it.



4. Now configure Nagios to start monitoring the multiple router instances
   we have available. These are from 10.10.254.1 through 10.10.254.26.

Let's create the first router in our file called routers.cfg. Add this line
to the bottom of the file:

define host {
    use         generic-host
    host_name   r1
    alias       router 1
    address     10.10.254.1
    parents     gw-254
}

Now create the remaining routers 2-26. Or, just create a few if you don't
want to spend too long on this particular part of the exercise. But, remember
which router instances you have defined!

Second router:

define host {
    use         generic-host
    host_name   r2
    alias       router 2
    address     10.10.254.2
    parents     gw-254
}

Repeat this until router number 26:

define host {
    use         generic-host
    host_name   r26
    alias       router 26
    address     10.10.254.26
    parents     gw-254
}

Save the file.


5. Look at your Nagios instance on the web. Note that "Status Map" gives
you a graphical view of the parent-child relationships you have defined.


PART III
Configure Service check for the classroom NOC
-----------------------------------------------------------------------------

0. Configuring

Now that we have our hardware configured we can start telling Nagios what services to monitor
on the configured hardware, how to group the hardware in interesting ways, how to group
services, etc.

1. Associate a service check for our classroom NOC

    # joe hostgroups_nagios2.cfg

    - Find the hostgroup named "ssh-servers". In the members section of the defintion 
      change the line:

members   localhost

    to

members   localhost,noc

Exit and save the file. 

Verify that your changes are OK:

 # nagios3 -v /etc/nagios3/nagios.cfg
 
Restart Nagios to see the new service assocation with your host:

 # /etc/init.d/nagios3 restart

Click on the "Service Detail" link in the Nagios web interface to see your new entry.


PART IV
Defining Services for all PCs
-----------------------------------------------------------------------------

0. For services, the default normal_check_interval is 5 (minutes) in
   generic-service_nagios2.cfg. You may wish to change this to 1 to speed up
   how quickly service issues are detected, at least in the workshop.

1. Determine what services to define for what devices

   - This is core to how you use Nagios and network monitoring tools in 
     general. So far we are simply using ping to verify that physical hosts
     are up on our network and we have started monitoring a single service on
     a single host (your PC). The next step is to decide what services you wish
     to monitor for each host in the classroom.

   - In this particular class we have:

     routers:  running ssh and snmp 
     switches: running telnet and possibly ssh as well as snmp
     pcs:      All PCs are running ssh and http and should be running snmp
               The NOC is currently running an snmp daemon
              
     So, let's configure Nagios to check for these services for these 
     devices.

2.) Verify that SSH is running on the routers and workshop PCs images

   - In the file services_nagios2.cfg there is already an entry for the SSH 
     service check, so you do not need to create this step. Instead, you 
     simply need to re-define the "ssh-servers" entry in the file
     /etc/nagios3/conf.d/hostgroups_nagios2.cfg. The initial entry in the file
     looked like:

# A list of your ssh-accessible servers
define hostgroup {
        hostgroup_name  ssh-servers
                alias           SSH servers
                members         localhost,noc
        }

     What do you think you should change? Correct, the "members" line. You should
     add in entries for all the classroom pcs, routers and  the switches that run ssh. 
     With this information and the network diagram you should be able complete this entry.
     
     The entry will look something like this:

define hostgroup {
        hostgroup_name  ssh-servers
                alias           SSH servers
                members         localhost,pc1,pc2,pc3,pc4....,noc,ap1,ap2,r1,r2,r3....
        }

    Note: leave in "localhost" - This is your PC and represents Nagios' network point of
    view. So, for instance, if you are on "pc3" you would not include "pc3" in the list
    of all the classroom pcs as it is represented by the "localhost" entry.
    
    The "members" entry will be a long line and will likely wrap on the screen.

         Remember to include all your PCs and all your routers that you have defined. Do no
         include any entries if they are not already defined in pcs.cfg, switches.cfg or
         routers.cfg.

    - Once you are done, run the pre-flight check:

    # nagios3 -v /etc/nagios3/nagios.cfg

    If everything looks good, then restart Nagios 

    # /etc/init.d/nagios3 stop
    # /etc/init.d/nagios3 start

    and view your changes in the Nagios web interface.

3.) Check that http is running on all the classroom PCs. 

    - This is almost identical to the previous exercise. Just make the change to the
      HTTP service adding in each PC (no routers or switches). Remember, you don't need
      to add your machine as it is already defined as "localhost".      

4.)  OPTIONAL EXTRA: as opposed to just checking that a web server is
     running on the classroom PCs, you could also check that the nagios3
     service is available, by requesting the /nagios3/ path. This means
     passing extra options to the check_http plugin.

     For a description of the available options, type this:

      # /usr/lib/nagios/plugins/check_http
      # /usr/lib/nagios/plugins/check_http --help

     and of course you can browse the online nagios documentation or google
     for information on check_http. You can even run the plugin by hand to
     perform a one-shot service check:

     # /usr/lib/nagios/plugins/check_http -H localhost -u /nagios3/

     So the goal is to configure nagios to call check_http in this way.

define command{
        command_name    check_http_arg
        command_line    /usr/lib/nagios/plugins/check_http -H '$HOSTADDRESS$' $ARG1$
        }

define service {
        hostgroup_name                  nagios-servers
        service_description             NAGIOS
        check_command                   check_http_arg!-u /nagios3/
        use                             generic-service
}

     and of course you'll need to create a hostgroup called nagios-servers to
     link to this service check.

     Once you have done this, check that Nagios warns you about failing
     authentication (because it's trying to fetch the page without providing
     the username/password). There's an extra parameter you can pass to
     check_http_arg to provide that info, see if you can find it.

      WARNING: in the tradition of "Debian Knows Best", their definition of the
      check_http command in /etc/nagios-plugins/config/http.cfg
      is *not* the same as that recommended in the nagios3 documentation.
      It is missing $ARG1$, so any parameters to pass to check_http are
      ignored. So you might think you are monitoring /nagios3/ but actually
      you are monitoring root!

     This is why we had to make a new command definition "check_http_arg".
     You could make a more specific one like "check_nagios", or you could
     modify the Ubuntu check_http definition to fit the standard usage.



PART V
Create More Host Groups
-----------------------------------------------------------------------------

0. In the web view, look at the pages "Hostgroup Overview", "Hostgroup
   Summary", "Hostgroup Grid". This gives a convenient way to group together
   hosts which are related (e.g. in the same site, serving the same purpose).

1. Update /etc/nagios3/conf.d/hostgroups_nagios2.cfg

    - For the following exercises it will be very useful if we have created
      or update the following hostgroups:

      debian-servers
      routers
      switches
 
      If you edit the file /etc/nagios3/conf.d/hostgroups_nagios2.cfg you
      will see an entry for debian-servers that just contains localhost. 
      Update this entry to include all the classroom PCs, including the
      noc (this assumes that you created a "noc" entry in your pcs.cfg
      file). Remember to skip your PC entry as it is represented by the
      localhost entry.

    # editor /etc/nagios3/conf.d/hostgroups_nagios2.cfg

     Update the entry that says:


# A list of your Debian GNU/Linux servers
define hostgroup {
        hostgroup_name  debian-servers
                alias           Debian GNU/Linux Servers
                members         localhost
        }
      
      So that the "members" parameter contains something like this. Use your
      classroom network diagram to confirm the exact number of machines and names
      in your workshop.

                members         localhost,pc1,pc2,pc3,pc4,pc5,pc6,pc7,pc8,pc9
                                pc10,pc11,pc12,pc13,pc14,pc15,pc16,pc17,pc18,
                                pc19,pc20,pc21,pc22,pc23,pc24,pc25,pc26

        Be sure that the line wraps and is not on two separate lines. Otherwise
        you will get an error when you go to restart Nagios. Remember that
        your own PC is "localhost".

      - Once you have done this, add in two more host groups, one for routers and 
        one for switches. Call these entries "routers" and "switches".

      - When you are done be sure to verify your work and restart Nagios.
  
2. Go back to the web interface and look at your new hostgroups


PART VI
Extended Host Information ("making your graphs pretty")
-----------------------------------------------------------------------------

1. Update extinfo_nagios2.cfg 

    - If you would like to use appropriate icons for your defined hosts in
      Nagios this is where you do this. We have the three types of devices:

      Cisco routers
      Cisco switches
      Ubuntu servers

      There is a fairly large repository of icon images available for you to
      use located here:

      /usr/share/nagios/htdocs/images/logos/

      these were installed by default as dependent packages of the nagios3
      package in Ubuntu. In some cases you can find model-specific icons for
      your hardware, but to make things simpler we will use the following 
      icons for our hardware:

      /usr/share/nagios/htodcs/images/logos/base/debian.*
      /usr/share/nagios/htdocs/images/logos/cook/router.*
      /usr/share/nagios/htdocs/images/logos/cook/switch.*

    - The next step is to edit the file /etc/nagios3/conf.d/extinfo_nagios2.cfg
      and tell nagios what image you would like to use to represent your devices.

    # editor /etc/nagios3/conf.d/extinfo_nagios2.cfg

      Here is what an entry for your routers looks like (there is already an entry
      for debian-servers that will work as is). Note that the router model (3600) 
      is not all that important. The image used represents a router in general.

define hostextinfo {
        hostgroup_name   routers 
        icon_image       cook/router.png
        icon_image_alt   Cisco Routers (3600) 
        vrml_image       router.png 
        statusmap_image  cook/router.gd2
}

      Now add an entry for your switches. Once you are done check your
      work and restart Nagios. Take a look at the Status Map in the web interface.
      It should be much nicer, with real icons instead of question marks.


PART VII
Create Service Groups
-----------------------------------------------------------------------------

1. Create service groups for ssh and http for each set of pcs.

   - The idea here is to create three service groups. Each service group will
     be for a quarter of the classroom. We want to see these PCs grouped together
     and include status of their ssh and http services. To do this edit
     and create the file:

   # editor /etc/nagios3/conf.d/servicegroups.cfg

     Here is a sample of the service group for group 1:

define servicegroup {
 servicegroup_name group1-servers
 alias   group 1 servers
 members   pc1,SSH,pc1,HTTP,pc2,SSH,pc2,HTTP,pc3,SSH,pc3,HTTP,pc4,SSH,pc4
        }

 - Note that the members line should wrap and not be on two lines.
 
 - Note that "SSH" and "HTTP" need to be uppercase as this is how the service_description is
   written in the file /etc/nagios3/conf.d/services_nagios2.cfg
   
 - You should create an entry for other groups of servers too

    - Save your changes, verify your work and restart Nagios. Now if you click on
      the Servicegroup menu items in the Nagios web interface you should see
      this information grouped together. 



PART VIII
Configure Guest Access to the Nagios Web Interface
-----------------------------------------------------------------------------

1. Edit /etc/nagios3/cgi.cfg to give read-only guest user access to the Nagios 
   web interface.

    - By default Nagios is configured to give full r/w access via the Nagios
      web interface to the user nagiosadmin. You can change the name of this
      user, add other users, change how you authenticate users, what users
      have access to what resources and more via the cgi.cfg file.

    - First, lets create a "guest" user and password in the htpasswd.users
      file.
      
    # htpasswd /etc/nagios3/htpasswd.users guest

      You can use any password you want (or none). A password of "guest" is 
      not a bad choice.

    - Next, edit the file /etc/nagios3/cgi.cfg and look for what type of access
      has been given to the nagiosadmin user. By default you will see the following 
      directives (note, there are comments between each directive):

      authorized_for_system_information=nagiosadmin
      authorized_for_configuration_information=nagiosadmin
      authorized_for_system_commands=nagiosadmin
      authorized_for_all_services=nagiosadmin
      authorized_for_all_hosts=nagiosadmin
      authorized_for_all_service_commands=nagiosadmin
      authorized_for_all_host_commands=nagiosadmin

      Now let's tell Nagios to allow the "guest" user some access to 
      information via the web interface. You can choose whatever you would
      like, but what is pretty typical is this:

      authorized_for_system_information=nagiosadmin,guest
      authorized_for_configuration_information=nagiosadmin,guest
      authorized_for_system_commands=nagiosadmin
      authorized_for_all_services=nagiosadmin,guest
      authorized_for_all_hosts=nagiosadmin,guest
      authorized_for_all_service_commands=nagiosadmin
      authorized_for_all_host_commands=nagiosadmin

    - Once you make the changes, save the file cgi.cfg, verify your 
      work and restart Nagios. 

    - To see if you can log in as the "guest" user you may need to clear 
      the cookies in your web browser. You will not notice any difference
      in the web interface. The difference is that a number of items that
      are available via the web interface (forcing a service/host check, 
      scheduling checks, comments, etc.) will not work for the guest 
      user.


OPTIONAL
--------

* Check that SNMP is running on the classroom NOC

    - First you will need to add in the appropriate service check for SNMP in the file
      /etc/nagios3/conf.d/services_nagios2.cfg. This is where Nagios is impressive. There
      are hundreds, if not thousands, of service checks available via the various Nagios
      sites on the web. You can see what plugins are installed by Ubuntu in the nagios3
      package that we've installed by looking in the following directory:

    # ls /usr/lib/nagios/plugins

      As you'll see there is already a check_snmp plugin available to us. If you are 
      interested in the options the plugin takes you can execute the plugin from the
      command line by typing:

    # /usr/lib/nagios/plugins/check_snmp
    # /usr/lib/nagios/plugins/check_snmp --help 

      to see what options are available, etc. You can use the check_snmp plugin and
      Nagios to create very complex or specific system checks.

    - Now to see all the various service/host checks that have been created using the
      check_snmp plugin you can look in /etc/nagios-plugins/config/snmp.cfg. You will
      see that there are a lot of preconfigured checks using snmp, including:

      snmp_load
      snmp_cpustats
      snmp_procname
      snmp_disk
      snmp_mem
      snmp_swap
      snmp_procs
      snmp_users
      snmp_mem2
      snmp_swap2
      snmp_mem3
      snmp_swap3
      snmp_disk2
      snmp_tcpopen
      snmp_tcpstats
      snmp_bgpstate
      check_netapp_uptime
      check_netapp_cupuload
      check_netapp_numdisks
      check_compaq_thermalCondition
      
      And, even better, you can create additional service checks quite easily.
      For the case of verifying that snmpd (the SNMP service on Linux) is running we
      need to ask SNMP a question. If we don't get an answer, then Nagios can assume
      that the SNMP service is down on that host. When you use service checks such as
      check_http, check_ssh and check_telnet this is what they are doing as well.

    - In our case, let's create a new service check and call it "check_system". This
      service check will connect with the specified host, use the private community 
      string we have defined in class and ask a question of snmp on that ask - in this
      case we'll ask about the System Description, or the OID "sysDescr.0" -

    - To do this start by editing the file /etc/nagios-plugins/config/snmp.cfg:

    # joe /etc/nagios-plugins/config/snmp.cfg

      At the top (or the bottom, your choice) add the following entry to the file:

# 'check_system' command definition
define command{
       command_name    check_system
       command_line    /usr/lib/nagios/plugins/check_snmp -H '$HOSTADDRESS$' -C
'$ARG1$' -o sysDescr.0
        }
      
      You may wish to copy and paste this vs. trying to type this out.

   Note that "command_line" is a single line. If you copy and paste in joe the line
   may not wrap properly and you may have to manually add the part:
   
     '$ARG1$' -o sysDescr.0
     
   to the end of the line.

    - Now you need to edit the file /etc/nagios3/conf.d/services_nagios2.cfg and add
      in this service check. We'll run this check against all our servers in the 
      classroom, or the hostgroup "debian-servers"

    - Edit the file /etc/nagios3/conf.d/services_nagios2.cfg

    # joe /etc/nagios3/conf.d/services_nagios2.cfg

      At the bottom of the file add the following definition:

# check that snmp is up on all servers
define service {
        hostgroup_name                  snmp-servers
        service_description             SNMP
        check_command                   check_system!xxxxxx
        use                             generic-service
        notification_interval           0 ; set > 0 if you want to be renotified
}

      The "xxxxxx" is the community string previously (or to be) defined in class.
     
      Note that we have included our private community string here vs. hard-coding
      it in the snmp.cfg file earlier. You must change the "xxxxx" to be the snmp
      community string given in class or this check will not work.
      
    - Now we must create the "snmp-servers" group in our hostgroups_nagios2.cfg file. 
      Edit the file /etc/nagios3/conf.d/hostgroups_nagios2.cfg and go to the end of the
      file. Add in the following hostgroup definition:
      
# A list of snmp-enabled devices on which we wish to run the snmp service check
define hostgroup {
    hostgroup_name snmp-servers
            alias snmp servers
            members noc
   }
   
 - Note that for "members" you could, also, add in the switches and routers for
   group 1 and 2. But, the particular item (MIB) we are checking for "sysDescr.0"
   may not be available on the switches and/or routers, so the check would then fail.

    - Now verify that your changes are correct and restart Nagios.

    - If you click on the Service Detail menu choice in web interface you should see
      the SNMP check appear for the noc host.
      
    - After we do the SNMP presentation and exercises in class, then you could come
      back to this exercise and add in all the classroom PCs to the members list in the
      hostgroups_nagios2.cfg file, snmp-servers hostgroup definition. Remember to list
      your PC as "localhost".






rsync with delete option and different ssh port

How to rsync e.g PIPELINE dir from Source to Destination? #rsync -avzr   --delete-before  -e "ssh -p $portNumber"  /local...