2011
05.10

Puppet Dashboad has a concept of classes, which can be really useful if you make use of external nodes and link it to the dashboard.
Unfortunately it doesn’t currently have a way to auto-import classes defined in your puppet manifests.

The following is a little bit of python hacked together to provide this functionality. It looks at a directory for a list of modules and the database details for puppet dashboard (only works for MySQL). It will add any modules to puppet dashboard that are not already defined and remove any extras that no longer exist in your modules path.

#!/usr/bin/python
import os
import MySQLdb
import datetime
#Path to Puppet modules directory
modulesdir="/etc/puppet/modules/"
#MySQL details:
user = ""
password = ""
host = ""
database = ""
availmodules = []
currentmodules = []

#Get list of avaible modules from filesystem
for item in os.listdir(modulesdir):
if os.path.isdir(os.path.join(modulesdir,item)) and not item.startswith('.'):
availmodules.append(item)
availmodules = set(availmodules)

#Get list of current modules from database
db = MySQLdb.connect(host=host,user=user,passwd=password,db=database)
cursor = db.cursor()
cursor.execute("SELECT name FROM node_classes")
for [name] in cursor.fetchall():
currentmodules.append(name)
currentmodules = set(currentmodules)

extramodules = currentmodules - availmodules

print "Availble:\t"+str(availmodules)
print "Current:\t"+str(currentmodules)
print "Extra:\t\t"+str(extramodules)

#Add or update current availble modules
currenttime =  datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
for module in availmodules:
if module in currentmodules:
#module already exists, just update timestamp
cursor.execute("UPDATE node_classes SET updated_at = %s WHERE name = %s",(currenttime,module))
else:
#module doesn't exist, insert it
print "Adding: "+str(module)
cursor.execute("INSERT INTO node_classes (name,created_at,updated_at) vALUES (%s,%s,%s)",(module,currenttime,currenttime))
#Delete any extra modules
for module in extramodules:
print "Deleting:"+str(module)
cursor.execute("DELETE from node_classes WHERE name = %s",(module))
db.commit()
cursor.close()
db.close()

There are better ways to get a list of classes, for example the interface utils, but this requires puppet 2.6.5 at a minimum and I’m running 2.6.2.
It would also be desirable to input data into the dashboard via an API instead of directly into the database, but AFAIK this functionality isn’t available yet.

I personally run the above as part of the post-commit hook on the puppet repository. But you could also run it via cron or manually.

2011
04.12

I recently attended UKUUG’s annual Large Installation Systems Administration (LISA) conference, this year hosted in Leeds over 2 days.

Overall I was very impressed with the event, at some points I was torn with that track to go on. In the end I attended the following:

  • A bird’s-eye view on DNSSEC (Jan-Piet Mens)
  • Managing distributed web services with OpenNMS (Dr. Craig Gallen)
  • Unbound as a caching validating DNSSEC resolver (Jan-Piet Mens)
  • IPv6 update from a broadband ISP (Adrian Kennard)
  • The Future of Configuration Management — Latest developments from Cfengine (Mark Burgess)
  • FusionInventory and GLPI (Fabrice Flore-Thebault)
  • The benefit of BGP for every service provider (Thomas Mangin)
  • DNSSEC zone-signing tool chest (Jan-Piet Mens)
  • Scale out Datacenter Architecture (Bernd Erk)
  • Digital Forensics in Large Scale Environments (Si Biles)
  • Large Scale file storage with MogileFS (Stuart Teasdale)
  • OpenStack: Towards a fully open cloud (Thierry Carrez)
  • PostgreSQL: New features for Large Systems Administration (Simon Riggs)

Some of the above were very interesting, which I hope to blog about in detail in the near future.

The evening was very enjoyable and I fully enjoyed taking to like minded people. Although not a HE based conference, I was pleasantly suprised to meet fellow sysadmins from other Universities. Some as close as Bath, yes it takes a trip up to the north to meet people from a city on my doorstep.

2011
02.10

Google Apps Revisited

Yesterday Microsoft and Google came to work to present their Cloud solutions to E-mail, Calendar and Collaboration (Live@edu & Google Apps for Education). The relative pros and cons regarding a University moving it’s collaboration tools to the cloud is a debate for another day (despite it being hard to hold back). But one thing the presentations did do is make me re-consider the free google apps service for my personal set-up.

Currently I run a bespoke set-up for me and a small number of friends that provides:

  • e-mail
  • Calendar
  • Contacts
  • Web interface
  • Synchronisation to mobile devices (syncML)
  • IM (Jabber + transports)

When I first looked at Google Apps a number of years ago, there were a couple of show stoppers that have mostly been resolved:

Mail Aliases

I use a lot of mail aliases. The only way I could previously see to emulate such a feature was to create a new user with the desired name and  forward all messages to the appropriate user. This was not a scaleable solution. Now Google Apps has Groups (think mailing lists), a group with just one member is an acceptable way of implementing aliases.

The other end of this requirement is to be able to sent mail from these addresses. Previously when sending a mail using googles SMTP servers with a From header that didn’t match the address on the account, the header was modified to include “On behalf of <foo@gmail.com>”. Now it appears you can explicitly add custom from addresses (although it doesn’t require a verification step).

SyncML Support

A number of my devices support contact syncronisation by syncML only. Google did not support this, but it appears now they do. Although this does not work for calendar. How well calendar sync via ical works on different devices is yet to be seen.

What Google does that I don’t / can’t

Now that my gripes with Google Apps are mostly gone. What incentive is there to move?

Better Web Interface – There is no doubt that the GMail web interface is the bee knees (wasps elbows?). No open source webmail / collaboration suite comes close.

More Resilient – As fantastic as my service uptime may be, I’m still just one guy hosting these services out of the good of my heart. Google on the other hand is a Giant with datacenters around the world and a lot of income from advertising.

More Attributes in the Address Book - Currently my address book doesn’t support some funky attributes like photos, although I could make this happen with some effort, google does it already.

Mailing Lists – Again, mailing lists are not hard to set-up, but I never got around to it. Google Groups provides this already.

What I provide that Google does not

On the flip-side I currently provide some services that Google does not offer.

SMTP gateway – For hosts, they need to be able to send mails for reporting. Although the hosts can be configured to authenticate against a (google) account, the fact the From address can change makes googles SMTP servers unsuitable.

Jabber Transports – Currently I make use of a number of Jabber transports (MSN, IRC, etc). GTalk does not provide these transports, but it is still possible to use external transports with GTalk.

Confidence that my data isn’t going to  disappear – Before you butt in, hear me out. Many anti-google folks complain about the lack of control of their data, that Google might share it with other parties. Although this could be true, I believe the chances are so small it almost not worrying about it. If you really have such sensitive data you should be encrypting it individually . My fear is losing my account and not being able to access my data, we have all read the horror stories and they are a real concern. The fact is that Google is providing these servies for “free”, if they get a complaint about a user that could cause them trouble (legal, technical, etc) is it cheaper/easier for them to lock that user out than to investigate it. The solution to this on a individual  level is to use the ical/imap interfaces to take a daily backup. But this can not be scaled up to be automatic for other users on the same domain.

The end result is that Google offer a very attractive service for free with some features that would be impossible to provide with open-source tools and the resources available to me. For the features they don’t provide, it’s possible for me to provide them in parallel. Maybe it is time for me to become a slave of the Google Monster. For those who know me, you will know how upsetting it is for me to say that.

2011
01.30

It is widely accepted that Mice belong with Computers. Many users believe they are vital for the smooth operation of their PC. Unfortunately this fact does not expand to other rodents, particularly Rats. A major part of my workplaces networking went down, after much effort searching for the problem, we finally found the following.

Even Rats need their 5-a-day

The above is 64-Cores of fibre optic cable, rendered useless by some rats. It’s surprising the outer shell doesn’t have rat poison.

2010
08.11

The location of subversion repositories at zem.org.uk has changed. Until now all repositories have been available via svn+ssh://zem.org.uk/srv/svn/<repo> on port 222. This has now been changed to https://zem.org.uk/svn/<repo>.

What do I need to do?

If you make use of the CLI subversion client, you should be able to run the following command while in the base of your working copy:

 $ svn switch --relocate svn+ssh://zem.org.uk/srv/svn/<repo> https://zem.org.uk/svn/<repo>

For other clients, please see it’s documentation. If you are having issues switching the repository and have no local modifications it may be simpler to checkout a new working copy and remove the old copy.

Why the Change?

  • Running subversion on a non-default ssh port causes confusion for many users.
  • Subversion over https gives the possibility of finer control of permissions.
2010
06.27

Castle Dan

Leach Lan 5 is just around the corner and I’ve managed to to borrow epic tent again. With 3 large sleeping pods and a communal area, it can comfortably sleep up to 9 people.
King Dan leaving his quarters

Obviously this luxury accommodation is in great demand, so before I say yes to too many people, I’m going to try and keep track of it’s residents here. If you are not on this list but want to be a citizen of Castle Dan for the weekend, say something to me!

  1. TP2k
  2. IRConan
  3. Ragzouken
  4. 2kah
  5. Fish-Face
  6. WildFire
  7. Jennycide
  8. jakexks
  9. BattlePope

(Plus: Kelsa sleeping in the day)

2009
10.03

Kerberos

The following describes how I configured Kerberos on Debian Lenny:

Server Config:

Install the required packages:

aptitude install krb5-kdc krb5-admin-server krb5-clients libpam-krb5

edit /etc/krb5.conf:

[libdefaults]
default_realm = ZEM.ORG.UK
# The following krb5.conf variables are only for MIT Kerberos.
krb4_config = /etc/krb.conf
krb4_realms = /etc/krb.realms
kdc_timesync = 1
ccache_type = 4
forwardable = true
proxiable = true
# The following encryption type specification will be used by MIT Kerberos
# if uncommented.  In general, the defaults in the MIT Kerberos code are
# correct and overriding these specifications only serves to disable new
# encryption types as they are added, creating interoperability problems.
#
# Thie only time when you might need to uncomment these lines and change
# the enctypes is if you have local software that will break on ticket
# caches containing ticket encryption types it doesn't know about (such as
# old versions of Sun Java).
#   default_tgs_enctypes = des3-hmac-sha1
#   default_tkt_enctypes = des3-hmac-sha1
#   permitted_enctypes = des3-hmac-sha1
# The following libdefaults parameters are only for Heimdal Kerberos.
v4_instance_resolve = false
v4_name_convert = {
host = {
rcmd = host
ftp = ftp
}
plain = {
something = something-else
}
}
fcc-mit-ticketflags = true
[realms]
ZEM.ORG.UK = {
kdc = ldap.zem.org.uk
admin_server = ldap.zem.org.uk
}
[domain_realm]
.zem.org.uk = ZEM.ORG.UK
zem.org.uk = ZEM.ORG.UK
[login]
krb4_convert = true
krb4_get_tickets = false
[logging]
kdc = FILE:/var/log/kerberos/krb5kdc.log
admin_server = FILE:/var/log/kerberos/kadmin.log
default = FILE:/var/log/kerberos/krb5lib.log

create the new realm and initialise some basic users:

krb5_newrealm
kadmin.local -q "ktadd -k /etc/krb5kdc/kadm5.keytab kadmin/admin"
kadmin.local -q "ktadd -k /etc/krb5kdc/kadm5.keytab kadmin/changepw"
kadmin.local -q "addprinc dan/admin@EXAMPLE.COM"
kadmin.local -q "addprinc ldapadm@EXAMPLE.COM"

edit /etc/krb5kdc/kadm5.acl

*/admin@ZEM.ORG.UK     *
*/*@ZEM.ORG.UK          i
*@ZEM.ORG.UK           i

restart services:

# /etc/init.d/krb5-kdc restart
Restarting Kerberos KDC: krb5kdc krb524d.
# /etc/init.d/krb5-admin-server restart
Restarting Kerberos administrative servers: kadmind

Client Config:

edit /etc/pam.d/common-auth:

auth    sufficient  pam_krb5.so use_first_pass ignore_root forwardable
auth    required        pam_unix.so nullok_secure

edit /etc/pam.d/common-session:

ession    required    pam_unix.so
session         sufficient      pam_krb5.so ignore_root
2009
10.03

So I took the plunge last week and re-installed my dedicated server. This time using openVZ for OS level visualization.

My main reason for this is that more people wanted shell access for various reasons and therefore I wanted a more isolated environment. I’ve got to say I’m generally impressed, although I still don’t fully understand the use of varous user_beancounters for memory management. Maybe if I stare at this diagram long enough it will all make sense.

Hopefully I will get around to noting how I set things up, but for now I must continue moving services to openvz containers.