Operational Relevance

Relevance is important. Operational relevance is critical to being successful at your job and creating a DevOps culture.

I’ve been working to emerge from my team’s silo and tear down the walls of the development team’s silo. I’ve also been working to improve my team’s outward communication about the effects of system and service outages on the organization’s productivity. My guys are used to talking about DNS servers, virtual clusters, and Internet bandwidth. None of those means anything to non-IT folk. Non-IT folk think we speak Greek and communication breaks down. As a result, we may not be in the loop on current and future projects (see slide 9 from 10+ Deploys Per Day; Ops Stereotype).

To help illustrate to my team what it means when we improve our communications about outages, I drew up this little XKCD-like sketch (via Skitch):

The point from the sketch is that we need to translate what and who are impacted by an outage, not just that an outage occurred.

My goal: Demonstrate operational relevance to the organization and increase the likelihood of being involved in all projects.

DevOps at Etsy

At this year’s PICC, I sat in on Michael Rembetsy’s (@mrembetsy) DevOps talk. I thoroughly enjoyed the session for two reasons:

  1. It affirmed for me that I’m heading in the right direction in my own organization, with respect to bridging development and operations.
  2. I recognized I’ve got a lot more work to do and I learned about new tools to help me get there (i.e. exposing what’s happening with our applications and systems to the developers).

I was inspired by Mike’s overview of the DevOps culture at Etsy and began researching them more to understand how they do what they do. Mike mentioned the leadership from John Allspaw and I came across this 3-year old presentation:

10 Deploys per Day

It’s still relevant today.

In all, Mike’s session energized me to continue forward and keep fighting the good fight. Hoo-rah!

Toddlers and TCP

While having a lengthy conversation with my 3-year old son one day, it occurred to me: TCP models toddler communication. For example, I’ve noticed that when my son speaks to me, if I don’t acknowledge him (immediately), he repeats himself:

Son: Daddy, what’s that?
Me: …
Son: Daddy, what’s that?
Me: …
Son: Daddy, what’s that?
Me: Oh, that’s a pony.

At which point, he moves on to the next query. Without the ACK, he will retransmit until his message gets through. He’s quite persistent.

Superstition as Policy

Sysadmins are a superstitious bunch. Even when we know better. Unfortunately, we’ve each had one of those experiences that causes us to wear that tired, 15-year old T-shirt every 3rd Thursday after a blue moon. Because if we don’t, the mail server might crash. It’s happened. At least twice.

We’re superstitious enough that it is embedded within our culture in some way shape or form. Usually, a policy is implemented that firmly institutes that superstition. It may be written or unwritten, but it’s there for a reason. For me, there is never any maintenance, upgrades, or system shutdowns prior to any major holiday or just before the person who would do the work goes on vacation. Too many times I’ve been bitten in that scenario and as a result, my superstition has become policy. Even with proper planning, somehow, some way, the universe wreaks havoc on my weekend if I don’t burn an offering to the switch rack or sacrifice a stuffed Tux on that 10-year old Windows 2000 server lurking in the dark corner of the data center.

I feel a blue moon coming…

Terms of Service Genius: 500px

File-sharing service 500px.com has one of the best terms of service (ToS) pages I’ve seen so far:

http://500px.com/terms

The legalese is there, to keep the law hounds at bay. And the terms are accompanied by plain Engish summaries. Genius.

I’d keep my eye out for flying computers, though (“Limitation Of Liability” section). It could net you $100!

OpenLDAP: Implementing the Password Policy Overlay

Sometimes I tend to be verbose. Click here to get to the meat of the post. Otherwise, read on.

It’s been some time since I configured OpenLDAP but I decided to use it on a new project I’m working on. Prior to this new effort, I used a simple database table to store user credentials because my requirements were, well, simple. Now, I need to extend the management capabilities of my project and delegate access according to group membership. In addition, I want to implement basic password management functions such as minimum password length, aging, etc. OpenLDAP’s password policy overlay is perfect for this second criterion.

Only, something is missing: documentation. How I loathe missing documentation. The only thing worse is incorrect, incomplete, or out-of-date documentation. That’s exactly what I found myself with when attempting to configure the password policy overlay for OpenLDAP using the the dynamic configuration backend: cn=config.

For more information on overlays, see http://www.openldap.org/doc/admin24/overlays.html. You’ll note, unfortunately, that the documentation for the password policy overlay still references the slapd.conf-method of configuring that overlay.

NOTE: I used slaptest to convert a temporary instance of slapd.conf into the cn=config backend objects I needed. Perhaps I’ll post more on that process later.

I dig the cn=config methodology. It makes sense to me and I’ve had no issues implementing custom backends for new directory instances. However, there is a dearth of clear, concise information available on how to configure the password policy overlay using the dynamic cn=config backend. Until now.

Assumptions

To get started, I’ll tell you what my assumptions are and what we’ll end up with:

  1. I’ll assume (onerously, conceitedly) you’re working on the same platform as me (AWS Micro Instance) running a generic Linux AMI.
  2. I’ll assume you’re familiar with the ldap* and slap* commands.
  3. I’ll assume that you like the name of my directory instance: snafoobar.com.
  4. You’ll end up with a working, simple password policy overlay that is used to create a default password policy for one of your directory instances.

The Meat

For this example, the configuration lives in /etc/openldap/slapd.d/cn=config/. Our directory instance is specified in a file named olcDatabase={3}snafoobar.com.ldif:

/etc/openldap/slapd.d/cn=config/
|
`-- olcDatabase={3}snafoobar.com.ldif

File Preparation

To create and configure the password policy overlay, start by creating a few files:

  • cn=module{0}.ldif: The attributes in this file will tell LDAP which library to pull in so that the password policy overlay support is available.
  • Create a directory named olcDatabase={3}snafoobar.com/ in the root of the cn=config directory. This directory is named similar to the LDIF file used to define the directory instance; this is not an accident.
  • olcDatabase={3}snafoobar.com/olcOverlay={0}ppolicy.ldif: This LDIF file will contain the necessary configuration to define that we’re implementing a password policy overlay, and, in turn, points to the actual password policy we will define..

The resulting file hierarchy is as follows:

/etc/openldap/slapd.d/cn=config/
|
|-- cn=module{0}.ldif
|-- olcDatabase={3}snafoobar.com
|   `-- olcOverlay={0}ppolicy.ldif
`-- olcDatabase={3}snafoobar.com.ldif

cn=module{0}.ldif

cn=module{0}.ldif specifies which shared library LDAP should load to support the password policy overlay. Its contents should be as follows:

dn: cn=module{0}
objectClass: olcModuleList
cn: module{0}
olcModuleLoad: {0}ppolicy.la
structuralObjectClass: olcModuleList

olcOverlay={0}ppolicy.ldif

olcOverlay={0}ppolicy.ldif should have the following attributes defined:

dn: olcOverlay={0}ppolicy
objectClass: olcOverlayConfig
objectClass: olcPPolicyConfig
olcOverlay: {0}ppolicy
olcPPolicyDefault: cn=passwordDefault,ou=policies,dc=snafoobar,dc=com
olcPPolicyHashCleartext: FALSE
olcPPolicyUseLockout: FALSE
olcPPolicyForwardUpdates: FALSE
structuralObjectClass: olcPPolicyConfig

NOTE: I’ve already created the ou=policies,dc=snafoobar,dc=com organizational unit in my directory instance. You’ll need to do the same to follow this example.

Restart slapd

Do it. Restart slapd. I’ll wait.

passwordPolicy.ldif

You’ll need to create an LDIF file that will contain the default password policy definition. I didn’t mention this above as this file is not part of the cn=config hierarchy. Create a file named passwordPolicy.ldif (anywhere, except within the cn=config/ hierarchy) and add the following contents:

dn: cn=passwordDefault,ou=policies,dc=snafoobar,dc=com
objectClass: pwdPolicy
objectClass: person
objectClass: top
cn: passwordDefault
sn: passwordDefault
pwdAttribute: userPassword
pwdCheckQuality: 0
pwdMinAge: 0
pwdMaxAge: 0
pwdMinLength: 5
pwdInHistory: 5
pwdMaxFailure: 3
pwdFailureCountInterval: 0
pwdLockout: TRUE
pwdLockoutDuration: 0
pwdAllowUserChange: TRUE
pwdExpireWarning: 0
pwdGraceAuthNLimit: 0
pwdMustChange: FALSE
pwdSafeModify: TRUE

NOTE: I’m leaving it as an exercise to you, the reader, to determine what the above attributes are and what is appropriate for your environment. I need these attributes configured with these values. You may require something different.

Add the entry into the directory instance (my LDIF lives in ~/ldif/):

[root@awsInstance ~/ldif]# ldapadd -c -f passwordPolicy.ldif -D "cn=root,dc=snafoobar,dc=com" -x -w cleverBoysNeverTell
adding new entry "cn=passwordDefault,ou=policies,dc=snafoobar,dc=org"

If you want to confirm that the new entry exists and a default password policy has been implemented, execute an ldapsearch:
[root@awsInstance ~/ldif]# ldapsearch -x -D cn=root,dc=snafoobar,dc=com -H ldap:// -b "dc=snafoobar,dc=com" -w cleverBoysNeverTell
# extended LDIF
#
# LDAPv3
# base <dc=snafoobar,dc=com> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

...

# policies, snafoobar.com
dn: ou=policies,dc=snafoobar,dc=com
ou: policies
objectClass: organizationalUnit
description: Default directory policies

# passwordDefault, policies, snafoobar.com
dn: cn=passwordDefault,ou=policies,dc=snafoobar,dc=com
objectClass: pwdPolicy
objectClass: person
objectClass: top
cn: passwordDefault
sn: passwordDefault
pwdAttribute: userPassword
pwdCheckQuality: 0
pwdMinAge: 0
pwdMaxAge: 0
pwdMinLength: 5
pwdInHistory: 5
pwdMaxFailure: 3
pwdFailureCountInterval: 0
pwdLockout: TRUE
pwdLockoutDuration: 0
pwdAllowUserChange: TRUE
pwdExpireWarning: 0
pwdGraceAuthNLimit: 0
pwdMustChange: FALSE
pwdSafeModify: TRUE

...

Fini!

You should now have an LDAP instance with a default password policy defined. Test it out by modifying user accounts’ passwords (i.e. via your favorite LDAP browser or custom code).

PICC 2012: Here We Come!

I’m excited! I’m going to PICC 2012 and taking two of my team with me. See you all there.

Rule #29

I just came across Tom Perrine’s blog and the page Tom’s Rules. Rule #29 really resonated with me since I implemented Request Tracker (RT) 6 months ago:

If you don’t have a ticketing system, you’re doing it wrong. If you have a bad ticketing system that you or your users hate, or makes it harder to get things done, you’re doing it wronger.

RT replaced a home-grown ticketing system that had me and my team doing more work to manage the system than fulfilling the requests that came into us.

Give the rules a read; they’re worth the time to help reflect on what we do every day as sysadmins.

(Over-) Communication

I stood in as chair of a recent committee meeting at work and had the misfortune of doubling as the secretary, tasked with noting and writing up the meeting’s minutes. For convenience, I recorded the meeting so that I could transcribe it later. Worse than the tedium of the transcription was listening to myself! I couldn’t stand listening to my voice drone on. I found that I often repeated myself, albeit in new and novel ways, in an attempt to either influence a discussion or hammer an idea home in the minds of my unfortunate peers.

I shared this bit of introspection with my boss and a small grin grew on his face. He simply replied, “I know. You also say ‘generally speaking’ a lot.” Generally speaking, he was right.

Since then I have vowed to make my points concisely. And only once. Over-communicating is just as bad as under-communicating; folks start to tune you out and your message is never heard.

Generally speaking, that’s a bad thing.

IANA Enterprise Number Assignments

Once upon a time, I thought it would be important to register an enterprise number with IANA. I had grandiose ideas of creating immense MIBs, namespaces, schema, etc. (i.e. for SNMP). I was young and ideologic; I didn’t have clue one about what content would go into those schema. Still, I registered enterprise numbers because I knew I could. In a brief fit of nostalgia, I looked them up at http://www.iana.org/assignments/enterprise-numbers/:


12285
  Acclamation Systems, Inc
    Ryan Frantz
      rfrantz&acclamation.com

...

27445
  InforMed, LLC
    Ryan Frantz
      ryanfrantz&informed-llc.com

If you too would like to register an enterprise number with IANA, you can do so here:

http://pen.iana.org/pen/PenApplication.page

I highly recommend, though, that you do so only if you plan to do something with it. I never did and it serves as a reminder to me that planning is crucial to achieving my goals. At least, in that context, those numbers serve some small purpose.