Trying out initramfs with selinux and grsec

Posted on January 15, 2012 by swift

I’m no fan of initramfs. All my systems boot up just fine without it, so I often see it as an additional layer of obfuscation. But there are definitely cases where initramfs is needed, and from the looks of it, we might be needing to push out some documentation and support for initramfs. Since my primary focus is to look at a hardened system, I started playing with initramfs together with Gentoo Hardened, grSecurity and SELinux. And what a challenge it was…

But first, a quick introduction to initramfs. The Linux kernel supports initrd images for quite some time. These images are best seen as loopback-mountable images containing a whole file system that the Linux kernel boots as the root device. On this initrd image, a set of tools and scripts then prepare the system and finally switch towards the real root device. The initrd feature was often used when the root device is a network-mounted location or on a file system that requires additional activities (like an encrypted file system or even on LVM. But it also had some difficulties with it.

Using a loopback-mountable image means that this is seen as a full device (with file system on it), so the Linux kernel also tries caching the files on it, which leads to some unwanted memory consumption. It is a static environment, so it is hard to grow or shrink it. Every time an administrator creates an initrd, he needs to carefully design (capacity-wise) the environment not to request too much or too little memory.

Enter initramfs. The concept is similar: an environment that the Linux kernel boots as a root device which is used to prepare for booting further from the real root file systems. But it uses a different approach. First of all, it is no longer a loopback-mountable image, but a cpio archive that is used on a tmpfs file system. Unlike initrd, tmpfs can grow or shrink as necessary, so the administrator doesn’t need to plan the capacity of the image. And because it is a tmpfs file system, the Linux kernel doesn’t try to cache the files in memory (as it knows they already are in memory).

There are undoubtedly more advantages to initramfs, but let’s stick to the primary objective of this post: talk about its implementation on a hardened system.

I started playing with dracut, a tool to create initramfs archives which is seen as a widely popular implementation (and suggested on the gentoo development mailinglist). It uses a simple, modular approach to building initramfs archives. It has a base, which includes a small init script and some device handling (based on udev), and modules that you can add depending on your situation (such as adding support for RAID devices, LVM, NFS mounted file systems etc.)

On a SELinux system (using a strict policy, enforcing mode) running dracut in the sysadm_t domain doesn’t work, so I had to create a dracut_t domain (which has been pushed to the Portage tree yesterday). But other than that, it is for me sufficient to call dracut to create an initramfs: 

# dracut -f "" 3.1.6-hardened

My grub then has an additional set of lines like so: 

title Gentoo Linux Hardened (initramfs)
root (hd0,0)
kernel /boot/vmlinuz-3.1.6-hardened root=/dev/vda1 console=ttyS0 console=tty0
initrd /boot/initramfs-3.1.6-hardened.img

Sadly, the bugger didn’t boot. The first problem I hit was that the Linux kernel I boot has chroot restrictions in it (grSecurity). These restrictions further tighten chroot environments so that it is much more difficult to “escape” a chroot. But dracut, and probably all others, use chroot to further prepare the bootup and eventually switch to the chrooted environment to boot up further. Having the chroot restrictions enabled effectively means that I cannot use initramfs environments. To work around, I enabled sysctl support for all the chroot restrictions and made sure that their default behavior is to be disabled. Then, when the system boots up, it enables the restrictions later in the boot process (through the sysctl.conf settings) and then locks these settings (thanks to grSecurity’s grsec_lock feature) so that they cannot be disabled anymore later.

But no, I did get further, up to the point that either the openrc init is called (which tries to load in the SELinux policy and then breaks) or that the initramfs tries to load the SELinux policy – and then breaks. The problem here is that there is too much happening before the SELinux policy is loaded. Files are created (such as device files) or manipulated, chroots are prepared, udev is (temporarily) ran, mounts are created, … all before a SELinux policy is loaded. As a result, the files on the system have incorrect contexts and the moment the SELinux policy is loaded, the processes get denied all access and other privileges they want against these (wrongly) labeled files. And since after loading the SELinux policy, the process runs in kernel_t domain, it doesn’t have the privileges to relabel the entire system, let alone call commands.

This is currently where I’m stuck. I can get the thing boot up, if you temporarily work in permissive mode. When the openrc init is eventually called, things proceed as usual and the moment udev is started (again, now from the openrc init) it is possible to switch to enforcing mode. All processes are running by then in the correct domain and there do not seem to be any files left with wrong contexts (since the initramfs is not reachable anymore and the device files in /dev are now set again by udev which is SELinux aware.

But if you want to boot up in enforcing straight away, there are still things to investigate. I think I’ll need to put the policy in the initramfs as well (which has the huge downside that every update on the policy requires a rebuild of the initramfs as well). In that case I can load the policy early up the chain and have the initramfs work further running in an enforced situation. Or I completely regard the initramfs as an “always trusted” environment and wait for openrc’s init to load the SELinux policy. In that case, I need to find a way to relabel the (temporarily created) /dev entries (like console, kmsg, …) before the policy is loaded.

Definitely to be continued…

Posted in Hardened, SELinux | 9 Comments

Unix domain sockets are files

Posted on December 31, 2011 by swift

Probably not a first for many seasoned Linux administrators, and probably not correct accordingly to more advanced users than myself, but I just found out that Unix domain sockets are files. Even when they’re not.

I have been looking at a weird SELinux denial I had occuring on my system: 

avc:  denied  { read write } for  pid=10012 comm="hostname" \
path="socket:[318867]" dev=sockfs ino=318867 \
scontext=system_u:system_r:hostname_t \
tcontext=system_u:system_r:dhcpc_t \
tclass=unix_stream_socket

I had a tough time trying to figure out why in earth the hostname application was trying to read/write to a socket that was owned by dhcpcd. Even more, I didn’t see a connectto attempt, and there is nothing in my policy that would allow the hostname_t domain to connect to a unix_stream_socket of dhcpc_t. But moreover I was intrigued why the given path was no real path, even though it has an inode.

So I dug up lsof, which returned the following on this socket: 

# lsof -p 10017
COMMAND   PID USER   FD      TYPE             DEVICE SIZE/OFF   NODE NAME
...
dhcpcd  10017 root    3u     unix 0x0000000000000000      0t0 318867 socket
dhcpcd  10017 root    4w      REG              252,3        6 268749 /var/run/dhcpcd-eth1.pid
dhcpcd  10017 root    5u     unix 0x0000000000000000      0t0 318869 socket

Still no luck in figuring out what that is. And even /proc/net/unix didn’t give anything back: 

# grep 318867 /proc/net/unix
Num               RefCount Protocol Flags    Type St Inode Path
0000000000000000: 00000002 00000000 00000000 0001 01 318867

So I started looking at Unix domain sockets, what they are, how they are used, etc. And I learned that

  • Unix domain sockets are just files. Well, most of the time. To use a socket (from server-perspective), a programmer first calls socket() to create a socket descriptor, which is a special type of file descriptor. It then bind()‘s the socket to a (socket)file on the file system, listen()‘s for incoming connections and eventually accept()‘s them. Clients also use socket() but then call connectto() to have its socket connected to a (socket)file and eventually read() and write() (or send() and recv()).
  • Linux supports an abstract namespace for sockets, so not all of these are actually bound/connected to a file. Instead, they connect to a “name” instead, which cannot be traced back to a file. For those interested, looking at /proc/net/unix or netstat -xa shows the abstract ones starting with an @ sign.
  • Not all Unix sockets (actually almost the majority of sockets on my system) can be traced back to either a file or abstract name.

And this latter is eating me up. I assume that these sockets were originally created on a file system, but immediately after they were bind()‘ed, the file is unlinked, making it harder (impossible?) to find what the socket file was called to begin with. I first thought it were sockets that were not bind()‘ed to, but many of them have the state CONNECTED displayed (in the netstat -xa output) so that’s not a likely scenario. In any case, if you know how these sockets can have an inode without a known path, please let me know.

But what has this to do with my previous investigation? Well, because the sockets are descriptors, they are passed when a process uses fork() and execve(). And looking at the source code of dhcpcd, I noticed that it does not close its file descriptors when it calls its hook scripts (through the exec_script() function of its sources). As a result, the open file descriptors (including the sockets) are passed on to the hook scripts – one of them calling hostname.

So what I saw in the AVC denials was a leaked socket (so there was no connectto originating from the hostname_t domain since the connection was made by dhcpc in the dhcpc_t domain) that is for some reason being read/written to. A leaked unix stream socket.

Posted in SELinux | Leave a comment

Gentoo WiKi & Knowledge Base

Posted on December 26, 2011 by swift

I have been playing with the Gentoo Wiki the last few days and am very impressed with the work that both the wiki teams as well as existing contributors have already done to the place. The look and feel is very slick and editing works just as expected. One of the changes I made was to move SELinux module information to the wiki. This documentation was originally intended to be published on the Gentoo SELinux Project page, but is easily accessible and maintainable on the wiki too.

So I went a step further and dug up my original GLEP 0051 – Gentoo Knowledge Base proposal and checked how far I could use the Gentoo WiKi for this purpose. From the looks of it, the WiKi can offer a great deal of leverage for this idea and although not everything is supported through the WiKi (like natural search language and such), that might have been overshooting a bit. So we received a Gentoo WiKi Knowledge Base namespace under which the Knowledgebase entries can reside.

Now what is the idea behind such a knowledge base? Well, first of all, the articles below this prefix should all follow the same structure (as explained in the main page) and be sufficiently specific so that the title of the entry should leave little room for misinterpretation. But other than that, there is no limit as to what the Knowledge Base could hold. To that respect, the knowledge base section then provides a (hopefully) thorough listing of common and less common issues with a good explanation why the problem occurred and how to resolve it.

For the time being, the location doesn’t hold that many entries yet, but I will add them as they come along. And of course, feedback is always appreciated ;-)

On a second note, I’d like to give my PoV on the wiki and its relation with the official Gentoo documentation. Unlike what might be circulating, I’m definitely not against the wiki for documentation, on the contrary. Wiki’s have proven to be a good resource for documentation, and because we can never have enough documentation writers, every method for getting more documentation is welcome. But because of its online nature, offline documentation development (which I frequently do) is not possible. Also, keeping translations in sync might be a bit more challenging compared to a file-based solution with version control (otoh, I have little experience with WiKi translations so I might be wrong here).

I strongly believe that the wiki will play a big role in Gentoo’s documentation assets. Many of the documents currently managed by the GDP or the subprojects might be suited to be hosted on the WiKi, especially when those documents are too specific (and as such would require a very specific developer profile to maintain the documents). In such cases, the maintainers of those documents should be able to pick the most efficient method. But for very generic documents, this might not be an easy option.

At least the Gentoo documents now support CC-BY-SA 3.0, so we can migrate documents from the wiki to the main site, and the 2.5 version currently used the most on the main site should be forward compatible with 3.0 (if I read the legalese text well) so we might be able to migrate documents from the main site to the wiki too.

Edit: a3li created the “Knowledge Base” namespace on the wiki, so I updated the links in my post. Thanks for the work on the wiki, a3li!

Posted in Gentoo | 2 Comments

Supporting fix scripts for XCCDF content and maintaining the documents

Posted on December 23, 2011 by swift

One of the features supported through OVAL (and Open-SCAP) is to generate fix scripts when a test has failed. The administrator can then verify this script (of course) and then execute it to correct wrong settings. So I decided to play around with this as well and enhanced the Gentoo Security Benchmark (XCCDF source) with some fixables (like for the sysctl settings). And lo and behold: the thing works ;-)

After evaluating the XCCDF (together with the OVAL document) against my system, I had Open-SCAP generate a fix script: 

# oscap xccdf generate fix --result-id OSCAP-Test-Gentoo-Default xccdf-results.xml
#!/bin/bash
# OpenSCAP fix generator output for benchmark: Gentoo Security Benchmark

# XCCDF rule: rule-sysctl-ipv4-forward
echo 0 > /proc/sys/net/ipv4/ip_forward

# generated: 2011-12-23T14:53:03+01:00
# END OF SCRIPT

Now isn’t that nice. But generating a fix script is one thing, maintaining the XCCDF and OVAL documents is a completely other picture.

One of the downsides that I talked about earlier already is that OVAL has quite an extensible language (it’s a large XML document). Although this extensibility is very flexible and powerful, when you want to add generic tests (like validating sysctl values or matching regular expressions in files) having to write over 30 lines of XML code for a single test is time-consuming at the least. So I quickly scripted something to help me maintain these settings.

The Generating OVAL documents with genoval.sh document explains this script (which is retrievable from my git repository) whose primary purpose is to transform a single line into the entire OVAL structure. With this script, I can now just say gentoo variable USE must contain ssl and it generates both the rules in the XCCDF as the OVAL statements in the OVAL document.

Okay, it’s a script, not a feature-full application, but at least it helps me (and perhaps others as well).

Posted in Gentoo | 2 Comments

SELinux Gentoo/Hardened state 2011-12-19

Posted on December 19, 2011 by swift

On december 14th, the Gentoo Hardened project had its monthly online meeting to discuss the current state of affairs of its projects and subprojects. Amongst them, the updates on the SELinux-front were presented as well.

Since last meeting, the follow topics passed the revue.

  • sec-policy/selinux-base-policy, which is the “master” of our SELinux policies and contains those SELinux modules that are somewhat indivisible (hence the name, “base”), is now at revision 8. I tend to describe the changes on the gentoo-hardened mailinglist, and this is not different for rev 8. I haven’t stabilized the rev 6 one yet although I promised too, I’ll try to find some time to do that this evening.
  • We had a regression with newrole for some time. Luckily, Jory “Anarchy” Pratt found the issue. Drop the setuid bit from the binary, and the application works again as it should. This will be included in the next policycoreutils bump.
  • The last available sudo package now builds with native SELinux support as well, which allows users to add ROLE= and TYPE= information in the sudoers file. As such, users do not need to call newrole when they need to transition to a specific role for just a single command – sudo can now take care of that.
  • The older selinux/v2refpolicy/* profiles have been deprecated. If you want to use a SELinux-enabled profile, you need to use a profile that ends with /selinux, such as default/linux/amd64/10.0/selinux or hardened/linux/amd64/selinux. Of course we prefer you to use a hardened profile ;-)
  • Documentation-wise,

That’s about it. Not a too busy month but progress anyhow.

Posted in Hardened, SELinux | 2 Comments

Supporting CC-BY-SA 3.0

Posted on November 29, 2011 by swift

Until now, documents on the Gentoo website all had to be licensed under the Creative Commons Attribution/Share Alike license, version 2.5. Why? Because at the time of the license choice, that was probably the latest version at hand. In the XML code itself, the license tagging was done using a <license /> tag. Simple and efficient. But things change, and so do license versions.

The folks over at Creative Commons have released version 3.0 somewhere in 2007. I’m not going to cover the differences here, but in general, the principle behind Gentoo’s choice for the CC-BY-SA license remains. But we didn’t change our licenses and there was no real need for it either.

Recently however, the Official Gentoo Wiki was announced, which uses the CC-BY-SA license as well… but the 3.0 version of it. You can’t blame them for taking the latest version, but that does made it a bit more difficult to share resources between the two repositories (wiki versus GuideXML-ified website). The solution? Support CC-BY-SA 3.0 for GuideXML too.

A few commits in our repository made that change happen. Nothing big though: the DTD is updated to allow for <license version="3.0"/> tags, the XSL is updated to support this attribute (and display the new license) and a few other files (supporting files, like the GuideXML Guide) have received the necessary updates.

The result of the change is that existing documents remain under the current 2.5 license (we are not allowed to bump versions of licenses as most documents are not copyrighted by Gentoo Foundation but by their respective authors) but new documents can now use the 3.0 license.

Edit: Sebastian Pipping mailed me to say that in the legal code of the CC-BY-SA 2.5 license there is a clausule about “… a later version of this license with the same License elements…”, so perhaps I might have a “take two” on this.

Posted in Gentoo | Leave a comment

SELinux Gentoo/Hardened state 2011-11-17

Posted on November 17, 2011 by swift

A small write-down on the Gentoo Hardened SELinux state-of-affairs, largely triggered because there was an online meeting for the Gentoo Hardened project today.

  • The SELinux policies offered in the sec-policy category are based on the latest refpolicy release. The older policies have been removed from the Portage tree. The patches that we include in our policies are sent upstream and are getting eventually merged. This way we ensure that we keep the policies manageable (larger development audience), secure (more eyes looking at policy changes) and usable for other SELinux-enabled distributions.
  • The userspace utilities to manage SELinux are also the latest ones available upstream; the older ones have been removed from the tree as well as to keep the number of ebuilds small enough.
  • The Gentoo profiles that enable SELinux support are currently the selinux/v2refpolicy ones and the hardened/*/selinux ones. The former are the older profiles and were a bit more difficult to maintain. The latter ones are the newer profiles which have been running for quite some time now. Alas, we will be deprecating the selinux/v2refpolicy profiles pretty soon now.
  • The various SELinux-related documents as offered on our subproject page are regularly crosschecked to ensure that they are up-to-date with the latest SELinux state-of-affairs. An additional guide will be created on how to report SELinux policy bugs in bugzilla to ensure that we have the information that is needed to get a policy patch accepted upstream as well.
  • On a HR-note: Matt Thode (known as “prometheanfire”) has joined the ranks of SELinux developers in Gentoo Hardened. I’ve also taken over the position as Gentoo Hardened SELinux subproject lead from Chris Pebenito.
Posted in Hardened | Leave a comment

Gentoo Security Benchmark with OVAL and Open-SCAP

Posted on November 16, 2011 by swift

A while ago, I got referred to the Open Vulnerability and Assessment Language, which seems to be an open specification (or even standard) for defining security content/information and being able to document such things in a way that tools can interpret it. Actually, it is a set of these specifications. But first:

tl;dr Gentoo Security Benchmark Guide with example report on automated tests based on XCCDF and OVAL, interpreted by Open-SCAP.

There; now that we have that out of our way, let’s continue on the somewhat more gory details. In this first post, I’d like to talk a bit about XCCDF and OVAL, which are both imo overly complex but interesting XML formats.

The first one, XCCDF, is better known as the Extensible Configuration Checklist Description Format and is an XML format in which you document settings (what should a system look like). By itself, that doesn’t warrant another XML format. However, the power of XCCDF is that you can define in the document profiles. Each of these profiles is then documented with the set of rules that applies to the profile. So you can have an XCCDF document on the configuration of BIND (the nameserver) and have two profiles: one for a single-server setup, and one for a multi-server (master/slave) setup.

These rules also define checks that you can have a tool performed against your configuration. These checks are documented in an OVAL XML file (Open Vulnerability and Assessment Language) which can be interpreted by an OVAL-compliant tool. A very simply put statement could be: “File /etc/ssh/sshd_config must have a line that matches ‘PermitRootLogin no’”.

Of course, XML doesn’t use simple statements. In the case of OVAL, a specific form of normalization occurs.

  • At the beginning, you define a definition that explains what you want to achieve (similar to the above statement) in plain text, and then refer to one or more criteria that needs to be passed if this line applies. Most checks in a configuration guide are simple criteria, but with OVAL you can create criteria like “If my system is a Gentoo x86_64 one, and I use the hardened profile, then criteria A must apply, but if my system does not use a hardened profile, it is criteria B”.
  • The criteria refers to a test that needs to be executed. This test can be a file expression match, partition information check, service state, installed software, etc. but does not allow executing commands that the user defines (it is not considered a safe practice that you execute commands that are stated in the XML file since most OVAL interpreters will run as root). This test is based on two additional aspects:

    • The object refers to the object or resource that is checked. This can be a partition or a file, or a list of lines that match an expression in a file, etc.
    • The state refers to the state that that object or resource should be in (or should match).

With this OVAL language in place, you can now refer to several tests to enhance your XCCDF document, and allow OVAL interpreters to test the various rules on your system. For me, this was the major reason to look into the language, since I had my hopes up to update or rewrite the Gentoo Linux Security Handbook but with a way for users to validate if their system adheres to most/all statements made in that document.

As a matter of exercise, I started making such a security benchmark of which you can find a HTML version online (it’s a preview URL, so might change in the future). And since it is written with XCCDF and OVAL, I’ve also added an example report on automated tests too. The sources of these documents are available as well (XCCDF and OVAL – download as txt but rename to XML then).

For those adventurous enough to play with them: install app-forensics/openscap so that you can parse the files. To generate the guide itself, use oscap xccdf generate scap-gentoo-xccdf.xml > guide.html. To run the tests associated with it, use oscap xccdf eval --oval-results --profile Gentoo-Default --report report.html scap-gentoo-xccdf.xml. Also take a look at the Open-SCAP website which is a good resource as well (and the mailinglists are low traffic but with good response times!).

So, what is the future on all this for me?

First, I’m going to work a bit further on the OVAL statements, so that I can automatically test the majority of settings that I currently have in the benchmark/guide. Only when I’m far enough will I continue on the content of the guide (since it is far from finished) and also see if this isn’t something that can be put on a somewhat more official location. If not, I’ll still continue developing it, but it’ll remain on my dev-page.

When I’m somewhat satisfied with that, I might check if I can’t have OVAL enhanced with some Gentoo-specific objects (there are already objects for RedHat-like and Debian-like distributions) so that we can write tests for Gentoo settings (like USE flags, profiles, enabled GCC specs, etc.) If we have that, then we can even write checks (XCCDF and OVAL based) to validate if a system is how it is supposed to be (wouldn’t that be great, an automated test that tells you if your system is properly set up according to our documents).

But XCCDF and OVAL isn’t the end. There are other formats available as well, like CCE (for configuration entries), CVE/CPE (to check vulnerability information and its target software/platform) and more. I know RedHat is already actively using OVAL for its security advisories, and other sites like Center for Internet Security are also using XCCDF and OVAL to document and work with security guides. So why would Gentoo not get on that train as well?

Posted in Gentoo, Hardened, Security | Leave a comment

Centers of Excellence

Posted on October 25, 2011 by swift

When dealing with software (I’ll talk about software here, but the information is applicable to most technologies, such as appliances and operating systems) many organizations want to have “centers of excellence” with respect to the software. These teams are responsible for positioning the software within the organization, supporting the software and if necessary, act as a link between the internal customer and the software vendor.

The approach on these “centers of excellence” is often described as a cost efficient way of handling the software within the organization. Sadly, many organizations go to the extreme and try to put as much support and services within those teams as possible, hoping that full consolidation of all service matters would yield an even better (financial) benefit.

Such further consolidation however has a negative side which is often overlooked: centralized teams are less aware of the internal customers’ requirements and situation. Most internal customers probably have their own IT teams that are much better informed about the criticality of the customers’ systems and the services that the customer requests. Those teams are then responsible for getting the right services from those “centers of excellence”. And that is where the difficulty lies.

“Centers of excellence” are based on products and technical services. They want to provide the best-in-class service with their products and as such keep enhancing their services in the hope that they can serve all internal customers. But by doing so, they are becoming more and more of a product vendor. In the end, they either focus on the product completely, or they focus on some frameworks and tooling that they have designed and developed to support the integration of the product within the organization. For the IT teams of the internal customer however, the “centers of excellence” are less seen as part of the organization and more as a vendor (or broker).

That doesn’t mean that the concept of these “centers of excellence” is wrong though, but they need to keep the organization itself in mind when dealing with the product. The detail on services that they offer need to be aligned with the organizations’ strategy and weighted to provide a cost efficient, yet qualitatively best-in-class service.

Of course, all that is easier said than done. So let me suggest an approach on software service(s) within an organization.

The lowest service that the organization must support for any software is assistance in the installation, upgrade, tracking and eventual removal of the software. Such a service is always needed and can be provided with little knowledge of the internal customer. A “center of excellence” should provide the means to (semi)automatically install the software – preferably using the organizations’ standard software deployment methods, upgrade the software (both for major releases, minor releases as well as security patches), remove and (not to forget) track where the software is installed. Especially when dealing with proprietary software, tracking is almost mandatory for organizations to keep track of the licenses needed.

This lowest service offering has almost only positive sides:

  • IT teams that need the software can easily (and without further assistance of other teams) install the software. If the installation method is automated, it can even be done in a fast manner, which is always to the liking of the customer.
  • The organization keeps its risks low by providing the security updates and product upgrades in a seem less manner.
  • By tracking deployments, the organization can keep track of licenses used (for instance, the “center of excellence” can provide regular reporting towards the financial departments) and, if the tracking is done right, can even suggest improvements in the architecture or deployments to further minimize license cost.
  • IT teams can freely focus on the solution that they are building for the customer without the need to duplicate software installation methods and different patching processes.

By providing these services from a “center of excellence”, you definitely reduce certain research & development costs – without this offering, each team would need to develop processes to deploy software and track its usage. This is independent of the internal customer and as such, consolidation is a definite win here.

Once this service is offered, the “center of excellence” can focus on the services that they can provide and which are mandatory for all internal customers (not “usable”, but mandatory) and for which little flexibility (in design or development) is possible. There are not that many cases here, and this is very specific to each technology and organization in which the software is made available.

As a hypothetical example, consider an LDAP service. The “center of excellence” might want to provide auditing (and alignment with an organization standard auditing system) if the organization has a policy that auditing is mandatory, regardless of the project for which LDAP is used. Of course, if the “center of excellence” wants to stop at this service offering (i.e. the previously mentioned installation/tracking, and now auditing) then this is most likely a best practice document geared towards the IT teams that need to implement it.

The benefit? The IT teams are in this case aware of the requirement (auditing must be enabled) and do not need to investigate how to do this anymore (it is already documented).

The third level of service offering that I see is the reusable, customer-independent services that one wants to provide on the software. For a database, this might mean alignment with the organizations’ backup infrastructure. For an LDAP, that might mean getting feeds from a central source (be it a central LDAP infrastructure, an Identity Management system, …).

When you consider providing this service (which is usually the case in larger environments), take special care that the service you want to offer is flexible enough so that any IT team can work with it. A service that is only applicable to 70% of your internal customers will not do it. For an LDAP service, this might mean that you provide out-of-the-box configuration templates, best practice information for its back-end infrastructure (which includes backup/restore operations), …

But make sure that you are not redesigning and re-developing what your product already provides. I have seen numerous cases where teams develop tools that should “shield the complexity” from the end user, but in effect are creating additional layers of clouds and complexity instead. If you want or need to abstract complexity from the user, make sure that this is only a single layer that you are introducing. The moment you are creating tools on top of previously written tools, you should reconsider your actions.

Often, “centers of excellence” want to rewrite documentation for the end users. They feel that the documentation available from the vendor is too complex for IT teams to use. Although I can relate to that, they should not underestimate the expertise within the IT teams. If the IT teams do not want to gain the knowledge or experience through the product guides, then they are less likely to properly maintain and troubleshoot the product.

In such cases, I would wager that it is more beneficial for the organization to look at their (human) resources and their relation to the software. In times like these, where cloud solutions play an important role, my suggestion would be to consolidate the software usage towards a SaaS principle (Software-as-a-Service) managed by an experienced team (or teams). This does not mean that the “center of excellence” has to play this role, but it does sound like a logical step for them. After all, if the “center of excellence” only defines additional services without actually consuming them, they might lose track of the product in real-case scenario’s.

Take a web server for example – Apache. You might have a “center of excellence” for Apache web servers, which provides easily deployable packages. They might provide example configuration files as well as pointers towards Apache’s documentation (and best practices). They track the deployments and ensure that security patches are available as soon as possible. But they should guard over the rest of the services that they want to offer.

Why would that team create a framework for auto-generating configuration files? What is the benefit of this for the entire organization, for each customer? If the IT team can take care of the configuration, let them be. But if those IT teams would rather not manage these web servers themselves, what is the point in creating additional frameworks to “hide the complexity”? It is, imo, much more efficient to see if you cannot provide web hosting services instead, and have the IT teams “buy” these (internal) web hosting services.

There are many advantages to this: the web hosting environment is managed by a team of experts, can be consolidated to keep the TCO low, can provide default configurations that are fully aligned with the organization but still offer the flexibility that individual customers might require.

Same goes for other software products: java application servers (like JBoss or IBM WebSphere AS), databases (Oracle, SQL Server, MySQL or Postgresql), messaging systems, log servers, LDAP services, file share services, …

So what about you – how do you position “centre of excellence” teams? Can you relate with such a “back to basics” approach, or would you rather see a fully integrated, standardized solution roll-out where IT teams only have experience with the organization-only frameworks?

Posted in Analysis | Leave a comment

SELinux’ 2011/07 releases now stable

Posted on October 23, 2011 by swift

A few minutes ago, I stabilized both the 2.20110726 policies as well as the SELinux userspace utilities that were stable (upstream) on 20110727. With the change, I also updated the Gentoo SELinux Handbook with the changes I presented on our gentoo-hardened mailinglist. After some time, I’ll remove the now obsoleted older policies and userspace utilities to keep the tree in a sane state.

There are a few policy packages whose stabilized version isn’t the latest (cfr earlier post), those are due within the proper designated period (about 1 month).

Posted in Hardened, SELinux | Leave a comment