But first, a quick introduction to initramfs. The Linux kernel supports initrd images for quite some time. These images are best seen as loopback-mountable images containing a whole file system that the Linux kernel boots as the root device. On this initrd image, a set of tools and scripts then prepare the system and finally switch towards the real root device. The initrd feature was often used when the root device is a network-mounted location or on a file system that requires additional activities (like an encrypted file system or even on LVM. But it also had some difficulties with it.

Using a loopback-mountable image means that this is seen as a full device (with file system on it), so the Linux kernel also tries caching the files on it, which leads to some unwanted memory consumption. It is a static environment, so it is hard to grow or shrink it. Every time an administrator creates an initrd, he needs to carefully design (capacity-wise) the environment not to request too much or too little memory.

Enter initramfs. The concept is similar: an environment that the Linux kernel boots as a root device which is used to prepare for booting further from the real root file systems. But it uses a different approach. First of all, it is no longer a loopback-mountable image, but a cpio archive that is used on a tmpfs file system. Unlike initrd, tmpfs can grow or shrink as necessary, so the administrator doesn’t need to plan the capacity of the image. And because it is a tmpfs file system, the Linux kernel doesn’t try to cache the files in memory (as it knows they already are in memory).

There are undoubtedly more advantages to initramfs, but let’s stick to the primary objective of this post: talk about its implementation on a hardened system.

I started playing with dracut, a tool to create initramfs archives which is seen as a widely popular implementation (and suggested on the gentoo development mailinglist). It uses a simple, modular approach to building initramfs archives. It has a base, which includes a small init script and some device handling (based on udev), and modules that you can add depending on your situation (such as adding support for RAID devices, LVM, NFS mounted file systems etc.)

On a SELinux system (using a strict policy, enforcing mode) running dracut in the sysadm_t domain doesn’t work, so I had to create a dracut_t domain (which has been pushed to the Portage tree yesterday). But other than that, it is for me sufficient to call dracut to create an initramfs:

# dracut -f "" 3.1.6-hardened

My grub then has an additional set of lines like so:

title Gentoo Linux Hardened (initramfs)
root (hd0,0)
kernel /boot/vmlinuz-3.1.6-hardened root=/dev/vda1 console=ttyS0 console=tty0
initrd /boot/initramfs-3.1.6-hardened.img

Sadly, the bugger didn’t boot. The first problem I hit was that the Linux kernel I boot has chroot restrictions in it (grSecurity). These restrictions further tighten chroot environments so that it is much more difficult to “escape” a chroot. But dracut, and probably all others, use chroot to further prepare the bootup and eventually switch to the chrooted environment to boot up further. Having the chroot restrictions enabled effectively means that I cannot use initramfs environments. To work around, I enabled sysctl support for all the chroot restrictions and made sure that their default behavior is to be disabled. Then, when the system boots up, it enables the restrictions later in the boot process (through the sysctl.conf settings) and then locks these settings (thanks to grSecurity’s grsec_lock feature) so that they cannot be disabled anymore later.

But no, I did get further, up to the point that either the openrc init is called (which tries to load in the SELinux policy and then breaks) or that the initramfs tries to load the SELinux policy – and then breaks. The problem here is that there is too much happening before the SELinux policy is loaded. Files are created (such as device files) or manipulated, chroots are prepared, udev is (temporarily) ran, mounts are created, … all before a SELinux policy is loaded. As a result, the files on the system have incorrect contexts and the moment the SELinux policy is loaded, the processes get denied all access and other privileges they want against these (wrongly) labeled files. And since after loading the SELinux policy, the process runs in kernel_t domain, it doesn’t have the privileges to relabel the entire system, let alone call commands.

This is currently where I’m stuck. I can get the thing boot up, if you temporarily work in permissive mode. When the openrc init is eventually called, things proceed as usual and the moment udev is started (again, now from the openrc init) it is possible to switch to enforcing mode. All processes are running by then in the correct domain and there do not seem to be any files left with wrong contexts (since the initramfs is not reachable anymore and the device files in /dev are now set again by udev which is SELinux aware.

But if you want to boot up in enforcing straight away, there are still things to investigate. I think I’ll need to put the policy in the initramfs as well (which has the huge downside that every update on the policy requires a rebuild of the initramfs as well). In that case I can load the policy early up the chain and have the initramfs work further running in an enforced situation. Or I completely regard the initramfs as an “always trusted” environment and wait for openrc’s init to load the SELinux policy. In that case, I need to find a way to relabel the (temporarily created) /dev entries (like console, kmsg, …) before the policy is loaded.

Definitely to be continued…

I have been looking at a weird SELinux denial I had occuring on my system:

avc:  denied  { read write } for  pid=10012 comm="hostname" \
path="socket:[318867]" dev=sockfs ino=318867 \
scontext=system_u:system_r:hostname_t \
tcontext=system_u:system_r:dhcpc_t \
tclass=unix_stream_socket

I had a tough time trying to figure out why in earth the hostname application was trying to read/write to a socket that was owned by dhcpcd. Even more, I didn’t see a connectto attempt, and there is nothing in my policy that would allow the hostname_t domain to connect to a unix_stream_socket of dhcpc_t. But moreover I was intrigued why the given path was no real path, even though it has an inode.

So I dug up lsof, which returned the following on this socket:

# lsof -p 10017
COMMAND   PID USER   FD      TYPE             DEVICE SIZE/OFF   NODE NAME
...
dhcpcd  10017 root    3u     unix 0x0000000000000000      0t0 318867 socket
dhcpcd  10017 root    4w      REG              252,3        6 268749 /var/run/dhcpcd-eth1.pid
dhcpcd  10017 root    5u     unix 0x0000000000000000      0t0 318869 socket

Still no luck in figuring out what that is. And even /proc/net/unix didn’t give anything back:

# grep 318867 /proc/net/unix
Num               RefCount Protocol Flags    Type St Inode Path
0000000000000000: 00000002 00000000 00000000 0001 01 318867

So I started looking at Unix domain sockets, what they are, how they are used, etc. And I learned that

• Unix domain sockets are just files. Well, most of the time. To use a socket (from server-perspective), a programmer first calls socket() to create a socket descriptor, which is a special type of file descriptor. It then bind()‘s the socket to a (socket)file on the file system, listen()‘s for incoming connections and eventually accept()‘s them. Clients also use socket() but then call connectto() to have its socket connected to a (socket)file and eventually read() and write() (or send() and recv()).
• Linux supports an abstract namespace for sockets, so not all of these are actually bound/connected to a file. Instead, they connect to a “name” instead, which cannot be traced back to a file. For those interested, looking at /proc/net/unix or netstat -xa shows the abstract ones starting with an @ sign.
• Not all Unix sockets (actually almost the majority of sockets on my system) can be traced back to either a file or abstract name.

And this latter is eating me up. I assume that these sockets were originally created on a file system, but immediately after they were bind()‘ed, the file is unlinked, making it harder (impossible?) to find what the socket file was called to begin with. I first thought it were sockets that were not bind()‘ed to, but many of them have the state CONNECTED displayed (in the netstat -xa output) so that’s not a likely scenario. In any case, if you know how these sockets can have an inode without a known path, please let me know.

But what has this to do with my previous investigation? Well, because the sockets are descriptors, they are passed when a process uses fork() and execve(). And looking at the source code of dhcpcd, I noticed that it does not close its file descriptors when it calls its hook scripts (through the exec_script() function of its sources). As a result, the open file descriptors (including the sockets) are passed on to the hook scripts – one of them calling hostname.

So what I saw in the AVC denials was a leaked socket (so there was no connectto originating from the hostname_t domain since the connection was made by dhcpc in the dhcpc_t domain) that is for some reason being read/written to. A leaked unix stream socket.

Since last meeting, the follow topics passed the revue.

• sec-policy/selinux-base-policy, which is the “master” of our SELinux policies and contains those SELinux modules that are somewhat indivisible (hence the name, “base”), is now at revision 8. I tend to describe the changes on the gentoo-hardened mailinglist, and this is not different for rev 8. I haven’t stabilized the rev 6 one yet although I promised too, I’ll try to find some time to do that this evening.
• We had a regression with newrole for some time. Luckily, Jory “Anarchy” Pratt found the issue. Drop the setuid bit from the binary, and the application works again as it should. This will be included in the next policycoreutils bump.
• The last available sudo package now builds with native SELinux support as well, which allows users to add ROLE= and TYPE= information in the sudoers file. As such, users do not need to call newrole when they need to transition to a specific role for just a single command – sudo can now take care of that.
• The older selinux/v2refpolicy/* profiles have been deprecated. If you want to use a SELinux-enabled profile, you need to use a profile that ends with /selinux, such as default/linux/amd64/10.0/selinux or hardened/linux/amd64/selinux. Of course we prefer you to use a hardened profile ;-)
• Documentation-wise,
  • the Gentoo Hardened SELinux Handbook has been updated to reflect the profile changes
  • the SELinux bugreporting guide has been put online to inform users what kind of information is needed for us to fix issues or denials that they might see
  • the SELinux FAQ has been updated with the questions Applications do not transition on a nosuid partition and Why do I always need to re-authenticate when operating init scripts?.

That’s about it. Not a too busy month but progress anyhow.

tl;dr Gentoo Security Benchmark Guide with example report on automated tests based on XCCDF and OVAL, interpreted by Open-SCAP.

There; now that we have that out of our way, let’s continue on the somewhat more gory details. In this first post, I’d like to talk a bit about XCCDF and OVAL, which are both imo overly complex but interesting XML formats.

The first one, XCCDF, is better known as the Extensible Configuration Checklist Description Format and is an XML format in which you document settings (what should a system look like). By itself, that doesn’t warrant another XML format. However, the power of XCCDF is that you can define in the document profiles. Each of these profiles is then documented with the set of rules that applies to the profile. So you can have an XCCDF document on the configuration of BIND (the nameserver) and have two profiles: one for a single-server setup, and one for a multi-server (master/slave) setup.

These rules also define checks that you can have a tool performed against your configuration. These checks are documented in an OVAL XML file (Open Vulnerability and Assessment Language) which can be interpreted by an OVAL-compliant tool. A very simply put statement could be: “File /etc/ssh/sshd_config must have a line that matches ‘PermitRootLogin no’”.

Of course, XML doesn’t use simple statements. In the case of OVAL, a specific form of normalization occurs.

• At the beginning, you define a definition that explains what you want to achieve (similar to the above statement) in plain text, and then refer to one or more criteria that needs to be passed if this line applies. Most checks in a configuration guide are simple criteria, but with OVAL you can create criteria like “If my system is a Gentoo x86_64 one, and I use the hardened profile, then criteria A must apply, but if my system does not use a hardened profile, it is criteria B”.
• The criteria refers to a test that needs to be executed. This test can be a file expression match, partition information check, service state, installed software, etc. but does not allow executing commands that the user defines (it is not considered a safe practice that you execute commands that are stated in the XML file since most OVAL interpreters will run as root). This test is based on two additional aspects:
  • The object refers to the object or resource that is checked. This can be a partition or a file, or a list of lines that match an expression in a file, etc.
  • The state refers to the state that that object or resource should be in (or should match).

With this OVAL language in place, you can now refer to several tests to enhance your XCCDF document, and allow OVAL interpreters to test the various rules on your system. For me, this was the major reason to look into the language, since I had my hopes up to update or rewrite the Gentoo Linux Security Handbook but with a way for users to validate if their system adheres to most/all statements made in that document.

As a matter of exercise, I started making such a security benchmark of which you can find a HTML version online (it’s a preview URL, so might change in the future). And since it is written with XCCDF and OVAL, I’ve also added an example report on automated tests too. The sources of these documents are available as well (XCCDF and OVAL – download as txt but rename to XML then).

For those adventurous enough to play with them: install app-forensics/openscap so that you can parse the files. To generate the guide itself, use oscap xccdf generate scap-gentoo-xccdf.xml > guide.html. To run the tests associated with it, use oscap xccdf eval --oval-results --profile Gentoo-Default --report report.html scap-gentoo-xccdf.xml. Also take a look at the Open-SCAP website which is a good resource as well (and the mailinglists are low traffic but with good response times!).

So, what is the future on all this for me?

First, I’m going to work a bit further on the OVAL statements, so that I can automatically test the majority of settings that I currently have in the benchmark/guide. Only when I’m far enough will I continue on the content of the guide (since it is far from finished) and also see if this isn’t something that can be put on a somewhat more official location. If not, I’ll still continue developing it, but it’ll remain on my dev-page.

When I’m somewhat satisfied with that, I might check if I can’t have OVAL enhanced with some Gentoo-specific objects (there are already objects for RedHat-like and Debian-like distributions) so that we can write tests for Gentoo settings (like USE flags, profiles, enabled GCC specs, etc.) If we have that, then we can even write checks (XCCDF and OVAL based) to validate if a system is how it is supposed to be (wouldn’t that be great, an automated test that tells you if your system is properly set up according to our documents).

But XCCDF and OVAL isn’t the end. There are other formats available as well, like CCE (for configuration entries), CVE/CPE (to check vulnerability information and its target software/platform) and more. I know RedHat is already actively using OVAL for its security advisories, and other sites like Center for Internet Security are also using XCCDF and OVAL to document and work with security guides. So why would Gentoo not get on that train as well?

There are a few policy packages whose stabilized version isn’t the latest (cfr earlier post), those are due within the proper designated period (about 1 month).

In related news, the Gentoo Hardened SELinux FAQ is updated with two entries:

1. Portage fails to label files because “setfiles” does not work anymore, and
2. Applications do not transition on a nosuid-mounted partition

The purpose and necessity of network firewalls is well known and understood: make sure that the service is only accessible from the right location, check if connections aren’t abused (or too many connections are made), etc. But what if the connection itself is valid? After all, most abuse of services is not because they originate from the wrong location or try to access the wrong service. Instead, such abuse comes from valid access to the application, but with less kosher intentions. So what can application firewalls do in this case?

• Because they perform inspection of the data that is transferred itself, application firewalls can detect malicious data fragments or attempts to abuse the service. These detection rules can be based on general, heuristic rules (well-known examples are detection rules for cross-site scripting attacks (XSS) or SQL Injection) but can also be very specific to a particular application.
• Because all data is transferred through the firewall and the firewall has knowledge of the application itself, these firewalls offer advanced auditing features since they can detect authentication steps, user data, application-specific transactions and more.
• With knowledge of the users’ session and behavior (application-level) and origin (network level), application firewalls can detect and prevent unauthorized sessions, such as the case with session hijacking or even man-in-the-middle attacks (based on behavior detection)

Implementing an application firewall however doesn’t only mean that you improve access controls on it. It has other advantages that make application firewalls an important part in many architectures:

• If all service access is forced through the application firewall (for instance through an IP filter on the service that only allows connections from the application firewalls) you can implement rules that deter known attacks/vulnerabilities without needing to fix the code itself (or if fixing is possible, lower the time pressure). For instance, for Apache-based services, such an application firewall could detect or even change the Range: header on malicious requests to lower the impact of this potentially nasty DoS vulnerability
• Depending on the complexity, some functional application bug fixing might even be possible. For instance changing content types on requests/replies (HTTP), adding a domain on an FTP accounts’ login statement, …
• Many application firewalls (or gateways) offer proxy functionality which might improve response times. This is not a sure-given, since most applications are session-aware so the advantage is only for session-agnostic requests (be it static content or specific SQL statements in case of a database firewall). But also in case of session-aware statements can an improvement be found. Consider a database firewall which translates SQL statements from an unsupported application towards better defined statements (for instance using proper indexes or materialized views).
• In some cases, you might even be able to upgrade a backend of an unsupported application (which previously required an outdated version of that database) by translating the backend requests when they are incompatible with the new backend version. So you can improve integration or support unsupported upgrades.
• In case of risk reduction, application firewalls also allow you to move a service elsewhere (even in the public cloud) and still keep the access under control.

Of course, it would be TGTBT (Too Good To Be True) if there isn’t an (important) downside: maintaining the application firewall is a daunting task. Because of its flexibility, you’ll need deep knowledge in the application firewall administration and development, keep track of all rules you have (and why you have them), do lots and lots of testing on each rule (since it might affect the functioning of the application) and still be aware that subtle differences introduced by the application firewall rules can pop up unexpectedly. Also, integrating an application firewall is another service between your customer and his service, which might influence performance but also makes the underlying architecture more complex. Finally, you’ll need to consider that an application firewall requires lots of resources (CPU/memory), especially when it needs to perform SSL/TLS termination. Oh, and they’re often expensive too.

Still, even with these downsides, application firewalls are an important part of the service isolation strategy, which is a key aspect in the risk mitigation strategy which this series started with. We’ve focused on three now: service isolation (network-wise), process isolation (through mandatory access control) and now access isolation through application firewalls. And with proper hardening in place, I believe that you have done all you can do to reduce the risks when running unsupported software (apart from upgrading it or switching towards supported software).

If you have other ideas that benefit risk mitigation, with specific focus on unsupported software, I would be glad to hear about them.

Standard access control on most popular operating systems is based on a limited set of privileges (such as read, write and execute) on a limited scale (user, group, everyone else). Recent developments are showing an increase in the privilege flexibility, with the advent of manageable capabilities (Linux/Unix) or Group Policies (Windows). However, these still lack some important features:

• Users are still able to delegate their privilege to others. A user with read access on a particular file can copy that file to a public readable location so others can read it as well. Privileges on his own files and directories are fully manageable by the owner. For our risk mitigation approach on unsupported software, that means that a vulnerability might be exploited so that the service “leaks” information. It is especially important in an attack that uses a sequence of vulnerabilities (such as in an advanced persistent threat) where low-risk vulnerabilities can be combined into a high-risk exploit.
• Privileges are still user-level privileges (including technical account users). In case of running services, this almost always means that the process has more privileges than it requires. Some software titles allow for dropping capabilities when not needed anymore. Most however are oblivious of the rights they possess. Abuse of the service (which includes use of features that the service offers but are not allowed policy-wise by the organization) cannot be prevented if hardening doesn’t disable it.
• Privileges are managed by many actors (such as the system administrators) and are not that easy to audit. Privilege denials are often not audited, causing issues to only come up when they occur, rather then when the attempt to provoke issues is started. In many cases, a malicious (or “playful, inventive”) user starts with investigating and trying out long before a way is found to abuse the service.

In case of a mandatory access control system, a security administrator is responsible for writing and managing a security policy which is enforced by the operating system (well, higher level enforcement would be even better, but is currently not realistic). Once enforced, the policy ensures that privileges are not delegated (unless allowed). Also, in most MAC systems, the policy allows for a much more detailed privilege granularity. And recent server operating systems have support for MAC – I personally work with SELinux for the (GNU/)Linux operating system.

But this more granular flexibility in privileges comes with some costs. First of all, it becomes much more complex to manage the policy. You’ll need highly experienced administrators to work with a MAC-enabled system. Second, a MAC model has a negative influence on performance since the system has to check many more accesses and access rules. To make MAC-enabled systems workable, operating systems offer a default policy which already covers many services. Also, developers on the MAC technology are continuously safe-guarding performance – I personally do not notice a performance degradation when using SELinux, and more realistic benchmarks suggest that the impact of SELinux is between 3% and 12% depending on the policy level.

But what does that mean towards the initial risk list that I identified in the beginning of this article series? Well, directly, very little: mandatory access control in this case is about reducing the impact of security vulnerabilities (and abuse of the service). It will not help you out in other ways. However, there are other things to gain from a mandatory access control than just threat reduction.

An advantage is – again – that you get to know your application well, especially if you had to write a security policy for it. Since you need to define what files it can access, which kind of accesses it is allowed to do, which commands it can execute, etc, it will give you insight in how the application operates. Bugs in the application might be solved faster and you’ll definitely learn more about how the application is integrated. Another one is that most mandatory access control systems have much more detailed auditing capabilities. Attempts to abuse the service will result in denials which are detected and on which you can then take proper action.

Taking a higher-level look at mandatory access control will show you that, in case of risk mitigation, it is much more like service isolation, but then on the operating system level. You isolate the processes, governing the accesses they are allowed to do.

But the one main issue – active exploits on the application service – cannot be hindered by neither service isolation (since the service is still accessible), hardening (although it might help) or mandatory access control (which reduces the actions an exploit can do). To make sure that vulnerabilities are less likely to be exploited, I’ll talk about application firewalls in the next post.

Security is, for some, also a religion. They see risks and vulnerabilities and what not everywhere. They’re always thinking every system in the world is or will be hacked in the near future and are frantically trying to secure every service they are running – and more. But security is also a real-life issue. If you take a look at the compromised GlobalSign website (who mentions that the website is an isolated one – as I described earlier) I hope that you look at security as being a functional requirement in architecturing and design (and not a non-functional one as many frameworks suggest).

And as you can see from the example, isolating services is not sufficient to prevent a successful exploit of an insecure or unsupported software (the reason why I started with this series). One additional measure that you can take is hardening the server and service.

The act of hardening a server and service is to configure the system so that it is as secure as possible, based on configuration entries. Many vendors and projects offer a security guide (like the Gentoo Security Handbook or the Fedora Security Guide) although most of them add this as part of their standard administrative documents (like the PostgreSQL “Server Setup and Operation” chapter).

But for some reason, you’ll find that default installations – even when following the instructions of the vendor – are not as secure as you want it to be. As a matter of fact, if you come in contact with auditors, you’ll probably fail any audit if you use a default installation. To help administrators to secure their services, you will find lots of third party sites offering advice on securing the operating system and the services running on it. These guides are what you will need to “harden” your system.

• OWASP, which stands for Open Web Application Security Project, hosts some hardening guides and suggestions together with test scenarios. For front-end application servers (mostly web application servers) you will find lots of interesting resources in the OWASP site (and surrounding community).
• Google is probably the best resource for finding hardening guides for your operating system or service. Just look for “hardening foo” and you will be reading for a week.
• CISecurity, or “Center for Internet Security”, is another one with a larger portfolio on hardening guides. Not only does it offer these guides (which it calls “benchmarks”), but organizations can also become a member and as such benefit from tooling that CISecurity supports for the validation of benchmarks (i.e. test if the system/deployment is compliant towards a particular benchmark). It does that by developing the benchmarks in a open specification called OVAL (the Open Vulnerability and Assessment Language) and XCCDF (XML Configuration Checklist Data Format). And CISecurity is not the only one there.
• Another such resource is the National Vulnerability Database (national for US residents, that is ;-) There you can find and download the OVAL/XCCDF resources for various software titles and operating systems. But as you can imagine from the abbreviations, the resources are XML files which are not made to be read by humans.

Although you can use the tool(s) that CISecurity offers, another possibility is to use Open-SCAP, an open source framework for handling SCAP, OVAL, XCCDF and other such open specifications on a system. Its documentation offers a first glance at what it can support.

However, this brings on he disadvantages of hardening services…

1. Hardening a system and its services is a time consuming job. Its only purpose is to reduce the impact of exploited vulnerabilities and reduce the “attack surface” so that exploits on unused functions are not possible.
2. Hardening a system and its services can impact the service. Make it too tight, and it might not behave anymore like you want it to.
3. Also, since there are many, many resources “out there” on hardening, you will have to manage your hardening rules, document them for yourself. It is also advisable to document the rules you are not implementing, if not just for future’s sake.
4. The hardening guides also require quite some expertise on the service. If you are not experienced with the service but you need to harden it, you can be lucky and just implement what is suggested and hope for the best, but usually you will need to dive deeper in the subject and make (tough) choices.

Although specifications like SCAP exist to help you in your hardening exercises, these are still difficult to manage (do not try to write OVAL/SCAP/XCCDF content in your favorite text editor). Its adoption however by Fedora and RedHat is showing a positive effect on the tools surrounding this specification. I will be writing about SCAP, OVAL and XCCDF later since I too see good use of it in organizations (or even free software projects).

Does that mean that hardening is not beneficial? On the contrary:

• You gain lots of knowledge in the matter, and also forces you to think about integration aspects. Since you are responsible for the service (or the damage that could be made if the service is exploited) being knowledgeable is definitely a good thing.
• A considerable amount of vulnerabilities that are and will be reported on the service (check CVE details to find out about publicly known vulnerabilities, documented in the CVE specification) will not have their effect on a well hardened service. Or put another way, you will reduce the number of real vulnerabilities in your service. You will not be able to exclude all vulnerabilities, but the projected number is high – a fully hardened Windows or Linux system can mitigate up to 90% of the exploits on the operating system. It will considerably reduce the risks that you and your organization are taking.
• A well defined hardening guide will also offer the means to automatically audit or check if the system is still compliant to the hardening setup you envisioned. Scheduled regularly, this will ensure that your configurations are not drifting away, back to a more vulnerable setup, for whatever reason.
• By removing the functions that the service should not offer, you make sure that the use of the service is per the organizations’ guidelines. (Internal) abuse of the service is made more difficult, so users are forced to take the regular way. Unlike service isolation, which allows you to keep track of data/service flows, hardening makes sure that side-functionality is not used without your consent. Or to put it more blunt, “Yes I know Oracle DB can be used to schedule tasks on the operating system, but no, you’re not allowed to use that function”.

And who knows, perhaps by optimizing the configuration, it might run faster with a lower resource footprint ;-) If it does, that’s perfect, since the next topic on risk mitigation will have a negative influence on performance: mandatory access control.

Internet: absolute communication, absolute isolation
~Paul Carvel

The quote might be ripped out of its context completely, since it wasn’t made when talking about risks and the assurance you might need to get in order to reduce risks. But it does give a nice introduction to the second part of this article series on risk mitigation. After all, if the unsupported software is offering services to the Internet, you really want to govern the communication and isolate the service.

When you are dealing with a product or software that is unsupported (be it that it will not get any patches and updates from its authors or vendor, or there is no time/budget to support the environment properly), it is in my opinion wise to isolate the service from the rest. My first post on the matter gave a high-level introduction on the risks that you might be taking when you run unsupported (or out-of-support) systems. Service isolation helps in reducing the risks that others have when you run such software on a shared infrastructure (like in the same network or even data centre).

By isolating the unsupported service from the rest, you create a sort-of quarantine environment where sudden mishaps are shielded from interfering with other systems. It provides insurance for others, knowing that their (supported) services cannot be influenced or jeopardized by issues with the unsupported ones. And if these services need to interact with the isolated service, the interface used is known and much more manageable (think about a well-defined TCP connection versus local communication or even Inter-Process Communication). But it goes beyond just providing insurance for others.

Isolation forces you to learn about the application and its interaction with other services. It is this phase that makes it extremely important in an environment, because not knowing how an application works, behaves or interacts creates more problems later when you need to debug issues, troubleshoot performance problems and more. Integration failures, as described in my previous post, can only be dealt with swiftly if you know how the service integrates with others.

Another advantage of proper service isolation is that you can fix its dependencies more easily. Remember that I talked about upgrade difficulties, where a necessary upgrade for one component impacted the functionalities of the other (unsupported) component? With good isolation, the dependencies are more manageable and controllable. Not only are (sub)component upgrades easier to schedule, it is also a lot easier to provide fall-back scenario’s in case problems occur. After all, the isolated service is the only user so you have little to fear if you need to roll-back a change.

But what is proper service isolation?

• First of all, it means that you focus on running the (unsupported) software alone on an operating system instance. Do not run other services on the same OS, not even if they too are unsupported. The only exception here is if the other services are tightly integrated with your service and cannot be installed on a separate OS. But usually, full service isolation is possible.
• Next, strip the operating system so it only runs what you need for managing the service. Put primary focus on services that are accepting incoming connections (“listening”) and secondary focus on allowed outgoing protocols/sessions (and the tools that initiate them).
• See if you can virtualize the environment. In most cases, the service does not require many resources so it would be a waste running it on a dedicated system. However, in my opinion, a much better reason for virtualization is hardware abstraction. Sure, all operating systems tell you that they have some sort of Hardware Abstraction Layer in them and that they can deal with hardware changes without you noticing it. But if you are an administrator, you know this is only partially true. Virtualization offers the advantage that the underlying hardware is virtual and can remain the same, even if you move the virtualized system to a much more powerful host. Another advantage is that you might be able to offload certain necessary services from the OS (like backup) to the host (snapshotting).
• Shield the operating system, network-wise, from other systems. Yes, that means putting a firewall BEFORE the operating system guest (and definitely not on the OS) which governs all traffic coming in and out of the environment. Only allow connections that are legit. If your organization has a huge network to manage, they might work with network segment filtering instead of IP-level filtering. See if you can get an exception to that – managing the rules should not give too much overhead since the system, being unsupported and all, is a lot less likely to get many connectivity updates.

But before finishing off, a hint about stripping an operating system. Stripping is much more than just removing services that are not used. It also means that you look for services that are needed, and see if you can externalize them. Common examples here are logging (send your logs to a remote system rather than keeping them local), e-mail (use simple “direct-out” mail) and backup (use a locally scheduled backup tool, or even offload to the host in virtualized systems), but many others exist.

Of course, service isolation is not unknown to most people. If you run a large(r) network with Internet-facing services, you probably isolate those in a DMZ environment. That is quite frankly (also) for the same “risk mitigation” reason. In case of a security breach, service unavailability or otherwise, you want to reduce the risk that this fault spreads to other systems (be it getting to internal documents or putting more services down).

Another aspect administrators do with systems in their DMZ is system hardening, which I will talk about in the third part.