Simplicity is a form of art…

The last isolation-related aspect on risk mitigation is called application firewalls. Like more “regular” firewalls, its purpose is to be put in front of a service, controlling which data/connections get through and which don’t. But unlike these regular firewalls, application firewalls work on higher-level protocols (like HTTP, FTP) that deal with user data rather than with connection routing. I’m going to call these firewalls “network firewalls”, although most modern network firewalls have some application firewall functionality as well.

The purpose and necessity of network firewalls is well known and understood: make sure that the service is only accessible from the right location, check if connections aren’t abused (or too many connections are made), etc. But what if the connection itself is valid? After all, most abuse of services is not because they originate from the wrong location or try to access the wrong service. Instead, such abuse comes from valid access to the application, but with less kosher intentions. So what can application firewalls do in this case?

• Because they perform inspection of the data that is transferred itself, application firewalls can detect malicious data fragments or attempts to abuse the service. These detection rules can be based on general, heuristic rules (well-known examples are detection rules for cross-site scripting attacks (XSS) or SQL Injection) but can also be very specific to a particular application.
• Because all data is transferred through the firewall and the firewall has knowledge of the application itself, these firewalls offer advanced auditing features since they can detect authentication steps, user data, application-specific transactions and more.
• With knowledge of the users’ session and behavior (application-level) and origin (network level), application firewalls can detect and prevent unauthorized sessions, such as the case with session hijacking or even man-in-the-middle attacks (based on behavior detection)

Implementing an application firewall however doesn’t only mean that you improve access controls on it. It has other advantages that make application firewalls an important part in many architectures:

• If all service access is forced through the application firewall (for instance through an IP filter on the service that only allows connections from the application firewalls) you can implement rules that deter known attacks/vulnerabilities without needing to fix the code itself (or if fixing is possible, lower the time pressure). For instance, for Apache-based services, such an application firewall could detect or even change the Range: header on malicious requests to lower the impact of this potentially nasty DoS vulnerability
• Depending on the complexity, some functional application bug fixing might even be possible. For instance changing content types on requests/replies (HTTP), adding a domain on an FTP accounts’ login statement, …
• Many application firewalls (or gateways) offer proxy functionality which might improve response times. This is not a sure-given, since most applications are session-aware so the advantage is only for session-agnostic requests (be it static content or specific SQL statements in case of a database firewall). But also in case of session-aware statements can an improvement be found. Consider a database firewall which translates SQL statements from an unsupported application towards better defined statements (for instance using proper indexes or materialized views).
• In some cases, you might even be able to upgrade a backend of an unsupported application (which previously required an outdated version of that database) by translating the backend requests when they are incompatible with the new backend version. So you can improve integration or support unsupported upgrades.
• In case of risk reduction, application firewalls also allow you to move a service elsewhere (even in the public cloud) and still keep the access under control.

Of course, it would be TGTBT (Too Good To Be True) if there isn’t an (important) downside: maintaining the application firewall is a daunting task. Because of its flexibility, you’ll need deep knowledge in the application firewall administration and development, keep track of all rules you have (and why you have them), do lots and lots of testing on each rule (since it might affect the functioning of the application) and still be aware that subtle differences introduced by the application firewall rules can pop up unexpectedly. Also, integrating an application firewall is another service between your customer and his service, which might influence performance but also makes the underlying architecture more complex. Finally, you’ll need to consider that an application firewall requires lots of resources (CPU/memory), especially when it needs to perform SSL/TLS termination. Oh, and they’re often expensive too.

Still, even with these downsides, application firewalls are an important part of the service isolation strategy, which is a key aspect in the risk mitigation strategy which this series started with. We’ve focused on three now: service isolation (network-wise), process isolation (through mandatory access control) and now access isolation through application firewalls. And with proper hardening in place, I believe that you have done all you can do to reduce the risks when running unsupported software (apart from upgrading it or switching towards supported software).

If you have other ideas that benefit risk mitigation, with specific focus on unsupported software, I would be glad to hear about them.

While I’m writing this post, my neighbor is shouting. He’s shouting so hard, that I was almost writing with CAPS on to make sure you could read me. But don’t worry, he’s not fighting – it is how he expresses his (positive) feelings about his religion.

Security is, for some, also a religion. They see risks and vulnerabilities and what not everywhere. They’re always thinking every system in the world is or will be hacked in the near future and are frantically trying to secure every service they are running – and more. But security is also a real-life issue. If you take a look at the compromised GlobalSign website (who mentions that the website is an isolated one – as I described earlier) I hope that you look at security as being a functional requirement in architecturing and design (and not a non-functional one as many frameworks suggest).

And as you can see from the example, isolating services is not sufficient to prevent a successful exploit of an insecure or unsupported software (the reason why I started with this series). One additional measure that you can take is hardening the server and service.

The act of hardening a server and service is to configure the system so that it is as secure as possible, based on configuration entries. Many vendors and projects offer a security guide (like the Gentoo Security Handbook or the Fedora Security Guide) although most of them add this as part of their standard administrative documents (like the PostgreSQL “Server Setup and Operation” chapter).

But for some reason, you’ll find that default installations – even when following the instructions of the vendor – are not as secure as you want it to be. As a matter of fact, if you come in contact with auditors, you’ll probably fail any audit if you use a default installation. To help administrators to secure their services, you will find lots of third party sites offering advice on securing the operating system and the services running on it. These guides are what you will need to “harden” your system.

• OWASP, which stands for Open Web Application Security Project, hosts some hardening guides and suggestions together with test scenarios. For front-end application servers (mostly web application servers) you will find lots of interesting resources in the OWASP site (and surrounding community).
• Google is probably the best resource for finding hardening guides for your operating system or service. Just look for “hardening foo” and you will be reading for a week.
• CISecurity, or “Center for Internet Security”, is another one with a larger portfolio on hardening guides. Not only does it offer these guides (which it calls “benchmarks”), but organizations can also become a member and as such benefit from tooling that CISecurity supports for the validation of benchmarks (i.e. test if the system/deployment is compliant towards a particular benchmark). It does that by developing the benchmarks in a open specification called OVAL (the Open Vulnerability and Assessment Language) and XCCDF (XML Configuration Checklist Data Format). And CISecurity is not the only one there.
• Another such resource is the National Vulnerability Database (national for US residents, that is ;-) There you can find and download the OVAL/XCCDF resources for various software titles and operating systems. But as you can imagine from the abbreviations, the resources are XML files which are not made to be read by humans.

Although you can use the tool(s) that CISecurity offers, another possibility is to use Open-SCAP, an open source framework for handling SCAP, OVAL, XCCDF and other such open specifications on a system. Its documentation offers a first glance at what it can support.

However, this brings on he disadvantages of hardening services…

1. Hardening a system and its services is a time consuming job. Its only purpose is to reduce the impact of exploited vulnerabilities and reduce the “attack surface” so that exploits on unused functions are not possible.
2. Hardening a system and its services can impact the service. Make it too tight, and it might not behave anymore like you want it to.
3. Also, since there are many, many resources “out there” on hardening, you will have to manage your hardening rules, document them for yourself. It is also advisable to document the rules you are not implementing, if not just for future’s sake.
4. The hardening guides also require quite some expertise on the service. If you are not experienced with the service but you need to harden it, you can be lucky and just implement what is suggested and hope for the best, but usually you will need to dive deeper in the subject and make (tough) choices.

Although specifications like SCAP exist to help you in your hardening exercises, these are still difficult to manage (do not try to write OVAL/SCAP/XCCDF content in your favorite text editor). Its adoption however by Fedora and RedHat is showing a positive effect on the tools surrounding this specification. I will be writing about SCAP, OVAL and XCCDF later since I too see good use of it in organizations (or even free software projects).

Does that mean that hardening is not beneficial? On the contrary:

• You gain lots of knowledge in the matter, and also forces you to think about integration aspects. Since you are responsible for the service (or the damage that could be made if the service is exploited) being knowledgeable is definitely a good thing.
• A considerable amount of vulnerabilities that are and will be reported on the service (check CVE details to find out about publicly known vulnerabilities, documented in the CVE specification) will not have their effect on a well hardened service. Or put another way, you will reduce the number of real vulnerabilities in your service. You will not be able to exclude all vulnerabilities, but the projected number is high – a fully hardened Windows or Linux system can mitigate up to 90% of the exploits on the operating system. It will considerably reduce the risks that you and your organization are taking.
• A well defined hardening guide will also offer the means to automatically audit or check if the system is still compliant to the hardening setup you envisioned. Scheduled regularly, this will ensure that your configurations are not drifting away, back to a more vulnerable setup, for whatever reason.
• By removing the functions that the service should not offer, you make sure that the use of the service is per the organizations’ guidelines. (Internal) abuse of the service is made more difficult, so users are forced to take the regular way. Unlike service isolation, which allows you to keep track of data/service flows, hardening makes sure that side-functionality is not used without your consent. Or to put it more blunt, “Yes I know Oracle DB can be used to schedule tasks on the operating system, but no, you’re not allowed to use that function”.

And who knows, perhaps by optimizing the configuration, it might run faster with a lower resource footprint ;-) If it does, that’s perfect, since the next topic on risk mitigation will have a negative influence on performance: mandatory access control.

Internet: absolute communication, absolute isolation
~Paul Carvel

The quote might be ripped out of its context completely, since it wasn’t made when talking about risks and the assurance you might need to get in order to reduce risks. But it does give a nice introduction to the second part of this article series on risk mitigation. After all, if the unsupported software is offering services to the Internet, you really want to govern the communication and isolate the service.

When you are dealing with a product or software that is unsupported (be it that it will not get any patches and updates from its authors or vendor, or there is no time/budget to support the environment properly), it is in my opinion wise to isolate the service from the rest. My first post on the matter gave a high-level introduction on the risks that you might be taking when you run unsupported (or out-of-support) systems. Service isolation helps in reducing the risks that others have when you run such software on a shared infrastructure (like in the same network or even data centre).

By isolating the unsupported service from the rest, you create a sort-of quarantine environment where sudden mishaps are shielded from interfering with other systems. It provides insurance for others, knowing that their (supported) services cannot be influenced or jeopardized by issues with the unsupported ones. And if these services need to interact with the isolated service, the interface used is known and much more manageable (think about a well-defined TCP connection versus local communication or even Inter-Process Communication). But it goes beyond just providing insurance for others.

Isolation forces you to learn about the application and its interaction with other services. It is this phase that makes it extremely important in an environment, because not knowing how an application works, behaves or interacts creates more problems later when you need to debug issues, troubleshoot performance problems and more. Integration failures, as described in my previous post, can only be dealt with swiftly if you know how the service integrates with others.

Another advantage of proper service isolation is that you can fix its dependencies more easily. Remember that I talked about upgrade difficulties, where a necessary upgrade for one component impacted the functionalities of the other (unsupported) component? With good isolation, the dependencies are more manageable and controllable. Not only are (sub)component upgrades easier to schedule, it is also a lot easier to provide fall-back scenario’s in case problems occur. After all, the isolated service is the only user so you have little to fear if you need to roll-back a change.

But what is proper service isolation?

• First of all, it means that you focus on running the (unsupported) software alone on an operating system instance. Do not run other services on the same OS, not even if they too are unsupported. The only exception here is if the other services are tightly integrated with your service and cannot be installed on a separate OS. But usually, full service isolation is possible.
• Next, strip the operating system so it only runs what you need for managing the service. Put primary focus on services that are accepting incoming connections (“listening”) and secondary focus on allowed outgoing protocols/sessions (and the tools that initiate them).
• See if you can virtualize the environment. In most cases, the service does not require many resources so it would be a waste running it on a dedicated system. However, in my opinion, a much better reason for virtualization is hardware abstraction. Sure, all operating systems tell you that they have some sort of Hardware Abstraction Layer in them and that they can deal with hardware changes without you noticing it. But if you are an administrator, you know this is only partially true. Virtualization offers the advantage that the underlying hardware is virtual and can remain the same, even if you move the virtualized system to a much more powerful host. Another advantage is that you might be able to offload certain necessary services from the OS (like backup) to the host (snapshotting).
• Shield the operating system, network-wise, from other systems. Yes, that means putting a firewall BEFORE the operating system guest (and definitely not on the OS) which governs all traffic coming in and out of the environment. Only allow connections that are legit. If your organization has a huge network to manage, they might work with network segment filtering instead of IP-level filtering. See if you can get an exception to that – managing the rules should not give too much overhead since the system, being unsupported and all, is a lot less likely to get many connectivity updates.

But before finishing off, a hint about stripping an operating system. Stripping is much more than just removing services that are not used. It also means that you look for services that are needed, and see if you can externalize them. Common examples here are logging (send your logs to a remote system rather than keeping them local), e-mail (use simple “direct-out” mail) and backup (use a locally scheduled backup tool, or even offload to the host in virtualized systems), but many others exist.

Of course, service isolation is not unknown to most people. If you run a large(r) network with Internet-facing services, you probably isolate those in a DMZ environment. That is quite frankly (also) for the same “risk mitigation” reason. In case of a security breach, service unavailability or otherwise, you want to reduce the risk that this fault spreads to other systems (be it getting to internal documents or putting more services down).

Another aspect administrators do with systems in their DMZ is system hardening, which I will talk about in the third part.