Simplicity is a form of art...

Comparing Hadoop with mainframe

At my work, I have the pleasure of being involved in a big data project that uses Hadoop as the primary platform for several services. As an architect, I try to get to know the platform's capabilities, its potential use cases, its surrounding ecosystem, etc. And although the implementation at work is not in its final form (yay agile infrastructure releases) I do start to get a grasp of where we might be going.

For many analysts and architects, this Hadoop platform is a new kid on the block so I have some work explaining what it is and what it is capable of. Not for the fun of it, but to help the company make the right decisions, to support management and operations, to lift the fear of new environments. One thing I've once said is that "Hadoop is the poor man's mainframe", because I notice some high-level similarities between the two.

Template was specified incorrectly

After reorganizing my salt configuration, I received the following error:

[ERROR   ] Template was specified incorrectly: False

Enabling some debugging on the command gave me a slight pointer why this occurred:

[DEBUG   ] Could not find file from saltenv 'testing', u'salt://top.sls'
[DEBUG   ] No contents loaded for env: testing
[DEBUG   ] compile template: False
[ERROR   ] Template was specified incorrectly: False

I was using a single top file as recommended by Salt, but apparently it was still looking for top files in the other environments.

Yet, if I split the top files across the environments, I got the following warning:

[WARNING ] Top file merge strategy set to 'merge' and multiple top files found. Top file merging order is undefined; for better results use 'same' option

So what's all this about?

Using salt-ssh with agent forwarding

Part of a system's security is to reduce the attack surface. Following this principle, I want to see if I can switch from using regular salt minions for a saltstack managed system set towards salt-ssh. This would allow to do some system management over SSH instead of ZeroMQ.

I'm not confident yet that this is a solid approach to take (as performance is also important, which is greatly reduced with salt-ssh), and the security exposure of the salt minions over ZeroMQ is also not that insecure (especially not when a local firewall ensures that only connections from the salt master are allowed). But playing doesn't hurt.

Trying out imapsync

Recently, I had to migrate mail boxes for a couple of users from one mail provider to another. Both mail providers used IMAP, so I looked into IMAP related synchronization methods. I quickly found the imapsync application, also supported through Gentoo's repository.

New cvechecker release

A short while ago I got the notification that pulling new CVE information was no longer possible. The reason was that the NVD site did not support uncompressed downloads anymore. The fix for cvechecker was simple, and it also gave me a reason to push out a new release (after two years) which also includes various updates by Christopher Warner.

So cvechecker 3.6 is now available for general consumption.

Switching focus at work

Since 2010, I was at work responsible for the infrastructure architecture of a couple of technological domains, namely databases and scheduling/workload automation. It brought me in contact with many vendors, many technologies and most importantly, many teams within the organization. The focus domain was challenging, as I had to deal with the strategy on how the organization, which is a financial institution, will deal with databases and scheduling in the long term.

Getting su to work in init scripts

While developing an init script which has to switch user, I got a couple of errors from SELinux and the system itself:

~# rc-service hadoop-namenode format
Authenticating root.
 * Formatting HDFS ...
su: Authentication service cannot retrieve authentication info
(Ignored)

Custom CIL SELinux policies in Gentoo

In Gentoo, we have been supporting custom policy packages for a while now. Unlike most other distributions, which focus on binary packages, Gentoo has always supported source-based packages as default (although binary packages are supported as well).

A recent commit now also allows CIL files to be used.

Using multiple OpenSSH daemons

I administer a couple of systems which provide interactive access by end users, and for this interactive access I position OpenSSH. However, I also use this for administrative access to the system, and I tend to have harder security requirements for OpenSSH than most users do.

For instance, on one system, end users with a userid + password use the sFTP server for publishing static websites. Other access is prohibited, so I really like this OpenSSH configuration to use chrooted users, internal sftp support, whereas a different OpenSSH is used for administrative access (which is only accessible by myself and some trusted parties).

Maintaining packages and backporting

A few days ago I committed a small update to policycoreutils, a SELinux related package that provides most of the management utilities for SELinux systems. The fix was to get two patches (which are committed upstream) into the existing release so that our users can benefit from the fixed issues without having to wait for a new release.