web.luchs.at News Service

Of File Servers and Compulsive Hoarders

Sat, 07 Jan 2012 01:13:31 CET

Digital assets have a major drawback compared to their solid counterparts in the analogue world - you cannot see them. While most people hail this attribute as one of the major building blocks of the digital age, most system administrators tend to disagree. This is especially true if you ever had to switch the hardware of file servers. Usually keeping the data is the goal of upgrades of any kind. When changing storage devices this involves a lot of copying and inspecting the logs of copied file and directory names reveals - your file server is dealing with compulsive hoarders!

We are fully aware that keeping track of every single file and directory on your typical computer, mobile phone or digital gadget is next to impossible. The same is true for networked environments where user data is stored centrally. The typical work environment features thousands of files. Settings, temporary files, icons, e-mails, caches, credentials, multimedia files, documents and lots of stuff needs to be stored for every user. Most people think that storing data is cheap and thus they don't care. Storage media keeps getting larger and cheaper, right?

Wrong! Let's state it clearly: Proper storage is not cheap!

But why? It depends how valuable the data is for you. Storage is only cheap if you can afford to lose data. If you don't care if your data disappears from one second to the next, then storage is really cheap. However if you want to keep your data for a longer period of time, then you need copies. Lots of copies! A typical storage environment features mirrored disks, a primary backup server, possibly a secondary backup server (or storage media) and an archive. Assuming you go with a minimal amount of mirroring (double) per stage, then you end up with a factor of 8! This means that multiplying your cheap disk from the store with a factor of 8 gets you much nearer to the real price of storing data. And we haven't even talked about data transport yet. Have you ever tried to copy your live data via USB 2.0 or 100 Mbit/s Ethernet? Both methods are cheap and widely used. Once you need to copy 500+ GB you start to think about faster data transports, thus increasing the effort for proper storage and raising the costs.

So, do you know what the cheapest part of proper storage is? Deleting data you do not need any more. You can do this any time. Please do it! Your system administrator and all devices in the backup chain will be glad. Deleting data is a small step for you, but it is a big step for your storage systems.

Note: Yes, we know about data deduplication and other features of storage subsystems, but this isn't the point. There is no silver bullet and we cannot use endless resources forever.

Avoid Wi-Fi Protected Setup

Fri, 30 Dec 2011 19:40:26 CET

Wireless network access points have a feature called Wi-Fi Protected Setup (WPS). The idea is to facilitate the configuration of the access point since abbreviations such as TKIP, AES, WPA2, PSK, etc. can be quite intimidating for anyone not used to network and security protocols. Security researchers have found a serious weakness in the WPS protocol.

Security researchers Stefan Viehböck and Craig Heffner has published a description of the vulnerability along with a tool to prove the existence of the weakness. Basically WPS substitutes the security of pass phrases with a PIN code. Due to the nature of WPS an attacker can guess this code very easily and obtain the configured pass phrases. The attacker only has to guess the correct code out of 11,000 codes which dramatically reduces the time for attacks.
If you have used WPS, please consider deactivating it. Pick a random string for your pass phrase (at least 16 characters, 63 is the maximum supported). We recommend 63 characters. You can create QR codes for entering this code when using mobile devices such as smart phones. Use the security setting WPA2 with a fixed pass phrase (PSK) and select AES encryption (sometimes abbreviated CCMP, most modern devices support AES, so TKIP is not needed any more). To sum everything up into a couple of steps.

Pick a random string for your pass phrase (16+ characters, 63 maximum)
Select WPA2.
Select AES/CCMP.
Enjoy.

While the options of configuring wireless security settings can be intimidating, please consider deactivating WPS and following the steps described. The alternative is to wait until vendors provide firmware updates for the wireless routers, but changing from WPS to a direct configuration can be done more quickly.

Ghosts Inside the Shell - Hardware Failures

Wed, 21 Dec 2011 00:38:00 CET

The fortune cookie collections claims that hardware consists of the parts that can be kicked. This is especially true if something fails. Apart from the material there's often firmware involved which can also fail (algorithms are human, they have stress, too). We have two stories for you involving failed hardware.

Using redundant arrays of independent disks (RAIDs) sounds like a good idea. Have plenty of copies of your data and less worries. That's the idea on the surface. Below you'll find that mirroring data can also mirror deletion of data equally well. Then there are more complex RAID algorithms that use parity and checksums in order to deduce lost data from spare information. Complex is bad, and if only the firmware knows where your data is you probably won't in an emergency.
And then there is silent data corruption. A combination of faulty firmware and faulty hardware can destroy your file system(s) without warning. This happened to a logical volume spanning two RAID1 mirrors. There were no errors in the logs of either Linux kernel, RAID controller or server BIOS. Instead the Linux kernel got I/O errors when accessing the RAID1 containers, but no disk was marked as faulty and no RAID volume was marked degraded. Finally the JFS on the volume suffered a catastrophic failure and could not use its transaction log after a hard reboot of the locked server. A port mortem analysis of the file system and the hardware yielded no indication for the cause.

A different case was presented by a GNU/Linux router/firewall system. The hardware was an Mini-ITX board with three network interface cards, 1 GB RAM and crypto-acceleration in the CPU. The system worked flawless for over two years until the machine froze spontaneously during operation. The console stayed black, no input and no reset by keyboard was possible. The network interface cards were not reachable, too. Logs on the system showed no entries around the time of the freeze. Timestamps on the file system and files with 0 bytes indicated that the crypto-acceleration might have been in use at the time of the failure. After rebooting the firewall system selected Netfilter rules stopped working (about 3 out of 500+) including the NAT rules for SIP packets on port 5060/UDP. One NAT rule could be „repaired“ by switching the IP address of one server in the DMZ.
After switching the hardware and using the same set of rules on a different system all rules worked again, also in the form prior to changing the server address.

The hardware in question still needs to be examined in depth. Regardless of the results, you cannot trust any component of your infrastructure without regular maintenance.

The Perils of App Stores and App Markets

Tue, 29 Nov 2011 16:21:28 CET

App stores and „markets“ for software are very popular these days. Most of the buyers and downloaders believe that the software in these stores is readily available and can be used at will. You just open the store application, browse the content and click/touch/smear to get your software. This is the theory. In practice the software offered can be very volatile and disappear without warning. Just take the VLC player or RedPhone/TextSecure as example.

The aquisition of Whispersystems by Twitter is the most recent case. Twitter acquired Whispersystems, the vendor of hardened Android software and secure communication tools. Immediately after the deal their apps were unavailable in the Android market. They even shut down the RedPhone servers, thus denying users around the globe secure end-to-end communication. TextSecure, a tool to use OTR-style encryption for text messages, continues to work, but unless you have backups you cannot install it any more. Why Twitter bought Whispersystems and why they took their software offline remains a mystery. The time of these events couldn't be worse. Whispersystems published the tools to help people communicate during the Arab Spring. Now they deny the same users the use of these tools during Egypt's election.

So beware of App Stores and Markets. Unless you get the source code, you don't own anything. Expect any software without disclosed source code to vanish at any given moment.

Trusted Computing is back to compromise your Systems

Sun, 30 Oct 2011 18:01:29 CET

Do you recall the discussions about the Trusted Computing (TC) platform introduced several years ago? The idea was to introduce a trust relationship for code that gets executed on your own computing devices. Ross Anderson has explained the mechanism and its consequences in plain English on his personal web site in 2003:

…TC provides a computing platform on which you can't tamper with the application software, and where these applications can communicate securely with their authors and with each other. The original motivation was digital rights management (DRM): Disney will be able to sell you DVDs that will decrypt and run on a TC platform, but which you won't be able to copy. The music industry will be able to sell you music downloads that you won't be able to swap. They will be able to sell you CDs that you'll only be able to play three times, or only on your birthday. … TC will also make it much harder for you to run unlicensed software. In the first version of TC, pirate software could be detected and deleted remotely. … TC will protect application software registration mechanisms, so that unlicensed software will be locked out of the new ecology. Furthermore, TC apps will work better with other TC apps, so people will get less value from old non-TC apps (including pirate apps). Also, some TC apps may reject data from old apps whose serial numbers have been blacklisted. If Microsoft believes that your copy of Office is a pirate copy, and your local government moves to TC, then the documents you file with them may be unreadable. TC will also make it easier for people to rent software rather than buy it; and if you stop paying the rent, then not only does the software stop working but so may the files it created. So if you stop paying for upgrades to Media Player, you may lose access to all the songs you bought using it.…

The problems introduced by TC do not stop here. TC can help with remote censorship and hide malicious software from you. The latter is especially interesting since the discovery of state-sponsored malicious software found in Germany. There is no trust if you cannot control your own hardware. The architecture of the Trusted Computing platform merely takes control away from you and gives it to the hardware and software vendors. This is not a trusted computing platform and it opens up a whole set of questions. Given then fact that the third-party trust model has been broken by the security breaches of several certificate authorities (such as Comodo and DigiNotar).

So we strongly support making UEFI secure boot available to all users and not only to the consortium of the Trusted Computing Alliance.

Secure your Communication Lines

Tue, 30 Aug 2011 19:21:14 CET

The events in Libya allow a rare glimpse behind the curtain of a government who uses digital surveillance technology against its citizens. It will take some time for the new government to assess the damage in terms of privacy violations, espionage and impact on the security of dissidents. The rebels found stashes of intercepted information ranging from instant messengers, videos, phone calls to e-mails. The eavesdropping was done by technology from companies in Western countries. This is a clear signal to businesses to secure their communication lines and to thoroughly scrutinise the promises of vendors.

Most of us take private conversations for granted. If you write an e-mail, a text message or make a phone call, then the gadgets you use give you the illusion of privacy. In reality next to none product and protocol takes measures to guard the information you are sending or receiving. Usually you have resort to extra effort, and in most cases you cannot reliably protect a communication line due to interoperability problems (maybe the end-point is still analogue or doesn't support certain protocols). You have to be aware of these issues and you have to define what communication lines can be used for which information. This is a very important step. Do not start with the technical issues. Start with an inventory of your communication habits and the data you usually transmit. Technical measures are always second. If you don't know how your internal processes use communication, then you can't do anything to improve their security.

Follow the events in Libya and learn about existing methods for compromising communication. The products are out there and they are not advertised publicly. Of course this doesn't mean that no one uses them. Keep an open mind and a tight grip on your company's digital assets.

Secure Communication for Businesses

Mon, 15 Aug 2011 15:57:11 CET

Every now and then there is talk. If you run a business, you know what we are talking about. Every day we communicate. There are phone calls, text messages, e-mails, web portals, bulk data transfers, faxes, and more transmissions we have to deal with during a normal day at work. A part of this communication transports important information such as logins/passwords, offers, invoices, reports or personal data (any piece of information linked to a person). Most of the time our communication channels are up and running. This has not to be the case. The recent events in UK, the San Francisco underground, the regime in Syria (and Libya and Egypt), or the discussion about the Internet kill switch show that communication lines are always a prime target for attackers. The motivation of the attacker doesn't play a role if your business is in risk of being cut-off from networks.

You might want to spend some time preparing for blackouts or eavesdroppers before you encounter any one of these threats. Travellers are well aware that you can be disconnected faster than you anticipate. Virtual private network (VPN) links do not work in all countries. Mobile phone networks must not be trusted for sensitive information in general. GSM has been successfully attacked already (and will probably follow the path of early Wi-Fi networks in terms of security). GPRS has been attacked as well. The results were presented at the Chaos Communication Camp. This is no news to Internet veterans or members of NGOs threatened in countries abroad and domestically.
The lesson is always the same: You have to add extra layers of security. You have to consider using your own keys and what to do with these keys. Secure communication between and inside groups boils down to proper key management. This means you can start working on your security by making sure you can organise the switch to secure protocols. Simply stating "Let's encrypt!" won't get you far.

You can try secure communication in small steps to get used to the complexity involved. We use GPG, a cryptographically secured drop box and TextSecure for Android among other things. Drop us some ciphertext.

Do you care about your data?

Fri, 10 Jun 2011 11:27:35 CET

Eran Feigenbaum, Google's security teleevangelist, has told the world that it doesn't matter where you data is. All that matters is securing your data, no matter where. In theory this is correct, but once your data is out of reach your choices for keeping track of your data and deploying security measures is limited. Don't get us wrong, this is not about Cloud Bashing. The ominous Cloud is part of IT infrastructure, and you can't whip up an elastic Cloud with high-performance computing out of your hat or basement. However Mr. Feigenbaum's view seems to be „clouded“ by marketing. One of his examples is the trace of an e-mail message that bounced through five countries. Certainly few people care about how their e-mails get transported, but some do, and most others don't e-mail databases with customer data around (some do though).

There are other problems connected with data locality and service providers. You have to tell your customers where their data is. If you can't tell, then you have to say so. If you rely on outsourcing, then you should have the guts to admit that maybe a crucial part of your business is out of your hands. If your only technical expertise is relying on a few lines of „SLA code“, then tell your customers; but please do not use smoke and mirrors and hide these little known facts.
During the past years the term „too big to fail“ has failed. The Cloud has lost data, „stable“ banking institutions have disappeared, earthquakes have created a triple nuclear meltdown, SecurID has issues, and many more similar events have happened. Frankly not caring where your data is, is the same as not caring at all.

We know where our data is. Customers hire us to make sure they know where their data is. Sometimes customers even approach us to retrieve data whose location is unknown. Make sure you build your information infrastructure on a solid foundation.

DropBox lied about security measures

Sat, 14 May 2011 12:43:06 CET

Using the Cloud (whatever this word really means) for storage has become more and more convenient. Cloud storage is hailed as secure, fast, cheap, stable, efficient and more; it is truly the dream of every marketing department. "Hassle-free, batteries not included, your mileage may vary, …", but let's not forget: "There ain't no such thing as a free lunch"

The DropBox' cloud storage service is advertised with the slogan "Your stuff is safe". However the company reviewed its website claim about security. They changed the sentence "All files stored on Dropbox servers are encrypted (AES256) and are inaccessible without your account password." to "All files stored on Dropbox servers are encrypted (AES 256)." which carries a vastly different meaning. There's more. Another part of the claims were changed, stating clearly that the service provider has access to the user's data: …we have a small number of employees who must be able to access user data for the reasons stated in our privacy policy…
The article where the quotes came from has a detailed review of the security claims and the actual security provided. It's well worth reading before fully trusting this service - which is true for all services provided by third-parties.

Remember, your data is only secure if you have the encryption keys and if you control who has access to these keys. If you use the Cloud or any other service not under your control, you will most certainly not possess the keys, thus someone else will always have access to your data. Keep this in mind. Security needs to be designed and reviewed. Make sure that you plan ahead before distributing your data all around the world.

Cloud Volatility in Production Environments

Thu, 10 Mar 2011 12:45:13 CET

You have probably heard of cloud computing already. The cloud can help you to save your own infrastructure by leasing from some other company, organisation or individual. It's basically the old Software As A Service (Saas) in new clothing with some added virtualisation and management technology. While we still use physical and virtual resources ourselves, some products of the Cloud are quite convenient. However you won't be safe if you don't put some planning into your infrastructure, regardless if it is your own or somebody else's. The following message was sent to us yesterday:

Dear customer,
unfortunately there was a problem with a defect RAID controller. The filesystem was destroyed and all data of the virtual servers were deleted. The RAID controller was replaced by a new one. We reinstall the hardwarenode at the moment, so that the customers can recreate their virtual servers and restore the data from their backup on the backup box.

We have no idea what the exact infrastructure of our recently deceased virtual server looks like, but we have a backup stored on our own infrastructure. Don't go around in Cloud Land and trust everybody. You have to roll-out and plan the use of rented infrastructure as well. True, you save some effort maintaining hardware, but that's about it. Be careful out there and get some advise from professionals before you put critical data at risk. Sometimes it's even worthwhile to build your own cloud. Let us know if you need help.

Apple exploits Free Software and denies it to its customers

Sun, 09 Jan 2011 21:09:05 CET

The Apple Mac OS X system features Free Software, just as many UNIX-like operating systems do. There's no harm in that. It is good practice to reuse code that has been widely tested and is actively developed. So its fair to say that Apple benefits from Free Software. Strangely it doesn't allow its customers to have these benefits. Apple explicitly forbids Free Software in the Apple App Store’s Terms of Service (ToS). One of the first victims are users of the popular VLC media player. The VLC player has been pulled from the App Store because of its Free Software licensing.

The opinion of the VLC developer community on this matter is divided. In October a developer named Rémi Denis-Courmont contacted Apple regarding the GPL and Apple's redistribution of the code. Apparently Apple has no interest in honouring the license and simply decided to remove software in order not do deal with the licensing issues. Brett Smith, FSF Licensing Compliance Engineer, summarises: „Apple ‘only’ allows you to do the activities in the list of Usage Rules, if an activity does not appear in this list, you’re not allowed to do it at all.“

So far the Android platform has not presented an incompatibility with Free Software. Furthermore you are not required to pay a fee for writing code for Android. Our recommendation is to stay clear off Apple and focus on other platform that do not forbid the development of Free Software.

Things you cannot do with a Cloud Desktop

Fri, 17 Dec 2010 11:33:29 CET

Google Cr-48 has created some news already. The idea is to have a lightweight laptop that boots from the network, doesn't store any data locally and lets you live in the cloud, desktop-wise. That's good news, isn't it? Well, in theory it is. The practice looks a lot different. Face it, you don't have quality Internet access everywhere. You may find 3G in cities, but more often than not your mobile Internet access will drop to 2G leaving you with ISDN speed (128 kbit/s). If you're lucky you may have a Wi-Fi network nearby, but these are often locked down in terms of access or quality. Finally there's the fading net neutrality which means that your ISP may soon charge extra for booting your fancy Chrome OS (it's content delivered by Google after all).

So, what's the problem? Well, here's a list of things you cannot do with your shiny Cloud OS:

You cannot work offline. No network, no work. True, most of us need Internet access for working, but right now I can work offline without problems - because I can boot without network and have all the software I need.
You cannot share your phone's data connection with a USB tether. This means you won't get online with your Cr-48 while you can get online easily with our cell phone.
You cannot use any file type you want. Google determines which file types can be opened, viewed and accessed. Right know this is .doc, .pdf, .html and various image file types. Amazing, isn't it?
You cannot access and manage USB storage devices. Your USB port is for mouse, Internet dongle and keyboard. No USB drives, no iPad, no Android phones, no USB sticks, nothing.
You cannot watch high-quality streaming video. The author of the source article tried, but it doesn't work.

It seems that having a "100% Web-powered" laptop doesn't make sense right now. Your computer has basically all the advantages of a full-frontal lobotomy. The only benefit is a certain coolness factor. Get a decent laptop, get a GNU/Linux operating system, use online storage such as Ubuntu One and you have much more than Cr-48 has to offer.

Chrome OS: 10 things you can (and can't) do with the Google PC

Upgrading Debian 5 to Debian 6

Sun, 05 Dec 2010 14:18:14 CET

Yesterday we tested the upgrade path of a Debian 5.06 desktop to Debian 6.0. The main interest was to see how the new system would boot and what would happen to the encrypted root partition formatted with Ext4 (the desktop ran a custom 2.6.36 kernel). The whole process took about three hours, most of it was due to the download of the packages. The upgrade itself went smoothly, but took about an hour as well (because of big packages such as OpenOffice and TeXLive).

There is one issue to report. The update replaced the grub 0.97 package with the new grub 2 package. Since the root file system uses the data=writeback option it was necessary to insert the appropriate rootflags boot parameter. Another problem was that the Ext4 driver was compiled as module and would not be loaded in time to mount the / partition. The system tried to mount it as Ext3 which failed due to specific Ext4 options present. The resulting error message was: couldn't mount because of unsupported optional features (40) Booting a Grml live CD and recreating the init RAM disk image with mkinitramfs reenabled the / partition after boot. This is not a serious bug, and it stems from the non-standard setup of running Debian 5 with Ext4.

Apart from minor changes and some adaptions everything works fine. The system runs faster, it boots faster and awaits further tests.

Web site, Jabber, DNS and Postfix now run IPv6 too

Sun, 26 Sep 2010 01:19:59 CET

Today we extended the configuration of some services. The main web server, selected mail transport agents, the Jabber server and the main DNS server were connected via IPv6 too. If you query the DNS you will find additional AAAA records pointing to the new IPv6 addresses. The names associated with the service and selected virtual hosts now have IPv4 A and IPv6 AAAA records. Provided you use properly configured networks you want feel the difference. Everything should work as expected. You can try installing the ShowIP add-on in Firefox to see the IP address(es) of the site you are surfing on.

So far we encountered no major problems configuring the changes. You have to take care of the IPv6 Neighbour Discovery protocol locally, and of course you will need a stateful firewall that is capable of dealing with IPv6. Once you run IPv6 a lot of administrative tasks get a lot easier since you don't need to open NAT holes in your firewall setup anymore. Filtering is easier too. You just configure the filter rule with the "real" IP addresses.

Some Internet service providers already offer native IPv6 connectivity. No matter if you get connected natively or via tunnels, we can help to find a smooth migration path.

Linux kernel security bug, workaround and defence

Sun, 19 Sep 2010 14:52:48 CET

You have probably heard about the bug in the Linux kernel affecting 64-bit systems. It affects the 32-bit compatibility mode of the kernel. Attackers can gain root rights when exploiting this bug. This is bad news. The good news is that there is a fix available. Most GNU/Linux distributions have already rolled out a new kernel package. If you do regular updates you should be fine by now.

The bug shows a crucial point of modern computer security. Distributions such as Fedora or OpenSuSE use a mixed architecture environment. This means that when installing a 64-bit system you get all the 64-bit libraries plus a lot of 32-bit libraries so you can run either 64-bit or 32-bit code on your machine. This is convenient, but this is not necessary for all your installations. A better approach is to set up a "pure" 64-bit environment and add 32-bit compatibility layers if you need them. Ubuntu and Debian follow this way. You get a true 64-bit system. Our configurations even disable the 32-bit compatibility mode of the kernel if the machine is never intended to run 32-bit code. Bear in mind that leaving the 32-bit mode on also leads to a compatibility layer for malicious applications that get passed around as binaries. If your system has a clearly defined task, then use a clearly defined configuration. Don't go for the "just in case" extras if security matters to you.

You can switch off the 32-bit compatibility support by providing a different shell for 32-bit ELF binaries. The Workaround for Ac1db1tch3z exploit article below shows how to do this.