web.luchs.at News Service

Happy New 2013!

Mon, 31 Dec 2012 16:40:26 CEST

We have been busy with work, research and recreation in the past month. Since the year is coming to an end today, we wish you a happy 2013!

2013
2012

Windows 8 ties Success to Hardware Secrets

Sun, 16 Sep 2012 15:55:54 CEST

The computing technology has abandoned the race for higher Gigahertz clock frequencies. Instead we now have multiple processor cores per physical processor. With the rise of portable devices such as laptops, mobile phones and tablets the race shifted to power saving modes in order to conserve battery power. And now that we have a whole variety of computing devices at hand, the vendors try to reduce the choices for customers again.

Intel's flagship when it comes to small and portable computers is the Atom™ CPU family. We like Atom™ CPUs. They offer sufficient performance for most applications, have a x86-64 mode for 64 bit code and feature multiple cores (or at least hyperthreading). Tablets and laptops run on Atom™ processors, too. Since the chip is part of the traditional x86 family, you can run any software on it you want to. So everything's perfect, right? Enter graphical processing units (GPUs) or graphic cards.

A processor won't be enough. You also want to see something, i.e. attach a monitor or a touchscreen. Usually this is not a problem - unless the vendor doesn't provide a graphic driver or the necessary documentation to write one. Stubborn verndors have plagued Linux developer for decades. Intel is entering the fight for being stubborn and short-sighted with the new Clover Trail-based Atom™ processor product line. These will be Systems on a Chip (SoC) and integrate CPU with the GPU on the same physical package. This saves space and power. Sadly Intel has decided only to support Microsoft's Windows 8 operating system. This means that you cannot buy certain tablets and laptops for installing an GNU/Linux OS on it. This is essentially a step backwards for customers. It also shows that Windows 8 probably cannot convince by its features and usability and that it has to be pushed by hardware tricks (just as Apple does with iOS).

Developers and customers should take care when selecting a platform. If possible select a platform that is fully supported by the vendor in terms of providing documentation for software development. Since the Atom™ SoC components are aimed at end user products, the server components are not affected (yet). Intel's decision is clearly aimed at Google's Android OS in order to artificially keep it from competing with Windows 8.

Customers might want to avoid Clover Trail-based Atom™ processors and look to the Valley View Atom™ SoC for proper GNU/Linux support.

Linux File System revisited

Sat, 25 Feb 2012 15:27:16 CEST

You probably use a couple of file systems and you rely on them. Since the Linux kernel is a very easy to deploy test platform you will find dozens of file systems in the vanilla kernel source from kernel.org. In production environments you will often encounter Ext2/Ext3/Ext4, XFS and JFS. Recently BTRFS has been declared stable by Oracle® and enters use on production servers. So, which file system would you use?

The Ext2/Ext3/Ext4 family follows a conservative design and features certain backward compatibilities and upgrade paths. Very few people realise that Ext4 is a temporary solution that addresses shortcomings of Ext3 and bridges the gap until BTRFS is stable and widely available in GNU/Linux distributions. Ext4 works well for most workloads and storage scenarios. If you face the problem of storing lots of data or big data you should consider XFS. The XFS development team has fixed some performance issues with metadata changes. Beginning with the Linux 3.x kernel XFS is really a very good choice, even for desktop systems (we're running XFS on laptops, too).
BTRFS is a good choice because of its features useful for file system administration. It includes some of ZFS' capabilities. Due to its complexity and copy-on-write design it is not very suitable for fsync-intensive operations. If you watch the talk from the XFS developer from the Linux.Conf.Au 2012 conference you will get a look at some interesting benchmarks regarding this issue. XFS tries to put as few barriers as possible between the application handling the data and the actual block devices.

Selecting a good file system can be difficult (not as difficult as writing one). If you go through a selection, make sure you are comfortable with the tools creating and maintaining the file system. When things go wrong it's always nice to know what to do and what not to do.

Common Unix Printing System turns into Apple's Printing System

Thu, 16 Feb 2012 13:56:29 CEST

The Common UNIX Printing System (CUPS) is a widely used suite of software and protocols used for printing documents and organising printer queues. CUPS is superior to the LPR system commonly used on UNIX® systems in the past. CUPS is developed by Apple since 2007. CUPS features its own network protocol and discovery of neighbouring CUPS servers (called „CUPS browsing“). This is about to change for some of CUPS' features will be dropped beginning with release 1.6. The mechanism will be moved to using DNS-SD instead of UDP broadcasts. If you rely on CUPS printer discovery, then you are in for a change.

In theory using DNS-SD for local resource discovery is not a bad idea. Multicast has its advantages over broadcasts. However just adding Avahi on GNU/Linux system won't do the trick. You still need the proper service definitions on machines providing these services. Getting the proper file for putting it into /etc/avahi/services/ can be a very frustrating experience. Even Mac OS X needs manual intervention to discover printers on the local network, contrary to all the rumours about its „point-and-click“ approach and user friendliness. And we haven't talked about Microsoft Windows® yet where the discovery of services works differently. You can install the Bonjour implementation for Windows®, but according to our tests it works worse than on GNU/Linux or Mac OS X.

If you do not deploy multicast routing or have all your clients in the same multicast segment, then you will miss CUPS browsing. A classic example being road-warriors connecting via VPN. Unless there is bridging involved, you won't get the list of printer queues. Of course, you can deploy DNS-SD routing or proxying. The question is why, because CUPS browsing can be configured to use HTTP and fetch all queues from your master CUPS server.

It's a small step for Apple, but it's a giant leap backwards for many others. Maybe the reason for the changes is the essence of this statement taken from the changes announcement on the Fedora development mailing list: „Apple is removing some filters from CUPS as they are not needed for Mac OS X.“ If it's of no use to OS X, then drop it. Let other deal with non-OS X stuff: „These filters will be maintained in a new "cups-filters" project at OpenPrinting.“ Interesting idea, but quite a deviation from a common printing system.

Of File Servers and Compulsive Hoarders

Sat, 07 Jan 2012 01:13:31 CEST

Digital assets have a major drawback compared to their solid counterparts in the analogue world - you cannot see them. While most people hail this attribute as one of the major building blocks of the digital age, most system administrators tend to disagree. This is especially true if you ever had to switch the hardware of file servers. Usually keeping the data is the goal of upgrades of any kind. When changing storage devices this involves a lot of copying and inspecting the logs of copied file and directory names reveals - your file server is dealing with compulsive hoarders!

We are fully aware that keeping track of every single file and directory on your typical computer, mobile phone or digital gadget is next to impossible. The same is true for networked environments where user data is stored centrally. The typical work environment features thousands of files. Settings, temporary files, icons, e-mails, caches, credentials, multimedia files, documents and lots of stuff needs to be stored for every user. Most people think that storing data is cheap and thus they don't care. Storage media keeps getting larger and cheaper, right?

Wrong! Let's state it clearly: Proper storage is not cheap!

But why? It depends how valuable the data is for you. Storage is only cheap if you can afford to lose data. If you don't care if your data disappears from one second to the next, then storage is really cheap. However if you want to keep your data for a longer period of time, then you need copies. Lots of copies! A typical storage environment features mirrored disks, a primary backup server, possibly a secondary backup server (or storage media) and an archive. Assuming you go with a minimal amount of mirroring (double) per stage, then you end up with a factor of 8! This means that multiplying your cheap disk from the store with a factor of 8 gets you much nearer to the real price of storing data. And we haven't even talked about data transport yet. Have you ever tried to copy your live data via USB 2.0 or 100 Mbit/s Ethernet? Both methods are cheap and widely used. Once you need to copy 500+ GB you start to think about faster data transports, thus increasing the effort for proper storage and raising the costs.

So, do you know what the cheapest part of proper storage is? Deleting data you do not need any more. You can do this any time. Please do it! Your system administrator and all devices in the backup chain will be glad. Deleting data is a small step for you, but it is a big step for your storage systems.

Note: Yes, we know about data deduplication and other features of storage subsystems, but this isn't the point. There is no silver bullet and we cannot use endless resources forever.

Avoid Wi-Fi Protected Setup

Fri, 30 Dec 2011 19:40:26 CEST

Wireless network access points have a feature called Wi-Fi Protected Setup (WPS). The idea is to facilitate the configuration of the access point since abbreviations such as TKIP, AES, WPA2, PSK, etc. can be quite intimidating for anyone not used to network and security protocols. Security researchers have found a serious weakness in the WPS protocol.

Security researchers Stefan Viehböck and Craig Heffner has published a description of the vulnerability along with a tool to prove the existence of the weakness. Basically WPS substitutes the security of pass phrases with a PIN code. Due to the nature of WPS an attacker can guess this code very easily and obtain the configured pass phrases. The attacker only has to guess the correct code out of 11,000 codes which dramatically reduces the time for attacks.
If you have used WPS, please consider deactivating it. Pick a random string for your pass phrase (at least 16 characters, 63 is the maximum supported). We recommend 63 characters. You can create QR codes for entering this code when using mobile devices such as smart phones. Use the security setting WPA2 with a fixed pass phrase (PSK) and select AES encryption (sometimes abbreviated CCMP, most modern devices support AES, so TKIP is not needed any more). To sum everything up into a couple of steps.

Pick a random string for your pass phrase (16+ characters, 63 maximum)
Select WPA2.
Select AES/CCMP.
Enjoy.

While the options of configuring wireless security settings can be intimidating, please consider deactivating WPS and following the steps described. The alternative is to wait until vendors provide firmware updates for the wireless routers, but changing from WPS to a direct configuration can be done more quickly.

Ghosts Inside the Shell - Hardware Failures

Wed, 21 Dec 2011 00:38:00 CEST

The fortune cookie collections claims that hardware consists of the parts that can be kicked. This is especially true if something fails. Apart from the material there's often firmware involved which can also fail (algorithms are human, they have stress, too). We have two stories for you involving failed hardware.

Using redundant arrays of independent disks (RAIDs) sounds like a good idea. Have plenty of copies of your data and less worries. That's the idea on the surface. Below you'll find that mirroring data can also mirror deletion of data equally well. Then there are more complex RAID algorithms that use parity and checksums in order to deduce lost data from spare information. Complex is bad, and if only the firmware knows where your data is you probably won't in an emergency.
And then there is silent data corruption. A combination of faulty firmware and faulty hardware can destroy your file system(s) without warning. This happened to a logical volume spanning two RAID1 mirrors. There were no errors in the logs of either Linux kernel, RAID controller or server BIOS. Instead the Linux kernel got I/O errors when accessing the RAID1 containers, but no disk was marked as faulty and no RAID volume was marked degraded. Finally the JFS on the volume suffered a catastrophic failure and could not use its transaction log after a hard reboot of the locked server. A port mortem analysis of the file system and the hardware yielded no indication for the cause.

A different case was presented by a GNU/Linux router/firewall system. The hardware was an Mini-ITX board with three network interface cards, 1 GB RAM and crypto-acceleration in the CPU. The system worked flawless for over two years until the machine froze spontaneously during operation. The console stayed black, no input and no reset by keyboard was possible. The network interface cards were not reachable, too. Logs on the system showed no entries around the time of the freeze. Timestamps on the file system and files with 0 bytes indicated that the crypto-acceleration might have been in use at the time of the failure. After rebooting the firewall system selected Netfilter rules stopped working (about 3 out of 500+) including the NAT rules for SIP packets on port 5060/UDP. One NAT rule could be „repaired“ by switching the IP address of one server in the DMZ.
After switching the hardware and using the same set of rules on a different system all rules worked again, also in the form prior to changing the server address.

The hardware in question still needs to be examined in depth. Regardless of the results, you cannot trust any component of your infrastructure without regular maintenance.

The Perils of App Stores and App Markets

Tue, 29 Nov 2011 16:21:28 CEST

App stores and „markets“ for software are very popular these days. Most of the buyers and downloaders believe that the software in these stores is readily available and can be used at will. You just open the store application, browse the content and click/touch/smear to get your software. This is the theory. In practice the software offered can be very volatile and disappear without warning. Just take the VLC player or RedPhone/TextSecure as example.

The aquisition of Whispersystems by Twitter is the most recent case. Twitter acquired Whispersystems, the vendor of hardened Android software and secure communication tools. Immediately after the deal their apps were unavailable in the Android market. They even shut down the RedPhone servers, thus denying users around the globe secure end-to-end communication. TextSecure, a tool to use OTR-style encryption for text messages, continues to work, but unless you have backups you cannot install it any more. Why Twitter bought Whispersystems and why they took their software offline remains a mystery. The time of these events couldn't be worse. Whispersystems published the tools to help people communicate during the Arab Spring. Now they deny the same users the use of these tools during Egypt's election.

So beware of App Stores and Markets. Unless you get the source code, you don't own anything. Expect any software without disclosed source code to vanish at any given moment.

Trusted Computing is back to compromise your Systems

Sun, 30 Oct 2011 18:01:29 CEST

Do you recall the discussions about the Trusted Computing (TC) platform introduced several years ago? The idea was to introduce a trust relationship for code that gets executed on your own computing devices. Ross Anderson has explained the mechanism and its consequences in plain English on his personal web site in 2003:

…TC provides a computing platform on which you can't tamper with the application software, and where these applications can communicate securely with their authors and with each other. The original motivation was digital rights management (DRM): Disney will be able to sell you DVDs that will decrypt and run on a TC platform, but which you won't be able to copy. The music industry will be able to sell you music downloads that you won't be able to swap. They will be able to sell you CDs that you'll only be able to play three times, or only on your birthday. … TC will also make it much harder for you to run unlicensed software. In the first version of TC, pirate software could be detected and deleted remotely. … TC will protect application software registration mechanisms, so that unlicensed software will be locked out of the new ecology. Furthermore, TC apps will work better with other TC apps, so people will get less value from old non-TC apps (including pirate apps). Also, some TC apps may reject data from old apps whose serial numbers have been blacklisted. If Microsoft believes that your copy of Office is a pirate copy, and your local government moves to TC, then the documents you file with them may be unreadable. TC will also make it easier for people to rent software rather than buy it; and if you stop paying the rent, then not only does the software stop working but so may the files it created. So if you stop paying for upgrades to Media Player, you may lose access to all the songs you bought using it.…

The problems introduced by TC do not stop here. TC can help with remote censorship and hide malicious software from you. The latter is especially interesting since the discovery of state-sponsored malicious software found in Germany. There is no trust if you cannot control your own hardware. The architecture of the Trusted Computing platform merely takes control away from you and gives it to the hardware and software vendors. This is not a trusted computing platform and it opens up a whole set of questions. Given then fact that the third-party trust model has been broken by the security breaches of several certificate authorities (such as Comodo and DigiNotar).

So we strongly support making UEFI secure boot available to all users and not only to the consortium of the Trusted Computing Alliance.

Secure your Communication Lines

Tue, 30 Aug 2011 19:21:14 CEST

The events in Libya allow a rare glimpse behind the curtain of a government who uses digital surveillance technology against its citizens. It will take some time for the new government to assess the damage in terms of privacy violations, espionage and impact on the security of dissidents. The rebels found stashes of intercepted information ranging from instant messengers, videos, phone calls to e-mails. The eavesdropping was done by technology from companies in Western countries. This is a clear signal to businesses to secure their communication lines and to thoroughly scrutinise the promises of vendors.

Most of us take private conversations for granted. If you write an e-mail, a text message or make a phone call, then the gadgets you use give you the illusion of privacy. In reality next to none product and protocol takes measures to guard the information you are sending or receiving. Usually you have resort to extra effort, and in most cases you cannot reliably protect a communication line due to interoperability problems (maybe the end-point is still analogue or doesn't support certain protocols). You have to be aware of these issues and you have to define what communication lines can be used for which information. This is a very important step. Do not start with the technical issues. Start with an inventory of your communication habits and the data you usually transmit. Technical measures are always second. If you don't know how your internal processes use communication, then you can't do anything to improve their security.

Follow the events in Libya and learn about existing methods for compromising communication. The products are out there and they are not advertised publicly. Of course this doesn't mean that no one uses them. Keep an open mind and a tight grip on your company's digital assets.

Secure Communication for Businesses

Mon, 15 Aug 2011 15:57:11 CEST

Every now and then there is talk. If you run a business, you know what we are talking about. Every day we communicate. There are phone calls, text messages, e-mails, web portals, bulk data transfers, faxes, and more transmissions we have to deal with during a normal day at work. A part of this communication transports important information such as logins/passwords, offers, invoices, reports or personal data (any piece of information linked to a person). Most of the time our communication channels are up and running. This has not to be the case. The recent events in UK, the San Francisco underground, the regime in Syria (and Libya and Egypt), or the discussion about the Internet kill switch show that communication lines are always a prime target for attackers. The motivation of the attacker doesn't play a role if your business is in risk of being cut-off from networks.

You might want to spend some time preparing for blackouts or eavesdroppers before you encounter any one of these threats. Travellers are well aware that you can be disconnected faster than you anticipate. Virtual private network (VPN) links do not work in all countries. Mobile phone networks must not be trusted for sensitive information in general. GSM has been successfully attacked already (and will probably follow the path of early Wi-Fi networks in terms of security). GPRS has been attacked as well. The results were presented at the Chaos Communication Camp. This is no news to Internet veterans or members of NGOs threatened in countries abroad and domestically.
The lesson is always the same: You have to add extra layers of security. You have to consider using your own keys and what to do with these keys. Secure communication between and inside groups boils down to proper key management. This means you can start working on your security by making sure you can organise the switch to secure protocols. Simply stating "Let's encrypt!" won't get you far.

You can try secure communication in small steps to get used to the complexity involved. We use GPG, a cryptographically secured drop box and TextSecure for Android among other things. Drop us some ciphertext.

Do you care about your data?

Fri, 10 Jun 2011 11:27:35 CEST

Eran Feigenbaum, Google's security teleevangelist, has told the world that it doesn't matter where you data is. All that matters is securing your data, no matter where. In theory this is correct, but once your data is out of reach your choices for keeping track of your data and deploying security measures is limited. Don't get us wrong, this is not about Cloud Bashing. The ominous Cloud is part of IT infrastructure, and you can't whip up an elastic Cloud with high-performance computing out of your hat or basement. However Mr. Feigenbaum's view seems to be „clouded“ by marketing. One of his examples is the trace of an e-mail message that bounced through five countries. Certainly few people care about how their e-mails get transported, but some do, and most others don't e-mail databases with customer data around (some do though).

There are other problems connected with data locality and service providers. You have to tell your customers where their data is. If you can't tell, then you have to say so. If you rely on outsourcing, then you should have the guts to admit that maybe a crucial part of your business is out of your hands. If your only technical expertise is relying on a few lines of „SLA code“, then tell your customers; but please do not use smoke and mirrors and hide these little known facts.
During the past years the term „too big to fail“ has failed. The Cloud has lost data, „stable“ banking institutions have disappeared, earthquakes have created a triple nuclear meltdown, SecurID has issues, and many more similar events have happened. Frankly not caring where your data is, is the same as not caring at all.

We know where our data is. Customers hire us to make sure they know where their data is. Sometimes customers even approach us to retrieve data whose location is unknown. Make sure you build your information infrastructure on a solid foundation.

DropBox lied about security measures

Sat, 14 May 2011 12:43:06 CEST

Using the Cloud (whatever this word really means) for storage has become more and more convenient. Cloud storage is hailed as secure, fast, cheap, stable, efficient and more; it is truly the dream of every marketing department. "Hassle-free, batteries not included, your mileage may vary, …", but let's not forget: "There ain't no such thing as a free lunch"

The DropBox' cloud storage service is advertised with the slogan "Your stuff is safe". However the company reviewed its website claim about security. They changed the sentence "All files stored on Dropbox servers are encrypted (AES256) and are inaccessible without your account password." to "All files stored on Dropbox servers are encrypted (AES 256)." which carries a vastly different meaning. There's more. Another part of the claims were changed, stating clearly that the service provider has access to the user's data: …we have a small number of employees who must be able to access user data for the reasons stated in our privacy policy…
The article where the quotes came from has a detailed review of the security claims and the actual security provided. It's well worth reading before fully trusting this service - which is true for all services provided by third-parties.

Remember, your data is only secure if you have the encryption keys and if you control who has access to these keys. If you use the Cloud or any other service not under your control, you will most certainly not possess the keys, thus someone else will always have access to your data. Keep this in mind. Security needs to be designed and reviewed. Make sure that you plan ahead before distributing your data all around the world.

Cloud Volatility in Production Environments

Thu, 10 Mar 2011 12:45:13 CEST

You have probably heard of cloud computing already. The cloud can help you to save your own infrastructure by leasing from some other company, organisation or individual. It's basically the old Software As A Service (Saas) in new clothing with some added virtualisation and management technology. While we still use physical and virtual resources ourselves, some products of the Cloud are quite convenient. However you won't be safe if you don't put some planning into your infrastructure, regardless if it is your own or somebody else's. The following message was sent to us yesterday:

Dear customer,
unfortunately there was a problem with a defect RAID controller. The filesystem was destroyed and all data of the virtual servers were deleted. The RAID controller was replaced by a new one. We reinstall the hardwarenode at the moment, so that the customers can recreate their virtual servers and restore the data from their backup on the backup box.

We have no idea what the exact infrastructure of our recently deceased virtual server looks like, but we have a backup stored on our own infrastructure. Don't go around in Cloud Land and trust everybody. You have to roll-out and plan the use of rented infrastructure as well. True, you save some effort maintaining hardware, but that's about it. Be careful out there and get some advise from professionals before you put critical data at risk. Sometimes it's even worthwhile to build your own cloud. Let us know if you need help.

Apple exploits Free Software and denies it to its customers

Sun, 09 Jan 2011 21:09:05 CEST

The Apple Mac OS X system features Free Software, just as many UNIX-like operating systems do. There's no harm in that. It is good practice to reuse code that has been widely tested and is actively developed. So its fair to say that Apple benefits from Free Software. Strangely it doesn't allow its customers to have these benefits. Apple explicitly forbids Free Software in the Apple App Store’s Terms of Service (ToS). One of the first victims are users of the popular VLC media player. The VLC player has been pulled from the App Store because of its Free Software licensing.

The opinion of the VLC developer community on this matter is divided. In October a developer named Rémi Denis-Courmont contacted Apple regarding the GPL and Apple's redistribution of the code. Apparently Apple has no interest in honouring the license and simply decided to remove software in order not do deal with the licensing issues. Brett Smith, FSF Licensing Compliance Engineer, summarises: „Apple ‘only’ allows you to do the activities in the list of Usage Rules, if an activity does not appear in this list, you’re not allowed to do it at all.“

So far the Android platform has not presented an incompatibility with Free Software. Furthermore you are not required to pay a fee for writing code for Android. Our recommendation is to stay clear off Apple and focus on other platform that do not forbid the development of Free Software.