Theodore Tso: How the LSB Helps You Behind the Scenes

By James Turner
September 22, 2008

You may also download this file. Running time: 00:23:20

Subscribe to this podcast series via iTunes. Or, visit the O'Reilly Media area at iTunes to find other podcasts from O'Reilly.

James Turner: This is James Turner with O'Reilly Media; I'm talking today with Theodore Tso, who is the Chief Platform Strategist at the Linux Foundation. Thank you for taking some time to talk to us.

Theodore Tso: Glad to.

James Turner: So why don't you start off by talking a little bit about your background and how you came to be involved with the Linux Foundation?

Theodore Tso: So I got involved with Linux starting in 1991; I believe or at least Linus tells me that I'm the first North American Linux kernel developer. I started in late August of '91 with the Linux kernel version 0.09 and been involved with Linux for a very long time. In 1999, I started making Linux my full-time job when I started working for VA Linux Systems and in 2001 I started working for IBM and about nine months ago I started a two-year assignment with the Linux Foundation so I'm still employed by IBM but they're letting me spend two years helping out the Linux Foundation, largely with the Linux standards space, but I'm also doing other things with them in terms of you know helping with the Kernel Summit which is now being run out of the Linux Foundation and all sorts of other things, but those are probably the two biggies.

James Turner: Okay; why don't we talk a little bit about the Linux Standard Base? Why don't you first of all start by describing exactly what it is?

Theodore Tso: Well the LSB is what we call an ABI, an application binary interface. Unlike an API, an application programming interface, its goal is to provide instead of source level compatibility binary level compatibility, so that a single executable can be assured of working the same way on any LSB certified distribution. So we actually certify applications as well as certify distributions and the goal is that if the application obeys certain rules and the distribution passes on our test suite that the application should work on that distribution.

book coverbook coverbook cover
For a complete list of all things Linux,

James Turner: Why is the LSB important and who are the major people who benefit from it?

Theodore Tso: So the major people who benefit from it are ISVs, independent software vendors, who will be able to save costs by not having to release different product images for different Linux distributions. Back in the late '80s when we had the Unix wars it's been commonly accepted that one of the reasons why Microsoft was able to take over the desktop was that there was no standard ABI and so software vendors that wanted to release products for Unix had to do so on half a dozen to a dozen different splintered Unix variants. And that cost a huge amount of money or they could just simply release on Windows. And so one of the reasons why the LSB was formed was to avoid a repeat of that situation. So it helps the independent software vendors because it saves them costs; hopefully that promotes a stronger software ecosystem for Linux for those people who are interested in purchasing software for their Linux servers or their Linux clients and it also helps the Linux industry as a whole because it's one of the things that helps keep compatibility between various Linux distributions and therefore it sort of strengthens Linux in a way that Unix really didn't have that benefit in the late '80s.

James Turner: When you talk about binary compatibility, is that binary compatibility on a given point release of Linux, on a given major release of Linux; Obviously there were big differences for instance between 2.4 and 2.6 in terms of libraries. What are the standards that are applied in there?

Theodore Tso: So it's not as much about the Linux kernel as it is about the entire Linux platform. In fact, it's most of the problem arise less from kernel although they're have been situations where there have been compatibility issues due to kernel version but more because of the shared libraries that distributions choose to ship and so what we do is the LSB is a rolling standard and so we specify different versions of the LSB. The currently released version of the LSB is LSB 3.2 and we are currently working on LSB 4.0 which will be released at the end of the year. And the compatibility really comes from the libraries and the interfaces of those libraries that we specify. In practice, there are multiple different versions of a particular shared library, such as the GNU libc Library that might satisfy the constraints of the LSB and that's because shared library maintainers do recognize the importance of backwards compatibility and that's actually one of the things that we do try to do is to work with the upstream developers so they use our test suites and can find compatibility problems earlier that perhaps they had unintentionally introduced and find it before the distributions find them, and also just to encourage people to write their libraries in a way that promotes this type of compatibility primarily by adding new interfaces and not changing already established interfaces.

James Turner: There's more than just binary compatibility to the way Linux works these days. Some people like /opt; some people like /usr/local; some people stick everything in /usr/bin. Has there--is there any effort to try to get some standardization around things like directory paths and other kinds of non-binary issues with Linux?

Theodore Tso: Part of the Linux standards base is the file system hierarchy standard which strongly encourages independent software vendors to install their binary in /opt and so that's what we do encourage. In practice, where data files live very often is a matter of the choice of the individual system administrator and so in my experience many of the application vendors will allow that to be something that is configurable via some type of configuration parameter maybe a config-file or environment variables, but if you take a look at Oracle for example, Oracle will allow the Oracle table space files to be located you know in the location of the system administrator's choosing. But in terms of where the binaries themselves are installed we strongly encourage /opt.

James Turner: Another obvious area where there isn't a lot of compatibility these days although there are less of them there used to be is in package management and it's probably one of the biggest roadblocks to having one software package that can install on any distribution. Again is that something that you'd like to see more standardization of?

Theodore Tso: I would--I think you're right; it's a very, very hard problem. We actually tried to address that problem in two different ways; one is in the LSB we provide a very limited sub-set of the RPM package format which we know is compatible across all of the distributions and also compatible to Debian and Ubuntu via a program called Alien that will actually convert RPMs into Debian packages and then install them. And then the other thing that we do is the LSB uses a single dependency on the LSB, so basically there's a meta-package LSB that guarantees all of the libraries that are specified by the LSB and that tends to work very well because it just sort of bypasses the issue of different distributions using different package names to name the various libraries that an RPM might depend upon. The other thing that we do see though is that some ISVs want to use their own delivery system, primarily so they can be portable across other platforms. So if you have a large software vendor whether it's Oracle or IBM Software Group, BEA, and so on they very often will use the same product documentation across multiple different Unix and Linux systems and so they want the installing experience for their customers to be the same regardless of platform. And so they've actually packaged their own installer media. One of the things that we've been working with them on is something called the Berlin API which is an optional specification that might allow distributions to get information from an ISV provided installer so that they can at least have an uninstall mechanism which is better integrated with the distribution. That's something that isn't going to be in LSB 4.0 but it's something we've been playing with. The reality is today for those systems what in practice happens is the ISV has been providing their own installer and it installs in /opt or whatever and there isn't currently a clean un-install mechanism, and that's one of the things that we've been trying to work with ISVs to try to provide some standardized way of you know making that kind of functionality available to Linux users.

James Turner: When I was looking at the LSB on the website, I noticed a mention of things like MySQL; how are those fitting into the LSB, you know that's a database product?

Theodore Tso: So MySQL has been working with us, and we've been using it mainly as a proof point to demonstrate that an application as complex as a MySQL database could in fact be packaged as an LSB application. It's useful to MySQL because they can just simply package if they were to choose to do so a single binary that would work across multiple Linux distributions. Now in practice, MySQL being Open Source, many of the distributions are of course already including MySQL in the distribution but so the only main reason why MySQL might want to do this is if they want to provide one standardized release level for their commercial support offering. But that's one place where, once they decide that that's the path they want to go down with the customer where the customer wants to get commercial support from MySQL, if they were to use the LSB to ship one product image for all distributions that are LSB certified it saves MySQL costs because they don't have to build one version of My SQL for Red Hat, another version of My SQL for SUSE, yet another for Ubuntu and so on.

James Turner: What do you consider some of the greater successes that the LSB has had to date?

Theodore Tso: So there's a certain amount of work that we've been doing behind the scenes that isn't necessarily visible to the rest of the world. But the fact that the distributions are largely compatible with one another is actually one of the side effects of the LSB. There have been times in the past where for example, the distributions were about to release different versions of g++ that turned out not to be backwards compatible or not compatible with each other and we detected that before the distributions had actually finalized their release plans and we were able to go to both of them and say look, the g++ people made an unintentional backwards compatibility problem in one of the standard c++ libraries, libstdc++ such that even though these two release versions were supposed to be compatible they're not and through some gentle negotiation with both distros one of the distros actually agreed to change what they shipped to avoid that particular backwards compatibility problem. And so there's a certain amount of the work that we do even if we don't have applications standardizing on LSB, we have a few applications that have actually certified to the LSB but to be honest, not as many as I would have liked where we are right now say three to five years ago. I actually thought we would have been further ahead in terms of ISV adoption and we haven't been. But nevertheless, there's been quite a bit of work that we've done in terms of using the test suites to detect problems upfront, actually find bugs in the various libraries and get them fixed, and keeping the LSB platform closer together.

James Turner: Conversely what would you think are the most vexing problems that still face you guys?

Theodore Tso: So one of the most vexing problems has been on the desktop where the Open Source community has been developing new desktop libraries faster than we can standardize them. And also ISVs want to use those latest desktop libraries even though they may not be stable yet and in some ways that's sort of us being a victim of our own success. The LSB desktop has been getting better and better and despite all the jokes that for every year since I don't know probably five years ago, every year has been promoted as the year of the Linux desktop. The fact of the matter is the Linux desktop has been making gains very, very quickly but sometimes as a result of that some of the bleeding edge interfaces for the Linux desktop haven't been as stable as say the C library. And so it's been challenging for ISVs because they want to actually ship products that will work across a wide range of Linux distributions and this is one of the places where the Linux upstream sources haven't stabilized themselves. And one of the issues with the LSB is if the upstream developers are still making changes there's nothing we can do on the LSB to really help the ISVs. And this is a problem which I think the desktop community is definitely coming to understand and they're trying to think ahead about you to make their interfaces more forward compatible so that as they add new features they don't need to go back and change pre-existing interfaces, but that's been an ongoing challenge.

James Turner: How do you decide if a given library or binary should be included in the LSB or not as new things come along?

Theodore Tso: So there are a couple of ways--it's a fairly complicated process; one of the first things that we do is we base it on requests from the software vendors--application programmers that say we really want to have some kind of sound interface in the LSB. In addition, we have a tool that used to be called the Application Test Kit and we've since re-branded it to be a little bit less geeky; it's now called the Linux Application Checker, which can be used to scan an application and determine number one, whether or not it's using interfaces that are outside of the current versions of the LSB and even if it is we'll actually analyze the libraries that it uses and give a warning to the programmer about how generically portable that application might be across different distributions. So even if their application isn't LSB compliant because it's using say some new desktop library that's not yet in the LSB, the Application Checker can actually tell them "Gee, you're using you know QT4 and so you'll work on these distributions but you won't work on Rel4 for example because it doesn't ship some library that you need but it will work on Rel5." Now of course that's not a guarantee that it will work but it's a whether or not the shared libraries and the interfaces that application needs would be satisfied by different distribution. One of the other things that can be done with the Linux Application Checker is if they click on the upload button the list of all of the interfaces they use can get uploaded to one of our servers and we can use that statistical information of how many applications are using which libraries and which interfaces of those libraries to help us determine what are the next libraries that we should add to the LSB.

James Turner: How does the LSB work when you're looking at environments like Mobile or Small Form Factor devices? Do you have special sub-sets of the standard when you're looking at them?

Theodore Tso: We are starting to work with some of the various embedded players for adapting the LSB to those environments. In fact, in those environments the model the LSB actually works very, very well because in these embedded environments very typically there is a very restricted set of libraries that will be present. And so this type of testing is especially important. Moblin, the Intel initiative, has actually been using our tools as part of their test procedure and we've--and that's fine; all of our tests, all of our tools are Open Source, so they're perfectly free to do that. And we're working with them to you know potentially help them with their certification efforts. Since they're using our tools we understand them very well and we think we can add value to Moblin but in fact there are some of these mobile platforms that are looking very, very hard or are already using our tools as part of their platform definitions.

James Turner: Theodore Tso, Chief Platform Strategist at the Linux Foundation; thank you very much for taking the time to speak to us and good luck with standardizing Linux.

Theodore Tso: Thanks.

You might also be interested in:

News Topics

Recommended for You

Got a Question?