Securing Linux at a University Through Securing a Linux Distribution: Technical and Social Aspects Jon Lasser University of Maryland, Baltimore County Abstract: This paper presents a case study of a university attempting to reduce Linux-related security incidents through customizing a localized, security-aware Linux distribution based on Red Hat Linux. After describing the university environment and the parameters of the UMBC Linux project, an overview of technical and social aspects of the project is given. Following this, extensive analysis dissects the UMBC Linux 5.2 project's successes and failures, especially the failure to harness the power of the Linux community to improve the project. Finally, future development plans, including the Bastille Linux project, are briefly described. Defining The 'Linux Problem' at UMBC By the spring of 1998, it was clear to the University Computing Services (UCS) staff of the University of Maryland, Baltimore County (UMBC) that Linux was spreading like wildfire across the campus. There was an active Linux Users Group, whose newsgroup and mailing list had replaced the umbc.unix newsgroup as the focus of public computing discussion; the physics department, the math department, and the computer science department all were running their own Linux boxes without any support or control from the UCS systems group. Security was nonexistent; break-ins were frequent, placing other university computers at risk. Something had to be done. UMBC, according to an NSF report, produces more IT graduates than any other research university in the country. There are several thousand computers on campus, including approximately 400 Unix workstations and servers. Machines run by UCS for general campus use run Kerberos 5 and depend heavily upon the Andrew File System (AFS), a distributed filesystem. Most of the machines in external departments (CS, Physics, Chemistry, etc.) are still being transitioned and are not yet dependent upon these services. We have more than 10,000 students on campus, and approximately 750 professors. Prior to the summer of 1998, the two officially supported Unix flavors on campus were Irix (5.3 and 6.[2-5]) and Solaris (2.[56]). It was clear that demand for Linux support was on the horizon. Computer Science students were demanding improved access to workstations for the development of their class projects. One of the 1998 summer plans was to acquire several labs' worth of new PCs on which to dual-boot Windows NT (for general use) and Linux. Additionally, the multiple-processor Irix servers which served shell accounts for students and faculty, primarily for e-mail, news, web page design and hosting, and occasionally for compilation of programs were heavily overloaded. UCS was interested in replacing or supplementing these machines with a large number of significantly less-expensive Intel-based PCs; Linux was the obvious choice, but we would have to integrate it into our campus-wide services. UMBC's 'Linux Problem' clearly broke along several lines: there were places we wanted to run Linux internally, such as our labs and servers, and there were places on campus external to us who were already running Linux or were interested in doing so. These external Linux users included both academic departments on campus and students living in the recently-networked dormitory rooms with high-speed Ethernet connections. Finally, the 'Linux Problem' broke along both technical and social lines too: not only did we have to solve our security concerns from a technical perspective, but we needed to convince our users to solve their security problems, even when we had no direct control over their systems. Another description of the 'Linux Problem' is that it was hard to do the Right Thing: Red Hat Linux is easy to install but difficult to administer in a secure fashion. We needed to make it as easy to lock down a Red Hat machine as it is to run one which is open to the world. UMBC Linux: The Quest For Carrots Generally, there are two categories of persuasion: carrots and sticks. Since university policy makes it exceedingly difficult to shut down machines run by academic departments, which takes away our main stick, we needed carrots. The obvious carrot, of course, is to make it easy to do the Right Thing: a Linux distribution which installs locked-down and customized for our campus environment would solve most of our security problems, it seemed. After all, the administrators of Linux boxes we are most concerned with are less experienced, unlikely to patch or update software, and are least likely to change defaults. Secure defaults provide at least a baseline level of security for inexperienced administrators uncomfortable with or unsure of how to configure the system. A second carrot we're able to offer is Linux support: we will agree to support departments running Linux only if they run our customized distribution. If people wish to reap the advantages of working with Linux without being forced to administer their machines, this policy provides a way for us to do so with a minimum of effort. To avoid compatibility concerns, we will agree to support hardware only if it is listed as Tier I or Tier II supported on Red Hat's hardware compatibility list. Unfortunately, at least until recently, this incentive has not been very successful. While some departments have switched to our Linux distribution, others have not done so. Their reasons range from a near-religious commitment to a specific Linux distribution on the part of faculty or staff to an 'If it ain't broke, don't fix it' attitude, with a good dose of 'but we can administer our own machines just fine, thank you' attitude which ignores repeated security breaches on those selfsame computers. Recently, this has started to change due to some personnel turnover in those external departments' support staff and the novelty of Unix systems administration wearing off for other such support people. Alas, a concern for systems security has not ranked as a reason to switch to UMBC's Linux distribution. The most successful incentive we've found to convince both individuals and departments to run our Linux distribution has been one of Linux's main advantages all along: accessibility. Our distribution is easy for staff, faculty, and students to acquire. In addition to being available over the campus network, any member of the UMBC community can stop by the UCS office and trade her or his blank CD-R for one which contains our distribution. Despite our campus connection to the high-speed VBNS (Internet II) backbone, most users seem to prefer to install via CD-ROM, or to at least have a copy of the media available for the future. By making our distribution available at virtually no cost, UMBC's Linux distribution has in fact become quite popular on campus. How We Built UMBC Linux Because one of the main targets for our distribution is beginning users, it made sense to base our distribution upon one which is easy to install without prior Unix experience. An open-source installation program was mandatory because we needed the freedom to modify the installer and distribute our modified installation program. Simplified administration tools would also be a plus for much of our target market, though they would almost certainly alienate some experienced users. (Of course, we are less concerned with the security of those self-styled expert users, who in any case could simply refuse to use the included administration tools.) For these reasons, and because of its ubiquity, we chose to base UMBC Linux on Red Hat Linux. Our first revision of UMBC Linux had a goal which was even more pressing than security: the need to 'ship' simply to prove that we could. We needed to integrate Kerberos and AFS into Linux before the beginning of the Fall 1998 semester and install it into one lab as proof-of-concept. At the time work on this distribution began, Red Hat Linux version 5.1 had been available for some time, and was quite stable with up-to-date packages. Our first release was simply Red Hat 5.1 with updated packages, Kerberos 5, the unofficial Linux port of AFS 3.3, and the addition of the K Desktop Environment (KDE) as the default X session, to ease the transition of Windows users to Linux. Used internally, it was also installed in one public lab for testing. This was announced to the Linux Users Group via their mailing list and newsgroup, in hopes that they would find and report bugs to us. To notable problems arose during this initial development phase: we needed to update a CD of the software in order to install lit, and we needed to install it in the lab in a uniform fashion. After significant experimentation on this latter point, it was decreed that, as the 'Ghost' program used to mirror the NT side of the lab could also handle, after a fashion, Linux's ext2 filesystem, we would simply use Ghost as well. An interesting problem developed for us as a result of this decision: because Ghost does not really understand ext2 filesystems, and in actuality only performs a sector-by-sector copy, our "lab install" partition on our development machine had to be identical in size and location to the lab machines themselves. While simple the first time, we were frustrated with the near-constant repartitioning of our one Linux development machine's hard drive as the Linux portion of the drive shrank and the NT partitions ballooned. We would simply dump the contents of the partitions over the network and restore after each repartitioning. Our other problem, that of updating a CD and the install program to deal with our changes, was simply a matter of documentation: at that time, there was little information on how to modify Red Hat's installer or how to burn bootable CD-ROMs. E-mail to a member of Red Hat's development team was met with a clear description of how to rebuild the catalog used by the installer via the `genhdlist` program included on the CD, as well as where other information used by the installer is stored. As for information regarding burning CD-ROMs, the mkisofs documentation eventually yielded the secrets of this method. Currently, all relevant documentation on these matters has been consolidated by $NAME in the $HOWTO-NAME, available at $HOWTO-LOCATION. Development continued on a 'real' release, with more substantial changes, during the Fall 1998 semester. Our target date for release was towards the beginning of winter break, so that all labs on campus with new PCs could have Linux installed by the beginning of the Spring 1999 semester. During this time we moved to Red Hat Linux version 5.2, which we had found to be primarily version 5.1 with stability fixes and a handful of additional software. We also integrated a number of packages from Red Hat's 'Powertools' collection, which consists of programs excluded from the main Linux distribution due to their less-than-polished state, non-Open Source license, limited usefulness, time pressure in the development process, or the whims of the developers. From this package, we included Mesa and Lesstif, to more closely simulate the SGI environment to which most CS students were accustomed; CDRecord and mkisofs, to ease our own development process; XEmacs, which is quite popular on UMBC's campus; and several games, as further incentive to run our distribution rather than stock Red Hat; and a handful of useful miscellaneous applications. Several packages which contained in-house scripts were added to our distribution. These scripts included a way to reboot the lab machines from the login prompt and a script intended to properly configure the Sound Blaster AWE64 sound cards in the lab computers. Several changes were made to improve the security of our distribution: we included the latest version of SSH 1, to encourage secured logins between computers; we altered the default configuration of the mtools package, so that any user could read or write from the floppy and Zip drives without being able to mount their disks; we changed permissions of several files and devices; and we changed several defaults in inittab and inetd.conf for security reasons. Finally, we included cfengine, a package which automates arbitrary configuration changes to the system to be performed automatically based on a configuration file. By placing a cfengine configuration file in AFS space and calling the program on boot, we gave ourselves the ability to make changes to the system with minimal exertion in a relatively unobtrusive fashion. We met our release date for UMBC Linux 5.2, codenamed Mercury. (We chose to key our version numbers to Red Hat's version numbers.) We got all of the labs installed before the spring semester began; and we were handing out CDs to the students, staff, and faculty as soon as they returned to school. We even made available a dual-Pentium II server with half a gigabyte of RAM for general login use to supplement our multi-processor SGI servers, on a public beta test basis. This server is expected to become an officially supported machine before the beginning of the Fall 1999 semester. What We Got Right We did in fact accomplish quite a bit on a tight schedule. We managed to gain credibility in the campus Linux community through close contact with its members, and we made Linux available on a wide scale in a somewhat more secure fashion than previously available to beginning users. We gave ourselves a way to fix mistakes as they are reported to us, which we've used in the lab situation on a fair number of occasions. We created mailing lists to report new security patches, and we announced all of our Linux support to the community through various newsgroups and UMBC's newsletters. Our distribution was quite well documented: an extensive README included on the CD provided a complete high-level overview of the differences between Red Hat Linux 5.2 and our distribution. This was helped by extensive notes made during the development phase of the project and has helped experienced users evaluate whether a switch to our distribution made sense. Linux itself has gained credibility at UMBC, our beta-test Linux general login server will be supplemented with either a larger machine or several other machines of equal size, and our project continues. All this was accomplished by one employee dedicating most of his time to the project, one other employee working part-time on the project in its later phases, and with the support of our local AFS and Kerberos guru and our management. Our only expenses were the purchase of a CD-R drive, accompanying SCSI card, and the dedicated use of a single PC identical to the lab machines. No known security breaches have occurred on any machine running UMBC Linux. That is certainly not to say that breaches have not occurred, but that a certain class of opportunistic break-ins by 'script kiddies' seem not to be effective or perhaps attempted, for the most part. We have not attempted to ascertain the underlying causes for this apparent success, but it nonetheless represents a situation fundamentally different from our initial status. What We Got Wrong Unfortunately, we got quite a bit wrong in the process as well: while we did integrate Kerberos and AFS into our distribution, we failed to do so either well or consistently; we failed to use our own resources effectively at times; our testing period was virtually nonexistent; and most importantly we failed to work with the larger Linux community, a problem which is currently being addressed. Red Hat version 4.2 and later use Pluggable Authentication Modules (hereafter referred to by the rather redundant term 'PAM modules') to authenticate logins and passwords throughout the distribution. For example, the xlock screen saver included with Red Hat Linux has been compiled (or modified to compile with) PAM support. Of course, the login program has been PAM-ified, as has FTP and other essential services. Unfortunately, by the release of UMBC Linux 5.2 we had been unable to locate PAM modules which worked with Kerberos 5 and AFS. In fact, we had found modules to do this on the web, but we were unable to build the modules. Furthermore, PAM documentation for system administrators was rather lacking, and we simply did not have the time to investigate further. Instead, we simply built alternative login and XDM packages for our distribution and included detailed instructions in our README file for manually installing AFS and Kerberos enabled login packages. Regrettably, these packages were rather dangerous: either package failed to permit logins if the network was misconfigured, and therefore neither package could be safely installed at boot time. We also updated the xlock package for Kerberos and AFS support. Worse yet, our customized login and xdm packages required modifying the Red Hat Linux packages which contained these programs -- and many others: login was part of the larger util-linux package, and xdm was part of the XFree86 package. We needed to split off a subpackage which contained login and xdm each in their own package. As a result of this change, we are forced to modify any updated XFree86 or util-linux packages provided by Red Hat Software even without any other reason to do so. This was a major failure in several respects: it was difficult and dangerous to kerberize logins and provide AFS tokens. Furthermore, we failed to update most packages which used authentication: FTP, xscreensaver, vlock, KDE's screen saver, and other packages were broken on our distribution as a result of this oversight on our part. In retrospect, it seems obvious that we could not anticipate password verification in all programs. Without effective PAM support, however, no other option appeared to present itself. The best we could do was, when one of these programs was reported as 'broken' in the labs, set cfengine to disable it upon the next boot of each workstation. Furthermore, bugs still exist in our support for AFS and Kerberos: one bug, whose cause has not yet been correctly identified, prevents more than one user from using Pine at any given time. On workstations, this is not significant, but it has proved a major obstacle with moving our login server into production. This bug is still present, even with the official Transarc release of AFS 3.5 for Linux. We are not certain if the bug lies in the AFS code, Pine, or our own modifications to Pine to provide AFS support. Ironically, another major difficulty with our development and administration process has been our failure to use our own resources, such as cfengine, in an appropriate fashion. Our cfengine configuration tree for our Linux labs was separate from our campus-wide setup for Irix and Solaris, rather than simply a set of special cases within the larger tree. As a result of this, as well as our relative inexperience with cfengine, one change introduced a bug which kept cfengine from processing any of our configuration files. This configuration error existed for weeks before it was noticed. Many security fixes available from Red Hat have not been installed, simply because we've found it difficult and time-consuming to put these changes in cfengine and to test them. As a result, our machines are not updated often enough to be truly secure. Similarly, we have not made effective use of the mailing lists created for the purpose of disseminating security notices and notices of upgraded packages. Initially we intended to simply forward Red Hat's notices to the list, when applicable, but we have not followed up on even this step. Our failure to have sufficient time to test our distribution before release can be attributed to several factors: a dearth of machines upon which we could perform testing, a lack of personnel capable of testing our distribution in any substantial fashion, a lack of any plan for formalized testing, and an utter lack of any leeway in our development cycle. Hopefully as Linux matures, fewer releases in general will be necessary, and we can alternate between development and maintainance in a more leisurely fashion. On the one hand, our failure to test our distribution extensively did not result in an incredible number of bugs. Partly, this was because we were building upon Red Hat Linux, version 5.2, which is quite well tested and which had been substantially patched by the time we released our distribution. Partly it is a testament to the abilities of UCS staffmembers. Partly it was just blind luck. What has resulted from a lack of testing, and a more general shortage of time for the project, is a fair number of obvious fixes which could have tightened up security much more than we managed to do. For example, our TCP Wrappers configuration was not locked down, as we had originally intended, and many other security enhancements might have been suggested during a testing phase, had one been provided for. Our most substantial failure, however, was not technical but social: despite the ever-growing use of Linux at universities and in commercial environments, we did not seek to interact with others performing similar work. Since our first significant development cycle ended we have come into contact with many other universities and institutions who have done the same work we did. Certainly, had we shared our plans and results with other schools, we would have duplicated less work and been able to accomplish more. One striking example of this is a site at MIT, $MIT_SITE, which has built AFS and Kerberos packages for Red Hat Linux versions 5.2 and 6.0, as well as many of the support packages and PAM modules. Had we been aware of this site earlier, we might have saved ourselves quite a bit of effort. Into the Future: Beyond UMBC Linux 5.2 At the SANS '99 conference, in May 1999 in Baltimore, we found the large number of educational institutions experiencing the same problems we had. During the Universities birds-of-a-feather session and a Secure Linux BoF, we realized how much of this work had been done before and how much could be accomplished if we worked together. SANS has signed on as a sponsor of our work, and VA Linux has agreed to host our website and mailing lists. UMBC Linux is no more; Bastille Linux, a cooperative effort between universities, private companies, and other interested parties, seeks to close the security gap. Our motto: "It wasn't the building, it was the administration!" While the Bastille Linux project expects to produce only minor gains for Linux security as a whole, we hope to improve security at large sites where users administer many machines through improving default settings and discouraging use of obsolete, dangerous technology such as the rsh suite of tools. Because of our focus on large sites, we also intend to improve the administrator's ability to produce site-specific distributions customized for local environments. UMBC expects to use the work we and others are contributing to Bastille Linux in our new distribution of UMBC Linux, version 6.0. While it will certainly differ in some respects from Bastille Linux 1.0, we will nevertheless have almost certainly saved a large amount of duplicated effort. Our future plans, as distinct from those of Bastille Linux, primarily involve integrating our cfengine tree with the rest of our cfengine configuration for campus, and integrating the application-building process for our /usr/local with that of our other platforms.