Releasing Free/Libre/Open Source Software (FLOSS) for Source Installation

David A. Wheeler

2009-04-13 (revised 2013-08-23)

If you’ve written (or started to write) some Free/Libre/Open Source Software (FLOSS), please follow the time-tested community standards for releasing FLOSS software when you want people to be able to install it from source code. This article will briefly explain why you should follow standard release conventions, and what they are.

Why Follow Standard Release Conventions?

First, the why. The quick answer is, “because it makes it easy to install your software - including creating packages to do so”. Modern Linux and BSD systems (as well as MacOS and MS Windows) normally install programs via “packages”; on Linux and BSD systems, this automatically installs all the necessary dependencies too. These packages are much easier to create and maintain if you follow standard release practices.

I’ve written a number of FLOSS programs (such as flawfinder and SLOCCount), and participated in other development, but I didn’t really understand the importance of the release standards until I started to create packages as part of my work on open proofs. I quickly learned that the packaging concept was simple, but some programs were hideously painful to package because they hadn’t followed the usual release standards. I knew some of these practices before, but I sometimes skimped on them before I understood the nightmare I was (unintentionally) inflicting on others.

I intend to modify my future releases to better meet usual release practices, now that I’ve “learned better”... and I hope to spread the word so that others will do the same.

What are the Standard Release Conventions?

Now, the what. These documents in particular describe good release practices:

  1. The GNU Coding Standards, especially the material on the release process. The makefile conventions section of the GNU make manual is illuminating about these.
  2. Software Release Practice HOWTO by Eric Steven Raymond.
  3. Filesystem Hierarchy Standard (FHS) — this describes where files are supposed to go.

If you’re releasing FLOSS, you want it to be easy to package, so it might be a very good idea to know about typical packaging guidelines. You can look at:

At least look over the first set (the GNU coding standards’ release process, Software Release Practice HOWTO, and the Filesystem Hierarchy Standard (FHS)).

A Quick Checklist

I know that the above is a lot information, so here is a quick checklist of some of the most important guidelines (in my opinion), based on problems that I’ve had:

  1. Pick a good, simple, and unique Google-able name for the package and its executables. Limit your package name to the lower case letters a-z, digits 0-9, and the four punctuation characters “-._+”. The Software Release Practice HOWTO’s chapter on good project and archive naming practice and the Fedora naming guidelines may help. Make sure the uppercase/lowercase distinction is irrelevant; some filesystems can’t distinguish them, and package names tend to be all lowercase anyway. Try to pick a name that’s unique — at least unique among packages, and preferably, one that doesn’t have any Google hits at all (so that people can find your package easily). A good example of a bad name is Why, because it is completely un-Googleable. If you don’t pick a good name, then in the best possible case, people will rename your program, often leading to confusion. At worst, they will never find your program at all. Similarly, try to make to make the name of the executable(s) unique; they will probably be installed in /usr/bin with all other user programs.
  2. Identify the version in MAJOR.MINOR.SUBMINOR format or by using dates in YYYYMMDD format. I especially recommend using the MAJOR.MINOR.SUBMINOR form defined as semantic versioning (with the added restriction: don't add "-..." at the end of a version name, because the rpm tool used by many Linux distributions does not allow hyphens in version names). People and software must be able to easily determine if a version is older or newer than another one. Using version numbers (e.g., 2.2.2) or date values in YYYYMMDD format (e.g., 20090413) makes that clear. The filenames of major releases should include the version number, e.g., “myprogram-1.2.7.tar.gz”, so that even after downloading there are no questions. Please use a version control system (such as git), but please make occasional “real” releases that are single-file archives; that way, people can easily know what version is ready to use, and how to get it.
  3. Use a standard, widely-used, GPL-compatible FLOSS license — and say so. One of the worst mistakes a FLOSS project can make is to use a non-standard license (especially a home-grown one) or fail to state a license at all. “Home-grown” licenses are often not even FLOSS, impede reuse of other code, can delay packaging or use by years, and will dissuade potential co-developers. Many organizations and distributions explicitly forbid using software that fails to include a license statement, because they create a legal hazard. As a result, failing to state a license makes the software useless to many. Use one of the standard FLOSS licenses (e.g., MIT, BSD-new, LGPL, or GPL), and be sure your license is GPL-compatible. Fedora’s licensing page has more info. You do not need to use the GPL itself, but using a GPL-incompatible license is a mistake, in part because most FLOSS is GPL’ed. Do not write your own FLOSS license. Clearly state in your documents - and code header files - what the license is.
  4. Follow good distribution-making practice, in particular, make sure tarballs always unpack into a single new subdirectory (of the current directory) named NAME-VERSION. The Software Release Practice HOWTO’s “Good distribution-making practice” goes into this. Make sure that tarballs unpack into a subdirectory, or unwary users will end up with unwanted junk in directories they use. By doing this, you’ll fulfill people’s expectation that the first two steps to do are (1) unpack (untar) the file, and (2) “cd” into its directory.
  5. Support the standard invocation to configure, build, and install it: ./configure --prefix=/usr ; make; make install. Many people expect that, after the unpack-and-cd steps, you’ll perform the normal configure, make, make install process. Please implement configuring, building, and installing this way, because this presumption is widespread (e.g., here, here, here, and here). You do not need to use the GNU autotools (autoconf, automake, and libtool) to implement this - just conform to the usual external interface. See the GNU Coding Standards’ “Managing the Release Process” for more. That said, you might consider using autotools, since it automates this; see the notes below on autotools. Don’t interleave the configuring, building, and installation steps; in particular, the build (“make”) step should create all the files to be installed, and the install step should do nothing but install the files created by the build step. The first two steps should not require root privileges; the latter one may require privileges depending on where files are being installed. The next three bullets discuss the assumptions of each of these three steps.
  6. Support the standard ./configure options like --prefix, --exec-prefix, --bindir, --libdir, and so on. By supporting the standard “configure” options you make it easy to install the program wherever the user wants it. Supporting “prefix” is especially important; “prefix” is prepended to all filenames to be installed, and should default to “/usr/local” (for a system-wide installation that’s not part of the system). When the program is packaged for a given system, the prefix is typically changed to “/usr” instead. You should support installation by a single unprivileged user, in which case the prefix would typically be “$HOME”. There is a reason that people need control over these values, for example, never assume that “libdir” is the same as “$(prefix)/lib” — on many 64-bit systems it is “$(prefix)/lib64”. Here’s how configuration should work. It’s nice to support --srcdir, but I wouldn’t worry about that; --srcdir can be awkward to support, and is far less important (storage space is plentiful nowadays, so you can just make a copy and then build). But options like --prefix and --libdir are vital.
  7. Create a makefile that can rebuild everything and uses makefile variables (including applicable standard standard makefile variable names and targets). By default, invoking “make” should rebuild/recompile everything (by tradition this target is named “all”). Makefiles should be able to rebuild everything; many distributions forbid using pre-built binaries or libraries in packages (since they can’t then fix problems). It’s okay to include pre-built binaries/libraries in a source distribution, as long as “make clean” (or “make maintainer-clean” or “make really-clean”) can get rid of them.

    Makefiles should maximally use makefile variables so they can be easily overridden where necessary, e.g., “make CC=mygcc”. GNU publishes useful makefile conventions. Overriding “make” values is very portable (it’s defined by the POSIX standard), with this precedence: (1) variables set on the “make” command line, (2) MAKEFLAGS, (3) variables set in the Makefile unless “-e” is set, (4) environment values, (5) default Makefile rules. in particular, variables set on the make line or via MAKEFLAGS take precedence over the rest (make will use the environment values if their values are not set in a makefile; “-e” makes the environment values take precedence).

    Use standard makefile variable names where they apply (e.g., prefix, exec_prefix, bindir, libdir, etc.).

    You don’t need to create a makefile directly; many programs use makefile generators, such as the one in Cmake and autoconf/automake. Just make sure that the generated makefile(s) support these conventions. It’s nice if you can create a portable makefile by sticking with just the capabilities in the POSIX standard, but the 2008 POSIX specification for “make” lacks so many capabilities that many people find it unacceptable (others have come to the same conclusion, e.g., Mozilla LLVM Mad Scientist). If you will not create a portable makefile, then use GNU make and its extensions; GNU make is open source software, free (in both senses), available everywhere, actively maintained, and full of useful capabilities. (Hopefully someday the POSIX standard for make will get fixed; I have submitted some comments to try to get things better.)

    Note: be sure to mark phony targets (like “all”, “clean”, and “install”) with the .PHONY mark, like this: 

         .PHONY : clean
    This is particularly important for “install”; without it, “make install” will probably do nothing on filesystems that ignore case distinctions (because the make will see that the “install” file already exists). In general, marking phony targets (targets that aren’t really files) will eliminate mysterious errors and document more clearly what is what.

    Even if you use a completely different build system that does not use make, at least include a tiny makefile that invokes the “real” make system. There’s no reason you have to use make directly if you don’t want to. For example, today’s FLOSS Java applications tend to use Ant or Maven. Cmake is a very useful cross-platform build system, with a lot of users (Cmake generates makefiles, so it automatically meets the requirement to provide a makefile). However, make sure you include a small makefile that can invoke your build and installation system. That way, people don’t need to figure out how to invoke your make system, and programs can be automatically built and installed regardless of your preferred build system.

  8. “make install” should not rebuild anything; simply copy files to locations as specified by DESTDIR, prefix, and so on. Most packaging systems, including Ubuntu, Debian, and Fedora’s, expect programs to be completely built by the steps above, and then split “installation” into two steps: An “install” to an intermediate directory (where the files will then be collected), and a final “install” done on the user’s actual system. The program will not be given root privileges in the first step, and in the second step it will not normally run commands at all, it will just copy files into predetermined positions. This is easily done if your installation program is invoked using “make install” and supports a few standard conventions (such as DESTDIR). The install rule should “not modify anything in the directory where the program was built, provided ‘make all’ has just been done... The commands should create all the directories in which files are to be installed, if they don’t already exist.” In most cases using “cat” in “make install” is a mistake; “make install” should simply make directories and copy already-built files into the appropriate directories, so think carefully before adding any command other than mkdir, install, cp, and ln. Please support the DESTDIR convention, that is, they should always install into the directory $(DESTDIR)$(prefix) where DESTDIR can be set by the user. This implies that you should always create directories before installing into them; $(DESTDIR)/usr/bin might not exist! Packaging can be very painful if you don’t support DESTDIR. Automating DESTDIR can be a pain, so it’s best if the program supports it to start with. If your package doesn’t support DESTDIR, at least make sure that the “make install” target only uses a few simple commands to place the files in their final positions (e.g., mkdir, install, cp, and ln), so that the package Auto-DESTDIR can automatically support DESTDIR. The GNU documents recommend not depending on mkdir supporting “-p”, but I think this is obsolete advice; “mkdir -p” has become very widespread, and it is even included in the Single Unix Standard version 3, so I think it’s okay to depend on it now. It’s often helpful to support other make targets like “clean” and “uninstall”.
  9. Do not require web access or interactivity for building, testing, or installing the software. Often these aren’t permitted or possible. If you must interact with a user before installation can complete, install all the files without interaction, and detect when a user tries to start the program for the first time. Normally you should do this by looking for a file or directory named $HOME/.PACKAGE_NAME. If that file/directory doesn’t exist, display information on how to complete the installation or interactively complete the installation, and in the process create the file or directory so this won’t happen next time.
  10. Let users easily use and update the system/local versions of external tools and libraries needed for building and running it, and document what those are.. Document in your README or INSTALL file the external tools/libraries you need for building and running your software, including a URL for each uncommon one. Most importantly, make it easy for users to separately install and update those tools/libraries!

    It’s okay if you include a copy of an externally-developed tool or library, as a convenience for people who don’t want to download it separately. But do not require people to use these embedded versions. Over time, separate tools/libraries get updated, and whatever version is embedded in your release will become obsolete. Even if you sent the latest version at the time, external programs will be upgraded later. This is especially an issue for security updates; people want to be able to download and install them once, and then have that upgrade work for everything. Try to ensure that users can use dynamic libraries to load the latest versions, if that’s practical. Unfortunately, many Java programs are especially bad at this. Many distribution guidelines forbid using bundled libraries in their packages, for example, see Fedora’s “No bundled libraries” rule and Debian’s policy on convenience copies explicitly forbid them. For example, Tom “spot” Callaway gave up packaging the Chromium web browser for Fedora for this reason. Debian works to try to identify packages that embed code from other projects, saying, “This is considered bad for fixing security flaws because the fix needs to be applied in multiple source packages.” Similar problems mean that the math system Sage is not easily installed (via their package managers) on Fedora or Ubuntu.

  11. If you support plugs-in/extensions, use a directory for them. If your application can be extended with “plug-ins” (variously called “extensions” or other terms), do not require plug-ins to modify the contents of any files. Instead, just create a directory for plug-ins to be placed in (the names of such directories often end in “.d” but that’s not required). Then, when a plug-in is to be added or removed, the relevant file(s) can simply be added or removed. Your program should simply use the contents of this directory to determine what plug-ins are available.
  12. If you patch an external library/tool, get the patch upstream. If you don’t, your “local” version will stagnate while the original tool gets many improvements.
  13. Use pkg-config when finding or installing libraries. Freedesktop.org’s pkg-config is relatively new but has swept the world of POSIX systems because it’s an easily and simple solution to a more complicated problem. If you’re compiling an application, it can tell you which flags to use. If you’re creating a library, please create and install the pkg-config .pc files, so others can use pkg-config.
  14. Use standard user interfaces. For command line tools, use “-” single-letter options, “--” long-name options, and “--” by itself to signal “no more options”. For GUI tools, provide a .desktop file. Make it easy for people to start up your application. GNU standards for command line interfaces notes standard option conventions for command line interfaces, including some standard long names like “--version” and “--help”. Once installed, GUI programs should be runnable from the main menu, which requires that a .desktop entry file be created and correctly installed; see freedesktop.org’s desktop entry specification.
  15. Where sensible, make the application useable as a library or via the command line. This makes it possible to build a larger program (using libraries or scripts) that uses your program/library.
  16. Include tests as “make check” and return its success or failure as an exit status. Users and packagers want to know if the program is working as intended, and anyone who makes an improvement to it will want to know if the program is still working. So, include an automated test suite, where you send in inputs, produce outputs, and check that the outputs are correct. It should be invocable with “make check”, it should “exit 0” if the tests succeeded, and it should “exit nonzero” if a test failed (so users will know that the program is not working properly). Tests that cover a significant amount of the program’s functionality and implementation are best, but even a cursory test (a “smoke test”) is better than nothing. Set up your test suite so that it’s easy to add new tests. Make sure the test will test the program built even if another version of the software is installed.
  17. Include documentation. Typically you’ll have a brief README (what is this program?) and INSTALL (how to install it and any requirements for tools). It’s a good idea to have a man page and additional separate documentation, sufficient to answer the questions: What does your program do (sufficient so the user can determine if they want it)? What are some sample invocations? What are its options? What is its input format or API? What is output format or result? What are its strengths or weaknesses? What is the project’s URL?
  18. Make it portable. If your software can port to different architectures and platforms, then it’s probably flexible enough to handle likely future changes. Follow formal and informal standards, such as the Single Unix Specification. Consider using checking tools like the Linux Foundation’s AppChecker.
  19. Support internationalization/locationalization (i18n/L10n). Use tools (like gettext) for any command-line or GUI interface so that people can use their native language in such interfaces. Do not assume that characters all fit in one or two bytes (they do not). I recommend using UTF-8 to encode characters in most cases, but in some cases (e.g., Java, Windows APIs) it may be easier to use a different encoding; if you do, beware of their gotchas (some glyphs take more than 2 bytes in UTF-16, and you have to worry about byte ordering and the byte-order mark).

Test by packaging

By far the best way to see if you’ve done a poor job of packaging is to create a package. If it isn’t trivial to create a package, then you haven’t released it properly.

For example, you could follow the instructions for creating a Fedora package. Just fill in the name, version number, and so on. Your prep, build, and install instructions should be simply: 

 %prep
 %setup -q  # This auto-unpacks tarballs in standard formats

 %build
 %configure # Configures with "--prefix=/usr", CCFLAGS=..., and so on.
 make %{?_smp_mflags}  # This makes, using configuration info.

 %install
 rm -rf %{buildroot}
 make install DESTDIR=%{buildroot}

Or, you can do the same thing with Debian or Ubuntu; create a package. If you have to do anything nonstandard, you are probably doing it wrong.

Yes, packagers can work around a lot, but in the long term that will create problems. If you release your software in a format that is easy to package, you will speed the release of your software out to end-users, and reduce the risk that there will be unintended problems.

What about Autotools?

The GNU autotools suite can help you automate creating a good build process for your users. In particular, autotools will automatically implement many of the conventions listed above.

Autotools is not the only way to create source code releases that are easily built and packaged. But it’s one of the most widely-used, especially for programs that use C or C++ (though it’s not limited to that). Anyone releasing software as non-Windows source code should know a little about autotools, so that they can determine if they should use it.

There isn’t actually a program called “autotools” — instead, this is the conventional name for a suite of GNU tools that can be used together for this purpose, including autoconf, automake, and libtool. The process of changing a program to use autotools is called autoconfiscation.

Years ago, autotools was hard for developers to use and it had lousy documentation. GNU autotools has been significantly improved over the years, and much better documentation is available too. Unfortunately, there’s a lot of really obsolete documentation (and a lot of obsolete complaints about autotools).

See my autotools (autoconf, automake, and friends) if you’re interested in more information. This page includes an autotools tutorial, pointers to other useful information, and warnings that may help you avoid wasting your time.

Recursive make considered harmful

Historically, when programs got big, people would divide the source code into directories, and create a makefile in each directory that would build that one directory. Then you would have makefiles recursively call down to other makefiles. This is the obvious way to scale up, and many projects have historically done this.

However, today many consider recursive makes a bad idea. The lucid “Recursive Make Considered Harmful” by Peter Miller shows that the traditional “recursive make” approach is actually a bad approach. Recursive make tends to produce wrong results, is slower, and is hard to maintain. It is far better to have a single make process (possibly “including” multiple makefiles into it) For example, erikd reports success, and Daniel Elstner has shown that non-recursive make can be significantly faster.

I think Miller is quite right that recursive make, though widely used, is a bad idea. I would add some caveats. Miller correctly notes the problems of “=” and the advantages of “:=”, but “:=” is not in the POSIX standard (it’s a GNU make extension). I worked with the POSIX standards folks to get “::=” into the POSIX standard; hopefully in the future this kind of assignment will be more portable. Still, you can learn a lot about correctly using make from this paper.

Related pages

Here are a few related pages:

There are efforts to help create packages for many different systems simultaneously. For example, the Linux Foundation announced in April 2009 that it would host a build service to build packages for multiple systems simultaneously (based on the OpenSUSE build service). But these are much easier if you follow the guidelines above.

2009: software installation in GNU/Linux is still broken -- and a path to fixing it argues that GNU/Linux distributions should change how application software is installed. That’s worth discussing, but I suspect that if there are any such changes, they will primarily work only for software that follows guidelines like the ones above.

Automating DESTDIR can be a pain, so it’s best if the program supports it to start with; my package Auto-DESTDIR can automatically support DESTDIR in some cases if the program installation does not support it to begin with.

Feel free to see my home page at https://dwheeler.com. You may also want to look at my paper Why OSS/FS? Look at the Numbers! and my book on how to develop secure programs.

(C) Copyright 2009 David A. Wheeler.