Lab 4 - Distributions, package management, compiling software

What is a distribution?

A Linux distribution is a combination of the Linux kernel, package management system, drivers, scheduler, and other software. They are the operating systems that use Linux as a kernel.

What should I use then?

Currently, there are around 300 actively developed distributions listed on DistroWatch. It might be a bit overwhelming with so many choices. However, there are three types of distributions that consist the bulk of most widely used Linux distributions: Debian, RHEL, and Arch.

What is Debian?

Debian is a distribution developed by the Debian Project beginning in 1993. This distribution strives to provide a completely free (libre) software experience. Many other distributions are based on Debian as Debian releases are usually very stable. One such distribution is Ubuntu, which is very popular and has spawned its many of its own derivatives. What unifies all of Debian-based distributions is the usage of the DEB package format and the dpkg package manager. This is what we’re using for this DeCal.

What is RHEL?

Red Hat Enterprise Linux (RHEL) is a distribution developed by Red Hat beginning in 1995 as Red Hat Linux. REHL is a commercial distribution, requiring a license to run. In 2004, Red Hat Linux was split into RHEL and Fedora, a cutting edge free community distribution. CentOS is a community derivative that was created to be binary compatible with RHEL and was recently taken up by Red Hat itself. RHEL and CentOS prides on being even more stable than Debian, with long release cycles. What RHEL-based distributions share alike is their usage of the RPM package format and the yum/dnf package manager/frontend.

What is Arch?

Arch Linux is a distribution that started development in 2002. It is notorious for being a distribution that that allows the users the experience of installing the distribution without an installer or install scripts. However, this model lends itself to be extremely flexible and customizable by the user. It uses the pacman package manager.

But what about these “package managers”?

If you come from the Windows or even MacOS ecosystem, you might be scratching your head thinking what a package managers is. On those operating systems, the most conventional and used method of installing programs is to run an installer that copies the files from an archive to the correct place (and makes changes to a registry of some sorts).

Enter the package manager, a centralized way to install, update, and remove software from repositories/sources. Most package managers work in the same way: the package manager gets or updates a list of packages and then fetches the packages that is referenced in the list if is requested to be installed. A package manager is like a librarian. When a patron wants to read a book, the librarian consults a catalog (which is the package database) and then fetches the book from the shelf and gives it to the patron (installing the package). Sometimes, the librarian has to update the catalog because new editions of the books were added (updating the package database).

Debian: An Example

In this class, we will be focused on using Debian. As noted before, Debian uses apt/dpkg as its package manager.

To package manager is invoked by using apt. Before you do anything, it is good package to update the package list so your package manager can find and fetch the most up-to-date packages. To do that, run:

apt update

To find the package to install:

apt-cache search [package|description]

To install a package, run:

apt install package

To remove a package, run:

apt remove package

Easy? Sometimes, you want to update the packages that you have installed, run:

apt dist-upgrade

There are also other commands, such as removing unneeded dependencies and purging packages but that is what the man pages are for.

Example

We are going to install GCC for the next step of the lab.

Simply run:

sudo apt install gcc

Now check if GCC is installed by typing the followng:

gcc -v

Now install the package ruby-dev and make for the next parts of the lab.

But what about software that isn’t in the repositories/package lists?

This is a very common and frustrating thing to occur. Just when you want to install a package, it appears that it is not listed in any of the repositories that are available. What can you do? There are two ways in going about this.

One way is the find a generic binary package from the developer. This comes in the form of shady shell scripts, or a binary tarball in which you can just extract it and run it, or an appimage, which is a special type of executable that brings its own dependencies.

The other way is to compile your own software from scratch. What does that mean? Free and open-source software usually has their source code somewhere publicly available (GitHub, GitLab, their website). If you fetch their source code, it won’t magically run out of the box. The source code is like the recipe, while the software itself is like food. A package is like a food that is put into a box but we won’t covering the details of making a package yourself (there are tools that do that for you and it varies from platform to platform).

So how do I compile?

Compiling software on Linux is a mixed bag. Sometimes, all the dependencies are installed on your computer and there is no fiddling around. Sometimes, the dependencies don’t even exist pre-compiled for your distribution so you have to compile those yourself in order to compile what’s at hand. But all of these steps all have one component in common: the makefile.

In most source tarballs, there is usually a makefile that contains a set of directives to compile a project. This is because in a project, there is usually multiple files that need to be compiled by the compiler. And even then, there are multiple compilers and compiler settings that can be used.

Many projects that have to be compiled are usually in C and/or C++. On most Linux distributions, there are usually three compiler “options”. There is the GNU Toolchain which uses GCC, LLVM which uses clang, and Intel’s proprietary toolkit. Both GCC and clang are open-source and free. However, clang was created with a permissive license in response to GCC’s more viral copyleft license. Although clang was made with the intention of being completely compatible with GCC, there are some quirks and most open-source software still target GCC.

To compile most software that you have the dependencies to, run:

make

This is will usually choose the correct compiler and compile the whole project. Usually when the compilation is done, the executables are in /bin. This utilizes Make which general purpose workflow program, mainly used for compilation. It runs all the compilers, cleans, and installation processes using different

Example: Make

We’re going to try and make FFTW, a popular Fast Fourier Transform library. To get it, run the following:

wget http://www.fftw.org/fftw-3.3.6-pl2.tar.gz

Now we are going to unpackage it by running the following:

tar -xvf fftw-3.3.6-pl2.tar.gz

Now we’re going to go into the directory:

cd fftw-3.3.6-pl2

And we are going to make a new folder called out and run the configure script so the Make will copy the output of the compilation to the right place later.

mkdir out

./configure --prefix=/home/[INSERT YOUR USERNAME HERE]/fftw-3.3.6-pl2/out

Now we will start compiling the whole thing using Make.

make

If you are on a multi-core machine, you can increase the number of cores that Make can run on by adding a -j[INSERT CORE NUMBER HERE]

After it’s done, you can have it copy all the binaries, libraries, and other files to out by running the following:

make install

If for some reason something fails and you want to redo the compilation again, run the following the clean out the compiled files:

make clean

Using GCC

GCC and clang also have compiler flags that allow certain features to be enabled. Usually the flags that actually matter are optimization flags. Depending on what you want to optimize for, either space or memory or speed, there is a flag for it.

Now, we will make a very simple application in C that prints “Hello Penguin!” named hellopenguin. Invoke:

nano hellopenguin.c

And then type in the following:

#include <stdio.h>

int main()

{

   printf("Hello Penguin!\n");

   return 0;

}

Now save and exit.

We will now compile the source file that you have just written:

gcc hellopenguin.c -o hellopenguin

What this does is to take in a source file hellopenguin.c and compile it to an executable named hellopenguin with the -o output flag.

How do I package stuff?

Packaging manually for Debian can be very hard and frustrating, especially for first timers. That’s why for this class, we’ll be using a really cool Ruby package called fpm which simplifies the task of packaging a lot. Please note that if you were going to package for Debian’s own repositories, this is NOT the correct/formal way to go about packaging. It is merely a great way to backport or package your own applications extremely quickly.

First make sure Ruby and it’s own package manager called Gem is installed using the commands from above. Now run the following to install fpm locally:

gem install fpm --user

or if you want to install it globally:

sudo gem install fpm

Try invoking fpm and if it doesn’t work, add ~/.gem/ruby/2.3.0/bin to your PATH. To do that, add this .bashrc or just type the following into Bash:

PATH=~/.gem/ruby/2.3.0/bin:$PATH

Now we will create a very simple package using the hellopenguin executable that you made above. First, we will make a new folder named packpenguin:

mkdir packpenguin

Now we will create the folder structure of where the executable show reside in. In Debian, user-level packages usually reside in the folder /usr/bin/.

mkdir packpenguin/usr

mkdir packpenguin/usr/bin

Now move your hellopenguin into the packpenguin/usr/bin/ folder.

mv hellopenguin packpenguin/usr/bin/

Now we will create a package called hellopenguin. Move into the parent directory of the hellopenguin folder and invoke the following:

fpm -s dir -t deb -n hellopenguin -v 1.0~ocf1 -C packpenguin

This specifies that you want to take in a directory, using the -s flag, and to output a .deb package using the -t flag. It takes in a directory called packpenguin, using the -C flag, and output a .deb file named hellopenguin, using the -n, with a version number of 1.0~ocf1, using the -v flag.

Now test it by invoking apt and installing it:

sudo apt install ./hellopenguin_1.0~ocf1_amd64.deb

Now you should be able to run hellopenguin by doing the following:

hellopenguin

If all of this works, you’re ready to be checked off!

If you have time left over, try and package FFTW. Note that the out folder will serve as your usr folder above. You will have to rename it to usr and then make a new folder and put that usr folder into it.