Beginner Lab 4 - Debian, packages, compiling software

What is a distribution?

A distribution is a combination of the a kernel, init system, libraries, drivers, and other critical software. There are many distributions, for Linux and other kernels as well, such as BSD and Solaris.

What should I use then?

Currently, there are around 300 actively developed Linux distributions listed on DistroWatch. It might be a bit overwhelming with so many choices. Popular distributions include Debian, Ubuntu, Arch Linux, CentOS, and Red Hat Enterprise Linux, but in this course, we’ll only be using Debian.

What is Debian?

Debian is a distribution developed by the Debian Project starting in 1993. This distribution strives to provide a completely free (libre) software experience. In the context of free software, “libre” means that the source code is available under a permissive license, enabling downstream modification and rediistribution. Many other distributions are based on Debian, as Debian releases are usually very stable, meaning that the software provided by the distribution has been tested to be as bug-free as possible. One such distribution is Ubuntu, which is very popular and has spawned its own family of distributions. What defines the Debian family of distributions is the use of the DEB package format and the dpkg package manager.

But what about these “package managers”?

If you come from the Windows or MacOS ecosystems, you might be wondering what a package manager is. On those operating systems, the most common method of installing new programs is to run an installer executable that unpacks and copies files into the correct location, and updates the OS registry to reflect this.

Enter the package manager, a centralized way to install, update, and remove software from your computer using verified sources called repositories. Most package managers work in the same way: the package manager gets a list of packages from the repository, and then, when asked to install a package, fetches the package from the repository, verifies that it is legit, and installs it. A package manager is like a librarian. When a patron wants to read a book, the librarian consults a catalog (which is the package database) and then fetches the book from the shelf and gives it to the patron (installing the package). Sometimes, the librarian has to update the catalog because new editions of the books were added (updating the package database). When the patron is done, the librarian knows where to return the book (cleaning up after removing a package). Contrast this to the approach on Windows, where dedicated uninstaller programs are occasionally necessary as the OS does not know where programs install their own files.

Debian: An Example

In this class, we will be focused on using Debian. As noted before, Debian uses apt or dpkg as its package manager.

We invoke the package manager by using apt. Before installing anything, you shoud update your local copy of packages and versions to ensure the package manager installs the latest and most up-to-date packages. To do that, run:

$ sudo apt update

To find the package to install:

$ apt search [package|description]

To install a package, run:

$ sudo apt install package

To remove a package, run:

$ sudo apt remove package

Easy? When you want to upgrade the packages that you have installed when new versions are released, you can do so by calling:

$ sudo apt upgrade

There are also other commands, such as removing unneeded dependencies (sudo apt autoremove) and purging packages (sudo apt purge) but that is what the man pages are for.

Example

We are going to install GCC for the next step of the lab.

This, and all other parts of the lab, should be done on your DigitalOcean VMs, as you cannot install packages on OCF machines.

Please note that you can only connect to your VM from inside the Berkeley network, so you will have to either be on campus wifi, or you will have to SSH through ssh.ocf.berkeley.edu (or a similar on-campus host) to access your VM. You should be able to connect to it at <OCF username>.decal.xcf.sh by running ssh <username>@<username>.decal.xcf.sh and the password you set when your first logged in, or using the email you received.

Simply run:

$ sudo apt install gcc

Now check if GCC is installed by running the following to check GCC’s version:

$ gcc --version

Now install the ruby-dev and ruby-ffi packages for the next parts of the lab.

But what about software that isn’t in the repositories?

Sometimes it happens that a program you want to install hasn’t been packaged for your distributions, or hasn’t been packaged at all. You have several options in this scenario to install the software you want:

Warning:

Installing software not from a repository carries the same risks as installing software from a random .exe or .msi on Windows.

Linux is popularly considered more secure than Windows or MacOS because of its use of repositories – but a malicious package installed manually can pwn your system as easily as malware on any other OSes. Use common sense! Only manually install from sources you trust.

(For users of Ubuntu: PPAs, or third-party repositories, carry the same risks of running malicious third-party code.)

If the developers provide a package compatible with your distribution (.deb for Debian-based distros), you can download that package and install it using:

$ sudo apt install ./package.deb

Another way is to find a generic binary package from the developer. This can come in the form of shady shell scripts, or a binary tarball that you can just extract it and run, or an appimage, which is a special type of executable that includes its own dependencies.

The last way is to compile your software from scratch. What does that mean? Open source software must have its source code publicly available somehow, (GitHub, GitLab, their website). If you fetch their source code, it won’t magically run out of the box. The source code is like the recipe, while the software itself is like food. A package is like a food that is put into a box but we won’t covering the details of making a package yourself (there are tools that do that for you and it varies from platform to platform).

So how do I compile?

Compiling software on Linux can be a mixed bag. Sometimes, all the dependencies (like libraries) are installed on your computer and there is no fiddling around. Sometimes, the dependencies don’t even exist pre-compiled for your distribution so you have to compile those yourself in order to compile what’s at hand. Most of the time, these steps are simplified through the use of a Makefile, which controls the Make build system.

In most source tarballs, there is usually a Makefile that contains a set of directives to compile a project. This is because there are usually multiple files across many directories that need to be compiled together into the final executable. On top of that, there are multiple settings that control, for example, optimizations, size of the executable, static vs. dynamic linking, and whether to link against system libraries or alternatives.

Many projects that have to be compiled are usually in C, C++, or similar lower-level languages. On most Linux distributions, there are usually three compiler “options”. There is the GNU Toolchain which provides gcc or the GNU C Compiler, LLVM which provides clang, and Intel’s proprietary toolkit, icc. Both gcc and clang are open-source and free software. While clang was designed to be a drop-in replacement for gcc, several quirks remain that make porting large programs nontrivial.

To compile software that provides a Makefile, assuming you have the dependencies, simply type:

$ make

This is will usually choose the correct compiler and compile the whole project. Once the compilation is done, the resulting executables are usually stored in ./bin or ./build.

GCC and clang also have compiler flags that allow certain features to be enabled. Usually the flags that actually matter are optimization flags. Depending on what you want to optimize for, either space or memory or speed, there is a flag for it.

Now, we will make a very simple application in C that prints “Hello Penguin!” named hellopenguin. Run:

$ nano hellopenguin.c

to create a file named hellopenguin.c, and type in the following:

#include <stdio.h>
int main()
{
    printf("Hello Penguin!");
    return 0;
}

Now save and exit.

We will now compile the source file that you have just written:

$ gcc hellopenguin.c -o hellopenguin

What this does is take the source file hellopenguin.c and compile it, writing the executable output to a file named hellopenguin.

How do I package stuff?

Packaging manually for Debian can frustrating, especially for first timers. That’s why for this class, we’ll be using a really cool Ruby called fpm which simplifies the task of packaging a lot. Please note that if you were going to package for Debian’s own repositories, this is NOT the correct/formal way to go about packaging. It is merely a great way to backport or package your own applications extremely quickly. First make sure ruby and its own package manager called gem is installed. If they aren’t, run sudo apt install ruby-dev ruby-ffi. Now run the following to install fpm locally:

$ gem install fpm --user

Try invoking fpm and if it doesn’t work, add ~/.gem/ruby/2.3.0/bin to your PATH. To do that, add this to your .bashrc, or just type the following into Bash to temporarily add it to your PATH:

$ export PATH=~/.gem/ruby/2.3.0/bin:$PATH

Now we will create a very simple package using the hellopenguin executable that you made above. First, we will make a new folder named packpenguin and move into it:

$ mkdir packpenguin

$ cd packpenguin

Now we will create the folder structure of where the executable will reside.

In Debian, user-level packages usually reside in the folder /usr/bin/.

$ mkdir -p usr/bin

Now move your hellopenguin into the packpenguin/usr/bin/ folder.

$ cd ../ # cd into the directory where the hellpenguin executable is
$ mv hellopenguin packpenguin/usr/bin/

Now we will create a package called hellopenguin. Move into the parent directory of the hellopenguin folder and invoke the following:

$ fpm -s dir -t deb -n hellopenguin -v 1.0~ocf1 -C packpenguin

This specifies that you want to take in a directory, using the -s flag, and to output a .deb package using the -t flag. It takes in a directory called packpenguin, using the -C flag, and output a .deb file named hellopenguin, using -n, with a version number of 1.0~ocf1, using the -v flag.

Now test it by invoking apt and installing it, replacing <version+arch> with the appropriate version and architecture that the package is built for, which will be provided by fpm when the package is built:

$ sudo apt install ./hellopenguin_<version+arch>.deb

Now you should be able to run hellopenguin by doing the following:

$ hellopenguin

If all of this works, you’re ready to be checked off!