Extend Amazon Linux with Software from other RPM-based Linux Distributions – Part 1 / 2

March 30, 2017 |

Recently we came across the need to install the R programming language, specifically version 3.3.2, on Amazon Linux. And at the time  of writing this article, Amazon Linux only carries version 3.2.2 [1]. After confirming the non-availability of version 3.3.2 in Amazon’s preview repository; we did some research on our end to install R 3.3.2 on Amazon Linux. In this two-part blog series, we share the process, with the belief that you might have a need for this or any other package that is not available in the Amazon Linux software repositories.

Principle Behind Amazon Linux

Let us start by understanding the design principle behind Amazon Linux. Amazon Linux is an RPM-based distribution, and by looking at the system configuration file “/etc/os-release”, we see that Amazon Linux is based on Red Hat Enterprise Linux (RHEL) [2], specifically version 6 [3] and Fedora [4]. As documented in [3], RHEL6 shipped on 10 November 2010 with the latest update, update 8, which shipped on 10 May 2016. The kernel versions that shipped in RHEL6 were between 2.6.32-71 – 2.6.32-642. However, the kernel versions in Amazon Linux is different and much newer, as shown by the results for system command “uname -a”. The following code block shows the results of these two system comments, with the relevant output highlighted using “BOLD ITALICS”.

pic 1

Using the above information and [4], we found that the Linux kernel is either from the Fedora 23 update release or the Fedora 24 beta release. This makes Amazon Linux a hybrid Linux distribution. In our view, we see that, Amazon wants to strike a balance between a slow-moving enterprise-class Linux distribution (i.e., RHEL) and fast-moving community-class Linux distribution (i.e., Fedora) by selectively including packages from either of them, as appropriate.

You can install R (and other packages) from source. However, this involves mastery of the build toolchain & process, satisfying prerequisites and more importantly being responsible for keeping the package up-to-date. Taking into consideration security updates and patch lifecycle, we do not recommend a source-based installation as the first choice, especially when the package is available from a well-maintained upstream repository. However, a valid reason for installing from source is that the package in an upstream repository might be out-of-date.

In the last section we will generalize the process so that you can install any package of your interest.

Installing R 3.3.2

As a first step we confirm that the R 3.3.2 package is not installed by default and is not available through the default repositories. This can be achieved using two bash commands. The following code block shows the output of these two bash commands, with the relevant parts highlighted using “BOLD ITALICS”, and were executed on an Amazon Linux Instance on EC2.

Pic 2

The previous terminal output shows that R is not installed and the version we are interested in is not available from the default repositories. Next, we look at the rpm repository setup in the system. This is achieved by looking at the directory “/etc/yum.repos.d”, as shown in the following code block (irrelevant parts redacted with …).

Pic 3

Each of these software repository configuration files contain one or more repositories and is enabled by setting the variable “enabled=1” or disabled by setting the same variable “enabled=0”. We now look into these files to see which of the repositories are enabled/disabled.

Pic 4

We can see that only the “amazon-main” and “amazon-updates” repositories are enabled. We will now enable the “amazon-preview” repository to check if R332 is available in there. The following code block shows that even the “amazon-preview” repository does not carry R332. Check the references [11] to learn how to enable repos using EPEL as an example. EPEL is a fedora project is called “Extra Packages for Enterprise Linux” and maintains packages for enterprise linux distributions including, but not limited to, RHEL, CentOS, Scientific Linux and Oracle Linux.

Pic 5

We will now restore the default repositories, i.e., disable the “amazon-preview” repository and continue to enable the EPEL repository.

Note: The reason we are enabling one repo at a time is to ensure that there are no package conflicts, as it is possible for different repositories to carry the same package and more commonly different versions of the same package.

Pic 6

We observe two things in the above output. First, there is still no R  available from EPEL. This is strange because the packages list of this epel.repo repository carries R332 packages as per the packages list available from [6]. This lead us to wonder why the package is not visible to the “yum” command. Upon further investigation we found that amazon repositories are allocated a higher priority than 3rd party priorities.

Pic 7

We now proceed to increase the priority of EPEL repositories. This can be done by either adding “priority” to EPEL.repo or removing the priority from amazon repositories, thus making them all the same priority. We proceed by adding “priority=10” to the EPEL repo. The following console shows the result.

Pic 8

The above output confirms that R332 is present in the EPEL repository. We will now proceed to install R332 by specifying the exact version as shown in the following command[12].

Pic 9

The above output shows that some dependencies are not being met by amazon or the EPEL repositories. This came as a surprise to us and set us thinking why would a package like R332 be included in an optional repository but not its prerequisites. In the follow-up post we will show you how to fix this situation by adding 3rd party repositories and selectively enabling only the required packages from those repositories.

Read more about this in part two of the blog series. If you are interested in having our team help you in your projects, please contact us at info@reancloud.com.  Also, if you like these kind of problems and want to join our team please contact us at careers@reancloud.com or visit us at http://www.reancloud.com/company/careers/.

References:

[1] Amazon Linux – https://aws.amazon.com/amazon-linux-ami/

[2] Red Hat Enterprise Linux – https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux

[3] RHEL 6 – https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#RHEL_6

[4] Fedora version history – https://en.wikipedia.org/wiki/Fedora_version_history

[5] EPEL – https://fedoraproject.org/wiki/EPEL

[6] EPEL packages list – https://dl.fedoraproject.org/pub/epel/6/x86_64/

[7] Missing Packages Error – https://forums.aws.amazon.com/thread.jspa?messageID=262860

[8] https://aws.amazon.com/amazon-linux-ami/2016.09-packages/

[9] http://mirror.raystedman.net/centos/6.8/os/x86_64/Packages/

[10] https://morecode.wordpress.com/2011/12/08/including-one-package-in-a-yum-repository/

[11] https://aws.amazon.com/amazon-linux-ami/faqs/

[12] http://unix.stackexchange.com/questions/151689/how-can-i-instruct-yum-to-install-a-specific-version-of-package-x

[13] https://cran.r-project.org/mirrors.html

[14] http://stackoverflow.com/questions/11488174/how-to-select-a-cran-mirror-in-r

[15] https://www.centos.org/keys/

Other Blog Posts

Blog

Top 5 Reasons to Utilize Cloud Computing in Financial Services
Read More
Blog

Is Migrating to the Cloud Safe for Financial Sector Companies?
Read More
Blog

REAN Cloud is one of the few AWS Premier Partners to achieve both AWS DevOps Competency and MSP Designation
Read More
Blog

7 Ways DevOps Can Save Your Company…Time and Money
Read More
Request Consultation
close slider