Extend Amazon Linux with Software from other RPM-based Linux Distributions – Part 2 / 2

March 30, 2017 |

Part 1 Recap

In part one of this blog series, we started installing a package (including a specific version) that is not available in the default Amazon Linux repositories. We took the example of installing the R programming language, version 3.3.2. We located this specific package in an optional repository, but some required prerequisites were not available. In this concluding blog post, we continue this journey to install the prerequisites and realize the R332 package.

Installing R 3.3.2 (continued)

In [8], amazon lists the packages that went into the Amazon Linux distribution release 2016-09. We confirmed that none of the above required packages were included in the distribution. Further, we also confirmed that the prerequisite packages were not present in the EPEL repository [6]. This step is required to ensure that the packages are not visible due to system configuration such as “priority”. The discussion thread in [7] suggested the possibility of adding the base CentOS repository to get the packages. We followed this route.

We now initiate an Amazon EC2 instance with CentOS 6.5 and observe the repositories that have been enabled by default. The bash commands and the corresponding results in following code block (retracted) shows this.

Blog 2 pic 1

We inferred a couple of things from the above console output.

  1. gpgcheck is ON. This means we will need to create a gpgkey file and copy the public key. We can copy the public key from this instance or from the Internet.
  2. We need to convert the variables used in the URLs to be static numbers since they no longer will apply to Amazon Linux.
  3. We also confirm that the required packages (tk, tk-devel and xdg-utils) are present in the CentOS repository using a mirror site [9].
  4. In order to avoid any potential package conflicts between the Amazon base and CentOS base repositories, we explicitly only include the required packages from the CentOS repository using the tip provided in [10].
  5. Finally, we also increase the priority to match that of Amazon base, EPEL repositories.
  6. Using appropriate substitution, we derive the following block that will be added to a brand new .repo file.

Blog 2 pic 2

Now, we will switch back to our Amazon Linux instance. We create a new file called “centos.repo” in the “/etc/yum.repos.d” and copy the above contents. We also make a new file “/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6” and copy the public key from the CentOS instance to this instance. Following this we will rerun the command “sudo yum -y install R“. Finally, as a test, we will now install the R ML package “caret” and verify that it installs successfully. The reason we are using caret is due to the fact that it requires R 3.3.2.

Blog 2 pic 3

In the above command, we have pre-selected an R repo that automatically redirects you to the nearest mirror. The url [13] shows all the mirrors that is available and you are free to choose a static one that is near to you. The url [14] shows the syntax for this “install.packages” command when selecting a mirror.

As with cent-os-6 repo configuration, we will also tighten the packages exposed from epel.repo by adding the required filters – “includepkgs=R*,libRmath*” to the epel.repo to avoid any package conflicts between epel, centos and amazon linux repos.

Summary

The result from the previous sections shows that we are able to install R332 on Amazon EMR and Amazon EC2 when we are using Amazon Linux. In this section, we will summarize all the steps that is required to get R332 up and running along with the various end states of the various configuration files. We will then generalize the steps so you can adapt it to install other packages of your interest.

Blog 2 pic 4

Blog 2 pic 5

Blog 2 pic 6

Blog 2 pic 7

We now provide you a generic checklist that we believe will be useful to install a package, and its prerequisites, on an Amazon Linux instance. Every subsequent step assumes that the answer to the previous step is a “no”.

  1. Is the package installed by default ? Use “which <packagename>” command to find out.
  2. Is the package available from default repositories ? Use “yum –showduplicates list <packagename>” to find out.
  3. Is the package available from one or more disabled repositories ? Inspect the repo configuration files in  “/etc/yum.repos.d” to find out.
  4. Enable one repo at a time by changing the configuration variable “enabled=1”. Ensure that the priority level is the same as that of the default repositories.
    1. Rerun step 2 to check if this exposes the package you are interested in.
  5. If the package is still not available, enable the the CentOS repository by adding the .repo file. Ensure the CentOS version is compatible with the Amazon Linux version.
  6. CentOS.repo by default does a GPG-based key check. Either disable this or add the public key listed above.
    1. You can also download the public key from the official website [15] if you don’t trust us 😉
  7. At this stage you should be able to install any package from Amazon Linux, EPEL and CentOS distributions. If you still can’t find your package, another popular RPM distribution that you can try is the Fedora Linux. YMMV.

Conclusion:

In summary, we hope that this post has been helpful in providing guidance on how you can install RPM based packages that are not carried by the Amazon Linux distribution. If you are interested in having our team help you with your projects, please contact us at info@reancloud.com.  Also, if you like these kind of problems and want to join our team please contact us at careers@reancloud.com or visit us at http://www.reancloud.com/company/careers/.

References:

[1] Amazon Linux – https://aws.amazon.com/amazon-linux-ami/

[2] Red Hat Enterprise Linux – https://www.redhat.com/en/technologies/linux-platforms/enterprise-linux

[3] RHEL 6 – https://en.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#RHEL_6

[4] Fedora version history – https://en.wikipedia.org/wiki/Fedora_version_history

[5] EPEL – https://fedoraproject.org/wiki/EPEL

[6] EPEL packages list – https://dl.fedoraproject.org/pub/epel/6/x86_64/

[7] missing packages error – https://forums.aws.amazon.com/thread.jspa?messageID=262860

[8] https://aws.amazon.com/amazon-linux-ami/2016.09-packages/

[9] http://mirror.raystedman.net/centos/6.8/os/x86_64/Packages/

[10] https://morecode.wordpress.com/2011/12/08/including-one-package-in-a-yum-repository/

[11] https://aws.amazon.com/amazon-linux-ami/faqs/

[12] http://unix.stackexchange.com/questions/151689/how-can-i-instruct-yum-to-install-a-specific-version-of-package-x

[13] https://cran.r-project.org/mirrors.html

[14] http://stackoverflow.com/questions/11488174/how-to-select-a-cran-mirror-in-r

[15] https://www.centos.org/keys/

Related Blog Posts

Blog

Top 5 Reasons to Utilize Cloud Computing in Financial Services
Read More
Blog

Is Migrating to the Cloud Safe for Financial Sector Companies?
Read More
Blog

REAN Cloud is one of the few AWS Premier Partners to achieve both AWS DevOps Competency and MSP Designation
Read More
Blog

7 Ways DevOps Can Save Your Company…Time and Money
Read More