If you work on a CIS computer, please simply open RStudio from the App Hub.
If you work on your own laptop, you will need recent versions of R and RStudio installed.
R is the actual programming engine. RStudio is simply an editor which is tailored to the use of R.
Both R and Rstudio are free software and can be downloaded and installed from the links given in this sentence.
Note that R needs to be installed first, then RStudio.
The full material for this tutorial includes
this html document (this is “knitted” R notebook);
a raw version of this notebook, ASMLTutorial.Rmd, which you will use for your practical work;
a complete version of this notebook with solutions (to be provided at the end of this meeting); also in HTML.
a Handout (PDF) giving basics of R programming.
using R Notebooks;
workspace handling;
reading in data;
working with vectors, matrices, and data frames;
basic programming devices (such as if…then, for, while, apply, functions);
application to real data sets.
In this tutorial several of us had problems with compiling Knit to HTML on CIS computers. It has turned out that this is a simple formatting problem, and related to the presence of the “5 little dashes” between subsections. We have now removed all these dashes from this document, and we believe that everything us now working. If problems continue, please send an e-mail to jochen.einbeck@durham.ac.uk
This is in fact the same formatting problem that some people had with executing pdflatex. It is noted that the use of pdflatex requires prior installation of MikTex (but this was not actually content and objective of this tutorial!)
Please create somewhere on your computer an empty directory with title ASMLTutorial. This will be our working directory.
Download the file ASMLTutorial.Rmd (use the right mouse button), and place it in the directory that you have just created. Then open it in RStudio, by double-clicking on that file. If you had already opened this file without having created the directory, please close it again, create the directory, place this file in there, and open the file as described.
We are now continuing our work using this .Rmd file. You can of course still use the html file alongside.
This is a R Notebook. It is based on R Markdown and can be used for parallel typesetting and coding. It is best used in RStudio.
Note: For the use of R Notebooks we will need the packages rmarkdown and knitr. In principle, RStudio should automatically install these if required. If this does not happen on your system for any reason (and you encounter any problems below), you can do this manually, via install.packages("rmarkdown", "knitr").
There are (basically) three ways of executing (`rendering’) code from a R Notebook.
x<-3
x## [1] 3D<- date()
D## [1] "Wed Mar 11 13:50:23 2020"DayofWeek<- substr(D, 1,x) # extracts the first x letters from date object
cat("Today's day of the week is:", DayofWeek)## Today's day of the week is: Wedwhich you can execute by clicking on the green triangle at the top right of the chunk. DO THIS.
Row-wise. Click on any row of the code and then CTRL-ENTER. DO THIS for any of the rows in the chunk above.
You can render (`knit’) the entire document, it produces a nice and tidy summary of all text, code, and results.
TRY THIS NOW, that is click on the ‘Knit’ button at the top of this window. You can choose any of PDF, HTML, Word, or Preview, as output options. [Note: Preview does not actually execute any chunks, it just shows pre-existing output. My recommendation would be to set this option to Knit to HTML. This will produce a .html file in your workspace that you can open separately.]
You can, of course, also edit this document yourself. Specifically, you can also create your own chunk, by clicking on the Insert icon (towards the right of top menu bar of this window). DO THIS.
The syntax of R Markdown is largely self-explaining, detailed explanations are available at https://rmarkdown.rstudio.com/authoring_basics.html.
There is no need to save or copy outputs of today’s work into Latex, MS Word, etc. This document itself will be able to reproduce all your work done today.
The best way to work through this tutorial is to go chunk-by-chunk.
Every R session uses a working directory. This is some directory on your hard drive (or USB key, etc.) which R will use in order to save images, data, and your workspace (see below). R will also assume by default that any data sets that you attempt to read in are stored in this directory. By default, R Notebook chunks will use the location of the .Rmd file as working directory.
Check the content (“workspace”) and location of the current working directory via
ls()## [1] "D" "DayofWeek" "x"getwd()## [1] "C:/Users/jeinb/OneDrive/Documents/DU/Institute for Data Science/Miscada/ASMLTutorial"This should return you the directory ASMLTutorial that you have created above. If this is not right, then the easiest way of fixing this is to close this .Rmd file and start again.
Notes:
setwd(``pathname''). However, this will not work for R Notebooks, as it will only change the directory of the current chunk! More experienced users can follow these instructions to change the working directory for all R chunks.You can, at any time, save the entire workspace for later use, by using the command save.image(``filename''). Let’s do this. Render
save.image("asmltut.RData")then close RStudio and open it again. Then load the saved workspace back via
load("asmltut.RData")
ls()## [1] "D" "DayofWeek" "x"and check whether everything is there! (In our case it should obviously just be x, D and DayofWeek.)
Important (but rather confusing): RStudio opens a new R session to knit your R Notebook file. That is, even if you have some other objects (for instance from previous work) in the global environment (see top right RStudio window) then those objects will not be available when you knit your notebook. To see this, type for instance test<-1 in your R console. You will see it shows up directly in the Global Environment. But, if you create a chunk which refers to test and then knit the notebook, you get an error.
The first data set that we are going to investigate give the energy use (kg of oil equivalent per capita) over 135 countries from 1960 to 2010.
Energy use is defined as the use of primary energy before transformation to other end-use fuels, which is equal to indigenous production plus imports and stock changes, minus exports and fuels supplied to ships and aircraft engaged in international transport.
Source: Worldbank
You can read the data in via
energy.use <-read.csv("http://www.maths.dur.ac.uk/~dma0je/Data/energy.csv", header=TRUE)Alternatively, you can download the data from the given web address, place them in your working directory, end then call energy.use <-read.csv("energy.csv", header=TRUE).
Check now whether things have gone right. Try
dim(energy.use)## [1] 135 52which should give you the dimension \(135 \times 12\). Also try
head(energy.use)in order to see the first six rows.
The object energy.use is a data frame. You can check whether or not an object is a data frame by typing class(object) or is.data.frame(object). Try this for the object energy.use in the chunk below.
# ... It is easy to access individual rows, columns, or elements of a data frame. For instance,
energy.use[127,] energy.use[,49]## [1] 693.70 1088.75 605.54 1850.19 925.65 5887.67 3996.85
## [8] 1387.90 11551.42 163.29 2890.85 5366.42 343.50 570.95
## [15] 1483.16 1068.47 1238.99 7189.78 2641.20 358.42 390.89
## [22] 8168.64 1850.79 1484.02 664.57 289.32 356.51 1069.57
## [29] 495.86 2100.54 884.02 2854.25 4427.55 3597.77 804.18
## [36] 884.81 839.94 799.77 150.80 4198.49 289.97 6895.24
## [43] 4257.74 1299.69 767.12 4026.64 415.46 2875.07 620.35
## [50] 285.70 661.40 1984.58 2657.97 15707.75 528.91 848.57
## [57] 2603.95 1104.80 3456.56 3058.87 3000.63 1852.16 4019.07
## [64] 1268.90 4292.25 484.84 774.41 4585.54 9463.13 556.47
## [71] 2051.76 959.29 2889.12 2740.24 8789.71 1482.47 2733.47
## [78] 2119.55 1750.20 909.89 1182.10 459.93 418.39 318.53
## [85] 744.97 337.76 11321.17 4909.32 3966.37 620.91 722.19
## [92] 5703.57 5677.66 512.15 844.66 685.86 493.85 450.64
## [99] 2547.47 2362.76 19504.15 1805.74 4730.04 6202.50 224.75
## [106] 2141.28 5830.54 3306.64 3631.59 2783.77 3207.52 463.97
## [113] 362.95 5511.75 3405.85 977.91 579.72 442.82 1552.58
## [120] 390.13 11505.66 864.22 1369.86 3631.02 2953.00 11832.50
## [127] 3465.18 7758.94 952.79 1811.91 2319.43 655.12 323.85
## [134] 604.36 758.92 energy.use[127,49] ## [1] 3465.18will give you the 127th row; 49th column; and the entry of the 127th row and the 49th column, respectively (this is the UK energy consumption in 2007). You can also access columns directly through their column names, such as
energy.use$X2007## [1] 693.70 1088.75 605.54 1850.19 925.65 5887.67 3996.85
## [8] 1387.90 11551.42 163.29 2890.85 5366.42 343.50 570.95
## [15] 1483.16 1068.47 1238.99 7189.78 2641.20 358.42 390.89
## [22] 8168.64 1850.79 1484.02 664.57 289.32 356.51 1069.57
## [29] 495.86 2100.54 884.02 2854.25 4427.55 3597.77 804.18
## [36] 884.81 839.94 799.77 150.80 4198.49 289.97 6895.24
## [43] 4257.74 1299.69 767.12 4026.64 415.46 2875.07 620.35
## [50] 285.70 661.40 1984.58 2657.97 15707.75 528.91 848.57
## [57] 2603.95 1104.80 3456.56 3058.87 3000.63 1852.16 4019.07
## [64] 1268.90 4292.25 484.84 774.41 4585.54 9463.13 556.47
## [71] 2051.76 959.29 2889.12 2740.24 8789.71 1482.47 2733.47
## [78] 2119.55 1750.20 909.89 1182.10 459.93 418.39 318.53
## [85] 744.97 337.76 11321.17 4909.32 3966.37 620.91 722.19
## [92] 5703.57 5677.66 512.15 844.66 685.86 493.85 450.64
## [99] 2547.47 2362.76 19504.15 1805.74 4730.04 6202.50 224.75
## [106] 2141.28 5830.54 3306.64 3631.59 2783.77 3207.52 463.97
## [113] 362.95 5511.75 3405.85 977.91 579.72 442.82 1552.58
## [120] 390.13 11505.66 864.22 1369.86 3631.02 2953.00 11832.50
## [127] 3465.18 7758.94 952.79 1811.91 2319.43 655.12 323.85
## [134] 604.36 758.92Data frames are very important as they are the standard form in which data are expected by many R functions, such as lm, glm,….
Let us now simplify the data frame a little bit, so that it is easier to use for the applied work. We reduce our interest to the energy consumption in the years 2001 and 2007. We do this via
energy <- energy.use[,c("X2001", "X2007")]Also, we would like to give the rows and columns of the new data frame meaningful names. Please type
rownames(energy)<- energy.use[, 1]
colnames(energy)<- c("use01", "use07")in order to specify row and column names, respectively. Then type energy to look at your final data frame.
This data frame allows to access information quickly. For instance,
energy["United Kingdom",]gives you the UK values of energy consumption. DO THIS for a couple of countries.
# ...One may be interested in looking at these data in a form in which they are ordered by their energy consumption. This can be done using
order(energy$use07)## [1] 39 10 105 50 26 41 84 133 86 13 27 20 113 120 21 47 83
## [18] 118 98 82 112 66 97 29 94 55 70 14 117 134 3 49 90 132
## [35] 51 25 96 1 91 85 135 45 67 38 35 37 95 56 122 31 36
## [52] 80 5 129 72 116 16 28 2 58 81 17 64 44 123 8 76 15
## [69] 24 119 79 102 130 4 23 62 52 71 30 78 106 131 100 99 57
## [86] 19 53 77 74 110 32 48 73 11 125 61 60 111 108 115 59 127
## [103] 34 124 109 89 7 63 46 40 43 65 33 68 103 88 12 114 93
## [120] 92 107 6 104 42 18 128 22 75 69 87 121 9 126 54 101which gives you a list of numbers. The first number tells you the index (here: 39) of the country with the smallest per-capita energy consumption (here: Eritrea), and typing energy[order(energy$use07),] gives you the full ordered list.
In the chunk below, save this ordered data frame into a new data frame `senergy’.
# ... Next, we wish to identify the nations with extremely large energy consumption, say, more than 10000 kg of oil per capita (Intuitively, what do you think, which countries will this be?). Calling
energy$use07 > 10000## [1] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE
## [12] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [23] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [34] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [45] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE
## [56] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [67] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [78] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE
## [89] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [100] FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE
## [111] FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE
## [122] FALSE FALSE FALSE FALSE TRUE FALSE FALSE FALSE FALSE FALSE FALSE
## [133] FALSE FALSE FALSEwill give you a vector of logical values, with a TRUE for each country for which this condition is met. The command
sum(energy$use07 > 10000)## [1] 6will tell you how many these are, and
which(energy$use07 > 10000)## [1] 9 54 87 101 121 126will give you the index numbers of these countries. From this, we would get the data rows corresponding to these countries via
energy[which(energy$use07 > 10000),]We would like to compare the energy use in 2001 and 2007. Do the same as above but now use the condition energy$use01 > energy$use07 instead. Observe and understand the information that you gain at each step.
# ... A very useful tool to carry out repeated operations is the for command (see Handout!).
Task: Implement a loop which, for all 135 countries, writes a text like
# ... Another command for repeated operations is while. It does not have a fixed number of loops, but proceeds until a certain condition is met. For instance, consider the ordered frame senergy created above. Assume we are interested in the following question: If we take exactly one person from each of the countries with the smallest energy use, i.e. one person from Eritrea, one person from Bangladesh, etc., then how many persons are needed in order to achieve the same use of energy as a single person in Qatar?
To answer this, create objects i and sum07 and assign them the initial value 0. Then use the while function (see Handout) with condition sum07< senergy["Qatar",2] and action i <- i+1; sum07 <- sum07+ senergy[i,2]. Make it clear to yourself what each row does. Also, interpret the result.
# ... Use apply to compute a vector which contains, for each country, the larger of the two energy consumption values given for 2001 and 2007. Consult the see Handout and the corresponding help file (via help(apply) or ?apply) if you are unsure how to do this.
# ... Use hist and boxplot to create histograms and boxplots of the variables use01 and use07. Comment on the distributional shape.
# ...Next, add logarithmic versions of these variables, say luse01 andluse07, to the data frame via
energy$luse01<- log(energy$use01)and foruse07 analogously. Repeat the previous question using the transformed variables. What can we say about the distribution of these transformed variables, compared to the original ones?
# ...Next, we consider a data set featuring \(n=82\) observations of galaxy velocities. Load the galaxies data, read the associated help file, and create a histogram using the option breaks =18 in function hist.
data(galaxies, package="MASS")
# ... For both data sets, the dominating feature is the presence of multiple modes or `clusters’. It is a relevant problem in Statistics and Machine Learning to identify such clusters, and also find the corresponding cluster centers. A simple method to do this is the k-means algorithm. See for instance this resource for a quick introduction into this algorithm.
In R, this algorithm is implemented in the function kmeans. The algorithm requires the specification of the number of clusters in advance, through the argument centers. Study the help file of kmeans and then apply this function onto the luse01 and galaxies data. You will need only the first two arguments of kmeans. For the choice of the number of clusters, you can use visual inspection as a guide for your choice.
#...Have a look at the produced output and try to understand and interpret it in the light of the graphical representations of the data presented earlier.
This was an example for the application a very simple clustering technique, in the one-dimensional (univariate) case.
In one of the lectures of ASML2, we will pick up from here and consider a more elaborated clustering method (mixture models).
Finally, note that clustering is an `unsupervised’ learning technique, since the algorithm needs to make decisions without having seen (not even for training samples) the true allocation of samples to clusters/classes. Such methods are called unsupervised learning. Most of the material dealt with in ASML will be set in the world of supervised learning, where training samples with true and known class labels (or output values) are available.
If you would like to do do a bit more to improve your R skills, we recommend the following resources:
ASML1: Ian H. Jermyn, i.h.jermyn@durham.ac.uk
ASML2: Jochen Einbeck, jochen.einbeck@durham.ac.uk
ASML3: Louis Aslett, louis.aslett@durham.ac.uk