---
title: "ETC1010: Data Modelling and Computing"
subtitle: "Lecture 6B: Functions"
author: "Dr. Nicholas Tierney & Professor Di Cook"
institute: "EBS, Monash U."
date: "`r Sys.Date()`"
output:
  xaringan::moon_reader:
    lib_dir: libs
    css: ["shinobi", "ninjutsu", "slides.css"]
    seal: true
    self_contained: false
    nature:
      ratio: "16:9"
      highlightStyle: github
      highlightLines: true
      countIncrementalSlides: false
---

```{r setup, include=FALSE}
library(emo)
library(tidyverse)
library(polite)
library(rvest)
library(countdown)
library(knitr)
library(lubridate)
library(tuneR)
library(gridExtra)
library(plotly)
knitr::opts_chunk$set(
  fig.width = 8,
  fig.height = 4.5,
  fig.retina = 3,
  fig.align = "center",
  out.width = "90%",
  message = FALSE,
  warning = FALSE,
  cache = FALSE,
  autodep = TRUE,
  hiline = TRUE
)

knitr::opts_hooks$set(fig.callout = function(options) {
  if (options$fig.callout) {
    options$echo <- FALSE
    options$out.height <- "99%"
    options$fig.width <- 16
    options$fig.height <- 8
  }
  options
})

options(
  htmltools.dir.version = FALSE,
  width = 90,
  max.print = 9999,
  knitr.table.format = "html"
)

as_table <- function(...) knitr::kable(..., format = "html", digits = 3)
```

class: bg-main1

# Recap

.huge[
- style (it's important!)
- ed quiz
]


---
class: bg-main3

# File Paths and organising yourself

.huge[
- It's important when you start working on your own machine that you understand _file storage hygiene_. 
- It helps prevent unexpected problems and makes you more productive 
- You'll spend less time fighting against strange file paths. 
- Not sure what a file path is? We'll explain that as well.
]

---
class: bg-main1

# Your Turn

.huge[
Discuss:

1. What your normal "workflow" is for starting a new project
2. Possible challenges that might arise when maintaining your project
]


```{r cd-your-turn, echo = FALSE}
# countdown(minutes = 2, play_sound = TRUE)
```


---
class: bg-main1

# A Mantra: When you start a new project \- Open a new RStudio project

.huge[
- This section is heavily influenced by [Jenny Bryan's great blog post on project based workflows.](https://www.tidyverse.org/articles/2017/12/workflow-vs-script/)
- Sometimes this is the first line of an R Script or R markdown file.

```r
setwd("c:/really/long/file/path/to/this/directory)
```
- What do you think the `setwd` code does?
]

---
class: bg-main1

# What does `setwd()` do?

.huge[
- "set my working directory to this specific working directory".

- It means that you can read in data and other things like this:

```r
data <- read_csv("data/mydata.csv")
```

- Instead of

```r
data <- read_csv("c:/really/long/file/path/to/this/directory/data/mydata.csv")
```
]

---
class: bg-main1

# What does `setwd()` do?

.huge[
- This has the effect of **making the file paths work in your file**
- This is a problem because, among other things, using `setwd()` like this:
  -  Has 0% chance of working on someone else's machine (**this includes you in >6 months**)
  - Your file is not self-contained and portable. (Think: _"What if this folder moved to /Downloads, or onto another machine?"_)
- To get this to work, you need to hand edit the file path to your machine. 
- This is painful. And when you do this all the time, it gets old, fast.
]


---
class: bg-main1

# What even is a file path?
  
.huge[
- The machine-readable directions to where files on your computer live.
- So, this file path:

```
/Users/njtierney/rmd4sci-materials/demo-gapminder.Rmd
```

Describes the location of the file "demo-gapminder.Rmd". 
]


---
class: bg-main1

# What even is a file path?

.huge[
We could visualise the path

```
/Users/njtierney/rmd4sci-materials/demo-gapminder.Rmd
```

as:

```
users
  └── njtierney
        └──  rmd4sci-materials
               └── demo-gapminder.Rmd
```
]

---
class: bg-main1

.huge[
- To read in the `gapminder.csv` file, you might need to write code like this:

```r
gapminder <- read_csv("/Users/njtierney/Desktop/rmd4sci-materials/data/gapminder.csv")
```
 
This is a problem, because this is not portable code.
]

--

.huge[
It has ZERO chance of working on someone else's machine
]

---
class: bg-main1

.huge[
If you have an RStudio project file inside the `rmd4sci-materials` folder, you can instead write the following:

```r
gapminder <- read_csv("data/gapminder.csv")
```
]

---
class: bg-main1

# Your Turn

.vlarge[
- What folders are above the `health.csv` file in the following given file path? 

`"/Users/miles/etc1010/week1/data/health.csv"`

- What would be the result of using the following code in `demo-gapminder.Rmd`, and then using the code, and then moving this to another location, say inside your C drive?

```r
setwd("Downloads/etc1010/week1/week1.Rmd)
```
]

---
class: bg-main1

# Is there an answer to the madness?

.huge[
- This file path situation is a real pain. 
]

--

.huge[
Yes!

When you start on a new idea, new research project, paper... Anything new. It should start its life as an **rstudio project**. 
]

---
class: bg-main1

# One more time: say it with me

.vhuge[
When you start on a new idea, new research project, paper... Anything new. It should start its life as an **rstudio project**. 

]

---
class: bg-main1

# Demo

---
class: bg-main1

# Rstudio projects help keep related work together in the same place.


.huge[

* Keep all your files together
* Set the working directory to the project directory
* Starts a new session of R
* Restore previously edited files into the editor tabs
* Restore other rstudio settings
* Allow for multiple R projects open at the same time.
]

---
class: bg-main1

# Rstudio projects help keep you sane, because:

.huge[
* Projects are independent. 
* You can work on different projects at the same time. 
* Objects & functions created in one project won't impact another.  
* You can refer to your data and other projects in a consistent way.
]

---
class: bg-main1

# Rstudio projects help keep you sane, because:

.huge[
And finally, the big one

**RStudio projects help resolve file path problems**, because they automatically set the working directory to the location of the rstudio project.
]

---
class: bg-main1

# A Bit more on file paths

---
class: bg-main1

# The "here" package

.huge[
- RStudio projects help resolve file path problems
- In some cases you might have many folders in your r project. To help navigate them appropriately, you can use the `here` package to provide the full path directory, in a compact way.

```{r show-here}
here::here("exercise")
```

]
---
class: bg-main1

# The `here` package

.huge[
```{r show-here-exercise}
here::here("exercise", "lecture-6b-exercise.Rmd")
```

]

---
class: bg-main1

.huge[

```r
here::here("exercise", "lecture-6b-exercise.Rmd")
```

You can read the above `here` code as:

> In the folder `exercise`, there is a file called `lecture-6b-exercise.Rmd`. Can you please give me the full path to that file? 

]


---
class: bg-main1

# The `here` package

.huge[
This is really handy for a few reasons:

1. It makes file paths _completely_ portable
1. Rmarkdown documents have a special way of looking for files, this helps eliminate file path pain.
1. If you decide to not use RStudio projects, you have code that will work on _any machine_
]

---
class: bg-main1

# Remember

.huge[
> If the first line of your R script is 

```
setwd("C:\Users\jenny\path\that\only\I\have")
```
> I will come into your office and SET YOUR COMPUTER ON FIRE 🔥.

-- Jenny Bryan
]

---
class: bg-main1

# Aside: How to create an RStudio project

.huge[
- Go to [section 5.12 of rmarkdown for scientists](https://rmd4sci.njtierney.com/workflow.html#aside-creating-an-rstudio-project)
]

---
class: bg-main1

# Summary of file paths and rstudio projects

.huge[
So far we have:

* Learnt what file paths are
* How to setup an rstudio project
* How to construct full file paths with the `here` package

]

---
class: bg-main1

# Go to ed and finish quiz 6b

---
class: bg-main1

# Motivating Functions

---
class: bg-main1

# Remember web scraping?

```{r show-st-poster, echo = FALSE, out.width = "40%"}
include_graphics("https://m.media-amazon.com/images/M/MV5BZGExYjQzNTQtNGNhMi00YmY1LTlhY2MtMTRjODg3MjU4YTAyXkEyXkFqcGdeQXVyMTkxNjUyNQ@@._V1_.jpg")
```

---
class: bg-main1

# How many episodes are there of Stranger Things?

```{r st-episode-count}
st_episode <- 
  bow("https://www.imdb.com/title/tt4574334/") %>%
  scrape() %>%
  html_nodes(".np_right_arrow .bp_sub_heading") %>%
  html_text() %>%
  str_remove(" episodes") %>%
  as.numeric()

st_episode
```

---
class: bg-main1

# How many episodes of stranger things? And ... Mindhunter? 

```{r mh-scrape}
st_episode <- bow("https://www.imdb.com/title/tt4574334/") %>%
  scrape() %>%
  html_nodes(".np_right_arrow .bp_sub_heading") %>%
  html_text() %>%
  str_remove(" episodes") %>%
  as.numeric()
st_episode

mh_episodes <- bow("https://www.imdb.com/title/tt4574334/") %>%
  scrape() %>%
  html_nodes(".np_right_arrow .bp_sub_heading") %>%
  html_text() %>%
  str_remove(" episodes") %>%
  as.numeric()
mh_episodes
```

---
class: bg-main1

# Why functions?

.huge[
- Automate common tasks in a power powerful and general way than copy-and-pasting:
    - Give a functions an evocative name that makes code easier to understand.
    - As requirements change, **you only need to update code in one place, instead of many**.
    - You eliminate the chance of making incidental mistakes when you copy and paste (i.e. updating a variable name in one place, but not in another).
]

---
class: bg-main1

# Why functions?

.huge[
- Down the line: Improve your reach as a data scientist by writing functions (and packages!) that others use
]

---
class: bg-main1

# Setup

```{r mbow-scrape-all}
library(tidyverse)
library(rvest)
library(polite)

st <- bow("http://www.imdb.com/title/tt4574334/") %>% scrape()

twd <- bow("http://www.imdb.com/title/tt1520211/") %>% scrape()

got <- bow("http://www.imdb.com/title/tt0944947/") %>% scrape()
```

---
class: bg-main1

# When should you write a function?

.huge[
Whenever you’ve copied and pasted a block of code more than twice.

When you want to clearly express some set of actions

(there are many other reasons as well!)
]

---
class: bg-main1

# Do you see any problems in the code below?

```{r scrape-problems, eval=FALSE}
st_episode <- st %>%
  html_nodes(".np_right_arrow .bp_sub_heading") %>%
  html_text() %>%
  str_replace(" episodes", "") %>%
  as.numeric()

got_episode <- got %>%
  html_nodes(".np_right_arrow .bp_sub_heading") %>%
  html_text() %>%
  str_replace(" episodes", "") %>%
  as.numeric()

twd_episode <- got %>%
  html_nodes(".np_right_arrow .bp_sub_heading") %>%
  html_text() %>%
  str_replace(" episodes", "") %>%
  as.numeric()
```

---
class: bg-main1

# Inputs

.huge[
How many inputs does the following code have?
]

```{r example-scrape, eval=FALSE}
st_episode <- st %>%
  html_nodes(".np_right_arrow .bp_sub_heading") %>%
  html_text() %>%
  str_replace(" episodes", "") %>%
  as.numeric()
```

---
class: bg-main1

# Turn the code into a function

.vlarge[
Pick a short but informative **name**, preferably a verb.
]

<br>
<br>
<br>
<br>

```r
scrape_episode <- 
  
  
  
  
  
  
```

---
class: bg-main1

# Turn your code into a function

.vlarge[
- Pick a short but informative **name**, preferably a verb.
- List inputs, or **arguments**, to the function inside `function`. If we had more the call would look like `function(x, y, z)`.
]

<br>

```{r example-scrape-formals, eval=FALSE}
scrape_episode <- function(x){
  
  
  
  
  
}  
```

---
class: bg-main1

# Turn your code into a function

.vlarge[
- Pick a short but informative **name**, preferably a verb.
- List inputs, or **arguments**, to the function inside `function`. If we had more the call would look like `function(x, y, z)`.
- Place the **code** you have developed in body of the function, a `{` block that immediately follows `function(...)`.
]

```{r first-scrape-episode-fun}
scrape_episode <- function(x){
  x %>%
    html_nodes(".np_right_arrow .bp_sub_heading") %>%
    html_text() %>%
    str_replace(" episodes", "") %>%
    as.numeric()
}
```

---
class: bg-main1

# Turn your code into a function

```{r first-scrape-episode-fun-print}
scrape_episode <- function(x){
  x %>%
    html_nodes(".np_right_arrow .bp_sub_heading") %>%
    html_text() %>%
    str_replace(" episodes", "") %>%
    as.numeric()
}
```

```{r scrape-episode-st-show}
scrape_episode(st)
```


---
class: bg-main1

# Check your function

.huge[
- Number of episodes in The Walking Dead
]

```{r scrape-epi-twd}
scrape_episode(twd)
```

.huge[
- Number of episodes in Game of Thrones
]

```{r scrape-epi-got}
scrape_episode(got)
```

---
class: bg-main1

# Naming functions (it's hard)

.huge[
> "There are only two hard things in Computer Science: cache invalidation and naming things." - Phil Karlton

]

---
class: bg-main1

# Naming functions (it's hard)

.huge[
- Names should be short but clearly evoke what the function does
]

--

.huge[
- Names should be verbs, not nouns
]

--

.huge[
- Multi-word names should be separated by underscores (`snake_case` as opposed to `camelCase`)
]

---
class: bg-main1

# Naming functions (it's hard!)

.huge[
- A family of functions should be named similarly (`scrape_title`, `scrape_episode`, `scrape_genre`, etc.)
]

--

.huge[
- Avoid overwriting existing (especially widely used) functions (e.g., `ggplot`)
]

---
class: bg-main1

# Scraping show info

```{r scrape-show-info-fun}
scrape_show_info <- function(x){

  title <- x %>%
    html_node("#title-overview-widget h1") %>%
    html_text() %>%
    str_trim()

  runtime <- x %>%
    html_node("time") %>%
    html_text() %>%
    str_replace("\\n", "") %>%
    str_trim()
  
  genres <- x %>%
    html_nodes(".txt-block~ .canwrap a") %>%
    html_text() %>%
    str_trim() %>%
    paste(collapse = ", ")
  
  tibble(title = title, runtime = runtime, genres = genres)
}
```

---
class: bg-main1

```{r scrape-all-show-info}
scrape_show_info(st)
scrape_show_info(twd)
```

---
class: bg-main1

# How to update this function to use page URL as argument?

```{r scrape-show-info-examle, eval=FALSE}
scrape_show_info <- function(x){

  title <- x %>%
    html_node("#title-overview-widget h1") %>%
    html_text() %>%
    str_trim()

  runtime <- x %>%
    html_node("time") %>%
    html_text() %>%
    str_replace("\\n", "") %>%
    str_trim()
  
  genres <- x %>%
    html_nodes(".txt-block~ .canwrap a") %>%
    html_text() %>%
    str_trim() %>%
    paste(collapse = ", ")
  
  tibble(title = title, runtime = runtime, genres = genres)
}
```

---
class: bg-main1

```{r scrape-show-info-add-bow}
scrape_show_info <- function(x){
  
y <- bow(x) %>% scrape()

  title <- y %>%
    html_node("#title-overview-widget h1") %>%
    html_text() %>%
    str_trim()

  runtime <- y %>%
    html_node("time") %>%
    html_text() %>%
    str_replace("\\n", "") %>%
    str_trim()

  genres <- y %>%
    html_nodes(".txt-block~ .canwrap a") %>%
    html_text() %>%
    str_trim() %>%
    paste(collapse = ", ")
  
  tibble(title = title, runtime = runtime, genres = genres)
}
```

---
class: bg-main1

## Let's check

```{r set-urls}
st_url <- "http://www.imdb.com/title/tt4574334/"
twd_url <- "http://www.imdb.com/title/tt1520211/"
```
--

```{r use-urls-scrape}
scrape_show_info(st_url)
scrape_show_info(twd_url)
```

---

class: center, middle

# Automation

---
class: bg-main1

.huge[
- You now have a function that will scrape the relevant info on shows given its URL. 
- Where can we get a list of URLs of top 100 most popular TV shows on IMDB? 
- Write the code for doing this in your teams.
]

---
class: bg-main1


```{r get-urls}
urls <- bow("http://www.imdb.com/chart/tvmeter") %>%
  scrape() %>%
  html_nodes(".titleColumn a") %>%
  html_attr("href") %>%
  paste("http://www.imdb.com", ., sep = "")
```

```{r print-urls, echo=FALSE}
urls
```


---
class: bg-main1

# Go to each page, scrape show info 

.vlarge[
- Now we need a way to programatically direct R to each page on the `urls` list and run the `scrape_show_info` function on that page.
]

```{r scrape-each}
scrape_show_info(urls[1])
scrape_show_info(urls[2])
scrape_show_info(urls[3])
```

---
class: bg-main1

# Go to each page, scrape show info 

.huge[
In other words, we want to **map** the `scrape_show_info` function to each element of `show_urls`:
]

```{r map-scrape, eval = FALSE}
top_100_shows <- map_df(urls, scrape_show_info)
```

.huge[
- This will hit the `urls` one after another, and grab the info.
]


---
class: bg-main1

# Passing functions to ... functions?

.huge[
- The fact that we can pass a function to another is a **big idea**, and is one of the things that makes R a **functional programming language**.
- It's a bit mind-bending, but it's an idea worth practicing and comfortable with
]

---
class: bg-main1

# aside: `list`s as an idea: first...vectors

.huge[
- `c()` creates a **vector** of one type
- e.g., `x <- c(1, 2, 3, "A")` contains:
- `[1] "1" "2" "3" "A"` 
- `class(x)` returns:
- `[1] "character"`
]

---
class: bg-main1

# aside: `list`s as an idea: first...vectors

.huge[
- You can look up vectors based on position with `[]`
- `x[1]` returns the first thing
- `x[2]` returns the second thing
- `x[1:2]` returns the first through to second thing
]

---
class: bg-main1

# aside: `list`s as an idea: second...lists

.huge[
- `list()` creates list, which can be any type

```r
y <- list(1,2,3,"x"); y
#> [[1]]
#> [1] 1
#> 
#> [[2]]
#> [1] 2
#> 
#> [[3]]
#> [1] 3
#> 
#> [[4]]
#> [1] "x"
```
]

---
class: bg-main1

# aside: `list`s as an idea: second...lists

.huge[
- You access positions of a list with `[[]]`
- So `y[[1]]` returns: `1`
]

---
class: bg-main1

# aside: a `data frame` is actually a list!


---
class: bg-main1

# calculate the mean for every column:

```{r map-mean}
map(mtcars, mean)
```

---
class: bg-main1

# calculate the mean for every column:

```{r map-dbl-mean}
map_dbl(mtcars, mean)
```


---
class: bg-main1

# Range for every column: writing a function

```{r map-dbl-mean-fun}
my_range <- function(x){
  max(x) - min(x)
}
map_dbl(mtcars, my_range)
```


---
class: bg-main1

# Range for every column: writing a function in map

```{r map-dbl-mean-inline-fun}
map_dbl(mtcars, .f = function(x) max(x) - min(x))
```

---
class: bg-main1

# Range for every column: writing a function in map

```{r map-dbl-mean-lambda}
map_dbl(mtcars, .f = ~(max(.) - min(.)))
```

---
class: bg-main1

# Your Turn: rstudio.cloud

.huge[
Take the lab quiz!
]

---
class: bg-main1

# Resources

.huge[
- Jenny Bryans blog post
- functions chapter of r4DS
- iteration section of r4ds
- [lists section in advanced R](https://adv-r.hadley.nz/vectors-chap.html#lists)
]

---
class: bg-main1

# Share and share alike

<a rel="license" href="http://creativecommons.org/licenses/by/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.
