I was handed a dataset in my PhD that had more than 65% of the data missing. We had to spend quite a bit of time exploring the missingness relationships and the whole process was very frustrating, this inspired me to develop methods for exploring missing data, which ended up being a substantial part of my thesis

But today we are going to talk about some example weather data from Melbourne.

But, like all good explanations, this one is simple - but the process to get there, where we describe it, probably was not.

In order to get to a position where you could generate this graphic, and show this data, you probably had to spend more time than you would have liked developing exploratory data analyses and models

It might seem obvious to point out the precise mechanism for generating the missingness, but this is kind of difficult when there is a lot of missing data, or when there are many variables.

Before we get started, we need to define missing values

Missing values are values that should have been recorded, but were not.

Think of it this way:

You might accidentally not record seeing a bird - this is a missing value. This is different to recording that there were no birds observed.

R stores missing values as NA, which stands for Not Available.

Missing values don't jump out and scream "I'm here!". They're usually hidden, like a needle in a haystack.

To detect missing values use any_na, which returns TRUE if there are any missings, and FALSE if there are none.

are_na asks "are these NA?" and returns TRUE/FALSE for each value

are_na shows us 3 TRUE values - 3 missing values.

To avoid counting each TRUE yourself, n_miss counts the number of missings

And prop_miss gives the proportion of missings, which gives important context: 50% of data is missing!

So what happens when we mix missing values with our calculations? We need to know what happens, so we can be primed to find these cases. The general rule is this:

Calculations with NA return NA.

Say you have the height of three friends: Sophie, Dan, and Fred.

The sum of their heights returns NA,

This is because we don't know the sum of a number and NA.

Now that you understand what missing values are, how to count them, and how they operate, let's scale these up to more detailed summaries of missingness.

We need to summarise missing data to identify variables, cases, or patterns of missingness, as these can bias our data analysis.

There are two main summaries: basic, and dataframe summaries.

Basic summaries return a single number, like the number of missing or complete values using n_miss or n_complete.

However, you will need more detailed missingness summaries to help you on your journey through a data analysis.

This lesson introduces you to missing data summaries.

naniar provides a family of functions all starting with miss_., which each provide different summaries of missingness, and return a dataframe.

This allows us to see features that can be difficult to articulate, or time consuming to calculate.

For example, miss_var_summary and miss_case_summary return the number and percentage of missings in each variable or case.

These summaries work with dplyr''s group_by, so you can fluidly explore missingness by each groups.

Use miss_var_summary to summarise the number of missings in each variable.

This returns a dataframe where each row is a variable. It also includes summaries of the number and percentage of missings for each variable in the dataset, and is sorted by the number of missings.

For example, Ozone has 37 missing values, and is about 24.2 percent missing.

Similar to miss_var_summary, miss_case_summary returns a summary dataframe, where each case represents a dataset row number.

Here, case 5 - the fifth row in the dataset - has 2 missing values, which means 33% of that case is missing.

Tabulation of missingness counts the number of times there are 0, 1, 2, 3, and so on, missings. They are very useful, compact summaries that reveal interesting structure.

miss_var_table returns a dataframe with the number of missings in a variable, and the number and percentage of variables affected.

For example, there are four variables with no missings detected, which corresponds to 66.7 percent of variables, and there was 1 variable with 7 missings, and 1 variable with 37 missings.

Similarly, miss_case_table returns the same information, but for cases.

Sometimes you are interested in missingness for groups in the data.

Each missingness summary function can be calculated by group, using group_by from dplyr.

For example, we can look at the missingness by Month in the airquality dataset.

Here we see that Month 5 for Ozone there are 5 missings, but for Month 6 Ozone has 21 missings.

We cover how to get a bird's eye view of the data, how to look at missings in the variables and cases, and how to generate visualizations for missing spans and across groups in the data.

We now know what missing values are, how they work, how to count and summarise them - now let's look at some of the built-in visualisations that come with naniar.

Data summaries are very useful, but sometimes an idea or a thought can be quickly captured with a visualisation.

naniar provides a friendly family of missing data visualisation functions, each presenting different visualisations missingness summaries.

In fact, each of these visualisations is a nice compact shorthand for the data summaries. While you could create similar and more complex visualisations using the summary information from the previous lesson, this can be repetitive. The visualisations in naniar reduce repetition and increase iteration, so you can operate closer to the speed of thought.

When you first get a dataset, it can be difficult to get a visceral sense of where the missings are.

To get an overview of the amount of missingness, use the vis_miss function from the visdat package.

vis_miss produces a "heatmap" of the missingness - like as if the plot corresponded to the dataset as a giant spreadsheet, with values coloured black for missing, and grey for present.

vis_miss also provides missingness summary statistics, showing the overall percentage of missingness in the legend, and the amount of missings in each variable.

These can be turned off in its options, described in the helpfile.

vis_miss also allows for clustering of the missing data by setting cluster = TRUE: this orders the rows by missingness to identify common co-occurrences.

To quickly show the missingness in variables and cases, we visualise them using gg_miss_var and gg_miss_case. Note that these are visual analogues of the miss_var_summary and miss_case_summary functions.

These plots show the amount of missingness on the x axis, and for gg_miss_var, each point represents the amount of missingness in that variable, and for gg_miss_case, each line represents the amount of missingness in that case.

Note that these visualisations are ordered so that the most missing is at the top. The ordsering in gg_miss_case can be turned off with option, order_cases = FALSE.

gg_miss_var and gg_miss_case also allow for facetting by one variable.

This means you can explore missingness in cases and variables across the levels of another group.

This plot is facetted by month, showing the number of missings in each variable for each month.

Here we see that Ozone in Month 6 has the most missings.

To visualise the common combinations of missingness - which variables and cases go missing together, use gg_miss_upset.

This powerful visualisation shows the number of combinations of missing values that co-occur.

An upset plot of the airquality dataset shows there are only missing values in Ozone and Solar.R, with 35 in only Ozone, 5 in Solar.R, and in both Ozone and Solar.R, there are 2 missing cases.

To explore how missingness in each variable changes across a factor, use gg_miss_fct.

This displays a heatmap visualisation showing the factors on the x axis, each other variable on the y axis, and the amount of missingness coloured from dark purple to yellow.

gg_miss_fct does not support facetting.

• Provides a consistent way to set the existing location of missing values.

• This has additional great benefits when combined with imputation

• Also allows for additional visualisations

One approach, (ggobi) is to shift missing values below the minimum value

This then means that they can be plotted on the same axis.

Typically, when exploring this data, you would do something like this:

The problem with this is that ggplot does not handle missings be default, and removes the missing values. This makes it hard to explore the missing values.

Instead, we have create a ggplot2 geom, geom_missing_point().

This geom allows for missing values to be displayed, and also works with the rest of ggplot2 - themes, and facets as well.

What do we learn?

• Two variables, air temperature and humidity, have a large number of missings.
• Year, latitude and longitude have no missings
• Sea temperature has a couple of missings
• Some rows have a lot of missings

Overall statistics, 3% of possible values are missing. That's not much. BUT, both air temperature and humidity have more than 10% missing which is too much to ignore.

This type of display is called a "heatmap", displays the data table, with cells coloured according to some other information. In this case it is type of variable, and missingness status. What do we learn?

• Most of the variables are character (text) variables, or integer variables.
• Missing values are located only in the counts, but it is in blocks, so perhaps corresponds to some category levels of the other variables.

and this is a problem, because results computed on data with missing values might be biased. for example, ggplot ignores them, but at least tells you its ignoring them:

What do we learn?

• There is a different missingness pattern in each of the years
• Year needs to be accounted for in finding good substitute values.

What do we learn?

• Oh, this is so wrong!
• The imputed values are nothing like the complete case values

What do we learn?

• The imputed values are closer to the complete case values
• However, they form a rigid line, mismatching the variation
• and they extend outside the range of complete values. This is a problem in that the imputed air temperature value for these high sea temperature cases is lower than we would expect, and thus possibly impeding good model fitting

Two minute challenge

• Change the code to plot sea temperature against humidity, with colour representing missing humidity values. What do you learn about the imputations?

Relationship with other variables

ggplot(data = tao_shadow,
       aes(x = wind_ew, 
           y=wind_ns, 
           colour=air_temp_c_NA)) +
       scale_colour_brewer(palette="Dark2") +
       geom_point(alpha=0.7) + theme(aspect.ratio=1)

What do we learn?

• The lowest values of east-west winds have no missing values. Maybe it is less likely to have air temperature missing values when there are light east-west winds?

Two minute challenge

• Generate the shadow matrix and make a plot of the winds, coloured by missingness on humidity.