Welcome to BLOCK 5: Statistics.
This is the companion video for understanding
and interpreting variability in data.
We will see the outlier concept. In practice,
supported by chart types that are useful
for the univariate analysis of data.
Okay.
So, data analysis
has to be the own using software.
There are many statistical packages, but the one that we will be
using in data analysis, in the Master of
Urban Mobility will be R and R language.
R language is free and it's an extension and evolution
of S plus language that was developed at Bell Laboratories
and in fact it's a reference
language and environment for the statisticians with many,
many probability distribution, many possibilities for statistical
modeling and it's extremely useful.
Let's take a look
at the chart types for numeric characteristics.
Okay.
So, we have seen that numeric numbers to summarize
the characteristics of a numeric variable
are mean, median, mode, variance, etc..
Okay. This is in the case.
The variance is a measure of dispersion, but the central trend is, mean and median.
Now, we have to see the basic plots, the basic charts that are useful,
and I will focus on the dotplot,
histogram and box plot,
because box plot is a very useful tool to detect extreme observations.
This is what we call outliers. Okay?
We will see the possibilities using the base library.
So, the basic library, the default installation in R, and
in some cases, I will show you how to enhance the graphics
using the ggplot2 library.
Ggplot2 is a language for graphics and you can
produce very, very nice graphics.
So let's see, this is a numeric characteristic
and in this case it is the consumption mpg of a car.
Here, we have a boxplot, sorry a dot plot. This chart shows a dot plot
and on the left, the dot plot for the base library in R.
So, what we see is on the X-axis
has the values of mpg, that is a numeric variable.
And then, we have dots
that are shown in vertical, increasing in vertical,
indicating the concentration and the number of observations.
In fact, we can use on the
Y-axis proportions, or just absolute numbers.
Okay.
In the case of ggplot.
So, the nicest way to show
this variable, it is on the right. I have not done
too many nice arrangements to
to customize these this geom_dotplot.
But, here can you see the same
and of course the value for each dot
depends on what you define in the specification.
In this case, as you see, it's nicer, but, it is the same concept.
Roughly what we have is a profile.
So, this is very useful when we have data
and the data is of discrete
type, but with many different values.
Okay.
So, sometimes
when we have these
data, it's convenient to add some sort of noise
to avoid the overlapping in the plots.
So, this is usually added
to the basic chart and it's 'to jitter'.
I'm not going to extend too much about this,
but in the stripchart method in the base library.
We can add the method jitter just to add some noise at the Y-axis.
Okay.
In ggplot tool, it's exactly the same.
We have sometimes
an interesting graphic that it's based on error bars.
So, this summarizes
some sort of confidence interval for a given measure.
For example, the mean.
It is usually the mean.
So this chart is a complement
of graphics, of some other graphics
usually histograms or in the discrete case, bar plots.
Now let's focus
on the most important or I don't know
if the first or the second best, chart
for the description of numerical data.
This is the popular box plot.
Box plot is in fact the combination
of summary of numeric parts, numeric data
and also this chart. We have a central box
and this central box is divided
by the median, on the left hand side.
If we take a look at the horizontal graphic that I show you on the left,
then the left hand side of the
of the box is defined by Q1.
So, the first quartile, and the right side is defined by the third quartile.
That it's the 75% percentile.
Okay.
And then we have some whiskers.
These whiskers take us from the extremes
of the sides of the boxes
to the minimum and the maximum.
Okay.
So, the important thing to highlight is that
the width
of the box is the interquartile range.
So, the difference between Q3 and Q1.
Okay.
So, those observations
that are outside the region of the whiskers.
So, in fact, these whiskers are taking us to
to define an area, a range of values.
Then those values that are outside the range of the whiskers are outliers.
Okay.
So, these outliers are extreme observations
because they are too big or too small
Detection. Applied
detection is extremely important in data analysis.
In fact, from a practical point of view, if we have to make
an assessment of data quality, we will see many problems with data.
For example, redundancy, inconsistencies
in the way of writing, addresses, names, etc..
We see also when combining different
sources, the application dependency.
So, this is from the computer
science point of view, but from the statistical point of view,
we will see also errors,
incorrect data, we will see outliers.
So, outliers are unusual values
because they are too big or too small and missing data.
Missing data is a nuisance.
This is a serious problem that can bias
the study or the study that we are
addressing after exploratory data analysis.
So, from a point of view of low quality,
all these problems, all these issues
can contribute. From the point of view of statistics,
we can treat outliers and missing data.
Okay.
Addressing missing data is not easy, but
addressing univariate outliers is very easy
taking into account the box plot tool.
So, let's see the context
of outliers, according to Douglas Hawkins.
An outlier is an observation which deviates so much
from the other observations
as to arouse suspicious
that it was generated by a different mechanism.
So, in this case, an observation.
Let's see. Univariate.
I'm assuming that it is generated
a random sample belonging to the standard normal
distribution, and we see the red points
that are shown in the density chart.
So in this case, these observations are too high.
The value is around seven.
Sorry, it's five, six and taking into account
that this is the standard normal distribution.
This data is extreme.
The probability is very low.
So, in this case, these would be outliers.
Okay.
In the definition and treatment of outliers,
we have to take into account two type of outliers, mild outliers
that are the ones that in the chart that I'm showing you.
In the case of horizontal chart, you will see the circles
that we have one circle,
circles in red and circles in white.
Okay, so, Mild outliers are circles in white.
An extreme outliers.
According to the name, they are far away and they are in red.
Okay.
So, an observation is clearly an extreme outlier.
If it is,
it lies outside of the interval
defined by three times the interquartile range
from the left of the box
or from the right of the box.
So, in this case, as we see
in the chart, we have the red values.
So, the mild outliers are the ones
that lie
between 1.5 and 3 times the interquartile range
from the extremes of the box that are Q1 and Q3.
Okay.
So, in this case, we have in between
what is normal
and what is extreme, we have the mild outliers.
Okay.
So, in the case, if you would like to know, in the case, of a standard normal
distributions, the probabilities for a
3 times or for extreme outliers, according to the definition,
are very low, from the probabilistic point of view.
Okay.
And it's of order ten minus six.
But in fact, this is just assuming that I'm
doing some sort of random
draw to define the column of data.
But, in fact, from a practical point of view, we have a column of data
and we have to detect those observations that are extreme, very small, or very large.
Next point,
histograms.
Histogram consists
of dividing from the minimum to the maximum
into intervals of equal length.
This is by default and in each interval
we count the number of observations.
So we have these bars
on the left, where you see this sort of bars.
And in fact, the terminology is bins.
Okay.
So, in this case, this is a counting.
And again, we have some sort of profile about the distribution.
The left chart that I show you is the chart
that has been produced using the basic library in R
And it is just the basic histogram.
Histogram has many parameters and some of these parameters
are described on the right chart. On the right chart,
the names of the limit for the Y variable, and the limits
and the name for the X variable
that it's the X-axis, the range of values.
And we see the name and the definition of
the bins as a number or just the intervals.
So, everything can be customized.
But this is quite important to use histograms
and sometimes a couple of parameters should be tuned
to obtain the best profile to understand the meaning of the data.
Histograms are very important for numeric variables.
Some statisticians prefer, instead of this,
a profile that it's described by the histogram
to use some sort of kernel
or local regression to define the density.
Okay.
This is in my opinion, a question of taste.
Sometimes counts, but it is the basic point.
First of all, we have to understand
and produce a histogram
and now we have to say something and move forward
in order to see the chart types
that apply for qualitative variables.
Qualitative or categorical variables, it's the same, synonymous.
Okay.
And we have the bar plot.
In the bar plot, clearly for each category,
in the example, we have some sort of sample that refers
to male and female,
and we have just the bar plot
where each of the bars
indicate, on the left,
Well, what we have is the counting.
So, the number of observations and on the right,
what they have is no..., this is exactly the same, the number of observations,
but it can be customized to to use the proportion, for example.
So ,this is for each category.
I have a bar indicating the absolute
or relative values in the sample.
If you
don't have too many categories, then it's useful
maybe to use instead of the basic
barplot, the pie chart, but the default pie chart in R
are not extremely nice, from my point of view.
Okay,
Now, let's see an example.
That is a practical example.
You're seeing R, RStudio.
So I'm taking this CPS1985 data set.
So this is a dataset that can be obtained just uploading the library
AER that stands for Applied Econometric Regression.
So, I upload this dataset.
This dataset contains around 10-12 variables.
So, in the columns and then the response variable,
it is the main variable that we are interested in.
This is the target variable, it's wage.
This is, how much money you earn per hour.
The quantity
is dollar per hour and they are quite old.
But anyway, what we have, first of all, it's
let's take wage
and summarize with summarize wage, using the summary method,
and the summary method is the 6-number summary
that I mention in the companion video, and that I mentioned before.
In this case, we have the minimum and the maximum.
Minimum is 1 and maximum 44.5$,
and we have the first quartile and third quartile. First quartile
is 5.2 and third is 11.2.
Okay.
And then the central, the central tendency, that it is median
almost 7.8
and mean 9, something above nine.
So, in this case, I see that median and mean
are not, roughly, the same quantity
and this means that my distribution,
and the profile of the data is asymmetric.
So, I have an assymmetry with respect
to the mean and this points,
given the mean is greater than the median,
to a skewness that it's positive.
So the coefficient of symmetry is positive, indicating a long tail to the right.
So, this is summary for the target.
If instead of summarizing with the 6-numbers,
according to the Tukey's terminology,
then we can use the quantiles,
but a particular set of quantiles, that are the deciles.
So, dividing the set of data into ten parts
and I have for example, in this case
I have 1, 4.98,... and so on.
Let's interpret,
for example, the 90% quantile,
the 90% percentile.
So, in this case is 15.275.
It means that
90% of the sample has a salary, a wage,
that is less than 15.275
and only 10% of the sample has a wage
that is greater than this quantity.
So, this is the meaning and it's very useful in order to see
how the values are distributed along the range.
The possible range.
Next one, I have the box plot.
Here, we have box plot, it is the chart on the left, it's a horizontal box plot
of wage and we have defined a line,
a vertical line at what it is three times the interquartile range
from the right side of the box in the box plot.
And we see this red line on the chart.
And in the chart, we determine, first of all, we have no problem
with lower value outliers.
So, the problem is on the upper outliers.
So extreme values, because they are too big.
In this case, we see the end of the whiskers,
and some dots in white indicating that, okay,
nothing to say because they are between the 1.5
and 3 times IQR, the interquartilar range.
So, nothing to say.
These are just mild outliers
and to the right of the red line,
what we have is one observation that is an extreme outlier.
So, $44.5
per hour is too much compared
to the range of the values for wage that we have in the sample.
Okay.
So this is an extreme outlier.
And finally,
we see the histogram,
a little bit customized with colors and in number of bins.
In this case, what we see is the profile of wage
indicating for each interval
the height of the bin, it is the frequency,
so the number of observations belonging to that interval.
In this case, what we see, as I suspected
before, it's an asymmetric distribution with a long right tail,
as it is shown in the histogram.
So, these are basic tools
in R language that are extremely useful
to summarize the basic statistics
and variability in data.
Thank you very much for your attention.