Welcome to the video,
this video belongs to BLOCK 5 Probability and Statistics
we will see the difference between
observation, data, inference, etc.
We will see measures of central tendency, measures of variability,
understanding, variability in data, and the link to the practical video
that contains charts that are useful for univariate
descriptive statistics.
So, terminology.
Analyzing data has a long history,
but in fact, nowadays it's related to statistics,
to artificial intelligence, and machine learning, and data analytics
The terminology that we are using nowadays, it is Data Science.
Is the same big data and data science?
Okay I wouldn't say this, but in fact
many of the techniques are the same.
But exactly is not, so
we don't get into details.
But in fact, let's take data analysis,
how to understand data.
And for students, what we have here, it is the software.
Absolutely free that are R and RStudio.
Then, we have Python; again
the same with many libraries related to
statistics, basic statistics and data analysis.
And we have very nice repositories around the world.
Okay,
The taxonomy
of data analysis and their statistical methods,
can be basically divided
into descriptive methods and inferential methods.
We will see the difference between descriptive and inference.
Then, in the part of descriptive descriptive methods,
we have univariate methods, bivariate
for two variables simultaneously, and multivariate methods.
So, in the inferential part: inferential methods,
we won't see anyone in this slides,
but they will come in the first course
about data analysis.
So, first of all, description of data.
Let's take one variable one variable each time
and in the case of numeric variable, what with what we see,
or what we do, it's to obtain a measure of the center of the variable
and also the variability; sometimes connected to what is called the scale.
Then we have shape.
This is the shape, the profile of the distribution.
And this is very, very related to what is called in
theoretical statistics: moments,
raw moments and centered moments.
For describing a categorical variable, a categorical set of data.
In fact, we have charts.
The frequency of each category or the percentage
of each category.
First of all, let's take a look at what it is called
in the R domain: the data frame.
A data frame is a bidimensional
structure containing rows and columns.
The rows represent individuals,
instances sometimes,
and for each individual we have many characteristics.
These characteristics are called attributes,
and the attributes constitute
the columns of this data frame.
Okay, so the terminology varies,
depending on the field; in machine learning, it is not the same
as in traditional statistics, but we have rows and columns,
so instances and variables (or features), depending on the domain.
Okay.
So, taking into account the possibilities
in qualitative variables, categorical variables, it is the same.
We have binary variables.
This is, two labels, two categories. Nominal is called, in general
when we have more than two labels, two categories.
And we have also the case where the labels
have a certain order, in the certain order
we have the ordinal qualitative variables
In the domain of numeric or continuous variables,
we have counts. Counts are discrete data,
but in fact, this discrete data has a meaning as a number.
So, we have the count of words in a sentence;
the number of telephone calls, etc.
And then we have intervals, for example, intervals
in temperature, or just proportions
in ordinary continuous data.
The possibilities are also enhanced
by different and extra data types and also the possibilities
of including multiple response in, for example,
in surveys and the questions that we address in the questionnaire.
But this is not the work topic, including textual data.
So what is data analysis?
Data analysis is just the process of making sense of the data.
So giving meaning to the data, that's all
and I think this summarizes quite well
the goal of this video.
The data corresponds to, in general,
to a sample of a given population.
And in fact, we analyze data from a sample.
But then the process of inferring
from the sample, the parameters in the population,
this is what we call inference: an inferential process.
Once said this,
let's focus on exploratory data analysis for numeric variables,
and we have an important reference
Dr. Tukey.
Tukyey worked for the Bell Laboratories.
He was the one to introduce
the basic techniques that it is the five numbers summary.
He also introduced, explored on some other charts that are useful.
The five numbers summary has been extended
to 6 in the basic R summary,
but roughly it's the same.
So, exploratory data analysis, let's
see numerical summaries.
Numeric representation.
First of all, we have to focus on the central
value that summarizes the set of observations.
The set of observations is for one column in the data frame and
we would like to have the central value.
So this is a summary and usually the summary
is undertaken with mean or median.
Okay, but this is not enough.
We have to enhance the set of summary numbers
to include some measures of data dispersion.
Data dispersion is how the data
is spread around the mean.
So, this is quite important
and we quantify the average distance to the mean.
We will see how,
For example, to understand a little bit the concept of data variability,
ee have two data sets, A and B,
so A, has three three units.
Okay.
The mean is 50.
In B, the mean is 50 too.
But, in fact, if we pay attention,
the spread of the data in dataset
A is greater than the spread of the data in data set B
because the distance of the observations to the mean
in the case of B is less than the distance
to the mean in dataset A.
So, this has to be quantified
because it is an important concept that complements the central value.
Let's focus first of all on the central value.
The center measured, the natural one is the mean.
So the mean in the sample;
in data for a column in the data frame.
So for us, this is the sample of a of a numeric column.
Then the mean is just adding up and dividing into the number of observations.
So, this can be written in a formal way as let x1, x2, ...xn,
and that are observations for a given variable and variable is notated as x.
So the mean of this variable that is notated as x bar
is just
adding up from the first to the end and dividing
into the number of observations.
The median is the next concept.
Median is the middle value
in an ordered sequence of observations.
So for example, what we have in the case of an even number
of observations, what we do is we take just
the mean of the two central observations.
Let's see this with an example that it's very easy.
We have a set of five numbers and first of all, what we have to do
is just order from the first to the last in a increasing order.
So, in this case we have an odd number of observations.
That is 5.
So the central value is 6.
I have to pay attention to the order of the sequence and the midpoint
that it's 6, it's observation number three, and this is the median.
In the case of
a sequence that has 6 observations, then I repeat the process.
So order the sequence, I have 6 numbers.
And the central point is in fact two points that are 5 and 6 values.
So I do the average of both
and we get that the median is 5.5.
Mode is not very useful in many applications in engineering,
but in fact is the value that it's observed at most frequently
that's all.
Let's extend
a little bit the concept of median.
The median is very important because it is a robust statistic.
It means that is less sensitive to outliers. Outlier is a concept
that is critical to understand
data analysis, the basic data analysis.
Let's see, for example, this sequence of five
four sorry, four numbers that is 20, 30,
40 and 990.
So, the mean is 270.
But in fact, the median would take the two central points.
So it's the average between 30 and 40 is 35.
So, in fact the median indicates
the central value of most of the observations.
In this case, most is just a small sample because this is an example.
But what we refer to robustness
is that the mean is biased by the large values.
And in this case I have a large value that is an extreme value,
what we will call an outlier.
Okay, but but in fact, the median is not affected
Variability.
The common statistics for variability
are range, variance, standard deviation, interquartile array, etc.
Range is very easy, it's just
the difference from the minimum to the maximum.
So, this is the range of values. it is a crude measure.
The second one is a traditional, traditional statistic
that it's variance. In this case, what we do is just to add up the squared differences
from the observations to the central point,
that it's the mean,
so this it's divided into, because it's an average,
by n or n - 1
in fact, n-1 it's what is returned by most of the statistical softwares
and in fact, it's the unbiased estimator.
So, what's the variance of this sequence?
5. 7 3. The mean is 5, the variance is applied.
In the formula that you see above.
So the variance is 4.
And the problem with the variance it's that it has a squared units
with respect to the values of the variable.
So, it's very common to produce
the squared root of the variance and this is the statistic
S, standard deviation.
Okay, so in the above example,
it would be the squared root of 4 that is 2.
Quartiles. Quartiles is based on
dividing the range of values into four parts.
And, this is the cut points at the quartiles.
The notation is a little bit
tough to see, but the example it's very easy.
So, the first quantile is Q1 and
it indicates the first 25% of the data.
So, it's the point that 25% of the data is less than Q1
and 75% is greater than Q1.
The second quartile is the median.
So, this is, this spreads the range of values into two parts.
So, median is the central part,
the central tendency, the third quartile is Q3.
Only 25% of the data in the column is greater than that value,
Q3. So, between Q1 and Q3,
we have 50% of the sample.
Let's do this with an
example, because it's the best way to to proceed.
Let's assume that we have 15 numbers, the ones that you see, just
right now on the screen and
the 15 numbers.
So, what we have to do is using the formula 15 + 1,
the number of observations + 1 divided into 4.
This is 4.
So, we have to divide the
the sample into four parts.
And the first quartile would be 11,
the second 40 and the third quartile 61.
Okay.
It's just a question of order and counting.
So observation number 4, number 8 and
number 12. This is.
So, the interquartile range
is the difference between Q1
and Q3, in fact it's Q3,
the 75% percentile
minus the 25%
percentile. In this example, it would be 61 - 11.
That it would be 50.
Deciles.
We extend the idea
of the quartiles, to the percentiles and the deciles.
Deciles would be to divide the sample into ten parts and
the percentiles would be to divide
the sample into 100 parts.
This is the most common, and more general point of view.
And this is the reason that Q1, the first quartile is
in fact the 25% percentile.
Okay.
And Q3, for example, it's the 75% percentile
In notation,
it can be written, but it is, again, a little bit hard
and in fact, it takes the number of observations plus one.
In order to finish this summary, we have the coefficient of variation
that it's the ratio between the standard deviation
of one column, numeric column, divided into the mean of the column,
usually multiplied by 100 to give us a percentage.
Okay.
So, descriptive versus inferential statistics.
The difference is that if
we have medians, we have means, etc.
but this does not generalize to anything
because it's just a summary of what we have in the sample.
But, this is the reason that we talk about exploratory data analysis
in the case, of inferential analysis,
what we do is to state hypothesis
and this hypothesis enabled
to make inferences about the population parameters.
So. In fact, the population is just
where the sample of data belongs to;
it can be a random draw or not, obtained with any technique.
Okay,
I think we have
seen everything for the summary of numbers.
And in the companion video, I will show you the chart types.
Thank you very much for your attention.