A frequent problem in statistics is deciding if two or more samples of data come from the same or different populations. Perhaps the context is trying to decide if a particular medical treatment is effective, or if a change to some industrial process has reduced the production of pollution, or maybe even if a change in teaching methodology has led to improved student performance? In each of these cases, you would need to interpret any change between before and after measurements, and see if there has been a significant change in outcome.

This is not always easy because of inherent randomness and errors in measurement. This is called noise. The techniques of Exploratory Data Analysis are designed to bring out the important characteristics of data, prior to resorting to in-depth quantitative analysis and modelling.



Exploring Data
A picture can tell a thousand words
The table below contains three sets of data, each giving a y value for a particular x. Considering these as three independent measurements, you can apply the following 3 step approach to attempt to understand and interpret these data:
  1. Look at the values and see how much you think they look the same,
  2. Use the buttons below to calculate some summary statistical values,
  3. Plot the data as points (the plot buttons will appear once all the summary statistics have been calculated).
Consider how your understanding of the data changes with each step.

Finally, enter your own values into the 4th column of the table. The average and standard deviation will be calculated automatically. Edit your values and see how the summary statistics change. You can also drag the points on the associated plot to adjust the values.

xy1y2y3my data

About the activity

It is quite remarkable that in many situations improperly understood data leads to wrong conclusions, or considerations of the same data lead to opposite conclusions (and consequently warring factions). This is because statistics is not exact, and interpretation plays a significant role. Developing good data analysis skills helps recognise and avoid problems such as these.

The initial proponent of Exploratory Data Analysis was the mathematician John Tukey. One of the tools he invented for the purpose was the Box-Plot (see the Understanding Outliers activity). However, he is most famous as one of the inventors of the Fast Fourier Transform, thought by many to be the most important computational algorithm of the 20th century.

A Fourier Transform takes an image or a signal and tells and breaks it into its frequency components. That is, it tells us how much high frequency, or fast changing parts, there are, and how much low frequency. Unfortunately, a direct calculation of a Fourier Transform gets extremely slow as the amount of data increases. In fact just 10 times as much data means 100 times longer to calculate. This quickly becomes impossible—even for computers.

The Fast Fourier Transform gives a better way to calculate a Fourier Transform, and is in general about 300 times faster. This reduces a computation time of 5 hours down to just 1 minute, and clearly makes a huge difference in our ability to apply this fundamentally important kind of analysis in technology.