What is Exploratory Data Analysis

• EDA is taking a preview of the data we will analyze
  • Small or large datasets
• Engineering Statistics Handbook
  • “EDA is an approach/philosophy for data analysis that employs a variety of techniques (mostly graphical) to”
    1. maximize insight into a data set
    2. uncover underlying structure
    3. extract important variables
    4. test underlying assumptions
    5. develop parsimonious models
    6. determine optimal factor settings

*http://www.itl.nist.gov/div898/handbook/eda/section1/eda11.htm*

Focus

• EDA is an approach
  • Not a set of techniques
  • It’s about how a data analysis should be done

Philosophy

• Not only statistical graphics
• Approach to data analysis that postpones usual assumptions about the kind of model that the data follows
• It allows data itself to reveal its underlying structure and model
• Not a collection of techniques but a philosophy about how to dissect a data set
  • What we look for
  • How we look
  • How we interpret
• For this, we use a set of techniques, many of them are statistical graphics

History

• Seminal work in EDA http://www.dtic.mil/dtic/tr/fulltext/u2/a266775.pdf
• Data Analysis and Regression, Mosteller and Tukey, 1977
• Interactive Data Analysis, Hoaglin, 1977
• The ABC’s of EDA, Velleman and Hoaglin, 1981

Show the way a data set should be analyzed

Techniques

• Mostly graphical but also quantitative techniques
• Graphical because main role of EDA is open-mindedly explore
  • Graphics good for this
  • Discover data structural secrets
  • Get new and maybe unsuspected insight into data
• Combine graphics with natural pattern recognition

Techniques

• Some techniques
  • Plotting raw data
    • data traces
    • histograms
    • bihistograms
    • probability plots
    • lag plots
    • block plots
    • youden plots

Techniques

• Some techniques
  • Plotting simple statistics
    • mean plots
    • standard deviation plots
    • box plots

Techniques

• Some techniques
  • Position plots to maximize our natural pattern-recognition abilities
    • Multiple plots per page
    • Place plots comparing features

Exploratory Data Analysis Checklist

1. Formulate your question
2. Read your data
3. Check the packaging
4. Run str()
5. Look at the top and botton of data
6. Check your “n”s
7. Validate with at least one external data source
8. Try the easy solution first
9. Challenge your solution
10. Follow up

this checklist by by Roger D. Peng

Formulate your question

• Our question would be a guide to the EDA process
• Limit the branches it could take
• A sharp question could help to reduce dimensions
  • Variables not immediately relevant to the question

Formulate your question

Some common questions from the Engineering Statistics Handbook (http://www.itl.nist.gov/div898/handbook/eda/section3/eda32.htm)

1. What is a typical value?
2. What is the uncertainty for a typical value?
3. What is a good distributional fit for a set of numbers?
4. What is a percentile?
5. Does an engineering modification have and effect?
6. Does a factor have an effect?
7. What are the most important factors?

Formulate your question

Some common questions from the Engineering Statistics Handbook (http://www.itl.nist.gov/div898/handbook/eda/section3/eda32.htm) Continues…

8. Are measurements coming from different laboratories equivalent?
9. What is the best function for relating a response variable to a set of factor variables?
10. What are the best settings for factors?
11. Can we separate signal from noise in time dependent data?
12. Can we extract any structure from multivariate data?
13. Does the data have outliers?

References

1. Engineering Statistics Handbook, “Exploratory Data Analysis”, (http://www.itl.nist.gov/div898/handbook/eda/eda.htm), last visited: January 27, 2016.
2. Roger D. Peng, Exploratory Data Analysis (https://leanpub.com/exdata), Lean Pubs, 2015.