Data¶

Data can be anything. It depends on the data engineer on what the input and output data is

Data = results of measurement

• Definition of measurand (quantity being measured)
• Measurement value
• number

• unit

• Experimental context

• Test method
• sampling technique

• environment

• Estimate of uncertainty

• Measurement uncertainty: estimate of dispersion of measurement values around true value
• Context uncertainty: uncertainty of controlled and uncontrolled input parameters
• Metrology/Measurement model: science of measurement; theory, assumptions and definitions used in making measurement

Types¶

• Structured
• Numbers
• Tables
• Unstructured
• Audio
• Image
• Video

Datasets¶

Collection of data in rows and columns

• Rows = Objects, Records, Samples, Instances
• Columns = Attributes, Variables, Dimensions, Features

Types¶

• Labelled has Target variable
• Unlabelled does not have target variable

Data Collection¶

Stages¶

• Motivation
• Composition
• Collection process
• Labelling
• Preprocessing
• Uses
• Distribution
• Maintenance

Metadata¶

Garbage-in, Garbage-out

• Manual Labelling
• Manually marking as cat/not cat, etc.
• Observing Behaviour
• taking data from user activity and seeing whether they purchased or not
• machine temperatures and observing for faults or not
• Download from the web

Mistakes¶

1. Waiting too long for implementing a data set
2. implement it early so that AI team can give feedback to the IT team
3. Not all data is valuable
4. Messy
5. Garbage in, garbage out
6. incorrect data
7. multiple types of data

Types of Attributes¶

• D = Discrete/Qualitative/Categorical
• N = Numerical/Quantitative/Continuous

Asymmetric Attributes¶

Attributes where only non-zero values are important. It can be

• Binary (0 or 1)
• Discrete (0, 1, 2, 3, …)
• Continuous (0, 33.35, 52.99, …)

Characteristics of Dataset¶

Minimum Sample Size¶

To learn effectively

where

• \(n =\) no of sample points
• \(k =\) no of input variables
• \(C =\) no of classes

Dimensionality¶

No of features

Sparseness¶

If majority of attributes have 0 as value, depending on the context

Resolution¶

Detail/Frequency of the data (hourly, daily, monthly, etc)

Types of Datasets¶

Records¶

Collection of records having fixed attributes, without any relationship with other records

Graph¶

Ordered¶

Relationships between attributes

Sequential/Temporal¶

Extension of record, where each record has a time associated with it.

Even this can be time series data, if recorded periodically.

Time-Associated¶

Sequence Data¶

Sequence of entities

Eg: Genomic sequence data

Time Series Data¶

Series of observations over time recorded periodically

Each record is a time series as well.

Spatial Data¶

Data has spatial attributes, such as positions/areas

Weather data collected for various locations

Spatio-Temporal Data¶

Data has both spatial and temporal attributes

Issues with Data Quality¶

Estimation¶

Data¶

Data can be structured/unstructured

• Each column = feature
• Each row = instance

Data Split¶

• Train-Inner Validation-Outer Validation-Test is usually 60:10:10:20
• Split should be mutually-exclusive, to ensure good out-of-sample accuracy

The size of test set is important; small test set implies statistical uncertainty around the estimated average test error, and hence cannot claim algo A is better than algo B for given task.

Random split is the best. However, random split will not work well all the time, where there is auto-correlation, for eg: time-series data

flowchart LR

td[(Training Data)] -->
|Training| m[Model] -->
|Validation| vd[(Validation)] -->
|Tuning| m --->
|Testing| testing[(Testing Data)]

Multi-Dimensional Data¶

can be hard to work with as

• requires more computing power
• harder to interpret
• harder to visualize

Feature Selection¶

Dimension Reduction¶

Using Principal Component Analysis

Deriving simplified features from existing features

Easy example: using area instead of length and breadth.

Categories of Data¶

“Fat-Tailedness”¶

Degree to which rare events drive the aggregate statistics of a distribution

• Lower \(\alpha \implies\) Fatter tails
• 
• Kurtosis (breaks down for \(\alpha \le 4\))
• Variance of Log-Normal distribution
• 
• Taleb’s \(\kappa\) metric
• 

Leverage¶

Leverage points = data points with extreme value of input variable(s)

Like outliers, high leverage data points can have outsize influence on learning

High leverage points have lower variance

$$ \text{var}(u_i) = \sigma^2_u (1-h_{ii}) \ \text{SE}(u_i) = \text{RMSE} \sqrt{1-h_{ii}} $$

Hence, when doing statistical tests on residuals (Grubbs’ test, skewness, etc.) you should only use externally-studentized residuals

Normalized Leverage¶

William’s Graph¶

To inspect for both outliers and high-leverage data, plot the ESR vs Normalized Leverage

Influence¶

They are of concern, due to fragility of conclusions: our conclusions may depend only on a few influential data points

We just identify influential points: We don’t remove/adjust highly influential points

\(\hat y_{j(i)}\) is \(\hat y_j\) without \(i\) in the training set

	Formula	Criterion \(n \le 20\) \(n > 20\)
Cook’s Distance	\(\begin{aligned} & D_i \\ & = \dfrac{\sum\limits_{j=1}^n (\hat y_{j (i)} - \hat y_j)}{k \times \text{MSE}} \\ &= \dfrac{u_i^2}{k \times \text{MSE}} \times \dfrac{h_{ii}}{(1-h_{ii})^2} \\ &= \dfrac{\text{isr}_i^2}{k} \times \dfrac{h_{ii}}{(1-h_{ii})} \end{aligned}\)	\(1\) \(4/n \quad \approx F(k, n-k)\).inv(0.5)
Difference in Beta	\(\begin{aligned} & \text{DFBETA}_{i, j} \\ &= \dfrac{\beta_j - \beta_{j(i)}}{\text{SE}(\beta_{k(i)})} \end{aligned}\)	\(1\) \(\sqrt{4/n}\)
Difference in Fit	\(\begin{aligned} &\text{DFFITS}_{i} \\ &= \dfrac{ \hat y - \hat y_{i(i)} }{ s_{u(i)} \sqrt{h_{ii}} } \\ &= \text{esr}_i \sqrt{ \dfrac{h_{ii}}{1-h_{ii}} } \end{aligned}\)	\(1\) \(\sqrt{4k/n}\)
Mahalanobis Distance

Tidy Data¶

Also called long data

Characteristic	Visual
Each variable has its own column
Each observation has its own row
Each value has its own cell

Comments