Mellouk A., Chebira A. (eds.) Machine Learning

Подождите немного. Документ загружается.

Linear Subspace Learning for Facial Expression Analysis

283

Fig. 17. (Best viewed in color) Images of data set S4 are mapped into 2D embedding spaces.

We also plot in the right side of Fig. 18 the averaged recognition rates of different subspace

techniques as the function of the reduced dimension when using BLBP features. It is

observed the performances of SLPP and LDA become comparable when the reduced

dimension increases. On the other hand, ONPP and PCA have similar performance. The

plots for S4 shows greater variations compared to those of S1 and S2 (shown in Fig. 12).

This may also be due to the small size of the training set.

Machine Learning

284

Table 6. Averaged recognition rates (with the standard deviation) of 7-class facial expression

recognition on data set S4.

Fig. 18. (Best viewed in color) (Left) Comparison of recognition rates on data set S4; (Right)

Averaged recognition accuracy versus dimensionality reduction (with BLBP features) on

data set S4.

5. Conclusions and discussions

In this chapter, we review and evaluate a number of linear subspace methods in the context

of automatic facial expression analysis, which included recently proposed LPP, SLPP, OLPP,

ONPP, LSDA, and the traditional PCA and LDA. These techniques are compared using

different facial feature representations on several databases. Our experiments demonstrate

that the supervised LPP performs best in modeling the underlying facial expression

subspace resulting in the best expression recognition performance. We believe that this

study is helpful and necessary for further research in linear subspace methods and facial

expression analysis.

It is believed that images of facial expressions lies on a non-linear low-dimensional

manifold. Therefore, although linear subspace learning methods have been shown to be

effective , non-linear manifold learning could potentially perform better for modeling facial

expression space. For future work, we would expect to see research on discriminant non-

linear manifold learning techniques for facial expression analysis.

6. Acknowledgments

We sincerely thank Prof. Jeffery Cohn, Dr. Maja Pantic, and Dr. Michael J. Lyons for

granting access to the Cohn-Kanade database, the MMI database, and the JAFFE database

respectively. We also thank E. Kokiopoulou for providing the source code of ONPP.

Linear Subspace Learning for Facial Expression Analysis

285

7. References

Ahonen T, Hadid A, Pietik¨ainen M (2004) Face recognition with local binary patterns. In:

European Conference on Computer Vision (ECCV), pp 469–481

Ambady N, Rosenthal R (1992) Thin slices of expressive behaviour as predictors of

interpersonal consequences: A meta-analysis. Psychological Bulletin 111(2):256–274

Balasubramanian VN, Krishna S, Panchanathan S (2008) Person-independent head pose

estimation using biased manifold embedding. EURASIP Journal of Advances in

Signal Processing 8(1)

Bartlett M, Littlewort G, Fasel I, Movellan R (2003) Real time face detection and facial

expression recognition: Development and application to human computer

interaction. In: IEEE Conference on Computer Vision and Pattern Recognition

Workshop, pp 53–53

Bartlett MS, Movellan JR, Sejnowski TJ (2002) Face recognition by independent component

analysis. IEEE Transactions on Neural Networks 13(6):1450–1464

Bartlett MS, Littlewort G, Frank M, Lainscsek C, Fasel I, Movellan J (2005) Recognizing facial

expression: Machine learning and application to spotaneous behavior. In: IEEE

Conference on Computer Vision and Pattern Recognition (CVPR), pp 568–573

Bassili JN (1979) Emotion recognition: The role of facial movement and the relative

importance of upper and lower area of the face. Journal of Personality and Social

Psychology 37(11):2049–2058

Belhumeur PN, Hespanha JP, Kriegman DJ (1997) Eigenfaces vs. fisherfaces: Recognition

using class specific linear projection. IEEE Transactions on Pattern Analysis and

Machine Intelligence 19(7):711–720

Belkin M, Niyogi P (2001) Laplacian eigenmaps and spectral techniques for embedding and

clustering. In: Advances in Neural Information Processing Systems (NIPS), pp 585–

591

Belkin M, Niyogi P (2003) Laplacian eigenmaps for dimensionality reduction and data

representation. Neural Computation 15(6):1373–1396

Cai D, He X, Han J, Zhang H (2006) Orthogonal laplacianfaces for face recognition. IEEE

Transactions on Image Processing 15(11):3608–3614

Cai D, He X, Zhou K, Han J, Bao H (2007) Locality sensitive discriminant analysis. In:

International Joint Conference on Artificial Intelligence (IJCAI), pp 708–713

Chang Y, Hu C, TurkM(2003) Mainfold of facial expression. In: IEEE International

Workshop on Analysis and Modeling of Faces and Gestures (AMFG), pp 28–35

Chang Y, Hu C, Turk M (2004) Probabilistic expression analysis on manifolds. In: IEEE

Conference on Computer Vision and Pattern Recognition (CVPR), pp 520– 527

Cohen I, Sebe N, Cozman F, Cirelo M, Huang T (2003a) Learning baysian network classifiers

for facial expression recognition using both labeled and unlabeled data. In: IEEE

Conference on Computer Vision and Pattern Recognition (CVPR)

Cohen I, Sebe N, Garg A, Chen L, Huang TS (2003b) Facial expression recognition from

video sequences: Temporal and static modeling. Computer Vision and Image

Understanding 91:160–187

Cohen I, Cozman F, Sebe N, Cirelo M, Huang TS (2004) Semisupervised learning of

classifiers: Theory, algorithms, and their application to human-computer

interaction. IEEE Transactions on Pattern Analysis and Machine Intelligence

26(12):1553–1566

Machine Learning

286

Darwin C (1872) The Expression of the Emotions in Man and Animals. John Murray,

London

Donato G, Bartlett M, Hager J, Ekman P, Sejnowski T (1999) Classifying facial actions. IEEE

Transactions on Pattern Analysis and Machine Intelligence 21(10):974–989

Ekman P, Friesen W (1976) Pictures of Facial Affect. Consulting Psychologists

Ekman P, Friesen WV, Hager JC (2002) The Facial Action Coding System: A Technique for

the Measurement of Facial Movement. San Francisco: Consulting Psychologist

Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recognition

36:259–275

Freund Y, Schapire RE (1997) A decision-theoretic generalization of on-line learning and an

application to boosting. Journal of Computer and System Sciences 55(1):119–139

Fu Y, Huang TS (2006) Graph embedded analysis for head pose estimation. In: IEEE

International Conference on Automatic Face & Gesture Recognition (FG), pp 3–8

Gritti T, Shan C, Jeanne V, Braspenning R (2008) Local features based facial expression

recognition with face registration errors. In: IEEE International Conference on

Automatic Face & Gesture Recognition (FG), Amsterdam, The Netherlands

He X, Niyogi P (2003) Locality preserving projections. In: Advances in Neural Information

Processing Systems (NIPS)

Jones M, Viola P (2003) Fast multi-view face detection. Tech. Rep. 096, MERL

Kaliouby RE, Robinson P (2004) Real-time inference of complex mental states from facial

expressions and head gestures. In: IEEE Conference on Computer Vision and

Pattern Recognition Workshop, pp 154–154

Kanade T, Cohn J, Tian Y (2000) Comprehensive database for facial expression analysis. In:

IEEE International Conference on Automatic Face& Gesture Recognition (FG), pp

46–53

Kokiopoulou E, Saad Y (2005) Orthogonal neighborhood preserving projections. In: IEEE

International Conference on Data Mining (ICDM)

Kokiopoulou E, Saad Y (2007) Orthogonal neighborhood preserving projections: A

projection-based dimensionality reduction technique. IEEE Transactions on Pattern

Analysis and Machine Intelligence 29(12):2143–2156

Li Y, Gong S, Liddell H (2003) Constructing facial identity surfaces for recognition.

International Journal of Computer Vision 53(1):71–92

Lienhart R, Kuranov D, Pisarevsky V (2003) Empirical analysis of detection cascades of

boosted classifiers for rapid object detection. In: DAGM’03, 25th Pattern

Recognition Symposium, Madgeburg, Germany, pp 297–304

Littlewort G, Bartlett M, Fasel I, Susskind J, Movellan J (2006) Dynamics of facial expression

extracted automatically from video. Image and Vision Computing 24(6):615–625

Lyons MJ, Budynek J, Akamatsu S (1999) Automatic classification of single facial images.

IEEE Transactions on Pattern Analysis and Machine Intelligence 21(12):1357–1362

Mehrabian A (1968) Communication without words. Psychology Today 2(4):53–56

Murphy-Chutorian E, Trivedi M (2008) Head pose estimation in computer vision: A survey.

IEEE Transactions on Pattern Analysis and Machine Intelligence

Ojala T, Pietikäinen M, Mäenpää T (2002) Multiresolution gray-scale and rotation invariant

texture classification with local binary patterns. IEEE Transactions on Pattern

Analysis and Machine Intelligence 24(7):971–987

Linear Subspace Learning for Facial Expression Analysis

287

Oliver N, Pentland A, Berard F (2000) Lafter: a real-time face and lips tracker with facial

expression recognition. Pattern Recognition 33:1369–1382

Pantic M, Bartlett MS (2007) Machine analysis of facial expressions. In: Kurihara K (ed) Face

Recognition, Advanced Robotics Systems, Vienna, Austria, pp 377– 416

Pantic M, Rothkrantz L (2000a) Automatic analysis of facial expressions: the state of art.

IEEE Transactions on Pattern Analysis and Machine Intelligence 22(12):1424–1445

Pantic M, Rothkrantz L (2000b) Expert system for automatic analysis of facial expression.

Image and Vision Computing 18(11):881–905

Pantic M, Rothkrantz L (2003) Toward an affect-sensitive multimodal humancomputer

interaction. In: Proceeding of the IEEE, vol 91, pp 1370–1390

Pantic M, Rothkrantz LJM (2004) Facial action recognition for facial expression analysis from

static face images. IEEE Transactions on Systems, Man, and Cybernetics 34(3):1449–

1461

Pantic M, Valstar M, Rademaker R, Maat L (2005) Web-based database for facial expression

analysis. In: IEEE International Conference on Multimedia and Expo (ICME), pp

317–321

Roweis ST, Saul LK (2000) Nonlinear dimensionality reduction by locally linear embedding.

Science 290:2323–2326

Russell JA (1994) Is there univeral recognition of emotion from facial expression.

Psychological Bulletin 115(1):102–141

Saul LK, Roweis ST (2003) Think globally, fit locally: Unsupervised learning of low

dimensional manifolds. Journal of Machine Learning Research 4:119–155

Schapire RE, Singer Y (1999) Improved boosting algorithms using confidencerated

predictions. Maching Learning 37(3):297–336

Shan C (2007) Inferring facial and body language. PhD thesis, Queen Mary, University of

London

Shan C, Gritti T (2008) Learning discriminative lbp-histogram bins for facial expression

recognition. In: British Machine Vision Conference (BMVC), Leeds, UK

Shan C, Gong S, McOwan PW (2005a) Appearance manifold of facial expression. In: Sebe N,

Lew MS, Huang TS (eds) Computer Vision in Human-Computer Interaction,

Lecture Notes in Computer Science, vol 3723, Springer, pp 221–230

Shan C, Gong S, McOwan PW (2005b) Conditional mutual information based boosting for

facial expression recognition. In: British Machine Vision Conference (BMVC),

Oxford, UK, vol 1, pp 399–408

Shan C, Gong S, McOwan PW (2005c) Robust facial expression recognition using local

binary patterns. In: IEEE International Conference on Image Processing (ICIP),

Genoa, Italy, vol 2, pp 370–373

Shan C, Gong S, McOwan PW (2006a) A comprehensive empirical study on linear subspace

methods for facial expression analysis. In: IEEE Conference on Computer Vision

and Pattern Recognition Workshop, New York, USA, pp 153–158

Shan C, Gong S, McOwan PW (2006b) Dynamic facial expression recognition using a

bayesian temporal manifold model. In: British Machine Vision Conference (BMVC),

Edinburgh, UK, vol 1, pp 297–306

Suwa M, Sugie N, Fujimora K (1978) A preliminary note on pattern recognition of human

emotional expression. In: International Joint Conference on Pattern Recognition, pp

408–410

Machine Learning

288

Tenenbaum JB, Silva V, Langford JC (2000) A global geometric framework for nonlinear

dimensionality reduction. Science 290:2319–2323

Tian Y (2004) Evaluation of face resolution for expression analysis. In: International

Workshop on Face Processing in Video, pp 82–82

Tian Y, Kanade T, Cohn J (2001) Recognizing action units for facial expression analysis. IEEE

Transactions on Pattern Analysis and Machine Intelligence 23(2):97– 115

Tian Y, Kanade T, Cohn J (2005) Handbook of Face Recognition, Springer, chap 11. Facial

Expression Analysis

Tong Y, Liao W, Ji Q (2006) Inferring facial action units with causal relations. In: IEEE

Conference on Computer Vision and Pattern Recognition (CVPR), pp 1623–1630

Turk M, Pentland AP (1991) Face recognition using eigenfaces. In: IEEE Conference on

Computer Vision and Pattern Recognition (CVPR)

Valstar M, Pantic M (2006) Fully automatic facial action unit detection and temporal

analysis. In: IEEE Conference on Computer Vision and Pattern Recognition

Workshop, p 149

Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In:

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 511–518

Wang Y, Ai H, Wu B, Huang C (2004) Real time facial expression recognition with adaboost.

In: International Conference on Pattern Recognition (ICPR), pp 926– 929

Whitehill J, Omlin C (2006) Harr features for facs au recognition. In: IEEE International

Conference on Automatic Face & Gesture Recognition (FG), pp 97–101

Yang MH, Kriegman DJ, Ahuja N (2002) Detecting faces in images: A survey. IEEE

Transactions on Pattern Analysis and Machine Intelligence 24(1):34–58

Yeasin M, Bullot B, Sharma R (2004) From facial expression to level of interests: A spatio-

temporal approach. In: IEEE Conference on Computer Vision and Pattern

Recognition (CVPR), pp 922–927

Zhang Y, Ji Q (2005) Active and dynamic information fusion for facial expression

understanding from image sequences. IEEE Transactions on Pattern Analysis and

Machine Intelligence 27(5):1–16

Zhang Z, Lyons MJ, Schuster M, Akamatsu S (1998) Comparison between geometry-based

and gabor-wavelets-based facial expression recognition using multi-layer

perceptron. In: IEEE International Conference on Automatic Face & Gesture

Recognition (FG), pp 454–461

Resampling Methods for Unsupervised

Learning from Sample Data

Ulrich Möller

Leibniz Institute for Natural Product Research and Infection Biology –

Hans Knöll Institute,

Germany

1. Introduction

Two important tasks of machine learning are the statistical learning from sample data (SL)

and the unsupervised learning from unlabelled data (UL) (Hastie et al., 2001; Theodoridis &

Koutroumbas, 2006). The synthesis of the two parts – the unsupervised statistical learning

(USL) – is frequently used in the cyclic process of inductive and deductive scientific

inference. This applies especially to those fields of science where promising, testable

hypotheses are unlikely to be obtained based on manual work, the use of human senses or

intuition. Instead, huge and complex experimental data have to be analyzed by using

machine learning (USL) methods to generate valuable hypotheses. A typical example is the

field of functional genomics (Kell & Oliver, 2004).

When machine learning methods are used for the generation of hypotheses, human

intelligence is replaced by artificial intelligence and the proper functioning of is this type of

‘intelligence’ has to be validated. This chapter is focused on the validation of cluster analysis

which is an important element of USL.

It is assumed that the data set is a sample from a mixture population which is statistically

modeled as a mixture distribution. Cluster analysis is used to ‘learn’ the number and

characteristics of the components of the mixture distribution (Hastie et al., 2001). For this

purpose, similar elements of the sample are assigned to groups (clusters).

Ideally, a cluster represents all of the elements drawn from one population of the mixture.

However, clustering results often contain errors due to lacking robustness of the algorithms.

Rather different partitions may result even for samples with small differences. That is, the

obtained clusters have a random character. In this case, the generalization from clusters of a

sample to the underlying populations is inappropriate. If a hypothesis derived from such

clustering results is used to design an experiment, the outcome of this experiment will

hardly lead to a model with a high predictive power. Thus, a new study has to be performed

to find a better hypothesis. Even a single cycle of hypothesis generation and hypothesis

testing can be time-consuming and expensive (e.g., a gene expression study in cancer

research, with 200 patients, lasts more than a year and costs more than 100.000 dollars).

Therefore, it is desirable to increase the efficiency and effectiveness of the scientific progress

by using suitable validation tools.

An approach for the statistical validation of clustering results is data resampling

(Lunneborg, 2000). It can be seen as a special Monte Carlo method that is, as a method for

Machine Learning

290

finding solutions to statistical problems by simulation (Borgelt & Kruse, 2006). The choice of

a suitable resampling method for any cluster validation task is not trivial. On the one hand,

such a method is expected to simulate random samples that have the same structure that

underlies the original sample – even though the true structure is unknown. On the other

hand, it is undesired that the method introduces any additional structure into the simulated

data, because this kind of error can not be recognized from the clustering results in the

absence of the ground truth.

Once, clustering results (partitions) have been generated for a set of resamples, three steps

are usually performed. i) The stability of the partitions under the influence of resampling is

calculated. When desired, stability scores can be obtained also for single clusters and

individual assignments of data points to clusters. ii) A consensus partition is determined that

best possible represents the characteristics which are common to the resample partitions. iii)

The number of clusters is estimated, typically based on the maximization of a partition

stability score. For methods that can be used to perform the steps i) to iii) see, for example,

(Strehl & Gosh, 2002; Topchy et al., 2005; Fred & Jain, 2006 and Ayad & Kamel, 2008).

Resampling-based cluster validation is not yet common standard. In many software tools for

cluster analysis, resampling methods are missing. Some new methods were published only

recently. The choice of the appropriate resampling technique depends on the data

properties, the goal and constraints of the study and on the clustering methods used. The

purpose of this contribution is to review available techniques, to summarize existing

benchmark results and to give recommendations for the selection and use of the methods.

Furthermore, a new method called nearest neighbor resampling is presented.

In statistics resampling schemes are subdivided into parametric and non-parametric

methods. The use of parametric methods for cluster validation will be briefly characterized

in section 2. In section 3 non-parametric methods will be reviewed. Section 4 is a summary

of benchmarking tests of different resampling techniques. Section 5 refers to results of the

new resampling method previously described in section 3.5. Finally, section 6 contains a

discussion of the described methods and conclusions for their future application.

2. Parametric resampling

Parametric resampling is also known as parametric bootstrapping. Methods of this type are

used to fit a parametric model to the data. That is, the hypothesis is made that the data

follow a theoretical distribution and certain parameters of this distribution (mean, variance

etc.) are estimated. Then resample data sets are sampled from the distribution with the

parameter values set to the obtained estimates. In cluster analysis a mixture distribution P =

is assumed, where P

, i = 1,…,C, are the C distributions generating C “true” clusters

respectively, and

is the probability that a sample point from P

is drawn.

In principle, this approach has attractive properties. Examples for the validation of

clustering results obtained from gene expression data are contained in (McLachlan & Khan,

2004). However, there exist also arguments against the use of parametric resampling for

cluster validation. One argument concerns the lack of justification for the (more or less

arbitrary) selection of a particular theoretical distribution as a model for real data with an

unknown distribution (Yu et al. 1999; Lunneborg, 2000). Hennig (2007) argued that

parametric bootstrapping does not suggest itself for the aim of cluster validation, because

parametric methods discover structures generated by the assumed model much better than

patterns in real data for which the model does not hold. This could lead to overoptimistic

Resampling Methods for Unsupervised Learning from Sample Data

291

assessments of the stability of clustering structures. If the original sample has clearly more

dimensions than data points, model fitting may be impaired by the “curse of

dimensionality”. Further arguments can be found in (Tseng & Wong, 2005). In the sequel,

we consider non-parametric resampling methods that may be used in cluster analysis.

3. Non-parametric resampling

3.1 Sampling from a sample

Several methods can be referred to as re-sampling in the literal sense according to the

common (non-statistical) definition of the word sample

. In such methods the data points of

a resample are drawn from the set of data points contained in the original sample.

Bootstrapping. The non-parametric version of bootstrapping is usually described as

“drawing with replacement”. That is, each bootstrap sample is obtained by drawing N data

points randomly and with replacement from the original sample, where N is the number of

data points in this sample. If the population size N

is finite and relatively small compared

to N, (i.e., N

/ N < 20), another procedure is conventionally used (see Lunneborg, 2000).

This procedure guarantees that the empirical distribution of the union of all bootstrap

resamples agrees accurately with the empirical distribution of the original sample. In any

case some original data points are likely represented more than once in a bootstrap sample,

while accordingly, other original points are missing in the resample.

It has been shown that for increasing values of N, the percentage of original data which are

not contained in a bootstrap sample converges to about 37%. If this information loss is

considered to large for an adequate recognition of the data structure, the bootstrap scheme

could be applied to M randomly selected points of the sample X (M < N), while the

resample is completed by the N−M points of X not used for the bootstrapping. This

modification would allow to control the degree of information loss associated with the

bootstrap scheme (Möller & Radke, 2006a). Moreover, this resampling version could be

performed by using random numbers M

for the generation of r = 1, 2, … bootstrap samples

with reasonable boundaries of the interval from which the values M

are drawn. This

selection could make the results less depending on the heuristic choice of parameter M.

Subsampling. The original data set X is used to draw random subsets Y

⊂ X, r = 1, 2,… The

size of a drawn subset, S = card(Y

), is a control parameter. Usually, S is fixed for all

subsamples to be used in an application. If S is not much smaller than the original sample

size N, clustering results of different subsamples may be very similar and not informative.

The choice of S clearly smaller than N can be recommended if the information retained on

average in a subsample is sufficient to obtain reasonable estimates of the unknown

underlying distribution. Resampling-based clustering methods have been introduced

including the subsampling of 70% (Tseng & Wong, 2005), 80% (Monti et al., 2003) and 90%

(Fred & Jain, 2006) of the data. It may not be easy to select an optimal subsample size in a

particular application. To avoid an inappropriate choice for this parameter, the subsample

size could be varied from subsample to subsample. For example, the subsample size is

uniformly drawn from an interval that represents 75-90% of the size of the original sample.

A sample of things is a number of them that are chosen at random out of a larger group

and then used to test ideas or to provide information about the whole group (Collins

Cobuild Dictionary, 1987).

Machine Learning

292

An alternative way, without an explicit specification of the subsampling size, would be to

generate a bootstrap sample and to discard the identically replicated points (Hennig, 2007).

Subdivision. The original sample X is split into two disjoint subsets Y ∪ Z = X. Clustering is

used to generate the partitions π

and π

, from Y and Z, respectively. In addition, a classifier

is build from the subset Y and the label set π

. Then C

is applied to the subset Z

providing the partition π

. Finally, the predictability of π

based on π

is assessed. For the

success of this strategy it has to be ensured that in general each subset Y and Z contain

sufficient information about the underlying distribution necessary to infer a reasonable

model from the data. Dudoit and Fridlyand (2003) presented an example, where the

‘training’ set Y and the ‘test’ set Z consist respectively of 2/3 and 1/3 of the original sample.

3.2 Jittering

Real data samples contain random measurement errors. Even if the same objects were

observed multiple times under the same experimental conditions, the data are likely to be

different. These differences can be simulated by generating copies of the original sample

and adding random values to each of these data sets. The normal distribution with zero

mean is traditionally used for this purpose. If estimates of the measurement error exist,

these information can be utilized to define the parameters of the error distribution.

Otherwise, heuristic rules can be applied.

Hennig (2007) defined such a resampling scheme as follows. 1) For all p dimensions of the

original sample data X = (x

,…, x

), compute the N−1 differences d

between neighboring

data values in dimension p: for i = 1,…, N−1, j = 1,…, p, d

is the difference between the (i+1)-

th and the i-th order statistic of the j-th dimension. For j = 1,…, p, let q

be the empirical

quantile of the d

, where q is a tuning constant. 2) Draw noise e

, n = 1,…, N, independent

and identically distributed from a normal distribution with a zero mean and a diagonal

matrix as covariance matrix with diagonal elements σ

= q

,…, σ

= q

and compute the

resample points y

= x

+ e

for n = 1,…, N. (For an example see section 4).

3.3 Combination of bootstrapping and jittering

When using the (non-parametric) bootstrapping scheme, about one third of the resample

points will be identical replicates of original sample points. Each group of such identical

points could be seen as a mini-cluster. The occurrence of these artificial clusters, generated

by a statistical analysis tool, may induce inappropriate models of the true data structure. In

particular, when clustering the resample data, the artificially replicated data points may be

misinterpreted as true clusters (Monti et al., 2003). Moreover, for some implementations of

clustering and multidimensional scaling methods the identical bootstrap replicates may

cause numerical problems. Hennig (2007) proposed the combination of bootstrapping and

jittering as a way to avoid or to reduce these problems.

3.4 Perturbation

Data sets for applications of statistical machine learning are usually generated with a

precision that is high enough to measure intra-population variability. Therefore, any data

point of a sample is likely to be different from any data point of another (disjoint) sample −

even if the measurement error was zero. This type of inter-sample differences is not

realistically simulated when using the above non-parametric methods. (Sampling from a

sample provides highly overlapping data sets that all consist of random selections from the