Machine Learning Glossary

Michael J. Pyrcz, Professor, The University of Texas at Austin

Twitter | GitHub | Website | GoogleScholar | Geostatistics Book | YouTube | Applied Geostats in Python e-book | Applied Machine Learning in Python e-book | LinkedIn

Chapter of e-book “Applied Machine Learning in Python: a Hands-on Guide with Code”.

Cite this e-Book as:

Pyrcz, M.J., 2024, Applied Machine Learning in Python: A Hands-on Guide with Code [e-book]. Zenodo. doi:10.5281/zenodo.15169139

The workflows in this book and more are available here:

Cite the MachineLearningDemos GitHub Repository as:

Pyrcz, M.J., 2024, MachineLearningDemos: Python Machine Learning Demonstration Workflows Repository (0.0.3) [Software]. Zenodo. DOI: 10.5281/zenodo.13835312. GitHub repository: GeostatsGuy/MachineLearningDemos

By Michael J. Pyrcz
© Copyright 2024.

This chapter is a summary of essential Machine Learning Terminology.

Motivation for this Glossary

Firstly, why create this glossary?

I received a request for a course glossary from students in my Subsurface Machine Learning graduate course. While I usually dedicate a definition slide in my lecture slide decks to important terms, various students requested a consolidated glossary of terminology as part of their course review materials. The e-book provides an excellent vehicle and motivation for creating this resource.

Let me begin with a confession. There is a Machine Learning Glossary written by Google developers. For those seeking the in depth, comprehensive list of machine learning terms please use this book! For those seeking a comprehensive and in-depth reference of geostatistical terminology, this book remains an excellent resource.

So why create another glossary?

• Scope - By writing my own glossary, I can limit the scope and descriptions to the concepts covered in this course. I believe many students would be overwhelmed by the size, breadth, and mathematical notation of a comprehensive geostatistics glossary.

• Statistical Linkages - At the same time in my opinion machine learning is statistical learning and I have taken liberty to include many standard statistical terms as a foundation for all things machine learing.

• Workflow Integration - By integrating the glossary directly into the e-book, I can link terminology to detailed chapter discussions, demonstrations, and examples. The goal is to eventually populate all chapters with hyperlinks to glossary entries, enabling students to move easily between concepts and applications.

• Evergreen Resource - Finally, like the rest of this e-book, I want the glossary to be an evergreen living document. It will continue to evolve with new concepts, improved explanations, and feedback from students and readers.

I put quite a bit of time into this project during summer 2026 and I am happy with the way that is has evolved,

• More than a glossary, it has become an evergreen network of machine learning concepts.

Machine Learning Workflow Design

Machine learning (and more generally, data science) workflows follow a logical sequence of steps that transform raw data into information that supports classification, prediction, interpretation, and decision making.

1. Define the Objective, for example,

• build a predictive model

• classify observations

• identify patterns in the data

• compare alternative recovery processes

2. Understand the Data, answering questions such as,

• what data are available?

• what information is missing?

• are the data representative and of sufficient quality?

• could additional data improve the analysis?

3. Design the Workflow, including steps such as,

• load and organize the data

• clean, validate, and preprocess the data

• engineer or transform features

• train statistical or machine learning models

• evaluate and compare model performance

• visualize, interpret, and communicate results

• support decision making

4. Validate and Improve the Workflow

• test the workflow using typical and edge cases

• identify when the workflow performs well and when it fails

• diagnose sources of error and uncertainty

• refine the data, features, models, or workflow design

5. Document and Communicate the Workflow

• record implementation details, assumptions, metadata, limitations, and future work

• document the flow of data and information throughout the workflow

• summarize sources of uncertainty and describe how they were quantified and incorporated

• communicate conclusions and recommendations

Used in: TBA - workflow design chapter

Margin

The margin in a support vector machine is the distance between the binary decision boundary and the closest observations from each class. SVM training is primarily influenced by observations near the margin, called support vectors, while observations well outside the margin have little or no influence on the final decision boundary.

• The margin is a symmetric distance around the binary decision boundary that defines a region of uncertainty between the two classes.

For classification problems with overlapping groups in predictor feature space, perfect separation by a decision boundary is often not practical.

No margin classification would require only that observations fall on the correct side of the decision boundary,

\[ y_i \left( x_i^T \beta + \beta_0 \right) \geq 0 \]

A hard-margin SVM requires,

\[ y_i \left( x_i^T \beta + \beta_0 \right) \geq 1 \]

where observations must be correctly classified and separated from the decision boundary by a minimum margin.

For real-world data with overlapping classes and noise, a soft-margin SVM allows some observations to violate the margin using slack variables, \(\xi_i\),

\[ y_i \left( x_i^T \beta + \beta_0 \right) \geq 1 - \xi_i \]

where,

• \(\xi_i\) quantifies the margin violation, or the distance an observation extends into the margin region or across the decision boundary.

• \(0 < \xi_i < 1\) indicates that an observation is correctly classified but lies inside the margin

• \(\xi_i > 1\) indicates that an observation is misclassified

The SVM optimization balances maximizing the margin and minimizing classification errors,

\[ \underset{\beta,\beta_0,\xi}{\text{min}} \left( \frac{1}{2}\|\beta\|^2 + C\sum_{i=1}^{N}\xi_i \right) \]

where,

• \(\frac{1}{2}\|\beta\|^2\) controls the margin width, since the margin is inversely proportional to \(\|\beta\|\)

• \(C\) is a hyperparameter controlling the penalty for observations violating the margin

A larger \(C\) emphasizes correct classification and produces a narrower margin, while a smaller \(C\) allows more violations and produces a wider margin.

The objective is therefore to,

• maximize the separation margin between classes

• minimize classification error through the slack variables weighted by \(C\)

Used in:

• Margin for Support Vector Machines Discussion and Demonstration

Also see:

• Support Vector

• Support Vector Machines

• Kernel Trick

Marginal Probability

Probability that considers only a single event occurring. For example, the probability of event \(A\),

\[ P(A) \]

Marginal probabilities may be calculated from joint probabilities through the process of marginalization,

\[ P(A) = \int_{-\infty}^{\infty} P(A,B) dB \]

where we integrate over all cases of the other event, \(B\), to remove its influence. Given discrete, categorical or binned continuous cases of event \(B\) we can simply sum the probabilities over all possible cases of \(B\),

\[ P(A) = \sum_{i=1}^{k_B} P(A,B) \]

Used in:

• Marginal Probability Definition and Demonstration

Contrast with:

• Conditional Probability

• Joint Probability

Markov Chain Monte Carlo

Known widely by the acronym MCMC, a family of algorithms used to estimate complicated probability distributions by generating samples from those distributions.

MCMC methods are based on,

• Markov chain - a sequence of samples where each sample depends only on the previous sample

• Markov property - the assumption that the current sample contains all required information from previous samples, such that future samples are conditionally independent of earlier samples given the current sample, this is also known as Markov screening

• Monte Carlo - a simulation process that uses random sampling to approximate probability distributions and expectations

The general workflow is,

1. Initialize - start at an initial point in the distribution.

2. Propose - generate a candidate new sample based on a proposal distribution.

3. Evaluate - calculate the probability of accepting the proposed sample relative to the current sample.

4. Accept or Reject - accept the new sample according to an acceptance probability. Higher probability samples are favored, while lower probability samples may still be accepted to explore the full distribution.

5. Repeat - generate many samples to form a Markov chain that approximates the target probability distribution.

There are a variety of MCMC methods, including,

• Metropolis-Hastings - proposes random steps and uses a stochastic acceptance rule to sample from the target distribution

• Gibbs Sampler - samples each feature or variable sequentially from its conditional probability distribution

Used in:

• Metropolis-Hastings to Sample Bayesian Linear Regression Posterior Description and Demonstration

Compare with:

• Metropolis-Hastings

Also see:

• Markov Chain Monte Carlo

• Metropolis-Hastings

• Gibbs Sampler

Matrix Scatter Plots

A composite plot containing all pair-wise scatter plots between features in a dataset.

• given \(m\) features, there are \(m \times m\) scatter plots

• the scatter plots are ordered with the y-axis feature from \(X_1,\ldots,X_m\) arranged over the rows and the x-axis feature from \(X_1,\ldots,X_m\) arranged over the columns

• the diagonal contains each feature plotted against itself and is often replaced with univariate histograms or probability density functions

Matrix scatter plots are used to,

• identify bivariate linear or nonlinear relationships between features

• identify bivariate homoscedasticity (constant conditional variance) and heteroscedasticity (changing conditional variance)

• identify bivariate constraints, such as sum constraints in compositional data

The remaining features are marginalized during each pair-wise comparison,

• therefore, a matrix scatter plot is not a visualization of the full \(m\)-dimensional feature space

Used in:

• exploratory data analysis (EDA)

• feature engineering and selection

• identifying feature relationships and redundancy

• diagnosing assumptions for statistical and machine learning models

Used in:

• Matrix Scatter Plots Description and Demonstration

Also see:

• Scatter Plot

• Bivariate

Maximum Likelihood Estimation

A statistical method for estimating unknown model parameters by selecting the parameter values that maximize the likelihood of the observed data.

Maximum likelihood estimation is one of the primary approaches for model parameter training and contrasts with Bayesian updating, which combines prior knowledge with observed data.

Also see:

• Model Parameter

• Model Parameter Training

• Bayesian Updating

Maximum Relevance Minimum Redundancy

A mutual information-based feature selection approach that identifies a subset of predictor features by maximizing relevance to the response feature while minimizing redundancy between selected features.

• one common formulation is a relevance minus redundancy criterion,

\[\begin{split} mRMR =\max\left[\frac{1}{|S|}\sum_{X_i \in S} I(X_i,Y)-\frac{1}{|S|^2}\sum_{X_i \in S}\sum_{\substack{X_j \in S \\ i \ne j}}I(X_i,X_j)\right] \end{split}\]

where,

• \(S\) is the selected predictor feature subset

• \(|S|\) is the number of features in subset \(S\)

• \(I(X_i,Y)\) is the mutual information between predictor feature \(X_i\) and reponse feature \(Y\), representing feature relevance

• \(I(X_i,X_j)\) is the mutual information between predictor features \(X_i\) and \(X_j\), representing feature redundancy

The objective is to select features that,

• contain strong information about the response variable

• provide complementary information rather than repeating information already contained in selected features

Used in:

• Maximum Relevance Minimum Redundancy for Feature Ranking

Also see:

• Mutual Information

Mean

The mean (or average) is a measure of central tendency. There are several useful interpretations of the mean,

• representative value - a single value to represent an entire distribution

• estimate - given a distribution of values, the mean minimizes the L2 norm (sum of squared error)

• scale-up - under linear averaging of a spatial feature, the mean is the correct upscaled value

• expectation - if all outcomes are equiprobable, the mean is equal to the expectation

For a sample, the mean is,

\[ \overline{x} = \frac{1}{n}\sum_{\alpha=1}^{n} x_{\alpha} \]

Note, the mean is quite sensitive to outliers.

Used in:

• Measures of Central Tendency Summarized

Also see:

• Variance

• Statistic

• Parameter

Mean Absolute Error

Prediction model performance metric calculated as the average absolute prediction error over all observations,

• commonly known by the acronym MAE.

The equation for mean absolute error is,

\[ MAE=\frac{1}{n}\sum_{i=1}^{n}\left|y_i-\hat{y}_i\right| \]

where \(y_i\) is the observed response feature value, \(\hat{y}_i\) is the model prediction, and \(n\) is the number of observations.

Mean absolute error has several important properties,

• all prediction errors contribute equally to the metric

• prediction errors increase linearly with error magnitude

• always non-negative, with \(MAE=0\) indicating perfect predictions

• reported in the original units of the response feature

Mean absolute error is commonly applied,

• as a performance metric for regression models

• when robustness to outliers is important

• to compare competing regression models on the same testing dataset

Compared with other error metrics,

• mean square error - penalizes large prediction errors more strongly by squaring the errors

• root mean square error - also emphasizes larger prediction errors but is reported in the original response feature units

Advantages of mean absolute error include,

• less sensitive to outliers than MSE and RMSE

• easier to interpret because it represents the average magnitude of prediction errors

Limitations of mean absolute error include,

• treats all prediction errors equally, regardless of their magnitude

• absolute values are not differentiable at zero, making MAE less convenient for optimization than MSE

• provides no indication whether prediction errors are biased or randomly distributed

Used in:

• Sum Absolute Value Loss in Shrinkage Term of LASSO Regression

Also see:

• Prediction

• Prediction Error

• Mean Square Error

• Root Mean Square Error

• Loss Function

• Norm

• Model Hyperparameter Tuning

Mean Square Error

Prediction model performance metric calculated as the average squared prediction error over all observations,

• commonly known by the acronym MSE.

Te equation for mean square error is,

\[ MSE=\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2 \]

where \(y_i\) is the observed response feature value, \(\hat{y}_i\) is the model prediction, and \(n\) is the number of observations.

Mean square error has several important properties,

• all prediction errors contribute to the metric

• squaring the errors penalizes large prediction errors much more strongly than small errors

• always non-negative, with \(MSE=0\) indicating perfect predictions

• reported in squared units of the response feature

Mean square error is commonly applied,

• as a loss function for training many regression models

• as an objective metric for model hyperparameter tuning

• to compare competing regression models on the same testing dataset

Compared with other error metrics,

• mean absolute error - more robust to outliers because prediction errors are not squared

• root mean square error - square root of MSE, reported in the original units of the response feature

Limitations of mean square error include,

• sensitive to outliers due to squaring of prediction errors

• should not be compared between response features with different units or scales without normalization

• provides no indication whether prediction errors are biased or randomly distributed

Used in:

• Mean Square Error Loss Function for Linear Regression Description and Demonstration

• Mean Square Error Loss Function for Ridge Regression Description and Demonstration

• Mean Square Error Loss Function for LASSO Regression Description and Demonstration

• Mean Square Error Loss Function for Decision Tree Description and Demonstration

• Mean Square Error Loss Function for Random Forest Description and Demonstration

Also see:

• Prediction

• Prediction Error

• Mean Absolute Error

• Root Mean Square Error

• Loss Function

• Norm

• Model Hyperparameter Tuning

Metropolis-Hastings

A Markov chain Monte Carlo sampler based on,

• proposing random steps in parameter space and using a stochastic acceptance rule to sample from a target probability distribution.

The basic steps of the Metropolis-Hastings MCMC sampler are:

For \(\ell = 1,\ldots,L\):

1. Initialize - assign random initial values for the model parameters,

\[ \theta^{(1)} = (\beta^{(1)}, b_0^{(1)}, \sigma^{2(1)}) \]

where \(\theta\) represents the complete set of model parameters.

2. Propose - generate new model parameters from a proposal distribution,

\[ \theta' \sim q(\theta'|\theta^{(\ell)}) \]

3. Calculate the acceptance probability - compute the probability of accepting the proposed parameters,

\[ \alpha = \min \left( \frac{P(\theta'|y,X)} {P(\theta^{(\ell)}|y,X)} \cdot \frac{q(\theta^{(\ell)}|\theta')} {q(\theta'|\theta^{(\ell)})}, 1 \right) \]

where,

• \(P(\theta|y,X)\) is the posterior probability of the model parameters given the data

• \(q(\theta'|\theta)\) is the proposal distribution probability of generating \(\theta'\) from the current state

4. Accept or reject - generate a random value,

\[ u \sim U(0,1) \]

If \(u < \alpha\), accept the proposal,

\[ \theta^{(\ell+1)}=\theta' \]

otherwise, retain the current sample,

\[ \theta^{(\ell+1)}=\theta^{(\ell)} \]

5. Repeat - continue sampling until a sufficient number of samples are generated.

The resulting Markov chain provides samples that approximate the target posterior distribution.

Used in:

• Metropolis-Hastings to Sample Bayesian Linear Regression Posterior Description and Demonstration

Compare with:

• Metropolis-Hastings

Also see:

• Markov Chain Monte Carlo

Minkowski Distance

A general distance metric where the well-known Manhattan and Euclidean distances are special cases.

\[ d_{(i,i')} =\left(\sum_{j=1}^{m}\left|x_{(j,i)} - x_{(j,i')}\right|^p\right)^{\frac{1}{p}} \]

where \(m\) is the number of features and \(p\) controls the distance metric.

Common cases include,

• \(p=1\) - Manhattan distance - sums absolute differences and is less dominated by large coordinate differences

• \(p=2\) - Euclidean distance - computes the straight-line distance between observations

• \(p=\infty\) - Chebyshev distance - considers only the maximum coordinate difference between observations

Algorithms that may use Minkowski distance include,

• k-Nearest Neighbours

• k-Means Clustering

Used in: TBA - in training and tuning chapter

Missing At Random

Missing feature values are considered Missing At Random (MAR) when the probability of a feature value being missing depends on other observed variables in the dataset, but not on the missing value itself after accounting for those observed variables.

For example,

1. permeability measurements may be missing more frequently in certain lithologies, depths, or sampling conditions where those related features are available

2. laboratory measurements may be unavailable due to known sample characteristics or collection conditions

3. samples may be selectively collected based on available information to reduce uncertainty and improve decision making

MAR differs from MCAR because missingness is not random across the entire dataset; however, the missingness mechanism can be explained using observed information.

Appropriate approaches for MAR data may include,

• model-based imputation

• multiple imputation

• including variables related to missingness in predictive models

If MAR data are ignored, missing values may result in,

• biased sample statistics

• biased model training and evaluation

• biased predictions with potentially no indication of the underlying bias

Used in:

• Missing Feature Imputation Discussion and Demonstration

Also see:

• Missing Completely At Random

• Missing Not At Random

Missing Completely At Random

Missing feature values are considered Missing Completely At Random (MCAR) when the probability of a feature value being missing is independent of both observed and unobserved data.

In this case,

• missing values occur randomly throughout the dataset

• samples with missing values are statistically representative of the complete dataset

• removing samples with missing values does not introduce systematic bias

Examples include,

1. accidental data recording failures

2. random equipment failures during measurement

3. randomly lost samples during data transfer or storage

MCAR is the most favorable missing data condition because standard approaches such as removing incomplete samples may not introduce bias.

However, true MCAR conditions are uncommon in real-world datasets because data collection processes often introduce systematic missingness.

Used in:

• Missing Feature Imputation Discussion and Demonstration

Also see:

• Missing At Random

• Missing Not At Random

Missing Feature Values

Missing values in a data table occur when samples do not contain measurements for all features.

There are many causes of missing feature values, for example,

1. Sampling cost - some measurements may be expensive, time-consuming, or impractical to collect, e.g., low permeability tests that require long-duration experiments

2. Sample recovery limitations - some samples may be difficult to recover or measure due to their physical properties, e.g., inability to recover weak mudstone samples

3. Targeted sampling strategies - samples may be collected to reduce uncertainty and maximize decision value rather than to achieve statistical representativity, e.g., dual-purpose samples collected for both information gain and production evaluation

Missing data consequences extend beyond reducing the amount of training and testing data. If missing values are not completely random, they may result in,

• biased sample statistics, leading to biased model training and evaluation

• biased models and predictions, potentially with no indication of the underlying bias

Used in:

• Missing Feature Imputation Discussion and Demonstration

Also see:

• Missing At Random

• Missing Completely At Random

• Missing Not At Random

Missing Not At Random

Missing feature values are considered Missing Not At Random (MNAR) when the probability of a feature value being missing depends on the missing value itself or other unobserved information.

For example,

1. permeability measurements may be missing because very low permeability samples require excessive testing time

2. weak mudstone samples may not be recovered because their physical properties prevent successful sampling

3. production-related measurements may be preferentially collected only from high-value or successful operations

MNAR is the most challenging missing data condition because the missingness mechanism contains information that is not directly observed.

Consequences of MNAR data include,

• biased sample statistics because missing observations are systematically different from available observations

• biased models and predictions

• uncertainty that is underestimated because the missingness mechanism is not fully represented

Addressing MNAR data typically requires additional assumptions, domain knowledge, sensitivity analysis, or explicit modeling of the missing data process.

Used in:

• Missing Feature Imputation Discussion and Demonstration

Also see:

• Missing At Random

• Missing Completely At Random

Model Bagging

The application of bootstrap resampling to generate \(B\) realizations of the training dataset,

\[ Y^b, X_1^b, \dots, X_m^b, \quad b = 1, \dots, B, \]

used to train an ensemble of predictive models,

\[ \hat{Y}^b = \hat{f}^b(X_1^b, \dots, X_m^b), \]

where,

• \((X_1^b, \dots, X_m^b)\) – the predictor features in the \(b^{th}\) bootstrap sample

• \(\hat{f}^b\) – the predictive model trained on the \(b^{th}\) bootstrap sample

• \(\hat{Y}^b\) – the prediction from the \(b^{th}\) model

The ensemble of predictions is then aggregated to reduce model variance. The aggregation depends on the prediction task:

• Regression – average the predictions,

\[ \hat{Y} = \frac{1}{B}\sum_{b=1}^{B}\hat{Y}^b \]

• Classification – assign the class receiving the majority vote,

\[ \hat{Y} = \operatorname{mode}\left(\hat{Y}^1,\hat{Y}^2,\ldots,\hat{Y}^B\right) \]

Bagging can be applied to almost any predictive model. In fact, the BaggingClassifier and BaggingRegressor classes in scikit-learn are wrappers that use any compatible prediction model as the base learner to construct a bagged ensemble.

Used in:

• Bagging Trees Description and Demonstration

Also see:

• Bootstrap

Model Bias

The additive component of the expected testing mean square error that is caused by a predictive model being too simple and inflexible,

\[ \mathbb{E}\!\left[\left(y_0-\hat{f}(x_{0,1}^{(i)},\ldots,x_{0,m}^{(i)})\right)^2\right] = \underbrace{\left(\mathbb{E}\!\left[\hat{f}(x_{0,1}^{(i)} \ldots,x_{0,m}^{(i)})\right]-f(x_{0,1}^{(i)},\ldots,x_{0,m}^{(i)})\right)^2}_{\text{Model Bias}^2} + \underbrace{\mathbb{E}\!\left[\left(\hat{f}(x_{0,1}^{(i)},\ldots,x_{0,m}^{(i)})-\mathbb{E}\!\left[\hat{f}(x_{0,1}^{(i)},\ldots,x_{0,m}^{(i)})\right]\right)^2\right]}_{\text{Model Variance}} + \underbrace{\sigma_{\epsilon}^{2}}_{\text{Irreducible Error}} \]

Model hyperparameter tuning is applied to balance model bias and model variance

• model bias variance trade-off - as model complexity increases, model variance increases and model bias decreases, and vice versa.

Used in:

• Tuning Model Hyperparameters

Also see:

• Model Bias

• Model Variance

• Model Bias–Variance Trade-off

Model Bias–Variance Trade-off

Model hyperparameter tuning is applied to balance the model bias and model variance components of the expected test mean square error.

The model bias–variance trade-off results in,

• as model complexity increases, model variance generally increases and model bias decreases

• as model complexity decreases, model variance generally decreases and model bias increases

The objective is to select a model complexity that minimizes expected prediction error by balancing,

• sufficient flexibility to capture important patterns in the data

• sufficient constraint to avoid excessive sensitivity to training data variation

Increasing model complexity cannot reduce the irreducible error component of prediction uncertainty.

Used in:

• Tuning Model Hyperparameters

Also see:

• Model Bias

• Model Variance

• Model Bias–Variance Trade-off

Model Checking

A set of critical steps in any spatial modeling workflow to ensure the models are ready to support decision making. Model checking evaluates whether the model honors the available information, accurately predicts known data, and provides a reliable representation of uncertainty.

Examples of model checks include,

1. Model Inputs - data and statistics integration

• check that the model honors the input data and statistical assumptions, generally evaluated over all realizations

• for example, compare output histograms, variograms, and correlation coefficients from the realizations against the input statistics

2. Accurate Spatial Estimates - ability of the model to predict away from available sample data

• evaluate predictive performance using cross validation, where some data are withheld and then predicted by the model

• predictive accuracy is generally summarized with a truth versus predicted cross plot and measures such as mean square error,

\[ MSE = \frac{1}{n} \sum_{\alpha = 1}^{n} \left(z^{*}(\mathbf{u}_{\alpha}) - z(\mathbf{u}_{\alpha}) \right)^2 \]

3. Accurate and Precise Uncertainty Modelss - the uncertainty model is consistent with the amount of information available and the sources of uncertainty

• evaluate uncertainty using cross validation by withholding data and checking whether the observed values occur within the predicted probability intervals at the expected frequency

• summarize uncertainty goodness with observed proportion within interval versus the predicted probability interval

• points on the 45 degree line indicate a good uncertainty model

• points above the 45 degree line indicate an overly conservative uncertainty model, where uncertainty intervals are too wide

• points below the 45 degree line indicate an under-estimated uncertainty model, where uncertainty intervals are too narrow or the model is biased

Used in:

• Model Checking Methods Definitions and Demonstrations

Model Complexity

Model complexity describes the capacity of a predictive machine learning model to fit patterns in data and the difficulty of interpreting the resulting model.

In general,

• increasing model complexity produces a more flexible model that can capture more complex patterns, but makes the model more difficult to interpret and may increase the risk of an overfit model

• decreasing model complexity produces a less flexible model that is easier to interpret, but may not capture important patterns in the data and may increase the risk of underfitting

A variety of concepts may be used to characterize model complexity, including,

• the number of predictor features included in the model, which influences the dimensionality of the feature space and may increase the number of model parameters

• the number of model parameters and the mathematical complexity of the model terms, e.g., linear terms, polynomial terms, thresholds, and interaction terms

• the structure and representation of the model, e.g., a compact equation in polynomial regression, nested conditional rules in decision trees, or thousands of structured weights and biases in neural networks

Examples of increased model complexity include,

• higher-order polynomial regression

• deeper decision trees

• neural networks with more layers and parameters

In general, more flexible models are more difficult to interpret,

• linear regression provides interpretable model coefficients that can be analyzed and used for feature importance or ranking

• support vector machines with radial basis function kernels are linear models in an implicit high-dimensional feature space, but the transformed representation makes interpretation of individual model parameters difficult

Used in:

• Model Complexity vs. Interprebility and Flexibility Discussion

Model Generalization

The ability of a predictive machine learning model to make accurate predictions beyond the specific observations used for training.

Model generalization may be evaluated for different types of prediction cases,

• interpolation cases - predictions within the range of the training data where similar examples are represented in the training dataset

• extrapolation and edge cases - predictions near or beyond the tails of the predictor feature distributions where limited examples are available

• black swan cases - predictions for unforeseen cases that are substantially different from the training data and may represent new regions of the feature space

A model with good generalization,

• learns the underlying structure and relationships in the data rather than memorizing individual training observations

• captures patterns that transfer to new observations and conditions

Models that do not generalize well include,

• overfit models - models that learn training data details or noise, resulting in high training accuracy but reduced performance on testing or new data

• underfit models - models that are too simple or inflexible to represent the underlying phenomenon, resulting in poor performance on both training and testing data

Used in:

• Generalization Strategies in Model Training and Tuning Discussion

Also see:

• Overfit Model

• Underfit Model

Model Hyperparameter

Model settings specified prior to training or estimation that control the structure, flexibility, complexity, or smoothness of a model.

• Hyperparameters are not directly estimated from the training data but are selected using approaches such as validation data, cross-validation, optimization procedures, or expert knowledge.

Examples include,

• regularization strength in regression models

• tree depth and minimum samples per split in decision trees

• nugget effect, range, and sill in variogram models (geostatistics)

• Trend Model order or complexity

• parameters controlling data conditioning, smoothing, or model flexibility in spatial models

For example, polynomial model complexity may be controlled by selecting the polynomial order,

\[ y = b_4 \cdot x^4 + b_3 \cdot x^3 + b_2 \cdot x^2 + b_1 \cdot x + b_0 \]

\[ y = b_3 \cdot x^3 + b_2 \cdot x^2 + b_1 \cdot x + b_0 \]

\[ y = b_2 \cdot x^2 + b_1 \cdot x + b_0 \]

\[ y = b_1 \cdot x + b_0 \]

where the polynomial order is the hyperparameter and the coefficients, \(b_i\), are model parameters estimated from the data.

The selected polynomial order controls model complexity,

• the first-order model is less flexible and has lower complexity

• the fourth-order model is more flexible and has higher complexity

Hyperparameters influence the balance between model flexibility, model bias, model variance, and model generalization.

Used in:

• Model Hyperparameters Discussion

Contrast with:

• Model Parameter

Model Hyperparameter Tuning

Workflow to identify the hyperparameter combination that provides the best predictive performance for a machine learning model,

• for data not used to train the model parameters

• performed with withheld data - testing data with train and test workflow, or validation data for train, validate and test workflows

Unlike model parameters, which are estimated during training, hyperparameters are specified before training and control the model complexity, flexibility, or training process.

The general workflow is,

1. specify a range of candidate hyperparameter values or combinations

2. train a model for each hyperparameter combination

3. evaluate prediction performance with withheld testing data

4. select the hyperparameter combination that minimizes the chosen prediction error metric

5. retrain the final model using the selected hyperparameters

Hyperparameter tuning commonly uses,

• a single train and test split, holdout cross validation

• repeated training and testing splits, such as K-fold cross validation

• out-of-bag error for model bagging-based ensemble methods

Examples of hyperparameters include,

• decision tree – maximum tree depth and minimum samples per leaf

• random forest – number of trees and number of candidate predictor features considered at each split

• ridge and LASSO regression – regularization parameter, \(\lambda\)

• support vector machine – cost parameter, \(C\), and kernel-specific hyperparameters

• artificial neural network – learning rate, batch size, number of hidden layers, number of nodes per layer, activation function, and number of training epochs

Additional comments,

• hyperparameter tuning estimates model complexity appropriate for the available data

• increasing model complexity generally reduces training error but may increase testing error due to overfitting

• the optimal hyperparameters are those that provide the best prediction accuracy on previously unseen data

• hyperparameter tuning and model evaluation should be based on the same prediction metric, such as MAE, MSE, RMSE, or classification accuracy

Used in:

• Hyperparameter Tuning Ridge Regression Demonstration

• Hyperparameter Tuning LASSO Regression Demonstration

• Hyperparameter Tuning k-Nearest Neighbours Demonstration

• Hyperparameter Tuning Decision Tree Demonstration

• Hyperparameter Tuning Random Forest Demonstration

Also see:

• Cross Validation

Contrast with:

• K-fold Cross Validation

Also see:

• Train and Test Split

• Cross Validation

• Model Parameter

• Overfit Model

• Underfit Model

• Random Forest

• Ridge Regression

• LASSO Regression

Model Parameter

Quantities estimated from data that define a model and control its fit to training data observations.

Model parameters are typically obtained through a process known as model parameter training based on,

• optimization

• analytical solutions

• statistical estimation

with methods such as,

• least squares

• maximum likelihood

• kriging.

Examples include regression coefficients, covariance values, and trend coefficients.

For example, for a polynomial model,

\[ y = b_3 \cdot x^3 + b_2 \cdot x^2 + b_1 \cdot x + b_0 \]

where \(b_3\), \(b_2\), \(b_1\), and \(b_0\) are model parameters.

Used in:

• Model Parameters Discussion

Contrast with:

• Model Hyperparameter

Model Parameter Training

Workflow to estimate the model parameters that minimize a specified loss function over the training data.

Model parameter training seeks parameter values that optimize a specified objective,

• minimizing a loss function

• maximizing a likelihood function

• satisfying analytical estimation equations, when available

Common parameter training methods include,

• analytical solutions, such as least squares

• iterative optimization using gradient-based optimization

• Bayesian estimation using posterior probability distributions

Unlike hyperparameters, which are specified before training, model parameters are learned automatically from the training data.

The general workflow is,

1. initialize the model parameters

2. calculate model predictions over the training data

3. evaluate the loss function

4. update the model parameters to reduce the loss

5. repeat until convergence or a stopping criterion is satisfied

The parameter training method depends on the prediction model,

• linear regression - analytical least-squares solution

• ridge regression - analytical regularized least-squares solution

• LASSO regression - iterative optimization

• decision tree - recursive partitioning of the predictor feature space

• support vector machine - constrained optimization

• artificial neural networks - backpropagation with gradient-based optimization

Examples of model parameters include,

• linear regression - feature weights and intercept

• decision tree - split locations and response values within terminal regions

• random forest - the parameters of each decision tree in the ensemble

• support vector machine - hyperplane coefficients

• artificial neural network - connection weights and node biases

Additional comments,

• model parameters are estimated separately for every realization of model training

• parameter training minimizes the specified loss function for fixed hyperparameter values

• changing the hyperparameters generally changes the resulting model parameters

• model parameter training and model hyperparameter tuning are typically repeated together until a satisfactory prediction model is obtained

Used in:

• Model Parameter Training for Linear Regression

• Model Parameter Training for Ridge Regression

• Model Parameter Training for k-Nearest Neighbours

• Model Parameter Training for Decision Tree

Also see:

• Model Parameter Training

• Model Hyperparameter Tuning

• Loss Function

• Model Parameter

• Gradient-based Optimization

• Backpropagation

Contrast with:

• Model Hyperparameter

• Model Hyperparameter Tuning

Model Variance

The component of the expected testing mean square error caused by a predictive model being overly sensitive to variations in the training data.

The bias–variance decomposition of the expected testing mean square error is,

\[ \mathbb{E}\!\left[\left(y_0-\hat{f}(x_{0,1}^{(i)},\ldots,x_{0,m}^{(i)})\right)^2\right] = \underbrace{ \left(\mathbb{E}\!\left[\hat{f}(x_{0,1}^{(i)},\ldots,x_{0,m}^{(i)})\right] - f(x_{0,1}^{(i)},\ldots,x_{0,m}^{(i)})\right)^2}_{\text{Model Bias}^2} + \underbrace{\mathbb{E}\!\left[\left(\hat{f}(x_{0,1}^{(i)},\ldots,x_{0,m}^{(i)})- \mathbb{E}\!\left[\hat{f}(x_{0,1}^{(i)},\ldots,x_{0,m}^{(i)})\right]\right)^2\right]}_{\text{Model Variance}} + \underbrace{\sigma_{\epsilon}^{2}}_{\text{Irreducible Error}} \]

where,

• model variance represents the variability in model predictions that arises from changes in the training data

• high model variance indicates that the model is sensitive to the particular observations used for training

• high model variance is commonly associated with an overfit model

Model hyperparameter tuning is used to balance model variance and model bias.

• model bias–variance trade-off - as model complexity increases, model variance generally increases while model bias decreases, and vice versa

Used in:

• Tuning Model Hyperparameters

Also see:

• Model Bias

• Model Variance

• Model Bias–Variance Trade-off

Momentum

An optimization technique that improves the stability and convergence of gradient descent by combining the current optimization step with information from previous optimization steps.

• momentum, \(\lambda\), controls how much of the previous optimization step is retained, while \(1-\lambda\) determines the influence of the current gradient step

A momentum update may be written as,

\[ s_t=\lambda s_{t-1}-(1-\lambda)\,r\,\nabla L_t \]

followed by the model parameter update,

\[ \theta_{t+1}=\theta_t+s_t \]

where,

• \(\theta\) represents the model parameters

• \(s_t\) is the optimization step at iteration \(t\)

• \(r\) is the learning rate

• \(\nabla L_t\) is the gradient of the loss function at iteration \(t\)

• \(\lambda\) is the momentum coefficient, typically between 0 and 1

The optimization step is often referred to as the velocity in the machine learning literature because momentum was originally derived from an analogy with Newtonian mechanics, where model parameters correspond to position and gradients act as forces.

• since optimization iterations occur at constant time intervals, the time increment is effectively unity, and the velocity is proportional to the optimization step.

• consequently, the terms velocity and optimization step are mathematically equivalent in this context.

Momentum provides several benefits,

• reduces the effect of noisy gradient estimates

• smooths the optimization trajectory by averaging successive optimization steps

• reduces zig-zag motion across narrow valleys of the loss function

• accelerates convergence in directions with consistent gradients

• helps the optimization continue through shallow local irregularities while remaining directed toward regions of lower loss

Used in:

• Momentum for Gradient Descent-based Training Neural Network

Also see:

• Gradient Boosting Model

• Gradient-based Optimization

Monte Carlo Method

Algorithm that solve computational problems through repeated random sampling and statistical analysis of the resulting samples.

• Monte Carlo methods approximate quantities that may be difficult to calculate analytically by using simulated random realizations from an underlying probability distribution. Common examples include,

• Numerical integration - estimating an integral by sampling random values,

\[ I=\int_a^b f(x)dx \]

using,

\[ I \approx (b-a)\frac{1}{N}\sum_{i=1}^{N}f(x_i) \]

• Uncertainty simulation - propagating uncertainty through a model by repeatedly sampling uncertain input features and evaluating the resulting outcomes,

\[ Y=f(X_1,X_2,\ldots,X_m) \]

to estimate statistics such as expected values, confidence intervals, and probability of exceeding thresholds.

• Bootstrap resampling - estimating uncertainty in statistical quantities by repeatedly sampling from an empirical distribution.

Monte Carlo methods are extended to,

• Markov Chain Monte Carlo, which uses a stochastic process to generate samples from complex probability distributions when direct sampling is not feasible.

Used in:

• Monte Carlo Simulation Definitions

• Propagating Uncertainty with the Monte Carlo Simulation Workflow

Also see:

• Monte Carlo Simulation

• Monte Carlo Simulation Workflow

• Bootstrap

Monte Carlo Simulation

A method for generating random samples from one or more statistical distributions. A random sample from a distribution is defined as a random variable, \(X\). The steps for Monte Carlo simulation are:

1. Model the feature cumulative distribution function, \(F_x(x)\).

2. Draw a random value from a uniform \([0,1]\) distribution, representing a random cumulative probability value, \(p^{\ell}\).

3. Apply the inverse cumulative distribution function to calculate the associated sample value,

\[ x^{\ell} = F_x^{-1}(p^{\ell}) \]

4. Repeat steps 2 and 3 to calculate enough realizations for subsequent analysis.

Monte Carlo simulation is a fundamental building block of stochastic simulation and uncertainty workflows. Examples include,

• Monte Carlo simulation workflow - apply Monte Carlo simulation over all uncertain features, then apply a transfer function to calculate a realization of the decision criteria. Repeat this process to generate many realizations and propagate uncertainty through the transfer function.

• Bootstrap - applies Monte Carlo simulation to generate realizations of the sample data, allowing estimation of uncertainty in sample statistics or ensembles of prediction models for ensemble-based machine learning.

• Monte Carlo method - use random sampling to approximate solutions to complex problems, with the solution generally converging as the number of random samples increases.

Used in:

• Monte Carlo Simulation Definitions

• Propagating Uncertainty with the Monte Carlo Simulation Workflow

Also see:

• Monte Carlo Method

• Monte Carlo Simulation Workflow

• Bootstrap

Monte Carlo Simulation Workflow

A general stochastic Monte Carlo simulation workflow for propagating uncertainty through a transfer function. The workflow includes the following steps,

1. Model the uncertainty distributions or cumulative distribution functions for all input features,

\[ F_{x_1}(x_1), \quad F_{x_2}(x_2), \quad \dots \quad , F_{x_m}(x_m) \]

2. Monte Carlo simulate realizations for all input features,

\[ x_1^{\ell}, \quad x_2^{\ell}, \quad \ldots \quad , x_m^{\ell} \]

3. Apply the transfer function to calculate a realization of the output, often the decision criteria,

\[ y^{\ell} = f \left(x_1^{\ell},x_2^{\ell}, \quad \ldots \quad, x_m^{\ell} \right) \]

4. Repeat steps 2 and 3 to calculate enough realizations to model the output uncertainty distribution,

\[ F_y(y) \]

The input feature realizations may be simulated independently or with relationships between features included through multivariate uncertainty models.

Used in:

• Monte Carlo Simulation Definitions

• Propagating Uncertainty with the Monte Carlo Simulation Workflow

Also see:

• Monte Carlo Simulation

• Bootstrap

Multidimensional Scaling

Machine learning method in inferential statistics and information visualization for exploring the similarity (or conversely the dissimilarity) between samples from a high-dimensional dataset.

Multidimensional scaling (MDS) projects data from \(m\) dimensions to a lower-dimensional space with \(p \ll m\),

• while attempting to preserve the pairwise distances or dissimilarities between samples

• ideally projecting to \(p=2\) or \(3\) dimensions for visualization and exploration

Unlike methods that operate directly on predictor features, multidimensional scaling requires only the pairwise distances or dissimilarities between samples.

For multidimensional scaling,

• the original feature values are not required; only the distance or dissimilarity matrix between samples

• as with any distance-based method, feature standardization is often applied so that features with larger variance do not dominate the distance calculations

• a variety of distance or dissimilarity measures may be used depending on the application

Comparison between multidimensional scaling and principal component analysis,

• Principal Component Analysis (PCA) - operates on the covariance matrix (\(m \times m\)) of the predictor features and finds orthogonal linear projections that maximize explained variance

• Multidimensional Scaling (MDS) - operates on the pairwise distance matrix (\(n \times n\)) between samples and finds a low-dimensional representation that best preserves those pairwise distances

The resulting low-dimensional representation provides new opportunities for data exploration, including,

• visualization of clusters and sample relationships

• analog selection based on sample proximity

• visualization of the diversity and coverage of training data

• visualization and conceptualization of uncertainty space

• identification of unusual observations and potential outliers

Used in:

• Multidimensional Scaling Description and Demonstration

Also see:

• Principal Component Analysis

Multiple Linear Regression

A regression model that extends linear regression by predicting a response feature from multiple predictor features using a linear combination of the predictor feature(s).

The general form is,

\[ \hat{Y} = \beta_0 + \beta_1X_1 + \beta_2X_2 + \ldots + \beta_pX_p \]

where \(\hat{Y}\) is the predicted response, \(\beta_0\) is the intercept, \(\beta_1,\ldots,\beta_p\) are regression coefficients, and \(X_1,\ldots,X_p\) are predictor features.

Multiple linear regression is widely used in statistical modeling and machine learning to quantify relationships between a response variable and multiple explanatory features, including,

• feature selection - regression coefficients (\(\beta\)) and standardized predictor feature effects

• inference - significance testing of linear relationships

• bivariate statistics - partial correlation coefficients

Used in:

• Multiple Linear Regression Exercise

Also see:

• Linear Regression

• Ridge Regression

• Polynomial Regression

• Model Parameter

Multiplication Rule

The joint probablity of \(A\) and \(B\) as the product of the conditional probability of \(B\) given \(A\) with the marginal probability of \(A\),

\[ P(A \cap B) = P(A,B) = P(B|A) \cdot P(A) \]

The multiplication rule is axiomatic as it is derived as a simple manipulation of the definition of conditional probability, in this case,

\[ P(B|A) = \frac{P(A,B)}{P(A)} \]

and the definition of conditional probability is readily obseved from a simple Venn diagram.

Used in: Probability Multiplication Rule Defintion and Demonstration

Multivariate

Involving more than two features (variables) considered together, often to study their relationships, dependence, or correlation.

For examples, see multivariate analysis.

Used in:

• Multivariate Analysis

Compare with:

• Univariate

• Bivariate

Multivariate Analysis

The analysis of more than two features (variables) measured over a collection of samples to investigate their relationships, dependence, and correlation.

Examples include:

• multivariate statistical models such as multiple linear regression

• scatter plot matrices showing relationships among multiple features

• covariance and correlation coefficient matrices

• principal component analysis

A common approach to multivariate analysis is to evaluate pairwise relationships among features using covariance, correlation, and scatter plots.

• note, this is a simplified multivariate analysis because it considers only pairwise relationships among features. Higher-order relationships involving three or more features simultaneously are not explicitly modeled.

Used in:

• Multivariate Analysis

• Multivariate Analysis with Correlation Matrix Description and Demonstration

Also see:

• Multivariate

Compare with:

• Univariate Statistic

• Bivariate Statistic

Mutual Information

Generalized statistic from information theory for quantifying statistical dependence between features without assuming a specific functional form.

Mutual information,

• quantifies the amount of information gained about one feature by observing another feature

• measures statistical dependence without assuming a linear or other parametric relationship

• is always non-negative

• is zero only when the two features are statistically independent

• is measured in bits when the logarithm is base 2 (or nats when the natural logarithm is used)

The calculation of mutual information is based on,

• comparing the observed joint probability with the joint probability expected if the features were statistically independent

• quantifying the departure from independence by comparing the joint probability, \(P(x,y)\), with the product of the marginal probabilities, \(P(x)\cdot P(y)\)

For statistically independent features,

\[ P(x,y) = P(x)\cdot P(y) \]

For discrete (or binned continuous) features, mutual information is calculated as,

\[ I(X;Y)=\sum_{y\in Y}\sum_{x\in X}P_{X,Y}(x,y)\log_2\left(\frac{P_{X,Y}(x,y)}{P_X(x)\,P_Y(y)}\right) \]

If the two features are statistically independent,

\[ \frac{P_{X,Y}(x,y)}{P_X(x)\,P_Y(y)}=1 \]

and therefore,

\[ \log_2\left(\frac{P_{X,Y}(x,y)}{P_X(x)\,P_Y(y)}\right)=0 \]

The joint probability, \(P_{X,Y}(x,y)\), acts as a weighting term, so regions of the feature space that occur more frequently contribute more to the mutual information than rare combinations.

For continuous (non-binned) features, the integral form is,

\[ I(X;Y)=\int_Y\int_XP_{X,Y}(x,y)\log_2\left(\frac{P_{X,Y}(x,y)}{P_X(x)\,P_Y(y)}\right)dx\,dy \]

Larger mutual information values indicate stronger statistical dependence between features, regardless of whether the relationship is linear or nonlinear.

Unlike the Correlation Coefficient, which measures only linear association, mutual information detects both linear and nonlinear statistical dependence.

Used in:

• Description and Demonstration of Mutual Information for Feature Ranking

Also see:

• Correlation Coefficient

Mutually Exclusive Events

Events that cannot occur together; they have no common outcomes. Using set notation, events \(A\) and \(B\) are mutually exclusive if,

\[ A \cap B = \{x: x \in A \text{ and } x \in B \} = \emptyset \]

Therefore, the probability of the intersection of mutually exclusive events is,

\[ P(A,B) = P(A \cap B) = 0.0 \]

For mutually exclusive events, the probability of a union simplifies to the sum of the individual probabilities,

\[ P(A \cup B) = P(A) + P(B) \]

Used in:

• Probability Operations Itemization and Demonstration

Naive Bayes

A classification machine learning model based on Bayesian updating with the simplifying assumption of conditional independence between predictor features.

The objective is to estimate the probability of a category, \(C_k\), given \(n\) predictor features,

\[ P(C_k|x_1,\ldots,x_n) \]

Using Bayes’ theorem,

\[ P(C_k|x_1,\ldots,x_n)=\frac{P(x_1,\ldots,x_n|C_k)P(C_k)}{P(x_1,\ldots,x_n)} \]

where,

• \(P(C_k)\) is the prior probability of category \(k\)

• \(P(x_1,\ldots,x_n|C_k)\) is the likelihood function of observing the features given category \(k\)

• \(P(x_1,\ldots,x_n)\) is the evidence term

The likelihood term can be expanded using the chain rule,

\[ P(x_1,\ldots,x_n|C_k)=P(x_1|x_2,\ldots,x_n,C_k)P(x_2|x_3,\ldots,x_n,C_k)\ldotsP(x_n|C_k) \]

The full likelihood requires knowledge of the joint conditional relationships between all predictor features. As the number of features increases, estimating this joint distribution requires increasingly large datasets.

The naive Bayes approach makes the simplifying assumption that all predictor features are conditionally independent given the category,

\[ P(x_i|x_{i+1},\ldots,x_n,C_k)=P(x_i|C_k) \]

for all \(i=1,\ldots,n\) features.

The likelihood then simplifies to,

\[ P(x_1,\ldots,x_n|C_k)=\prod_{i=1}^{n}P(x_i|C_k) \]

and the posterior probability becomes,

\[ P(C_k|x_1,\ldots,x_n)=\frac{P(C_k)\prod_{i=1}^{n}P(x_i|C_k)}{P(x_1,\ldots,x_n)} \]

The evidence term, \(P(x_1,\ldots,x_n)\), depends only on the observed features and is constant across categories. Therefore, classification can be performed by comparing the unnormalized posterior probabilities,

\[ P(C_k)\prod_{i=1}^{n}P(x_i|C_k) \]

and normalizing the results across all possible categories.

The naive Bayes approach,

• requires only prior probabilities, \(P(C_k)\), and individual conditional distributions, \(P(x_i|C_k)\), rather than the full joint feature distribution

• is computationally efficient and practical for high-dimensional problems

• performs well with relatively small datasets because fewer probability distributions must be estimated

Although the conditional independence assumption is often unrealistic, naive Bayes can still provide effective classification performance when the assumption is approximately valid or when the simplified model generalizes better than a more complex alternative.

Used in:

• Naive Bayes Description and Demonstration

Also see:

• Prior

• Likelihood Function

• Posterior

• Bayesian Updating for Classification

Neural Network

A flexible, nonlinear machine learning model inspired by the structure of biological nervous systems.

The biological analogy includes,

• neurons \(\rightarrow\) nodes

• synapses \(\rightarrow\) weighted connections between nodes

Feed-forward fully connected neural networks consist of,

• input layer - receives the predictor features

• hidden layers - progressively transform the information

• output layer - produces one or more response feature(s), prediction(s)

Information flows forward through the network,

• each node forms a weighted sum of the incoming information, adds a bias, and applies an activation function,

\[ a_j=g\left(\sum_{i=1}^{m}w_{ij}x_i+b_j\right) \]

where \(w_{ij}\) are the connection weights, \(b_j\) is the node bias, and \(g(\cdot)\) is the activation function.

This process is repeated over all hidden layers until the output layer produces the prediction.

Flexible neural network architectures include,

• fully connected neural network (FCNN) - every node is connected to every node in the next layer

• convolutional neural network (CNN) - convolution operators and feature maps summarize local spatial patterns while preserving spatial relationships

• recurrent neural network (RNN) - feedback connections provide memory of previous inputs for sequential or temporal data

• autoencoder (AE) - bottleneck architecture learns a compact latent representation before reconstructing the original data

Training neural networks,

• model parameters consist of all connection weights and node biases

• model parameters are estimated by backpropagation, where gradients of the loss function are propagated backward through the network

• gradient-based optimization updates the model parameters to minimize the loss function

Neural networks are characterized by,

• universal function approximators - with sufficient complexity they can approximate a wide variety of nonlinear relationships

• high model complexity - often containing thousands to millions of trainable parameters

• low interpretability - predictions arise from many interacting parameters and information pathways

Used in:

• Neural Network Description and Demonstration

Also see:

• Backpropagation

• Activation Function

• Convolutional Neural Network

• Recurrent Neural Network

• Autoencoder

ndarray

The fundamental N-dimensional array data structure in the Python numerical computing package, NumPy, used to efficiently store and manipulate collections of numerical data.

An ndarray represents data organized along one or more dimensions,

• 1D arrays - vectors

• 2D arrays - tables, images, or maps

• 3D arrays - volumes, time series of maps, or subsurface models

• higher-dimensional arrays - tensors and multidimensional scientific datasets

For geoscience applications, ndarray provides a convenient structure for working with exhaustive, regularly spaced gridded data over a 2D area of interest or 3D volume of interest, representing maps and models.

Advantages of ndarray include,

• efficient storage, access, and manipulation of multidimensional numerical data

• vectorized mathematical operations without requiring explicit loops

• built-in methods to calculate multidimensional summary statistics

• built-in methods for data queries, filtering, and conditional selection

• built-in methods for data manipulation, cleaning, reshaping, and reformatting

• support for loading and converting data from a variety of file formats and Python objects

• attributes describing the array structure, including size, shape, number of dimensions, and data type

An ndarray contains,

• the array values stored in a multidimensional structure

• metadata describing the array dimensions and numerical representation

Used in:

• Specifying the Model Grid Discussion and Demonstration

Also see:

• DataFrame

Node

There are 3 distinct uses for the term node,

1. entity or object in a graph that represents a point connected to other nodes through edges.

2. component of a decision tree representing either a decision point that splits data based on a feature value or a terminal point that provides a prediction.

3. basic computational unit of a neural network that receives information from connected nodes, performs a computation, and passes the result to other nodes.

A neural network is a graph, where each node,

• receives weighted inputs through one or more connections

• combines the inputs, typically by computing a weighted sum with a bias

• applies an activation function to produce an output

Nodes are organized into layers,

• input layer - receives the predictor feature values

• hidden layers - transform information through learned computations

• output layer - pproduces one or more response feature(s), prediction(s)

Used in:

• Node Description and Demonstration

• Nodes for CNNs Description and Demonstration

• Nodes Applied in an Autoencoder

• Nodes Applied in a GAN

Also see:

• Neural Network

• Connection

• Weight

• Activation Function

Nonparametric Model

A predictive machine learning model that does not assume a predefined functional form with a fixed number of model parameters.

Instead, a nonparametric model,

• learns the structure and shape of the relationship from the training data

• provides greater flexibility to represent complex and nonlinear natural systems

• allows model complexity to increase with the amount of available data

Unlike parametric models, nonparametric models do not define the model complexity in advance by specifying a fixed mathematical relationship.

Nonparametric models often require more data because,

• they estimate more flexible relationships directly from observations

• the effective number of model parameters can increase with the amount of training data

• greater flexibility can increase the risk of an overfit model without sufficient data or appropriate regularization

Examples of nonparametric models include,

• k-Nearest Neighbours

• Decision Tree

• Kernel-based methods such as support vector machine

• Neural Network

Contrast with:

• Parametric Model

Norm

Norm of a vector maps the vector components \([1,\ldots,n]\) to a single summary measure in the range \([0,\infty)\) that indicates,

• the size or length of the vector

To train predictive machine learning models to training data, we require a single summary measure of mismatch between the model predictions and the training observations, called the training error.

The error is observed at each training data location,

\[ \Delta y_i = y_i - \hat{y}_i, \quad \forall \quad i=1,\ldots,n \]

and together these errors form an error vector,

\[ \Delta y = [\Delta y_1,\ldots,\Delta y_n] \]

The norm of the error vector, called the error norm, provides,

• a single value to summarize the mismatch over all training data observations

• an loss function that can be minimized during model training

There are a variety of norms that may be applied in machine learning. The general \(p\)-norm (Minkowski norm) for a vector \(\mathbf{x}\) is,

\[ \|\mathbf{x}\|_p=\left(\sum_{i=1}^{n}|x_i|^p\right)^{\frac{1}{p}} \]

where,

• \(p\) controls the type of norm

• \(n\) is the number of dimensions in the vector

Commonly applied error norm cases include,

• \(p=1\) - \(L_1\) norm (Manhattan norm)

\[ \|\mathbf{x}\|_1=\sum_{i=1}^{n}|x_i| \]

• \(p=2\) - \(L_2\) norm (Euclidean norm)

\[ \|\mathbf{x}\|_2=\sqrt{\sum_{i=1}^{n}x_i^2} \]

• \(p\rightarrow\infty\) - \(L_{\infty}\) norm (Chebyshev norm)

\[ \|\mathbf{x}\|_{\infty}=\max_i |x_i| \]

Norm selection has an important impact the training of our model parameters, here’s a comparison of commonly applied error norms,

Property	Least Absolute Deviations (L1)	Least Squares (L2)
Objective	Minimize absolute errors	Minimize squared errors
Robustness to outliers	Robust	Less robust
Solution stability	May be unstable	Stable
Number of solutions	Possibly multiple optimal solutions	Unique analytical solution*
Feature selection	Built-in (promotes sparsity)	No feature selection
Model parameters	Sparse solutions	Non-sparse solutions
Analytical solution	Generally not available	Available for linear regression

* Assuming the predictor features are linearly independent.

Used in:

• Norm Discussion and Comparison

• Comparison of L1 and L2 Norms

Also see:

• Loss Function

Normalization

A distribution rescaling method that transforms feature values to a specified range, commonly \([0,1]\).

Min-max normalization can be interpreted as shifting and stretching or squeezing a univariate distribution (e.g., histogram) to enforce a minimum value of 0.0 and a maximum value of 1.0,

\[ y_i=\frac{x_i-\min(x)}{\max(x)-\min(x)}, \quad \forall \quad i=1,\ldots,n \]

This transformation is linear and therefore,

• preserves the rank ordering of observations

• does not change the relative shape of the distribution

• changes the scale and location of the feature values

Normalization is useful when feature magnitude influences model behavior, for example,

• distance-based methods where large-scale features can dominate distance calculations, such as k-Means Clustering clustering and k-Nearest Neighbours

• models where feature coefficients are compared for interpretation or feature ranking

• artificial neural networks where input scaling improves numerical optimization and avoids reduced sensitivity caused by activation function saturation

Normalization is applied to predictor features and may also be applied to response features, with inverse transformation used to return predictions to the original units.

Used in:

• Feature Normalization Discussion and Demonstration

Also see:

• Standardization

Normalized Histogram

A bar chart of the univariate statistical distribution with probability over an exhaustive set of bins over the range of possible values. These are the steps to build a normalized histogram,

1. Divide the continuous feature range of possible values into \(K\) equal size bins, \(\delta x\):

\[ \Delta x = \left( \frac{x_{max} - x_{min}}{K} \right) \]

or use available categories for categorical features.

2. Count the number of samples (frequency) in each bin, \(n_k\), \(\forall k=1,\ldots,K\)

3. Divide each by the total number of data, \(n\), to calculate the probability of each bin,

\[ p_k = \frac{n_k}{n}, \forall \quad k = 1,\ldots,L \]

4. Plot the probability vs. the bin label (use bin centroid if continuous)

Additional comments:

• step 3 converts a standard histogram to a normalized histogram with a y-axis of probability instead of frequency

• for categorical features, a normalized histogram represents an empirical probability mass function

• for continuous features, the normalized histogram represents the empirical probability of each bin interval and is an approximation to the underlying continuous distribution

Used in:

• Normalized Histogram Description and Demonstration

Also see:

• Histogram

One-Hot Encoding

A feature transformation method applied to categorical features that converts each category into a binary indicator vector.

Given a categorical feature with \(K\) possible categories (cardinality \(K\)), one-hot encoding creates a vector of length \(K\) where,

• a value of 1 indicates that the sample belongs to that category

• a value of 0 indicates that the sample does not belong to that category

For example, a categorical feature with three possible outcomes,

\[ \text{Rock Type}=[\text{Sandstone},\text{Shale},\text{Limestone}] \]

is transformed as,

\[ \text{Sandstone}=[1,0,0] \]

\[ \text{Shale}=[0,1,0] \]

\[ \text{Limestone}=[0,0,1] \]

One-hot encoding is equivalent to applying a categorical indicator transformation.

One-hot encoding is used for,

• integrating nominal categorical features into machine learning prediction models

• avoiding the incorrect assumption of ordering between categorical outcomes

• providing numerical feature representations required by many machine learning algorithms

Considerations include,

• the number of predictor features increases from one categorical feature to \(K\) binary indicator features

• high-cardinality categorical features may create large numbers of sparse predictor features

Used in:

• One-hot Encoding Description and Demonstration

Also see:

• k-bins Discretization

• categorical feature

Out-of-Bag

In bootstrap resampling, each model realization is trained using a sample of the original data selected with replacement.

For a bootstrap sample of size \(n\), approximately \(\frac{2}{3}\) of the original observations are included in expectation,

\[ 1-\left(1-\frac{1}{n}\right)^n \rightarrow 1-e^{-1}\approx0.632 \]

and approximately \(\frac{1}{3}\) of the observations are not selected.

These unused observations are called out-of-bag (OOB) samples or observations.

For model bagging-based ensemble prediction models,

• each model realization has a unique set of out-of-bag observations

• each response observation, \(y_{\alpha}\), receives predictions from only the ensemble members where that observation was out-of-bag

• with \(B\) bootstrap realizations, each observation receives approximately \(\frac{B}{3}\) out-of-bag predictions, \(\hat{y}^{*,b}_{\alpha}\)

The out-of-bag predictions are aggregated to calculate a single out-of-bag prediction,

• average for regression

• mode for classification

For regression,

\[ \hat{y}^{OOB}_{\alpha}=\frac{1}{B_{\alpha}}\sum_{b\in OOB_{\alpha}}\hat{y}^{*,b}_{\alpha} \]

where \(B_{\alpha}\) is the number of bootstrap models where observation \(\alpha\) was out-of-bag.

The out-of-bag mean square error is calculated as,

\[ MSE_{OOB}=\frac{1}{n}\sum_{\alpha=1}^{n}\left(\hat{y}^{OOB}_{\alpha}-y_{\alpha}\right)^2 \]

Out-of-bag error provides an internal validation estimate for bootstrap and model bagging-based ensemble models.

Advantages include,

• avoids the need for a separate training and testing split

• provides an estimate of model generalization during model hyperparameter tuning

• uses all available observations for model training across the ensemble

Considerations include,

• the effective validation proportion is approximately fixed at approximately 1/3

• the OOB validation samples are generated through bootstrap sampling and may not represent the difficulty of the intended model application

• a separate validation dataset may still be required when the deployment conditions differ substantially from the training data

Used in:

• Out-of-Bag Cross Validation Description and Demonstration

Also see:

• Model Bagging

• Random Forest

Outlier

An observation that differs substantially from the majority of the available samples and may have an unusually large influence on statistical analysis or machine learning models.

Outliers may result from,

• measurement or recording errors

• rare but valid observations

• previously unobserved processes or populations

Important considerations include,

• sensitivity - methods based on squared errors, such as mean square error and the L2 norm, are particularly sensitive to outliers

• detection - identifying outliers requires consideration of the data distribution, context, and application rather than a single universal criterion

Used in:

• Outlier Impact on Correlation Coefficients

Contrast with:

• Inlier

Overfit Model

A predictive machine learning model that learns the noise, sampling variability, or specific idiosyncrasies of the training data rather than the underlying relationship in the natural system.

An overfit model demonstrates,

• high prediction accuracy with training data but poor prediction accuracy with withheld testing data

• excessive dependence on the specific training observations, effectively memorizing aspects of the training dataset

During model hyperparameter tuning, the overfit region is characterized by,

• increasing model complexity that continues to reduce training error while increasing testing error

• a divergence between training and testing performance as the model becomes increasingly flexible

This behavior represents the high model variance region of the model bias–variance trade-off.

Issues associated with an overfit machine learning model include,

• more model complexity and flexibility than can be justified by the available data quantity, accuracy, frequency, and coverage

• high accuracy during training but poor accuracy during testing, indicating limited ability to generalize to new cases

Overfitting may be reduced through,

• reducing model complexity

• increasing the amount or quality of training data

• applying regularization or other constraints on model flexibility

• improving validation methods to better represent the intended model application

Used in:

• Overfit and Underfit Discussion

Contrast with:

• Underfit Model

Parameter

A numerical quantity that describes a population or probability model.

Examples include,

• population average

• population variance or standard deviation

• distribution parameters, such as the mean and variance of a Gaussian distribution

• variogram parameters, such as the nugget effect, range, and sill

Population parameters are generally unknown because the entire population is rarely observed. Instead, they are inferred from available sample statistics.

Used in:

• TBA

Also see:

• Model Parameter

Contrast with:

• Statistic

Parametric Model

Machine Learning Concepts: a model that makes an assumption about the functional form, shape of the natural system.

• we gain simplicity and advantage of only a few parameters

• for is a linear model we only have \(m+1\) model parameters

There is a risk that our model is quite different than the natural setting, resulting in a poor model, for example, a linear model applied to a nonlinear phenomenon.

Used in:

• Parameteric Model Training for Linear Regression

• Parameter Model Training for Ridge Regression

Constrast with:

• Nonparametric Model

Partial Correlation Coefficient

Correlation analysis metric that quantifies the correlation between \(X\) and \(Y\) while controlling for the linear influence of other features, \(Z_1,\ldots,Z_{m-2}\), on both \(X\) and \(Y\).

The \(m-2\) notation accounts for removing the two features of interest, \(X\) and \(Y\), from the complete set of \(m\) features.

The partial correlation coefficient is written as,

\[ \rho_{X,Y\cdot Z_1,\ldots,Z_{m-2}} \]

and measures the remaining linear association between \(X\) and \(Y\) after removing the linear contribution of the control features.

To calculate the partial correlation coefficient, the following steps are applied,

1. Perform linear least-squares regression to predict \(X\) from the control features,

\[ X^*=f(Z_1,\ldots,Z_{m-2}) \]

where \(X^*\) is the estimated value of \(X\) from the regression model.

2. Perform linear least-squares regression to predict \(Y\) from the control features,

\[ Y^*=f(Z_1,\ldots,Z_{m-2}) \]

where \(Y^*\) is the estimated value of \(Y\) from the regression model.

3. Calculate the residuals from the regression of \(X\),

\[ X-X^* \]

4. Calculate the residuals from the regression of \(Y\),

\[ Y-Y^* \]

5. Calculate the correlation coefficient between the residuals,

\[ \rho_{X-X^*,Y-Y^*} \]

This correlation between residuals represents the partial correlation between \(X\) and \(Y\) after controlling for \(Z_1,\ldots,Z_{m-2}\).

Interpretation,

• a partial correlation near 1 indicates a strong positive linear relationship remaining between \(X\) and \(Y\) after removing the influence of the control features

• a partial correlation near -1 indicates a strong negative linear relationship remaining between \(X\) and \(Y\)

• a partial correlation near 0 indicates little remaining linear relationship after controlling for the other features

Assumptions of the partial correlation coefficient include,

• \(X,Y,Z_1,\ldots,Z_{m-2}\) have approximately linear relationships

• no significant univariate or bivariate outliers, since partial correlation is sensitive to extreme values similar to regular correlation

• for statistical inference, approximately Gaussian distributed variables and homoscedastic linear relationships provide the most reliable interpretation

Partial correlation removes only the linear influence of the control features. Nonlinear relationships between the variables are not captured by the standard partial correlation coefficient,

• extensions based on nonlinear residualization models are possible, but these are no longer the standard partial correlation coefficient.

Used in:

• Partial Correlation Coefficients for Feature Ranking Description and Demonstration

Also see:

• Correlation Coefficient

• Multiple Linear Regression

Partitional Clustering

A family of clustering methods that divides a dataset into a specified number of non-overlapping groups, producing a single partition of the data.

Partitional clustering methods,

• assign each sample to one cluster group

• optimize a single clustering solution rather than producing multiple nested solutions

For example, k-Means clustering is a partitional clustering method that iteratively updates cluster assignments and cluster prototypes (centroids) while optimizing a single partition of the data.

Compared with hierarchical clustering,

• partitional clustering produces one final clustering solution

• hierarchical clustering produces a hierarchy of nested cluster solutions that can be viewed at different levels of grouping

Examples of partitional clustering methods include,

• k-Means clustering - groups samples by minimizing the distance between samples and a specified number of cluster prototypes (centroids)

• k-Medoid clustering - similar to k-Means clustering, but cluster prototypes are selected from actual samples

• k-Modes clustering - extension of k-Means clustering for categorical data using category-based similarity measures

A helpful summary of clustering methods,

• partitional clustering - find one best partition of the data

• hierarchical clustering - build a tree of nested clusters

• density-based clustering - identify regions of high sample density separated by lower-density regions

Used in:

• Partitional Clustering with K-Means Clustering Description and Demonstration

Also see:

• k-Means Clustering

Contrast with:

• Density-based Clustering

• Hierarchical Clustering

Polygonal Declustering

A declustering method to assign weights to spatial samples based on local sampling density, such that the weighted statistics are likely more representative of the population. Data weights are assigned so that,

• samples in densely sampled areas receive less weight

• samples in sparsely sampled areas receive more weight

Polygonal declustering proceeds as follows:

1. Split up the area of interest with Voronoi polygons. These are constructed by intersected perpendicular bisectors between adjacent data points. The polygons group the area of interest by nearest data point

2. Assign weight to each datum proportional to the area of the associated Voronoi polygon

\[ w(\bf{u}_j) = n \cdot \frac{A_j}{\sum_{j=1}^n} \]

where \(w(\bf{u}_j)\) is the weight for the \(j\) data. Note, the sum of the weights is \(n\); therefore, \(w(\bf{u}_j)\) is nominal weight of 1.0, sample density if the data were equally spaced over the area of interest.

Here are some highlights for polygonal declustering,

• polygonal declustering is sensitive to the boundaries of the area of interest; therefore, the weights assigned to the data near the boundary of the area of interest may change radically as the area of interest is expanded or contracted

• polygonal declustering is the same as the Theissen polygon method for calculation of precipitation averages developed by Afred H. Thiessen in 1911.

Also see:

• Cell-based Declustering

• Kriging-based Declustering

Polynomial Regression

A basis expansion method applied to predictor features before linear regression to introduce nonlinear relationships between predictors and response.

Polynomial regression transforms predictor features into polynomial basis features,

\[ y=\sum_{l=1}^{k}\sum_{j=1}^{m}\beta_{j,l}h_l(X_j)+\beta_0 \]

where the basis transforms over training data observations, \(i=1,\ldots,n\), are,

\[ h_1(x_i)=x_i,\quad h_2(x_i)=x_i^2,\quad h_3(x_i)=x_i^3,\quad h_4(x_i)=x_i^4,\ldots,\quad h_k(x_i)=x_i^k \]

up to the specified polynomial order \(k\).

For example, with a single predictor feature, \(m=1\), and a fourth-order polynomial,

\[ y=\beta_{1,1}X+\beta_{1,2}X^2+\beta_{1,3}X^3+\beta_{1,4}X^4+\beta_0 \]

After applying the basis expansion, the model remains linear in the parameters and the analytical linear regression solution can still be applied.

Polynomial regression assumes,

• the response is a linear combination of transformed basis features

• the relationship between predictors and response can be approximated by a polynomial function

The model parameters now describe transformed predictor features rather than the original predictor features,

• for example, interpretation of a coefficient associated with permeability\(^4\) is not straightforward

• higher-order polynomial models often have increased model variance and may produce unstable interpolation and extrapolation

More on polynomial regression assumptions,

• fixed predictor features - predictor features and their basis expansions are treated as known values rather than random variables

• constant variance - response error variance is constant over the range of predictor feature values

• linearity - response is a linear combination of the polynomial basis features

• polynomial relationship - the underlying relationship between predictor and response can be represented by polynomial functions

• independence of errors - response errors are uncorrelated with each other

• no severe multicollinearity - polynomial basis features are not excessively redundant with each other

Extensions of polynomial regression may include interaction terms between predictor features, such as \(X_1X_2\), to represent coupled nonlinear relationships.

Used in:

• Polynomial Regression Description and Demonstration

Also see:

• Basis Function

• Basis Expansion

• Polynomial Regression

• Hermite polynomial

Population

The complete set of values for a feature over the 2D area of interest or 3D volume of interest, represented at sufficient resolution to support decision making.

For example,

• the exhaustive set of porosity values at every location within a reservoir

• the exhaustive set of gold grades throughout an ore body

In practice, the entire population is rarely observed. Instead, a limited sample is collected and used to infer population parameters.

Used in:

• Comparison of Population and Sample Discussion

Contrast with:

• Sample

Posterior

In Bayes’ theorem, the posterior probability represents updated knowledge or uncertainty about possible model assumptions, states, or parameters after incorporating new data. The posterior is calculated by combining the prior probability with the likelihood function describing the compatibility of observed data with possible models,

\[ P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)} \]

where:

• \(P(A)\) is the prior probability representing uncertainty before observing new information,

• \(P(B|A)\) is the likelihood function describing the compatibility of observations \(B\) with state or parameter \(A\),

• \(P(B)\) is the evidence term used to normalize the posterior probability,

• \(P(A|B)\) is the posterior probability representing updated uncertainty after incorporating observations.

Used in:

• Bayesian Probability Definition and Demonstration

• Bayesian Probability Example Problems

• Bayesian Linear Regression Description and Demonstration

• Naive Bayes Description and Demonstration

Also see:

• Bayes’ Theorem

• Prior

• Likelihood Function

• Evidence

Power Law Average

A flexible family of averaging methods used to scale a feature from a smaller volume support, \(v\), to a larger support, \(V\), by calculating an effective value representative of the larger volume.

The power law average is,

\[ z_V =\left[ \frac{1}{n} \sum_{i=1}^{n} z_{v,i}^{\,\omega} \right]^{\frac{1}{\omega}} \]

where \(\omega\) is the averaging power.

Special cases include,

• \(\omega = 1\) — arithmetic average

• \(\omega = -1\) — harmonic average

• \(\omega \rightarrow 0\) — geometric average (obtained in the limit)

The choice of averaging power depends on the physical process being modeled. For example, for permeability,

• arithmetic averaging is appropriate for flow parallel to bedding

• harmonic averaging is appropriate for flow perpendicular to bedding

• near-geometric averaging is often appropriate for oblique flow directions

Used in: TBA - new scale and volume variance chapter

Precision

Categorical classification prediction model performance metric that summarizes, for each category \(k\), the conditional probability that an observation truly belongs to category \(k\) given that the model predicts category \(k\).

• “When the model predicts positive, how often is it correct?”

In other words,

• the ratio of true positives (TP) to all observations predicted as category \(k\), i.e., true positives (TP) plus false positives (FP)

• a summarization over the columns in a confusion matrix (truth on the y-axis and predicted categories on the x-axis)

\[ \text{Precision}_k=P(C=k\,|\,C^*=k)=\frac{TP_k}{TP_k+FP_k} \]

where,

• a precision of 1.0 indicates that every observation predicted as category \(k\) is actually category \(k\)

• a low precision indicates that many observations predicted as category \(k\) are actually other categories (false positives)

Used in:

• Precision Description

• Precision Checking Naive Bayes Classifier

Also see:

• Recall

• F1-Score

• Confusion Matrix

Prediction

Estimate unknown or future sample values given assumptions about, or a model of, the population. For example,

• given a model of the reservoir, predict the porosity, permeability, or production rate at a proposed well location before drilling

• given historical production data, predict next month’s production rate

• statistical or data-driven prediction uses inferred relationships from data, while physics-based prediction uses governing equations and physical models

Prediction is concerned with estimating unknown values, rather than inferring the parameterss of the underlying population.

Used in:

• Comparison of Inference and Prediction

Compare with:

• Inference

Prediction Error

The difference between an observed response feature value and the corresponding prediction from a predictive machine learning model.

For each training sample,

\[ e_i=y_i-\hat{y}_i, \quad \forall \quad i=1,\ldots,n \]

where \(y_i\) is the observed response feature value and \(\hat{y}_i\) is the model prediction.

Prediction errors are combined over all training data to evaluate model performance,

• the collection of prediction errors forms the error vector

• error norms summarize the prediction errors with a single value for model training and comparison

• prediction errors are the basis for common loss functions, including mean absolute error (MAE), mean square error, and root mean square error

Prediction errors are analyzed to,

• evaluate model accuracy

• compare competing prediction models

• detect systematic bias or trends in model performance

• quantify predictive uncertainty

Prediction error should be evaluated separately for,

• training data - to assess how well the model fits the available data

• testing data - to estimate how well the model generalizes to previously unseen data

A well-performing prediction model,

• has small prediction errors on both training and testing data

• exhibits prediction errors that are approximately random, with no systematic trends relative to predictor features or predicted values

Used in:

• Derivation of Expected Test Mean Square Error

Also see:

• Loss Function

• Mean Square Error

• Mean Absolute Error

• Root Mean Square Error

• Norm

Prediction Interval

An uncertainty range for a future prediction represented by lower and upper bounds based on a specified probability level, known as the confidence level.

For example, a 95% prediction interval may be communicated as,

• given predictor feature values, \(X_1=x_1,\ldots,X_m=x_m\), there is a 95% probability that a future reservoir NTG observation will fall between 13% and 17%.

A prediction interval represents uncertainty in the next observation and integrates,

• uncertainty in the estimated model prediction, \(\hat{Y}|X=x\)

• irreducible variability or error in the response around the model prediction, \(Y-\hat{Y}\)

Therefore, prediction intervals are wider than confidence intervals for the estimated mean response because they include both model uncertainty and observation uncertainty.

The prediction interval depends on,

• uncertainty in model parameters

• uncertainty in the estimated conditional mean response

• variability of observations around the conditional mean response

Used in:

• Analytical Prediction Error Description and Demonstration

Contrast with:

• Confidence Interval

Predictor Feature

A feature used as an input to predict a response feature in a predictive model.

A predictive machine learning model may be represented as,

\[ y = \hat{f}(x_1,\ldots,x_m) + \epsilon \]

where \(y\) is the response feature, \(x_1,\ldots,x_m\) are the predictor features, and \(\epsilon\) represents model error.

Additional comments,

• predictor features are also commonly called input features or explanatory features

• traditional statistical modeling often uses the term independent variable, although predictor feature is preferred because predictor features are not necessarily statistically independent

Used in:

• Comparison of Predictor and Response Features

Contrast with:

• Response Feature

Predictor Feature Space

The multiple variate space represented by the ranges and possible combinations of all features for our problem. The term is commonly truncated as feature space. Refers to the predictor features and does not include the response feature(s); therefore, it is,

• all possible combinations of predictor features for which we need to make predictions

• may be referred to as predictor feature space.

Typically, we train and test our machines’ predictions over the predictor feature space,

• the space is typically a hypercuboid with each axis representing a predictor feature and extending from the minimum to maximum, over the range of each predictor feature

• more complicated shapes of predictor feature space are possible, e.g., we could mask or remove subsets with poor data coverage.

Used in:

• Sampling and Coverage Curse of Dimensionality Discussion

Also see:

• Curse of Dimensionality

• Feature Projection

• Feature Selection

Primary Data

Data samples of the feature being modeled.

• the target feature for a geostatistical model

• the response feature for a predictive machine learning model

Primary data are the observations that the model directly estimates or simulates.

For example,

• porosity measurements from core are used to build a 3D geostatistical porosity model, supported by a 2D seismic acoustic impedance map. The core porosity measurements are the primary data because porosity is the feature being modeled.

Used in:

• TBD

Contrast with:

• Secondary Data

Principal Component Analysis

A machine learning method in inferential statistics and information visualization for exploring and representing the structure of high-dimensional datasets.

Principal component analysis (PCA) is commonly applied in machine learning workflows for,

• dimensionality reduction

• reducing linear redundancy between predictor features

• visualization of high-dimensional datasets

Salient points of principal component analysis include,

• orthogonal transformation - converts a set of correlated features into a set of linearly uncorrelated variables called principal components

• variance maximization - identifies orthogonal directions that explain the greatest possible variability in the data

• distance preservation - the full PCA transformation is a rotation and preserves pairwise distances; reduced PCA representations approximately preserve the original structure, although often a \(p\) subset is retained for dimensionality reduction resulting in distance projection error.

The number of principal components available is,

\[ k\leq\min(n-1,m) \]

where \(n\) is the number of observations and \(m\) is the number of predictor features.

Principal components are ordered by the amount of variance explained,

• the first principal component describes the largest possible variance in the dataset

• each subsequent principal component describes the largest remaining variance while remaining orthogonal to previous components

• additional components continue until the maximum number of available components is reached

PCA is based on eigenvalues and eigenvectors of the data covariance matrix.

The covariance matrix contains the pairwise covariance between all combinations of predictor features. Eigen decomposition of the covariance matrix provides,

• eigenvalues - variance explained by each principal component

• eigenvectors - directions or loadings that define each principal component

Comparison between principal component analysis and multidimensional scaling,

• Principal Component Analysis (PCA) - operates on the covariance matrix (\(m\times m\)) of predictor features and finds orthogonal linear projections that maximize explained variance

• Multidimensional Scaling (MDS) - operates on the pairwise distance matrix (\(n\times n\)) between samples and finds a low-dimensional representation that best preserves sample-to-sample dissimilarity

Benefits of working in a reduced-dimensional representation include,

1. reduced data storage and computational requirements

2. easier visualization of high-dimensional datasets

3. reduced multicollinearity between predictor features

Used in:

• Principal Component Analysis Description and Demonstration

Also see:

• Multidimensional Scaling

• Covariance

• Eigenvalue

• Eigenvector

Prior

In Bayes’ theorem, the prior probability represents knowledge or uncertainty about possible model assumptions, states, or parameters before considering new data. The prior is combined with the likelihood function describing new observations to calculate the posterior probability,

\[ P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)} \]

where:

• \(P(A)\) is the prior probability representing uncertainty before observing new information,

• \(P(B|A)\) is the likelihood function describing the compatibility of observations \(B\) with state or parameter \(A\),

• \(P(B)\) is the evidence term used to normalize the posterior probability,

• \(P(A|B)\) is the posterior probability representing updated uncertainty after incorporating observations.

Used in:

• Bayesian Probability Definition and Demonstration

• Bayesian Probability Example Problems

• Bayesian Linear Regression Description and Demonstration

• Naive Bayes Description and Demonstration

Also see:

• Bayes’ Theorem.

• Likelihood Function

• Posterior

• Evidence

Probability Closure

The normalization requirement of probability measures stating that the total probability over the entire sample space \(\Omega\) is equal to 1:

\[ P(\Omega) = 1 \]

This property ensures that all possible outcomes collectively account for the entire probability mass.

Useful examples include:

• Closure for complements:

\[ P(A) + P(A^c) = 1 \]

• Conditional complements:

\[ P(A|B) + P(A^c|B) = 1 \]

• Probability density functions:

\[ \int_{-\infty}^{\infty} f_X(x)\,dx = 1 \]

• Probability mass functions:

\[ \sum_{k=1}^{K} p_k = 1 \]

• Cumulative distribution functions:

\[ \lim_{x \to \infty} F_X(x) = 1 \]

Used in:

• Probability Operations Itemization and Demonstration

Also see:

• Evidence

Probability Constraints

The fundamental requirements for valid measures of probability include,

1. Boundedness, probabilities must be between zero and one,

\[ 0.0 \le P(A) \le 1.0 \]

2. Closure, the total probability over the entire sample space, \(\Omega\), is one,

\[ P(\Omega) = 1.0 \]

3. Null set, the probability of the empty set is zero,

\[ P(\emptyset) = 0.0 \]

4. Additivity, the probability of mutually exclusive events is the sum of their individual probabilities,

\[ P\left(\bigcup_i A_i\right) = \sum_i P(A_i) \]

These constraints are closely related to the Kolmogorov probability axioms.

Used in:

• Probability Operations Itemization and Demonstration

Probability Density Function

A representation of a continuous statistical distribution with a density function, \(f(x)\), describing the relative density over the range of possible feature values, \(x\).

A univariate probability density function is denoted by,

\[ f_X(x) \]

and a bivariate probability density function is denoted by,

\[ f_{X,Y}(x,y) \]

and extends to multivariate distributions.

For example, the Gaussian probability density function is specified as,

\[ X \sim \mathcal{N}(\mu,\sigma^2) \]

with probability density function,

\[ f_X(x) = \frac{1}{\sigma\sqrt{2\pi}}\exp \left(-\frac{(x-\mu)^2}{2\sigma^2} \right), \qquad -\infty < x < \infty \]

parameterized by average, \(\mu\), and variance, \(\sigma^2\).

These are requirements for a valid probability density function,

• non-negativity constraint, the density cannot be negative,

\[ 0.0 \le f_X(x) \]

• the density value may be greater than 1.0 because density is not probability

• integrate density over a range of \(x\) to calculate probability,

\[ 0 \le \int_a^b f_X(x) dx = P(a \le x \le b) \le 1.0 \]

• probability closure, the total area under the PDF curve is equal to 1.0,

\[ \int_{-\infty}^{\infty} f_X(x) dx = 1.0 \]

Nonparametric PDFs are commonly calculated with kernels (usually a small Gaussian distribution) that are summed over all data. Therefore, there is an implicit scale (smoothness) parameter when calculating a PDF.

• too large of kernels will smooth out important information about the univariate distribution

• too narrow a kernel will result in an overly noisy PDF that is difficult to interpret

This is analogous to the choice of bin size for a histogram or normalized histogram.

Parametric PDFs require model fitting to the data. The steps are,

1. Select a parametric distribution, e.g., Gaussian, lognormal, etc.

2. Calculate the parameters for the parametric distribution based on available data, using methods such as least squares or maximum likelihood.

It is very common to use the acronym PDF for probability density function.

Used in:

• Probability Density Function Description and Demonstration

Contrast with:

• Cumulative Distribution Function

• Probability Mass Function

Probability Mass Function

A function that describes the probability distribution of a discrete feature. The probability mass function assigns a probability to each possible discrete outcome,

\[ p_X(x) = P(X=x) \]

with the requirements,

\[ p_X(x) \geq 0, \quad \forall x \]

and probability closure,

\[ \sum_x p_X(x) = 1.0 \]

The normalized histogram of a discrete feature is an empirical probability mass function.

Contrast with:

• Probability Density Function

• Cumulative Distribution Function

Probability Of Acceptance

A probability used in stochastic sampling algorithms to determine whether a proposed sample or model realization is added to the generated sample.

The probability of acceptance is applied in methods such as,

• Monte Carlo method

• Simulated annealing

• Metropolis-Hastings

In the Metropolis-Hastings algorithm, the acceptance probability is calculated as,

\[ \alpha = \min\left(\frac{P(\beta'|y,X)}{P(\beta|y,X)}\frac{P(\beta|\beta')}{P(\beta'|\beta)},1\right) \]

where \(\alpha\) represents the probability of accepting the proposed sample.

The acceptance rule is,

• if \(\alpha \geq 1\), accept the proposed sample

• if \(\alpha < 1\), conditionally accept the proposed sample by drawing,

\[ p\sim U[0,1] \]

and accepting the proposed sample if,

\[ p\leq\alpha \]

The stochastic acceptance step allows the Markov chain to explore lower-probability regions of the target distribution while still converging to the desired probability distribution.

Used in:

• Probability of Acceptance in Metropolis-Hastings Sampler Description and Demonstration

Also see:

• Markov Chain Monte Carlo

• Monte Carlo Method

Probability Operators

A list of useful, common probability operators that are essential for working with probability and uncertainty problems.

1. Union of Events - the union of outcomes, the probability of \(A\) or \(B\) is calculated with the probability addition rule ,

\[ P(A \cup B) = P(A) + P(B) - P(A,B) \]

\(\quad\) where the intersection probability is subtracted to avoid double counting outcomes common to both events.

2. Intersection of Events - the intersection of outcomes, the probability of \(A\) and \(B\) is represented as,

\[ P(A \cap B) = P(A,B) \]

\(\quad\) Under the assumption of independence of \(A\) and \(B\), the intersection probability can be calculated from the marginal probabilities,

\[ P(A,B) = P(A) \cdot P(B) \]

\(\quad\) If there is dependence between \(A\) and \(B\), then conditional probability is required,

\[ P(A,B) = P(A|B) \cdot P(B) \]

3. Complementary Events - the NOT operator for probability. If we define event \(A\), then the complement \(A^c\) represents all outcomes that are not \(A\).

\(\quad\) The resulting closure relationship is,

\[ P(A) + P(A^c) = 1.0 \]

\(\quad\) Complementary events extend naturally to conditional probabilities, for example,

\[ P(A|B) + P(A^c|B) = 1.0 \]

\(\quad\) Note, the conditioning event must remain the same.

4. Mutually Exclusive Events - events that do not intersect and have no common outcomes. Using set notation,

\[ \{x: x \in A \text{ and } x \in B \} = \emptyset \]

\(\quad\) and the joint probability is,

\[ P(A \cap B) = P(A,B) = 0 \]

Used in:

• Probability Operations Itemization and Demonstration

Also see:

• Union of Events

• Intersection of Events

• Complementary Events

• Mutually Exclusive Events

Probability Perspectives

The three primary perspectives for interpreting and calculating probability are:

1. Long-term frequencies - probability as the ratio of observed outcomes from repeated experiments. This perspective requires repeatable experiments and observations, and is the basis for frequentist probability.

2. Physical tendencies or propensities - probability based on knowledge of, or models for, the physical system. For example, the probability of a heads outcome from a coin toss can be known from the physical properties of the coin without performing repeated experiments.

3. Degrees of belief - probability representing our uncertainty about an outcome or proposition, allowing probabilities to be updated as new information becomes available. This perspective is the basis for Bayesian probability.

Used in: TBA

Production Data

Spatiotemporal subsurface engineering data including bottom hole pressure, fluid production rates, fluid composition, and temperatures.

Production data are important dynamic observations used to evaluate and calibrate subsurface models.

Some additional comments,

• production from a single well may be commingled over multiple producing intervals unless production logging tool (PLT) data are available to allocate production by interval

• production data provide important ground truth for matching reservoir model forecasts through the model calibration process known as history matching

As model outputs from a flow simulation transfer function, production data integration requires,

• an inversion approach known as history matching, which is challenging because the system is ill-posed and the solution is generally nonunique

Used in:

• Production Data Feature in Synthetic Datasets

Proportions

The proportion of each possible category relative to the total number of observations. Proportions describe the categorical distribution of a feature and are equivalent to the probability of occurrence when observations are considered representative.

For a sample, the proportion of category \(k\) is,

\[ p_k = \frac{n_k}{n} \]

where \(n_k\) is the number of observations in category \(k\) and \(n\) is the total number of observations.

The proportions satisfy probability closure,

\[ \sum_{k=1}^{K} p_k = 1.0 \]

where \(K\) is the total number of categories.

Proportions are,

• the y-axis of a categorical feature probability density function

• the marginal probability’s for a categorical feature

• commonly used to summarize lithology, facies, rock type, mineral class, land use, or any categorical variable

Facies proportions are central to geostatistical modeling because facies often define stationary domains with distinct statistical and spatial characteristics. For example,

• sand — high porosity and permeability occurring in relatively large connected bodies

• shale — low porosity and permeability occurring in drapes and thin, laterally continuous beds

The determination of facies proportions is one of the most important modeling decisions because they control,

• expected volumetrics of each facies

• connectivity and geological architecture

• flow simulation and production forecasts

• uncertainty in downstream decision making

Facies proportions may be estimated from,

• well or drill-hole observations

• interpreted seismic data

• geological analogs

• conceptual geological models

Because available data are sparse, the proportions themselves are uncertain. This uncertainty is commonly represented with multiple scenarios by varying the global proportions within plausible limits and evaluating the impact on the resulting subsurface models.

Also see:

• Probability Density Function

• Marginal Probability

• Indicator Transform

• Scenarios

Prototype

In machine learning, prototype methods represent groups, clusters, or classes of data using a small set of representative vectors or exemplar observations called prototypes.

The general concepts of prototype methods include,

• representation compression - reducing a dataset to a manageable set of representative vectors or samples,

\[ W_1,W_2,\ldots,W_k \]

where \(W\) represents the set of prototypes.

• similarity-based assignment - observations are assigned to prototypes based on a similarity or distance measure, such as Euclidean, Manhattan, or cosine distance

• Voronoi tessellation - for distance-based prototype methods, the feature space is divided into regions where each region contains the observations closest to one prototype

Examples include,

• k-Means clustering - prototypes are cluster centroids calculated as the average location of all cluster members in feature space and are not necessarily actual samples

• k-Medoid clustering - prototypes are selected as actual samples from the dataset that best represent each cluster

Prototype methods are useful for,

• reducing complex datasets to representative examples

• efficient similarity-based prediction and classification

• interpretable representation of groups or clusters

Used in:

• k-Means Clustering Description and Demonstration

Also see:

• k-Means clustering

• Partitional Clustering

Contrast with:

• Density-based Clustering

• Hierarchical Clustering

Qualitative Feature

Feature described by labels rather than numerical quantities. Qualitative features represent information that requires interpretation or classification and the values do not have inherent numerical meaning.

• typically qualitative features cannot be directly measured from rock, but instead require interpretation steps

Examples of qualitative features include,

• rock type = sandstone

• facies = channel sand, levee, floodplain

• zonation = bornite-chalcopyrite-gold higher grade copper zone

Qualitative features may be encoded numerically for analysis, but the numerical codes represent categories and do not imply magnitude or order. For example,

• sandstone = 1 and shale = 2 are category labels, not measurements where shale is greater than sandstone.

In geostatistics and machine learning, qualitative features are commonly transformed using approaches such as indicator transform or one-hot encoding before modeling.

Used in: TBD

Contrast with:

• Quantitative Feature

Quantitative Feature

A feature that can be measured and represented by numerical values with meaningful magnitude.

Examples of quantitative features include,

• age = 10 Ma (millions of years)

• porosity = 0.134 (fraction of volume is void space)

• saturation = 80.5% (volume percentage)

Similar to qualitative feature, quantitative features often require interpretation. For example,

• total porosity may be directly measured, but effective porosity may require geological interpretation or a petrophysical model.

Quantitative features may be continuous or discrete depending on whether the possible values are measured along a continuum or occur as countable values.

Used in: TBD

Contrast with:

• Qualitative Feature

Query

A component of the attention mechanism that allows a model to dynamically focus on the most relevant information while reducing the influence of less relevant information.

• A Query (\(Q\)) is a representation of the information required for the current prediction or calculation.

The attention mechanism,

• compares each Query with all Keys to calculate similarity scores, which are then used to derive attention weights.

Some additional comments,

• each Key–Value pair represents a stored piece of information, where the Key describes the information and the Value contains the associated content.

Conceptually,

• Query (\(Q\)) asks, “What information do I need right now?”

Used in:

• Definition and Demonstrations of Attention Mechanism

Also see:

• Key

• Value

R Squared

Stated as coeficient of determination, “R-squared” or in mathatical notations as \(R^2\), a statistical measure of the proportion of the variability in a response feature that is explained by a regression model.

For ordinary least-squares linear regression with an intercept, the response feature variance can be partitioned into,

\[ \sigma^2_{tot}=\sigma^2_{reg}+\sigma^2_{res} \]

where,

• \(\sigma^2_{tot}\) is the variance of the response feature, \(y_i\)

• \(\sigma^2_{reg}\) is the variance explained by the regression model

• \(\sigma^2_{res}\) is the residual variance, variance of prediction error

The coefficient of determination is therefore,

\[ r^2=\frac{\sigma^2_{reg}}{\sigma^2_{tot}}=1-\frac{\sigma^2_{res}}{\sigma^2_{tot}} \]

where,

• \(R^2=0\) indicates the regression explains none of the response variability

• \(R^2=1\) indicates the regression explains all of the response variability

For ordinary least-squares linear regression with a single predictor feature,

\[ R^2=\rho_{x,y}^2 \]

where \(\rho_{x,y}\) is the correlation coefficient between the predictor and response.

Some additional comments,

• for many nonlinear regression models an \(R^2\) statistic can still be calculated, but it should be interpreted as a relative reduction in squared prediction error rather than a strict variance decomposition

• \(R^2\) does not indicate whether predictions are unbiased or whether the model generalizes well to unseen data; prediction error metrics such as mean square error, root mean square error, or mean absolute error are required to evaluate predictive performance.

Used in:

• Linear Regression Description and Demonstration

Also see:

• Correlation Coefficient

• Linear Regression

Random Function

A set of random variables correlated over space or time. In geostatistics, a random function provides the mathematical framework for representing spatial uncertainty and variability.

The key concept introduced by Matheron is that a geological phenomenon is viewed as a realization of an underlying random function. The observed spatial data represent one possible outcome from this random process.

Important points on random function nomenclature,

• random variables are denoted with upper-case, e.g., \(X\)

• random functions are denoted with upper-case with location vectors, e.g.,

\[ X(\mathbf{u}_1), X(\mathbf{u}_2), \ldots, X(\mathbf{u}_n) \]

• joint outcomes called realizations, or data samples are represented with lower case, e.g.,

\[ x(\mathbf{u}_1), x(\mathbf{u}_2), \ldots, x(\mathbf{u}_n) \]

• realizations with the \(\ell\) notation, e.g.,

\[ x^{\ell}(\mathbf{u}_1), x^{\ell}(\mathbf{u}_2), \ldots, x^{\ell}(\mathbf{u}_n) \]

for \(\ell = 1,\ldots,L\) realizations.

Also see:

• random variable

Random Forest

Ensemble prediction model based on,

• decision tree - each estimator in the ensemble is a decision tree

• model bagging - each tree is trained on a bootstrap realization of the training data and predictions are aggregated across the ensemble

• tree decorrelation - only a random subset of the \(m\) available predictor features is considered at each decision tree split, \(p < m\)

The reduction in model variance by ensemble estimation (averaging for regression) may be represented by the standard error of the mean,

\[ \sigma_{\overline{x}}^2=\frac{\sigma_s^2}{n} \]

where \(\sigma_s^2\) is the variance of an individual estimator and \(n\) is the number of estimators in the ensemble. This result assumes that the estimators are independent (uncorrelated).

In general, the variance of the ensemble prediction depends on the correlation between the decision trees,

\[ \mathrm{Var}(\overline{X})=\frac{\sigma^2}{n}\left[1+(n-1)\rho\right] \]

where \(\rho\) is the average pairwise correlation between trees.

This equation explains the motivation for Random Forest,

• if \(\rho=0\), the trees are uncorrelated and

\[ \mathrm{Var}(\overline{X})=\frac{\sigma^2}{n} \]

providing the maximum possible reduction in model variance

• if \(\rho=1\), the trees are perfectly correlated and

\[ \mathrm{Var}(\overline{X})=\sigma^2 \]

so averaging provides no reduction in model variance

One issue with tree model bagging is that decision trees in the ensemble may become highly correlated,

• when one predictor feature is dominant, it is repeatedly selected for the upper tree splits

• consequently many trees become very similar and averaging provides much less reduction in model variance than expected

By restricting the candidate predictor features considered at each split,

• each tree evolves differently, reducing the correlation between trees

• lower tree correlation produces greater reduction in ensemble prediction variance

Common default values for the number of candidate predictor features include,

\[ p=\sqrt{m} \]

for classification, and

\[ p\approx\frac{m}{3} \]

for regression.

If \(p=m\), every predictor feature is considered at every split and Random Forest reduces to tree bagging.

Used in:

• Random Forest Description and Demonstration

Also see:

• Decision Tree

• Model Bagging

Random Sample

A representative spatial sampling method from a:

• population – sample for statistical inference

• data set – bootstrap sample generated by sampling with replacement

• model – stochastic realization

Random sampling requires that:

• every possible sample of the specified size is equally likely to be selected

• each selection is made randomly and is not influenced by previous selections or outcomes

This minimizes selection bias and supports valid statistical inference.

Used in:

• Discussion of Bootstrap Philosophy Discussion

• Sampling the Posterior for Bayesian Linear Regression Description and Demonstration

Random Variable

A mathematical representation of uncertainty where the value of a feature is unknown and can take a range of possible outcomes described by a statistical distribution, probability density function, or cumulative distribution function.

A random variable is denoted with upper-case notation, e.g., \(X\), while possible outcomes or observed values are represented with lower-case notation, e.g., \(x_{\alpha}\) or realization \(x^{\ell}\).

For spatial phenomena, a location vector, \(\mathbf{u}\), is added to represent the random variable at a specific location,

• spatial random variable:

\[ X(\mathbf{u}_{\alpha}) \]

• spatial data measure:

\[ x(\mathbf{u}_{\alpha}) \]

• spatial realization:

\[ x^{\ell}(\mathbf{u}_{\alpha}) \]

The collection of correlated spatial random variables over many locations forms a random function.

Used in:

• Probability Density Function

Also see:

• Random Function

Realization

An outcome from a random variable or a joint outcome from a random function.

• an outcome from a random variable, \(X\), or a joint set of outcomes from a random function

• represented with lower case notation, e.g., \(x\)

• for spatial settings it is common to include a location vector, \(\mathbf{u}\), to describe the location, e.g., \(x(\mathbf{u})\), corresponding to the random variable \(X(\mathbf{u})\)

• generated by simulation methods, e.g., Monte Carlo simulation, geostistical sequential Gaussian simulation, or any other method that samples jointly from a random function

• in general, stochastic simulation assumes realizations are equiprobable, meaning each realization is considered an equally likely outcome of the modeled uncertainty

Used in:

• Drawing a Single Monte Carlo Realization

• Bootstrap Uncertainty Models

Also see:

• Monte Carlo Simulation

Realizations

A realization ensemble of spatial models generated by stochastic simulation by holding input parameters and model choices constant while changing only the random number seed.

• realizations explore the spatial offset from data component of uncertainty modeling

A realizations represents spatial uncertainty by sampling multiple possible outcomes from the same random function.

For example,

• hold the porosity average, variogram model, conditioning data, and simulation parameters constant

• generate multiple porosity models by changing only the random number seed

• differences between the realizations represent spatial uncertainty in porosity away from conditioning data

Contrast with:

• Scenarios

Reasons to Learn Some Coding

Professor Pyrcz’s reasons why every scientist and engineer should learn some coding,

• Transparency – no compiler accepts hand waving! Coding exposes every assumption and every step of your logic for others to review.

• Reproducibility – run it and get an answer. Share it with a colleague and they should obtain the same answer. Reproducibility is a cornerstone of the scientific method.

• Quantification – computers require numbers. Coding encourages us to move from qualitative descriptions to quantitative analysis and often reveals new ways to understand a problem.

• Open Source – leverage a world of brilliance. Thousands of scientists and engineers freely share software, algorithms, and ideas for everyone to build upon.

• Break Down Barriers – don’t throw your work over the fence. Work directly with software developers and contribute your subject matter expertise to build better tools.

• Deployment – share your code with others and multiply your impact. Whether measured by performance metrics or simply helping others, one script can benefit thousands of people.

• Efficiency – automate repetitive tasks. Build reusable workflows so you spend less time repeating work and more time solving scientific and engineering problems.

• Always Time to Do it Again! – many tasks are performed repeatedly. Although scripting and automation often take longer the first time, the investment usually pays for itself many times over.

• Think Like a Programmer – learning to code changes the way you approach problems. Rather than being limited by existing software, you begin designing your own solutions.

Recall

Categorical classification prediction model performance metric that summarizes, for each category \(k\), the fraction of observations belonging to category \(k\) that are correctly identified by the prediction model.

• “Of all the actual positives, how many did the model find?”

In other words,

• the ratio of true positives (TP) to all actual observations in category \(k\), i.e., true positives (TP) plus false negatives (FN)

• a summarization over the rows in a confusion matrix (truth on y-axis and predicticted categories on x-axis)

\[ \text{recall}_k=\frac{n_{k,\text{true positives}}}{n_k}=\frac{TP_k}{TP_k+FN_k} \]

where,

• a recall of 1.0 indicates that every observation belonging to category \(k\) is correctly identified

• a low recall indicates that many observations of category \(k\) are missed (false negatives)

Used in:

• Precision Description

• Precision Checking Naive Bayes Classifier

Also see:

• Recall

• F1-Score

• Confusion Matrix

Recursive Feature Elimination

Model-based feature ranking workflow that recursively removes the least important predictor feature until all features have been ranked.

The general workflow is,

1. Train a prediction model using the current set of predictor features and calculate a feature ranking metric, such as feature importance.

2. Remove the lowest-ranked predictor feature.

3. Repeat Steps 1 and 2 until only one predictor feature remains.

The final feature ranks, \(1,\ldots,m\), are obtained from the reverse order of elimination,

• last remaining feature is the most important

• second last remaining feature is the second most important

\(\vdots\)

• second feature removed is the second least important

• first feature removed is the least important

Benefits of recursive feature elimination include,

• redundant predictor features - removing one member of a redundant feature group often increases the apparent importance of the remaining members, reducing the chance that an entire redundant group is eliminated

• computational efficiency - each successive model is trained with fewer predictor features, reducing the computational cost of later iterations

• improved feature ranking - repeatedly retraining the model allows feature importance to adapt as redundant and noisy predictor features are removed

Limitations of recursive feature elimination include,

• computational cost - requires training approximately \(m\) prediction models for \(m\) predictor features

• model dependent - feature rankings are only as reliable as the underlying prediction model and its feature importance metric

• no automatic stopping criterion - produces a complete feature ranking, but does not determine the optimal number of predictor features to retain

Used in:

• Recursive Feature Elimination for Feature Ranking Description and Demonstration

Also see:

• Feature Importance

• Feature Selection

Recurrent Neural Network

A neural network architecture designed for sequential data, such as sentences, audio, signals, or time-series data, where previous information influences predictions at subsequent sequence positions.

Unlike standard feed-forward neural networks,

• recurrent neural networks maintain a hidden state that stores information from previous sequence positions, introducing recursion into the network

• predictions are made sequentially, with information from earlier steps passed forward to influence later predictions

The fundamental concept of recurrent neural networks is,

• for sequential data, the context from previous observations should influence the current prediction

Instead of treating each observation as independent,

• the network processes observations in sequence

• a recurrent weighted connection passes information from the previous hidden state into the current hidden state

• the hidden state acts as a learned memory of previous information

For a sequence position \(t\),

\[ h_t=f(W_xx_t+W_hh_{t-1}+b) \]

where \(x_t\) is the current input, \(h_t\) is the current hidden state, \(h_{t-1}\) is the previous hidden state, and \(W_x\), \(W_h\), and \(b\) are trainable network parameters.

Training recurrent neural networks,

• network weights and biases must account for the influence of each parameter over all subsequent sequence positions

• training is performed using backpropagation through time (BPTT), where gradients are propagated through the recurrent connections over the sequence

Recurrent neural networks are commonly applied for,

• natural language processing (NLP) - text generation, translation, and language modeling

• speech recognition - converting audio sequences into text

• time-series analysis - forecasting signals and detecting temporal patterns

• healthcare - monitoring sequential patient measurements and detecting anomalies

Variants of recurrent neural networks include,

• long short-term memory (LSTM) - introduces memory gates to reduce vanishing gradients and retain important long-term information

• gated recurrent unit (GRU) - simplified gated architecture with improved computational efficiency

• bidirectional recurrent neural network (BRNN) - processes sequences in both forward and backward directions to incorporate preceding and future context

Training recurrent neural networks,

• network weights and biases are estimated using backpropagation through time and gradient-based optimization

• training is more computationally complex than feed-forward neural networks due to the repeated application of the network over sequence positions

Used in:

• Recursion in Recurrent Neural Network Architecture Description and Demonstration

Also see:

• Neural Network

• Activation Function

• Backpropagation

Recursion

From Wikipedia recursion, “recursion occurs when the definition of a concept or process depends on a simpler or previous version of itself.”

A recursive process applies the same operation repeatedly to a simpler version of the problem until reaching a stopping condition, known as the base case.

A simple example of recursion is calculating a factorial,

\[ n!=n\cdot(n-1)\cdot\ldots\cdot2\cdot1 \]

which can also be written recursively as,

\[ n!=n\cdot(n-1)! \]

with the base case,

\[ 0!=1 \]

A Python implementation of the recursive factorial calculation,

def factorial(n):
    if n == 0 or n == 1:
        return 1
    else:
        return n * factorial(n-1)

where the function calls itself with a simpler input, \(n-1\), until reaching the base case.

Other examples of recursion include,

• Fibonacci sequence - each number is calculated from the two preceding numbers,

\[ F_n=F_{n-1}+F_{n-2} \]

def fibonacci(n):
    if n == 0:
        return 0
    if n == 1:
        return 1
    else:
        return fibonacci(n-1)+fibonacci(n-2)

• fractal geometry - complex geometric structures generated by repeatedly applying the same construction rule at smaller scales, for example the Sierpiński triangle

In programming, recursion is useful when a problem can naturally be divided into smaller versions of itself. However, recursive solutions may require additional computational memory and may not always be the most efficient implementation.

Used in:

• Recursion in Recurrent Neural Network Architecture Description and Demonstration

Also see:

• Recurrent Neural Network

Regression

A supervised machine learning method that predicts a continuous response feature from one or more predictor features.

Regression learns a relationship between predictor features and a response feature that is a continuous feature,

• given predictor features, \(X_1,\ldots,X_m\), the model estimates a continuous response value, \(\hat{Y}\)

• model predictions are evaluated by comparing predicted values, \(\hat{Y}\), with observed response values, \(Y\)

Common regression methods include,

• linear regression

• ridge regression

• lasso regression

• regression decision trees and neural networks

Used in:

• Regression with Linear Regression Description and Demonstration

• Regression with Ridge Regression Description and Demonstration

• Regression with LASSO Regression Description and Demonstration

• Regression with Decision Tree Description and Demonstration

• Regression with Random Forest Description and Demonstration

Contrast with:

• Classification

Regular Sampling

A representative spatial sampling method from a:

• population – observations collected at fixed intervals

• data set – every kth observation

• model – values extracted on a regular grid of predictor features

Regular sampling:

• collects samples at fixed intervals in space, time, or sequence

• provides uniform coverage of the population or domain

• may introduce bias if the sampling interval aligns with periodic patterns in the population, data set, or model

Used in:

• Cell-based Declustering Description and Demonstration

Also see:

• Representative Spatial Sampling

• Cell-based Declustering

• Polygonal Declustering

• Kriging-based Declustering

Representative Spatial Sampling

The sample and resulting sample statistics are representative of the population, by sampling theory we have 2 options:

1. Random sampling - each potential sample from the population is equally likely to be sampled as samples are collected. This includes,

• selecting a specific location has no impact on the selection of subsequent locations.

• assumption that the population size that is much larger than the sample size; therefore, significant correlation between samples is not imposed due to without replacement sampling (the constraint that you can only sample a location once). Note, generally this is not an issue for the subsurface due to the sparsely sampled massive populations

2. Regular Sample - sampling at equal space or time intervals. While random sampling is prefered, regular sampling is robust as long as,

• the regular sampling intervals do not align with natural periodicity in the data, e.g., the crests are systemally sampling resulting in biased high sample statistics

Used in:

• Spatial Sample Selection Discussion

Contrast with:

• Biased Spatial Sampling

• Clustered Spatial Sampling

Residual Sum of Squares

The sum of the squared prediction errors over all observations, commonly used as an objective function for regression.

The residual sum of squares is,

\[ RSS=\sum_{i=1}^{n}(y_i-\hat{y}_i)^2, \]

where \(y_i\) is the observed response feature value and \(\hat{y}_i\) is the corresponding model prediction.

Residual sum of squares,

• measures the total squared prediction error over all observations

• is minimized during model parameter training for ordinary least squares linear regression

• forms the basis for several regression error metrics, including the mean squared error

Used in:

• Residual Sum of Squares Loss for Decision Trees Description and Discussion

Also see:

• Prediction Error

• Mean Squared Error

• Mean Absolute Error

• Cross-Entropy Loss

• Gradient-based Optimization

• Model Parameter

Response Feature

The output or target feature for a predictive machine learning model. A predictive machine learning model can be generalized as,

\[ y = \hat{f}(x_1,\ldots,x_m) + \epsilon \]

where the response feature is \(y\), the predictor features are \(x_1,\ldots,x_m\), and \(\epsilon\) represents model error or unexplained variability.

The response feature is the quantity that the predictive model attempts to estimate or predict.

• traditional statistical modeling uses the term “dependent variable” instead of response feature

Used in:

• Comparison of Predictor and Response Features Discussion

Contrast with:

• Predictor Feature

Ridge Regression

A regularized version of linear regression with the same prediction equation,

\[ y=\sum_{\alpha=1}^{m}b_{\alpha}x_{\alpha}+b_0 \]

where the trainable model parameters are \(b_{\alpha}\), the feature weights, and \(b_0\), the constant intercept term.

The model parameters are estimated by minimizing a regularized least squares loss function that combines the residual sum of squares (RSS) with a shrinkage penalty,

\[ \sum_{i=1}^{n}\left(y_i-\left(\sum_{\alpha=1}^{m}b_{\alpha}x_{\alpha,i}+b_0\right)\right)^2+\lambda\sum_{\alpha=1}^{m}b_{\alpha}^2 \]

where \(y_i\) is the observed response feature value and \(\sum_{\alpha=1}^{m}b_{\alpha}x_{\alpha,i}+b_0\) is the model prediction for training sample \(i\).

Ridge regression introduces the hyperparameter \(\lambda\), which controls the strength of the L2 regularization penalty,

\[ \lambda\sum_{\alpha=1}^{m}b_{\alpha}^2 \]

where,

• larger values of \(\lambda\) increase the penalty on large slope coefficients

• the intercept term \(b_0\) is not penalized because it represents the response mean rather than predictor influence

Ridge regression integrates two competing goals during model training,

• minimize prediction error with respect to the training data

• minimize the magnitude of the slope parameters toward zero

The hyperparameter \(\lambda\) controls the model bias–variance trade-off,

• as \(\lambda\rightarrow0\), the solution approaches ordinary linear regression with no additional regularization bias, but potentially higher model variance

• as \(\lambda\) increases, model variance decreases and model bias increases as the model becomes less flexible

• as \(\lambda\rightarrow\infty\), the slope parameters \(b_1,\ldots,b_m\) approach zero and predictions approach the response feature mean

Ridge regression is also known as Tikhonov regularization.

The assumptions of ridge regression include,

• Error-free predictors - predictor features are treated as known values, not random variables

• Linearity - the expected response is a linear combination of predictor features

• Constant variance - the variance of response error is constant over predictor feature values (homoscedasticity)

• Independence of error - errors in the response are uncorrelated with each other

• No multicollinearity - predictor features are not linearly redundant with each other

Used in:

• Ridge Regression Description and Demonstration

Also see:

• Linear Regression

• LASSO Regression

• Polynomial Regression

• Model Parameter

Root Mean Square Error

Prediction model performance metric calculated as the square root of the mean square error over all observations,

\[ RMSE=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i-\hat{y}_i)^2} \]

where \(y_i\) is the observed response feature value, \(\hat{y}_i\) is the model prediction, and \(n\) is the number of observations.

Root mean square error has several important properties,

• all prediction errors contribute to the metric

• larger prediction errors are penalized more strongly than smaller errors because the errors are squared before averaging

• always non-negative, with \(RMSE=0\) indicating perfect predictions

• reported in the same units as the response feature

Root mean square error is commonly applied,

• to summarize regression model prediction accuracy

• to compare competing regression models on the same testing dataset

• when prediction error should be interpreted in the original response feature units

Compared with other error metrics,

• mean square error - reports the average squared prediction error and is commonly used as a loss function

• mean absolute error- is more robust to outliers because prediction errors are not squared

Advantages of root mean square error include,

• easily interpreted because it has the same units as the response feature

• emphasizes larger prediction errors while remaining directly comparable to the response variable

Limitations of root mean square error include,

• sensitive to outliers because large prediction errors are squared before averaging

• should not be compared between response features with different units or scales without normalization

• provides no indication whether prediction errors are biased or randomly distributed

Used in:

• Mean Square Error Discussion

Also see:

• Prediction Error

• Mean Square Error

• Mean Absolute Error

• Loss Function

• Norm

• Model Hyperparameter Tuning

Comments

This was a basic introduction to geostatistics. If you would like more on these fundamental concepts I recommend the Introduction, Modeling Principles and Modeling Prerequisites chapters from my text book, Geostatistical Reservoir Modeling{cite}`pyrcz2014’.

I hope this is helpful,

Michael

The Author:

Michael Pyrcz, Professor, The University of Texas at Austin Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers’ and geoscientists’ impact in subsurface resource development.

For more about Michael check out these links:

Twitter | GitHub | Website | GoogleScholar | Geostatistics Book | YouTube | Applied Geostats in Python e-book | Applied Machine Learning in Python e-book | LinkedIn

Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

• Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I’d be happy to drop by and work with you!

• Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

• I can be reached at mpyrcz@austin.utexas.edu.

I’m always happy to discuss,

Michael

Michael Pyrcz, Ph.D., P.Eng. Professor, Cockrell School of Engineering and The Jackson School of Geosciences, The University of Texas at Austin

More Resources Available at: Twitter | GitHub | Website | GoogleScholar | Geostatistics Book | YouTube | Applied Geostats in Python e-book | Applied Machine Learning in Python e-book | LinkedIn