Machine Learning Glossary A to C#

Michael J. Pyrcz, Professor, The University of Texas at Austin

Twitter | GitHub | Website | GoogleScholar | Geostatistics Book | YouTube | Applied Geostats in Python e-book | Applied Machine Learning in Python e-book | LinkedIn

Chapter of e-book “Applied Machine Learning in Python: a Hands-on Guide with Code”.

Cite this e-Book as:

Pyrcz, M.J., 2024, Applied Machine Learning in Python: A Hands-on Guide with Code [e-book]. Zenodo. doi:10.5281/zenodo.15169139 DOI

The workflows in this book and more are available here:

Cite the MachineLearningDemos GitHub Repository as:

Pyrcz, M.J., 2024, MachineLearningDemos: Python Machine Learning Demonstration Workflows Repository (0.0.3) [Software]. Zenodo. DOI: 10.5281/zenodo.13835312. GitHub repository: GeostatsGuy/MachineLearningDemos DOI

By Michael J. Pyrcz
© Copyright 2024.

This chapter is a summary of essential Machine Learning Terminology.

Motivation for this Glossary#

Firstly, why create this glossary?

I received a request for a course glossary from students in my Subsurface Machine Learning graduate course. While I usually dedicate a definition slide in my lecture slide decks to important terms, various students requested a consolidated glossary of terminology as part of their course review materials. The e-book provides an excellent vehicle and motivation for creating this resource.

Let me begin with a confession. There is a Machine Learning Glossary written by Google developers. For those seeking the in depth, comprehensive list of machine learning terms please use this book! For those seeking a comprehensive and in-depth reference of geostatistical terminology, this book remains an excellent resource.

So why create another glossary?

  • Scope - By writing my own glossary, I can limit the scope and descriptions to the concepts covered in this course. I believe many students would be overwhelmed by the size, breadth, and mathematical notation of a comprehensive geostatistics glossary.

  • Statistical Linkages - At the same time in my opinion machine learning is statistical learning and I have taken liberty to include many standard statistical terms as a foundation for all things machine learing.

  • Workflow Integration - By integrating the glossary directly into the e-book, I can link terminology to detailed chapter discussions, demonstrations, and examples. The goal is to eventually populate all chapters with hyperlinks to glossary entries, enabling students to move easily between concepts and applications.

  • Evergreen Resource - Finally, like the rest of this e-book, I want the glossary to be an evergreen living document. It will continue to evolve with new concepts, improved explanations, and feedback from students and readers.

I put quite a bit of time into this project during summer 2026 and I am happy with the way that is has evolved,

  • More than a glossary, it has become an evergreen network of machine learning concepts.

Activation Function#

A nonlinear transformation, \(\alpha(\cdot)\), applied at every neural network node after combining the weighted input signals and node bias,

\[ a_j=\alpha \left(\sum_{i=1}^{m}w_{i,j}x_i+b_j\right) \]

where \(w_{i,j}\) are the connection weights, \(x_i\) are the input signals, \(b_j\) is the node bias, and \(a_j\) is the node output.

Activation functions serve several important purposes, including,

  • introduce nonlinearity, allowing neural networks to learn complex nonlinear relationships

  • prevent multiple linear layers from collapsing into an equivalent single linear model

  • control how information propagates through the network during prediction and training

Activation functions are also known as transfer function, (not to be confused with transfer function for decision making).

Common activation functions include,

  • Logistic (Sigmoid) - smooth nonlinear mapping to the range \([0,1]\), commonly used for binary classification outputs

  • Hyperbolic Tangent (tanh) - smooth nonlinear mapping to the range \([-1,1]\), centered on zero to improve optimization

  • Rectified Linear Unit (ReLU) - outputs zero for negative inputs and the input value otherwise, computationally efficient and often results in faster training

  • Softmax - transforms the output layer values into probabilities that sum to one for multi-class classification

Selection of the activation function influences,

  • training stability

  • computational efficiency

  • gradient propagation during backpropagation

Used in:

Also see:

Addition Rule#

Method to calculate the probability of any event (the union of outcomes, represented by “or” grammar). For example, the probability of \(A\) or \(B\) is calculated with the probability addition rule,

\[ P(A \cup B) = P(A) + P(B) - P(A,B) \]

given mutually exclusive events we can generalize the addition rule as,

\[ P\left( \bigcup_{i=1}^k A_i \right) = \sum_{i=1}^k P(A_i) \]

Used in:

Adjacency Matrix#

A matrix representing a graph that records the pairwise connections between all pairs of nodes representing samples.

  • the entries are indicators, with 0 indicating no connection and 1 indicating a connection

Adjacency matrices are commonly used in,

For example, in spectral clustering,

  • the adjacency matrix transforms a collection of samples into a graph, allowing clustering based on global connectivity rather than only pairwise similarity or distance

Some additional comments,

  • node self connection - the diagonal entries (self-connections) are typically set to 0

  • for undirected graphs - the adjacency matrix is symmetric

  • alternative - that stores the strength of the connection rather than only connected/not connected is the Affinity Matrix

Used in:

Also see:

Affine Correction#

A distribution rescaling method that applies a shift and linear scaling (stretching or squeezing) to a univariate distribution, for example, histogram.

For the case of affine correction of feature \(X\) to feature \(Y\),

\[ f_{X}(x) \rightarrow f_{Y}(y) \]

we correct \(X\) to match the mean and variance of \(Y\):

\[ y_i = \frac{\sigma_y}{\sigma_x}(x_i - \overline{x}) + \overline{y}, \quad i = 1,\ldots,n \]

where \(\overline{x}\) and \(\sigma_x\) are the mean and standard deviation of the original distribution, and \(\overline{y}\) and \(\sigma_y\) are the target mean and standard deviation.

The procedure,

  1. centers the data by subtracting the original mean

  2. rescales the deviations by the ratio of standard deviations

  3. shifts the result to the target mean

Affine correction does not alter the distribution shape; it preserves the relative ordering of values and applies only a linear transformation. For transformations that modify distribution shape, see distribution transformation.

Used in:

Compare with:

Affinity Matrix#

A matrix representing a graph that stores the pairwise similarity (affinity) between all pairs of nodes (samples).

  • the entries are continuous values, with 0 indicating no similarity and larger values indicating greater similarity or stronger affinity

Affinity matrices are commonly used in,

For example, in spectral clustering,

  • the affinity matrix transforms a collection of samples into a weighted graph, allowing clustering based on global connectivity rather than only pairwise similarity or distance

Some additional comments,

  • the diagonal entries are often set to the maximum affinity (self-similarity), although some implementations set them to 0 before constructing the graph Laplacian

  • affinity values are commonly calculated from a distance measure, for example with a Gaussian (RBF) kernel,

\[ A_{ij}=\exp\left(-\frac{d_{ij}^{2}}{2\sigma^{2}}\right) \]

where \(d_{ij}\) is the distance between samples \(i\) and \(j\) and \(\sigma\) controls the neighborhood size

  • an alternative that records only whether nodes are connected is the adjacency matrix

Used in:

Also see:

Attention Mechanism#

A mechanism that allows a model to dynamically focus on the most relevant information while reducing the influence of less relevant information by answering,

  • what information is relevant?

  • where is the relevant information located?

  • how strongly should each piece of information influence the prediction?

Rather than relying only on nearby information, attention allows each query to compare itself with many candidate observations, features, or patterns and retrieve information according to their similarity. The three components of attention are,

  • Query (Q) – what information is needed right now?

  • Key (K) – a descriptor used to determine whether stored information is relevant.

  • Value (V) – the information associated with each key.

The attention mechanism proceeds by,

  1. comparing each Query with all Keys to calculate similarity scores,

  2. converting the similarity scores into Attention Weights,

  3. calculating a weighted average of the Values using the attention weights.

Some additional comments,

  • each Key–Value pair represents a stored piece of information, where the key describes it and the value contains the associated content.

  • similarity between Query and Key vectors is commonly calculated with a scaled dot product.

  • attention allows models such as transformers and large language models (LLMs) to learn long-range relationships and contextual dependencies in very large datasets.

Used in:

Also see:

Attention Weights#

Normalized similarity scores that determine how strongly each Value contributes to the output for a specific Query.

The similarity score for each Query–Key pair is commonly calculated with a scaled dot product,

\[ s_i = \frac{QK_i^{T}}{\sqrt{d_k}} \]

where \(d_k\) is the dimension of the Key vectors.

The similarity scores are then normalized with the softmax activation function to obtain the attention weights,

\[ w_i=\frac{e^{s_i}}{\sum_{j=1}^{N}e^{s_j}} \]

The attention weights satisfy,

\[ \sum_{i=1}^{N} w_i = 1 \]

The output of the attention mechanism is the weighted average of the Values,

\[ \mathrm{Attention}(Q,K,V)=\sum_{i=1}^{N} w_i V_i \]

How do we interpret the attention weights?

  • high weights – highly relevant information for the current Query.

  • low weights – less relevant information for the current Query.

Because the weights sum to one, they behave like a probability distribution over the available information, although they are learned importance weights rather than probabilities.

Used in:

Also see:

Area of Interest#

The 2D spatial domain that is being characterized, modeled, and evaluated to support subsurface decision making. In general, the area of interest,

  • is the subsurface reservoir for oil and gas, the ore body for mining, or the aquifer for hydrogeological applications

  • may include volume away from the reservoir or ore body to support data integration and extraction modeling

  • may be further subdivided into local regions or facies and modeled separately

  • is represented by a grid with features populated from data, estimation, or simulation

  • in 3D modeling is commonly called the volume of interest

  • the extent and grid cell size are selected based on a trade-off between model accuracy and computational complexity

Used in:

See also:

Autoencoder#

A neural network architecture designed to learn a compact representation of data by reconstructing the original input through an encoder-decoder structure.

An autoencoder consists of three main components,

  • encoder - sequence of layers that transforms the input data into progressively lower-dimensional representations

  • bottleneck (latent layer) - lowest-dimensional representation of the input information, forcing the network to compress the data in a latent space representation

  • decoder - sequence of layers that reconstructs the original input from the latent representation, often with a structure that mirrors the encoder

Training an autoencoder,

  • the input and output features are the same, with each output node corresponding to the matching input node

  • the network parameters are optimized to minimize the reconstruction error between the input and output

\[ L=\sum_{i=1}^{n}(x_i-\hat{x}_i)^2 \]

where \(x_i\) is the input value and \(\hat{x}_i\) is the reconstructed output.

Once trained, the bottleneck layer provides,

  • a latent representation - compressed features that capture the most important patterns in the original data

  • a lower-dimensional feature space for inference, visualization, and subsequent machine learning workflows

Autoencoders are commonly applied for,

  • dimensionality reduction - learning nonlinear feature representations beyond linear methods such as principal component analysis

  • denoising - reconstructing clean information from corrupted input data

  • anomaly detection - identifying observations that cannot be accurately reconstructed

  • latent-space modeling - performing inference, clustering, or prediction using learned compact representations

Variants of autoencoders include,

  • convolutional autoencoder - uses convolutional layers to preserve and learn spatial patterns in images and gridded data

  • variational autoencoder (VAE) - learns a probabilistic latent representation for generative modeling

Training autoencoders,

Used in:

Also see:

Average#

The average is a measure of central tendency. There are several useful interpretations of the average,

  • representative value - a single value to represent an entire distribution

  • estimate - given a distribution of values, the average minimizes the L2 norm (sum of squared error)

  • scale-up - under linear averaging of a spatial feature, the average is the correct upscaled value

  • expectation - if all outcomes are equiprobable, the average is equal to the expectation

For a sample, the average is,

\[ \overline{x} = \frac{1}{n}\sum_{\alpha=1}^{n} x_{\alpha} \]

Note, the average is quite sensitive to outliers.

Used in:

Also see:

Backpropagation#

An efficient algorithm that propagates the error gradient backward through a neural network to calculate the partial derivatives of the loss function with respect to every network weight and bias.

Backpropagation,

  • applies the chain rule from calculus to efficiently calculate gradients through multiple network layers

  • calculates the partial derivatives of the loss function with respect to all trainable weights and biases

  • provides the gradients required for gradient-based optimization

For example, the gradient at a hidden layer node, \(H_4\), is calculated from the gradient at an output node, \(O_6\), by repeated application of the chain rule,

\[ \frac{\partial L}{\partial H_4}=\frac{\partial O_{6_{\text{in}}}}{\partial H_4}\cdot\frac{\partial O_6}{\partial O_{6_{\text{in}}}}\cdot\frac{\partial L}{\partial O_6}=\lambda_{4,6}\cdot\left((1-O_6)O_6\right)\cdot(O_6-y) \]

After the gradients have been calculated,

  • gradient-based optimization updates the weights and biases using the learning rate

  • the process is repeated over many training iterations until the loss function converges

Additional comments,

  • backpropagation calculates gradients—it does not update the model parameters

  • parameter updates are performed by optimization methods such as gradient descent or stochastic gradient-based optimization

  • the bookkeeping is complicated by the many possible information pathways through the network, but the chain rule provides an efficient computational solution

Used in:

Also see:

Basis Expansion#

A technique in statistics and machine learning that transforms predictor features into a new set of, usually higher-dimensional, features called basis functions. Basis expansion increases the flexibility of a model by allowing linear methods to represent nonlinear relationships for regression and classification.

  • in mathematics, basis expansion represents a complex function as a linear combination of simpler basis functions that make the problem easier to solve

  • in machine learning, basis expansion transforms the original predictor features into a higher-dimensional feature space while still fitting a model that is linear in the transformed features

For a predictor feature \(x_i\), the transformed feature vector is

\[ h(x_i)=\left(h_1(x_i),h_2(x_i),\ldots,h_k(x_i)\right) \]

where \(h_j(x_i)\) is the \(j^{th}\) basis function.

For example, polynomial basis expansion used in polynomial regression uses the basis functions

\[ h_{1}(x_i)=x_i,\quad h_{2}(x_i)=x_i^2,\quad h_{3}(x_i)=x_i^3,\quad h_{4}(x_i)=x_i^4,\quad \dots,\quad h_{k}(x_i)=x_i^k. \]

Another example is Hermite polynomial basis expansion, which uses a set of orthogonal polynomial functions. Hermite basis functions are commonly used when the predictor variables follow a Gaussian distribution or when orthogonality between basis functions is desirable.

The first few Hermite polynomial basis expansion functions are,

\[ h_1(x_i)=1, \quad h_2(x_i)=x_i, \quad h_3(x_i)=x_i^2-1, \quad \text{and} h_4(x_i)=x_i^3-3x_i \]

and a linear model can then combine these basis functions to approximate nonlinear relationships in the original predictor feature space. After the transformation, a linear model can fit nonlinear relationships in the original predictor feature by learning a linear combination of the basis functions.

Used in:

Also see:

Basis Function#

A basis function is a mathematical building block used to construct more complex functions. It operates on the principle of linear combination:

  • by multiplying basis functions by learned weights and adding them together, we can approximate complex relationships and represent functions within a chosen function space.

A general basis expansion is written as,

\[ f(x) = \sum_{j=1}^{k} w_j h_j(x) \]

where,

  • \(h_j(x)\) is the \(j^{th}\) basis function

  • \(w_j\) is the weight assigned to the \(j^{th}\) basis function

  • \(k\) is the number of basis functions

Common examples of basis functions include:

In machine learning, basis functions allow simple models, such as linear regression, to represent nonlinear relationships by transforming the original predictor features into a richer feature space.

Used in:

Also see:

Bayes Theorem#

A theorem that relates conditional probabilities and provides the mathematical basis for Bayesian updating of uncertainty models given new information.

\[ P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)} \]

where:

  • \(P(A)\) is the prior probability representing uncertainty before considering new information,

  • \(P(B|A)\) is the likelihood function describing the compatibility of observations \(B\) with possible states or parameters \(A\),

  • \(P(B)\) is the evidence term used to normalize the posterior probability,

  • \(P(A|B)\) is the posterior probability representing updated uncertainty after incorporating observations.

Commonly applied in:

Also see:

Compare with:

Used in:

Bayesian Linear Regression#

The Bayesian formulation of the linear regression model. Start with the frequentist formulation of linear regression,

\[ y = b_1x + b_0 + \epsilon \]

where \(x\) is the predictor feature, \(b_1\) is the slope parameter, \(b_0\) is the intercept parameter, and \(\epsilon\) is the random error term.

  • in ordinary least squares regression, the model parameters are estimated by minimizing the \(L^2\) norm of the residual error vector between observed and predicted values.

In the Bayesian formulation, the regression model is expressed as a probability distribution for the response feature \(Y\),

\[ Y \sim N(\beta^T X,\sigma^2 I) \]

where \(\beta\) is the vector of regression weights, \(X\) is the design matrix of predictor features, and \(\sigma^2\) is the homoscedastic error variance.

Instead of estimating a single set of regression parameters, Bayesian linear regression treats the model parameters as random variables and estimates their posterior distribution using Bayes’ theorem,

\[ p(\beta|y,X)=\frac{p(y|X,\beta)p(\beta)}{p(y|X)} \]

where,

  • \(p(\beta)\) is the prior distribution of the regression parameters

  • \(p(y|X,\beta)\) is the likelihood of observing the training data given the model parameters

  • \(p(\beta|y,X)\) is the posterior distribution of the regression parameters

How do we solve this?

  • for standard linear regression with conjugate prior assumptions, the posterior distribution can be calculated analytically.

  • for more complex models or non-conjugate priors, sampling methods such as Markov chain Monte Carlo can be used to approximate the posterior distribution.

Used in: Bayesian Linear Regression Description and Demonstration

Compare with:

Also see:

Bayesian Probability#

Probability framework that represents uncertainty using prior knowledge and new information. Prior information may be based on,

  • expert judgment

  • experience

  • historical data

  • physical understanding, or

  • previous observations.

Bayesian probability provides a formal approach to updating uncertainty as new information becomes available. The general approach,

  1. start with a prior probability distribution representing uncertainty before considering new information

  2. formulate a likelihood function describing the compatibility of new observations with possible states or parameters

  3. combine the prior and likelihood using Bayes’ theorem to calculate the posterior probability distribution

  4. continue updating uncertainty as additional information becomes available

Bayesian probability is applied to solve probability problems where prior knowledge, limited data, or sequential information updates are important,

  • Bayesian probability contrasts with the frequentist probability approach, which interprets probability primarily through long-run frequencies of repeated experiments

Used in:

Contrast with:

Also see:

Bayesian Updating#

The process of revising a prior probability distribution using new observations or evidence to obtain a posterior probability distribution according to Bayes’ theorem.

The Bayesian updating workflow is:

  1. Specify a prior distribution representing existing knowledge.

  2. Observe new data.

  3. Evaluate the likelihood of the observations.

  4. Apply Bayes’ Theorem to obtain the posterior distribution.

  5. Use the posterior as the prior when additional observations become available.

Used in:

  • Bayesian Inference

  • Sequential Learning

  • Data Assimilation

Used in:

Also see:

Bayesian Updating for Classification#

Classification prediction problem from the perspective of Bayesian updating, based on the conditional probability of a category, \(k\), given \(n\) features, \(x_1, \dots , x_n\).

\[ P(C_k | x_1, \dots , x_n) \]

we can solve for this posterior with Bayesian updating,

\[ P(C_k | x_1, \dots , x_n) = \frac{P(x_1, \dots , x_n | C_k) P(C_k)}{P(x_1, \dots , x_n)} \]

let’s combine the likelihood function and prior for the moment,

\[ P(x_1, \dots , x_n | C_k) P(C_k) = P(x_1, \dots , x_n, C_k) \]

we can expand the full joint distribution recursively as follows,

\[ P(x_1, \dots , x_n, C_k) \]

expansion of the joint with the conditional and prior,

\[ P(x_1 | x_2, \dots , x_n, C_k) P(x_2, \dots , x_n, C_k) \]

continue recursively expanding,

\[ P(x_1 | x_2, \dots , x_n, C_k) P(x_2 | x_3, \dots , x_n, C_k) P(x_3, \dots , x_n, C_k) \]

we can generalize as,

\[ P(C_k | x_1, \dots , x_n) = P(x_1 | x_2, \dots , x_n, C_k) P(x_2 | x_3, \dots , x_n, C_k) P(x_3 | x_4, \dots , x_n, C_k) \ldots P(x_{n-1} | x_n, C_k) (x_{n} | C_k) P(C_k) \]

Used in:

Also see:

Bias#

A systematic tendency that causes results, decisions, or model outputs to deviate from an intended or objective representation.

Common uses include,

  1. Cognitive bias — a systematic tendency in human reasoning that leads to irrationally ignoring, emphasizing, or weighting information.

  2. Sampling bias — a systematic error caused by a non-representative sample that results in biased statistics or misleading conclusions.

  3. Neural network bias — a trainable additive model parameter associated with a node that shifts the activation function output independently of the input weights.

More on the last use case. Information flows forward through the network,

  • each node forms a weighted sum of the incoming information, adds a bias, and applies an activation function

\[ a_j=g\left(\sum_{i=1}^{m}w_{ij}x_i+b_j\right) \]

where \(b_j\) is the bias associated with node \(j\).

The bias determines the baseline activation of a node,

  • it shifts the weighted sum before the activation function is applied

  • it allows the activation threshold to be adjusted independently of the connection weights

During training,

Contrast with:

Also see:

Biased Spatial Sampling#

Spatial sampling such that the sample statistics are not representative of the population parameters. For example,

  • the sample mean is not the same as the population mean

  • the sample variance is not the same as the population variance

Of course, the population parameters are not accessible, so we cannot directly calculate sampling bias, i.e., the difference between the sample statistics and the population parameters. Methods we can use to check for biased sampling,

  • evaluate the samples for preferential sampling, clustering, filtering, or survivorship bias.

  • apply declustering as a diagnostic to check for biased sampling

Used in:

Contrast with:

Big Data#

Identification of big data is based on a combination of these criteria:

  1. Data Volume - many data samples and features, difficult to store, transmit and visualize

  2. Data Velocity - high-rate collection, continuous data collection relative to decision making cycles, challenges keeping up with the new data while updating the models

  3. Data Variety - data form various sources, with various types of data, types of information, and scales

  4. Data Variability - data acquisition changes during the project, even for a single feature there may be multiple vintages of data with different scales, distributions, and veracity

  5. Data Veracity - data has various levels of accuracy, the data is not certain

For common subsurface applications most, if not all, of these criteria are met. Subsurface engineering and geoscience are often working with big data!

Used in:

Also see:

Big Data Analytics#

The process of examining big data using statistical, computational, and machine learning methods to discover patterns, extract insights, and support decision making.

Used in:

Also see:

Binary Transform#

Indicator coding a random variable to a probability relative to a category or a threshold,

If \(i(\bf{u}:z_k)\) is an indicator for a categorical variable,

  • what is the probability of a realization equal to a category?

\[\begin{split} i(\bf{u}; z_k) = \begin{cases} 1, & \text{if } Z(\bf{u}) = z_k \\ 0, & \text{if } Z(\bf{u}) \ne z_k \end{cases} \end{split}\]

for example,

  • given threshold, \(z_2 = 2\), and data at \(\bf{u}_1\), \(z(\bf{u}_1) = 2\), then \(i(bf{u}_1; z_2) = 1\)

  • given threshold, \(z_1 = 1\), and a RV away from data, \(Z(\bf{u}_2)\) then is calculated as \(F^{-1}_{\bf{u}_2}(z_1)\) of the RV as \(i(\bf{u}_2; z_1) = 0.23\)

If \(I\{\bf{u}:z_k\}\) is an indicator for a continuous variable,

  • what is the probability of a realization less than or equal to a threshold?

\[\begin{split} i(\bf{u}; z_k) = \begin{cases} 1, & \text{if } Z(\bf{u}) \le z_k \\ 0, & \text{if } Z(\bf{u}) > z_k \end{cases} \end{split}\]

for example,

  • given threshold, \(z_1 = 6\%\), and data at \(\bf{u}_1\), \(z(\bf{u}_1) = 8\%\), then \(i(\bf{u}_1; z_1) = 0\)

  • given threshold, \(z_4 = 18\%\), and a RV away from data, \(Z(\bf{u}_2) = N\left[\mu = 16\%,\sigma = 3\%\right]\) then \(i(\bf{u}_2; z_4) = 0.75\)

The indicator coding may be applied over an entire random function by indicator transform of all the random variables at each location.

Used in: -Binary and Indicator Transform Description and Demonstration

Also see:

Bivariate#

Involving two features (variables) simultaneously, often to study their relationship, dependence, or correlation. For examples see bivariate statistic.

Used in:

Compare with:

Also see:

Bivariate Statistic#

A summary measure calculated from two features (variables) measured over a collection of samples. Bivariate statistics describe the relationship, dependence, or correlation between two variables.

Examples include:

  • Scatter Plot – to visualize the relationship between two variables.

  • Joint Probability – to quantify the probability of outcomes from two variables occurring together.

  • Joint Probability Density Function – a complete probabilistic model of the relationship between two variables.

Used in:

Compare with:

Also see:

Boosting Model#

The sequential addition of multiple weak learners to build a stronger predictive model.

  • a weak learner is one that offers predictions only marginally better than random selection

Boosting proceeds with the following steps,

  1. build a simple model with a relatively high error rate; although inaccurate, the model captures the general trend

  2. calculate the prediction error (residuals)

  3. fit another weak learner to the residuals

  4. update the ensemble model with the new weak learner and repeat until a stopping criterion is reached

Now the steps with equations, the general workflow for predicting \(Y\) from \(X_1,\ldots,X_m\) is,

  1. fit an initial weak learner,

\[ \hat{F}_1(X) \]

  1. calculate the residuals at the training data,

\[ r_i^{(k)} = y_i - \hat{F}_k(x_i) \]

  1. fit a new weak learner to the residuals,

\[ h_k(X) \]

  1. update the ensemble model,

\[ \hat{F}_{k+1}(X)=\hat{F}_k(X)+\eta\,h_k(X) \]

where \(\eta\) is the learning rate that controls the contribution of each new weak learner.

Additional comments,

  • the final regression estimator is the sum of the weak learners,

\[ \hat{Y}=\hat{F}_K(X)=\sum_{k=1}^{K}\eta_k h_k(X) \]

where \(\eta_k\) is the learning rate for the \(k^{th}\) learner (often a constant \(\eta\)).

  • the number of estimators, \(K\), is a hyperparameter that controls model flexibility and complexity

  • the learning rate, \(\eta\), controls how much each weak learner contributes to the final model

  • increasing \(K\) generally improves model performance, but too many estimators or an excessively large learning rate can result in an Overfit Model

Used in:

Contrast with:

Also see:

Bootstrap#

A statistical resampling procedure used to quantify uncertainty in a calculated statistic by repeatedly resampling from the available sample data. Some general comments,

  • sampling with replacement - \(n\) (number of data samples) Monte Carlo simulations from the empirical distribution of the dataset produces a realization of the resampled data.

  • simulates the data collection process - the fundamental idea is to approximate repeated sampling from the population by repeatedly resampling from the available data instead of collecting new samples.

  • bootstrap any statistic - the bootstrap is flexible because uncertainty can be characterized for almost any calculated statistic.

  • computationally efficient - repeated resampling generates realizations of the statistic that can be used to build an uncertainty distribution. A large number of realizations, \(L\), improves characterization of the uncertainty model.

  • calculates the entire uncertainty distribution - for any statistic, summary statistics of the uncertainty distribution can be calculated, such as the mean, P10, and P90 uncertainty in an estimated mean.

  • model bagging for machine learning - bagging applies bootstrap resampling to create multiple training datasets, train multiple predictive models, and aggregate predictions from the model ensemble to reduce prediction variance.

What are the limitations of bootstrap?

  • biased sample data will likely result in a biased bootstrap uncertainty model; samples should first be corrected for known bias, e.g., declustering

  • bootstrap requires a sufficient sample size to reliably characterize uncertainty

  • classical bootstrap quantifies uncertainty due to limited sampling but does not explicitly account for spatial context, including sample locations, volume of interest, or spatial continuity

  • a variant called spatial bootstrap accounts for spatial relationships during resampling

Used in:

Contrast with:

Also see:

Categorical Feature#

A feature or variable that can take one of a limited and usually fixed number of possible categories. Categories often represent qualitative classes and generally do not have inherent numerical meaning or ordering, unless categorical ordinal feature,

  • categories may have qualitative names, but are often represented by integer labels or binary indicator variables for computational analysis and modeling.

Used in:

Categorical Nominal Feature#

A categorical feature without a natural ordering relationship between categories. Examples include,

  • facies = {boundstone, wackestone, packstone, breccia}

  • minerals = {quartz, feldspar, calcite}

Categories may be assigned labels or integer codes for analysis, but the labels do not represent magnitude or ranking.

Used in:

Contrast with:

Opposite:

Categorical Ordinal Feature#

A categorical feature with a natural ordering relationship between categories. The ordering provides relative ranking information, but category differences may not represent equal numerical intervals. Examples include,

  • geologic age = {Miocene, Pliocene, Pleistocene} - ordered from older to younger rock

  • Mohs hardness = \(\{1, 2, \ldots, 10\}\) - ordered from softer to harder minerals

Used in:

Contrast with:

Causation#

A relationship in which a change in one feature or variable directly produces a change in another feature or variable through an underlying causal mechanism.

Some important characteristics of causal relationships are,

  1. Temporal precedence – the cause precedes the effect in time.

  2. Non-spuriousness – the relationship is not explained by random chance, confounding variables, or selection bias.

  3. Mechanism – a plausible physical, biological, economic, or other process exists to explain how the cause produces the effect.

  4. Consistency – the relationship is observed across different conditions, populations, times, or studies.

  5. Strength of evidence – stronger and more reproducible relationships provide greater support for causation, although strength alone does not establish causality.

Establishing causation is considerably more difficult than identifying association.

  • in this book we generally focus on prediction and association rather than causal inference

  • remember, correlation does not imply causation; two variables may be highly correlated without one causing the other

Cell-based Declustering#

A declustering method that assigns weights to spatial samples based on local sampling density to reduce sampling bias and produce statistics that are more representative of the population (i.e., improve geostatistical sampling representativity]) in the presence of clustered spatial sampling. Data weights are assigned such that,

  • samples in densely sampled areas receive less weight

  • samples in sparsely sampled areas receive more weight

The goal of declustering is to reduce the influence of uneven sample locations on statistical estimates. For example, infill drilling or blast hole samples should not significantly change statistics for the area of interest simply because some locations have been sampled more densely.

Cell-based declustering proceeds as follows:

  1. a cell mesh is placed over the spatial data and initial weights are assigned proportional to the inverse of the number of samples in each cell

  2. the cell mesh size is varied, and a cell size is selected based on the resulting declustered statistics. Typically, the cell size that minimizes the declustered mean is selected when the sample mean is biased high, and the cell size that maximizes the declustered mean is selected when the sample mean is biased low

  3. to reduce sensitivity to cell mesh position, the cell mesh is randomly shifted multiple times and the resulting declustering weights are averaged for each datum

The weights are calculated as:

\[ w(\bf{u}_j) = \frac{1}{n_l} \cdot \frac{n}{L_o} \]

where \(n_l\) is the number of data in the current cell, \(L_o\) is the number of cells containing data, and \(n\) is the total number of data.

Some highlights for cell-based declustering,

  • expert judgement to assign cell size based on nominal sample spacing (e.g., data spacing before infill drilling) may improve performance compared with automated cell size selection based only on minimizing or maximizing the declustered mean

  • cell-based declustering does not account for boundaries of the area of interest; therefore, samples near the boundary may appear more sparsely sampled and receive larger weights

  • cell-based declustering was introduced by Professor André Journel in 1983, and remains a foundational geostatistical declustering method.

Used in:

Also see:

Classification#

A supervised machine learning method that predicts a categorical response feature from one or more predictor features.

Classification learns a relationship between predictor features and response feature that is a categorical feature or discrete feature with categories,

  • given predictor features, \(X_1,\ldots,X_m\), the model estimates a class assignment, \(\hat{Y}\)

  • model predictions are evaluated by comparing predicted classes, \(\hat{Y}\), with observed classes, \(Y\)

Common classification methods include,

Contrast with:

Used in:

Clustered Spatial Sampling#

Spatial samples with locations preferentially selected or concentrated in certain areas, resulting in potentially biased statistics.

  • spatial samples are often clustered in locations associated with higher or more desirable values, for example, high porosity and permeability, good quality shale for unconventional reservoirs, or low acoustic impedance indicating higher porosity

Because the true population parameters are generally unknown, sampling bias cannot be directly calculated as the difference between sample statistics and population parameters. Methods to diagnose and address biased sampling include,

  • evaluate samples for preferential sampling and spatial clustering

  • apply declustering as a diagnostic method to evaluate the impact of clustered sampling on statistics

Used in:

Contrast with:

Cognitive Biases#

Cognitive biases are automatic mental shortcuts, or heuristics, that influence human reasoning and decision making. These shortcuts help humans efficiently process information under uncertainty, but they can also systematically distort interpretation of data, scientific evidence, and engineering decisions.

Common cognitive biases include:

  1. Anchoring Bias - excessive influence of initial information or assumptions, even when later information suggests alternatives.

  2. Availability Heuristic - overestimating the importance of information that is easily recalled or available, such as anecdotes.

  3. Bandwagon Effect - increasing confidence in a belief because many others hold the same belief.

  4. Blind-spot Effect - failing to recognize one’s own cognitive biases.

  5. Choice-supportive Bias - favoring information that supports previous decisions or commitments.

  6. Clustering Illusion - perceiving patterns in random data.

  7. Confirmation Bias - preferentially considering information that supports existing beliefs or models.

  8. Conservatism Bias - favoring established information over new evidence.

  9. Recency Bias - giving excessive weight to recently acquired information.

  10. Survivorship Bias - focusing only on successful or visible examples while ignoring missing cases.

Mitigating cognitive biases requires deliberate uncertainty analysis, quantitative evaluation of evidence, diverse perspectives, and critical review of assumptions.

Used in:

  • Cognitive Biases Itemization and Definitions TBA

Complementary Events#

The logical NOT relationship in probability. For a simple example, if we define event \(A\), then the complementary event, \(A^c\), represents NOT \(A\), and the resulting probability closure relationship is,

\[ P(A) + P(A^c) = 1.0 \]

Complementary events can also be considered for multivariate and conditional probabilities. For example, for a bivariate conditional relationship,

\[ P(A|B) + P(A^c|B) = 1.0 \]

Note that the conditioning event must remain the same for complementary probability closure.

Used in:

Computational Complexity#

A measure of the computational resources required to execute an algorithm or workflow as the size of the problem increases.

  • in machine learning, computational complexity helps us understand how algorithms scale with the number of training samples, predictor features, or model parameters

Computational complexity is commonly represented using “Big-O notation”,

\[ O(f(n)) \]

where \(n\) is the size of the problem and \(f(n)\) describes how the computational cost grows asymptotically.

There are two primary components of computational complexity,

  • time complexity – the computational time required by an algorithm as the problem size increases

  • space complexity – the computer memory required by an algorithm as the problem size increases

For example, if the time complexity is \(O(n^3)\), where \(n\) is the number of training samples, then doubling the size of the dataset increases the computational time by approximately a factor of eight.

Additional important aspects of computational complexity,

  • worst-case complexity** – Big-O notation typically represents the upper bound on the computational cost for a given problem size

  • asymptotic complexity** – complexity describes the growth rate as \(n\) becomes large, ignoring constant factors and lower-order terms

  • algorithm assumptions** – complexity assumes a specific computational model and may depend on assumptions such as whether the data are already sorted

Common examples of time complexity include,

  • constant time, \(O(1)\) – accessing an element of an array by index

  • logarithmic time, \(O(\log n)\) – binary search on a sorted array

  • linear time, \(O(n)\) – finding the minimum value in an unsorted array

  • linearithmic time, \(O(n\log n)\) – merge sort and heapsort

  • quadratic time, \(O(n^2)\) – bubble sort or algorithms with nested loops over the data

  • cubic time, \(O(n^3)\) – multiplying two dense \(n \times n\) matrices using the classical algorithm

  • exponential time, \(O(2^n)\) – exhaustive search over all subsets of \(n\) variables

Used in: TBA

Conditional Independence#

Two random variables or events are conditionally independent given a third variable if, after accounting for the observed value of the third variable, knowledge of one provides no additional information about the other.

Firstly, recall random variables \(X\) and \(Y\) are independent if the joint probability is equal to the products of the marginal probabilities,

\[ P(X,Y) = P(X) \cdot P(Y) \]

Similarly, random variables \(X\) and \(Y\) are conditionally independent given \(Z\) if,

\[ P(X,Y \mid Z) = P(X \mid Z) \cdot P(Y \mid Z) \]

or equivalently, random variables \(X\) and \(Y\) are independent if the conditional probability is equal to the marginal probability,

\[ P(X \mid Y) = P(X) \]

Similarly, random variables \(X\) and \(Y\) are conditionally independent given \(Z\) if,

\[ P(X \mid Y,Z) = P(X \mid Z). \]

Suppose a patient has the flu (\(Z\)). Given that the patient has the flu, having a fever (\(X\)) provides no additional information about whether they have body aches (\(Y\)); both symptoms are explained by the flu. Thus,

\[ P(\text{Fever},\text{Body Aches}\mid\text{Flu}) = P(\text{Fever}\mid\text{Flu}) P(\text{Body Aches}\mid\text{Flu}). \]

Conditional independence is fundamental in,

where it simplifies inference and reduces computational complexity.

Used in:

Also see:

Conditional Probability#

The probability of an event given that another event has occurred. For example,

\[ P(A|B) = \frac{P(A,B)}{P(B)} \]

We read \(P(A|B)\) as the probability of \(A\) “given” \(B\) has occurred. Conditional probability is calculated as the joint probability divided by the marginal probability of the conditioning event.

Conditional probabilities can be extended to multivariate cases by including additional conditioning events. For example,

\[ P(C|A,B) = \frac{P(A,B,C)}{P(A,B)} \]

Used in:

Confidence Interval#

A range of values that quantifies the uncertainty in a population parameter estimated from a sample. The range is constructed so that, over many repeated random samples, a specified proportion of the intervals contain the true population parameter. This proportion is called the confidence level.

Confidence intervals are commonly, but incorrectly, communicated as,

  • there is a 95% probability that the model slope, \(b_1\), is between 0.5 and 0.7.

Instead, confidence intervals should be interpreted as,

  • if we repeatedly drew random samples from the same population and calculated a 95% confidence interval for each sample, approximately 95% of those intervals would contain the true population slope.

The probability applies to the procedure used to construct the interval,

  • not to the unknown parameter itself. Once the interval has been calculated, the true parameter either lies within the interval or it does not.

To make the probabilistic statement that there is a 95% probability the parameter lies within an interval, Bayesian methods are required to calculate a credible interval.

Confidence intervals may be calculated with,

  • analytical methods, when available. For example, the confidence interval for the population mean is

\[ \bar{x}\pm t_{1-\alpha/2,\,n-1}\frac{s}{\sqrt{n}}, \]

where \(\bar{x}\) is the sample mean, \(s\) is the sample standard deviation, \(n\) is the sample size, and \(t_{1-\alpha/2,\,n-1}\) is the critical value from the Student’s \(t\)-distribution.

  • bootstrap methods, which estimate confidence intervals directly from repeated resampling of the observed data and can be applied when analytical solutions are unavailable or difficult to derive.

Used in:

Contrast with:

Confusion Matrix#

A matrix of frequencies that compares the predicted (columns) and actual (rows) categories of a categorical response feature to evaluate the performance of a classification model.

Confusion matrices are used to,

  • visualize and diagnose all combinations of correct and incorrect classifications; for example, category 1 may frequently be misclassified as category 3

  • identify systematic classification errors and confusion between classes

  • assess model performance, where perfect classification places all observations on the diagonal of the confusion matrix

  • calculate summary metrics of classification performance, such as precision, recall, and the F1-score

For binary classification, the confusion matrix is composed of,

  • True Positive (TP) – positive observations correctly classified as positive

  • False Positive (FP) – negative observations incorrectly classified as positive

  • True Negative (TN) – negative observations correctly classified as negative

  • False Negative (FN) – positive observations incorrectly classified as negative

Common summary metrics from the confusion matrix include,

  • classification precision – the proportion of predicted positive observations that are actually positive,

\[ \text{Precision}=\frac{TP}{TP+FP} \]

  • classification recall (sensitivity) – the proportion of actual positive observations that are correctly classified,

\[ \text{Recall}=\frac{TP}{TP+FN} \]

  • classification F1-score – the harmonic mean of precision and recall,

\[ F_1= 2\, \frac{\text{Precision}\times\text{Recall}}{\text{Precision}+\text{Recall}} \]

The F1 score balances precision and recall and is especially useful when the classes are imbalanced.

Used in:

Also see:

Connection#

A pathway that transfers information between nodes in a neural network.

Each connection,

  • carries the output of one node to one or more nodes in the next computation step

  • has an associated trainable weight that controls the influence of the transmitted information

Connections depend on the neural network architecture,

  • fully connected neural network (FCNN) - every node is connected to every node in the next layer

  • convolutional neural network (CNN) - connections are local, defined by the convolution filter as it moves across the feature map

  • recurrent neural network (RNN) - connections may pass information both forward and through recurrent feedback loops, allowing information from previous computations to influence future predictions

During training,

Used in:

Compare with:

Also see:

Continuous Feature#

A feature that can take any value within a continuous range of possible values. For example,

  • porosity = \(\{13.01\%, 5.23\%, 24.62\%\}\)

  • gold grade = \(\{4.56 \text{ g/t}, 8.72 \text{ g/t}, 12.45 \text{ g/t}\}\)

Used in:

Contrast with:

Continuous Interval Feature#

A continuous feature where differences between values are meaningful and equally spaced, but the zero point is arbitrary and does not represent the absence of the quantity.

For example,

  • Celsius temperature scale (the zero point is defined by convention)

  • calendar year (there is no objective zero year)

Continuous interval features can be compared using addition and subtraction operations, but multiplication and division comparisons are not meaningful.

Used in:

Contrast with:

Continuous Ratio Feature#

A continuous feature where differences between values are meaningful, the zero point represents absence of the measured quantity, and ratios are physically meaningful.

For example,

  • Kelvin temperature scale

  • porosity

  • permeability

  • saturation

Because ratio features have a true zero, multiplication and division operations are meaningful. For example, a permeability of 200 mD can be described as twice the permeability of 100 mD.

Used in:

Contrast with:

Continuously Differentiable#

A function is continuously differentiable if it satisfies two conditions:

  1. Differentiability – the derivative of the function exists at every point in its domain.

  2. Continuous derivative – the derivative is itself continuous, with no jumps or discontinuities.

A continuously differentiable function is said to belong to the class \(C^1\).

For example,

  • the \(L^2\) norm (more precisely, the squared \(L^2\) norm used in least squares regression) is continuously differentiable. As a result, linear regression and ridge regression have smooth loss functions, allowing partial derivatives to be used to derive closed-form solutions for the model parameters.

  • the \(L^1\) norm is not continuously differentiable because it is not differentiable at zero. Consequently, the LASSO regression objective does not have a closed-form solution. Instead, iterative optimization algorithms, such as coordinate descent or proximal gradient methods, are used to estimate the model parameters.

Used in:

Also see:

Convolution#

A mathematical operation that combines two functions by integrating (or summing) the product of one function with a shifted and reflected version of the other.

One interpretation is smoothing a function, where a weighting function (kernel), \(f(\Delta)\), is applied to calculate a weighted average of another function, \(g(x)\),

\[ (f*g)(x)=\int_{-\infty}^{\infty}f(\Delta)\,g(x-\Delta)\,d\Delta. \]

The convolution operation extends naturally to multiple dimensions. For three dimensions,

\[ (f*g)(x,y,z)= \iiint f(\Delta_x,\Delta_y,\Delta_z) g(x-\Delta_x,y-\Delta_y,z-\Delta_z) \,d\Delta_x\,d\Delta_y\,d\Delta_z. \]

For discrete data, such as images, convolution is written as a summation,

\[ (f*g)[i,j]=\sum_m\sum_n f[m,n]\,g[i-m,j-n] \]

Convolution is commutative, so either function may be shifted,

\[ (f*g)(x)=\int_{-\infty}^{\infty}f(\Delta)\,g(x-\Delta)\,d\Delta=\int_{-\infty}^{\infty}f(x-\Delta)\,g(\Delta)\,d\Delta \]

If the reflection is omitted, the operation becomes cross-correlation, a measure of similarity between two signals as a function of displacement.

Applications in machine learning include,

  • convolutional neural networks (CNNs) – learn multiple kernels to extract spatial features from images and other structured data. In practice, most CNN implementations perform cross-correlation rather than strict mathematical convolution.

  • kernel smoothing – apply a moving kernel, such as a Gaussian kernel, to smooth noisy observations and estimate trend models.

  • image processing – perform filtering operations such as blurring, sharpening, and edge detection.

Used in:

Also see:

Convolutional Neural Network#

A neural network architecture designed for data with spatial structure, especially images and gridded data, through the use of,

  • feature maps - intermediate representations that preserve the spatial arrangement of information through network layers

  • convolution - mathematical operation that applies learnable filters to summarize local spatial patterns, textures, and arrangements

Unlike fully connected neural networks,

  • convolutional neural networks preserve the spatial relationships between neighboring data values

  • connections are local, only considering a neighborhood defined by the convolution filter

  • the same filter weights are reused throughout the feature map, significantly reducing the number of trainable parameters

The convolution operation applies a filter kernel over the input feature map,

  • early network layers typically learn simple patterns such as edges, orientations, and textures

  • deeper network layers combine simpler patterns into increasingly complex features and structures

Convolutional neural networks commonly include,

  • convolution layers - learn spatial filters that transform input feature maps

  • activation functions - introduce nonlinear relationships

  • pooling layers - reduce spatial resolution while retaining important features

  • fully connected layers - combine learned features for final prediction or classification

Training convolutional neural networks,

Applications include,

  • image classification

  • object detection and segmentation

  • spatial pattern recognition

  • analysis of gridded scientific data, including geological models and remote sensing imagery

Used in:

Also see:

Core#

The primary direct sampling method for characterizing subsurface resources.

  • In oil and gas exploration and development, core samples are obtained by replacing the drill bit with a specialized core barrel to recover a continuous rock sample. This process is expensive and time-consuming, so core data are typically sparsely and selectively acquired, often targeting specific geological intervals of interest.

  • In mining exploration and grade control, core drilling (commonly diamond drilling with a core barrel) is widely used because ore bodies may be accessed through surface drilling or underground workings such as drifts or stopes. As a result, core data are often more common than indirect measurements such as well logs in these settings.

  • In soft sediment environments, gravity, piston, and similar coring methods are used to sample unconsolidated sediments in lakes and oceans.

What do we learn from core data?

  • Petrological properties (e.g., sedimentary structures, mineralogy, and grade), petrophysical properties (e.g., porosity and permeability), and geomechanical properties (e.g., elastic moduli and Poisson’s ratio).

  • Stratigraphic relationships and geological geometry through direct observation and spatial interpolation between wells and drill holes.

Core data are critical for subsurface resource interpretation. They provide the most direct observations available, anchor geological and reservoir models, and provide calibration data for indirect measurements.

Used in: TBD

Also see:

Core Point#

A sample that has a sufficient number of neighboring samples within a specified radius to represent a dense region of the feature space.

For DBSCAN, a point \(x_i\) is a core point if,

\[ |N_{\epsilon}(x_i)| \geq minPts \]

where \(N_{\epsilon}(x_i)\) is the set of samples within the \(\epsilon\) neighborhood of \(x_i\), and \(minPts\) is the minimum number of samples required to define a dense region.

Important aspects,

  • core points are the starting points for DBSCAN cluster growth

  • core points can expand a cluster by connecting to other core points and their neighboring samples

  • a sample that is not a core point may still belong to a cluster as a border point if it is within the \(\epsilon\) neighborhood of a core point

  • samples that are neither core points nor border points are classified as noise or outliers

The DBSCAN cluster growth process is based on,

\[ \text{Core Point} \rightarrow \text{Density-Reachable} \rightarrow \text{Density-Connected} \rightarrow \text{Density-Based Cluster} \]

Used in:

Also see:

Correlation Coefficient#

A standardized measure of the strength and direction of the linear relationship between two features.

To understand the correlation coefficient, start with variance, a measure of the dispersion of a single feature,

\[ \sigma^2_x = \frac{\sum_{\alpha=1}^{n}(x_{\alpha}-\overline{x})^2}{n-1} \]

We can replace one squared deviation with the deviation of a second feature, \(y\), to obtain the covariance,

\[ C_{xy} = \frac{\sum_{\alpha=1}^{n}(x_{\alpha}-\overline{x})(y_{\alpha}-\overline{y})}{n-1} \]

Covariance measures how features \(x\) and \(y\) vary together. However, covariance depends on the units and scale of both features. We standardize covariance by the product of the standard deviations of \(x\) and \(y\) to calculate the correlation coefficient,

\[ \rho_{xy} = \frac{C_{xy}}{\sigma_x\sigma_y} \]

or equivalently,

\[ \rho_{xy} = \frac{\sum_{\alpha=1}^{n}(x_{\alpha}-\overline{x})(y_{\alpha}-\overline{y})}{(n-1)\sigma_x\sigma_y}, \quad -1.0 \leq \rho_{xy} \leq 1.0 \]

The correlation coefficient ranges from \(-1.0\) to \(1.0\),

  • \(\rho_{xy}=1.0\) indicates a perfect positive linear relationship

  • \(\rho_{xy}=-1.0\) indicates a perfect negative linear relationship

  • \(\rho_{xy}=0.0\) indicates no linear relationship

The correlation coefficient is useful because it is,

  • independent of the dispersion or standard deviation of both features

  • dimensionless, allowing comparison of relationships between features with different units and scales

  • related to the coefficient of determination, R-squared, or \(R^2\), for a linear regression model with an intercept, where \(R^2=\rho_{xy}^2\)

  • when we replace covariance with a covariance function, correlation becomes the correlogram, \(\rho(\mathbf{h})\), a measure of spatial correlation over separation distance

Some cautionary notes about the correlation coefficient,

  1. correlation does not imply causation

  • causal analysis requires careful experiments with sufficient replicates and control of confounding features.

  1. correlation is sensitive to outliers - a single extreme value can substantially change the magnitude and direction of the correlation coefficient.

Used in:

Also see:

Correlogram#

A spatial univariate measure of similarity between a feature and itself separated by a lag vector. The correlogram is the covariance between values separated by lag distance normalized by the feature variance,

\[ \rho_x(\bf{h}) = \frac{1}{n \cdot \sigma_x^2} \sum_{\alpha=1}^{n} (x(\bf{u}_{\alpha})-\bar{x}) (x(\bf{u}_{\alpha}+\bf{h})-\bar{x}) \]

For standardized features with a mean of 0.0 and variance of 1.0, the correlogram simplifies to,

\[ \rho_x(\bf{h}) = \frac{1}{n} \sum_{\alpha=1}^{n} x(\bf{u}_{\alpha}) x(\bf{u}_{\alpha}+\bf{h}) \]

The correlogram is the covariance normalized by the variance,

\[ \rho_x(\bf{h}) = \frac{C_x(\bf{h})}{C_x(0)} = \frac{C_x(\bf{h})}{\sigma_x^2} \]

where \(C_x(\bf{h})\) is the covariance function and \(C_x(0)\) is the variance. Therefore, for standardized features with variance equal to 1.0,

\[ \rho_x(\bf{h}) = C_x(\bf{h}) \]

The correlogram is also related to the variogram,

\[ \gamma_x(\bf{h}) = \sigma_x^2(1-\rho_x(\bf{h})) \]

and for standardized features,

\[ \rho_x(\bf{h}) = 1-\gamma_x(\bf{h}) \]

The correlogram is easy to interpret since it represents the correlation between samples separated by a specified lag distance.

Used in: TBD

Also see:

Covariance#

A bivariate measure of how two features vary together.

  • positive covariance indicates the features tend to increase together

  • negative covariance indicates that as one feature increases, the other tends to decrease

  • covariance near zero indicates little or no linear relationship

For a sample,

\[ C_{x,y} = \frac{1}{n-1}\sum_{\alpha=1}^{n}(x_{\alpha}-\overline{x})(y_{\alpha}-\overline{y}) \]

The covariance can be interpreted relative to the more familiar correlation coefficient.

  • the correlation coefficient is the covariance standardized by the product of the standard deviations of the two features

\[ \rho_{x,y} = \frac{C_{x,y}}{s_x\,s_y} \]

Some other observations about correlation,

  • unlike the correlation coefficient, covariance depends on the units of the two features and is therefore most useful for mathematical calculations rather than direct interpretation.

  • by replacing the second feature with the same feature offset in space, we get the covariance function, a useful measure of spatial similarity.

Used in:

Also see:

Covariance Function#

A spatial covariance as a measure of similarity between a feature and itself separated by a lag vector. The covariance function is calculated as the average product of deviations from the mean for values separated by the lag vector,

\[ C_x(\bf{h}) = \frac{1}{n} \sum_{\alpha = 1}^{n} (x(\bf{u}_{\alpha})-\overline{x}) (x(\bf{u}_{\alpha}+\bf{h})-\overline{x}) \]

The covariance function, \(C_z(\bf{h})\), is the variogram, \(\gamma_z(\bf{h})\), flipped upside down relative to the sill, \(\sigma_z^2\),

\[ C_z(\bf{h}) = \sigma_z^2 - \gamma_z(\bf{h}) \]

The covariance function is also related to the correlogram,

\[ C_x(\bf{h}) = \sigma_x^2 \rho_x(\bf{h}) \]

where \(\rho_x(\bf{h})\) is the correlogram. For standardized features with a mean of 0.0 and variance of 1.0,

\[ C_x(\bf{h}) = \rho_x(\bf{h}) \]

We model variograms, but inside kriging and simulation methods they are often converted to covariance values for numerical convenience,

  • Covariance matrices are typically diagonally dominant because the variance occurs on the diagonal, improving numerical stability when solving the linear systems used to calculate kriging weights.

Also see:

Credible Interval#

A range of values that quantifies the uncertainty in an unknown parameter using its posterior probability distribution. The interval is constructed so that a specified probability of the posterior distribution lies within the interval.

Credible intervals are interpreted probabilistically, for the example of Bayesian Linear Regression,

  • there is a 95% probability that the model slope, \(b_1\), lies between 0.5 and 0.7, given the observed data and the prior information.

Unlike a frequentist confidence interval, the probability applies directly to the unknown parameter,

  • because Bayesian inference represents uncertainty about the parameter with a posterior probability distribution.

Credible intervals are calculated from the posterior distribution,

  • analytically, when a closed-form posterior distribution is available

  • numerically, using methods such as Markov chain Monte Carlo when analytical solutions are unavailable.

Used in:

Contrast with:

Also see:

Cross-Entropy Loss#

A loss function that quantifies the difference between predicted class probabilities and the observed class labels for classification.

For binary classification, the cross-entropy loss is,

\[ L=-\frac{1}{n}\sum_{i=1}^{n}\left[y_i\log(\hat{p}_i)+(1-y_i)\log(1-\hat{p}_i)\right], \]

where \(y_i\) is the observed class label and \(\hat{p}_i\) is the predicted probability of the positive class.

Cross-entropy loss,

  • is minimized when predicted probabilities closely match the observed class labels

  • strongly penalizes confident but incorrect predictions

  • is commonly used to train logistic regression, neural networks, and other classification models

Used in:

Also see:

Cross Validation#

A model evaluation approach that estimates predictive performance by repeatedly withholding portions of the data for validation, training the model with the retained data, and evaluating predictions over samples not used to estimate the model parameters.

  • holdout cross validation - a simple validation approach is a train-test split, where typically \(15\% - 30\%\) of the data are withheld for testing

  • cross validation fairness - cross validation should be conceptualized as a dress rehearsal for real-world model use; the data split must be representative and fair, resulting in a similar prediction difficulty to the planned application of the model (Salazar et al., 2022)

More advanced cross validation designs, include,

  • folds - multiple cross validation cycles that allow for testing over all samples

  • stratification - test splits that preserve class balance in training

  • groups - ensure that a single group is completely in train or test to avoid data leakage

  • time series - sliding windows and only foreward forecasts

Cross validation is commonly applied to evaluate model prediction accuracy and may also be used to assess uncertainty model performance, such as uncertainty calibration and reliability (Maldonado-Cruz and Pyrcz, 2021)

Cross validation methods include,

Used in:

Compare with:

Also see:

Cumulative Distribution Function#

Commonly known by its acronym CDF, it describes the accumulation of probability up to a specified value. The CDF is calculated as the cumulative sum of a discrete probability mass function or the integral of a continuous probability density function.

Important concepts about CDFs,

  • the CDF is stated as \(F_x(x)\), while the PDF is stated as \(f_x(x)\)

  • the CDF is the probability that a random sample, \(X\), is less than or equal to a specific value \(x\); therefore, the y-axis represents cumulative probability,

\[ F_x(x)=P(X \le x)=\int_{-\infty}^{x} f_x(u)\,du \]

  • for discrete distributions, the CDF is calculated by summing probabilities up to and including the value \(x\)

  • for CDFs there is no bin assumption; therefore, bins are defined by the resolution of the available data

  • the CDF is a monotonically non-decreasing function because a negative slope would indicate decreasing cumulative probability over an interval

The requirements for a valid CDF include,

  1. bounded probability:

\[ 0.0 \le F_x(x) \le 1.0, \quad \forall x \]

  1. non-decreasing probability:

\[ \frac{dF_x(x)}{dx} \ge 0.0, \quad \forall x \]

  1. probability closure at the limits:

\[ \lim_{x\rightarrow-\infty}F_x(x)=0.0 \]
\[ \lim_{x\rightarrow+\infty}F_x(x)=1.0 \]

Used in:

Curse of Dimensionality#

The collection of challenges associated with working in high-dimensional feature spaces, where the number of predictor features becomes large relative to the available data.

Challenges associated with high-dimensional spaces include,

  • visualization difficulty – data and model behavior become impossible to directly visualize as dimensionality increases

  • sampling difficulty – the amount of data required to adequately sample and represent a predictor feature space increases rapidly with dimensionality

  • sparse feature coverage – available training data occupy only a small fraction of the possible high-dimensional feature space, reducing the ability to generalize predictions

  • distance distortion – geometric intuition breaks down as dimensions increase; distances become less informative and observations tend to become similarly distant from each other

  • increased feature redundancy – multicollinearity and correlated predictor features become more likely as the number of features increases

The curse of dimensionality motivates,

  • feature selection – reducing the number of predictor features by retaining only the most informative variables

  • feature projection – transforming the original features into a lower-dimensional representation that combines information and reduces redundancy

Used in:

Also see:

Comments#

This was a basic introduction to geostatistics. If you would like more on these fundamental concepts I recommend the Introduction, Modeling Principles and Modeling Prerequisites chapters from my text book, Geostatistical Reservoir Modeling{cite}`pyrcz2014’.

I hope this is helpful,

Michael

The Author:#

Michael Pyrcz, Professor, The University of Texas at Austin Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers’ and geoscientists’ impact in subsurface resource development.

For more about Michael check out these links:

Twitter | GitHub | Website | GoogleScholar | Geostatistics Book | YouTube | Applied Geostats in Python e-book | Applied Machine Learning in Python e-book | LinkedIn

Want to Work Together?#

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

  • Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I’d be happy to drop by and work with you!

  • Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

  • I can be reached at mpyrcz@austin.utexas.edu.

I’m always happy to discuss,

Michael

Michael Pyrcz, Ph.D., P.Eng. Professor, Cockrell School of Engineering and The Jackson School of Geosciences, The University of Texas at Austin

More Resources Available at: Twitter | GitHub | Website | GoogleScholar | Geostatistics Book | YouTube | Applied Geostats in Python e-book | Applied Machine Learning in Python e-book | LinkedIn