Machine Learning Glossary#

Michael J. Pyrcz, Professor, The University of Texas at Austin

Twitter | GitHub | Website | GoogleScholar | Geostatistics Book | YouTube | Applied Geostats in Python e-book | Applied Machine Learning in Python e-book | LinkedIn

Chapter of e-book “Applied Machine Learning in Python: a Hands-on Guide with Code”.

Cite this e-Book as:

Pyrcz, M.J., 2024, Applied Machine Learning in Python: A Hands-on Guide with Code [e-book]. Zenodo. doi:10.5281/zenodo.15169139 DOI

The workflows in this book and more are available here:

Cite the MachineLearningDemos GitHub Repository as:

Pyrcz, M.J., 2024, MachineLearningDemos: Python Machine Learning Demonstration Workflows Repository (0.0.3) [Software]. Zenodo. DOI: 10.5281/zenodo.13835312. GitHub repository: GeostatsGuy/MachineLearningDemos DOI

By Michael J. Pyrcz
© Copyright 2024.

This chapter is a summary of essential Machine Learning Terminology.

Motivation for this Glossary#

Firstly, why create this glossary?

I received a request for a course glossary from students in my Subsurface Machine Learning graduate course. While I usually dedicate a definition slide in my lecture slide decks to important terms, various students requested a consolidated glossary of terminology as part of their course review materials. The e-book provides an excellent vehicle and motivation for creating this resource.

Let me begin with a confession. There is a Machine Learning Glossary written by Google developers. For those seeking the in depth, comprehensive list of machine learning terms please use this book! For those seeking a comprehensive and in-depth reference of geostatistical terminology, this book remains an excellent resource.

So why create another glossary?

  • Scope - By writing my own glossary, I can limit the scope and descriptions to the concepts covered in this course. I believe many students would be overwhelmed by the size, breadth, and mathematical notation of a comprehensive geostatistics glossary.

  • Statistical Linkages - At the same time in my opinion machine learning is statistical learning and I have taken liberty to include many standard statistical terms as a foundation for all things machine learing.

  • Workflow Integration - By integrating the glossary directly into the e-book, I can link terminology to detailed chapter discussions, demonstrations, and examples. The goal is to eventually populate all chapters with hyperlinks to glossary entries, enabling students to move easily between concepts and applications.

  • Evergreen Resource - Finally, like the rest of this e-book, I want the glossary to be an evergreen living document. It will continue to evolve with new concepts, improved explanations, and feedback from students and readers.

I put quite a bit of time into this project during summer 2026 and I am happy with the way that is has evolved,

  • More than a glossary, it has become an evergreen network of machine learning concepts.

Gaussian Anamorphosis#

A quantile transformation that maps any univariate distribution to a Gaussian distribution,

  • also known as a normal score transform

The transformation maps feature values through their cumulative probabilities,

\[ y = G_y^{-1}\left( F_x(x)\right) \]

where \(F_x\) is the cumulative distribution function of the original feature distribution and \(G_y\) is the Gaussian CDF.

The Gaussian probability density function is,

\[ f(x) = \frac{1}{\sigma \sqrt{2 \pi}} exp \left[-\frac{1}{2} \left(\frac{x-\mu}{\sigma} \right)^2 \right] \]

A shorthand notation for a normal distribution is,

\[ N[\mu,\sigma^2] \]

for example, \(N[0,1]\) is the standard normal distribution.

Properties of Gaussian distributions,

  • much of natural variation and measurement error can be approximately represented by Gaussian distributions

  • parameterized completely by the mean, variance, and correlation coefficients for multivariate Gaussian distributions

  • unbounded distribution with no minimum or maximum values; extreme values are increasingly unlikely, and practical applications often apply truncation limits

Warning, many workflows apply univariate Gaussian anamorphosis and then assume a bivariate or multivariate Gaussian distribution. This assumption is generally not correct, but transforming data to a true multivariate Gaussian distribution is often computationally difficult.

Methods that benefit from or require Gaussian distributed features,

  • Pearson product-moment correlation coefficients completely characterize multivariate relationships when data follow a multivariate Gaussian distribution

  • partial correlation coefficients have their strongest statistical interpretation under multivariate normality

  • sequential Gaussian simulation (SGS) assumes Gaussian distributions to reproduce spatial variability and the global distribution

  • Student’s t-test for differences in means assumes normally distributed populations

  • chi-square distributions are derived from sums of squares of Gaussian distributed random variables

  • Gaussian naive Bayes classification assumes Gaussian conditional distributions

Used in:

Also see:

Constrast with:

Generative Adversarial Network#

Machine learning architecture where two neural networks are trained competitively against each other in an adversarial learning process,

  • generator - neural network that generates synthetic data attempting to produce samples that the discriminator classifies as real

  • discriminator - neural network that classifies samples as real (from the training dataset) or generated (from the generator)

The adversarial training concept is,

  • the generator learns to create increasingly realistic synthetic samples

  • the discriminator learns to distinguish generated samples from real training samples

  • the generator improves based on feedback from the discriminator

The adversarial objective can be expressed as a minimax optimization problem,

\[ \min_G \max_D V(D,G)=E_{x\sim p_{data}(x)}[\log D(x)]+E_{z\sim p_z(z)}[\log(1-D(G(z)))] \]

where \(G\) is the generator, \(D\) is the discriminator, \(x\) represents real training data, and \(z\) is random input noise.

Additional points about generative adversarial networks,

  • the generator does not directly observe the real training samples; instead, it learns from the discriminator’s feedback about generated samples

  • the discriminator provides a learned objective function that guides the generator toward realistic data generation

  • directly minimizing distance to training samples may cause the model to reproduce existing samples rather than generate diverse new realizations

  • adversarial training encourages the generator to reproduce the statistical characteristics and variability of the training data

Training generative adversarial networks,

  • the generator starts with random weights and produces random synthetic samples

  • the discriminator starts with random weights and initially performs poorly at distinguishing real and generated samples

  • generator and discriminator training must remain balanced; a discriminator that becomes too accurate can provide weak learning signals to the generator

  • gradients from the discriminator are propagated backward through the network to update generator weights

Applications of generative adversarial networks include,

  • image enhancement and super resolution - increasing image resolution and improving visual quality

  • synthetic image generation - generating realistic images and scientific models

  • geostatistical modeling - generating heterogeneous subsurface realizations that reproduce spatial patterns and geological variability

Variants of generative adversarial networks include,

  • conditional generative adversarial network (cGAN) - incorporates labels or conditioning information to control generated outputs

  • cycleGAN - learns transformations between two image domains without paired examples

  • pix2pix - performs image-to-image translation using paired training examples

Training generative adversarial networks,

  • network weights and biases are estimated using backpropagation and gradient-based optimization

  • gradients pass through the discriminator and provide learning signals to update the generator

  • training can be challenging due to unstable optimization and balancing the competing objectives of the two networks

Used in:

Also see:

Geostatistics#

A branch of applied statistics that integrates,

  1. spatial (geological) context

  2. spatial relationships and continuity

  3. volume support and scale

  4. uncertainty

Geostatistics provides methods to characterize, model, predict, and simulate spatial phenomena by incorporating the spatial structure of the data and the uncertainty in the subsurface or other spatial systems to support optimum decision making.

The boundary between geostatistics and spatial statistics is debated. In this course,

  • geostatistics includes many spatial statistics methods because, in practice, useful approaches for modeling spatial phenomena are adopted and integrated into the geostatistical toolkit.

Geostatistics is an expanding and evolving field of study that continues to incorporate new statistical, computational, and data-driven approaches.

Gibbs Sampler#

A Markov chain Monte Carlo simulation algorithm that generates samples from a target probability distribution such that, after convergence, the ensemble of samples reproduces the statistics of the target distribution.

The Gibbs sampler is based on,

  • sequentially sampling from conditional probability distributions

Since only conditional probability density functions are required, the sampling process is simplified because the full joint probability density function is not directly required.

The basic steps of the Gibbs MCMC sampler for a bivariate case are,

  1. Assign initial random values for \(X(0)\), \(Y(0)\)

  2. Sample from \(f(X|Y(0))\) to obtain \(X(1)\)

  3. Sample from \(f(Y|X(1))\) to obtain \(Y(1)\)

  4. Repeat the sequential conditional sampling steps to generate samples,

\[ \ell = 1,\ldots,L \]

After sufficient iterations and convergence, the resulting samples reproduce the target joint distribution,

\[ f(X,Y) \]

The Gibbs sampler is particularly useful when the joint distribution is complex, but the conditional distributions are easier to sample from.

Used in: TBA - with Bayesian Linear Regression

Compare with:

Also see:

Global Accuracy#

Honoring (matching) global measures calculated over the entire volume of interest, for example,

Global accuracy is a primary objective for simulation, where the realizations should reproduce the input global statistics in expectation.

Used in:

Contrast with:

Global Measure#

A statistical or spatial summary calculated over the entire volume of interest. Examples include,

Global measures characterize the overall behavior of a feature or model and are commonly used for model checking and validation.

Used in:

Contrast with:

Gradient Boosting Model#

Machine learning prediction model that results from posing a boosting model as a gradient descent optimization problem.

For boosting,

  • at each step, \(k\), a new model is fit to improve the current estimator by reducing the prediction error

  • the model \(h_k(X_1,\ldots,X_m)\) is fit to the residual errors or, more generally, the negative gradient of the loss function

We assign a loss function, \(L\),

\[ L\left(y,F(X)\right) = \frac{\left(y - F(X)\right)^2}{2} \]

and minimize the \(\ell_2\) loss function,

\[ J = \sum_{i=1}^{n} L\left(y_i,F_k(X_i)\right) \]

by adjusting our model estimator \(F(X)\) over the training data.

We can take the partial derivative of the loss with respect to our model estimate,

\[ \frac{\partial J}{\partial F(X_i)} = F(X_i)-y_i \]

The residuals can be interpreted as negative gradients,

\[ y_i-F(X_i)=-\frac{\partial J}{\partial F(X_i)} \]

Therefore, fitting a new model to the residuals is equivalent to fitting a model to the negative gradient of the loss function.

The gradient descent update is,

\[ F_{k+1}(X_i)=F_k(X_i)+h_k(X_i) \]

where the new model \(h_k(X_i)\) approximates the negative gradient,

\[ F_{k+1}(X_i)=F_k(X_i)+y_i-F_k(X_i) \]

or equivalently,

\[ F_{k+1}(X_i)=F_k(X_i)-\frac{\partial J}{\partial F_k(X_i)} \]

The general gradient descent form is,

\[ \phi_{k+1}=\phi_k-\rho\frac{\partial J}{\partial \phi_k} \]

where \(\phi_k\) is the current state, \(\rho\) is the learning rate, \(J\) is the loss function, and \(\phi_{k+1}\) is the updated estimator state.

The prediction residual at the training data provides the gradient direction that guides each new model. Therefore, gradient boosting performs,

  • fitting a sequence of models to the negative gradients of the loss function

By approaching boosting as a gradient descent problem, we can apply different loss functions depending on the prediction problem and desired robustness.

  • \(\ell_2\) loss,

\[ L(y,F(X))=\frac{(y-F(X))^2}{2} \]

is commonly used because it provides a smooth optimization objective, but it is sensitive to outliers.

The negative gradient is,

\[ -\frac{\partial J}{\partial F_k(X_i)}=y_i-F_k(X_i) \]

  • \(\ell_1\) loss,

\[ L(y,F(X))=|y-F(X)| \]

is more robust to outliers.

The negative subgradient is,

\[ -\frac{\partial J}{\partial F_k(X_i)} = sign(y_i-F_k(X_i)) \]

  • other loss functions include Huber loss, which combines the robustness of \(\ell_1\) loss with the smooth optimization properties of \(\ell_2\) loss

Gradient boosting can be conceptionalized as,

  • a sequence of addivite weak learners, where each learner is trained to correct the remaining errors of the previous ensemble.

Used in:

Also see:

Gradient-based Optimization#

A method to train model parameters by iteratively minimizing a loss function. The general steps include,

  1. initialize model parameters with random or informed starting values

  2. calculate the loss function for the current model parameters

  3. calculate the loss function gradient, which indicates the direction of steepest increase in loss. For many models, the analytical gradient is not available, and numerical approximation or automatic differentiation may be used.

For numerical calculation of a local loss function derivative,

\[ \nabla L(y_{\alpha}, F(X_{\alpha}, b_1)) = \frac{L(y_{\alpha},F(X_{\alpha},b_1+\epsilon))-L(y_{\alpha},F(X_{\alpha},b_1-\epsilon))}{2\epsilon} \]

  1. update the parameter estimate by stepping in the direction that decreases the loss function,

\[ \hat{b}_{1,t+1}=\hat{b}_{1,t}-r\nabla L(y_{\alpha},F(X_{\alpha},b_1)) \]

where \(r\) is the learning rate or step size, \(\hat{b}_{1,t}\) is the current model parameter estimate, and \(\hat{b}_{1,t+1}\) is the updated parameter estimate.

Important concepts in gradient-based optimization include,

  • gradient search convergence - the optimization process attempts to find a minimum of the loss function; depending on the loss landscape, this may be a local minimum, global minimum, or stationary point

  • gradient search step size - the learning rate \(r\) controls the size of each optimization step; if \(r\) is too small, convergence may be slow, while if \(r\) is too large, the optimization may overshoot the minimum or diverge

  • multiple model parameters - gradients are calculated for all model parameters and represented as a gradient vector,

\[\begin{split} \nabla L(y_{\alpha},F(X_{\alpha},b_1,b_2))=\left[\begin{matrix}\frac{\partial L}{\partial b_1}\\\frac{\partial L}{\partial b_2}\end{matrix}\right] \end{split}\]

  • exploration of parameter space - optimization for machine learning model training is an exploration of a high-dimensional parameter space to identify parameter values that minimize the loss function

  • loss function definition - the loss function is typically an error metric that summarizes prediction error over the training data

Used in:

Also see:

Graph#

A mathematical structure that represents data and relationships between data samples, where each sample is represented as a node and connections between samples are represented as edges.

For a graph,

  • nodes - represent samples, entities, or observations

  • edges - represent pairwise relationships or connections between nodes

  • edge weights - can represent the strength, distance, similarity, or other characteristics of the relationship

For an undirected graph, edges are bidirectional, meaning the relationship is symmetric between connected nodes.

Graph applications include,

  • providing a convenient format to represent and summarize complex data structures and relationships

  • informing clustering analysis methods, such as spectral clustering

  • enabling prediction models that incorporate relational information, such as graph neural network

Used in:

Also see:

Graph Laplacian#

Matrix representing a graph by integrating connections between graph nodes, samples, number of connections for each graph nodes, samples. Calculated as,

\[ L = D - A \]

degree matrix, \(D\), minus adjacency matrix, \(A\), where,

Graph aplacian matrices are commonly used in,

Used in:

Also see:

Graph Neural Network#

A neural network architecture designed to learn from graph-structured data by exchanging information between connected nodes.

Graph neural networks learn representations by,

Applications include,

  • social and communication networks

  • molecular and biological networks

  • transportation and infrastructure networks

  • geological fault, fracture, and well networks

Contrast with:

Also see:

Gridded Data#

Data represented at regularly spaced locations over a 2D area of interest or 3D volume of interest, commonly used to represent maps and spatial models.

Gridded data are characterized by,

  • regularly spaced locations defined by grid dimensions and cell spacing

  • exhaustive coverage over the modeled domain, where every grid location contains a value or assigned missing value

  • implicit spatial relationships between neighboring grid locations

Gridded data may be stored as,

  • a .csv comma-delimited file, where data may be represented as a matrix with \(n_y\) rows and \(n_x\) columns

  • binary formats for more compact storage and faster processing, although these files are not human readable

Gridded data are commonly visualized directly, for example,

  • image representations using functions such as matplotlib’s imshow

  • contour maps and other spatial visualization methods

Examples of gridded data include,

  • geological interpretation-based maps

  • acoustic impedance models inverted from seismic reflection surveys

For spatial modeling problems,

  • model outputs or response features are often estimated over a regular grid to create continuous maps or 3D models

Also see:

Hard Data#

Data that is treated as certain due to having a high degree of certainty relative to other available information sources. Hard data usually comes from direct measurement or observation of the feature of interest, for example,

  • Core-based porosity, permeability, mineralogy, and facies observations

  • direct measurements of grade from drill core samples

  • laboratory measurements of rock or fluid properties

Hard data is considered sufficiently reliable that uncertainty in the measurement is commonly not explicitly modeled or integrated into subsequent workflows. Note, hard data is not necessarily error-free;

  • it represents information that is treated as certain for the purpose of the analysis.

Hard data generally has high resolution (small scale, volume support), but poor spatial coverage because only an extremely small proportion of the population is directly sampled. For example,

  • Core coverage deepwater oil and gas - well core may sample only one five hundred millionth to one five billionth of a deepwater reservoir, assuming 3 inch diameter cores with 10% core coverage in vertical wells with 500 m to 1,500 m spacing

  • Core coverage mining grade control - diamond drill hole cores may sample one eight thousandth to one thirty thousandth of an ore body, assuming HQ 63.5 mm diameter cores with 100% core coverage in vertical drill holes with 5 m to 10 m spacing

Hard data provides the most direct calibration of subsurface models but must be integrated with other information sources to overcome limited spatial coverage.

Used in: TBD

Contrast with:

Harmonic Mean#

A measure of central tendency that calculates the reciprocal of the arithmetic mean of reciprocals, commonly used when,

  • averaging rates or ratios

  • scale up, change of volume support for permeability, given flow across beds

For \(n\) values, the harmonic mean is,

\[ H = \frac{n}{\sum_{i=1}^{n}\frac{1}{x_i}} \]

The harmonic mean gives greater weight to smaller values and is useful when low values should strongly influence the average,

Also see:

Hermite Polynomial#

A family of orthogonal polynomials on the real number line, commonly used for representing functions with Gaussian-distributed variables.

Order

Hermite Polynomial \(H_n(x)\)

0th Order

\(H_0(x)=1\)

1st Order

\(H_1(x)=x\)

2nd Order

\(H_2(x)=x^2-1\)

3rd Order

\(H_3(x)=x^3-3x\)

4th Order

\(H_4(x)=x^4-6x^2+3\)

These polynomials are orthogonal with respect to the weighting function,

\[ w(x)=e^{-\frac{x^2}{2}} \]

which is proportional to the standard Gaussian probability density function without the scaling factor,

\[ \frac{1}{\sqrt{2\pi}} \]

The definition of orthogonality is,

\[ \int_{-\infty}^{\infty} H_m(x)H_n(x)w(x)\,dx=0 \]

for \(m \neq n\).

Therefore, Hermite polynomials are orthogonal over the interval \([-\infty,\infty]\) with respect to the standard normal probability distribution.

By applying Hermite polynomials instead of regular polynomial terms for polynomial basis expansion in polynomial regression, we reduce multicollinearity between the predictor features,

  • recall, polynomial basis expansion creates correlated predictor features because terms such as \(x\), \(x^2\), \(x^3\) are not independent

  • orthogonal polynomial bases create transformed features that are uncorrelated under the specified weighting function

Orthogonal polynomial bases can improve numerical stability and interpretation of polynomial regression models, especially when predictors follow approximately Gaussian distributions.

Used in:

Also see:

Heuristic Algorithm#

A practical algorithm that uses approximations, rules, or strategies to find a good solution to a difficult problem without guaranteeing the optimal solution.

A heuristic algorithm represents a compromise between optimality, accuracy, computational cost, and practicality,

  • trading an exact solution for a faster solution that is often sufficiently accurate for the application

Heuristic algorithms are commonly applied in,

  • machine learning

  • computer science

  • mathematical optimization

For example,

  • the optimal solution for \(k\)-means clustering involves searching a large solution space of possible cluster assignments. Since the number of possible assignments grows exponentially with the number of samples, a heuristic iterative algorithm is applied to efficiently find a practical clustering solution.

Used in:

Also see:

Hierarchical Clustering#

A family of clustering methods that construct a hierarchy of nested clusters, typically represented by a dendrogram,

  • representing the cluster solutions for \(k = [1, 2, \ldots, n-1, n]\) simultaneously

  • dendrogram provides a visual representation of the cluster hierarchy and allows the user to select the number of clusters by choosing a linkage distance (cut height)

The linkage distance depends on the linkage criterion used:

  • single linkage - minimum distance between clusters

  • complete linkage - maximum distance between clusters

  • average linkage - average pairwise distance

  • Ward’s linkage - increase in within-cluster variance (often the most popular)

Hierarchical clustering determines cluster groups through a sequence of merges or splits, rather than directly optimizing a fixed partition of the data.

Common approaches include,

  • agglomerative hierarchical clustering - start with \(n\) clusters, where each data sample is assigned to its own cluster, and iteratively merge the most similar clusters until a stopping criterion is reached

  • divisive hierarchical clustering - start with all data assigned to a single cluster, and iteratively divide clusters into smaller clusters until a stopping criterion is reached

Compared with partitional clustering methods,

  • hierarchical clustering produces a hierarchy of cluster solutions rather than a single clustering solution

  • \(k\)-means clustering is a partitional clustering method that iteratively updates cluster assignments while directly optimizing a single partition of the data

  • once clusters have been merged or split, hierarchical clustering generally does not revisit earlier decisions, making the method computationally efficient but potentially sensitive to early decisions

A helpful summary of clustering methods,

  • partitional clustering \(\rightarrow\) Find one best partition of the data.

  • hierarchical clustering \(\rightarrow\) Build a tree of nested clusters.

  • density-based clustering \(\rightarrow\) Find regions of high sample density.

Also see:

Histogram#

A bar chart representation of a univariate statistical distribution showing the frequency of samples over an exhaustive set of bins spanning the range of possible values.

These are the steps to build a histogram,

  1. Divide the continuous feature range of possible values into \(K\) equal size bins, \(\Delta x\):

\[ \Delta x = \left( \frac{x_{max} - x_{min}}{K} \right) \]

or use available category labels for categorical features.

  1. Count the number of samples (frequency) in each bin, \(n_k\), \(\forall k=1,\ldots,K\).

  2. Plot frequency versus the bin label (use bin centroid for continuous features).

The histogram y-axis represents frequency. When normalized by the total number of samples,

\[ p_k = \frac{n_k}{n} \]

the result is a normalized histogram, where the y-axis represents probability.

Additional comments about histograms,

  • typically plotted as bar charts

  • 2D histograms use a orthonormal view columns with 2 axes for features and along with the frequency axis, but not often used to attempt to visualize bivariate relationships

Used in:

Also see:

Holdout Cross Validation#

A cross validation method that partitions the available data into one training subset and one withheld testing subset.

Holdout validation is simple and computationally efficient but may produce variable performance estimates because the evaluation depends on a single train-test split.

Used in:

Also see:

Contrast with:

Hybrid Model#

A model that combines both deterministic model and stochastic model components.

Hybrid models separate predictable structure from uncertain variability by combining,

  • deterministic components - representing known relationships, physical processes, expert interpretation, or data-driven trends

  • stochastic components - representing spatial variability, uncertainty, and unresolved processes that cannot be deterministically modeled

Most geostatistical models are hybrid models. For example, an additive deterministic trend model and stochastic residual model:

\[ Z(\mathbf{u}) = m(\mathbf{u}) + R(\mathbf{u}) \]

where \(m(\mathbf{u})\) represents the deterministic trend and \(R(\mathbf{u})\) represents the stochastic residual.

Other examples include,

  • deterministic geological frameworks combined with stochastic property simulation

  • physics-based models calibrated or conditioned with stochastic uncertainty models

  • data-driven predictive models combined with probabilistic uncertainty models

Used in: TBA - deterministic trends and data nonstationarity discussion

Independence#

Two random events, \(A\) and \(B\), are independent if knowledge of one event provides no information about the likelihood of the other event. Mathematically, events are independent if and only if the following equivalent relationships are true,

  1. Joint probability:

\[ P(A \cap B) = P(A) \cdot P(B) \]

  1. Conditional probability:

\[ P(A|B) = P(A) \]

  1. Conditional probability:

\[ P(B|A) = P(B) \]

If any of these relationships are violated, then the events are dependent, indicating that some form of relationship exists between them.

  • Note that dependence does not necessarily imply causation.

In spatial modeling, independence indicates that knowing one feature or event provides no additional information about another feature or event. For example,

  • spatially independent samples have no correlation over the specified lag distance.

Used in:

Indicator Transform#

Indicator coding converts a random variable into a probability relative to a category or a threshold.

For a categorical feature, the indicator answers the question,

  • what is the probability that the data value or realization belongs to a specific category?

The indicator transform is,

\[\begin{split} i(\mathbf{u}; z_k) = \begin{cases} 1, & \text{if } Z(\mathbf{u}) = z_k \\ 0, & \text{if } Z(\mathbf{u}) \ne z_k \end{cases} \end{split}\]

For example,

  • given category \(z_2 = 2\), and data at \(\mathbf{u}_1\) with \(z(\mathbf{u}_1)=2\), then \(i(\mathbf{u}_1;z_2)=1\)

  • given category \(z_1 = 1\), and a random variable away from data at \(\mathbf{u}_2\), the probability that the realization belongs to category \(z_1\) is \(P(Z(\mathbf{u}_2)=z_1)=0.23\), therefore \(i(\mathbf{u}_2;z_1)=0.23\)

For a continuous feature, the indicator answers the question,

  • what is the probability that the realization is less than or equal to a threshold?

The indicator transform is,

\[\begin{split} i(\mathbf{u}; z_k) = \begin{cases} 1, & \text{if } Z(\mathbf{u}) \le z_k \\ 0, & \text{if } Z(\mathbf{u}) > z_k \end{cases} \end{split}\]

For example,

  • given threshold \(z_1 = 6\%\), and data at \(\mathbf{u}_1\) with \(z(\mathbf{u}_1)=8\%\), then \(i(\mathbf{u}_1;z_1)=0\)

  • given threshold \(z_4 = 18\%\), and a random variable away from data, \(Z(\mathbf{u}_2)\sim N(\mu=16\%,\sigma=3\%)\), then

\[ i(\mathbf{u}_2;z_4)=P\!\left(Z(\mathbf{u}_2)\le18\%\right)=0.75 \]

The indicator transform may be applied to an entire random function by transforming the random variable at every location. Indicator transforms provide the foundation for indicator kriging, indicator variograms, and indicator simulation.

Note, the indicator transform,

  • may be applied to encode data softness, in this case the data values are not strickly 0 or 1 with respect to each threshold or category at the a data location.

  • may also be applied to encode a constriant relationship, e.g., data value cannot be category 1, but there is no information about category 2 nor 3, or data value is between 0.1 and 0.4, but there is no information within this interval.

  • is analogous to the one-hot encoding approach for feature engineering commonly used to deal with categorical features in machine learning.

Used in:

Also see:

Inertia#

A label for the k-Means clustering loss function that summarizes the total within-cluster variation by measuring the squared distance between each sample and the prototype (centroid) of its assigned cluster,

\[ I = \sum_{i=1}^{K} \sum_{x_j \in C_i} \|x_j - \mu_i\|^2 \]

where,

  • \(K\) is the total number of clusters,

  • \(C_i\) represents the set of samples assigned to the \(i^{th}\) cluster,

  • \(x_j\) represents a data sample belonging to cluster \(C_i\),

  • \(\mu_i\) is the prototype (centroid) of cluster \(C_i\),

  • \(\|x_j - \mu_i\|^2\) is the squared Euclidean distance between sample \(x_j\) and the cluster prototype \(\mu_i\).

The samples, prototypes, and distance calculations are performed in \(m\)-dimensional feature space, where each observation is represented by features \(X_1,\ldots,X_m\).

  • by minimizing inertia, k-means finds clusters with the smallest possible within-cluster sum of squared distances, producing compact groups of similar samples.

  • since the total variance of the dataset is fixed, reducing within-cluster variation generally results in greater separation between clusters.

Used in:

Also see:

Inference#

The process of using a sample drawn from a population to infer properties of the entire population. For example,

  • given sparsely sampled well data (the sample) with porosity well log measurements, infer the porosity histogram of the entire reservoir (the population)

  • given sparsely sampled drill hole data (the sample) with gold grade measurements, infer the gold grade distribution and spatial variability throughout the ore body (the population)

  • in geostatistics, inference includes estimating statistical properties (e.g., histograms, variograms, and trends) as well as spatial models away from the sampled locations

  • the statistical discipline devoted to inference is known as statistical inference

Inference is a broad topic encompassing many statistical methods. For this e-book, we adopt this simplified, practical definition focused on subsurface characterization and modeling.

Used in:

Compare with:

Inlier#

A regression model accuracy metric based on the proportion of testing data with prediction errors within a specified margin, \(\epsilon\).

The inlier ratio is,

\[ I_R=\frac{1}{n_{\text{test}}}\sum_{i=1}^{n_{\text{test}}}I(y_i,\hat{y}_i) \]

where the indicator function is,

\[\begin{split} I(y_i,\hat{y}_i)=\begin{cases}1, & \text{if } |y_i-\hat{y}_i|\leq\epsilon \\0, & \text{otherwise}\end{cases} \end{split}\]

This provides a simple and intuitive measure of prediction accuracy by reporting the proportion of testing (or training) data with predictions that are sufficiently accurate for the application.

Important considerations include,

  • choice or margin - the choice of the acceptable prediction margin, \(\epsilon\), is subjective and should reflect the accuracy requirements of the specific application

  • impact of outliers - unlike metrics such as mean square error or mean absolute error, the inlier ratio does not distinguish between small and large errors once the prediction falls outside the specified margin

Used in:

Contrast with:

Instance-based Learning#

Family of predictive machine learning models that,

  • makes predictions by comparing a new observation, represented by the predictor features \(x_1,\ldots,x_m\), with similar observations stored in the training data.

  • also known as memory-based learning

Instance-based learning is characterized by,

  • lazy learning training strategy - instance-based methods generally defer learning until prediction time by storing the training data rather than fitting an explicit predictive model, the model is the training data with the hyperparameters

  • analogy-based prediction - predictions are made directly from the most similar training observations rather than from an explicit mathematical model

  • similarity measure - predictions depend on a measure of similarity or distance between observations, such as Euclidean distance

  • computational complexity - prediction cost generally increases with the number of training observations, \(n\), the number of predictor features, \(m\), and, for nearest-neighbor methods, the number of neighbors, \(k\)

Examples of instance-based learning predictive machine learning models include,

  • \(k\)-nearest neighbors (KNN) regression

  • \(k\)-nearest neighbors (KNN) classification

Used in:

Contrast with:

Intersection of Events#

The event in which two or more events occur together. For two events, \(A\) and \(B\), the intersection is denoted by,

\[ A \cap B \]

The probability of the intersection is,

\[ P(A \cap B) = P(A,B) = P(A \text{ and } B) \]

where we may state, “probability of A and B”. If \(A\) and \(B\) are independent, then the joint probability simplifies to,

\[ P(A,B) = P(A) \cdot P(B) \]

Without the assumption of independence, we apply the multiplication rule,

\[ P(A,B) = P(B|A) \cdot P(A) \]

or equivalently,

\[ P(A,B) = P(A|B) \cdot P(B) \]

Used in:

See also:

Irreducible Error#

The additive component of the expected testing mean square error that cannot be eliminated by improving the prediction model.

\[ \mathbb{E}\!\left[\left(y_0-\hat{f}(x_{0,1}^{(i)},\ldots,x_{0,m}^{(i)})\right)^2\right] = \underbrace{\left(\mathbb{E}\!\left[\hat{f}(x_{0,1}^{(i)},\ldots,x_{0,m}^{(i)})\right]-f(x_{0,1}^{(i)},\ldots,x_{0,m}^{(i)})\right)^2}_{\text{Model Bias}^2} + \underbrace{\mathbb{E}\!\left[\left(\hat{f}(x_{0,1}^{(i)},\ldots,x_{0,m}^{(i)})-\mathbb{E}\!\left[\hat{f}(x_{0,1}^{(i)},\ldots,x_{0,m}^{(i)})\right]\right)^2\right]}_{\text{Model Variance}} + \underbrace{\sigma_{\epsilon}^{2}}_{\text{Irreducible Error}} \]

Irreducible error represents uncertainty that cannot be explained by the available predictor features. Sources include,

Irreducible error is the only additive component of the expected testing mean square error that is independent of model complexity.

Irreducible error may be reduced by,

  • collecting additional or more informative predictor features

  • increasing the quantity and representativeness of the training data

  • improving measurement quality to reduce observational error

The component of irreducible error associated with inherent randomness cannot be eliminated by any prediction model.

Used in:

Also see:

Joint Probability#

Probability of an event in which two or more events occur together,

\[ A \cap B \]

The probability of the intersection is,

\[ P(A \cap B) = P(A,B) = P(A \text{ and } B) \]

where we may state, “probability of A and B”. If \(A\) and \(B\) are independent, then the joint probability simplifies to,

\[ P(A,B) = P(A) \cdot P(B) \]

Without the assumption of independence, we apply the multiplication rule,

\[ P(A,B) = P(B|A) \cdot P(A) \]

or equivalently,

\[ P(A,B) = P(A|B) \cdot P(B) \]

Used in:

See also:

K-Bins Discretization#

A feature transformation method based on dividing a continuous feature into \(K\) discrete intervals (bins) over the feature range, then assigning each sample to a bin. For one-hot encoding bin representation, a value of 1 is assigned if the sample belongs to a bin and 0 if it is outside the bin.

A feature transformation method based on dividing a continuous feature into \(K\) discrete intervals (bins) over the feature range, then assigning each sample to a bin. The output may be either,

  • a single bin label indicating the assigned interval

  • a one-hot encoding representation with a value of 1 if the sample belongs to a bin and 0 if it is outside the bin

Binning strategies include:

  • uniform width bins - divides the feature range into \(K\) intervals of equal width,

  • quantile bins - divides the feature values into \(K\) intervals containing approximately the same number of samples.

This is a continuous range is partitioned into discrete intervals to simplify representation,

  • analogous to constructing a histogram, where the count of samples within each bin forms the height of each bar.

  • analogous to posterization in image processing, where continuous pixel intensity or color values are reduced to a smaller number of discrete levels.

Methods and applications that utilize \(K\) bins discretization include,

  • basis expansion to represent nonlinear relationships by transforming features into a higher-dimensional space,

  • discretization of continuous features into categorical features for categorical methods such as the naive Bayes classification,

  • histogram construction and Chi-square tests for comparing differences between distributions,

  • mutual information-based feature discretization and feature selection.

Used in:

Also see:

K-Fold Cross Validation#

A cross validation method based on partitioning the data into \(K\) folds and looping over the folds,

  • withhold the current fold as validation data and train the model using the remaining \(K-1\) folds,

  • calculate the model performance metric over the withheld validation fold,

  • aggregate the validation performance over all folds to estimate model generalization performance.

Relating K-fold cross validation to the holdout method with a single train and test split,

  • K-fold cross validation provides a more robust performance estimate than a single holdout split by allowing all samples to be used for both training and validation, and averaging performance over folds reduces the sensitivity to a particular train-test partition.

  • the relative size of the training and validation datasets is controlled by \(K\); for example, \(K=4\) produces approximately \(25\%\) validation data per fold and \(75\%\) training data, while \(K=5\) produces approximately \(20\%\) validation data per fold and \(80\%\) training data.

  • K-fold cross validation is commonly applied for estimating model prediction accuracy and may also be used to assess uncertainty in model performance and goodness (Maldonado-Cruz and Pyrcz, 2021).

Used in:

Compare with:

Also see:

k-Means Clustering#

An unsupervised machine learning method for cluster analysis that assigns unlabeled data samples into groups by minimizing dissimilarity within clusters, represented by the inertia loss function,

\[ I = \sum_{i=1}^{K} \sum_{\alpha \in C_i} ||X_{\alpha} - \mu_i||^2 \]

where,

  • \(i\) is the cluster index,

  • \(K\) is the total number of clusters,

  • \(C_i\) represents the set of samples assigned to cluster \(i\),

  • \(\alpha\) is the data sample index,

  • \(X_{\alpha}\) is a data sample,

  • \(\mu_i\) is the prototype (centroid) of cluster \(i\),

  • \(||X_{\alpha}-\mu_i||^2\) is the squared Euclidean distance between a sample and its cluster prototype in \(M\)-dimensional feature space calculated as,

\[ ||X_{\alpha}-\mu_i|| = \sqrt{\sum_{m=1}^{M}(X_{m,\alpha}-\mu_{m,i})^2} \]

Here is a summary of important aspects for k-means clustering,

  • partitional clustering - provides a single set of group assignments by partitioning samples into \(K\) clusters.

  • number of clusters (\(K\)) - specified as a model hyperparameter that controls the number of groups.

  • exhaustive and mutually exclusive groups - every data sample is assigned to exactly one cluster.

  • prototype method - represents the training data using a set of synthetic prototypes in feature space. For k-means clustering, \(K\) prototypes are assigned and iteratively updated.

  • unsupervised learning - training data are unlabeled and assigned cluster labels based on proximity to prototypes in feature space. The underlying assumption is that samples that are similar, represented by proximity in feature space, should belong to the same cluster.

  • feature weighting - k-means depends on Euclidean distance between training samples and prototypes. Distance is treated as an inverse measure of similarity. If features have significantly different magnitudes or ranges, features with larger values dominate the loss function and control the resulting clusters. Common approaches include standardization or normalization of features, while unequal feature weighting may also be applied. In this demonstration, features are normalized to a range from 0.0 to 1.0.

  • heuristic algorithm iterative solution - initial prototypes are assigned in the feature space, sample labels are updated by assigning each sample to the nearest prototype, and prototypes are recalculated as the centroid of their assigned samples. This process repeats until cluster assignments no longer change.

Assumptions and limitations of k-means clustering,

  1. spherical, data convexity, isotropic clusters - minimizes distance between samples and their group prototype

  2. equal variance for all features - requires reliable measures of distance in feature space

  3. equal prior probability for all clusters - naïve a priori clustering membership information (uniform distribution)

Used in:

Also see:

Contrast with:

k-Nearest Neighbours#

Nonparametric predictive machine learning model based on a local weighting applied to the \(k\) nearest training data, resulting in a method that is,

  • very simple, interpretable and flexible - the summary of the \(k\) nearest training data can be easily explained

  • mapping - analogous to mapping in the predictor feature space

  • adaptive - in regions with few training data the result is smoother than regions with denser training data available

  • instance-based, lazy learning method - the model training is postponed until prediction is required, no precalculation of the model. i.e., prediction requires access to the data.

The k-nearest neighbours approach is conceptually related to convolution and kernel-based smoothing used in spatial interpolation. Both methods estimate a prediction at a location by applying weights to nearby information.

In convolution, a weighting function (kernel) is shifted across a function to calculate a weighted average,

\[ (f*g)(x)=\int_{-\infty}^{\infty} f(\Delta)g(x-\Delta)d\Delta \]

where the weighting function controls the contribution of neighboring values.

Similarly, k-nearest neighbours applies locally adaptive weighting by selecting the \(k\) closest training samples in feature space and combining their values for prediction.

  • unlike standard convolution, where the weighting function and window size are predefined, k-nearest neighbours uses a data-driven neighborhood size determined by \(k\).

The hyperparameters include,

  • \(k\) - number of nearest data to utilize for prediction

  • data weighting - for example uniform weighting with the local training data average, or inverse distance weighting

Note, for the case of inverse distance weighting, the method is analogous to inverse distance weighted interpolation with a maximum number of local data constraint commonly applied for spatial interpolation.

  • inverse distance is available in GeostatsPy for spatial mapping.

Too find the k-nearest data a distance metric is needed,

  • training data within the predictor feature space are ranked by distance (closest to farthest)

  • a variety of distance metrics may be applied, including:

  1. Euclidian distance

\[ d_i = \sqrt{\sum_{\alpha = 1}^{m} \left(x_{\alpha,i} - x_{\alpha,0}\right)^2} \]

  1. Minkowski Distance - a general expression for distance with well-known Manhattan and Euclidean distances are special cases,

\[ d_{(i,i')} = \left( \sum_{j=1}^{m} \left( x_{(j,i)} - x_{(j,i')} \right)^p \right)^{\frac{1}{p}} \]

  • when \(p=2\), this becomes the Euclidean distance

  • when \(p=1\) it becomes the Manhattan distance

Used in:

Also see:

Kernel Trick#

It is possible to incorporate a basis expansion in our method without ever needing to transform the training data to this higher dimensional space,

\[ h(x) \]

Instead, we only need the inner product over the predictor features,

\[ h(x) \left( h(x') \right)^T = \langle h(x), h(x') \rangle \]

instead of the actual values in the higher dimensional space, we just need the ‘similarity’ between all available training data in that transformed space! To emphasize this,

  • the radial basis function (Gaussian) kernel with Taylor series expansion has inifite components, i.e., infinite dimensional space, but all we need is this similarity,

\[ K(\mathbf{x},\mathbf{x}') = \exp\left(-\frac{\|\mathbf{x}-\mathbf{x}'\|^2}{2\sigma^2}\right) \]

An example application of the kernel trick for machine learning,

  • training our support vector machines with only a similarity matrix between training data that will be projected to the higher dimensional space

Used in:

Also see:

Key#

A component of the attention mechanism that allows a model to dynamically focus on the most relevant information while reducing the influence of less relevant information.

  • A Key (\(K\)) is a descriptor used to determine whether stored information is relevant to a given Query.

The attention mechanism,

  • compares each Query with all Keys to calculate similarity scores, which are then used to derive attention weights.

Some additional comments,

  • each Key–Value pair represents a stored piece of information, where the Key describes the information and the Value contains the associated content.

Used in:

Also see:

Kolmogorov Probability Axioms#

The three axioms proposed by Andrey Kolmogorov that establish the rigorous mathematical foundation for probability theory.

  1. Probability of an event is a non-negative number,

\[ P(A) \ge 0 \]

  1. Probability of the entire sample space, all possible outcomes \(\Omega\), is one (unity), also known as probability closure,

\[ P(\Omega) = 1 \]

  1. Additivity of mutually exclusive events for unions,

\[ P\left(\bigcup_{i=1}^{\infty} A_i\right) = \sum_{i=1}^{\infty} P(A_i) \]

For example, the probability of two mutually exclusive events \(A_1\) and \(A_2\) is,

\[ P(A_1 \cup A_2) = P(A_1) + P(A_2) \]

Used in:

Kriging#

Spatial estimation approach that relies on linear weights that account for spatial continuity, data closeness and redundancy. The kriging estimate is,

\[ z^*(\bf{u}) = \sum_{\alpha = 1}^{n} \lambda_{\alpha} \cdot z(\bf{u}_{\alpha}) + \left( 1.0 - \sum_{\alpha=1}^n \lambda_{\alpha} \right) \cdot m_z \]

  • the right term is the unbiasedness constraint, where one minus the sum of the weights is applied to the global mean.

In the case where the trend, \(t(\bf{u})\), is removed, we now have a residual, \(y(\bf{u})\),

\[ y(\bf{u}) = z(\bf{u}) - t(\bf{u}) \]

the residual mean is zero so we can simplfy our kriging estimate as,

\[ y^*(\bf{u}) = \sum_{\alpha = 1}^{n} \lambda_{\alpha} \cdot y(\bf{u}_{\alpha}) \]

The simple kriging weights are calculated by solving a linear system of equations,

\[ \sum_{j=1}^n \lambda_j C(\bf{u}_i,\bf{u}_j) = C(\bf{u},\bf{u}_i), \quad i=1,\ldots,n \]

that may be represented with matrix notation as,

\[\begin{split} \begin{bmatrix} C(\bf{u}_1,\bf{u}_1) & C(\bf{u}_1,\bf{u}_2) & \dots & C(\bf{u}_1,\bf{u}_n) \\ C(\bf{u}_2,\bf{u}_1) & C(\bf{u}_2,\bf{u}_2) & \dots & C(\bf{u}_2,\bf{u}_n) \\ \vdots & \vdots & \ddots & \vdots \\ C(\bf{u}_n,\bf{u}_1) & C(\bf{u}_n,\bf{u}_2) & \dots & C(\bf{u}_n,\bf{u}_n) \\ \end{bmatrix} \cdot \begin{bmatrix} \lambda_1 \\ \lambda_2 \\ \vdots \\ \lambda_n \\ \end{bmatrix} = \begin{bmatrix} C(\bf{u}_1,\bf{u}) \\ C(\bf{u}_2,\bf{u}) \\ \vdots \\ C(\bf{u}_n,\bf{u}) \\ \end{bmatrix} \end{split}\]

This system may be derived by substituting the equation for kriging estimates into the equation for estimation variance, and then setting the partial derivative with respect to the weights to zero.

  • we are optimizing the weights to minimize the estimation variance

this system integrates the,

  • spatial continuity - as quantified by the variogram (and covariance function to calculate the covariance, \(C\), values)

  • redundancy - the degree of spatial continuity between all of the available data with themselves, \(C(\bf{u}_i,\bf{u}_j)\)

  • closeness - the degree of spatial continuity between the available data and the estimation location, \(C(\bf{u}_i,\bf{u})\)

Kriging provides a measure of estimation accuracy known as kriging variance (a specific case of estimation variance).

\[ \sigma^{2}_{E}(\bf{u}) = C(0) - \sum^{n}_{\alpha = 1} \lambda_{\alpha} C(\bf{u}_0 - \bf{u}_{\alpha}) \]

Kriging estimates are best in that they minimize the above estimation variance.

Properties of kriging estimates include,

  • Exact interpolator - kriging estimates with the data values at the data locations

  • Kriging variance - a measure of uncertainty in a kriging estimate. Can be calculated before getting the sample information, as the kriging estimation variance is not dependent on the values of the data nor the kriging estimate, i.e. the kriging estimator is homoscedastic.

  • Spatial context - kriging takes integrates spatial continuity, closeness and redundancy; therefore, kriging accounts for the configuration of the data and structural continuity of the feature being estimated.

  • Scale - kriging by default assumes the estimate and data are at the same point support, i.e., mathematically represented as points in space with zero volume. Kriging may be generalized to account for the support volume of the data and estimate,

  • Multivariate - kriging may be generalized to account for multiple secondary data in the spatial estimate with the cokriging system. We will cover this later.

  • Smoothing effect - of kriging can be forecasted as the missing variance. The missing variance over local estimates is the kriging variance.

Kriging Variance#

A measure of accuracy and uncertainty for a kriging estimate, expressed as,

\[ \sigma^{2}_{z}(\bf{u}) = C(0) - \sum^{n}_{\alpha = 1} \lambda_{\alpha} C(\bf{u}_0 - \bf{u}_{\alpha}) \]

Kriging variance is a specific case of estimation variance,

\[ \sigma^{2}_{E}(\bf{u}) = E \left[ \left(z(\bf{u}) - z^{*}(\bf{u}) \right)^2 \right] \]

Can be calculated before getting the sample information,

  • the kriging estimation variance is not dependent on the values of the data nor the kriging estimate, i.e. the kriging estimator is homoscedastic.

Kriging-based Declustering#

A declustering method to assign weights to spatial samples based on local sampling density using the spatial continuity model from a variogram model. The objective is to assign weights such that the weighted statistics are intended to be more representative of the population.

  • samples in densely sampled areas receive less weight

  • samples in sparsely sampled areas receive more weight

Kriging-based declustering proceeds as follows:

  1. calculate and model the experimental variogram

  2. apply kriging to calculate estimates over a high-resolution grid covering the volume of interest

  3. accumulate the kriging weights assigned to each data sample over the entire grid

  4. assign declustering weights proportional to the accumulated kriging weights

The kriging-based declustering weight for data sample \(j\) is calculated as,

\[ w(\bf{u}_j) = n \cdot\frac{\sum_{iy=1}^{n_y}\sum_{ix=1}^{n_x}\lambda_{j,ix,iy}}{\sum_{j=1}^{n}\sum_{iy=1}^{n_y}\sum_{ix=1}^{n_x}\lambda_{j,ix,iy}} \]

where \(n\) is the number of data samples, \(n_x\) and \(n_y\) are the number of grid cells in the declustering grid, and \(\lambda_{j,ix,iy}\) is the kriging weight assigned to data sample \(j\) when estimating grid cell \(ix,iy\).

The resulting weights sum to the number of samples,

\[ \sum_{j=1}^{n} w(\bf{u}_j)=n \]

which allows the weights to be directly applied in weighted statistics.

Important considerations for kriging-based declustering,

  • like polygonal declustering, kriging-based declustering is sensitive to the boundaries of the area of interest; therefore, samples near the boundary may receive substantially different weights as the area of interest is expanded or contracted.

  • kriging-based declustering integrates the spatial continuity model from the variogram. Therefore, the variogram model selection can significantly impact the resulting weights.

  • if there is a 100% relative nugget effect, there is no spatial continuity and all samples receive equal weight. In this case, the kriging weights contain no spatial information and the declustering calculation must be handled separately to avoid division by zero.

  • geometric anisotropy may significantly impact the weights because data aligned along preferred directions may be considered spatially closer or farther based on the covariance function model.

Also see:

LASSO Regression#

A regularized version of linear regression. The acronym stands for,

  • Least Absolute Shrinkage and Selection Operator (LASSO),

and it has the same prediction equation as linear regression,

\[ y=\sum_{\alpha=1}^{m}b_{\alpha}x_{\alpha}+b_0 \]

where the trainable model parameters are \(b_{\alpha}\), the feature weights, and \(b_0\), the constant intercept term.

The model parameters are estimated by minimizing a regularized least squares loss function that combines the residual sum of squares (RSS) with an L1 shrinkage penalty,

\[ \sum_{i=1}^{n}\left(y_i-\left(\sum_{\alpha=1}^{m}b_{\alpha}x_{\alpha,i}+b_0\right)\right)^2+\lambda\sum_{\alpha=1}^{m}|b_{\alpha}| \]

where \(y_i\) is the observed response feature value and \(\sum_{\alpha=1}^{m}b_{\alpha}x_{\alpha,i}+b_0\) is the model prediction for training sample \(i\).

LASSO regression introduces the hyperparameter \(\lambda\), which controls the strength of the L1 regularization penalty,

\[ \lambda\sum_{\alpha=1}^{m}|b_{\alpha}| \]

where,

  • larger values of \(\lambda\) increase the penalty on large slope coefficients

  • the intercept term \(b_0\) is not penalized because it represents the response mean rather than predictor influence

LASSO regression integrates two competing goals during model training,

  • minimize prediction error with respect to the training data

  • minimize the magnitude of the slope parameters toward zero

The hyperparameter \(\lambda\) controls the model bias–variance trade-off,

  • as \(\lambda\rightarrow0\), the solution approaches ordinary linear regression with no additional regularization bias, but potentially higher model variance

  • as \(\lambda\) increases, model variance decreases and model bias increases as the model becomes less flexible

  • as \(\lambda\rightarrow\infty\), the slope parameters \(b_1,\ldots,b_m\) approach zero and predictions approach the response feature mean

Due to the L1 regularization penalty, LASSO regression performs feature selection,

  • as \(\lambda\) increases, some model parameters are forced exactly to zero, effectively removing the corresponding predictor features from the model

Compared with ridge regression,

  • Ridge regression uses L2 regularization and shrinks coefficients toward zero, but generally retains all predictor features

  • LASSO regression uses L1 regularization and can shrink some coefficients exactly to zero, performing automatic feature selection

The assumptions of LASSO regression include,

  • Error-free predictors - predictor features are treated as known values, not random variables

  • Linearity - the expected response is a linear combination of predictor features

  • Constant variance - the variance of response error is constant over predictor feature values (homoscedasticity)

  • independence of error - errors in the response are uncorrelated with each other

  • Redundant predictors - highly correlated predictor features may reduce the stability of feature selection

Used in:

Also see:

Latent Feature#

A feature that is not directly observed in the original data but is learned or inferred by a model to represent underlying patterns, structure, or relationships.

Latent features are typically combinations or transformations of the original predictor features and often provide a more compact or meaningful representation of the data.

Used in:

Also see:

Latent Space#

A lower-dimensional representation of data in which the coordinates correspond to Latent Feature that capture the underlying structure or patterns in the original data.

Latent features are not directly observed but are learned or inferred from the data during model training.

Data represented in a latent space often preserves important relationships while reducing dimensionality, enabling tasks such as visualization, clustering, compression, and generation.

Used in:

Also see:

Lazy Learning#

A machine learning approach where the training data are retained and generalization is performed when prediction queries are made.

  • after selecting model hyperparameters, the model retains the original training data and uses them directly during prediction

  • the computational effort is concentrated during the prediction phase, while training is typically fast

Examples include,

Used in:

Also see:

Contrast with:

Leave-One-Out Cross Validation#

A K-fold Cross Validation-based cross validation method with one sample withheld for testing in each fold, such that the number of folds equals the number of samples, \(K=n\).

Leave-one-out cross validation uses nearly all available data for training in each fold but may have high computational cost and high variance in the estimated predictive performance.

Contrast with:

Also see:

Likelihood Function#

In Bayes’ Theorem, the likelihood function describes the compatibility of new data with possible states or inferred model parameters. It quantifies how likely the observed data are for different model assumptions and is combined with the prior probability to calculate the posterior probability,

\[ P(A|B) = \frac{P(B|A) \cdot P(A)}{P(B)} \]

where:

  • \(P(A)\) is the prior probability representing uncertainty before observing new information,

  • \(P(B|A)\) is the likelihood function describing the compatibility of observations \(B\) with state or parameter \(A\),

  • \(P(B)\) is the evidence term used to normalize the posterior probability,

  • \(P(A|B)\) is the posterior probability representing updated uncertainty after incorporating observations.

Used in:

Also see:

Linear Regression#

Simple, linear parametric prediction model based on a linear weighted combination of the predictor features with a constant term,

\[ y=\sum_{\alpha=1}^{m}b_{\alpha}x_{\alpha}+b_0 \]

where the trainable model parameters are \(b_{\alpha}\), the feature weights, and \(b_0\), the constant intercept term.

The analytical solution for the model parameters, \(b_1,\ldots,b_m,b_0\), is available by minimizing the L2 norm loss function. This is known as least squares because the residual errors are squared and summed over all training data,

\[ RSS=\sum_{i=1}^{n}\left(y_i-\left(\sum_{\alpha=1}^{m}b_{\alpha}x_{\alpha,i}+b_0\right)\right)^2 \]

where \(y_i\) is the observed response feature value and \(\sum_{\alpha=1}^{m}b_{\alpha}x_{\alpha,i}+b_0\) is the model prediction for training sample \(i\).

The assumptions of linear regression include,

  • Error-free predictors - predictor features are treated as known values, not random variables

  • Linearity - the expected response is a linear combination of predictor features

  • Constant variance - the variance of the response error is constant over predictor feature values (homoscedasticity)

  • Iindependence of error - errors in the response are uncorrelated with each other

  • No multicollinearity - predictor features are not linearly redundant with each other

Used in:

Also see:

Local Accuracy#

Accuracy of a spatial estimate at individual locations, typically assessed by minimizing the local estimation uncertainty or estimation error.

For kriging, local accuracy is achieved by minimizing the estimation variance,

\[ \sigma^2_E(\mathbf{u}) = \text{Var}\left[ Z^*(\mathbf{u}) - Z(\mathbf{u}) \right] \]

where \(Z^*(\mathbf{u})\) is the estimated value and \(Z(\mathbf{u})\) is the unknown true value.

Some general observations about local accuracy,

  • estimation methods prioritize local accuracy by honoring nearby data and minimizing estimation variance.

  • locally accurate estimates are often smooth because they represent the best estimate at each location, but they generally do not reproduce the spatial variability of the modeled feature.

  • estimates optimized for local accuracy may not reproduce global measures, such as the histogram, variogram, or correlation coefficient.

Used in:

Contrast with:

Local Measure#

A statistical measure that evaluates model performance at individual locations or over a local neighborhood.

Local measures are used to assess local accuracy.

  • local measures focus on location-specific agreement between estimates and observations or local uncertainty.

  • local measures do not assess reproduction of global measures, such as the entire histogram, variogram, or correlation coefficient.

Used in:

Contrast with:

Logistic Regression#

A supervised machine learning method that predicts the probability of membership in a categorical feature by applying a logistic function to a linear combination of the predictor features.

  • unlike linear regression, logistic regression predicts probabilities between 0 and 1, which may then be converted to class assignments.

Model parameters are estimated by minimizing the cross-entropy loss during model parameter training.

Applications include,

  • binary classification

  • multiclass classification (with suitable extensions)

Used in: TBA

Also see:

Loss Function#

Mathematical function that quantifies and summarizes the loss associated with the prediction errors over the training data and provides the objective minimized during model training.

For supervised learning, the general form is,

\[ L(Y,\hat{Y}) \]

where \(Y\) is the observed response feature values and \(\hat{Y}\) are the corresponding model predictions.

A loss function,

  • quantifies the agreement between model predictions and the observed response feature values

  • provides a single objective for model parameter optimization

  • is minimized during model training

The total loss over the training data is commonly calculated by aggregating the individual sample losses,

\[ L=\sum_{i=1}^{n}L(y_i,\hat{y}_i) \]

or by averaging,

\[ L=\frac{1}{n}\sum_{i=1}^{n}L(y_i,\hat{y}_i) \]

depending on the learning algorithm.

Common regression loss functions include,

  • Mean Squared Error (MSE) - squares prediction errors, emphasizing larger errors

  • Mean Absolute Error (MAE) - uses absolute prediction errors and is more robust to outliers

  • Huber Loss - combines the robustness of MAE with the smooth optimization properties of MSE

Common classification loss functions include,

  • Cross-Entropy Loss (Log Loss) - compares predicted class probabilities with the observed categories

  • Hinge Loss - used by support vector machines to maximize the classification margin

Additional comments,

  • the choice of loss function influences the model parameters learned during training

  • the loss function used for training does not have to be the same metric used to evaluate model performance

  • regularization terms are often added to the loss function to reduce model complexity and improve generalization

Examples include,

Used in:

Also see:

Comments#

This was a basic introduction to geostatistics. If you would like more on these fundamental concepts I recommend the Introduction, Modeling Principles and Modeling Prerequisites chapters from my text book, Geostatistical Reservoir Modeling{cite}`pyrcz2014’.

I hope this is helpful,

Michael

The Author:#

Michael Pyrcz, Professor, The University of Texas at Austin Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers’ and geoscientists’ impact in subsurface resource development.

For more about Michael check out these links:

Twitter | GitHub | Website | GoogleScholar | Geostatistics Book | YouTube | Applied Geostats in Python e-book | Applied Machine Learning in Python e-book | LinkedIn

Want to Work Together?#

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

  • Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I’d be happy to drop by and work with you!

  • Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

  • I can be reached at mpyrcz@austin.utexas.edu.

I’m always happy to discuss,

Michael

Michael Pyrcz, Ph.D., P.Eng. Professor, Cockrell School of Engineering and The Jackson School of Geosciences, The University of Texas at Austin

More Resources Available at: Twitter | GitHub | Website | GoogleScholar | Geostatistics Book | YouTube | Applied Geostats in Python e-book | Applied Machine Learning in Python e-book | LinkedIn