Machine Learning Glossary#
Michael J. Pyrcz, Professor, The University of Texas at Austin
Twitter | GitHub | Website | GoogleScholar | Geostatistics Book | YouTube | Applied Geostats in Python e-book | Applied Machine Learning in Python e-book | LinkedIn
Chapter of e-book “Applied Machine Learning in Python: a Hands-on Guide with Code”.
Cite this e-Book as:
Pyrcz, M.J., 2024, Applied Machine Learning in Python: A Hands-on Guide with Code [e-book]. Zenodo. doi:10.5281/zenodo.15169139 
The workflows in this book and more are available here:
Cite the MachineLearningDemos GitHub Repository as:
Pyrcz, M.J., 2024, MachineLearningDemos: Python Machine Learning Demonstration Workflows Repository (0.0.3) [Software]. Zenodo. DOI: 10.5281/zenodo.13835312. GitHub repository: GeostatsGuy/MachineLearningDemos 
By Michael J. Pyrcz
© Copyright 2024.
This chapter is a summary of essential Machine Learning Terminology.
Motivation for this Glossary#
Firstly, why create this glossary?
I received a request for a course glossary from students in my Subsurface Machine Learning graduate course. While I usually dedicate a definition slide in my lecture slide decks to important terms, various students requested a consolidated glossary of terminology as part of their course review materials. The e-book provides an excellent vehicle and motivation for creating this resource.
Let me begin with a confession. There is a Machine Learning Glossary written by Google developers. For those seeking the in depth, comprehensive list of machine learning terms please use this book! For those seeking a comprehensive and in-depth reference of geostatistical terminology, this book remains an excellent resource.
So why create another glossary?
Scope - By writing my own glossary, I can limit the scope and descriptions to the concepts covered in this course. I believe many students would be overwhelmed by the size, breadth, and mathematical notation of a comprehensive geostatistics glossary.
Statistical Linkages - At the same time in my opinion machine learning is statistical learning and I have taken liberty to include many standard statistical terms as a foundation for all things machine learing.
Workflow Integration - By integrating the glossary directly into the e-book, I can link terminology to detailed chapter discussions, demonstrations, and examples. The goal is to eventually populate all chapters with hyperlinks to glossary entries, enabling students to move easily between concepts and applications.
Evergreen Resource - Finally, like the rest of this e-book, I want the glossary to be an evergreen living document. It will continue to evolve with new concepts, improved explanations, and feedback from students and readers.
I put quite a bit of time into this project during summer 2026 and I am happy with the way that is has evolved,
More than a glossary, it has become an evergreen network of machine learning concepts.
Activation Function#
A nonlinear transformation, \(\alpha(\cdot)\), applied at every neural network node after combining the weighted input signals and node bias,
where \(w_{i,j}\) are the connection weights, \(x_i\) are the input signals, \(b_j\) is the node bias, and \(a_j\) is the node output.
Activation functions serve several important purposes, including,
introduce nonlinearity, allowing neural networks to learn complex nonlinear relationships
prevent multiple linear layers from collapsing into an equivalent single linear model
control how information propagates through the network during prediction and training
Activation functions are also known as transfer function, (not to be confused with transfer function for decision making).
Common activation functions include,
Logistic (Sigmoid) - smooth nonlinear mapping to the range \([0,1]\), commonly used for binary classification outputs
Hyperbolic Tangent (tanh) - smooth nonlinear mapping to the range \([-1,1]\), centered on zero to improve optimization
Rectified Linear Unit (ReLU) - outputs zero for negative inputs and the input value otherwise, computationally efficient and often results in faster training
Softmax - transforms the output layer values into probabilities that sum to one for multi-class classification
Selection of the activation function influences,
training stability
computational efficiency
gradient propagation during backpropagation
Used in:
Also see:
Addition Rule#
Method to calculate the probability of any event (the union of outcomes, represented by “or” grammar). For example, the probability of \(A\) or \(B\) is calculated with the probability addition rule,
given mutually exclusive events we can generalize the addition rule as,
Used in:
Adjacency Matrix#
A matrix representing a graph that records the pairwise connections between all pairs of nodes representing samples.
the entries are indicators, with 0 indicating no connection and 1 indicating a connection
Adjacency matrices are commonly used in,
graph neural networks
graph autoencoders
Isomap dimensionality reduction
For example, in spectral clustering,
the adjacency matrix transforms a collection of samples into a graph, allowing clustering based on global connectivity rather than only pairwise similarity or distance
Some additional comments,
node self connection - the diagonal entries (self-connections) are typically set to 0
for undirected graphs - the adjacency matrix is symmetric
alternative - that stores the strength of the connection rather than only connected/not connected is the Affinity Matrix
Used in:
Also see:
Affine Correction#
A distribution rescaling method that applies a shift and linear scaling (stretching or squeezing) to a univariate distribution, for example, histogram.
For the case of affine correction of feature \(X\) to feature \(Y\),
we correct \(X\) to match the mean and variance of \(Y\):
where \(\overline{x}\) and \(\sigma_x\) are the mean and standard deviation of the original distribution, and \(\overline{y}\) and \(\sigma_y\) are the target mean and standard deviation.
The procedure,
centers the data by subtracting the original mean
rescales the deviations by the ratio of standard deviations
shifts the result to the target mean
Affine correction does not alter the distribution shape; it preserves the relative ordering of values and applies only a linear transformation. For transformations that modify distribution shape, see distribution transformation.
Used in:
Compare with:
Affinity Matrix#
A matrix representing a graph that stores the pairwise similarity (affinity) between all pairs of nodes (samples).
the entries are continuous values, with 0 indicating no similarity and larger values indicating greater similarity or stronger affinity
Affinity matrices are commonly used in,
graph neural networks
graph autoencoders
isomap dimensionality reduction
For example, in spectral clustering,
the affinity matrix transforms a collection of samples into a weighted graph, allowing clustering based on global connectivity rather than only pairwise similarity or distance
Some additional comments,
the diagonal entries are often set to the maximum affinity (self-similarity), although some implementations set them to 0 before constructing the graph Laplacian
affinity values are commonly calculated from a distance measure, for example with a Gaussian (RBF) kernel,
where \(d_{ij}\) is the distance between samples \(i\) and \(j\) and \(\sigma\) controls the neighborhood size
an alternative that records only whether nodes are connected is the adjacency matrix
Used in:
Also see:
Attention Mechanism#
A mechanism that allows a model to dynamically focus on the most relevant information while reducing the influence of less relevant information by answering,
what information is relevant?
where is the relevant information located?
how strongly should each piece of information influence the prediction?
Rather than relying only on nearby information, attention allows each query to compare itself with many candidate observations, features, or patterns and retrieve information according to their similarity. The three components of attention are,
Query (Q) – what information is needed right now?
Key (K) – a descriptor used to determine whether stored information is relevant.
Value (V) – the information associated with each key.
The attention mechanism proceeds by,
comparing each Query with all Keys to calculate similarity scores,
converting the similarity scores into Attention Weights,
calculating a weighted average of the Values using the attention weights.
Some additional comments,
each Key–Value pair represents a stored piece of information, where the key describes it and the value contains the associated content.
similarity between Query and Key vectors is commonly calculated with a scaled dot product.
attention allows models such as transformers and large language models (LLMs) to learn long-range relationships and contextual dependencies in very large datasets.
Used in:
Also see:
Attention Weights#
Normalized similarity scores that determine how strongly each Value contributes to the output for a specific Query.
The similarity score for each Query–Key pair is commonly calculated with a scaled dot product,
where \(d_k\) is the dimension of the Key vectors.
The similarity scores are then normalized with the softmax activation function to obtain the attention weights,
The attention weights satisfy,
The output of the attention mechanism is the weighted average of the Values,
How do we interpret the attention weights?
high weights – highly relevant information for the current Query.
low weights – less relevant information for the current Query.
Because the weights sum to one, they behave like a probability distribution over the available information, although they are learned importance weights rather than probabilities.
Used in:
Also see:
Area of Interest#
The 2D spatial domain that is being characterized, modeled, and evaluated to support subsurface decision making. In general, the area of interest,
is the subsurface reservoir for oil and gas, the ore body for mining, or the aquifer for hydrogeological applications
may include volume away from the reservoir or ore body to support data integration and extraction modeling
may be further subdivided into local regions or facies and modeled separately
is represented by a grid with features populated from data, estimation, or simulation
in 3D modeling is commonly called the volume of interest
the extent and grid cell size are selected based on a trade-off between model accuracy and computational complexity
Used in:
See also:
Autoencoder#
A neural network architecture designed to learn a compact representation of data by reconstructing the original input through an encoder-decoder structure.
An autoencoder consists of three main components,
encoder - sequence of layers that transforms the input data into progressively lower-dimensional representations
bottleneck (latent layer) - lowest-dimensional representation of the input information, forcing the network to compress the data in a latent space representation
decoder - sequence of layers that reconstructs the original input from the latent representation, often with a structure that mirrors the encoder
Training an autoencoder,
the input and output features are the same, with each output node corresponding to the matching input node
the network parameters are optimized to minimize the reconstruction error between the input and output
where \(x_i\) is the input value and \(\hat{x}_i\) is the reconstructed output.
Once trained, the bottleneck layer provides,
a latent representation - compressed features that capture the most important patterns in the original data
a lower-dimensional feature space for inference, visualization, and subsequent machine learning workflows
Autoencoders are commonly applied for,
dimensionality reduction - learning nonlinear feature representations beyond linear methods such as principal component analysis
denoising - reconstructing clean information from corrupted input data
anomaly detection - identifying observations that cannot be accurately reconstructed
latent-space modeling - performing inference, clustering, or prediction using learned compact representations
Variants of autoencoders include,
convolutional autoencoder - uses convolutional layers to preserve and learn spatial patterns in images and gridded data
variational autoencoder (VAE) - learns a probabilistic latent representation for generative modeling
Training autoencoders,
network weights and biases are estimated using backpropagation and gradient-based optimization
Used in:
Also see:
Average#
The average is a measure of central tendency. There are several useful interpretations of the average,
representative value - a single value to represent an entire distribution
estimate - given a distribution of values, the average minimizes the L2 norm (sum of squared error)
scale-up - under linear averaging of a spatial feature, the average is the correct upscaled value
expectation - if all outcomes are equiprobable, the average is equal to the expectation
For a sample, the average is,
Note, the average is quite sensitive to outliers.
Used in:
Also see:
Backpropagation#
An efficient algorithm that propagates the error gradient backward through a neural network to calculate the partial derivatives of the loss function with respect to every network weight and bias.
Backpropagation,
applies the chain rule from calculus to efficiently calculate gradients through multiple network layers
calculates the partial derivatives of the loss function with respect to all trainable weights and biases
provides the gradients required for gradient-based optimization
For example, the gradient at a hidden layer node, \(H_4\), is calculated from the gradient at an output node, \(O_6\), by repeated application of the chain rule,
After the gradients have been calculated,
gradient-based optimization updates the weights and biases using the learning rate
the process is repeated over many training iterations until the loss function converges
Additional comments,
backpropagation calculates gradients—it does not update the model parameters
parameter updates are performed by optimization methods such as gradient descent or stochastic gradient-based optimization
the bookkeeping is complicated by the many possible information pathways through the network, but the chain rule provides an efficient computational solution
Used in:
Also see:
Basis Expansion#
A technique in statistics and machine learning that transforms predictor features into a new set of, usually higher-dimensional, features called basis functions. Basis expansion increases the flexibility of a model by allowing linear methods to represent nonlinear relationships for regression and classification.
in mathematics, basis expansion represents a complex function as a linear combination of simpler basis functions that make the problem easier to solve
in machine learning, basis expansion transforms the original predictor features into a higher-dimensional feature space while still fitting a model that is linear in the transformed features
For a predictor feature \(x_i\), the transformed feature vector is
where \(h_j(x_i)\) is the \(j^{th}\) basis function.
For example, polynomial basis expansion used in polynomial regression uses the basis functions
Another example is Hermite polynomial basis expansion, which uses a set of orthogonal polynomial functions. Hermite basis functions are commonly used when the predictor variables follow a Gaussian distribution or when orthogonality between basis functions is desirable.
The first few Hermite polynomial basis expansion functions are,
and a linear model can then combine these basis functions to approximate nonlinear relationships in the original predictor feature space. After the transformation, a linear model can fit nonlinear relationships in the original predictor feature by learning a linear combination of the basis functions.
Used in:
Also see:
Basis Function#
A basis function is a mathematical building block used to construct more complex functions. It operates on the principle of linear combination:
by multiplying basis functions by learned weights and adding them together, we can approximate complex relationships and represent functions within a chosen function space.
A general basis expansion is written as,
where,
\(h_j(x)\) is the \(j^{th}\) basis function
\(w_j\) is the weight assigned to the \(j^{th}\) basis function
\(k\) is the number of basis functions
Common examples of basis functions include:
polynomial functions - regular polynomial regression
Hermite polynomial - polynomial regression with reduced feature correlation
Fourier functions
radial basis functions
spline functions.
In machine learning, basis functions allow simple models, such as linear regression, to represent nonlinear relationships by transforming the original predictor features into a richer feature space.
Used in:
Also see:
Bayes Theorem#
A theorem that relates conditional probabilities and provides the mathematical basis for Bayesian updating of uncertainty models given new information.
where:
\(P(A)\) is the prior probability representing uncertainty before considering new information,
\(P(B|A)\) is the likelihood function describing the compatibility of observations \(B\) with possible states or parameters \(A\),
\(P(B)\) is the evidence term used to normalize the posterior probability,
\(P(A|B)\) is the posterior probability representing updated uncertainty after incorporating observations.
Commonly applied in:
Bayesian Inference
Also see:
Compare with:
Used in:
Bayesian Linear Regression#
The Bayesian formulation of the linear regression model. Start with the frequentist formulation of linear regression,
where \(x\) is the predictor feature, \(b_1\) is the slope parameter, \(b_0\) is the intercept parameter, and \(\epsilon\) is the random error term.
in ordinary least squares regression, the model parameters are estimated by minimizing the \(L^2\) norm of the residual error vector between observed and predicted values.
In the Bayesian formulation, the regression model is expressed as a probability distribution for the response feature \(Y\),
where \(\beta\) is the vector of regression weights, \(X\) is the design matrix of predictor features, and \(\sigma^2\) is the homoscedastic error variance.
Instead of estimating a single set of regression parameters, Bayesian linear regression treats the model parameters as random variables and estimates their posterior distribution using Bayes’ theorem,
where,
\(p(\beta)\) is the prior distribution of the regression parameters
\(p(y|X,\beta)\) is the likelihood of observing the training data given the model parameters
\(p(\beta|y,X)\) is the posterior distribution of the regression parameters
How do we solve this?
for standard linear regression with conjugate prior assumptions, the posterior distribution can be calculated analytically.
for more complex models or non-conjugate priors, sampling methods such as Markov chain Monte Carlo can be used to approximate the posterior distribution.
Used in: Bayesian Linear Regression Description and Demonstration
Compare with:
Also see:
Bayesian Probability#
Probability framework that represents uncertainty using prior knowledge and new information. Prior information may be based on,
expert judgment
experience
historical data
physical understanding, or
previous observations.
Bayesian probability provides a formal approach to updating uncertainty as new information becomes available. The general approach,
start with a prior probability distribution representing uncertainty before considering new information
formulate a likelihood function describing the compatibility of new observations with possible states or parameters
combine the prior and likelihood using Bayes’ theorem to calculate the posterior probability distribution
continue updating uncertainty as additional information becomes available
Bayesian probability is applied to solve probability problems where prior knowledge, limited data, or sequential information updates are important,
Bayesian probability contrasts with the frequentist probability approach, which interprets probability primarily through long-run frequencies of repeated experiments
Used in:
Contrast with:
Also see:
Bayesian Updating#
The process of revising a prior probability distribution using new observations or evidence to obtain a posterior probability distribution according to Bayes’ theorem.
The Bayesian updating workflow is:
Specify a prior distribution representing existing knowledge.
Observe new data.
Evaluate the likelihood of the observations.
Apply Bayes’ Theorem to obtain the posterior distribution.
Use the posterior as the prior when additional observations become available.
Used in:
Bayesian Inference
Sequential Learning
Data Assimilation
Used in:
Also see:
Bayesian Updating for Classification#
Classification prediction problem from the perspective of Bayesian updating, based on the conditional probability of a category, \(k\), given \(n\) features, \(x_1, \dots , x_n\).
we can solve for this posterior with Bayesian updating,
let’s combine the likelihood function and prior for the moment,
we can expand the full joint distribution recursively as follows,
expansion of the joint with the conditional and prior,
continue recursively expanding,
we can generalize as,
Used in:
Also see:
Bias#
A trainable model parameter associated with a neural network node that shifts the weighted input before applying the activation function.
Information flows forward through the network,
each node forms a weighted sum of the incoming information, adds a bias, and applies an activation function
where \(b_j\) is the bias associated with node \(j\).
The bias determines the baseline activation of a node,
it shifts the weighted sum before the activation function is applied
it allows the activation threshold to be adjusted independently of the connection weights
During training,
the node biases are updated by backpropagation and gradient-based optimization to minimize the loss function
together with the connection weights, the biases store the information learned from the training data
Contrast with:
Also see:
Biased Spatial Sampling#
Spatial sampling such that the sample statistics are not representative of the population parameters. For example,
the sample variance is not the same as the population variance
Of course, the population parameters are not accessible, so we cannot directly calculate sampling bias, i.e., the difference between the sample statistics and the population parameters. Methods we can use to check for biased sampling,
evaluate the samples for preferential sampling, clustering, filtering, or survivorship bias.
apply declustering as a diagnostic to check for biased sampling
Used in:
Contrast with:
Big Data#
Identification of big data is based on a combination of these criteria:
Data Volume - many data samples and features, difficult to store, transmit and visualize
Data Velocity - high-rate collection, continuous data collection relative to decision making cycles, challenges keeping up with the new data while updating the models
Data Variety - data form various sources, with various types of data, types of information, and scales
Data Variability - data acquisition changes during the project, even for a single feature there may be multiple vintages of data with different scales, distributions, and veracity
Data Veracity - data has various levels of accuracy, the data is not certain
For common subsurface applications most, if not all, of these criteria are met. Subsurface engineering and geoscience are often working with big data!
Used in:
Also see:
Big Data Analytics#
The process of examining big data using statistical, computational, and machine learning methods to discover patterns, extract insights, and support decision making.
Used in:
Also see:
Binary Transform#
Indicator coding a random variable to a probability relative to a category or a threshold,
also known as the indicator transform
If \(i(\bf{u}:z_k)\) is an indicator for a categorical variable,
what is the probability of a realization equal to a category?
for example,
given threshold, \(z_2 = 2\), and data at \(\bf{u}_1\), \(z(\bf{u}_1) = 2\), then \(i(bf{u}_1; z_2) = 1\)
given threshold, \(z_1 = 1\), and a RV away from data, \(Z(\bf{u}_2)\) then is calculated as \(F^{-1}_{\bf{u}_2}(z_1)\) of the RV as \(i(\bf{u}_2; z_1) = 0.23\)
If \(I\{\bf{u}:z_k\}\) is an indicator for a continuous variable,
what is the probability of a realization less than or equal to a threshold?
for example,
given threshold, \(z_1 = 6\%\), and data at \(\bf{u}_1\), \(z(\bf{u}_1) = 8\%\), then \(i(\bf{u}_1; z_1) = 0\)
given threshold, \(z_4 = 18\%\), and a RV away from data, \(Z(\bf{u}_2) = N\left[\mu = 16\%,\sigma = 3\%\right]\) then \(i(\bf{u}_2; z_4) = 0.75\)
The indicator coding may be applied over an entire random function by indicator transform of all the random variables at each location.
Used in: -Binary and Indicator Transform Description and Demonstration
Also see:
Bivariate#
Involving two features (variables) simultaneously, often to study their relationship, dependence, or correlation. For examples see bivariate statistic.
Used in:
Compare with:
Also see:
Bivariate Statistic#
A summary measure calculated from two features (variables) measured over a collection of samples. Bivariate statistics describe the relationship, dependence, or correlation between two variables.
Examples include:
Scatter Plot – to visualize the relationship between two variables.
Joint Probability – to quantify the probability of outcomes from two variables occurring together.
Joint Probability Density Function – a complete probabilistic model of the relationship between two variables.
Used in:
Compare with:
Also see:
Boosting Model#
The sequential addition of multiple weak learners to build a stronger predictive model.
a weak learner is one that offers predictions only marginally better than random selection
Boosting proceeds with the following steps,
build a simple model with a relatively high error rate; although inaccurate, the model captures the general trend
calculate the prediction error (residuals)
fit another weak learner to the residuals
update the ensemble model with the new weak learner and repeat until a stopping criterion is reached
Now the steps with equations, the general workflow for predicting \(Y\) from \(X_1,\ldots,X_m\) is,
fit an initial weak learner,
calculate the residuals at the training data,
fit a new weak learner to the residuals,
update the ensemble model,
where \(\eta\) is the learning rate that controls the contribution of each new weak learner.
Additional comments,
the final regression estimator is the sum of the weak learners,
where \(\eta_k\) is the learning rate for the \(k^{th}\) learner (often a constant \(\eta\)).
the number of estimators, \(K\), is a hyperparameter that controls model flexibility and complexity
the learning rate, \(\eta\), controls how much each weak learner contributes to the final model
increasing \(K\) generally improves model performance, but too many estimators or an excessively large learning rate can result in an Overfit Model
Used in:
Contrast with:
Also see:
Bootstrap#
A statistical resampling procedure used to quantify uncertainty in a calculated statistic by repeatedly resampling from the available sample data. Some general comments,
sampling with replacement - \(n\) (number of data samples) Monte Carlo simulations from the empirical distribution of the dataset produces a realization of the resampled data.
simulates the data collection process - the fundamental idea is to approximate repeated sampling from the population by repeatedly resampling from the available data instead of collecting new samples.
bootstrap any statistic - the bootstrap is flexible because uncertainty can be characterized for almost any calculated statistic.
computationally efficient - repeated resampling generates realizations of the statistic that can be used to build an uncertainty distribution. A large number of realizations, \(L\), improves characterization of the uncertainty model.
calculates the entire uncertainty distribution - for any statistic, summary statistics of the uncertainty distribution can be calculated, such as the mean, P10, and P90 uncertainty in an estimated mean.
model bagging for machine learning - bagging applies bootstrap resampling to create multiple training datasets, train multiple predictive models, and aggregate predictions from the model ensemble to reduce prediction variance.
What are the limitations of bootstrap?
biased sample data will likely result in a biased bootstrap uncertainty model; samples should first be corrected for known bias, e.g., declustering
bootstrap requires a sufficient sample size to reliably characterize uncertainty
classical bootstrap quantifies uncertainty due to limited sampling but does not explicitly account for spatial context, including sample locations, volume of interest, or spatial continuity
a variant called spatial bootstrap accounts for spatial relationships during resampling
Used in:
Contrast with:
Also see:
Categorical Feature#
A feature or variable that can take one of a limited and usually fixed number of possible categories. Categories often represent qualitative classes and generally do not have inherent numerical meaning or ordering, unless categorical ordinal feature,
categories may have qualitative names, but are often represented by integer labels or binary indicator variables for computational analysis and modeling.
Used in:
Categorical Nominal Feature#
A categorical feature without a natural ordering relationship between categories. Examples include,
facies = {boundstone, wackestone, packstone, breccia}
minerals = {quartz, feldspar, calcite}
Categories may be assigned labels or integer codes for analysis, but the labels do not represent magnitude or ranking.
Used in:
Contrast with:
Opposite:
Categorical Ordinal Feature#
A categorical feature with a natural ordering relationship between categories. The ordering provides relative ranking information, but category differences may not represent equal numerical intervals. Examples include,
geologic age = {Miocene, Pliocene, Pleistocene} - ordered from older to younger rock
Mohs hardness = \(\{1, 2, \ldots, 10\}\) - ordered from softer to harder minerals
Used in:
Contrast with:
Causation#
A relationship in which a change in one feature or variable directly produces a change in another feature or variable through an underlying causal mechanism.
Some important characteristics of causal relationships are,
Temporal precedence – the cause precedes the effect in time.
Non-spuriousness – the relationship is not explained by random chance, confounding variables, or selection bias.
Mechanism – a plausible physical, biological, economic, or other process exists to explain how the cause produces the effect.
Consistency – the relationship is observed across different conditions, populations, times, or studies.
Strength of evidence – stronger and more reproducible relationships provide greater support for causation, although strength alone does not establish causality.
Establishing causation is considerably more difficult than identifying association.
in this book we generally focus on prediction and association rather than causal inference
remember, correlation does not imply causation; two variables may be highly correlated without one causing the other
Cell-based Declustering#
A declustering method that assigns weights to spatial samples based on local sampling density to reduce sampling bias and produce statistics that are more representative of the population (i.e., improve geostatistical sampling representativity]) in the presence of clustered spatial sampling. Data weights are assigned such that,
samples in densely sampled areas receive less weight
samples in sparsely sampled areas receive more weight
The goal of declustering is to reduce the influence of uneven sample locations on statistical estimates. For example, infill drilling or blast hole samples should not significantly change statistics for the area of interest simply because some locations have been sampled more densely.
Cell-based declustering proceeds as follows:
a cell mesh is placed over the spatial data and initial weights are assigned proportional to the inverse of the number of samples in each cell
the cell mesh size is varied, and a cell size is selected based on the resulting declustered statistics. Typically, the cell size that minimizes the declustered mean is selected when the sample mean is biased high, and the cell size that maximizes the declustered mean is selected when the sample mean is biased low
to reduce sensitivity to cell mesh position, the cell mesh is randomly shifted multiple times and the resulting declustering weights are averaged for each datum
The weights are calculated as:
where \(n_l\) is the number of data in the current cell, \(L_o\) is the number of cells containing data, and \(n\) is the total number of data.
Some highlights for cell-based declustering,
expert judgement to assign cell size based on nominal sample spacing (e.g., data spacing before infill drilling) may improve performance compared with automated cell size selection based only on minimizing or maximizing the declustered mean
cell-based declustering does not account for boundaries of the area of interest; therefore, samples near the boundary may appear more sparsely sampled and receive larger weights
cell-based declustering was introduced by Professor André Journel in 1983, and remains a foundational geostatistical declustering method.
Used in:
Also see:
Classification#
A supervised machine learning method that predicts a categorical response feature from one or more predictor features.
Classification learns a relationship between predictor features and response feature that is a categorical feature or discrete feature with categories,
given predictor features, \(X_1,\ldots,X_m\), the model estimates a class assignment, \(\hat{Y}\)
model predictions are evaluated by comparing predicted classes, \(\hat{Y}\), with observed classes, \(Y\)
Common classification methods include,
classification decision trees and neural networks
Contrast with:
Used in:
Clustered Spatial Sampling#
Spatial samples with locations preferentially selected or concentrated in certain areas, resulting in potentially biased statistics.
spatial samples are often clustered in locations associated with higher or more desirable values, for example, high porosity and permeability, good quality shale for unconventional reservoirs, or low acoustic impedance indicating higher porosity
Because the true population parameters are generally unknown, sampling bias cannot be directly calculated as the difference between sample statistics and population parameters. Methods to diagnose and address biased sampling include,
evaluate samples for preferential sampling and spatial clustering
apply declustering as a diagnostic method to evaluate the impact of clustered sampling on statistics
Used in:
Contrast with:
Cognitive Biases#
Cognitive biases are automatic mental shortcuts, or heuristics, that influence human reasoning and decision making. These shortcuts help humans efficiently process information under uncertainty, but they can also systematically distort interpretation of data, scientific evidence, and engineering decisions.
Common cognitive biases include:
Anchoring Bias - excessive influence of initial information or assumptions, even when later information suggests alternatives.
Availability Heuristic - overestimating the importance of information that is easily recalled or available, such as anecdotes.
Bandwagon Effect - increasing confidence in a belief because many others hold the same belief.
Blind-spot Effect - failing to recognize one’s own cognitive biases.
Choice-supportive Bias - favoring information that supports previous decisions or commitments.
Clustering Illusion - perceiving patterns in random data.
Confirmation Bias - preferentially considering information that supports existing beliefs or models.
Conservatism Bias - favoring established information over new evidence.
Recency Bias - giving excessive weight to recently acquired information.
Survivorship Bias - focusing only on successful or visible examples while ignoring missing cases.
Mitigating cognitive biases requires deliberate uncertainty analysis, quantitative evaluation of evidence, diverse perspectives, and critical review of assumptions.
Used in:
Cognitive Biases Itemization and Definitions TBA
Complementary Events#
The logical NOT relationship in probability. For a simple example, if we define event \(A\), then the complementary event, \(A^c\), represents NOT \(A\), and the resulting probability closure relationship is,
Complementary events can also be considered for multivariate and conditional probabilities. For example, for a bivariate conditional relationship,
Note that the conditioning event must remain the same for complementary probability closure.
Used in:
Computational Complexity#
A measure of the computational resources required to execute an algorithm or workflow as the size of the problem increases.
in machine learning, computational complexity helps us understand how algorithms scale with the number of training samples, predictor features, or model parameters
Computational complexity is commonly represented using “Big-O notation”,
where \(n\) is the size of the problem and \(f(n)\) describes how the computational cost grows asymptotically.
There are two primary components of computational complexity,
time complexity – the computational time required by an algorithm as the problem size increases
space complexity – the computer memory required by an algorithm as the problem size increases
For example, if the time complexity is \(O(n^3)\), where \(n\) is the number of training samples, then doubling the size of the dataset increases the computational time by approximately a factor of eight.
Additional important aspects of computational complexity,
worst-case complexity** – Big-O notation typically represents the upper bound on the computational cost for a given problem size
asymptotic complexity** – complexity describes the growth rate as \(n\) becomes large, ignoring constant factors and lower-order terms
algorithm assumptions** – complexity assumes a specific computational model and may depend on assumptions such as whether the data are already sorted
Common examples of time complexity include,
constant time, \(O(1)\) – accessing an element of an array by index
logarithmic time, \(O(\log n)\) – binary search on a sorted array
linear time, \(O(n)\) – finding the minimum value in an unsorted array
linearithmic time, \(O(n\log n)\) – merge sort and heapsort
quadratic time, \(O(n^2)\) – bubble sort or algorithms with nested loops over the data
cubic time, \(O(n^3)\) – multiplying two dense \(n \times n\) matrices using the classical algorithm
exponential time, \(O(2^n)\) – exhaustive search over all subsets of \(n\) variables
Used in: TBA
Conditional Independence#
Two random variables or events are conditionally independent given a third variable if, after accounting for the observed value of the third variable, knowledge of one provides no additional information about the other.
Firstly, recall random variables \(X\) and \(Y\) are independent if the joint probability is equal to the products of the marginal probabilities,
Similarly, random variables \(X\) and \(Y\) are conditionally independent given \(Z\) if,
or equivalently, random variables \(X\) and \(Y\) are independent if the conditional probability is equal to the marginal probability,
Similarly, random variables \(X\) and \(Y\) are conditionally independent given \(Z\) if,
Suppose a patient has the flu (\(Z\)). Given that the patient has the flu, having a fever (\(X\)) provides no additional information about whether they have body aches (\(Y\)); both symptoms are explained by the flu. Thus,
Conditional independence is fundamental in,
Bayesian networks
probabilistic graphical models
Bayesian inference - for example naive Bayes classification
where it simplifies inference and reduces computational complexity.
Used in:
Also see:
Conditional Probability#
The probability of an event given that another event has occurred. For example,
We read \(P(A|B)\) as the probability of \(A\) “given” \(B\) has occurred. Conditional probability is calculated as the joint probability divided by the marginal probability of the conditioning event.
Conditional probabilities can be extended to multivariate cases by including additional conditioning events. For example,
Used in:
Confidence Interval#
A range of values that quantifies the uncertainty in a population parameter estimated from a sample. The range is constructed so that, over many repeated random samples, a specified proportion of the intervals contain the true population parameter. This proportion is called the confidence level.
Confidence intervals are commonly, but incorrectly, communicated as,
there is a 95% probability that the model slope, \(b_1\), is between 0.5 and 0.7.
Instead, confidence intervals should be interpreted as,
if we repeatedly drew random samples from the same population and calculated a 95% confidence interval for each sample, approximately 95% of those intervals would contain the true population slope.
The probability applies to the procedure used to construct the interval,
not to the unknown parameter itself. Once the interval has been calculated, the true parameter either lies within the interval or it does not.
To make the probabilistic statement that there is a 95% probability the parameter lies within an interval, Bayesian methods are required to calculate a credible interval.
Confidence intervals may be calculated with,
analytical methods, when available. For example, the confidence interval for the population mean is
where \(\bar{x}\) is the sample mean, \(s\) is the sample standard deviation, \(n\) is the sample size, and \(t_{1-\alpha/2,\,n-1}\) is the critical value from the Student’s \(t\)-distribution.
bootstrap methods, which estimate confidence intervals directly from repeated resampling of the observed data and can be applied when analytical solutions are unavailable or difficult to derive.
Used in:
Contrast with:
Confusion Matrix#
A matrix of frequencies that compares the predicted (columns) and actual (rows) categories of a categorical response feature to evaluate the performance of a classification model.
Confusion matrices are used to,
visualize and diagnose all combinations of correct and incorrect classifications; for example, category 1 may frequently be misclassified as category 3
identify systematic classification errors and confusion between classes
assess model performance, where perfect classification places all observations on the diagonal of the confusion matrix
calculate summary metrics of classification performance, such as precision, recall, and the F1-score
For binary classification, the confusion matrix is composed of,
True Positive (TP) – positive observations correctly classified as positive
False Positive (FP) – negative observations incorrectly classified as positive
True Negative (TN) – negative observations correctly classified as negative
False Negative (FN) – positive observations incorrectly classified as negative
Common summary metrics from the confusion matrix include,
classification precision – the proportion of predicted positive observations that are actually positive,
classification recall (sensitivity) – the proportion of actual positive observations that are correctly classified,
The F1 score balances precision and recall and is especially useful when the classes are imbalanced.
Used in:
Also see:
Connection#
A pathway that transfers information between nodes in a neural network.
Each connection,
carries the output of one node to one or more nodes in the next computation step
has an associated trainable weight that controls the influence of the transmitted information
Connections depend on the neural network architecture,
fully connected neural network (FCNN) - every node is connected to every node in the next layer
convolutional neural network (CNN) - connections are local, defined by the convolution filter as it moves across the feature map
recurrent neural network (RNN) - connections may pass information both forward and through recurrent feedback loops, allowing information from previous computations to influence future predictions
During training,
the connection weights are updated by backpropagation and gradient-based optimization
stronger positive or negative weights increase the influence of information flowing along that connection
Used in:
Compare with:
Also see:
Continuous Feature#
A feature that can take any value within a continuous range of possible values. For example,
porosity = \(\{13.01\%, 5.23\%, 24.62\%\}\)
gold grade = \(\{4.56 \text{ g/t}, 8.72 \text{ g/t}, 12.45 \text{ g/t}\}\)
Used in:
Contrast with:
Continuous Interval Feature#
A continuous feature where differences between values are meaningful and equally spaced, but the zero point is arbitrary and does not represent the absence of the quantity.
For example,
Celsius temperature scale (the zero point is defined by convention)
calendar year (there is no objective zero year)
Continuous interval features can be compared using addition and subtraction operations, but multiplication and division comparisons are not meaningful.
Used in:
Contrast with:
Continuous Ratio Feature#
A continuous feature where differences between values are meaningful, the zero point represents absence of the measured quantity, and ratios are physically meaningful.
For example,
Kelvin temperature scale
porosity
permeability
saturation
Because ratio features have a true zero, multiplication and division operations are meaningful. For example, a permeability of 200 mD can be described as twice the permeability of 100 mD.
Used in:
Contrast with:
Continuously Differentiable#
A function is continuously differentiable if it satisfies two conditions:
Differentiability – the derivative of the function exists at every point in its domain.
Continuous derivative – the derivative is itself continuous, with no jumps or discontinuities.
A continuously differentiable function is said to belong to the class \(C^1\).
For example,
the \(L^2\) norm (more precisely, the squared \(L^2\) norm used in least squares regression) is continuously differentiable. As a result, linear regression and ridge regression have smooth loss functions, allowing partial derivatives to be used to derive closed-form solutions for the model parameters.
the \(L^1\) norm is not continuously differentiable because it is not differentiable at zero. Consequently, the LASSO regression objective does not have a closed-form solution. Instead, iterative optimization algorithms, such as coordinate descent or proximal gradient methods, are used to estimate the model parameters.
Used in:
Also see:
Convolution#
A mathematical operation that combines two functions by integrating (or summing) the product of one function with a shifted and reflected version of the other.
One interpretation is smoothing a function, where a weighting function (kernel), \(f(\Delta)\), is applied to calculate a weighted average of another function, \(g(x)\),
The convolution operation extends naturally to multiple dimensions. For three dimensions,
For discrete data, such as images, convolution is written as a summation,
Convolution is commutative, so either function may be shifted,
If the reflection is omitted, the operation becomes cross-correlation, a measure of similarity between two signals as a function of displacement.
Applications in machine learning include,
convolutional neural networks (CNNs) – learn multiple kernels to extract spatial features from images and other structured data. In practice, most CNN implementations perform cross-correlation rather than strict mathematical convolution.
kernel smoothing – apply a moving kernel, such as a Gaussian kernel, to smooth noisy observations and estimate trend models.
image processing – perform filtering operations such as blurring, sharpening, and edge detection.
Used in:
Also see:
Convolutional Neural Network#
A neural network architecture designed for data with spatial structure, especially images and gridded data, through the use of,
feature maps - intermediate representations that preserve the spatial arrangement of information through network layers
convolution - mathematical operation that applies learnable filters to summarize local spatial patterns, textures, and arrangements
Unlike fully connected neural networks,
convolutional neural networks preserve the spatial relationships between neighboring data values
connections are local, only considering a neighborhood defined by the convolution filter
the same filter weights are reused throughout the feature map, significantly reducing the number of trainable parameters
The convolution operation applies a filter kernel over the input feature map,
early network layers typically learn simple patterns such as edges, orientations, and textures
deeper network layers combine simpler patterns into increasingly complex features and structures
Convolutional neural networks commonly include,
convolution layers - learn spatial filters that transform input feature maps
activation functions - introduce nonlinear relationships
pooling layers - reduce spatial resolution while retaining important features
fully connected layers - combine learned features for final prediction or classification
Training convolutional neural networks,
connection weights in convolution filters and other network parameters are estimated by backpropagation and gradient-based optimization
Applications include,
image classification
object detection and segmentation
spatial pattern recognition
analysis of gridded scientific data, including geological models and remote sensing imagery
Used in:
Also see:
Core#
The primary direct sampling method for characterizing subsurface resources.
In oil and gas exploration and development, core samples are obtained by replacing the drill bit with a specialized core barrel to recover a continuous rock sample. This process is expensive and time-consuming, so core data are typically sparsely and selectively acquired, often targeting specific geological intervals of interest.
In mining exploration and grade control, core drilling (commonly diamond drilling with a core barrel) is widely used because ore bodies may be accessed through surface drilling or underground workings such as drifts or stopes. As a result, core data are often more common than indirect measurements such as well logs in these settings.
In soft sediment environments, gravity, piston, and similar coring methods are used to sample unconsolidated sediments in lakes and oceans.
What do we learn from core data?
Petrological properties (e.g., sedimentary structures, mineralogy, and grade), petrophysical properties (e.g., porosity and permeability), and geomechanical properties (e.g., elastic moduli and Poisson’s ratio).
Stratigraphic relationships and geological geometry through direct observation and spatial interpolation between wells and drill holes.
Core data are critical for subsurface resource interpretation. They provide the most direct observations available, anchor geological and reservoir models, and provide calibration data for indirect measurements.
For example, core data are often used to calibrate well log responses, such as facies classification and porosity estimation regression.
Used in: TBD
Also see:
Core Point#
A sample that has a sufficient number of neighboring samples within a specified radius to represent a dense region of the feature space.
For DBSCAN, a point \(x_i\) is a core point if,
where \(N_{\epsilon}(x_i)\) is the set of samples within the \(\epsilon\) neighborhood of \(x_i\), and \(minPts\) is the minimum number of samples required to define a dense region.
Important aspects,
core points are the starting points for DBSCAN cluster growth
core points can expand a cluster by connecting to other core points and their neighboring samples
a sample that is not a core point may still belong to a cluster as a border point if it is within the \(\epsilon\) neighborhood of a core point
samples that are neither core points nor border points are classified as noise or outliers
The DBSCAN cluster growth process is based on,
Used in:
Also see:
Correlation Coefficient#
A standardized measure of the strength and direction of the linear relationship between two features.
To understand the correlation coefficient, start with variance, a measure of the dispersion of a single feature,
We can replace one squared deviation with the deviation of a second feature, \(y\), to obtain the covariance,
Covariance measures how features \(x\) and \(y\) vary together. However, covariance depends on the units and scale of both features. We standardize covariance by the product of the standard deviations of \(x\) and \(y\) to calculate the correlation coefficient,
or equivalently,
The correlation coefficient ranges from \(-1.0\) to \(1.0\),
\(\rho_{xy}=1.0\) indicates a perfect positive linear relationship
\(\rho_{xy}=-1.0\) indicates a perfect negative linear relationship
\(\rho_{xy}=0.0\) indicates no linear relationship
The correlation coefficient is useful because it is,
independent of the dispersion or standard deviation of both features
dimensionless, allowing comparison of relationships between features with different units and scales
related to the coefficient of determination, R-squared, or \(R^2\), for a linear regression model with an intercept, where \(R^2=\rho_{xy}^2\)
when we replace covariance with a covariance function, correlation becomes the correlogram, \(\rho(\mathbf{h})\), a measure of spatial correlation over separation distance
Some cautionary notes about the correlation coefficient,
correlation does not imply causation
causal analysis requires careful experiments with sufficient replicates and control of confounding features.
correlation is sensitive to outliers - a single extreme value can substantially change the magnitude and direction of the correlation coefficient.
Used in:
Also see:
Correlogram#
A spatial univariate measure of similarity between a feature and itself separated by a lag vector. The correlogram is the covariance between values separated by lag distance normalized by the feature variance,
For standardized features with a mean of 0.0 and variance of 1.0, the correlogram simplifies to,
The correlogram is the covariance normalized by the variance,
where \(C_x(\bf{h})\) is the covariance function and \(C_x(0)\) is the variance. Therefore, for standardized features with variance equal to 1.0,
The correlogram is also related to the variogram,
and for standardized features,
The correlogram is easy to interpret since it represents the correlation between samples separated by a specified lag distance.
Used in: TBD
Also see:
Covariance#
A bivariate measure of how two features vary together.
positive covariance indicates the features tend to increase together
negative covariance indicates that as one feature increases, the other tends to decrease
covariance near zero indicates little or no linear relationship
For a sample,
The covariance can be interpreted relative to the more familiar correlation coefficient.
the correlation coefficient is the covariance standardized by the product of the standard deviations of the two features
Some other observations about correlation,
unlike the correlation coefficient, covariance depends on the units of the two features and is therefore most useful for mathematical calculations rather than direct interpretation.
by replacing the second feature with the same feature offset in space, we get the covariance function, a useful measure of spatial similarity.
Used in:
Also see:
Covariance Function#
A spatial covariance as a measure of similarity between a feature and itself separated by a lag vector. The covariance function is calculated as the average product of deviations from the mean for values separated by the lag vector,
The covariance function, \(C_z(\bf{h})\), is the variogram, \(\gamma_z(\bf{h})\), flipped upside down relative to the sill, \(\sigma_z^2\),
The covariance function is also related to the correlogram,
where \(\rho_x(\bf{h})\) is the correlogram. For standardized features with a mean of 0.0 and variance of 1.0,
We model variograms, but inside kriging and simulation methods they are often converted to covariance values for numerical convenience,
Covariance matrices are typically diagonally dominant because the variance occurs on the diagonal, improving numerical stability when solving the linear systems used to calculate kriging weights.
Also see:
Credible Interval#
A range of values that quantifies the uncertainty in an unknown parameter using its posterior probability distribution. The interval is constructed so that a specified probability of the posterior distribution lies within the interval.
Credible intervals are interpreted probabilistically, for the example of Bayesian Linear Regression,
there is a 95% probability that the model slope, \(b_1\), lies between 0.5 and 0.7, given the observed data and the prior information.
Unlike a frequentist confidence interval, the probability applies directly to the unknown parameter,
because Bayesian inference represents uncertainty about the parameter with a posterior probability distribution.
Credible intervals are calculated from the posterior distribution,
analytically, when a closed-form posterior distribution is available
numerically, using methods such as Markov chain Monte Carlo when analytical solutions are unavailable.
Used in:
Contrast with:
Also see:
Cross-Entropy Loss#
A loss function that quantifies the difference between predicted class probabilities and the observed class labels for classification.
For binary classification, the cross-entropy loss is,
where \(y_i\) is the observed class label and \(\hat{p}_i\) is the predicted probability of the positive class.
Cross-entropy loss,
is minimized when predicted probabilities closely match the observed class labels
strongly penalizes confident but incorrect predictions
is commonly used to train logistic regression, neural networks, and other classification models
Used in:
Also see:
Cross Validation#
A model evaluation approach that estimates predictive performance by repeatedly withholding portions of the data for validation, training the model with the retained data, and evaluating predictions over samples not used to estimate the model parameters.
holdout cross validation - a simple validation approach is a train-test split, where typically \(15\% - 30\%\) of the data are withheld for testing
cross validation fairness - cross validation should be conceptualized as a dress rehearsal for real-world model use; the data split must be representative and fair, resulting in a similar prediction difficulty to the planned application of the model (Salazar et al., 2022)
More advanced cross validation designs, include,
folds - multiple cross validation cycles that allow for testing over all samples
stratification - test splits that preserve class balance in training
groups - ensure that a single group is completely in train or test to avoid data leakage
time series - sliding windows and only foreward forecasts
Cross validation is commonly applied to evaluate model prediction accuracy and may also be used to assess uncertainty model performance, such as uncertainty calibration and reliability (Maldonado-Cruz and Pyrcz, 2021)
Cross validation methods include,
Holdout Cross Validation - a single split with one train and one withheld test subset of the data applied
K-fold Cross Validation - entire process repeated \(K\) times so that all data are tested and error is the average over all folds resulting in a smoother model hyperparameter tuning response
Leave-One-Out Cross Validation - K-fold cross validation with \(K=n\), resulting in only one sample withheld for each fold; it can have high computational cost and high variance in the estimated performance because each validation set contains only one sample.
Stratified K-fold Cross Validation - K-fold cross validation that preserves class proportions in each fold
Used in:
Tuning Shrinkage Term for Ridge Regression Description and Demonstration
Tuning Shrinkage Term for LASSO Regression Description and Demonstration
Tuning k for k-nearest Neighbours Regression Description and Demonstration
Compare with:
Also see:
Cumulative Distribution Function#
Commonly known by its acronym CDF, it describes the accumulation of probability up to a specified value. The CDF is calculated as the cumulative sum of a discrete probability mass function or the integral of a continuous probability density function.
Important concepts about CDFs,
the CDF is stated as \(F_x(x)\), while the PDF is stated as \(f_x(x)\)
the CDF is the probability that a random sample, \(X\), is less than or equal to a specific value \(x\); therefore, the y-axis represents cumulative probability,
for discrete distributions, the CDF is calculated by summing probabilities up to and including the value \(x\)
for CDFs there is no bin assumption; therefore, bins are defined by the resolution of the available data
the CDF is a monotonically non-decreasing function because a negative slope would indicate decreasing cumulative probability over an interval
The requirements for a valid CDF include,
bounded probability:
non-decreasing probability:
probability closure at the limits:
Used in:
Curse of Dimensionality#
The collection of challenges associated with working in high-dimensional feature spaces, where the number of predictor features becomes large relative to the available data.
Challenges associated with high-dimensional spaces include,
visualization difficulty – data and model behavior become impossible to directly visualize as dimensionality increases
sampling difficulty – the amount of data required to adequately sample and represent a predictor feature space increases rapidly with dimensionality
sparse feature coverage – available training data occupy only a small fraction of the possible high-dimensional feature space, reducing the ability to generalize predictions
distance distortion – geometric intuition breaks down as dimensions increase; distances become less informative and observations tend to become similarly distant from each other
increased feature redundancy – multicollinearity and correlated predictor features become more likely as the number of features increases
The curse of dimensionality motivates,
feature selection – reducing the number of predictor features by retaining only the most informative variables
feature projection – transforming the original features into a lower-dimensional representation that combines information and reduces redundancy
Used in:
Also see:
Data#
Data are observations collected to characterize a population or process. In spatial data analytics and geostatistics, three fundamental aspects determine the value of a dataset:
Data coverage - what proportion of the population has been sampled? In general, hard data have high resolution (small volume support), but poor spatial coverage. For example,
Core coverage in deepwater oil and gas may sample only one five hundred millionth to one five billionth of a reservoir, assuming 3-inch diameter cores with 10% core recovery in vertical wells spaced 500–1,500 m apart.
Core coverage for mining grade control may sample approximately one eight thousandth to one thirty thousandth of an ore body, assuming HQ (63.5 mm) cores with complete recovery in drill holes spaced 5–10 m apart.
\(\quad\) In contrast, soft data often provide excellent (sometimes complete) spatial coverage, but lower resolution, for example,
Seismic and other remote sensing measurements commonly cover the entire area of interest but have substantially lower spatial resolution, generally decreasing with depth.
Volume support (data scale) - what volume or scale is represented by each measurement? Examples include,
core tomography imaging at the pore scale (approximately 1–50 \(\mu\)m)
gamma ray well log sampled every 0.3 m with approximately 1 m radial investigation
ground-based gravity gradiometry with an effective resolution of approximately 20 m × 20 m × 100 m
Information content - what does the dataset tell us about the subsurface? Examples include,
grain size distributions used to calibrate permeability and saturation
fluid contacts used to identify oil-water contacts
structural dip and continuity used to infer reservoir connectivity
mineral grades used to delineate ore shells for mine planning
Used in: TBD
Data Analytics#
In this book, data analytics is used synonymously with:
Data Convexity#
A subset, \(A\), of Euclidean feature space is convex if for any two points \(x_1\) and \(x_2\) within \(A\), the entire line segment connecting these points is also contained within \(A\).
Mathematically,
where \(\lambda\) defines any location along the line segment between \(x_1\) and \(x_2\).
Considering data samples, \(x_i^k\), where \(i\) represents the sample index and \(k\) represents the predictor feature, the data occupy a region in predictor feature space where each axis corresponds to a predictor feature.
if for any two data samples \(x_1\) and \(x_2\) the entire line segment connecting them lies within the region occupied by the data, then the data distribution is convex
if portions of the connecting line segment fall outside the data region, the data distribution is non-convex
Data convexity is important in machine learning because many algorithms rely on assumptions about the structure of the feature space,
non-convex data structures may require nonlinear models or methods capable of capturing complex decision boundaries.
For example,
Used in:
Also see:
Data Preparation#
The workflow of transforming raw data into a reliable, consistent, and model-ready dataset for analysis and machine learning.
In many applied studies, especially in subsurface applications,
the majority of project effort is often dedicated to data preparation, quality control, and integration
data preparation requires domain knowledge combined with robust application of statistics and data analytics
Data preparation is essential due to common challenges with real-world data, including,
data curation – establishing format standards, version control, storage, transmission, security, and documentation practices
large data volumes – challenges with visualization, accessibility, computational management, and data exploration
large volumes of metadata – lack of consistent platforms, standards, and formats for describing data
engineering integration – combining diverse data sources with different scales, interpretations, measurement uncertainties, and levels of quality
data quality issues – missing values, inconsistent formats, measurement errors, duplicates, and incorrect records
Clean and well-documented databases are a prerequisite for successful data analytics and machine learning.
machine learning performance is limited by the quality and representativeness of the input data
garbage in, garbage out
Used in:
DataFrame#
A convenient Pandas data structure for working with tabular data. A DataFrame is a two-dimensional labeled data structure with rows representing samples or observations and columns representing features or variables.
DataFrames provide a convenient structure to,
store, access, and manipulate tabular data
load data from a variety of sources, including files, Python objects, databases, and Excel spreadsheets
calculate summary statistics and visualize data
perform data queries, sorting, filtering, and selection operations
complete data manipulation tasks, including cleaning, transformation, merging, and reformatting
store metadata and information about the dataset, such as dimensions, column names, data types, and missing values
Used in:
Compare with:
DBSCAN#
A density-based clustering algorithm that discovers groups in feature space by identifying regions with sufficient sample density.
DBSCAN is an acronym for Density-Based Spatial Clustering of Applications with Noise (Ester et al., 1996).
Clusters are expanded from dense regions using hyperparameters that define the neighborhood scale and minimum required density.
The primary DBSCAN hyperparameters are,
\(\epsilon\) – the radius of the local neighborhood measured in normalized feature space. This defines the scale or resolution of the clusters. If \(\epsilon\) is too small, many samples remain unassigned as outliers and clusters may be fragmented. If \(\epsilon\) is too large, distinct clusters may merge.
\(minPts\) – the minimum number of samples required within an \(\epsilon\) neighborhood to define a core point. Core points initialize and expand cluster groups.
Density is evaluated by counting the number of samples within an \(\epsilon\) neighborhood. In high-dimensional feature spaces, this neighborhood is defined using a distance metric applied across all dimensions.
An automated or guided approach for estimating \(\epsilon\) is available using a k-distance plot,
calculate the \(k\) nearest neighbor distance for every sample in normalized feature space
sort the distances in ascending order
select \(\epsilon\) near the point of maximum curvature (the elbow), representing a transition between dense clusters and sparse outliers
Salient aspects of DBSCAN clustering include,
Advantages – requires minimal prior knowledge of the number of clusters, can identify clusters with arbitrary shapes, and can efficiently identify outliers
Density-based cluster growth – samples are initially unassigned and clusters are iteratively expanded by connecting density-reachable core points and their neighboring samples
Mutually exclusive – like k-Means clustering, each assigned sample belongs to only one cluster group,
Non-exhaustive – some samples may remain unassigned and are classified as noise or outliers,
Used in:
Contrast with:
Also see:
Debiasing with Secondary Data#
When the full range of a primary feature is not sampled, declustering alone cannot remove sampling bias because parts of the feature distribution are completely missing. Instead, we use,
a secondary data feature that provides coverage over the entire area of interest
the relationship between the Primary Data feature and the secondary feature
to infer the unsampled portion of the primary feature distribution.
The relationship between the primary and secondary features may be established using,
a statistical model that extrapolates the primary feature into the unsampled range
a physical model based on scientific or engineering understanding
expert knowledge of the underlying process
Unlike declustering, which corrects for clustered spatial sampling, debiasing with secondary data addresses situations where part of the primary feature distribution has not been sampled at all.
Mentioned but not demonstrated (to be added later) in: TBD
Compare with: TBA
Decision Criteria#
An engineered feature or metric calculated from one or more subsurface models to support decision making. Decision criteria quantify the consequences of alternative decisions and may represent economic value, technical performance, environmental impact, health and safety, or combinations of these objectives. For example,
contaminant recovery rate to support the design of a pump-and-treat soil remediation project
oil in place to determine whether a reservoir should be developed
Lorenz coefficient as a heterogeneity measure to classify a reservoir and identify appropriate analogs
recovery factor or production rate to schedule production and optimize facilities
recovered mineral grade and tonnage to determine the economic ultimate pit shell
In quantitative decision workflows, the decision criterion is used to rank competing alternatives. Common approaches include,
maximizing a profit metric
minimizing a loss function
Best practice is to define decision criteria that directly represent project value. For example,
rather than stopping at hydrocarbon in place, continue through engineering and economics to estimate project profit (currency).
Used in:
Also see:
Decision Making#
The ultimate objective of data science, including machine-learning, geostatistics and data analytics is to support better decisions. Estimation, prediction, uncertainty modeling , and machine learning are intermediate steps whose value is realized only when they improve decisions.
Decision making involves selecting the best,
estimate
choice
from a set of alternatives.
In quantitative decision workflows, a decision criteria is used to rank competing alternatives. Common approaches include,
maximizing a profit metric
minimizing a loss function
to identify the optimum estimate or decision.
when accounting for uncertainty, this optimization is performed over an ensemble of subsurface realizations and scenarios.
Used in:
Also see:
Decision Tree#
An intuitive supervised regression and classification machine learning model that divides the predictor feature space, \(X_1,\ldots,X_m\), into \(J\) mutually exclusive and exhaustive regions, \(R_j\).
mutually exclusive – any combination of predictor values belongs to only one region, \(R_j\)
exhaustive – all possible combinations of predictor values belong to one of the regions, meaning the regions cover the entire feature space
For regression, the prediction within each region is the mean of the training responses in that region,
where \(n_j\) is the number of training samples within region \(R_j\).
For classification, the prediction within each region is the most common class, determined by the mode or argmax operator,
where \(k\) represents the possible response classes.
Other salient points about decision trees,
supervised learning – the response feature label, \(Y\), is available for the training data and used to construct the model
hierarchical, binary segmentation – the predictor feature space begins as a single region and is sequentially divided into smaller regions through binary splits
compact, interpretable model – because each split is based on a single predictor feature, the model can be represented as a tree structure with binary branches. The resulting model can be implemented as nested if statements, for example,
if porosity > 0.15:
if brittleness < 20:
initial_production = 1000
else:
initial_production = 7000
else:
if brittleness < 40:
initial_production = 500
else:
initial_production = 3000
The decision tree is constructed from the top down. We begin with a single region that covers the entire feature space and then proceed with a sequence of splits,
scan all possible splits over all regions and all predictor features
greedy optimization – select the split that provides the greatest improvement in prediction accuracy. For regression trees, this is achieved by minimizing the residual sum of squares (RSS),
where \(\hat{y}_{R_j}\) is the mean response prediction within region \(R_j\).
for classification trees, splits are selected by minimizing a class impurity measure, such as Gini impurity,
where \(p_k\) is the proportion of samples belonging to class \(k\) within region \(R_j\).
A pure classification region contains only one class,
while higher values indicate greater mixing of classes.
Each split is optimized locally using a greedy algorithm without considering future splits.
Hyperparameters include,
number of regions – controls the complexity of the tree and directly determines the number of terminal nodes
minimum reduction in RSS or impurity – prevents splits that provide insufficient improvement; however, stopping too early may prevent later beneficial splits
minimum number of training samples in each region – controls the reliability of regional predictions and helps prevent an overfit model
maximum tree depth – limits the number of sequential splits and controls model complexity
Used in:
Declustered Statistics#
Once declustering weights are calculated for a spatial dataset, the unweighted (also called naive) statistics are replaced with weighted statistics that account for the declustering weights. These corrected statistics are then used as input for all subsequent analysis and machine learning modeling to mitigate sampling bias. For example,
the declustered mean is used as the stationary global mean for simple kriging.
the weighted cumulative distribution function is used in sequential Gaussian simulation so that the back-transformed realizations reproduce the declustered distribution.
Any sample statistic can be computed using declustering weights, including the entire cumulative distribution function (CDF). Examples include:
weighted sample mean,
where \(n\) is the number of data.
weighted variance,
where \(\overline{x}_{wt}\) is the declustered mean.
weighted Covariance,
where \(\overline{x}_{wt}\) and \(\overline{y}_{wt}\) are the declustered means for features \(X\) and \(Y\).
weighted cumulative distribution function (CDF),
This expression represents the empirical weighted CDF evaluated at the observed data values. Between observations, the CDF is obtained by interpolation.
No declustering method can guarantee improved estimates of the population parameters for every dataset.
however, when preferential sampling is present, declustering methods generally reduce sampling bias and provide improved statistical estimates in expectation.
Used in:
Also see:
Declustering#
A family of methods that assign weights to spatial samples based on local sampling density so that weighted statistics are more representative of the inaccessible population. Data weights are assigned so that,
samples in densely sampled areas receive less weight
samples in sparsely sampled areas receive more weight
There are various declustering methods:
It is important to note that no declustering method can prove that for every data set the resulting weighted statistics will improve the prediction of the population parameters, but in expectation these methods tend to reduce the bias.
these data weights may be applied to improve machine learning model prediction accuracy
Used in:
Also see:
Degree Matrix#
A diagonal matrix representing the number of connections (degree) for each node in a graph.
For a graph with \(n\) nodes, the degree matrix \(D\) is defined as,
the diagonal elements contain the degree of each node, representing the number of connections to other nodes
all off-diagonal elements are zero
where \(A\) is the adjacency matrix and \(D_{ii}\) is the degree of node \(i\).
The entries of the degree matrix are integers,
\(0\) indicates a node with no connections
larger values indicate nodes with more connections
The degree matrix is used in graph-based machine learning methods, including spectral clustering, where it contributes to the calculation of the graph Laplacian,
where \(L\) is the graph Laplacian, \(D\) is the degree matrix and \(A\) is the affinity matrix.
Used in:
Also see:
Density-Based Cluster#
A nonempty set of points where every pair of points is density-connected with respect to the DBSCAN parameters \(\epsilon\) and \(minPts\).
Important aspects,
clusters are formed by groups of density-connected points
density-based clusters can have arbitrary shapes because cluster membership is determined by density connectivity rather than distance from a central point
points that are not density-connected to any cluster are classified as noise or outliers
Used in:
Also see:
Contrast with:
Density-Based Clustering#
A family of clustering methods that identifies groups as contiguous regions of high sample density in predictor feature space, separated by regions of low sample density.
Rather than assigning every sample to a cluster,
dense regions are identified as clusters
isolated observations are often classified as outliers or noise
Compared with k-means clustering, density-based clustering works well when populations,
overlap in predictor feature space
have irregular or non-spherical shapes
contain noise or outliers that should not belong to any cluster
have an unknown number of groups
General characteristics of density-based clustering,
cluster shape - clusters may take arbitrary shapes rather than being approximately spherical
automatic group count - the number of clusters is determined from the data rather than specified in advance
outlier detection - isolated observations are naturally identified as noise
One limitation of density-based clustering,
performance may decrease when clusters have substantially different densities, making it difficult to identify a single density threshold appropriate for all groups
Examples of density-based clustering methods include,
DBSCAN - clusters are formed from connected dense neighborhoods using minimum density criteria
OPTICS - extension of DBSCAN that accommodates varying cluster densities
Used in:
Also see:
Contrast with:
Density-Connected#
Two points \(A\) and \(B\) are density-connected if there exists a point \(Z\) such that both \(A\) and \(B\) are density-reachable from \(Z\).
Important aspects,
density-connectedness is a symmetric relationship, meaning if \(A\) is density-connected to \(B\), then \(B\) is density-connected to \(A\)
density-connected points belong to the same density-based cluster because they share a common density-reachable path through core points
Used in:
Contrast with:
Also see:
Density-Reachable#
A point \(Y\) is density-reachabl* from a point \(A\) if there exists a sequence of points connecting \(A\) to \(Y\) such that each point in the sequence is within the \(\epsilon\) neighborhood of the previous core point.
A density-reachable path requires a chain of core points where each core point is density-connected to the next point, and the final point \(Y\) may be a core point or a border point.
Important aspects,
density-reachability depends on the direction of the starting point, meaning if \(Y\) is density-reachable from \(A\), \(A\) may not be density-reachable from \(Y\)
applied by DBSCAN to grow clusters from core points through regions of sufficient density
Used in:
Contrast with:
Also see:
Deterministic Model#
A model that assumes a system or process is completely specified such that the same inputs always produce the same outputs. Deterministic models do not explicitly represent uncertainty in the system or process; therefore,
uncertainty is neglected and the system is treated as known or certain.
Deterministic models may be based on,
engineering and geoscience physics
expert interpretation and knowledge
data-driven estimation methods
Examples include,
numerical flow simulation for a specified set of reservoir properties
stratigraphic bounding surfaces interpreted from seismic data
kriging estimates
machine learning prediction models that return a single prediction
Advantages:
integrates physics, expert knowledge, and available data
integrates multiple information sources
often straightforward to interpret and apply
Disadvantages:
provides a single model or prediction without explicitly representing uncertainty
may underestimate decision risk when uncertainty is significant
often time consuming to construct, calibrate, and validate
Used in: TBS
Contrast with:
Dimensionality Reduction#
Methods to reduce the number of predictor features within a data science workflow. There are 2 primary methods,
features Selection – find the subset of original features that are most important for the problem
feature projection – transform the data from a higher to lower dimensional space
Known as dimension reduction or dimensionality reduction,
motivated by the curse of dimensionality and multicollinearity
applied in statistics, machine learning and information theory
Used in:
Also see:
Directly Density-Reachable#
A point \(X\) is directly density-reachable from point \(A\) if \(A\) is a core point and \(X\) belongs to the \(\epsilon\)-neighborhood of \(A\).
Mathematically,
and,
where \(N_{\epsilon}(A)\) is the set of samples within distance \(\epsilon\) of point \(A\), and \(minPts\) is the minimum number of samples required for \(A\) to be a core point.
Important aspects,
direct density reachability is the fundamental connection used to grow DBSCAN clusters
the relationship is directional; if \(X\) is directly density-reachable from \(A\), \(A\) may not be directly density-reachable from \(X\) unless \(X\) is also a core point
the starting point must be a core point, but \(X\) may be either a core point or a border point
chains of directly density-reachable points create density-reachable paths used for cluster growth
Used in:
Also see:
Discrete Feature#
A feature that can only take one of a countable set of distinct values. Discrete features may be naturally discrete (categorical feature) or created by grouping (or binning) a continuous feature. For example,
binned continuous feature – porosity between 0% and 20% assigned to 10 bins:
\(\quad\) represented by the bin centroids,
categorical feature – facies:
ordinal feature – Mohs hardness:
Used in:
Contrast with:
Distribution Transformation#
A mapping from one probability distribution to another through corresponding percentile values, also called a quantile transformation. The transformation preserves the rank ordering of the data while changing the distributional shape, resulting in new,
Distribution transformations are commonly applied in geostatistical methods and workflows because,
inference - to transform a sample distribution toward an expected distribution when data are sparse, biased, or insufficient to characterize the full distribution
theory - to satisfy a distributional assumption required by a workflow step, for example, a Gaussian distribution with mean 0.0 and variance 1.0 is required for sequential Gaussian simulation
data preparation - to reduce the influence of extreme values by mapping them into the target distribution while preserving their rank relationship
improve model performance - to improve sensitivity of the machine learning model, for example, sensitivity of an activation function.
How do we perform distribution transformations?
Values are transformed from the original cumulative distribution function, \(F_X\), to a target CDF, \(G_Y\), using percentile matching. This quantile transformation is applied to all sample values:
Forward transform:
Reverse transform:
This approach may be applied to any distribution, including parametric and nonparametric distributions, as long as percentile values can be mapped between the distributions.
The key property is:
rank preserving transform - the percentile position of a value is maintained, for example, P25 remains P25 after transformation
Contrast with affine correction, which only adjusts distribution location and scale (mean and variance),
distribution transformation modifies the complete distribution, including higher-order statistics and distribution shape
Used in:
Compare with:
Dot-product#
A linear algebra operation that measures the alignment between two vectors.
For two vectors \(a\) and \(b\),
The dot product can also be interpreted geometrically,
where \(\theta\) is the angle between the two vectors.
The dot product answers the question,
how much do two vectors align?
Interpretation,
large positive value \(\rightarrow\) strong alignment, vectors point in similar directions
zero \(\rightarrow\) orthogonal vectors, directions are perpendicular
negative value \(\rightarrow\) opposing directions
For similarity measures, the dot product can be interpreted as,
similar vectors \(\rightarrow\) large positive dot product
unrelated vectors \(\rightarrow\) small dot product
opposing vectors \(\rightarrow\) negative dot product
Applications in machine learning include,
attention mechanism – the dot product between Query and Key vectors is used to measure the relevance or similarity between elements
linear models – the prediction is often calculated as a weighted dot product between feature vectors and model parameters,
similarity search – vector representations are compared using dot products to identify similar patterns or observations
Used in:
Compare with:
Drill Cuttings#
Direct samples of subsurface material generated during drilling operations.
drill cuttings are fragments of rock produced by the drill bit and continuously transported to the surface, where they are collected, described, and logged during drilling.
Drill cuttings provide broader spatial coverage than core data because they are commonly recovered along much of the well or borehole trajectory during routine drilling operations. However, compared with core data, drill cuttings,
represent small, irregular, and mixed rock fragments rather than a continuous sample volume. Individual fragments may range approximately from 0.1 mm to 5 cm, although larger fragments (cavings) may occur due to mechanical failure along the borehole or wellbore.
lose orientation and large-scale structural information during recovery and transport because fragments are mixed and disrupted during pneumatic or hydraulic lifting from the borehole or well.
provide lithological and compositional information but generally cannot preserve continuous sedimentary structures, fracture orientations, or fine-scale spatial relationships.
Drill cuttings represent a trade-off between core and indirect measurements; they provide extensive direct sampling coverage but with reduced spatial resolution and geological context.
Used in: TBD
Also see:
Eager Learning#
A machine learning approach where a generalized model is constructed during a training phase before prediction queries are made.
after model parameter training and model hyperparameter tuning, the model is independent of the original training data and can calculate new predictions without accessing the training dataset
the computational effort is concentrated during the training phase, while prediction is typically fast
Examples include,
Logistic Regression - TBA
Used in:
Linear Regression Analytical Solution Description and Demonstration
LASSO Regression Interative Solution Description and Demonstration
Bayesian Linear Regression MCMC for Sampling Posterior Model Parameters
Also see:
Contrast with:
Eigenvalue#
A scalar that quantifies the amount of scaling associated with an eigenvector during a matrix transformation. Eigen value is the,
amount that the eigenvector is stretched or squished by during that transformation
An eigenvalue, \(\lambda\), satisfies
where \(\mathbf{A}\) is a matrix and \(\mathbf{v}\) is the corresponding eigenvector.
In Principal Component Analysis,
the eigenvalues of the covariance matrix quantify the variance explained by the corresponding principal components.
Also see:
Eigenvector#
A nonzero vector, special direction that remains unchanged when a matrix transformation is applied,
including stretching, rotating, or shearing a grid)
An eigenvector, \(\mathbf{v}\), satisfies
where \(\mathbf{A}\) is a matrix and \(\lambda\) is the corresponding eigenvalue.
In Principal Component Analysis,
the eigenvectors of the covariance matrix define the principal component directions, directions with most variance explained.
Also see:
Ergodic Fluctuations#
Statistical fluctuations observed when calculating statistics from finite simulated realizations of an ergodic random function. The statistics calculated from an individual realization are expected to vary around the input model statistics. For example,
the histogram of an individual realization may not exactly reproduce the input histogram
the variogram of an individual realization may not exactly reproduce the input variogram
the correlation coefficient between primary and secondary paired realizations may not exactly reproduce the input correlation coefficient
Some general observations about ergodic fluctuations,
part of the uncertainty model - fluctuations in statistical reproduction, along with scenarios, are an important part of the uncertainty model because they represent natural variability among possible realizations
magnitude - controlled by the ratio of spatial continuity range to the size of the model domain
minimized - when the model domain is large relative to the spatial continuity range, providing many effective independent spatial samples
maximized - when the model domain is small relative to the spatial continuity range, providing fewer effective independent spatial samples
When checking simulated realizations, some fluctuation in the histogram, variogram, and correlation coefficients should be expected.
best practice is to evaluate the expectation of these statistics over many realizations and compare the ensemble statistics with the input model statistics
Used in:
Estimation#
The paradigm and process of obtaining a single best value to represent a feature or variable at an unsampled location or time.
the “best” estimate is determined by an objective criterion, such as minimizing estimation error.
Some additional estimation concepts,
local accuracy - estimation methods prioritize honoring local data and minimizing local uncertainty, often at the expense of reproducing the full range of global spatial variability
deterministic model - the same inputs always produce the same outputs
smoothness - estimation methods commonly produce values that are smoother than the true variability because local averaging reduces variance
nonlinear response - smooth estimates may not be appropriate when applying transforms or decision criteria that are sensitive to heterogeneity, such as flow response, connectivity, recovery, or economic metrics
examples - inverse distance weighting and kriging
many predictive machine learning models focus on estimation, including k-nearest neighbours, decision trees, and random forests
Used in: TBA - estimation vs. simulation in concepts
Contrast with the simulation paradigm:
Evidence#
In Bayes’ Theorem, the evidence term represents the overall probability of observing the data. It provides the normalization required to ensure probability closure of the updated posterior probability.
where:
\(P(A)\) is the prior probability representing uncertainty before observing new information,
\(P(B|A)\) is the likelihood function describing the compatibility of observations \(B\) with state or parameter \(A\),
\(P(B)\) is the evidence term representing the total probability of observing data \(B\) and normalizing the posterior probability,
\(P(A|B)\) is the posterior probability representing updated uncertainty after incorporating observations.
Used in:
Also see:
Expectation#
The expected value is the probability-weighted average outcome of a random variable. It is a measure of central tendency that represents the average value accounting for the likelihood of all possible outcomes.
for a discrete random variable, the expectation is the sum of all possible outcomes weighted by their probabilities,
for a continuous random variable, the expectation is the integral of all possible values weighted by the probability density function,
Expectation is also the mathematical foundation for the average when all realizations are considered equiprobable.
Expectation is extremely useful for doing mathematics with random variables,
expectation of a constant,
expectation of a random variable plus a constant,
expectation of a constant multiplied by a random variable,
expectation of the addition of two random variables,
Expectation is widely used in data science,
for optimum decision making in the presence of uncertainty, i.e., selecting the choice that maximizes expected profit.
for the decomposition of expected test mean scquare error into model variance, model bias and irreducible error components
Used in:
Also see:
Expected Test Mean Square Error#
The expected test mean square error measures the expected prediction error of a model for observations that were not used during training,
where,
\(y_0\) is the true response for a new observation not included in the training data
\(\hat{f}(x_{0,1}^{(i)},\ldots,x_{0,m}^{(i)})\) is the model prediction for that observation
Because the training data are considered a random sample, the model predictions vary from one training dataset to another. The expectation is therefore taken over all possible training datasets.
Under standard assumptions, the expected test mean square error can be decomposed into three additive components,
These components may be summarized as,
Model Bias - systematic prediction error resulting from restrictive model assumptions
Model Variance - prediction variability caused by changes in the training data
Irreducible Error - uncertainty that cannot be explained by the available predictor features
This decomposition demonstrates the model bias–variance trade-off,
increasing model complexity generally decreases model bias but increases model variance
decreasing model complexity generally decreases model variance but increases model bias
The objective of model hyperparameter tuning is to select a model complexity that minimizes the expected test mean square error.
Used in:
Also see:
F1-score#
A categorical classification prediction model metric that summarizes the balance between precision and recall as a single metric calculated from the confusion matrix.
The F1-score is the harmonic mean of precision and recall,
or equivalently,
where \(k\) represents the category or class.
For binary classification, the confusion matrix is composed of,
True Positive (TP) – positive observations correctly classified as positive
False Positive (FP) – negative observations incorrectly classified as positive
True Negative (TN) – negative observations correctly classified as negative
False Negative (FN) – positive observations incorrectly classified as negative
Common summary metrics from the confusion matrix include,
a summarization over the columns and rows in a confusion matrix (truth on y-axis and predicticted categories on x-axis)
classification precision – the proportion of predicted positive observations that are actually positive,
classification recall (sensitivity) – the proportion of actual positive observations that are correctly classified,
F1-score – the harmonic mean of precision and recall,
The F1-score balances precision and recall and is especially useful when classes are imbalanced. Unlike accuracy, the F1-score does not allow the majority class to dominate the performance metric.
For multiclass classification, the F1-score may be summarized over all \(k\) categories using,
macro F1-score – average F1-score over all categories, giving equal importance to each class
weighted F1-score – average F1-score weighted by the number of samples in each category
micro F1-score – calculates precision and recall globally over all samples
Used in:
Also see:
Facies#
A method of grouping rock into discrete categories, creating a new categorical feature. Facies are used to represent geological variability in a manner that improves,
characterization through statistics, e.g., distributions and variograms
prediction of subsurface features, e.g., porosity and permeability away from wells
For oil and gas, the term facies is commonly used, while mining commonly uses terms such as rock types or zones. In subsurface modeling, multiple types of facies may be considered,
lithofacies - based on rock-related characteristics, including lithology, sedimentary structures, and small-scale geological features that influence porosity and permeability, for example, shale, sandstone, dolomite, limestone, laminated sandstone, hummocky cross-stratification, etc.
depofacies - integrate multiple lithofacies with depositional geometry and reservoir-scale architecture that impacts flow behavior and well connectivity, for example, channel axis, channel margin, outer sheet, etc.
seismic facies - large-scale classifications based on acoustic and elastic properties and seismic geomorphological expressions that define the reservoir framework, for example, parallel continuous high amplitude, chaotic amplitudes, mounded discontinuous low amplitudes, truncation, onlap, offlap, etc.
Here are some important considerations for determining facies,
facies or rock type is an important decision for subsurface modeling. Facies determination should remain a collaborative decision integrating expertise from the entire project team (Geologists, Reservoir Modelers, Reservoir Engineers, Petro- and Geophysicists).
facies or rock types must improve subsurface prediction away from the data or they do not add value.
the number of facies is a balancing act between geological realism, statistical inference, and modeling effort.
reservoir modeling is often hierarchical, for example, geological elements contain multiple depofacies, depofacies contain multiple lithofacies, and lithofacies have specific porosity and permeability distributions.
often 80-90% of reservoir-scale heterogeneity may be captured by the facies model.
Here is a summary of criteria for facies, rock types, or any discrete grouping used in a subsurface model,
Separation of rock properties - facies must be separable based on features that impact subsurface environmental and economic performance, for example, grade, porosity, permeability, etc.
Identifiable in data - facies must be identifiable from the most commonly available data. For example, facies identifiable only from cores are not useful if most wells only have well logs.
Map-able away from data - facies must be easier to predict away from data than the rock properties of interest directly; otherwise, facies do not improve prediction.
Sufficient sampling - there must be enough data to infer reliable statistics within each facies, i.e., by-facies statistics.
Used in:
Also see:
Feature#
A property measured, observed, or calculated (i.e., engineered feature) for analysis in a study.
also known as a variable
Features represent the information used to characterize, model, or predict a system. Examples include,
porosity, permeability, mineral concentrations, saturations, contaminant concentration, etc.
derived properties such as seismic attributes, ratios, transformations, or other calculated quantities.
Different fields use different terminology,
in data mining and machine learning this is commonly called a feature
in statistics this is commonly called a variable
in geoscience this is often called a property or attribute
Feature values may require significant measurement, processing, interpretation, and analysis before they are suitable for modeling.
When features are modified, combined, or transformed to improve model performance, this is called feature engineering.
Used in:
Also see:
Feature Engineering#
The process of creating, modifying, combining, transforming, or selecting features to improve model performance, interpretation, or statistical inference. Feature engineering may incorporate domain knowledge, physical understanding, and data analysis. Examples include,
adjusting total porosity to effective porosity
combining porosity and permeability into a single rock quality index measure
transforming data to account for volume support differences using a volume-variance relations model
weighting data samples for improved spatial representativity using declustering
transforming a data distribution to standard normal in geostatistical sequential Gaussian simulation
Feature engineering is commonly applied before modeling to create inputs that better represent the controlling processes and improve prediction or estimation
Used in:
Feature Importance#
Some machine learning methods provide convenient measures of feature importance, i.e., the impact of each predictor feature on the prediction(s). For example,
linear regression applied to standardized features - the standardized model coefficients,
\(\quad\) where the value of each coefficient indicates how much the target feature changes for a 1-unit increase in the corresponding standardized predictor feature.
decision trees - the cumulative reduction in mean square error attributable to splits on each feature, summarized as,
\(\quad\) where \(T_f\) is the set of all nodes that split on feature \(x\), \(N_t\) is the number of training samples reaching node \(t\), \(N\) is the total number of training samples, and \(\Delta_{MSE_t}\) is the reduction in MSE produced by the split at node \(t\).
\(\quad\) Note, feature importance is calculated similarly for classification decision trees by replacing MSE with Gini impurity (or another classification impurity measure).
Feature importance may be used for model-based feature ranking, but remember,
the reliability of feature importance depends on the predictive accuracy of the model, i.e., an inaccurate model will likely produce misleading feature importance estimates.
Used in:
Also see:
Feature Imputation#
Replacing missing feature values in a data table with plausible values for several reasons,
enable statistical calculations and machine learning methods that require complete data tables, i.e., cannot work with missing feature values
maximize model accuracy by increasing the number of reliable samples available for training and testing
mitigate model bias that may occur with listwise deletion when feature values are not missing at random
Feature imputation methods include,
constant value imputation - replace missing feature values with a constant statistic, such as the feature mean, median, or mode
model-based imputation - replace missing feature values with predictions from a model trained using the available feature values for the same sample
There are also iterative methods that depend on convergence,
Multiple Imputation by Chained Equations (MICE) - initialize missing values, then iteratively update them by predicting each missing feature from the remaining available and previously imputed feature values
The goal of feature imputation is to obtain reasonable values that preserve the relationships among the features while minimizing the bias and uncertainty introduced by missing data.
Used in:
Also see:
Feature Map#
A multidimensional representation of learned features produced by applying a convolution filter or other neural network operation to input data, including 2D images and three-dimensional or higher-dimensional models.
Feature maps preserve the spatial arrangement of the input while transforming the information into increasingly useful representations,
each feature map emphasizes a particular learned pattern or characteristic
spatial locations in the feature map correspond to spatial locations in the input
multiple feature maps allow different features to be learned simultaneously
Multiple feature maps arise through,
channels - multiple feature maps produced within the same layer, each generated by a different convolution filter
layers - successive feature maps at increasing levels of abstraction, where early layers commonly learn edges and textures, intermediate layers learn shapes and objects, and deeper layers learn arrangements and higher-level structures
As information flows through a convolutional neural network,
early feature maps commonly represent simple features such as edges, orientations, and textures
deeper feature maps combine simpler features into increasingly complex shapes, objects, and spatial structures
Feature maps are transformed by convolution, activation, and pooling layers before being used for final prediction or classification.
Used in:
Also see:
Feature Projection#
Methods that transform the original \(m\) features into \(p\) projected features, where \(p \ll m\), for dimensionality reduction and to reduce or remove predictor feature redundancy.
For example,
given \(m\) features, \(X_1,\ldots,X_m\), we require \(\binom{m}{2}=\frac{m(m-1)}{2}\) two-dimensional scatter plot s to visualize all pairwise feature relationships
these visualizations do not capture structures in more than two dimensions
once we have four or more features, understanding the relationships within the data becomes very difficult. Recall the curse of dimensionality.
Example machine learning methods for feature projection include linear methods,
principal component analysis - maximize the variance explained
factor analysis - explain variability with a smaller set of latent features
random projection - project data onto randomly generated directions; most effective for very high-dimensional datasets
non-linear methods, also known as manifold learning,
multidimensional scaling (classical or non-metric MDS) - preserve inter-sample distances
t-distributed stochastic neighbor embedding (t-SNE) - preserve local structure and data clusters
Uniform Manifold Approximation and Projection (UMAP) - preserve local and global structure
kernel principal component analysis (Kernel PCA) - project data into a higher-dimensional feature space where linear separation may be possible
and deep learning,
autoencoder - compress data through a neural network bottleneck
Alternative methods for dimensionality reduction include,
feature selection - retain the most relevant features while minimizing redundancy
feature aggregation - combine redundant or highly correlated features
Used in:
Also see:
Feature Ranking#
A collection of methods that quantify the relative importance of predictor features by measuring their contribution to predicting a response feature,
feature ranking is primarily motivated by the curse of dimensionality, seeking the smallest set of predictor features that retains the maximum predictive information while reducing redundancy, improving model interpretability, and often enhancing predictive performance.
As part of feature engineering, feature ranking assigns an importance score to each predictor feature based on its,
high relevance - the amount of useful information the feature provides for predicting the response feature
low redundancy - the extent to which this information is unique and not duplicated by other predictor features
The general classes of feature ranking methods considered in this book include,
Visual Inspection - examination of data distributions, scatter plot s, violin plots, and other graphical summaries
Statistical Summaries - quantitative measures of feature-response relationships, including correlation analysis, partial correlation coefficient, mutual information and maximum relevance minimum redundancy
Model-based Methods - feature importance scores derived from predictive models, including global Shapley values
Exploration of Feature Subsets - systematic evaluation of multiple feature subsets using trial-and-error search with cross-validation on withheld testing data to identify an optimal predictor feature set, for example recursive feature elimination
Used in:
Also see:
Feature Selection#
A dimensionality reduction method that improves model performance, and reduces model complexity while improving model interpretability by selecting a subset of the original predictor features.
Feature selection,
is motivated by the curse of dimensionality
retains a subset of the original features rather than creating new features
commonly uses feature ranking methods to identify the most informative features
Contrast with:
Also see:
Feature Space#
The multiple variate space represented by the ranges and possible combinations of all features for our problem. Commonly feature space only refers to the predictor features and does not include the response feature(s); therefore, it is,
all possible combinations of predictor features for which we need to make predictions
may be referred to as predictor feature space.
Typically, we train and test our machines’ predictions over the predictor feature space,
the space is typically a hypercuboid with each axis representing a predictor feature and extending from the minimum to maximum, over the range of each predictor feature
more complicated shapes of predictor feature space are possible, e.g., we could mask or remove subsets with poor data coverage.
Used in:
Also see:
Feature Transformation#
A feature engineering step involving mathematical operations applied to predictor feature(s) to create a representation that is more suitable for a machine learning workflow. For example,
feature truncation
feature normalization or standardization
feature distribution transformation
There are many reasons that we may perform feature transformations,
to make features consistent in scale and representation for visualization, comparison, and interpretation
to avoid bias or impose feature weighting for methods that rely on distances calculated in predictor feature space, for example k-nearest neighbours regression
to satisfy assumptions or requirements of specific methods, for example, artificial neural networks may perform better when features are normalized to a common range such as \([-1,1]\), and statistical methods based on correlation may benefit from approximately Gaussian distributed features
Feature transformation changes the representation of the predictor features while preserving the underlying information content, allowing machine learning methods to more effectively identify patterns and relationships.
Used in:
Also see:
Fourth Paradigm#
The data-driven paradigm for scientific discovery that builds upon the previous scientific paradigms,
First Paradigm - empirical science - experiments, observations, and the collection of measured data
Second Paradigm - theoretical science - analytical expressions, mathematical models, and fundamental principles
Third Paradigm - computational science - numerical simulation and computational experimentation
The Fourth Paradigm augments, rather than replaces, the previous paradigms. Each scientific paradigm builds upon and is supported by the earlier paradigms, for example,
theoretical science builds upon empirical observations to develop and validate mathematical relationships
computational science integrates analytical expressions, physical models, and calibrated equations derived from experimental observations
data-driven science combines large-scale datasets with computational methods to discover patterns, relationships, and predictive models that may be difficult to identify through traditional approaches
The Fourth Paradigm represents a shift from primarily hypothesis-driven discovery toward data-intensive discovery, while continuing to rely on empirical evidence, theory, and computation.
Used in:
Frequentist Probability#
A measure of the probability that an event occurs based on the long-run relative frequency observed from repeated experiments or repeated sampling. For random experiments and well-defined settings (such as coin tosses),
where:
\(n(A)\) = number of times event \(A\) occurred
\(n\) = number of trials
The frequentist interpretation assumes that probability represents an objective property of a repeatable process. Examples include,
probability of drilling a dry hole for the next well
probability of encountering sandstone at a location (\(\bf{u}_{\alpha}\))
probability of exceeding a rock porosity of \(15\%\) at a location (\(\bf{u}_{\alpha}\))
In geoscience, many processes cannot be repeated exactly; therefore, frequentist probabilities are often estimated from available samples under assumptions of representativity and stationarity.
Used in:
Contrast with:
Gaussian Anamorphosis#
A quantile transformation that maps any univariate distribution to a Gaussian distribution,
also known as a normal score transform
The transformation maps feature values through their cumulative probabilities,
where \(F_x\) is the cumulative distribution function of the original feature distribution and \(G_y\) is the Gaussian CDF.
The Gaussian probability density function is,
A shorthand notation for a normal distribution is,
for example, \(N[0,1]\) is the standard normal distribution.
Properties of Gaussian distributions,
much of natural variation and measurement error can be approximately represented by Gaussian distributions
parameterized completely by the mean, variance, and correlation coefficients for multivariate Gaussian distributions
unbounded distribution with no minimum or maximum values; extreme values are increasingly unlikely, and practical applications often apply truncation limits
Warning, many workflows apply univariate Gaussian anamorphosis and then assume a bivariate or multivariate Gaussian distribution. This assumption is generally not correct, but transforming data to a true multivariate Gaussian distribution is often computationally difficult.
Methods that benefit from or require Gaussian distributed features,
Pearson product-moment correlation coefficients completely characterize multivariate relationships when data follow a multivariate Gaussian distribution
partial correlation coefficients have their strongest statistical interpretation under multivariate normality
sequential Gaussian simulation (SGS) assumes Gaussian distributions to reproduce spatial variability and the global distribution
Student’s t-test for differences in means assumes normally distributed populations
chi-square distributions are derived from sums of squares of Gaussian distributed random variables
Gaussian naive Bayes classification assumes Gaussian conditional distributions
Used in:
Also see:
Constrast with:
Generative Adversarial Network#
Machine learning architecture where two neural networks are trained competitively against each other in an adversarial learning process,
generator - neural network that generates synthetic data attempting to produce samples that the discriminator classifies as real
discriminator - neural network that classifies samples as real (from the training dataset) or generated (from the generator)
The adversarial training concept is,
the generator learns to create increasingly realistic synthetic samples
the discriminator learns to distinguish generated samples from real training samples
the generator improves based on feedback from the discriminator
The adversarial objective can be expressed as a minimax optimization problem,
where \(G\) is the generator, \(D\) is the discriminator, \(x\) represents real training data, and \(z\) is random input noise.
Additional points about generative adversarial networks,
the generator does not directly observe the real training samples; instead, it learns from the discriminator’s feedback about generated samples
the discriminator provides a learned objective function that guides the generator toward realistic data generation
directly minimizing distance to training samples may cause the model to reproduce existing samples rather than generate diverse new realizations
adversarial training encourages the generator to reproduce the statistical characteristics and variability of the training data
Training generative adversarial networks,
the generator starts with random weights and produces random synthetic samples
the discriminator starts with random weights and initially performs poorly at distinguishing real and generated samples
generator and discriminator training must remain balanced; a discriminator that becomes too accurate can provide weak learning signals to the generator
gradients from the discriminator are propagated backward through the network to update generator weights
Applications of generative adversarial networks include,
image enhancement and super resolution - increasing image resolution and improving visual quality
synthetic image generation - generating realistic images and scientific models
geostatistical modeling - generating heterogeneous subsurface realizations that reproduce spatial patterns and geological variability
Variants of generative adversarial networks include,
conditional generative adversarial network (cGAN) - incorporates labels or conditioning information to control generated outputs
cycleGAN - learns transformations between two image domains without paired examples
pix2pix - performs image-to-image translation using paired training examples
Training generative adversarial networks,
network weights and biases are estimated using backpropagation and gradient-based optimization
gradients pass through the discriminator and provide learning signals to update the generator
training can be challenging due to unstable optimization and balancing the competing objectives of the two networks
Used in:
Also see:
Geostatistics#
A branch of applied statistics that integrates,
spatial (geological) context
spatial relationships and continuity
volume support and scale
uncertainty
Geostatistics provides methods to characterize, model, predict, and simulate spatial phenomena by incorporating the spatial structure of the data and the uncertainty in the subsurface or other spatial systems to support optimum decision making.
The boundary between geostatistics and spatial statistics is debated. In this course,
geostatistics includes many spatial statistics methods because, in practice, useful approaches for modeling spatial phenomena are adopted and integrated into the geostatistical toolkit.
Geostatistics is an expanding and evolving field of study that continues to incorporate new statistical, computational, and data-driven approaches.
Gibbs Sampler#
A Markov chain Monte Carlo simulation algorithm that generates samples from a target probability distribution such that, after convergence, the ensemble of samples reproduces the statistics of the target distribution.
The Gibbs sampler is based on,
sequentially sampling from conditional probability distributions
Since only conditional probability density functions are required, the sampling process is simplified because the full joint probability density function is not directly required.
The basic steps of the Gibbs MCMC sampler for a bivariate case are,
Assign initial random values for \(X(0)\), \(Y(0)\)
Sample from \(f(X|Y(0))\) to obtain \(X(1)\)
Sample from \(f(Y|X(1))\) to obtain \(Y(1)\)
Repeat the sequential conditional sampling steps to generate samples,
After sufficient iterations and convergence, the resulting samples reproduce the target joint distribution,
The Gibbs sampler is particularly useful when the joint distribution is complex, but the conditional distributions are easier to sample from.
Used in: TBA - with Bayesian Linear Regression
Compare with:
Also see:
Global Accuracy#
Honoring (matching) global measures calculated over the entire volume of interest, for example,
Global accuracy is a primary objective for simulation, where the realizations should reproduce the input global statistics in expectation.
Used in:
Contrast with:
Global Measure#
A statistical or spatial summary calculated over the entire volume of interest. Examples include,
Global measures characterize the overall behavior of a feature or model and are commonly used for model checking and validation.
Used in:
Contrast with:
Gradient Boosting Model#
Machine learning prediction model that results from posing a boosting model as a gradient descent optimization problem.
For boosting,
at each step, \(k\), a new model is fit to improve the current estimator by reducing the prediction error
the model \(h_k(X_1,\ldots,X_m)\) is fit to the residual errors or, more generally, the negative gradient of the loss function
We assign a loss function, \(L\),
and minimize the \(\ell_2\) loss function,
by adjusting our model estimator \(F(X)\) over the training data.
We can take the partial derivative of the loss with respect to our model estimate,
The residuals can be interpreted as negative gradients,
Therefore, fitting a new model to the residuals is equivalent to fitting a model to the negative gradient of the loss function.
The gradient descent update is,
where the new model \(h_k(X_i)\) approximates the negative gradient,
or equivalently,
The general gradient descent form is,
where \(\phi_k\) is the current state, \(\rho\) is the learning rate, \(J\) is the loss function, and \(\phi_{k+1}\) is the updated estimator state.
The prediction residual at the training data provides the gradient direction that guides each new model. Therefore, gradient boosting performs,
fitting a sequence of models to the negative gradients of the loss function
By approaching boosting as a gradient descent problem, we can apply different loss functions depending on the prediction problem and desired robustness.
\(\ell_2\) loss,
is commonly used because it provides a smooth optimization objective, but it is sensitive to outliers.
The negative gradient is,
\(\ell_1\) loss,
is more robust to outliers.
The negative subgradient is,
other loss functions include Huber loss, which combines the robustness of \(\ell_1\) loss with the smooth optimization properties of \(\ell_2\) loss
Gradient boosting can be conceptionalized as,
a sequence of addivite weak learners, where each learner is trained to correct the remaining errors of the previous ensemble.
Used in:
Also see:
Gradient-based Optimization#
A method to train model parameters by iteratively minimizing a loss function. The general steps include,
initialize model parameters with random or informed starting values
calculate the loss function for the current model parameters
calculate the loss function gradient, which indicates the direction of steepest increase in loss. For many models, the analytical gradient is not available, and numerical approximation or automatic differentiation may be used.
For numerical calculation of a local loss function derivative,
update the parameter estimate by stepping in the direction that decreases the loss function,
where \(r\) is the learning rate or step size, \(\hat{b}_{1,t}\) is the current model parameter estimate, and \(\hat{b}_{1,t+1}\) is the updated parameter estimate.
Important concepts in gradient-based optimization include,
gradient search convergence - the optimization process attempts to find a minimum of the loss function; depending on the loss landscape, this may be a local minimum, global minimum, or stationary point
gradient search step size - the learning rate \(r\) controls the size of each optimization step; if \(r\) is too small, convergence may be slow, while if \(r\) is too large, the optimization may overshoot the minimum or diverge
multiple model parameters - gradients are calculated for all model parameters and represented as a gradient vector,
exploration of parameter space - optimization for machine learning model training is an exploration of a high-dimensional parameter space to identify parameter values that minimize the loss function
loss function definition - the loss function is typically an error metric that summarizes prediction error over the training data
Used in:
Gradient Descent Optimization for Training LASSO Regression Description and Demonstration
Gradient Descent Optimization for Training Neural Networks Regression Description and Demonstration
Also see:
Graph#
A mathematical structure that represents data and relationships between data samples, where each sample is represented as a node and connections between samples are represented as edges.
For a graph,
nodes - represent samples, entities, or observations
edges - represent pairwise relationships or connections between nodes
edge weights - can represent the strength, distance, similarity, or other characteristics of the relationship
For an undirected graph, edges are bidirectional, meaning the relationship is symmetric between connected nodes.
Graph applications include,
providing a convenient format to represent and summarize complex data structures and relationships
informing clustering analysis methods, such as spectral clustering
enabling prediction models that incorporate relational information, such as graph neural network
Used in:
Also see:
Graph Laplacian#
Matrix representing a graph by integrating connections between graph nodes, samples, number of connections for each graph nodes, samples. Calculated as,
degree matrix, \(D\), minus adjacency matrix, \(A\), where,
degree matrix, \(𝐷\) - degree of connection for each node
adjacency matrix, \(𝐴\) - specific connections between nodes
Graph aplacian matrices are commonly used in,
Used in:
Also see:
Graph Neural Network#
A neural network architecture designed to learn from graph-structured data by exchanging information between connected nodes.
Graph neural networks learn representations by,
aggregating information from neighboring nodes through graph connections
updating node representations using learnable weights, activation functions, and backpropagation
Applications include,
social and communication networks
molecular and biological networks
transportation and infrastructure networks
geological fault, fracture, and well networks
Contrast with:
Also see:
Gridded Data#
Data represented at regularly spaced locations over a 2D area of interest or 3D volume of interest, commonly used to represent maps and spatial models.
Gridded data are characterized by,
regularly spaced locations defined by grid dimensions and cell spacing
exhaustive coverage over the modeled domain, where every grid location contains a value or assigned missing value
implicit spatial relationships between neighboring grid locations
Gridded data may be stored as,
a .csv comma-delimited file, where data may be represented as a matrix with \(n_y\) rows and \(n_x\) columns
binary formats for more compact storage and faster processing, although these files are not human readable
Gridded data are commonly visualized directly, for example,
image representations using functions such as matplotlib’s imshow
contour maps and other spatial visualization methods
Examples of gridded data include,
geological interpretation-based maps
acoustic impedance models inverted from seismic reflection surveys
For spatial modeling problems,
model outputs or response features are often estimated over a regular grid to create continuous maps or 3D models
Also see:
Hard Data#
Data that is treated as certain due to having a high degree of certainty relative to other available information sources. Hard data usually comes from direct measurement or observation of the feature of interest, for example,
Core-based porosity, permeability, mineralogy, and facies observations
direct measurements of grade from drill core samples
laboratory measurements of rock or fluid properties
Hard data is considered sufficiently reliable that uncertainty in the measurement is commonly not explicitly modeled or integrated into subsequent workflows. Note, hard data is not necessarily error-free;
it represents information that is treated as certain for the purpose of the analysis.
Hard data generally has high resolution (small scale, volume support), but poor spatial coverage because only an extremely small proportion of the population is directly sampled. For example,
Core coverage deepwater oil and gas - well core may sample only one five hundred millionth to one five billionth of a deepwater reservoir, assuming 3 inch diameter cores with 10% core coverage in vertical wells with 500 m to 1,500 m spacing
Core coverage mining grade control - diamond drill hole cores may sample one eight thousandth to one thirty thousandth of an ore body, assuming HQ 63.5 mm diameter cores with 100% core coverage in vertical drill holes with 5 m to 10 m spacing
Hard data provides the most direct calibration of subsurface models but must be integrated with other information sources to overcome limited spatial coverage.
Used in: TBD
Contrast with:
Harmonic Mean#
A measure of central tendency that calculates the reciprocal of the arithmetic mean of reciprocals, commonly used when,
averaging rates or ratios
scale up, change of volume support for permeability, given flow across beds
For \(n\) values, the harmonic mean is,
The harmonic mean gives greater weight to smaller values and is useful when low values should strongly influence the average,
Also see:
Hermite Polynomial#
A family of orthogonal polynomials on the real number line, commonly used for representing functions with Gaussian-distributed variables.
Order |
Hermite Polynomial \(H_n(x)\) |
|---|---|
0th Order |
\(H_0(x)=1\) |
1st Order |
\(H_1(x)=x\) |
2nd Order |
\(H_2(x)=x^2-1\) |
3rd Order |
\(H_3(x)=x^3-3x\) |
4th Order |
\(H_4(x)=x^4-6x^2+3\) |
These polynomials are orthogonal with respect to the weighting function,
which is proportional to the standard Gaussian probability density function without the scaling factor,
The definition of orthogonality is,
for \(m \neq n\).
Therefore, Hermite polynomials are orthogonal over the interval \([-\infty,\infty]\) with respect to the standard normal probability distribution.
By applying Hermite polynomials instead of regular polynomial terms for polynomial basis expansion in polynomial regression, we reduce multicollinearity between the predictor features,
recall, polynomial basis expansion creates correlated predictor features because terms such as \(x\), \(x^2\), \(x^3\) are not independent
orthogonal polynomial bases create transformed features that are uncorrelated under the specified weighting function
Orthogonal polynomial bases can improve numerical stability and interpretation of polynomial regression models, especially when predictors follow approximately Gaussian distributions.
Used in:
Also see:
Heuristic Algorithm#
A practical algorithm that uses approximations, rules, or strategies to find a good solution to a difficult problem without guaranteeing the optimal solution.
A heuristic algorithm represents a compromise between optimality, accuracy, computational cost, and practicality,
trading an exact solution for a faster solution that is often sufficiently accurate for the application
Heuristic algorithms are commonly applied in,
machine learning
computer science
mathematical optimization
For example,
the optimal solution for \(k\)-means clustering involves searching a large solution space of possible cluster assignments. Since the number of possible assignments grows exponentially with the number of samples, a heuristic iterative algorithm is applied to efficiently find a practical clustering solution.
Used in:
Also see:
Hierarchical Clustering#
A family of clustering methods that construct a hierarchy of nested clusters, typically represented by a dendrogram,
representing the cluster solutions for \(k = [1, 2, \ldots, n-1, n]\) simultaneously
dendrogram provides a visual representation of the cluster hierarchy and allows the user to select the number of clusters by choosing a linkage distance (cut height)
The linkage distance depends on the linkage criterion used:
single linkage - minimum distance between clusters
complete linkage - maximum distance between clusters
average linkage - average pairwise distance
Ward’s linkage - increase in within-cluster variance (often the most popular)
Hierarchical clustering determines cluster groups through a sequence of merges or splits, rather than directly optimizing a fixed partition of the data.
Common approaches include,
agglomerative hierarchical clustering - start with \(n\) clusters, where each data sample is assigned to its own cluster, and iteratively merge the most similar clusters until a stopping criterion is reached
divisive hierarchical clustering - start with all data assigned to a single cluster, and iteratively divide clusters into smaller clusters until a stopping criterion is reached
Compared with partitional clustering methods,
hierarchical clustering produces a hierarchy of cluster solutions rather than a single clustering solution
\(k\)-means clustering is a partitional clustering method that iteratively updates cluster assignments while directly optimizing a single partition of the data
once clusters have been merged or split, hierarchical clustering generally does not revisit earlier decisions, making the method computationally efficient but potentially sensitive to early decisions
A helpful summary of clustering methods,
partitional clustering \(\rightarrow\) Find one best partition of the data.
hierarchical clustering \(\rightarrow\) Build a tree of nested clusters.
density-based clustering \(\rightarrow\) Find regions of high sample density.
Also see:
Histogram#
A bar chart representation of a univariate statistical distribution showing the frequency of samples over an exhaustive set of bins spanning the range of possible values.
These are the steps to build a histogram,
Divide the continuous feature range of possible values into \(K\) equal size bins, \(\Delta x\):
or use available category labels for categorical features.
Count the number of samples (frequency) in each bin, \(n_k\), \(\forall k=1,\ldots,K\).
Plot frequency versus the bin label (use bin centroid for continuous features).
The histogram y-axis represents frequency. When normalized by the total number of samples,
the result is a normalized histogram, where the y-axis represents probability.
for discrete features, the normalized histogram is an empirical probability mass function
for continuous features, the normalized histogram represents probability mass over intervals; dividing by bin width provides an estimate of the probability density function
Additional comments about histograms,
typically plotted as bar charts
2D histograms use a orthonormal view columns with 2 axes for features and along with the frequency axis, but not often used to attempt to visualize bivariate relationships
Used in:
Also see:
Holdout Cross Validation#
A cross validation method that partitions the available data into one training subset and one withheld testing subset.
Holdout validation is simple and computationally efficient but may produce variable performance estimates because the evaluation depends on a single train-test split.
Used in:
Also see:
Contrast with:
Hybrid Model#
A model that combines both deterministic model and stochastic model components.
Hybrid models separate predictable structure from uncertain variability by combining,
deterministic components - representing known relationships, physical processes, expert interpretation, or data-driven trends
stochastic components - representing spatial variability, uncertainty, and unresolved processes that cannot be deterministically modeled
Most geostatistical models are hybrid models. For example, an additive deterministic trend model and stochastic residual model:
where \(m(\mathbf{u})\) represents the deterministic trend and \(R(\mathbf{u})\) represents the stochastic residual.
Other examples include,
deterministic geological frameworks combined with stochastic property simulation
physics-based models calibrated or conditioned with stochastic uncertainty models
data-driven predictive models combined with probabilistic uncertainty models
Used in: TBA - deterministic trends and data nonstationarity discussion
Independence#
Two random events, \(A\) and \(B\), are independent if knowledge of one event provides no information about the likelihood of the other event. Mathematically, events are independent if and only if the following equivalent relationships are true,
Conditional probability:
If any of these relationships are violated, then the events are dependent, indicating that some form of relationship exists between them.
Note that dependence does not necessarily imply causation.
In spatial modeling, independence indicates that knowing one feature or event provides no additional information about another feature or event. For example,
spatially independent samples have no correlation over the specified lag distance.
Used in:
Indicator Transform#
Indicator coding converts a random variable into a probability relative to a category or a threshold.
For a categorical feature, the indicator answers the question,
what is the probability that the data value or realization belongs to a specific category?
The indicator transform is,
For example,
given category \(z_2 = 2\), and data at \(\mathbf{u}_1\) with \(z(\mathbf{u}_1)=2\), then \(i(\mathbf{u}_1;z_2)=1\)
given category \(z_1 = 1\), and a random variable away from data at \(\mathbf{u}_2\), the probability that the realization belongs to category \(z_1\) is \(P(Z(\mathbf{u}_2)=z_1)=0.23\), therefore \(i(\mathbf{u}_2;z_1)=0.23\)
For a continuous feature, the indicator answers the question,
what is the probability that the realization is less than or equal to a threshold?
The indicator transform is,
For example,
given threshold \(z_1 = 6\%\), and data at \(\mathbf{u}_1\) with \(z(\mathbf{u}_1)=8\%\), then \(i(\mathbf{u}_1;z_1)=0\)
given threshold \(z_4 = 18\%\), and a random variable away from data, \(Z(\mathbf{u}_2)\sim N(\mu=16\%,\sigma=3\%)\), then
The indicator transform may be applied to an entire random function by transforming the random variable at every location. Indicator transforms provide the foundation for indicator kriging, indicator variograms, and indicator simulation.
Note, the indicator transform,
may be applied to encode data softness, in this case the data values are not strickly 0 or 1 with respect to each threshold or category at the a data location.
may also be applied to encode a constriant relationship, e.g., data value cannot be category 1, but there is no information about category 2 nor 3, or data value is between 0.1 and 0.4, but there is no information within this interval.
is analogous to the one-hot encoding approach for feature engineering commonly used to deal with categorical features in machine learning.
Used in:
Also see:
Inertia#
A label for the k-Means clustering loss function that summarizes the total within-cluster variation by measuring the squared distance between each sample and the prototype (centroid) of its assigned cluster,
where,
\(K\) is the total number of clusters,
\(C_i\) represents the set of samples assigned to the \(i^{th}\) cluster,
\(x_j\) represents a data sample belonging to cluster \(C_i\),
\(\mu_i\) is the prototype (centroid) of cluster \(C_i\),
\(\|x_j - \mu_i\|^2\) is the squared Euclidean distance between sample \(x_j\) and the cluster prototype \(\mu_i\).
The samples, prototypes, and distance calculations are performed in \(m\)-dimensional feature space, where each observation is represented by features \(X_1,\ldots,X_m\).
by minimizing inertia, k-means finds clusters with the smallest possible within-cluster sum of squared distances, producing compact groups of similar samples.
since the total variance of the dataset is fixed, reducing within-cluster variation generally results in greater separation between clusters.
Used in:
Also see:
Inference#
The process of using a sample drawn from a population to infer properties of the entire population. For example,
given sparsely sampled well data (the sample) with porosity well log measurements, infer the porosity histogram of the entire reservoir (the population)
given sparsely sampled drill hole data (the sample) with gold grade measurements, infer the gold grade distribution and spatial variability throughout the ore body (the population)
in geostatistics, inference includes estimating statistical properties (e.g., histograms, variograms, and trends) as well as spatial models away from the sampled locations
the statistical discipline devoted to inference is known as statistical inference
Inference is a broad topic encompassing many statistical methods. For this e-book, we adopt this simplified, practical definition focused on subsurface characterization and modeling.
Used in:
Compare with:
Inlier#
A regression model accuracy metric based on the proportion of testing data with prediction errors within a specified margin, \(\epsilon\).
The inlier ratio is,
where the indicator function is,
This provides a simple and intuitive measure of prediction accuracy by reporting the proportion of testing (or training) data with predictions that are sufficiently accurate for the application.
Important considerations include,
choice or margin - the choice of the acceptable prediction margin, \(\epsilon\), is subjective and should reflect the accuracy requirements of the specific application
impact of outliers - unlike metrics such as mean square error or mean absolute error, the inlier ratio does not distinguish between small and large errors once the prediction falls outside the specified margin
Used in:
Contrast with:
Instance-based Learning#
Family of predictive machine learning models that,
makes predictions by comparing a new observation, represented by the predictor features \(x_1,\ldots,x_m\), with similar observations stored in the training data.
also known as memory-based learning
Instance-based learning is characterized by,
lazy learning training strategy - instance-based methods generally defer learning until prediction time by storing the training data rather than fitting an explicit predictive model, the model is the training data with the hyperparameters
analogy-based prediction - predictions are made directly from the most similar training observations rather than from an explicit mathematical model
similarity measure - predictions depend on a measure of similarity or distance between observations, such as Euclidean distance
computational complexity - prediction cost generally increases with the number of training observations, \(n\), the number of predictor features, \(m\), and, for nearest-neighbor methods, the number of neighbors, \(k\)
Examples of instance-based learning predictive machine learning models include,
\(k\)-nearest neighbors (KNN) regression
\(k\)-nearest neighbors (KNN) classification
Used in:
Contrast with:
Intersection of Events#
The event in which two or more events occur together. For two events, \(A\) and \(B\), the intersection is denoted by,
The probability of the intersection is,
where we may state, “probability of A and B”. If \(A\) and \(B\) are independent, then the joint probability simplifies to,
Without the assumption of independence, we apply the multiplication rule,
or equivalently,
Used in:
See also:
Irreducible Error#
The additive component of the expected testing mean square error that cannot be eliminated by improving the prediction model.
Irreducible error represents uncertainty that cannot be explained by the available predictor features. Sources include,
missing or unmeasured predictor features
incomplete sampling of the predictor feature space
measurement error in the predictor feature(s) or response features
inherent randomness or stochastic variation in the underlying process
Irreducible error is the only additive component of the expected testing mean square error that is independent of model complexity.
Irreducible error may be reduced by,
collecting additional or more informative predictor features
increasing the quantity and representativeness of the training data
improving measurement quality to reduce observational error
The component of irreducible error associated with inherent randomness cannot be eliminated by any prediction model.
Used in:
Also see:
Machine Learning Workflow Design#
Machine learning (and more generally, data science) workflows follow a logical sequence of steps that transform raw data into information that supports classification, prediction, interpretation, and decision making.
Define the Objective, for example,
build a predictive model
classify observations
identify patterns in the data
compare alternative recovery processes
Understand the Data, answering questions such as,
what data are available?
what information is missing?
are the data representative and of sufficient quality?
could additional data improve the analysis?
Design the Workflow, including steps such as,
load and organize the data
clean, validate, and preprocess the data
engineer or transform features
train statistical or machine learning models
evaluate and compare model performance
visualize, interpret, and communicate results
support decision making
Validate and Improve the Workflow
test the workflow using typical and edge cases
identify when the workflow performs well and when it fails
diagnose sources of error and uncertainty
refine the data, features, models, or workflow design
Document and Communicate the Workflow
record implementation details, assumptions, metadata, limitations, and future work
document the flow of data and information throughout the workflow
summarize sources of uncertainty and describe how they were quantified and incorporated
communicate conclusions and recommendations
Used in: TBA - workflow design chapter
Margin#
The margin in a support vector machine is the distance between the binary decision boundary and the closest observations from each class. SVM training is primarily influenced by observations near the margin, called support vectors, while observations well outside the margin have little or no influence on the final decision boundary.
The margin is a symmetric distance around the binary decision boundary that defines a region of uncertainty between the two classes.
For classification problems with overlapping groups in predictor feature space, perfect separation by a decision boundary is often not practical.
No margin classification would require only that observations fall on the correct side of the decision boundary,
A hard-margin SVM requires,
where observations must be correctly classified and separated from the decision boundary by a minimum margin.
For real-world data with overlapping classes and noise, a soft-margin SVM allows some observations to violate the margin using slack variables, \(\xi_i\),
where,
\(\xi_i\) quantifies the margin violation, or the distance an observation extends into the margin region or across the decision boundary.
\(0 < \xi_i < 1\) indicates that an observation is correctly classified but lies inside the margin
\(\xi_i > 1\) indicates that an observation is misclassified
The SVM optimization balances maximizing the margin and minimizing classification errors,
where,
\(\frac{1}{2}\|\beta\|^2\) controls the margin width, since the margin is inversely proportional to \(\|\beta\|\)
\(C\) is a hyperparameter controlling the penalty for observations violating the margin
A larger \(C\) emphasizes correct classification and produces a narrower margin, while a smaller \(C\) allows more violations and produces a wider margin.
The objective is therefore to,
maximize the separation margin between classes
minimize classification error through the slack variables weighted by \(C\)
Used in:
Also see:
Marginal Probability#
Probability that considers only a single event occurring. For example, the probability of event \(A\),
Marginal probabilities may be calculated from joint probabilities through the process of marginalization,
where we integrate over all cases of the other event, \(B\), to remove its influence. Given discrete, categorical or binned continuous cases of event \(B\) we can simply sum the probabilities over all possible cases of \(B\),
Used in:
Contrast with:
Markov Chain Monte Carlo#
Known widely by the acronym MCMC, a family of algorithms used to estimate complicated probability distributions by generating samples from those distributions.
MCMC methods are based on,
Markov chain - a sequence of samples where each sample depends only on the previous sample
Markov property - the assumption that the current sample contains all required information from previous samples, such that future samples are conditionally independent of earlier samples given the current sample, this is also known as Markov screening
Monte Carlo - a simulation process that uses random sampling to approximate probability distributions and expectations
The general workflow is,
Initialize - start at an initial point in the distribution.
Propose - generate a candidate new sample based on a proposal distribution.
Evaluate - calculate the probability of accepting the proposed sample relative to the current sample.
Accept or Reject - accept the new sample according to an acceptance probability. Higher probability samples are favored, while lower probability samples may still be accepted to explore the full distribution.
Repeat - generate many samples to form a Markov chain that approximates the target probability distribution.
There are a variety of MCMC methods, including,
Metropolis-Hastings - proposes random steps and uses a stochastic acceptance rule to sample from the target distribution
Gibbs Sampler - samples each feature or variable sequentially from its conditional probability distribution
Used in:
Compare with:
Also see:
Matrix Scatter Plots#
A composite plot containing all pair-wise scatter plots between features in a dataset.
given \(m\) features, there are \(m \times m\) scatter plots
the scatter plots are ordered with the y-axis feature from \(X_1,\ldots,X_m\) arranged over the rows and the x-axis feature from \(X_1,\ldots,X_m\) arranged over the columns
the diagonal contains each feature plotted against itself and is often replaced with univariate histograms or probability density functions
Matrix scatter plots are used to,
identify bivariate linear or nonlinear relationships between features
identify bivariate homoscedasticity (constant conditional variance) and heteroscedasticity (changing conditional variance)
identify bivariate constraints, such as sum constraints in compositional data
The remaining features are marginalized during each pair-wise comparison,
therefore, a matrix scatter plot is not a visualization of the full \(m\)-dimensional feature space
Used in:
exploratory data analysis (EDA)
feature engineering and selection
identifying feature relationships and redundancy
diagnosing assumptions for statistical and machine learning models
Used in:
Also see:
Maximum Relevance Minimum Redundancy#
A mutual information-based feature selection approach that identifies a subset of predictor features by maximizing relevance to the response feature while minimizing redundancy between selected features.
one common formulation is a relevance minus redundancy criterion,
where,
\(S\) is the selected predictor feature subset
\(|S|\) is the number of features in subset \(S\)
\(I(X_i,Y)\) is the mutual information between predictor feature \(X_i\) and reponse feature \(Y\), representing feature relevance
\(I(X_i,X_j)\) is the mutual information between predictor features \(X_i\) and \(X_j\), representing feature redundancy
The objective is to select features that,
contain strong information about the response variable
provide complementary information rather than repeating information already contained in selected features
Used in:
Also see:
Mean#
The mean (or average) is a measure of central tendency. There are several useful interpretations of the mean,
representative value - a single value to represent an entire distribution
estimate - given a distribution of values, the mean minimizes the L2 norm (sum of squared error)
scale-up - under linear averaging of a spatial feature, the mean is the correct upscaled value
expectation - if all outcomes are equiprobable, the mean is equal to the expectation
For a sample, the mean is,
Note, the mean is quite sensitive to outliers.
Used in:
Also see:
Mean Absolute Error#
Prediction model performance metric calculated as the average absolute prediction error over all observations,
commonly known by the acronym MAE.
The equation for mean absolute error is,
where \(y_i\) is the observed response feature value, \(\hat{y}_i\) is the model prediction, and \(n\) is the number of observations.
Mean absolute error has several important properties,
all prediction errors contribute equally to the metric
prediction errors increase linearly with error magnitude
always non-negative, with \(MAE=0\) indicating perfect predictions
reported in the original units of the response feature
Mean absolute error is commonly applied,
as a performance metric for regression models
when robustness to outliers is important
to compare competing regression models on the same testing dataset
Compared with other error metrics,
mean square error - penalizes large prediction errors more strongly by squaring the errors
root mean square error - also emphasizes larger prediction errors but is reported in the original response feature units
Advantages of mean absolute error include,
less sensitive to outliers than MSE and RMSE
easier to interpret because it represents the average magnitude of prediction errors
Limitations of mean absolute error include,
treats all prediction errors equally, regardless of their magnitude
absolute values are not differentiable at zero, making MAE less convenient for optimization than MSE
provides no indication whether prediction errors are biased or randomly distributed
Used in:
Also see:
Mean Square Error#
Prediction model performance metric calculated as the average squared prediction error over all observations,
commonly known by the acronym MSE.
Te equation for mean square error is,
where \(y_i\) is the observed response feature value, \(\hat{y}_i\) is the model prediction, and \(n\) is the number of observations.
Mean square error has several important properties,
all prediction errors contribute to the metric
squaring the errors penalizes large prediction errors much more strongly than small errors
always non-negative, with \(MSE=0\) indicating perfect predictions
reported in squared units of the response feature
Mean square error is commonly applied,
as a loss function for training many regression models
as an objective metric for model hyperparameter tuning
to compare competing regression models on the same testing dataset
Compared with other error metrics,
mean absolute error - more robust to outliers because prediction errors are not squared
root mean square error - square root of MSE, reported in the original units of the response feature
Limitations of mean square error include,
sensitive to outliers due to squaring of prediction errors
should not be compared between response features with different units or scales without normalization
provides no indication whether prediction errors are biased or randomly distributed
Used in:
Mean Square Error Loss Function for Linear Regression Description and Demonstration
Mean Square Error Loss Function for Ridge Regression Description and Demonstration
Mean Square Error Loss Function for LASSO Regression Description and Demonstration
Mean Square Error Loss Function for Decision Tree Description and Demonstration
Mean Square Error Loss Function for Random Forest Description and Demonstration
Also see:
Metropolis-Hastings#
A Markov chain Monte Carlo sampler based on,
proposing random steps in parameter space and using a stochastic acceptance rule to sample from a target probability distribution.
The basic steps of the Metropolis-Hastings MCMC sampler are:
For \(\ell = 1,\ldots,L\):
Initialize - assign random initial values for the model parameters,
where \(\theta\) represents the complete set of model parameters.
Propose - generate new model parameters from a proposal distribution,
Calculate the acceptance probability - compute the probability of accepting the proposed parameters,
where,
\(P(\theta|y,X)\) is the posterior probability of the model parameters given the data
\(q(\theta'|\theta)\) is the proposal distribution probability of generating \(\theta'\) from the current state
Accept or reject - generate a random value,
If \(u < \alpha\), accept the proposal,
otherwise, retain the current sample,
Repeat - continue sampling until a sufficient number of samples are generated.
The resulting Markov chain provides samples that approximate the target posterior distribution.
Used in:
Compare with:
Also see:
Minkowski Distance#
A general distance metric where the well-known Manhattan and Euclidean distances are special cases.
where \(m\) is the number of features and \(p\) controls the distance metric.
Common cases include,
\(p=1\) - Manhattan distance - sums absolute differences and is less dominated by large coordinate differences
\(p=2\) - Euclidean distance - computes the straight-line distance between observations
\(p=\infty\) - Chebyshev distance - considers only the maximum coordinate difference between observations
Algorithms that may use Minkowski distance include,
Used in: TBA - in training and tuning chapter
Missing At Random#
Missing feature values are considered Missing At Random (MAR) when the probability of a feature value being missing depends on other observed variables in the dataset, but not on the missing value itself after accounting for those observed variables.
For example,
permeability measurements may be missing more frequently in certain lithologies, depths, or sampling conditions where those related features are available
laboratory measurements may be unavailable due to known sample characteristics or collection conditions
samples may be selectively collected based on available information to reduce uncertainty and improve decision making
MAR differs from MCAR because missingness is not random across the entire dataset; however, the missingness mechanism can be explained using observed information.
Appropriate approaches for MAR data may include,
model-based imputation
multiple imputation
including variables related to missingness in predictive models
If MAR data are ignored, missing values may result in,
biased sample statistics
biased model training and evaluation
biased predictions with potentially no indication of the underlying bias
Used in:
Also see:
Missing Completely At Random#
Missing feature values are considered Missing Completely At Random (MCAR) when the probability of a feature value being missing is independent of both observed and unobserved data.
In this case,
missing values occur randomly throughout the dataset
samples with missing values are statistically representative of the complete dataset
removing samples with missing values does not introduce systematic bias
Examples include,
accidental data recording failures
random equipment failures during measurement
randomly lost samples during data transfer or storage
MCAR is the most favorable missing data condition because standard approaches such as removing incomplete samples may not introduce bias.
However, true MCAR conditions are uncommon in real-world datasets because data collection processes often introduce systematic missingness.
Used in:
Also see:
Missing Feature Values#
Missing values in a data table occur when samples do not contain measurements for all features.
There are many causes of missing feature values, for example,
Sampling cost - some measurements may be expensive, time-consuming, or impractical to collect, e.g., low permeability tests that require long-duration experiments
Sample recovery limitations - some samples may be difficult to recover or measure due to their physical properties, e.g., inability to recover weak mudstone samples
Targeted sampling strategies - samples may be collected to reduce uncertainty and maximize decision value rather than to achieve statistical representativity, e.g., dual-purpose samples collected for both information gain and production evaluation
Missing data consequences extend beyond reducing the amount of training and testing data. If missing values are not completely random, they may result in,
biased sample statistics, leading to biased model training and evaluation
biased models and predictions, potentially with no indication of the underlying bias
Used in:
Also see:
Missing Not At Random#
Missing feature values are considered Missing Not At Random (MNAR) when the probability of a feature value being missing depends on the missing value itself or other unobserved information.
For example,
permeability measurements may be missing because very low permeability samples require excessive testing time
weak mudstone samples may not be recovered because their physical properties prevent successful sampling
production-related measurements may be preferentially collected only from high-value or successful operations
MNAR is the most challenging missing data condition because the missingness mechanism contains information that is not directly observed.
Consequences of MNAR data include,
biased sample statistics because missing observations are systematically different from available observations
biased models and predictions
uncertainty that is underestimated because the missingness mechanism is not fully represented
Addressing MNAR data typically requires additional assumptions, domain knowledge, sensitivity analysis, or explicit modeling of the missing data process.
Used in:
Also see:
Model Bagging#
The application of bootstrap resampling to generate \(B\) realizations of the training dataset,
used to train an ensemble of predictive models,
where,
\((X_1^b, \dots, X_m^b)\) – the predictor features in the \(b^{th}\) bootstrap sample
\(\hat{f}^b\) – the predictive model trained on the \(b^{th}\) bootstrap sample
\(\hat{Y}^b\) – the prediction from the \(b^{th}\) model
The ensemble of predictions is then aggregated to reduce model variance. The aggregation depends on the prediction task:
Regression – average the predictions,
Classification – assign the class receiving the majority vote,
Bagging can be applied to almost any predictive model. In fact, the BaggingClassifier and BaggingRegressor classes in scikit-learn are wrappers that use any compatible prediction model as the base learner to construct a bagged ensemble.
Used in:
Also see:
Model Bias#
The additive component of the expected testing mean square error that is caused by a predictive model being too simple and inflexible,
Model hyperparameter tuning is applied to balance model bias and model variance
model bias variance trade-off - as model complexity increases, model variance increases and model bias decreases, and vice versa.
Used in:
Also see:
Model Bias–Variance Trade-off#
Model hyperparameter tuning is applied to balance the model bias and model variance components of the expected test mean square error.
The model bias–variance trade-off results in,
as model complexity increases, model variance generally increases and model bias decreases
as model complexity decreases, model variance generally decreases and model bias increases
The objective is to select a model complexity that minimizes expected prediction error by balancing,
sufficient flexibility to capture important patterns in the data
sufficient constraint to avoid excessive sensitivity to training data variation
Increasing model complexity cannot reduce the irreducible error component of prediction uncertainty.
Used in:
Also see:
Model Checking#
A set of critical steps in any spatial modeling workflow to ensure the models are ready to support decision making. Model checking evaluates whether the model honors the available information, accurately predicts known data, and provides a reliable representation of uncertainty.
Examples of model checks include,
Model Inputs - data and statistics integration
check that the model honors the input data and statistical assumptions, generally evaluated over all realizations
for example, compare output histograms, variograms, and correlation coefficients from the realizations against the input statistics
Accurate Spatial Estimates - ability of the model to predict away from available sample data
evaluate predictive performance using cross validation, where some data are withheld and then predicted by the model
predictive accuracy is generally summarized with a truth versus predicted cross plot and measures such as mean square error,
Accurate and Precise Uncertainty Modelss - the uncertainty model is consistent with the amount of information available and the sources of uncertainty
evaluate uncertainty using cross validation by withholding data and checking whether the observed values occur within the predicted probability intervals at the expected frequency
summarize uncertainty goodness with observed proportion within interval versus the predicted probability interval
points on the 45 degree line indicate a good uncertainty model
points above the 45 degree line indicate an overly conservative uncertainty model, where uncertainty intervals are too wide
points below the 45 degree line indicate an under-estimated uncertainty model, where uncertainty intervals are too narrow or the model is biased
Used in:
Model Complexity#
Model complexity describes the capacity of a predictive machine learning model to fit patterns in data and the difficulty of interpreting the resulting model.
In general,
increasing model complexity produces a more flexible model that can capture more complex patterns, but makes the model more difficult to interpret and may increase the risk of an overfit model
decreasing model complexity produces a less flexible model that is easier to interpret, but may not capture important patterns in the data and may increase the risk of underfitting
A variety of concepts may be used to characterize model complexity, including,
the number of predictor features included in the model, which influences the dimensionality of the feature space and may increase the number of model parameters
the number of model parameters and the mathematical complexity of the model terms, e.g., linear terms, polynomial terms, thresholds, and interaction terms
the structure and representation of the model, e.g., a compact equation in polynomial regression, nested conditional rules in decision trees, or thousands of structured weights and biases in neural networks
Examples of increased model complexity include,
higher-order polynomial regression
deeper decision trees
neural networks with more layers and parameters
In general, more flexible models are more difficult to interpret,
linear regression provides interpretable model coefficients that can be analyzed and used for feature importance or ranking
support vector machines with radial basis function kernels are linear models in an implicit high-dimensional feature space, but the transformed representation makes interpretation of individual model parameters difficult
Used in:
Model Generalization#
The ability of a predictive machine learning model to make accurate predictions beyond the specific observations used for training.
Model generalization may be evaluated for different types of prediction cases,
interpolation cases - predictions within the range of the training data where similar examples are represented in the training dataset
extrapolation and edge cases - predictions near or beyond the tails of the predictor feature distributions where limited examples are available
black swan cases - predictions for unforeseen cases that are substantially different from the training data and may represent new regions of the feature space
A model with good generalization,
learns the underlying structure and relationships in the data rather than memorizing individual training observations
captures patterns that transfer to new observations and conditions
Models that do not generalize well include,
overfit models - models that learn training data details or noise, resulting in high training accuracy but reduced performance on testing or new data
underfit models - models that are too simple or inflexible to represent the underlying phenomenon, resulting in poor performance on both training and testing data
Used in:
Also see:
Model Hyperparameter#
Model settings specified prior to training or estimation that control the structure, flexibility, complexity, or smoothness of a model.
Hyperparameters are not directly estimated from the training data but are selected using approaches such as validation data, cross-validation, optimization procedures, or expert knowledge.
Examples include,
regularization strength in regression models
tree depth and minimum samples per split in decision trees
nugget effect, range, and sill in variogram models (geostatistics)
Trend Model order or complexity
parameters controlling data conditioning, smoothing, or model flexibility in spatial models
For example, polynomial model complexity may be controlled by selecting the polynomial order,
where the polynomial order is the hyperparameter and the coefficients, \(b_i\), are model parameters estimated from the data.
The selected polynomial order controls model complexity,
the first-order model is less flexible and has lower complexity
the fourth-order model is more flexible and has higher complexity
Hyperparameters influence the balance between model flexibility, model bias, model variance, and model generalization.
Used in:
Contrast with:
Model Hyperparameter Tuning#
Workflow to identify the hyperparameter combination that provides the best predictive performance for a machine learning model,
for data not used to train the model parameters
performed with withheld data - testing data with train and test workflow, or validation data for train, validate and test workflows
Unlike model parameters, which are estimated during training, hyperparameters are specified before training and control the model complexity, flexibility, or training process.
The general workflow is,
specify a range of candidate hyperparameter values or combinations
train a model for each hyperparameter combination
evaluate prediction performance with withheld testing data
select the hyperparameter combination that minimizes the chosen prediction error metric
retrain the final model using the selected hyperparameters
Hyperparameter tuning commonly uses,
a single train and test split, holdout cross validation
repeated training and testing splits, such as K-fold cross validation
out-of-bag error for model bagging-based ensemble methods
Examples of hyperparameters include,
decision tree – maximum tree depth and minimum samples per leaf
random forest – number of trees and number of candidate predictor features considered at each split
ridge and LASSO regression – regularization parameter, \(\lambda\)
support vector machine – cost parameter, \(C\), and kernel-specific hyperparameters
artificial neural network – learning rate, batch size, number of hidden layers, number of nodes per layer, activation function, and number of training epochs
Additional comments,
hyperparameter tuning estimates model complexity appropriate for the available data
increasing model complexity generally reduces training error but may increase testing error due to overfitting
the optimal hyperparameters are those that provide the best prediction accuracy on previously unseen data
hyperparameter tuning and model evaluation should be based on the same prediction metric, such as MAE, MSE, RMSE, or classification accuracy
Used in:
Also see:
Contrast with:
Also see:
Model Parameter#
Quantities estimated from data that define a model and control its fit to training data observations.
Model parameters are typically obtained through a process known as model parameter training based on,
optimization
analytical solutions
statistical estimation
with methods such as,
least squares
maximum likelihood
Examples include regression coefficients, covariance values, and trend coefficients.
For example, for a polynomial model,
where \(b_3\), \(b_2\), \(b_1\), and \(b_0\) are model parameters.
Used in:
Contrast with:
Model Parameter Training#
Workflow to estimate the model parameters that minimize a specified loss function over the training data.
Model parameter training seeks parameter values that optimize a specified objective,
minimizing a loss function
maximizing a likelihood function
satisfying analytical estimation equations, when available
Common parameter training methods include,
analytical solutions, such as least squares
iterative optimization using gradient-based optimization
Bayesian estimation using posterior probability distributions
Unlike hyperparameters, which are specified before training, model parameters are learned automatically from the training data.
The general workflow is,
initialize the model parameters
calculate model predictions over the training data
evaluate the loss function
update the model parameters to reduce the loss
repeat until convergence or a stopping criterion is satisfied
The parameter training method depends on the prediction model,
linear regression - analytical least-squares solution
ridge regression - analytical regularized least-squares solution
LASSO regression - iterative optimization
decision tree - recursive partitioning of the predictor feature space
support vector machine - constrained optimization
artificial neural networks - backpropagation with gradient-based optimization
Examples of model parameters include,
linear regression - feature weights and intercept
decision tree - split locations and response values within terminal regions
random forest - the parameters of each decision tree in the ensemble
support vector machine - hyperplane coefficients
artificial neural network - connection weights and node biases
Additional comments,
model parameters are estimated separately for every realization of model training
parameter training minimizes the specified loss function for fixed hyperparameter values
changing the hyperparameters generally changes the resulting model parameters
model parameter training and model hyperparameter tuning are typically repeated together until a satisfactory prediction model is obtained
Used in:
Also see:
Contrast with:
Model Variance#
The component of the expected testing mean square error caused by a predictive model being overly sensitive to variations in the training data.
The bias–variance decomposition of the expected testing mean square error is,
where,
model variance represents the variability in model predictions that arises from changes in the training data
high model variance indicates that the model is sensitive to the particular observations used for training
high model variance is commonly associated with an overfit model
Model hyperparameter tuning is used to balance model variance and model bias.
model bias–variance trade-off - as model complexity increases, model variance generally increases while model bias decreases, and vice versa
Used in:
Also see:
Momentum#
An optimization technique that improves the stability and convergence of gradient descent by combining the current optimization step with information from previous optimization steps.
momentum, \(\lambda\), controls how much of the previous optimization step is retained, while \(1-\lambda\) determines the influence of the current gradient step
A momentum update may be written as,
followed by the model parameter update,
where,
\(\theta\) represents the model parameters
\(s_t\) is the optimization step at iteration \(t\)
\(r\) is the learning rate
\(\nabla L_t\) is the gradient of the loss function at iteration \(t\)
\(\lambda\) is the momentum coefficient, typically between 0 and 1
The optimization step is often referred to as the velocity in the machine learning literature because momentum was originally derived from an analogy with Newtonian mechanics, where model parameters correspond to position and gradients act as forces.
since optimization iterations occur at constant time intervals, the time increment is effectively unity, and the velocity is proportional to the optimization step.
consequently, the terms velocity and optimization step are mathematically equivalent in this context.
Momentum provides several benefits,
reduces the effect of noisy gradient estimates
smooths the optimization trajectory by averaging successive optimization steps
reduces zig-zag motion across narrow valleys of the loss function
accelerates convergence in directions with consistent gradients
helps the optimization continue through shallow local irregularities while remaining directed toward regions of lower loss
Used in:
Also see:
Monte Carlo Method#
Algorithm that solve computational problems through repeated random sampling and statistical analysis of the resulting samples.
Monte Carlo methods approximate quantities that may be difficult to calculate analytically by using simulated random realizations from an underlying probability distribution. Common examples include,
Numerical integration - estimating an integral by sampling random values,
using,
Uncertainty simulation - propagating uncertainty through a model by repeatedly sampling uncertain input features and evaluating the resulting outcomes,
to estimate statistics such as expected values, confidence intervals, and probability of exceeding thresholds.
Bootstrap resampling - estimating uncertainty in statistical quantities by repeatedly sampling from an empirical distribution.
Monte Carlo methods are extended to,
Markov Chain Monte Carlo, which uses a stochastic process to generate samples from complex probability distributions when direct sampling is not feasible.
Used in:
Also see:
Monte Carlo Simulation#
A method for generating random samples from one or more statistical distributions. A random sample from a distribution is defined as a random variable, \(X\). The steps for Monte Carlo simulation are:
Model the feature cumulative distribution function, \(F_x(x)\).
Draw a random value from a uniform \([0,1]\) distribution, representing a random cumulative probability value, \(p^{\ell}\).
Apply the inverse cumulative distribution function to calculate the associated sample value,
Repeat steps 2 and 3 to calculate enough realizations for subsequent analysis.
Monte Carlo simulation is a fundamental building block of stochastic simulation and uncertainty workflows. Examples include,
Monte Carlo simulation workflow - apply Monte Carlo simulation over all uncertain features, then apply a transfer function to calculate a realization of the decision criteria. Repeat this process to generate many realizations and propagate uncertainty through the transfer function.
Bootstrap - applies Monte Carlo simulation to generate realizations of the sample data, allowing estimation of uncertainty in sample statistics or ensembles of prediction models for ensemble-based machine learning.
Monte Carlo method - use random sampling to approximate solutions to complex problems, with the solution generally converging as the number of random samples increases.
Used in:
Also see:
Monte Carlo Simulation Workflow#
A general stochastic Monte Carlo simulation workflow for propagating uncertainty through a transfer function. The workflow includes the following steps,
Model the uncertainty distributions or cumulative distribution functions for all input features,
Monte Carlo simulate realizations for all input features,
Apply the transfer function to calculate a realization of the output, often the decision criteria,
Repeat steps 2 and 3 to calculate enough realizations to model the output uncertainty distribution,
The input feature realizations may be simulated independently or with relationships between features included through multivariate uncertainty models.
Used in:
Also see:
Multidimensional Scaling#
Machine learning method in inferential statistics and information visualization for exploring the similarity (or conversely the dissimilarity) between samples from a high-dimensional dataset.
Multidimensional scaling (MDS) projects data from \(m\) dimensions to a lower-dimensional space with \(p \ll m\),
while attempting to preserve the pairwise distances or dissimilarities between samples
ideally projecting to \(p=2\) or \(3\) dimensions for visualization and exploration
Unlike methods that operate directly on predictor features, multidimensional scaling requires only the pairwise distances or dissimilarities between samples.
For multidimensional scaling,
the original feature values are not required; only the distance or dissimilarity matrix between samples
as with any distance-based method, feature standardization is often applied so that features with larger variance do not dominate the distance calculations
a variety of distance or dissimilarity measures may be used depending on the application
Comparison between multidimensional scaling and principal component analysis,
Principal Component Analysis (PCA) - operates on the covariance matrix (\(m \times m\)) of the predictor features and finds orthogonal linear projections that maximize explained variance
Multidimensional Scaling (MDS) - operates on the pairwise distance matrix (\(n \times n\)) between samples and finds a low-dimensional representation that best preserves those pairwise distances
The resulting low-dimensional representation provides new opportunities for data exploration, including,
visualization of clusters and sample relationships
analog selection based on sample proximity
visualization of the diversity and coverage of training data
visualization and conceptualization of uncertainty space
identification of unusual observations and potential outliers
Used in:
Also see:
Multiple Linear Regression#
A regression model that extends linear regression by predicting a response feature from multiple predictor features using a linear combination of the predictor feature(s).
The general form is,
where \(\hat{Y}\) is the predicted response, \(\beta_0\) is the intercept, \(\beta_1,\ldots,\beta_p\) are regression coefficients, and \(X_1,\ldots,X_p\) are predictor features.
Multiple linear regression is widely used in statistical modeling and machine learning to quantify relationships between a response variable and multiple explanatory features, including,
feature selection - regression coefficients (\(\beta\)) and standardized predictor feature effects
inference - significance testing of linear relationships
Used in:
Also see:
Multiplication Rule#
The joint probablity of \(A\) and \(B\) as the product of the conditional probability of \(B\) given \(A\) with the marginal probability of \(A\),
The multiplication rule is axiomatic as it is derived as a simple manipulation of the definition of conditional probability, in this case,
and the definition of conditional probability is readily obseved from a simple Venn diagram.
Used in: Probability Multiplication Rule Defintion and Demonstration
Multivariate#
Involving more than two features (variables) considered together, often to study their relationships, dependence, or correlation.
For examples, see multivariate analysis.
Used in:
Compare with:
Multivariate Analysis#
The analysis of more than two features (variables) measured over a collection of samples to investigate their relationships, dependence, and correlation.
Examples include:
multivariate statistical models such as multiple linear regression
scatter plot matrices showing relationships among multiple features
covariance and correlation coefficient matrices
A common approach to multivariate analysis is to evaluate pairwise relationships among features using covariance, correlation, and scatter plots.
note, this is a simplified multivariate analysis because it considers only pairwise relationships among features. Higher-order relationships involving three or more features simultaneously are not explicitly modeled.
Used in:
Also see:
Compare with:
Mutual Information#
Generalized statistic from information theory for quantifying statistical dependence between features without assuming a specific functional form.
Mutual information,
quantifies the amount of information gained about one feature by observing another feature
measures statistical dependence without assuming a linear or other parametric relationship
is always non-negative
is zero only when the two features are statistically independent
is measured in bits when the logarithm is base 2 (or nats when the natural logarithm is used)
The calculation of mutual information is based on,
comparing the observed joint probability with the joint probability expected if the features were statistically independent
quantifying the departure from independence by comparing the joint probability, \(P(x,y)\), with the product of the marginal probabilities, \(P(x)\cdot P(y)\)
For statistically independent features,
For discrete (or binned continuous) features, mutual information is calculated as,
If the two features are statistically independent,
and therefore,
The joint probability, \(P_{X,Y}(x,y)\), acts as a weighting term, so regions of the feature space that occur more frequently contribute more to the mutual information than rare combinations.
For continuous (non-binned) features, the integral form is,
Larger mutual information values indicate stronger statistical dependence between features, regardless of whether the relationship is linear or nonlinear.
Unlike the Correlation Coefficient, which measures only linear association, mutual information detects both linear and nonlinear statistical dependence.
Used in:
Also see:
Mutually Exclusive Events#
Events that cannot occur together; they have no common outcomes. Using set notation, events \(A\) and \(B\) are mutually exclusive if,
Therefore, the probability of the intersection of mutually exclusive events is,
For mutually exclusive events, the probability of a union simplifies to the sum of the individual probabilities,
Used in:
Joint Probability#
Probability of an event in which two or more events occur together,
The probability of the intersection is,
where we may state, “probability of A and B”. If \(A\) and \(B\) are independent, then the joint probability simplifies to,
Without the assumption of independence, we apply the multiplication rule,
or equivalently,
Used in:
See also:
K-Bins Discretization#
A feature transformation method based on dividing a continuous feature into \(K\) discrete intervals (bins) over the feature range, then assigning each sample to a bin. For one-hot encoding bin representation, a value of 1 is assigned if the sample belongs to a bin and 0 if it is outside the bin.
A feature transformation method based on dividing a continuous feature into \(K\) discrete intervals (bins) over the feature range, then assigning each sample to a bin. The output may be either,
a single bin label indicating the assigned interval
a one-hot encoding representation with a value of 1 if the sample belongs to a bin and 0 if it is outside the bin
Binning strategies include:
uniform width bins - divides the feature range into \(K\) intervals of equal width,
quantile bins - divides the feature values into \(K\) intervals containing approximately the same number of samples.
This is a continuous range is partitioned into discrete intervals to simplify representation,
analogous to constructing a histogram, where the count of samples within each bin forms the height of each bar.
analogous to posterization in image processing, where continuous pixel intensity or color values are reduced to a smaller number of discrete levels.
Methods and applications that utilize \(K\) bins discretization include,
basis expansion to represent nonlinear relationships by transforming features into a higher-dimensional space,
discretization of continuous features into categorical features for categorical methods such as the naive Bayes classification,
histogram construction and Chi-square tests for comparing differences between distributions,
mutual information-based feature discretization and feature selection.
Used in:
Also see:
K-Fold Cross Validation#
A cross validation method based on partitioning the data into \(K\) folds and looping over the folds,
withhold the current fold as validation data and train the model using the remaining \(K-1\) folds,
calculate the model performance metric over the withheld validation fold,
aggregate the validation performance over all folds to estimate model generalization performance.
Relating K-fold cross validation to the holdout method with a single train and test split,
K-fold cross validation provides a more robust performance estimate than a single holdout split by allowing all samples to be used for both training and validation, and averaging performance over folds reduces the sensitivity to a particular train-test partition.
the relative size of the training and validation datasets is controlled by \(K\); for example, \(K=4\) produces approximately \(25\%\) validation data per fold and \(75\%\) training data, while \(K=5\) produces approximately \(20\%\) validation data per fold and \(80\%\) training data.
K-fold cross validation is commonly applied for estimating model prediction accuracy and may also be used to assess uncertainty in model performance and goodness (Maldonado-Cruz and Pyrcz, 2021).
Used in:
Compare with:
Also see:
k-Means Clustering#
An unsupervised machine learning method for cluster analysis that assigns unlabeled data samples into groups by minimizing dissimilarity within clusters, represented by the inertia loss function,
where,
\(i\) is the cluster index,
\(K\) is the total number of clusters,
\(C_i\) represents the set of samples assigned to cluster \(i\),
\(\alpha\) is the data sample index,
\(X_{\alpha}\) is a data sample,
\(\mu_i\) is the prototype (centroid) of cluster \(i\),
\(||X_{\alpha}-\mu_i||^2\) is the squared Euclidean distance between a sample and its cluster prototype in \(M\)-dimensional feature space calculated as,
Here is a summary of important aspects for k-means clustering,
partitional clustering - provides a single set of group assignments by partitioning samples into \(K\) clusters.
number of clusters (\(K\)) - specified as a model hyperparameter that controls the number of groups.
exhaustive and mutually exclusive groups - every data sample is assigned to exactly one cluster.
prototype method - represents the training data using a set of synthetic prototypes in feature space. For k-means clustering, \(K\) prototypes are assigned and iteratively updated.
unsupervised learning - training data are unlabeled and assigned cluster labels based on proximity to prototypes in feature space. The underlying assumption is that samples that are similar, represented by proximity in feature space, should belong to the same cluster.
feature weighting - k-means depends on Euclidean distance between training samples and prototypes. Distance is treated as an inverse measure of similarity. If features have significantly different magnitudes or ranges, features with larger values dominate the loss function and control the resulting clusters. Common approaches include standardization or normalization of features, while unequal feature weighting may also be applied. In this demonstration, features are normalized to a range from 0.0 to 1.0.
heuristic algorithm iterative solution - initial prototypes are assigned in the feature space, sample labels are updated by assigning each sample to the nearest prototype, and prototypes are recalculated as the centroid of their assigned samples. This process repeats until cluster assignments no longer change.
Assumptions and limitations of k-means clustering,
spherical, data convexity, isotropic clusters - minimizes distance between samples and their group prototype
equal variance for all features - requires reliable measures of distance in feature space
equal prior probability for all clusters - naïve a priori clustering membership information (uniform distribution)
Used in:
Also see:
Contrast with:
k-Nearest Neighbours#
Nonparametric predictive machine learning model based on a local weighting applied to the \(k\) nearest training data, resulting in a method that is,
very simple, interpretable and flexible - the summary of the \(k\) nearest training data can be easily explained
mapping - analogous to mapping in the predictor feature space
adaptive - in regions with few training data the result is smoother than regions with denser training data available
instance-based, lazy learning method - the model training is postponed until prediction is required, no precalculation of the model. i.e., prediction requires access to the data.
The k-nearest neighbours approach is conceptually related to convolution and kernel-based smoothing used in spatial interpolation. Both methods estimate a prediction at a location by applying weights to nearby information.
In convolution, a weighting function (kernel) is shifted across a function to calculate a weighted average,
where the weighting function controls the contribution of neighboring values.
Similarly, k-nearest neighbours applies locally adaptive weighting by selecting the \(k\) closest training samples in feature space and combining their values for prediction.
unlike standard convolution, where the weighting function and window size are predefined, k-nearest neighbours uses a data-driven neighborhood size determined by \(k\).
The hyperparameters include,
\(k\) - number of nearest data to utilize for prediction
data weighting - for example uniform weighting with the local training data average, or inverse distance weighting
Note, for the case of inverse distance weighting, the method is analogous to inverse distance weighted interpolation with a maximum number of local data constraint commonly applied for spatial interpolation.
inverse distance is available in GeostatsPy for spatial mapping.
Too find the k-nearest data a distance metric is needed,
training data within the predictor feature space are ranked by distance (closest to farthest)
a variety of distance metrics may be applied, including:
Euclidian distance
Minkowski Distance - a general expression for distance with well-known Manhattan and Euclidean distances are special cases,
when \(p=2\), this becomes the Euclidean distance
when \(p=1\) it becomes the Manhattan distance
Used in:
Also see:
Kernel Trick#
It is possible to incorporate a basis expansion in our method without ever needing to transform the training data to this higher dimensional space,
Instead, we only need the inner product over the predictor features,
instead of the actual values in the higher dimensional space, we just need the ‘similarity’ between all available training data in that transformed space! To emphasize this,
the radial basis function (Gaussian) kernel with Taylor series expansion has inifite components, i.e., infinite dimensional space, but all we need is this similarity,
An example application of the kernel trick for machine learning,
training our support vector machines with only a similarity matrix between training data that will be projected to the higher dimensional space
Used in:
Also see:
Key#
A component of the attention mechanism that allows a model to dynamically focus on the most relevant information while reducing the influence of less relevant information.
A Key (\(K\)) is a descriptor used to determine whether stored information is relevant to a given Query.
The attention mechanism,
compares each Query with all Keys to calculate similarity scores, which are then used to derive attention weights.
Some additional comments,
each Key–Value pair represents a stored piece of information, where the Key describes the information and the Value contains the associated content.
Used in:
Also see:
Kolmogorov Probability Axioms#
The three axioms proposed by Andrey Kolmogorov that establish the rigorous mathematical foundation for probability theory.
Probability of an event is a non-negative number,
Probability of the entire sample space, all possible outcomes \(\Omega\), is one (unity), also known as probability closure,
Additivity of mutually exclusive events for unions,
For example, the probability of two mutually exclusive events \(A_1\) and \(A_2\) is,
Used in:
Kriging#
Spatial estimation approach that relies on linear weights that account for spatial continuity, data closeness and redundancy. The kriging estimate is,
the right term is the unbiasedness constraint, where one minus the sum of the weights is applied to the global mean.
In the case where the trend, \(t(\bf{u})\), is removed, we now have a residual, \(y(\bf{u})\),
the residual mean is zero so we can simplfy our kriging estimate as,
The simple kriging weights are calculated by solving a linear system of equations,
that may be represented with matrix notation as,
This system may be derived by substituting the equation for kriging estimates into the equation for estimation variance, and then setting the partial derivative with respect to the weights to zero.
we are optimizing the weights to minimize the estimation variance
this system integrates the,
spatial continuity - as quantified by the variogram (and covariance function to calculate the covariance, \(C\), values)
redundancy - the degree of spatial continuity between all of the available data with themselves, \(C(\bf{u}_i,\bf{u}_j)\)
closeness - the degree of spatial continuity between the available data and the estimation location, \(C(\bf{u}_i,\bf{u})\)
Kriging provides a measure of estimation accuracy known as kriging variance (a specific case of estimation variance).
Kriging estimates are best in that they minimize the above estimation variance.
Properties of kriging estimates include,
Exact interpolator - kriging estimates with the data values at the data locations
Kriging variance - a measure of uncertainty in a kriging estimate. Can be calculated before getting the sample information, as the kriging estimation variance is not dependent on the values of the data nor the kriging estimate, i.e. the kriging estimator is homoscedastic.
Spatial context - kriging takes integrates spatial continuity, closeness and redundancy; therefore, kriging accounts for the configuration of the data and structural continuity of the feature being estimated.
Scale - kriging by default assumes the estimate and data are at the same point support, i.e., mathematically represented as points in space with zero volume. Kriging may be generalized to account for the support volume of the data and estimate,
Multivariate - kriging may be generalized to account for multiple secondary data in the spatial estimate with the cokriging system. We will cover this later.
Smoothing effect - of kriging can be forecasted as the missing variance. The missing variance over local estimates is the kriging variance.
Kriging Variance#
A measure of accuracy and uncertainty for a kriging estimate, expressed as,
Kriging variance is a specific case of estimation variance,
Can be calculated before getting the sample information,
the kriging estimation variance is not dependent on the values of the data nor the kriging estimate, i.e. the kriging estimator is homoscedastic.
Kriging-based Declustering#
A declustering method to assign weights to spatial samples based on local sampling density using the spatial continuity model from a variogram model. The objective is to assign weights such that the weighted statistics are intended to be more representative of the population.
samples in densely sampled areas receive less weight
samples in sparsely sampled areas receive more weight
Kriging-based declustering proceeds as follows:
calculate and model the experimental variogram
apply kriging to calculate estimates over a high-resolution grid covering the volume of interest
accumulate the kriging weights assigned to each data sample over the entire grid
assign declustering weights proportional to the accumulated kriging weights
The kriging-based declustering weight for data sample \(j\) is calculated as,
where \(n\) is the number of data samples, \(n_x\) and \(n_y\) are the number of grid cells in the declustering grid, and \(\lambda_{j,ix,iy}\) is the kriging weight assigned to data sample \(j\) when estimating grid cell \(ix,iy\).
The resulting weights sum to the number of samples,
which allows the weights to be directly applied in weighted statistics.
Important considerations for kriging-based declustering,
like polygonal declustering, kriging-based declustering is sensitive to the boundaries of the area of interest; therefore, samples near the boundary may receive substantially different weights as the area of interest is expanded or contracted.
kriging-based declustering integrates the spatial continuity model from the variogram. Therefore, the variogram model selection can significantly impact the resulting weights.
if there is a 100% relative nugget effect, there is no spatial continuity and all samples receive equal weight. In this case, the kriging weights contain no spatial information and the declustering calculation must be handled separately to avoid division by zero.
geometric anisotropy may significantly impact the weights because data aligned along preferred directions may be considered spatially closer or farther based on the covariance function model.
Also see:
LASSO Regression#
A regularized version of linear regression. The acronym stands for,
Least Absolute Shrinkage and Selection Operator (LASSO),
and it has the same prediction equation as linear regression,
where the trainable model parameters are \(b_{\alpha}\), the feature weights, and \(b_0\), the constant intercept term.
The model parameters are estimated by minimizing a regularized least squares loss function that combines the residual sum of squares (RSS) with an L1 shrinkage penalty,
where \(y_i\) is the observed response feature value and \(\sum_{\alpha=1}^{m}b_{\alpha}x_{\alpha,i}+b_0\) is the model prediction for training sample \(i\).
LASSO regression introduces the hyperparameter \(\lambda\), which controls the strength of the L1 regularization penalty,
where,
larger values of \(\lambda\) increase the penalty on large slope coefficients
the intercept term \(b_0\) is not penalized because it represents the response mean rather than predictor influence
LASSO regression integrates two competing goals during model training,
minimize prediction error with respect to the training data
minimize the magnitude of the slope parameters toward zero
The hyperparameter \(\lambda\) controls the model bias–variance trade-off,
as \(\lambda\rightarrow0\), the solution approaches ordinary linear regression with no additional regularization bias, but potentially higher model variance
as \(\lambda\) increases, model variance decreases and model bias increases as the model becomes less flexible
as \(\lambda\rightarrow\infty\), the slope parameters \(b_1,\ldots,b_m\) approach zero and predictions approach the response feature mean
Due to the L1 regularization penalty, LASSO regression performs feature selection,
as \(\lambda\) increases, some model parameters are forced exactly to zero, effectively removing the corresponding predictor features from the model
Compared with ridge regression,
Ridge regression uses L2 regularization and shrinks coefficients toward zero, but generally retains all predictor features
LASSO regression uses L1 regularization and can shrink some coefficients exactly to zero, performing automatic feature selection
The assumptions of LASSO regression include,
Error-free predictors - predictor features are treated as known values, not random variables
Linearity - the expected response is a linear combination of predictor features
Constant variance - the variance of response error is constant over predictor feature values (homoscedasticity)
independence of error - errors in the response are uncorrelated with each other
Redundant predictors - highly correlated predictor features may reduce the stability of feature selection
Used in:
Also see:
Latent Feature#
A feature that is not directly observed in the original data but is learned or inferred by a model to represent underlying patterns, structure, or relationships.
the feature exists in a latent space
Latent features are typically combinations or transformations of the original predictor features and often provide a more compact or meaningful representation of the data.
Used in:
Also see:
Latent Space#
A lower-dimensional representation of data in which the coordinates correspond to Latent Feature that capture the underlying structure or patterns in the original data.
Latent features are not directly observed but are learned or inferred from the data during model training.
Data represented in a latent space often preserves important relationships while reducing dimensionality, enabling tasks such as visualization, clustering, compression, and generation.
Used in:
Also see:
Lazy Learning#
A machine learning approach where the training data are retained and generalization is performed when prediction queries are made.
after selecting model hyperparameters, the model retains the original training data and uses them directly during prediction
the computational effort is concentrated during the prediction phase, while training is typically fast
Examples include,
locally weighted regression
case-based reasoning
Used in:
Also see:
Contrast with:
Leave-One-Out Cross Validation#
A K-fold Cross Validation-based cross validation method with one sample withheld for testing in each fold, such that the number of folds equals the number of samples, \(K=n\).
Leave-one-out cross validation uses nearly all available data for training in each fold but may have high computational cost and high variance in the estimated predictive performance.
Contrast with:
Also see:
Likelihood Function#
In Bayes’ Theorem, the likelihood function describes the compatibility of new data with possible states or inferred model parameters. It quantifies how likely the observed data are for different model assumptions and is combined with the prior probability to calculate the posterior probability,
where:
\(P(A)\) is the prior probability representing uncertainty before observing new information,
\(P(B|A)\) is the likelihood function describing the compatibility of observations \(B\) with state or parameter \(A\),
\(P(B)\) is the evidence term used to normalize the posterior probability,
\(P(A|B)\) is the posterior probability representing updated uncertainty after incorporating observations.
Used in:
Also see:
Linear Regression#
Simple, linear parametric prediction model based on a linear weighted combination of the predictor features with a constant term,
where the trainable model parameters are \(b_{\alpha}\), the feature weights, and \(b_0\), the constant intercept term.
The analytical solution for the model parameters, \(b_1,\ldots,b_m,b_0\), is available by minimizing the L2 norm loss function. This is known as least squares because the residual errors are squared and summed over all training data,
where \(y_i\) is the observed response feature value and \(\sum_{\alpha=1}^{m}b_{\alpha}x_{\alpha,i}+b_0\) is the model prediction for training sample \(i\).
The assumptions of linear regression include,
Error-free predictors - predictor features are treated as known values, not random variables
Linearity - the expected response is a linear combination of predictor features
Constant variance - the variance of the response error is constant over predictor feature values (homoscedasticity)
Iindependence of error - errors in the response are uncorrelated with each other
No multicollinearity - predictor features are not linearly redundant with each other
Used in:
Also see:
Local Accuracy#
Accuracy of a spatial estimate at individual locations, typically assessed by minimizing the local estimation uncertainty or estimation error.
For kriging, local accuracy is achieved by minimizing the estimation variance,
where \(Z^*(\mathbf{u})\) is the estimated value and \(Z(\mathbf{u})\) is the unknown true value.
Some general observations about local accuracy,
estimation methods prioritize local accuracy by honoring nearby data and minimizing estimation variance.
locally accurate estimates are often smooth because they represent the best estimate at each location, but they generally do not reproduce the spatial variability of the modeled feature.
estimates optimized for local accuracy may not reproduce global measures, such as the histogram, variogram, or correlation coefficient.
Used in:
Contrast with:
Local Measure#
A statistical measure that evaluates model performance at individual locations or over a local neighborhood.
Local measures are used to assess local accuracy.
local measures focus on location-specific agreement between estimates and observations or local uncertainty.
local measures do not assess reproduction of global measures, such as the entire histogram, variogram, or correlation coefficient.
Used in:
Contrast with:
Logistic Regression#
A supervised machine learning method that predicts the probability of membership in a categorical feature by applying a logistic function to a linear combination of the predictor features.
unlike linear regression, logistic regression predicts probabilities between 0 and 1, which may then be converted to class assignments.
Model parameters are estimated by minimizing the cross-entropy loss during model parameter training.
Applications include,
binary classification
multiclass classification (with suitable extensions)
Used in: TBA
Also see:
Loss Function#
Mathematical function that quantifies and summarizes the loss associated with the prediction errors over the training data and provides the objective minimized during model training.
For supervised learning, the general form is,
where \(Y\) is the observed response feature values and \(\hat{Y}\) are the corresponding model predictions.
A loss function,
quantifies the agreement between model predictions and the observed response feature values
provides a single objective for model parameter optimization
is minimized during model training
The total loss over the training data is commonly calculated by aggregating the individual sample losses,
or by averaging,
depending on the learning algorithm.
Common regression loss functions include,
Mean Squared Error (MSE) - squares prediction errors, emphasizing larger errors
Mean Absolute Error (MAE) - uses absolute prediction errors and is more robust to outliers
Huber Loss - combines the robustness of MAE with the smooth optimization properties of MSE
Common classification loss functions include,
Cross-Entropy Loss (Log Loss) - compares predicted class probabilities with the observed categories
Hinge Loss - used by support vector machines to maximize the classification margin
Additional comments,
the choice of loss function influences the model parameters learned during training
the loss function used for training does not have to be the same metric used to evaluate model performance
regularization terms are often added to the loss function to reduce model complexity and improve generalization
Examples include,
ridge regression - residual sum of squares (RSS) plus an L2 regularization penalty
LASSO regression - residual sum of squares (RSS) plus an L1 regularization penalty
neural network - Mean Squared Error (MSE) for regression or cross-entropy for classification
Used in:
Loss Function for Linear Regression Description and Demonstration
Loss Function for Ridge Regression Description and Demonstration
Loss Function for LASSO Regression Description and Demonstration
Loss Function for Decision Tree Description and Demonstration
Loss Function for Random Forest Description and Demonstration
Also see:
Naive Bayes#
A classification machine learning model based on Bayesian updating with the simplifying assumption of conditional independence between predictor features.
The objective is to estimate the probability of a category, \(C_k\), given \(n\) predictor features,
Using Bayes’ theorem,
where,
\(P(C_k)\) is the prior probability of category \(k\)
\(P(x_1,\ldots,x_n|C_k)\) is the likelihood function of observing the features given category \(k\)
\(P(x_1,\ldots,x_n)\) is the evidence term
The likelihood term can be expanded using the chain rule,
The full likelihood requires knowledge of the joint conditional relationships between all predictor features. As the number of features increases, estimating this joint distribution requires increasingly large datasets.
The naive Bayes approach makes the simplifying assumption that all predictor features are conditionally independent given the category,
for all \(i=1,\ldots,n\) features.
The likelihood then simplifies to,
and the posterior probability becomes,
The evidence term, \(P(x_1,\ldots,x_n)\), depends only on the observed features and is constant across categories. Therefore, classification can be performed by comparing the unnormalized posterior probabilities,
and normalizing the results across all possible categories.
The naive Bayes approach,
requires only prior probabilities, \(P(C_k)\), and individual conditional distributions, \(P(x_i|C_k)\), rather than the full joint feature distribution
is computationally efficient and practical for high-dimensional problems
performs well with relatively small datasets because fewer probability distributions must be estimated
Although the conditional independence assumption is often unrealistic, naive Bayes can still provide effective classification performance when the assumption is approximately valid or when the simplified model generalizes better than a more complex alternative.
Used in:
Also see:
Neural Network#
A flexible, nonlinear machine learning model inspired by the structure of biological nervous systems.
The biological analogy includes,
neurons \(\rightarrow\) nodes
synapses \(\rightarrow\) weighted connections between nodes
Feed-forward fully connected neural networks consist of,
input layer - receives the predictor features
hidden layers - progressively transform the information
output layer - produces one or more response feature(s), prediction(s)
Information flows forward through the network,
each node forms a weighted sum of the incoming information, adds a bias, and applies an activation function,
where \(w_{ij}\) are the connection weights, \(b_j\) is the node bias, and \(g(\cdot)\) is the activation function.
This process is repeated over all hidden layers until the output layer produces the prediction.
Flexible neural network architectures include,
fully connected neural network (FCNN) - every node is connected to every node in the next layer
convolutional neural network (CNN) - convolution operators and feature maps summarize local spatial patterns while preserving spatial relationships
recurrent neural network (RNN) - feedback connections provide memory of previous inputs for sequential or temporal data
autoencoder (AE) - bottleneck architecture learns a compact latent representation before reconstructing the original data
Training neural networks,
model parameters consist of all connection weights and node biases
model parameters are estimated by backpropagation, where gradients of the loss function are propagated backward through the network
gradient-based optimization updates the model parameters to minimize the loss function
Neural networks are characterized by,
universal function approximators - with sufficient complexity they can approximate a wide variety of nonlinear relationships
high model complexity - often containing thousands to millions of trainable parameters
low interpretability - predictions arise from many interacting parameters and information pathways
Used in:
Also see:
ndarray#
The fundamental N-dimensional array data structure in the Python numerical computing package, NumPy, used to efficiently store and manipulate collections of numerical data.
An ndarray represents data organized along one or more dimensions,
1D arrays - vectors
2D arrays - tables, images, or maps
3D arrays - volumes, time series of maps, or subsurface models
higher-dimensional arrays - tensors and multidimensional scientific datasets
For geoscience applications, ndarray provides a convenient structure for working with exhaustive, regularly spaced gridded data over a 2D area of interest or 3D volume of interest, representing maps and models.
Advantages of ndarray include,
efficient storage, access, and manipulation of multidimensional numerical data
vectorized mathematical operations without requiring explicit loops
built-in methods to calculate multidimensional summary statistics
built-in methods for data queries, filtering, and conditional selection
built-in methods for data manipulation, cleaning, reshaping, and reformatting
support for loading and converting data from a variety of file formats and Python objects
attributes describing the array structure, including size, shape, number of dimensions, and data type
An ndarray contains,
the array values stored in a multidimensional structure
metadata describing the array dimensions and numerical representation
Used in:
Also see:
Node#
The basic computational unit of a neural network that receives information from connected nodes, performs a computation, and passes the result to other nodes.
More generally, a node (or vertex) is a fundamental element of a graph that represents an entity connected to other entities by edges.
Each node,
receives weighted inputs through one or more connections
combines the inputs, typically by computing a weighted sum with a bias
applies an activation function to produce an output
Nodes are organized into layers,
input layer - receives the predictor feature values
hidden layers - transform information through learned computations
output layer - pproduces one or more response feature(s), prediction(s)
Used in:
Also see:
Nonparametric Model#
A predictive machine learning model that does not assume a predefined functional form with a fixed number of model parameters.
Instead, a nonparametric model,
learns the structure and shape of the relationship from the training data
provides greater flexibility to represent complex and nonlinear natural systems
allows model complexity to increase with the amount of available data
Unlike parametric models, nonparametric models do not define the model complexity in advance by specifying a fixed mathematical relationship.
Nonparametric models often require more data because,
they estimate more flexible relationships directly from observations
the effective number of model parameters can increase with the amount of training data
greater flexibility can increase the risk of an overfit model without sufficient data or appropriate regularization
Examples of nonparametric models include,
Kernel-based methods such as support vector machine
Contrast with:
Norm#
Norm of a vector maps the vector components \([1,\ldots,n]\) to a single summary measure in the range \([0,\infty)\) that indicates,
the size or length of the vector
To train predictive machine learning models to training data, we require a single summary measure of mismatch between the model predictions and the training observations, called the training error.
The error is observed at each training data location,
and together these errors form an error vector,
The norm of the error vector, called the error norm, provides,
a single value to summarize the mismatch over all training data observations
an loss function that can be minimized during model training
There are a variety of norms that may be applied in machine learning. The general \(p\)-norm (Minkowski norm) for a vector \(\mathbf{x}\) is,
where,
\(p\) controls the type of norm
\(n\) is the number of dimensions in the vector
Commonly applied error norm cases include,
\(p=1\) - \(L_1\) norm (Manhattan norm)
\(p=2\) - \(L_2\) norm (Euclidean norm)
\(p\rightarrow\infty\) - \(L_{\infty}\) norm (Chebyshev norm)
Norm selection has an important impact the training of our model parameters, here’s a comparison of commonly applied error norms,
Property |
Least Absolute Deviations (L1) |
Least Squares (L2) |
|---|---|---|
Objective |
Minimize absolute errors |
Minimize squared errors |
Robustness to outliers |
Robust |
Less robust |
Solution stability |
May be unstable |
Stable |
Number of solutions |
Possibly multiple optimal solutions |
Unique analytical solution* |
Feature selection |
Built-in (promotes sparsity) |
No feature selection |
Model parameters |
Sparse solutions |
Non-sparse solutions |
Analytical solution |
Generally not available |
Available for linear regression |
* Assuming the predictor features are linearly independent.
Used in:
Also see:
Normalization#
A distribution rescaling method that transforms feature values to a specified range, commonly \([0,1]\).
Min-max normalization can be interpreted as shifting and stretching or squeezing a univariate distribution (e.g., histogram) to enforce a minimum value of 0.0 and a maximum value of 1.0,
This transformation is linear and therefore,
preserves the rank ordering of observations
does not change the relative shape of the distribution
changes the scale and location of the feature values
Normalization is useful when feature magnitude influences model behavior, for example,
distance-based methods where large-scale features can dominate distance calculations, such as k-Means Clustering clustering and k-Nearest Neighbours
models where feature coefficients are compared for interpretation or feature ranking
artificial neural networks where input scaling improves numerical optimization and avoids reduced sensitivity caused by activation function saturation
Normalization is applied to predictor features and may also be applied to response features, with inverse transformation used to return predictions to the original units.
Used in:
Also see:
Normalized Histogram#
A bar chart of the univariate statistical distribution with probability over an exhaustive set of bins over the range of possible values. These are the steps to build a normalized histogram,
Divide the continuous feature range of possible values into \(K\) equal size bins, \(\delta x\):
or use available categories for categorical features.
Count the number of samples (frequency) in each bin, \(n_k\), \(\forall k=1,\ldots,K\)
Divide each by the total number of data, \(n\), to calculate the probability of each bin,
Plot the probability vs. the bin label (use bin centroid if continuous)
Additional comments:
step 3 converts a standard histogram to a normalized histogram with a y-axis of probability instead of frequency
for categorical features, a normalized histogram represents an empirical probability mass function
for continuous features, the normalized histogram represents the empirical probability of each bin interval and is an approximation to the underlying continuous distribution
Used in:
Also see:
One-Hot Encoding#
A feature transformation method applied to categorical features that converts each category into a binary indicator vector.
Given a categorical feature with \(K\) possible categories (cardinality \(K\)), one-hot encoding creates a vector of length \(K\) where,
a value of 1 indicates that the sample belongs to that category
a value of 0 indicates that the sample does not belong to that category
For example, a categorical feature with three possible outcomes,
is transformed as,
One-hot encoding is equivalent to applying a categorical indicator transformation.
One-hot encoding is used for,
integrating nominal categorical features into machine learning prediction models
avoiding the incorrect assumption of ordering between categorical outcomes
providing numerical feature representations required by many machine learning algorithms
Considerations include,
the number of predictor features increases from one categorical feature to \(K\) binary indicator features
high-cardinality categorical features may create large numbers of sparse predictor features
Used in:
Also see:
Out-of-Bag#
In bootstrap resampling, each model realization is trained using a sample of the original data selected with replacement.
For a bootstrap sample of size \(n\), approximately \(\frac{2}{3}\) of the original observations are included in expectation,
and approximately \(\frac{1}{3}\) of the observations are not selected.
These unused observations are called out-of-bag (OOB) samples or observations.
For model bagging-based ensemble prediction models,
each model realization has a unique set of out-of-bag observations
each response observation, \(y_{\alpha}\), receives predictions from only the ensemble members where that observation was out-of-bag
with \(B\) bootstrap realizations, each observation receives approximately \(\frac{B}{3}\) out-of-bag predictions, \(\hat{y}^{*,b}_{\alpha}\)
The out-of-bag predictions are aggregated to calculate a single out-of-bag prediction,
average for regression
mode for classification
For regression,
where \(B_{\alpha}\) is the number of bootstrap models where observation \(\alpha\) was out-of-bag.
The out-of-bag mean square error is calculated as,
Out-of-bag error provides an internal validation estimate for bootstrap and model bagging-based ensemble models.
Advantages include,
avoids the need for a separate training and testing split
provides an estimate of model generalization during model hyperparameter tuning
uses all available observations for model training across the ensemble
Considerations include,
the effective validation proportion is approximately fixed at approximately 1/3
the OOB validation samples are generated through bootstrap sampling and may not represent the difficulty of the intended model application
a separate validation dataset may still be required when the deployment conditions differ substantially from the training data
Used in:
Also see:
Outlier#
An observation that differs substantially from the majority of the available samples and may have an unusually large influence on statistical analysis or machine learning models.
Outliers may result from,
measurement or recording errors
rare but valid observations
previously unobserved processes or populations
Important considerations include,
sensitivity - methods based on squared errors, such as mean square error and the L2 norm, are particularly sensitive to outliers
detection - identifying outliers requires consideration of the data distribution, context, and application rather than a single universal criterion
Used in:
Contrast with:
Overfit Model#
A predictive machine learning model that learns the noise, sampling variability, or specific idiosyncrasies of the training data rather than the underlying relationship in the natural system.
An overfit model demonstrates,
high prediction accuracy with training data but poor prediction accuracy with withheld testing data
excessive dependence on the specific training observations, effectively memorizing aspects of the training dataset
During model hyperparameter tuning, the overfit region is characterized by,
increasing model complexity that continues to reduce training error while increasing testing error
a divergence between training and testing performance as the model becomes increasingly flexible
This behavior represents the high model variance region of the model bias–variance trade-off.
Issues associated with an overfit machine learning model include,
more model complexity and flexibility than can be justified by the available data quantity, accuracy, frequency, and coverage
high accuracy during training but poor accuracy during testing, indicating limited ability to generalize to new cases
Overfitting may be reduced through,
reducing model complexity
increasing the amount or quality of training data
applying regularization or other constraints on model flexibility
improving validation methods to better represent the intended model application
Used in:
Contrast with:
Parameter#
A numerical quantity that describes a population or probability model.
Examples include,
population variance or standard deviation
distribution parameters, such as the mean and variance of a Gaussian distribution
variogram parameters, such as the nugget effect, range, and sill
Population parameters are generally unknown because the entire population is rarely observed. Instead, they are inferred from available sample statistics.
Used in:
TBA
Also see:
Contrast with:
Parametric Model#
Machine Learning Concepts: a model that makes an assumption about the functional form, shape of the natural system.
we gain simplicity and advantage of only a few parameters
for is a linear model we only have \(m+1\) model parameters
There is a risk that our model is quite different than the natural setting, resulting in a poor model, for example, a linear model applied to a nonlinear phenomenon.
Used in:
Constrast with:
Partial Correlation Coefficient#
Correlation analysis metric that quantifies the correlation between \(X\) and \(Y\) while controlling for the linear influence of other features, \(Z_1,\ldots,Z_{m-2}\), on both \(X\) and \(Y\).
The \(m-2\) notation accounts for removing the two features of interest, \(X\) and \(Y\), from the complete set of \(m\) features.
The partial correlation coefficient is written as,
and measures the remaining linear association between \(X\) and \(Y\) after removing the linear contribution of the control features.
To calculate the partial correlation coefficient, the following steps are applied,
Perform linear least-squares regression to predict \(X\) from the control features,
where \(X^*\) is the estimated value of \(X\) from the regression model.
Perform linear least-squares regression to predict \(Y\) from the control features,
where \(Y^*\) is the estimated value of \(Y\) from the regression model.
Calculate the residuals from the regression of \(X\),
Calculate the residuals from the regression of \(Y\),
Calculate the correlation coefficient between the residuals,
This correlation between residuals represents the partial correlation between \(X\) and \(Y\) after controlling for \(Z_1,\ldots,Z_{m-2}\).
Interpretation,
a partial correlation near 1 indicates a strong positive linear relationship remaining between \(X\) and \(Y\) after removing the influence of the control features
a partial correlation near -1 indicates a strong negative linear relationship remaining between \(X\) and \(Y\)
a partial correlation near 0 indicates little remaining linear relationship after controlling for the other features
Assumptions of the partial correlation coefficient include,
\(X,Y,Z_1,\ldots,Z_{m-2}\) have approximately linear relationships
no significant univariate or bivariate outliers, since partial correlation is sensitive to extreme values similar to regular correlation
for statistical inference, approximately Gaussian distributed variables and homoscedastic linear relationships provide the most reliable interpretation
Partial correlation removes only the linear influence of the control features. Nonlinear relationships between the variables are not captured by the standard partial correlation coefficient,
extensions based on nonlinear residualization models are possible, but these are no longer the standard partial correlation coefficient.
Used in:
Also see:
Partitional Clustering#
A family of clustering methods that divides a dataset into a specified number of non-overlapping groups, producing a single partition of the data.
Partitional clustering methods,
assign each sample to one cluster group
optimize a single clustering solution rather than producing multiple nested solutions
For example, k-Means clustering is a partitional clustering method that iteratively updates cluster assignments and cluster prototypes (centroids) while optimizing a single partition of the data.
Compared with hierarchical clustering,
partitional clustering produces one final clustering solution
hierarchical clustering produces a hierarchy of nested cluster solutions that can be viewed at different levels of grouping
Examples of partitional clustering methods include,
k-Means clustering - groups samples by minimizing the distance between samples and a specified number of cluster prototypes (centroids)
k-Medoid clustering - similar to k-Means clustering, but cluster prototypes are selected from actual samples
k-Modes clustering - extension of k-Means clustering for categorical data using category-based similarity measures
A helpful summary of clustering methods,
partitional clustering - find one best partition of the data
hierarchical clustering - build a tree of nested clusters
density-based clustering - identify regions of high sample density separated by lower-density regions
Used in:
Also see:
Contrast with:
Polygonal Declustering#
A declustering method to assign weights to spatial samples based on local sampling density, such that the weighted statistics are likely more representative of the population. Data weights are assigned so that,
samples in densely sampled areas receive less weight
samples in sparsely sampled areas receive more weight
Polygonal declustering proceeds as follows:
Split up the area of interest with Voronoi polygons. These are constructed by intersected perpendicular bisectors between adjacent data points. The polygons group the area of interest by nearest data point
Assign weight to each datum proportional to the area of the associated Voronoi polygon
where \(w(\bf{u}_j)\) is the weight for the \(j\) data. Note, the sum of the weights is \(n\); therefore, \(w(\bf{u}_j)\) is nominal weight of 1.0, sample density if the data were equally spaced over the area of interest.
Here are some highlights for polygonal declustering,
polygonal declustering is sensitive to the boundaries of the area of interest; therefore, the weights assigned to the data near the boundary of the area of interest may change radically as the area of interest is expanded or contracted
polygonal declustering is the same as the Theissen polygon method for calculation of precipitation averages developed by Afred H. Thiessen in 1911.
Also see:
Polynomial Regression#
A basis expansion method applied to predictor features before linear regression to introduce nonlinear relationships between predictors and response.
Polynomial regression transforms predictor features into polynomial basis features,
where the basis transforms over training data observations, \(i=1,\ldots,n\), are,
up to the specified polynomial order \(k\).
For example, with a single predictor feature, \(m=1\), and a fourth-order polynomial,
After applying the basis expansion, the model remains linear in the parameters and the analytical linear regression solution can still be applied.
Polynomial regression assumes,
the response is a linear combination of transformed basis features
the relationship between predictors and response can be approximated by a polynomial function
The model parameters now describe transformed predictor features rather than the original predictor features,
for example, interpretation of a coefficient associated with permeability\(^4\) is not straightforward
higher-order polynomial models often have increased model variance and may produce unstable interpolation and extrapolation
More on polynomial regression assumptions,
fixed predictor features - predictor features and their basis expansions are treated as known values rather than random variables
constant variance - response error variance is constant over the range of predictor feature values
linearity - response is a linear combination of the polynomial basis features
polynomial relationship - the underlying relationship between predictor and response can be represented by polynomial functions
independence of errors - response errors are uncorrelated with each other
no severe multicollinearity - polynomial basis features are not excessively redundant with each other
Extensions of polynomial regression may include interaction terms between predictor features, such as \(X_1X_2\), to represent coupled nonlinear relationships.
Used in:
Also see:
Population#
The complete set of values for a feature over the 2D area of interest or 3D volume of interest, represented at sufficient resolution to support decision making.
For example,
the exhaustive set of porosity values at every location within a reservoir
the exhaustive set of gold grades throughout an ore body
In practice, the entire population is rarely observed. Instead, a limited sample is collected and used to infer population parameters.
Used in:
Contrast with:
Posterior#
In Bayes’ theorem, the posterior probability represents updated knowledge or uncertainty about possible model assumptions, states, or parameters after incorporating new data. The posterior is calculated by combining the prior probability with the likelihood function describing the compatibility of observed data with possible models,
where:
\(P(A)\) is the prior probability representing uncertainty before observing new information,
\(P(B|A)\) is the likelihood function describing the compatibility of observations \(B\) with state or parameter \(A\),
\(P(B)\) is the evidence term used to normalize the posterior probability,
\(P(A|B)\) is the posterior probability representing updated uncertainty after incorporating observations.
Used in:
Also see:
Power Law Average#
A flexible family of averaging methods used to scale a feature from a smaller volume support, \(v\), to a larger support, \(V\), by calculating an effective value representative of the larger volume.
The power law average is,
where \(\omega\) is the averaging power.
Special cases include,
\(\omega = 1\) — arithmetic average
\(\omega = -1\) — harmonic average
\(\omega \rightarrow 0\) — geometric average (obtained in the limit)
The choice of averaging power depends on the physical process being modeled. For example, for permeability,
arithmetic averaging is appropriate for flow parallel to bedding
harmonic averaging is appropriate for flow perpendicular to bedding
near-geometric averaging is often appropriate for oblique flow directions
Used in: TBA - new scale and volume variance chapter
Precision#
Categorical classification prediction model performance metric that summarizes, for each category \(k\), the conditional probability that an observation truly belongs to category \(k\) given that the model predicts category \(k\).
“When the model predicts positive, how often is it correct?”
In other words,
the ratio of true positives (TP) to all observations predicted as category \(k\), i.e., true positives (TP) plus false positives (FP)
a summarization over the columns in a confusion matrix (truth on the y-axis and predicted categories on the x-axis)
where,
a precision of 1.0 indicates that every observation predicted as category \(k\) is actually category \(k\)
a low precision indicates that many observations predicted as category \(k\) are actually other categories (false positives)
Used in:
Also see:
Prediction#
Estimate unknown or future sample values given assumptions about, or a model of, the population. For example,
given a model of the reservoir, predict the porosity, permeability, or production rate at a proposed well location before drilling
given historical production data, predict next month’s production rate
statistical or data-driven prediction uses inferred relationships from data, while physics-based prediction uses governing equations and physical models
Prediction is concerned with estimating unknown values, rather than inferring the parameterss of the underlying population.
Used in:
Compare with:
Prediction Error#
The difference between an observed response feature value and the corresponding prediction from a predictive machine learning model.
For each training sample,
where \(y_i\) is the observed response feature value and \(\hat{y}_i\) is the model prediction.
Prediction errors are combined over all training data to evaluate model performance,
the collection of prediction errors forms the error vector
error norms summarize the prediction errors with a single value for model training and comparison
prediction errors are the basis for common loss functions, including mean absolute error (MAE), mean square error, and root mean square error
Prediction errors are analyzed to,
evaluate model accuracy
compare competing prediction models
detect systematic bias or trends in model performance
quantify predictive uncertainty
Prediction error should be evaluated separately for,
training data - to assess how well the model fits the available data
testing data - to estimate how well the model generalizes to previously unseen data
A well-performing prediction model,
has small prediction errors on both training and testing data
exhibits prediction errors that are approximately random, with no systematic trends relative to predictor features or predicted values
Used in:
Also see:
Prediction Interval#
An uncertainty range for a future prediction represented by lower and upper bounds based on a specified probability level, known as the confidence level.
For example, a 95% prediction interval may be communicated as,
given predictor feature values, \(X_1=x_1,\ldots,X_m=x_m\), there is a 95% probability that a future reservoir NTG observation will fall between 13% and 17%.
A prediction interval represents uncertainty in the next observation and integrates,
uncertainty in the estimated model prediction, \(\hat{Y}|X=x\)
irreducible variability or error in the response around the model prediction, \(Y-\hat{Y}\)
Therefore, prediction intervals are wider than confidence intervals for the estimated mean response because they include both model uncertainty and observation uncertainty.
The prediction interval depends on,
uncertainty in model parameters
uncertainty in the estimated conditional mean response
variability of observations around the conditional mean response
Used in:
Contrast with:
Predictor Feature#
A feature used as an input to predict a response feature in a predictive model.
A predictive machine learning model may be represented as,
where \(y\) is the response feature, \(x_1,\ldots,x_m\) are the predictor features, and \(\epsilon\) represents model error.
Additional comments,
predictor features are also commonly called input features or explanatory features
traditional statistical modeling often uses the term independent variable, although predictor feature is preferred because predictor features are not necessarily statistically independent
Used in:
Contrast with:
Predictor Feature Space#
The multiple variate space represented by the ranges and possible combinations of all features for our problem. The term is commonly truncated as feature space. Refers to the predictor features and does not include the response feature(s); therefore, it is,
all possible combinations of predictor features for which we need to make predictions
may be referred to as predictor feature space.
Typically, we train and test our machines’ predictions over the predictor feature space,
the space is typically a hypercuboid with each axis representing a predictor feature and extending from the minimum to maximum, over the range of each predictor feature
more complicated shapes of predictor feature space are possible, e.g., we could mask or remove subsets with poor data coverage.
Used in:
Also see:
Primary Data#
Data samples of the feature being modeled.
the target feature for a geostatistical model
the response feature for a predictive machine learning model
Primary data are the observations that the model directly estimates or simulates.
For example,
porosity measurements from core are used to build a 3D geostatistical porosity model, supported by a 2D seismic acoustic impedance map. The core porosity measurements are the primary data because porosity is the feature being modeled.
Used in:
TBD
Contrast with:
Principal Component Analysis#
A machine learning method in inferential statistics and information visualization for exploring and representing the structure of high-dimensional datasets.
Principal component analysis (PCA) is commonly applied in machine learning workflows for,
reducing linear redundancy between predictor features
visualization of high-dimensional datasets
Salient points of principal component analysis include,
orthogonal transformation - converts a set of correlated features into a set of linearly uncorrelated variables called principal components
variance maximization - identifies orthogonal directions that explain the greatest possible variability in the data
distance preservation - the full PCA transformation is a rotation and preserves pairwise distances; reduced PCA representations approximately preserve the original structure, although often a \(p\) subset is retained for dimensionality reduction resulting in distance projection error.
The number of principal components available is,
where \(n\) is the number of observations and \(m\) is the number of predictor features.
Principal components are ordered by the amount of variance explained,
the first principal component describes the largest possible variance in the dataset
each subsequent principal component describes the largest remaining variance while remaining orthogonal to previous components
additional components continue until the maximum number of available components is reached
PCA is based on eigenvalues and eigenvectors of the data covariance matrix.
The covariance matrix contains the pairwise covariance between all combinations of predictor features. Eigen decomposition of the covariance matrix provides,
eigenvalues - variance explained by each principal component
eigenvectors - directions or loadings that define each principal component
Comparison between principal component analysis and multidimensional scaling,
Principal Component Analysis (PCA) - operates on the covariance matrix (\(m\times m\)) of predictor features and finds orthogonal linear projections that maximize explained variance
Multidimensional Scaling (MDS) - operates on the pairwise distance matrix (\(n\times n\)) between samples and finds a low-dimensional representation that best preserves sample-to-sample dissimilarity
Benefits of working in a reduced-dimensional representation include,
reduced data storage and computational requirements
easier visualization of high-dimensional datasets
reduced multicollinearity between predictor features
Used in:
Also see:
Prior#
In Bayes’ theorem, the prior probability represents knowledge or uncertainty about possible model assumptions, states, or parameters before considering new data. The prior is combined with the likelihood function describing new observations to calculate the posterior probability,
where:
\(P(A)\) is the prior probability representing uncertainty before observing new information,
\(P(B|A)\) is the likelihood function describing the compatibility of observations \(B\) with state or parameter \(A\),
\(P(B)\) is the evidence term used to normalize the posterior probability,
\(P(A|B)\) is the posterior probability representing updated uncertainty after incorporating observations.
Used in:
Also see:
Probability Closure#
The normalization requirement of probability measures stating that the total probability over the entire sample space \(\Omega\) is equal to 1:
This property ensures that all possible outcomes collectively account for the entire probability mass.
Useful examples include:
Closure for complements:
Conditional complements:
Used in:
Also see:
Probability Constraints#
The fundamental requirements for valid measures of probability include,
Boundedness, probabilities must be between zero and one,
Closure, the total probability over the entire sample space, \(\Omega\), is one,
Null set, the probability of the empty set is zero,
Additivity, the probability of mutually exclusive events is the sum of their individual probabilities,
These constraints are closely related to the Kolmogorov probability axioms.
Used in:
Probability Density Function#
A representation of a continuous statistical distribution with a density function, \(f(x)\), describing the relative density over the range of possible feature values, \(x\).
A univariate probability density function is denoted by,
and a bivariate probability density function is denoted by,
and extends to multivariate distributions.
For example, the Gaussian probability density function is specified as,
with probability density function,
parameterized by average, \(\mu\), and variance, \(\sigma^2\).
These are requirements for a valid probability density function,
non-negativity constraint, the density cannot be negative,
the density value may be greater than 1.0 because density is not probability
integrate density over a range of \(x\) to calculate probability,
probability closure, the total area under the PDF curve is equal to 1.0,
Nonparametric PDFs are commonly calculated with kernels (usually a small Gaussian distribution) that are summed over all data. Therefore, there is an implicit scale (smoothness) parameter when calculating a PDF.
too large of kernels will smooth out important information about the univariate distribution
too narrow a kernel will result in an overly noisy PDF that is difficult to interpret
This is analogous to the choice of bin size for a histogram or normalized histogram.
Parametric PDFs require model fitting to the data. The steps are,
Select a parametric distribution, e.g., Gaussian, lognormal, etc.
Calculate the parameters for the parametric distribution based on available data, using methods such as least squares or maximum likelihood.
It is very common to use the acronym PDF for probability density function.
Used in:
Contrast with:
Probability Mass Function#
A function that describes the probability distribution of a discrete feature. The probability mass function assigns a probability to each possible discrete outcome,
with the requirements,
and probability closure,
The normalized histogram of a discrete feature is an empirical probability mass function.
Contrast with:
Probability Of Acceptance#
A probability used in stochastic sampling algorithms to determine whether a proposed sample or model realization is added to the generated sample.
The probability of acceptance is applied in methods such as,
Simulated annealing
In the Metropolis-Hastings algorithm, the acceptance probability is calculated as,
where \(\alpha\) represents the probability of accepting the proposed sample.
The acceptance rule is,
if \(\alpha \geq 1\), accept the proposed sample
if \(\alpha < 1\), conditionally accept the proposed sample by drawing,
and accepting the proposed sample if,
The stochastic acceptance step allows the Markov chain to explore lower-probability regions of the target distribution while still converging to the desired probability distribution.
Used in:
Also see:
Probability Operators#
A list of useful, common probability operators that are essential for working with probability and uncertainty problems.
Union of Events - the union of outcomes, the probability of \(A\) or \(B\) is calculated with the probability addition rule ,
\(\quad\) where the intersection probability is subtracted to avoid double counting outcomes common to both events.
Intersection of Events - the intersection of outcomes, the probability of \(A\) and \(B\) is represented as,
\(\quad\) Under the assumption of independence of \(A\) and \(B\), the intersection probability can be calculated from the marginal probabilities,
\(\quad\) If there is dependence between \(A\) and \(B\), then conditional probability is required,
Complementary Events - the NOT operator for probability. If we define event \(A\), then the complement \(A^c\) represents all outcomes that are not \(A\).
\(\quad\) The resulting closure relationship is,
\(\quad\) Complementary events extend naturally to conditional probabilities, for example,
\(\quad\) Note, the conditioning event must remain the same.
Mutually Exclusive Events - events that do not intersect and have no common outcomes. Using set notation,
\(\quad\) and the joint probability is,
Used in:
Also see:
Probability Perspectives#
The three primary perspectives for interpreting and calculating probability are:
Long-term frequencies - probability as the ratio of observed outcomes from repeated experiments. This perspective requires repeatable experiments and observations, and is the basis for frequentist probability.
Physical tendencies or propensities - probability based on knowledge of, or models for, the physical system. For example, the probability of a heads outcome from a coin toss can be known from the physical properties of the coin without performing repeated experiments.
Degrees of belief - probability representing our uncertainty about an outcome or proposition, allowing probabilities to be updated as new information becomes available. This perspective is the basis for Bayesian probability.
Used in: TBA
Production Data#
Spatiotemporal subsurface engineering data including bottom hole pressure, fluid production rates, fluid composition, and temperatures.
Production data are important dynamic observations used to evaluate and calibrate subsurface models.
Some additional comments,
production from a single well may be commingled over multiple producing intervals unless production logging tool (PLT) data are available to allocate production by interval
production data provide important ground truth for matching reservoir model forecasts through the model calibration process known as history matching
As model outputs from a flow simulation transfer function, production data integration requires,
an inversion approach known as history matching, which is challenging because the system is ill-posed and the solution is generally nonunique
Used in:
Proportions#
The proportion of each possible category relative to the total number of observations. Proportions describe the categorical distribution of a feature and are equivalent to the probability of occurrence when observations are considered representative.
For a sample, the proportion of category \(k\) is,
where \(n_k\) is the number of observations in category \(k\) and \(n\) is the total number of observations.
The proportions satisfy probability closure,
where \(K\) is the total number of categories.
Proportions are,
the y-axis of a categorical feature probability density function
the marginal probability’s for a categorical feature
commonly used to summarize lithology, facies, rock type, mineral class, land use, or any categorical variable
Facies proportions are central to geostatistical modeling because facies often define stationary domains with distinct statistical and spatial characteristics. For example,
sand — high porosity and permeability occurring in relatively large connected bodies
shale — low porosity and permeability occurring in drapes and thin, laterally continuous beds
The determination of facies proportions is one of the most important modeling decisions because they control,
expected volumetrics of each facies
connectivity and geological architecture
flow simulation and production forecasts
uncertainty in downstream decision making
Facies proportions may be estimated from,
well or drill-hole observations
interpreted seismic data
geological analogs
conceptual geological models
Because available data are sparse, the proportions themselves are uncertain. This uncertainty is commonly represented with multiple scenarios by varying the global proportions within plausible limits and evaluating the impact on the resulting subsurface models.
Also see:
Prototype#
In machine learning, prototype methods represent groups, clusters, or classes of data using a small set of representative vectors or exemplar observations called prototypes.
The general concepts of prototype methods include,
representation compression - reducing a dataset to a manageable set of representative vectors or samples,
where \(W\) represents the set of prototypes.
similarity-based assignment - observations are assigned to prototypes based on a similarity or distance measure, such as Euclidean, Manhattan, or cosine distance
Voronoi tessellation - for distance-based prototype methods, the feature space is divided into regions where each region contains the observations closest to one prototype
Examples include,
k-Means clustering - prototypes are cluster centroids calculated as the average location of all cluster members in feature space and are not necessarily actual samples
k-Medoid clustering - prototypes are selected as actual samples from the dataset that best represent each cluster
Prototype methods are useful for,
reducing complex datasets to representative examples
efficient similarity-based prediction and classification
interpretable representation of groups or clusters
Used in:
Also see:
Contrast with:
Qualitative Feature#
Feature described by labels rather than numerical quantities. Qualitative features represent information that requires interpretation or classification and the values do not have inherent numerical meaning.
typically qualitative features cannot be directly measured from rock, but instead require interpretation steps
Examples of qualitative features include,
rock type = sandstone
facies = channel sand, levee, floodplain
zonation = bornite-chalcopyrite-gold higher grade copper zone
Qualitative features may be encoded numerically for analysis, but the numerical codes represent categories and do not imply magnitude or order. For example,
sandstone = 1 and shale = 2 are category labels, not measurements where shale is greater than sandstone.
In geostatistics and machine learning, qualitative features are commonly transformed using approaches such as indicator transform or one-hot encoding before modeling.
Used in: TBD
Contrast with:
Quantitative Feature#
A feature that can be measured and represented by numerical values with meaningful magnitude.
Examples of quantitative features include,
age = 10 Ma (millions of years)
porosity = 0.134 (fraction of volume is void space)
saturation = 80.5% (volume percentage)
Similar to qualitative feature, quantitative features often require interpretation. For example,
total porosity may be directly measured, but effective porosity may require geological interpretation or a petrophysical model.
Quantitative features may be continuous or discrete depending on whether the possible values are measured along a continuum or occur as countable values.
Used in: TBD
Contrast with:
Query#
A component of the attention mechanism that allows a model to dynamically focus on the most relevant information while reducing the influence of less relevant information.
A Query (\(Q\)) is a representation of the information required for the current prediction or calculation.
The attention mechanism,
compares each Query with all Keys to calculate similarity scores, which are then used to derive attention weights.
Some additional comments,
each Key–Value pair represents a stored piece of information, where the Key describes the information and the Value contains the associated content.
Conceptually,
Query (\(Q\)) asks, “What information do I need right now?”
Used in:
Also see:
R Squared#
Stated as coeficient of determination, “R-squared” or in mathatical notations as \(R^2\), a statistical measure of the proportion of the variability in a response feature that is explained by a regression model.
For ordinary least-squares linear regression with an intercept, the response feature variance can be partitioned into,
where,
\(\sigma^2_{tot}\) is the variance of the response feature, \(y_i\)
\(\sigma^2_{reg}\) is the variance explained by the regression model
\(\sigma^2_{res}\) is the residual variance, variance of prediction error
The coefficient of determination is therefore,
where,
\(R^2=0\) indicates the regression explains none of the response variability
\(R^2=1\) indicates the regression explains all of the response variability
For ordinary least-squares linear regression with a single predictor feature,
where \(\rho_{x,y}\) is the correlation coefficient between the predictor and response.
Some additional comments,
for many nonlinear regression models an \(R^2\) statistic can still be calculated, but it should be interpreted as a relative reduction in squared prediction error rather than a strict variance decomposition
\(R^2\) does not indicate whether predictions are unbiased or whether the model generalizes well to unseen data; prediction error metrics such as mean square error, root mean square error, or mean absolute error are required to evaluate predictive performance.
Used in:
Also see:
Random Function#
A set of random variables correlated over space or time. In geostatistics, a random function provides the mathematical framework for representing spatial uncertainty and variability.
The key concept introduced by Matheron is that a geological phenomenon is viewed as a realization of an underlying random function. The observed spatial data represent one possible outcome from this random process.
Important points on random function nomenclature,
random variables are denoted with upper-case, e.g., \(X\)
random functions are denoted with upper-case with location vectors, e.g.,
joint outcomes called realizations, or data samples are represented with lower case, e.g.,
realizations with the \(\ell\) notation, e.g.,
for \(\ell = 1,\ldots,L\) realizations.
Also see:
Random Forest#
Ensemble prediction model based on,
decision tree - each estimator in the ensemble is a decision tree
model bagging - each tree is trained on a bootstrap realization of the training data and predictions are aggregated across the ensemble
tree decorrelation - only a random subset of the \(m\) available predictor features is considered at each decision tree split, \(p < m\)
The reduction in model variance by ensemble estimation (averaging for regression) may be represented by the standard error of the mean,
where \(\sigma_s^2\) is the variance of an individual estimator and \(n\) is the number of estimators in the ensemble. This result assumes that the estimators are independent (uncorrelated).
In general, the variance of the ensemble prediction depends on the correlation between the decision trees,
where \(\rho\) is the average pairwise correlation between trees.
This equation explains the motivation for Random Forest,
if \(\rho=0\), the trees are uncorrelated and
providing the maximum possible reduction in model variance
if \(\rho=1\), the trees are perfectly correlated and
so averaging provides no reduction in model variance
One issue with tree model bagging is that decision trees in the ensemble may become highly correlated,
when one predictor feature is dominant, it is repeatedly selected for the upper tree splits
consequently many trees become very similar and averaging provides much less reduction in model variance than expected
By restricting the candidate predictor features considered at each split,
each tree evolves differently, reducing the correlation between trees
lower tree correlation produces greater reduction in ensemble prediction variance
Common default values for the number of candidate predictor features include,
for classification, and
for regression.
If \(p=m\), every predictor feature is considered at every split and Random Forest reduces to tree bagging.
Used in:
Also see:
Random Sample#
A representative spatial sampling method from a:
population – sample for statistical inference
data set – bootstrap sample generated by sampling with replacement
model – stochastic realization
Random sampling requires that:
every possible sample of the specified size is equally likely to be selected
each selection is made randomly and is not influenced by previous selections or outcomes
This minimizes selection bias and supports valid statistical inference.
Used in:
Random Variable#
A mathematical representation of uncertainty where the value of a feature is unknown and can take a range of possible outcomes described by a statistical distribution, probability density function, or cumulative distribution function.
A random variable is denoted with upper-case notation, e.g., \(X\), while possible outcomes or observed values are represented with lower-case notation, e.g., \(x_{\alpha}\) or realization \(x^{\ell}\).
For spatial phenomena, a location vector, \(\mathbf{u}\), is added to represent the random variable at a specific location,
spatial random variable:
spatial data measure:
spatial realization:
The collection of correlated spatial random variables over many locations forms a random function.
Used in:
Also see:
Realization#
An outcome from a random variable or a joint outcome from a random function.
an outcome from a random variable, \(X\), or a joint set of outcomes from a random function
represented with lower case notation, e.g., \(x\)
for spatial settings it is common to include a location vector, \(\mathbf{u}\), to describe the location, e.g., \(x(\mathbf{u})\), corresponding to the random variable \(X(\mathbf{u})\)
generated by simulation methods, e.g., Monte Carlo simulation, geostistical sequential Gaussian simulation, or any other method that samples jointly from a random function
in general, stochastic simulation assumes realizations are equiprobable, meaning each realization is considered an equally likely outcome of the modeled uncertainty
Used in:
Also see:
Realizations#
A realization ensemble of spatial models generated by stochastic simulation by holding input parameters and model choices constant while changing only the random number seed.
realizations explore the spatial offset from data component of uncertainty modeling
A realizations represents spatial uncertainty by sampling multiple possible outcomes from the same random function.
For example,
hold the porosity average, variogram model, conditioning data, and simulation parameters constant
generate multiple porosity models by changing only the random number seed
differences between the realizations represent spatial uncertainty in porosity away from conditioning data
Contrast with:
Reasons to Learn Some Coding#
Professor Pyrcz’s reasons why every scientist and engineer should learn some coding,
Transparency – no compiler accepts hand waving! Coding exposes every assumption and every step of your logic for others to review.
Reproducibility – run it and get an answer. Share it with a colleague and they should obtain the same answer. Reproducibility is a cornerstone of the scientific method.
Quantification – computers require numbers. Coding encourages us to move from qualitative descriptions to quantitative analysis and often reveals new ways to understand a problem.
Open Source – leverage a world of brilliance. Thousands of scientists and engineers freely share software, algorithms, and ideas for everyone to build upon.
Break Down Barriers – don’t throw your work over the fence. Work directly with software developers and contribute your subject matter expertise to build better tools.
Deployment – share your code with others and multiply your impact. Whether measured by performance metrics or simply helping others, one script can benefit thousands of people.
Efficiency – automate repetitive tasks. Build reusable workflows so you spend less time repeating work and more time solving scientific and engineering problems.
Always Time to Do it Again! – many tasks are performed repeatedly. Although scripting and automation often take longer the first time, the investment usually pays for itself many times over.
Think Like a Programmer – learning to code changes the way you approach problems. Rather than being limited by existing software, you begin designing your own solutions.
Recall#
Categorical classification prediction model performance metric that summarizes, for each category \(k\), the fraction of observations belonging to category \(k\) that are correctly identified by the prediction model.
“Of all the actual positives, how many did the model find?”
In other words,
the ratio of true positives (TP) to all actual observations in category \(k\), i.e., true positives (TP) plus false negatives (FN)
a summarization over the rows in a confusion matrix (truth on y-axis and predicticted categories on x-axis)
where,
a recall of 1.0 indicates that every observation belonging to category \(k\) is correctly identified
a low recall indicates that many observations of category \(k\) are missed (false negatives)
Used in:
Also see:
Recursive Feature Elimination#
Model-based feature ranking workflow that recursively removes the least important predictor feature until all features have been ranked.
The general workflow is,
Train a prediction model using the current set of predictor features and calculate a feature ranking metric, such as feature importance.
Remove the lowest-ranked predictor feature.
Repeat Steps 1 and 2 until only one predictor feature remains.
The final feature ranks, \(1,\ldots,m\), are obtained from the reverse order of elimination,
last remaining feature is the most important
second last remaining feature is the second most important
\(\vdots\)
second feature removed is the second least important
first feature removed is the least important
Benefits of recursive feature elimination include,
redundant predictor features - removing one member of a redundant feature group often increases the apparent importance of the remaining members, reducing the chance that an entire redundant group is eliminated
computational efficiency - each successive model is trained with fewer predictor features, reducing the computational cost of later iterations
improved feature ranking - repeatedly retraining the model allows feature importance to adapt as redundant and noisy predictor features are removed
Limitations of recursive feature elimination include,
computational cost - requires training approximately \(m\) prediction models for \(m\) predictor features
model dependent - feature rankings are only as reliable as the underlying prediction model and its feature importance metric
no automatic stopping criterion - produces a complete feature ranking, but does not determine the optimal number of predictor features to retain
Used in:
Also see:
Recurrent Neural Network#
A neural network architecture designed for sequential data, such as sentences, audio, signals, or time-series data, where previous information influences predictions at subsequent sequence positions.
Unlike standard feed-forward neural networks,
recurrent neural networks maintain a hidden state that stores information from previous sequence positions, introducing recursion into the network
predictions are made sequentially, with information from earlier steps passed forward to influence later predictions
The fundamental concept of recurrent neural networks is,
for sequential data, the context from previous observations should influence the current prediction
Instead of treating each observation as independent,
the network processes observations in sequence
a recurrent weighted connection passes information from the previous hidden state into the current hidden state
the hidden state acts as a learned memory of previous information
For a sequence position \(t\),
where \(x_t\) is the current input, \(h_t\) is the current hidden state, \(h_{t-1}\) is the previous hidden state, and \(W_x\), \(W_h\), and \(b\) are trainable network parameters.
Training recurrent neural networks,
network weights and biases must account for the influence of each parameter over all subsequent sequence positions
training is performed using backpropagation through time (BPTT), where gradients are propagated through the recurrent connections over the sequence
Recurrent neural networks are commonly applied for,
natural language processing (NLP) - text generation, translation, and language modeling
speech recognition - converting audio sequences into text
time-series analysis - forecasting signals and detecting temporal patterns
healthcare - monitoring sequential patient measurements and detecting anomalies
Variants of recurrent neural networks include,
long short-term memory (LSTM) - introduces memory gates to reduce vanishing gradients and retain important long-term information
gated recurrent unit (GRU) - simplified gated architecture with improved computational efficiency
bidirectional recurrent neural network (BRNN) - processes sequences in both forward and backward directions to incorporate preceding and future context
Training recurrent neural networks,
network weights and biases are estimated using backpropagation through time and gradient-based optimization
training is more computationally complex than feed-forward neural networks due to the repeated application of the network over sequence positions
Used in:
Also see:
Recursion#
From Wikipedia recursion, “recursion occurs when the definition of a concept or process depends on a simpler or previous version of itself.”
A recursive process applies the same operation repeatedly to a simpler version of the problem until reaching a stopping condition, known as the base case.
A simple example of recursion is calculating a factorial,
which can also be written recursively as,
with the base case,
A Python implementation of the recursive factorial calculation,
def factorial(n):
if n == 0 or n == 1:
return 1
else:
return n * factorial(n-1)
where the function calls itself with a simpler input, \(n-1\), until reaching the base case.
Other examples of recursion include,
Fibonacci sequence - each number is calculated from the two preceding numbers,
def fibonacci(n):
if n == 0:
return 0
if n == 1:
return 1
else:
return fibonacci(n-1)+fibonacci(n-2)
fractal geometry - complex geometric structures generated by repeatedly applying the same construction rule at smaller scales, for example the Sierpiński triangle
In programming, recursion is useful when a problem can naturally be divided into smaller versions of itself. However, recursive solutions may require additional computational memory and may not always be the most efficient implementation.
Used in:
Also see:
Regression#
A supervised machine learning method that predicts a continuous response feature from one or more predictor features.
Regression learns a relationship between predictor features and a response feature that is a continuous feature,
given predictor features, \(X_1,\ldots,X_m\), the model estimates a continuous response value, \(\hat{Y}\)
model predictions are evaluated by comparing predicted values, \(\hat{Y}\), with observed response values, \(Y\)
Common regression methods include,
regression decision trees and neural networks
Used in:
Regression with Linear Regression Description and Demonstration
Regression with Ridge Regression Description and Demonstration
Regression with LASSO Regression Description and Demonstration
Contrast with:
Regular Sampling#
A representative spatial sampling method from a:
population – observations collected at fixed intervals
data set – every kth observation
model – values extracted on a regular grid of predictor features
Regular sampling:
collects samples at fixed intervals in space, time, or sequence
provides uniform coverage of the population or domain
may introduce bias if the sampling interval aligns with periodic patterns in the population, data set, or model
Used in:
Also see:
Representative Spatial Sampling#
The sample and resulting sample statistics are representative of the population, by sampling theory we have 2 options:
Random sampling - each potential sample from the population is equally likely to be sampled as samples are collected. This includes,
selecting a specific location has no impact on the selection of subsequent locations.
assumption that the population size that is much larger than the sample size; therefore, significant correlation between samples is not imposed due to without replacement sampling (the constraint that you can only sample a location once). Note, generally this is not an issue for the subsurface due to the sparsely sampled massive populations
Regular Sample - sampling at equal space or time intervals. While random sampling is prefered, regular sampling is robust as long as,
the regular sampling intervals do not align with natural periodicity in the data, e.g., the crests are systemally sampling resulting in biased high sample statistics
Used in:
Contrast with:
Residual Sum of Squares#
The sum of the squared prediction errors over all observations, commonly used as an objective function for regression.
The residual sum of squares is,
where \(y_i\) is the observed response feature value and \(\hat{y}_i\) is the corresponding model prediction.
Residual sum of squares,
measures the total squared prediction error over all observations
is minimized during model parameter training for ordinary least squares linear regression
forms the basis for several regression error metrics, including the mean squared error
Used in:
Also see:
Response Feature#
The output or target feature for a predictive machine learning model. A predictive machine learning model can be generalized as,
where the response feature is \(y\), the predictor features are \(x_1,\ldots,x_m\), and \(\epsilon\) represents model error or unexplained variability.
The response feature is the quantity that the predictive model attempts to estimate or predict.
traditional statistical modeling uses the term “dependent variable” instead of response feature
Used in:
Contrast with:
Ridge Regression#
A regularized version of linear regression with the same prediction equation,
where the trainable model parameters are \(b_{\alpha}\), the feature weights, and \(b_0\), the constant intercept term.
The model parameters are estimated by minimizing a regularized least squares loss function that combines the residual sum of squares (RSS) with a shrinkage penalty,
where \(y_i\) is the observed response feature value and \(\sum_{\alpha=1}^{m}b_{\alpha}x_{\alpha,i}+b_0\) is the model prediction for training sample \(i\).
Ridge regression introduces the hyperparameter \(\lambda\), which controls the strength of the L2 regularization penalty,
where,
larger values of \(\lambda\) increase the penalty on large slope coefficients
the intercept term \(b_0\) is not penalized because it represents the response mean rather than predictor influence
Ridge regression integrates two competing goals during model training,
minimize prediction error with respect to the training data
minimize the magnitude of the slope parameters toward zero
The hyperparameter \(\lambda\) controls the model bias–variance trade-off,
as \(\lambda\rightarrow0\), the solution approaches ordinary linear regression with no additional regularization bias, but potentially higher model variance
as \(\lambda\) increases, model variance decreases and model bias increases as the model becomes less flexible
as \(\lambda\rightarrow\infty\), the slope parameters \(b_1,\ldots,b_m\) approach zero and predictions approach the response feature mean
Ridge regression is also known as Tikhonov regularization.
The assumptions of ridge regression include,
Error-free predictors - predictor features are treated as known values, not random variables
Linearity - the expected response is a linear combination of predictor features
Constant variance - the variance of response error is constant over predictor feature values (homoscedasticity)
Independence of error - errors in the response are uncorrelated with each other
No multicollinearity - predictor features are not linearly redundant with each other
Used in:
Also see:
Root Mean Square Error#
Prediction model performance metric calculated as the square root of the mean square error over all observations,
where \(y_i\) is the observed response feature value, \(\hat{y}_i\) is the model prediction, and \(n\) is the number of observations.
Root mean square error has several important properties,
all prediction errors contribute to the metric
larger prediction errors are penalized more strongly than smaller errors because the errors are squared before averaging
always non-negative, with \(RMSE=0\) indicating perfect predictions
reported in the same units as the response feature
Root mean square error is commonly applied,
to summarize regression model prediction accuracy
to compare competing regression models on the same testing dataset
when prediction error should be interpreted in the original response feature units
Compared with other error metrics,
mean square error - reports the average squared prediction error and is commonly used as a loss function
mean absolute error- is more robust to outliers because prediction errors are not squared
Advantages of root mean square error include,
easily interpreted because it has the same units as the response feature
emphasizes larger prediction errors while remaining directly comparable to the response variable
Limitations of root mean square error include,
sensitive to outliers because large prediction errors are squared before averaging
should not be compared between response features with different units or scales without normalization
provides no indication whether prediction errors are biased or randomly distributed
Used in:
Also see:
Sample#
A subset of values and locations measured from a population and used to infer parameter(s) of the population.
Examples include,
sparse spatial samples - 1,000 porosity measures from well log data in a reservoir with high measurement precision, but very small volume support and limited spatial coverage.
dense spatial samples - 1,000,000 acoustic impedance measurements over a 1,000 x 1,000 2D grid for a reservoir unit of interest with lower measurement precision and larger volume support.
In spatial modeling, the information content of a sample depends not only on the number of measurements, but also on the spatial distribution, measurement precision, and volume support.
also the amount of information in the sample set may be related to data locations and spatial continuity, i.e., the degree of redundancy between the samples.
The above states, sample is the collection of extracted values or observation, but there are two other important uses of the term sample,
verb - the act of extracting one or more values from a population, data set, or model, i.e., collecting data or realizations
singular noun - a single extracted value or observation
Used in:
Contrast with:
Scatter Plot#
A common data visualization plot that displays paired observations of two features as points to visualize their relationship, dependence, trends, clusters, and outliers.
For paired samples of two features, each observation is represented as a point,
where one feature is plotted on the x-axis and the other feature is plotted on the y-axis.
Scatter plots are used to visually assess,
relationship and dependence between features
trends and nonlinear patterns
clusters and populations within the data
outliers and anomalous observations
Note that association observed in a scatter plot does not necessarily imply causation.
Used in:
Compare with:
Scenarios#
Multiple subsurface models calculated by changing input parameters, assumptions, or modeling choices to represent uncertainty due to incomplete knowledge of the system.
Examples of scenario uncertainty include,
changing the input feature distributions, for example, modeling low, mid, and high porosity mean scenarios and generating subsurface models from each distribution
changing geological interpretations, such as alternative facies proportions, structural interpretations, or depositional models
changing model parameters, such as variogram parameters, trend models, or spatial continuity assumptions
Each scenario may include an ensemble of realizations generated by varying the random number seed in stochastic simulation to represent spatial uncertainty within that scenario.
Used in:
Contrast with:
Secondary Data#
Data samples of a feature other than the feature being modeled, used to improve estimation or simulation of the primary feature.
Secondary data are integrated through a model of the relationship between the secondary and primary data.
For example,
acoustic impedance measurements from seismic data (secondary data) are used to support calculation of a 3D porosity model, where porosity is the feature of interest
porosity measurements (secondary data) are used to support calculation of a permeability model, where permeability is the feature of interest
Secondary data may provide additional spatial information, trends, or constraints, but are not the direct observations of the feature being modeled.
Used in:
TBD
Contrast with:
Seismic#
A geophysical measurement technique that uses controlled acoustic sources and receivers to measure subsurface reflections and infer geological structure and rock properties.
Reflection seismic data provide high spatial coverage but generally lower resolution compared with direct measurements such as well log and core data.
Some important details include,
seismic reflection amplitudes are processed and inverted to estimate rock properties, such as acoustic impedance, calibrated and positionally aligned with well sonic logs
seismic provides a geological framework by identifying bounding surfaces, structural features, and reservoir extents
seismic provides soft information for reservoir properties, such as porosity and facies, through relationships established between seismic attributes and available primary data
In geostatistical modeling, seismic is commonly used as secondary data to improve spatial prediction and uncertainty models.
Used in:
TBD
Shapley Value#
A local measure of feature importance from explainable machine learning, borrowed from cooperative game theory. In cooperative game theory, Shapley values are used to,
partition the winnings among players
assign the contribution of each player to the outcome of the game
The motivation for Shapley values is that many predictive machine learning models are highly accurate but difficult to interpret. Two general approaches are available to improve interpretability,
reduce model complexity
apply model-agnostic interpretation methods, such as Shapley values
To apply Shapley values to predictive machine learning,
players \(\leftarrow\) predictor features
game \(\leftarrow\) prediction model
winnings \(\leftarrow\) prediction above (or below) the average response
For an individual prediction, the Shapley value of each predictor feature is calculated as its average marginal contribution over all possible combinations (coalitions) of the remaining predictor features.
The prediction is decomposed into,
where \(\bar{y}\) is the average response feature value and \(\phi_j\) is the Shapley value for predictor feature \(j\).
Therefore,
positive Shapley values push the prediction above the response feature average
negative Shapley values pull the prediction below the response feature average
the Shapley values sum exactly to the prediction minus the response feature average
For feature ranking, a global measure of feature importance is obtained by averaging the magnitude (absolute value) of the Shapley values over many predictions,
larger average absolute Shapley values indicate greater overall feature importance
Shapley value summary,
model agnostic - applicable to virtually any predictive machine learning model
local explanation - explains an individual prediction
global feature importance - obtained by summarizing local Shapley values over many predictions
units are the same as the response feature
In addition to model explainability, Shapley values are used for,
feature ranking based on the average absolute Shapley per predictor feature over various predictions
Used in:
Also see:
Simpson’s Paradox#
A statistical phenomenon where a trend observed within individual groups reverses or disappears when the groups are combined into a single dataset.
For example,
each group may have a negative correlation between two features, while the combined dataset has a positive correlation
conversely, each group may show a positive relationship while the combined dataset shows a negative relationship
Simpson’s paradox occurs because,
a confounding feature influences both the grouping of the data and the relationship between the variables of interest
aggregating over the confounding feature changes the weighting of observations and can produce a misleading overall trend
As a result,
exploratory data analysis should examine both the complete dataset and meaningful subgroups
apparent relationships in aggregated data should be interpreted with caution when confounding variables may be present
Used in:
Also see:
Simulation#
A stochastic process of obtaining one or more possible values of a feature at unsampled locations that are consistent with available data and a multivariate, temporal and spatial uncertainty model.
Simulation models are designed to reproduce global characteristics and spatial variability, known as global accuracy, where the model reproduces specified global measures, including,
feature univariate distributions, including proportions, mean, variance, and the complete cumulative distribution function
feature spatial models, including the variogram, training image, or geological object geometries
Unlike estimation methods, simulation produces multiple equiprobable realizations that represent uncertainty rather than a single optimal prediction.
Examples of simulation models include,
geostatistical subsurface heterogeneity models, including sequential Gaussian simulation, sequential indicator simulation, multiple point simulation, and object-based simulation
uncertainty propagation through a transfer function, including Monte Carlo simulation
Use simulation when,
reproducing feature distributions is important, especially when extreme values influence decisions
realistic spatial models are required for applications such as flow simulation
uncertainty in the decision criteria must be quantified through multiple possible models
Used in:
Contrast with:
Soft Data#
Data with significant uncertainty such that the information is represented probabilistically and uncertainty must be integrated into the spatial model.
For example,
a local porosity probability density function calibrated from acoustic impedance measurements
a probability of facies’s occurrence interpreted from seismic data
Soft data integration requires workflows that incorporate uncertainty in the conditioning information, such as,
indicator kriging
sequential indicator simulation
p-field simulation
workflows that randomize or transform soft information into data realizations compatible with simulation methods that traditionally assume hard data, such as sequential Gaussian simulation
Soft data integration is an advanced topic and an active area of research; however, many standard subsurface modeling workflows and commercial software packages include approaches for incorporating soft information.
Used in:
TBD
Contrast with:
Spatial Estimation#
The process of obtaining a single best value to represent a feature at an unsampled location or time, \(\bf{u}\).
Given spatial data, \(z(\bf{u}_1), \dots, z(\bf{u}_n)\), we estimate the unknown feature value at location \(\bf{u}\) with a linear combination of the available data,
An unbiasedness constraint may be added by assigning the remainder of the weight (one minus the sum of weights) to the global average. Therefore, if no informative data are available, the estimate approaches the global average of the feature,
Some additional concepts,
local accuracy takes precedence over global accuracy, meaning that spatial estimation methods prioritize matching nearby observations over reproducing global statistics such as the histogram and variogram
spatial estimation maps and models generally have reduced variance and increased spatial continuity, resulting in smoother models than the true heterogeneous feature distribution
estimation models are not appropriate for transfer functions that are sensitive to heterogeneity and feature distributions, such as flow simulation or economic optimization
spatial estimation produces a single deterministic model and therefore does not provide multiple realizations required to sample uncertainty in the decision criteria; simulation methods are required for comprehensive uncertainty modeling and decision support
Examples of spatial estimation methods include,
inverse distance
There are also general non-spatial estimation methods; for example, many predictive machine learning models perform estimation by focusing on local predictive accuracy rather than global distribution reproduction,
Contrast with:
Spatial Sample Selection#
The process of selecting locations for collecting subsurface samples to reduce uncertainty and support resource development decisions.
For subsurface resource exploration and development, sample locations are selected to achieve two primary objectives:
Reduce uncertainty - by collecting information to answer key geological and engineering questions, for example,
how far does the contaminant plume extend? – sample the plume periphery to define its extent
where is the fault? – collect data guided by seismic interpretation and geological hypotheses
where are the highest mineral grades? – sample areas with potential economic significance
how far does the reservoir extend? – offset drilling to define reservoir boundaries
Maximize net present value - by collecting information while advancing development objectives, for example,
maximize production rates
maximize recoverable resource or mineral tonnage
Therefore, subsurface samples are often collected for dual purposes: reducing uncertainty and supporting development. For example,
exploration and appraisal wells provide geological and reservoir information that can subsequently be incorporated into the production system
production wells provide operational data while also becoming valuable conditioning data for future reservoir models
Used in:
Spectral Clustering#
Spectral Clustering: a partitional clustering method that utilizes the spectrum, eigenvalues and eigenvectors, of a matrix that represents the pairwise relationships between the data.
dimensionality reduction from data samples pairwise relationships characterized by the graph Laplacian matrix
eigenvalues, eigenvectors are equivalent to principal component analysis dimensionality reduction by linear, orthogonal feature projection and rotation to best describe the variance
Advantages of spectral clustering,
the ability to encode pairwise relationships, integrate expert knowledge.
eigenvalues provide useful information on the number of clusters, based on the degree of ‘cutting’ required to make k clusters
lower dimensional representation for the sample data pairwise relationships
the resulting eigenvalues and eigenvectors can be interpreted, eigenvalues describe the amount of connection for each number of groups and eigenvectors are grouped to form the clusters
Used in:
Standard Deviation#
The square root of the variance. Standard deviation measures the spread of a feature about its average in the same units as the original feature.
Given the sample variance,
the sample standard deviation is,
The equivalent population parameters are, the population variance,
the population standard deviation is,
Used in:
TBD
Also see:
Standardization#
Distribution rescaling that can be thought of as shifting, and stretching or squeezing of a univariate distribution (e.g., histogram) to a mean of 0.0 and a standard deviation of 1.0.
a special case of an affine transformation
For each data value, \(x_i\), the following transformation maps it to the standardized value, \(y_i\),
where \(\overline{x}\) is the original mean and \(\sigma_x\) is the original standard deviation.
This transformation is linear and therefore,
preserves the rank ordering of observations
does not change the shape of the distribution
changes only the location and scale of the feature values
Standardization is useful when feature magnitude influences model behavior, for example,
distance-based methods where predictor features with larger variance can dominate distance calculations, such as k-means clustering clustering and k-Nearest Neighbours
models where feature coefficients are compared for interpretation or feature ranking, such as linear regression
artificial neural networks where input scaling improves numerical optimization and reduces activation function saturation
Standardization is commonly applied to predictor features and may also be applied to response features. Predictions are then back-transformed to the original units for interpretation.
Used in:
Also see:
Stationarity#
The decision that a subset of the subsurface is the same “stuff” and therefore can be pooled to calculate statistics and build models.
Replicates are required to calculate any statistic. In many applications, replicates are obtained by repeated measurements through time, for example,
air or water samples collected repeatedly from a monitoring station
For subsurface resource models,
repeated samples are generally not available at the same location; only one sample is available at each location
instead of pooling measurements through time, we must pool samples over space to calculate statistics
Why must we pool data? Ultimately, it is required to make inference about the population from a limited sample,
to calculate statistics
to build spatial models
The choice of stationary domain is an expert geological decision. Without a stationarity decision, we are restricted to the measured locations (well bores or drill holes) and cannot calculate statistics or make predictions between samples.
An example geological definition of stationarity could be:
The rock within the stationary domain is sourced, deposited, preserved, and post-depositionally altered in a similar manner. The domain is mappable and may be used for local prediction or as information for analogous locations within the subsurface; therefore, information may be pooled over this expert-defined volume of the subsurface.
This expert geological interpretation defines a domain over which statistical stationarity is assumed for modeling.
There are two aspects of any stationarity decision:
Import license - the choice of which samples are allowed to contribute to the calculation of a statistic
Export license - the choice of where the resulting statistic is applicable within the subsurface
To state a stationarity decision, we must specify:
the statistic assumed stationary, for example, the mean, variance, cumulative distribution function , or spatial continuity
the spatial domain over which the statistic is assumed stationary, for example, the entire model, a facies, a depositional environment, or a geological region
Examples of statistical definitions of stationarity include:
stationary mean
stationary cumulative distribution function
stationary semivariogram
The stationarity decision may be extended to any statistic of interest, including,
bivariate distributions
multiple point statistics
Additional considerations for stationarity include:
Stationarity is a decision, not a hypothesis - therefore, it is not directly tested. Instead, data may demonstrate that a chosen stationarity decision is inappropriate.
Stationarity depends on scale - the appropriate modeling scale should be selected based on the geological process, decision objective, and project requirements.
A stationarity decision cannot be avoided - without stationarity, spatial statistics cannot be calculated and modeling cannot progress beyond measured locations. Conversely, assuming broad stationarity over very large regions of the Earth is generally unrealistic.
Geomodeling stationarity is a domain decision - defining (1) where data may be pooled (import license) and (2) where resulting statistics may be applied (export license).
Nonstationary trends may be modeled explicitly - deterministic trends can be removed and the remaining stationary residual variation can be modeled stochastically. This is the hybrid modeling approach.
Statistic#
A function of sample data that summarizes a property of the sample. Examples include,
sample mean - \(\overline{x}\)
sample standard deviation - \(s\)
sample variance - \(s^2\)
cumulative distribution function - \(F_x(x)\)
experimental variogram - \(\gamma(\mathbf{h})\)
Statistics are calculated from available samples because the complete population is generally unknown.
How do we use statistics?
Inference - calculate statistics from a sample and use them to estimate unknown population parameters
Prediction - use a model of population parameters and relationships to predict future observations or outcomes
Used in:
Compare with:
Statistical Distribution#
A description of the frequency or probability behavior of a feature over the range of possible values.
A univariate statistical distribution describes how feature values are distributed without considering their spatial or temporal arrangement. We represent the statistical distribution with,
What do we learn from a statistical distribution? For example,
what are the minimum and maximum values?
what is the most common range of values?
do we have many low values?
do we have many high values?
are there outliers or values that do not make geological or physical sense and require explanation?
what is the variability and uncertainty in the feature values?
Statistical distributions are fundamental for inference, simulation, and uncertainty modeling.
Used in:
Statistics#
The theory and practice for collecting, organizing, and interpreting data, as well as drawing conclusions and making decisions.
Used in: Entire book
Same as:
Stochastic Gradient-based Optimization#
Optimization method commonly applied in machine learning to estimate model parameters by iteratively minimizing a loss function.
Compared with full gradient-based optimization, stochastic gradient-based optimization improves computational efficiency by calculating gradients from random batches of the training data rather than the complete dataset.
a batch is a random subset of the training data with size \(n_{batch}\)
the batch provides a stochastic approximation of the full loss function gradient
each optimization step is less accurate than using the complete dataset, but much faster to calculate
increasing \(n_{batch}\) improves gradient accuracy while decreasing stochasticity
decreasing \(n_{batch}\) increases stochasticity and reduces computational cost per optimization step
The general workflow is,
initialize the model parameters, \(\mathbf{b}\)
randomly select a batch of training data
calculate the loss function and its gradient over the batch,
update the model parameters,
where \(r\) is the learning rate (step size).
This process is repeated until a stopping criterion is satisfied, such as,
maximum number of optimization iterations
sufficiently small change in the loss function
sufficiently small parameter updates
Common extentions include,
momentum - to dampen oscillation and between track large scale gradients in the loss function
adaptive learning rates - with methods such as AdaGrad, RMSProp, and Adam
Stochastic Model#
A model of a system or process that includes uncertainty and is represented by multiple possible outcomes, including realizations and scenarios, constrained by available data, statistics, and modeling assumptions.
Stochastic models represent uncertainty by describing a range of plausible outcomes rather than a single deterministic prediction.
Examples include,
data-driven models that integrate uncertainty, such as geostatistical simulation models
Monte Carlo models that propagate uncertainty through a transfer function
ensemble machine learning models that represent prediction uncertainty
Advantages:
computational speed compared with many physics-based models
explicit uncertainty assessment
ability to report confidence Intervals, prediction intervals, and risk measures
ability to integrate many sources of data and information
flexible data-driven approaches
Disadvantages:
limited representation of underlying physics unless explicitly incorporated
dependence on statistical model assumptions and simplifications
uncertainty models may be incomplete if important processes or information are not represented
Contrast with:
Stratified K-fold Cross Validation#
A K-fold Cross Validation-based cross validation method that preserves the class proportions of the response feature within each fold.
Stratification provides more representative training and testing subsets for classification problems, particularly when classes are imbalanced.
Contrast with:
Subsurface Modeling Workflow#
A common geostatistical workflow for integrating subsurface data, modeling uncertainty, and supporting development decision making. The workflow proceeds from data to decisions through the following steps:
Integrate all available information to build multiple subsurface scenarios and realizations that sample the uncertainty space.
Apply all realizations through the transfer function to sample the uncertainty in the decision criteria.
Assemble the distribution of the decision criteria from the ensemble of realizations and scenarios.
Make the optimum reservoir development decisions while accounting for the modeled uncertainty.
Supervised Learning#
Machine learning methods that learn relationships between predictor features and labeled response features.
Supervised learning uses both predictor features and response features,
response feature values, \(Y\), are provided with corresponding predictor features, \(X_1,\ldots,X_m\)
the machine learns a mapping from predictor features to response features
learned relationships may include prediction functions, decision boundaries, probability models, or other representations of the relationship between inputs and outputs
Supervised learning focuses on prediction of a response feature rather than inference of the natural system,
estimating response feature values for new observations
evaluating model performance using unseen data
Common supervised learning methods include,
regression - predicting continuous response features
classification - predicting categorical response features
time series forecasting - predicting future response feature values from historical observations
In this course we use the terms,
predictive machine learning - supervised learning methods focused on predicting response features from predictor features
inferential machine learning - unsupervised learning methods focused on discovering patterns and structure in data
Contrast with:
Support Vector#
For a support vector machine, support vectors are the training observations that determine the location of the decision boundary.
Support vectors include observations that,
lie on or inside the margin
are misclassified (soft-margin SVM)
Training observations well outside the margin,
have no influence on the fitted decision boundary
Only the support vectors contribute to the optimization of the support vector machine, giving the method its name.
Used in:
Also see:
Support Vector Machine#
Predictive, binary classification machine learning method designed to perform well when categorical groups have poor separation in the original predictor feature space.
Conceptually, support vector machines project the original predictor features into a higher-dimensional feature space where a linear decision boundary (a plane or hyperplane) can separate the categories,
where \(\beta\) is the vector of model parameters, \(\beta_0\) is the intercept, and \(x\) is the predictor feature vector in the higher-dimensional feature space.
The predicted category is determined by,
where,
\(f(x)\) is proportional to the signed distance from the decision boundary
\(f(x)=0\) lies exactly on the decision boundary
\(G(x)=-1\) and \(G(x)=+1\) indicate the two classification categories
The kernel trick makes this practical,
rather than explicitly constructing the higher-dimensional feature space, support vector machines calculate inner products between observations using a kernel function
the classifier therefore behaves as though it operates in the higher-dimensional feature space while avoiding the associated computational cost
For perfectly separable data, the decision boundary must satisfy,
where the class labels are encoded as \(y_i\in\{-1,+1\}\).
For real-world problems with overlapping groups and noisy observations, a soft-margin support vector machine allows margin violations through slack variables,
where,
\(\xi_i=0\) indicates an observation outside the margin and correctly classified
\(0<\xi_i<1\) indicates an observation inside the margin but correctly classified
\(\xi_i>1\) indicates a misclassified observation
The optimization problem is,
subject to,
The optimization balances two competing objectives,
maximize the decision margin
minimize margin violations through the slack variables
Only a subset of the training data influences the fitted classifier,
observations on or within the margin are known as support vectors
observations well outside the margin have no influence on the fitted decision boundary
The hyperparameter \(C\) controls the model bias–variance trade-off,
larger \(C\) places greater emphasis on correct classification, resulting in a smaller margin and potentially higher model variance (overfitting)
smaller \(C\) allows more margin violations, resulting in a wider margin and potentially higher model bias (underfitting)
Support vector machines become nonlinear through the choice of kernel function. Common kernels include,
Linear kernel
Polynomial kernel
Radial basis function (RBF) kernel
Sigmoid kernel
Additional kernel hyperparameters control the flexibility of the classifier, for example,
Polynomial kernel - polynomial order
Radial basis function kernel - \(\gamma\), controlling the distance over which training observations influence the decision boundary
Support vector machine summary,
effective for high-dimensional predictor feature spaces
depends only on the support vectors near the decision boundary
maximizes the decision margin while penalizing margin violations
uses the kernel trick to efficiently construct nonlinear classifiers
Used in:
Also see:
Tabular Data#
A data representation where observations are organized into a table with,
rows representing individual samples or observations
columns representing predictor features, response features, identifiers, or metadata
Tabular data is the most common data format for machine learning, statistics, and data analytics. Examples include,
spatial data with one row for each sampled location
temporal data with one row for each time observation
multivariate laboratory measurements with one row for each sample
Pandas’ ‘DataFrame’ is the standard Python class for working with tabular data due to,
convenient storage, access, and manipulation of tabular data
built-in methods to load data from a variety of file formats, databases, and spreadsheets
built-in methods for summary statistics, visualization, grouping, filtering, sorting, and joining tables
built-in methods for cleaning, reshaping, and transforming data
built-in attributes describing the data structure, for example, dimensions, column names, data types, and missing values
Compare with:
Also see:
Train and Test Split#
Model cross validation, prior to predictive model training, withholds a proportion of the data as testing data.
training data are applied to estimate the model parameters
withheld testing data are applied to evaluate prediction performance and tune the model hyperparameters
Model hyperparameter tuning selects the combination that minimizes the error norm over the withheld testing data.
The most common approach is random selection; however, this may not provide fair testing.
The testing difficulty should be similar to the intended real-world use of the model,
too easy – testing cases are the same as, or very similar to, training cases. Random sampling is often too easy.
too hard – testing cases are very different from the training cases, requiring severe extrapolation beyond the available data.
Cross validation may use a single train and test split or multiple splits, for example,
Holdout Cross Validation - one training subset and one withheld testing subset
K-fold Cross Validation - \(K\) splits with every observation withheld once for testing
Leave-One-Out Cross Validation - \(n\) splits, each time withholding a single observation for testing
Used in:
Training Image#
A 2D training image is a dense conceptual representation of expected geological patterns, connectivity, and morphology over a 2D area of interest.
Training images provide a library of geological patterns used to inform,
geostatistical multiple point simulation, where complex spatial relationships are learned from examples rather than only two-point statistics.
genAI models
Common aspects of training images include,
represent prior geological knowledge and conceptual understanding of spatial patterns
do not include local information or conditioning data before simulation conditioning
must have the same cell size as the simulation model
do not need to have the same extent (number of model cells in each dimension), but should be large enough to provide sufficient examples of geological patterns
larger training images provide more pattern examples and greater representation of geological variability, but increase computational complexity
2D training images and 3D training models are used as conceptual pattern libraries for machine learning, generative AI models, and geostatistics,
generative models learn the statistical structure of patterns from training images and models are used as conceptual pattern libraries for machine learning, generative AI models, and geostatistics,
generative models learn the statistical structure of patterns from training images and models
generative models learn the statistical structure of patterns from training images
generated realizations reproduce learned spatial relationships while creating new possible outcomes
Used in:
Also see:
Training Model#
A 3D training model is a dense conceptual representation of expected geological patterns, connectivity, and morphology over a 3D volume of interest.
Training models provide a library of geological patterns used to inform,
geostatistical multiple point simulation, where complex spatial relationships are learned from examples rather than only two-point statistics.
genAI models
Common aspects of training models include,
represent prior geological knowledge and conceptual understanding of spatial patterns
do not include local information or conditioning data before simulation conditioning
must have the same cell size as the simulation model
do not need to have the same extent (number of model cells in each dimension), but should be large enough to provide sufficient examples of geological patterns
larger training models provide more pattern examples and greater representation of geological variability, but increase computational complexity
2D Training images and 3D training models are used as conceptual pattern libraries for machine learning, generative AI models, and geostatistics,
generative models learn the statistical structure of patterns from training images and models
generated realizations reproduce learned spatial relationships while creating new possible outcomes
Used in:
Also see:
Transfer Function#
A model, process, or calculation applied to spatial subsurface model realizations and scenarios to transform uncertain subsurface properties into a decision criteria.
The transfer function connects subsurface uncertainty models to decision making by calculating metrics that represent value, risk, health, environment, safety, or operational constraints.
Transfer functions may be physics-based, data-driven, or hybrid. Example transfer functions include,
transport and bioattenuation - numerical simulation to model soil contaminant concentrations over time during a pump-and-treat operation
volumetric calculation - estimate total oil-in-place from reservoir property models
heterogeneity metrics - calculate indicators related to recovery factor and estimate reserves from resources
flow simulation - generate pre-drill production forecasts for a planned well
Whittle pit optimization - calculate mineral resources and ultimate pit shell designs
Transformer#
A neural network architecture that uses attention mechanisms to learn relationships between elements of sequential or structured data.
Transformers learn contextual representations by:
computing attention-based relationships between input elements
combining information from multiple elements without requiring sequential processing
learning compact feature representations through stacked attention and neural network layers
Transformers are widely used in natural language processing, computer vision, and other machine learning applications.
Common transformer-based models include:
large language models (LLMs) - attention to learn large-scale, contextual relationships among language elements
vision transformers (ViTs) - attention to learn large-scale spatial structures in images
multimodal foundation models - attention to simultaneously process diverse data types to address diverse problems
Contrast with:
Trend#
An interpretation that a spatial feature is nonstationary over space, meaning that one or more statistics of the feature systematically change over the 2D area of interest or 3D volume of interest.
For example,
porosity decreases with depth
copper grade increases toward a highly faulted zone
Trend in spatial data may be identified by,
integrating expert geological knowledge and physical understanding
calculating bivariate statistics, such as conditional means given a spatial coordinate
fitting a deterministic non-stationarity model with location as a predictor feature and evaluating model significance
calculating an experimental variogram and checking for trend structure
Trend is also used to describe a deterministic model of nonstationarity in a statistic or metric of interest, as in Trend Model.
Used in:
Also see:
Trend and Residual Workflow#
Most geostatistical modeling methods assume stationarity in the feature mean. Yet, nonstationarity, trend, in the mean is commonly observed in the subsurface.
to address this limitation, the common hybrid workflow is to deconvolve the spatial data into 2 components:
known - deterministic trend model
unknown - stochastic residual model
The known trend is calculated and then subtracted from the data, leaving a residual that is modelled stochastically with uncertainty (treated as unknown). The following steps are applied:
model the nonstationary, spatial, deterministic trend for a feature of interest
subtract the trend from the data to calculate the residual
model the residual with geostatistical spatial estimation or simulation
add the deterministic trend to the geostatistical (deterministic if kriging or stochastic if simulation) residual
check the model
Also see:
Trend Model#
A determistic model representing the spatial trend in a statistic that is applied as an input for a spatial simulation method, for example,
a linear function for reduction in average porosity with depth, based on local data and regional compaction trends
a moving window local average copper grade model to model the increase in copper grade toward the highly faulted zone
This provides a local value of the statistic at all model grid cells, so the simulation can apply the trend model to relax the assumption of statistionarity in the statistic.
a trend model may be calculated and applied to applied to any statistic used in the simulation model, e.g., mean, variogram range, variogram major direction, correlation coefficient, etc.
Also see:
Uncertainty Modeling#
Characterization of the range of plausible values for a feature at a location, jointly over the entire subsurface model, or propagated through a transfer function to support decision making.
Uncertainty may be considered at different levels:
a single location - realizations from a random variable
the entire subsurface model - realizations from a random function
decision criteria - realizations from a transfer function applied to the subsurface model
Common sources of uncertainty include:
Data imprecision - measurement error, interpretation uncertainty, and imperfect observations
Spatial offset from data - uncertainty from estimating unsampled locations away from available spatial data
Model parameter inference - uncertainty in inferred parameters such as global mean, variance, variogram, and correlation structure
Conceptual model uncertainty - uncertainty from choices about geological framework, modeling approach, and assumptions about the subsurface system
Uncertainty models are represented with ensembles of scenarios and realizations:
Scenarios - multiple spatial subsurface models calculated by changing input parameters or other modeling choices to represent uncertainty from model parameters and conceptual choices
Realizations - multiple spatial subsurface models calculated by holding input parameters and modeling choices constant and changing only the random number seed
How can we address each source of uncertainty?
data imprecision - model data uncertainty through data realizations, soft data integration, or indicator transforms
spatial offset from data - calculate multiple stochastic realizations by varying the simulation random number seed
model parameter inference - calculate scenarios by varying inferred model parameters
conceptual model uncertainty - develop and compare alternative geological interpretations or modeling workflows
Important considerations for uncertainty modeling,
uncertainty modeling is critical for quantifying limitations in sample precision and model predictions
uncertainty is itself a model; there is no objective uncertainty independent of assumptions, data, and modeling choices. Failure to recognize this leads to the circular pursuit of “uncertainty in the uncertainty”
uncertainty results from sparse sampling, measurement error, interpretation uncertainty, bias, and geological heterogeneity
uncertainty reflects our limited ability to observe subsurface features with sufficient accuracy, resolution, and coverage; it is not an intrinsic property of the geology itself
Used in:
Underfit Model#
A predictive machine learning model that fails to learn the underlying relationship in the natural system due to insufficient flexibility or excessive constraints.
An underfit model demonstrates,
low prediction accuracy with training data and poor prediction accuracy with withheld testing data
insensitivity to the specific training observations, resulting in systematic prediction errors
During model hyperparameter tuning, the underfit region is characterized by,
increasing model complexity reduces both training error and testing error
strong agreement between training and testing performance, but both with high error
This behavior represents the high model bias region of the model bias–variance trade-off.
Issues associated with an underfit machine learning model include,
insufficient model complexity and flexibility relative to the available data quantity, accuracy, frequency, and coverage
failure to capture important patterns and relationships in the natural system
Underfitting may be reduced through,
increasing model complexity
reducing regularization or other constraints on model flexibility
including additional informative predictor features
More about underfit models,
underfit models often approach the global mean of the response feature
underfit models have high error over both training and testing data
increasing model complexity generally decreases error over the underfit region
the underfit region occurs before the optimal model complexity, where training and testing errors are both decreasing
Used in:
Contrast with:
Union of Events#
The union of events represents all outcomes where event \(A\) occurs, event \(B\) occurs, or both events occur. The probability of the union is calculated with the probability addition rule,
The intersection probability is subtracted because outcomes where both \(A\) and \(B\) occur are included in both \(P(A)\) and \(P(B)\) and would otherwise be counted twice.
For mutually exclusive events, the intersection probability is zero,
and the probability addition rule simplifies to,
Used in:
Univariate#
Involving a single feature or event only.
Examples include:
univariate statistics - summary measures describing one feature, such as mean, variance, or histogram
univariate statistical distributions - probability models describing the possible values of one feature, such as a probability density function or cumulative distribution function
Used in:
Compare with:
Univariate Parameter#
A univariate population summary measure describing a single feature.
Examples include:
population mean, \(\mu\)
population variance, \(\sigma^2\)
population cumulative distribution function, \(F_x(x)\)
In practice, the complete population is rarely available, so univariate parameters are inferred from available univariate statistics calculated from samples.
Used in:
TBD
Contrast with:
Univariate Statistic#
A summary measure calculated from samples of a single feature.
Examples include:
sample standard deviation, \(s\) - measure of dispersion
Histogram - visualization of the univariate distribution
Marginal probability - measure of univariate likelihood
Univariate statistics describe the available sample and are used to infer the corresponding ppopulation parameter.
Used in:
Contrast with:
Unsupervised Learning#
Machine learning methods that learn patterns, structure, or representations from data without labeled response features.
Unsupervised learning uses only predictor features,
no response feature, \(Y\), is provided; instead only predictor features, \(X_1,\ldots,X_m\), are available
the machine learns by discovering regularities, patterns, and compact representations of the data
learned structures may include feature projections, group assignments, latent neural network features, probability distributions, or other representations of the data
Unsupervised learning focuses on inference of the natural system rather than prediction of a response feature,
understanding the structure, variability, and relationships within the available data
identifying patterns that may support interpretation, exploration, or future modeling
Common unsupervised learning methods include,
clustering - identifying groups or regions of similar observations
dimensionality reduction - finding lower-dimensional representations that preserve important information
density estimation - modeling the distribution and probability structure of the data
In this course we use the terms,
inferential machine learning - unsupervised learning methods focused on discovering patterns and structure in data
predictive machine learning - supervised learning methods focused on predicting response features from predictor features
Contrast with:
Value#
A component of the attention mechanism that allows a model to dynamically focus on the most relevant information while reducing the influence of less relevant information.
A Value (\(V\)) is the stored information that may contribute to the current prediction or calculation.
The attention mechanism,
compares each Query with all Keys to calculate similarity scores, which are then used to derive attention weights.
the attention weights are applied to the Values and combined to provide the information used for the current prediction or calculation.
Some additional comments,
each Key–Value pair represents a stored piece of information, where the Key describes the information and the Value contains the associated content.
Conceptually,
Value (\(V\)) answers, “Here is the information to use.”
Used in:
Also see:
Variable#
Any property measured or observed in a study, for example,
porosity, permeability, mineral concentrations, saturations, contaminant concentration
in data mining / machine learning this is known as a feature
often requires significant analysis, interpretation, etc.
Used in:
Everywhere
Same as:
Variance#
A measure of distribution dispersion, the spread or variability of a feature about its average. Larger variance indicates greater variability.
For a sample, the variance is,
The equivalent population parameter is the population variance,
Some comments about variance,
units - the units of variance are squares units of the feature, for more intuitive units consider using the standard deviation
additivity - variance are additive, enabling a lot of workflows like analysis of variance and trend + residual workflows, for example given \(X_{residual} + X_{trend} = X_{total}\), the variance is calculated as,
momments - variance is the \(2^{nd}\) centered momment
outliers - variance is very sensitive to outliers
Used in:
Also see:
Variance Inflation Factor#
Feature ranking or often feature filtering metric based on the linear multicollinearity between a predictor feature (\(X_i\)) and all other predictor features (\(X_j, \forall j \ne i\)).
Variance inflation factor is often applied as a first-pass filter to remove highly redundant predictor features before subsequent feature ranking or model training.
To calculate variance inflation factor,
Build a linear regression model to predict one predictor feature from all other predictor features,
Determine the coefficient of determination, R-squared or \(R^2\), for this regression model,
this \(R^2\) represents how well the remaining predictor features explain the feature \(X_i\)
Calculate the variance inflation factor,
where \(R_i^2\) is the coefficient of determination from predicting \(X_i\) using all other predictor features.
The interpretation of variance inflation factor,
\(VIF=1\) - no linear redundancy with other predictor features
larger \(VIF\) values indicate increasing multicollinearity and reduced independent information from the predictor feature
large \(VIF\) values indicate that the uncertainty in estimated model coefficients is inflated due to redundant predictor features
Common guidelines include,
\(VIF < 5\) - often considered acceptable multicollinearity
\(VIF > 5\) or \(10\) - often considered evidence of problematic multicollinearity
Comments about variance inflation factor,
redundancy - accounts for linear relationships between a predictor feature and all other predictor features
relevance - does not account for any relationship between the predictor feature and the response feature
linearity - only identifies linear redundancy; nonlinear feature relationships may not be detected
Used in:
Also see:
Variance Reduction Factor#
A convenient factor used to correct variance when changing from the data volume support to a larger model volume support.
The variance reduction factor, \(f\), is defined as the ratio of variance at the larger volume support, \(v\), to the variance at the original data volume support, \(\cdot\):
Using volume-variance relations, this can be calculated as,
where \(\overline{\gamma}(v,v)\) is the average variogram value within the volume support \(v\) and \(\sigma^2\) is the variance at the original data support.
Equivalently, using dispersion variance,
The variance reduction factor is applied to adjust the data histogram to represent the reduced variability expected at a larger model scale.
Without volume support correction, the original data-scale distribution will have excessive variance when applied directly to a larger model volume.
Used in:
Also see:
Variogram#
A scatterplot with axes of difference or variance over distance.
Experimental variogram is calculated over integer multiples of the unit lag distance and then plotted as points, then permissible variogram models are fit to the experimental variogram while integrating other domain and local knowledge.
the variogram is calculated as one half the average squared difference over lag distance, 𝐡, over all possible pairs of data,
the precise term is semivariogram (or variogram if you remove the \frac{1}{2} in the equation above), but in practice, the semivariogram is only used and the term variogram is always used for the semivariogram
the \(\frac{1}{2}\) term is added to the semivariogram so that the covariance function, \(C_z(\bf{u})\), and variogram, \(\gamma_z(\bf{h})\), may be related as:
Note the correlogram, \(\rho_z(\bf{u})\), is related to the covariance function, \(C_z(\bf{u})\), as:
Here are some general observations about the variogram,
Often increasing - as the lag Distance, \(\bf{h}\), increases, variability over the lag distance increase (in general).
Not a local measure - the variogram is calculated with over all possible pairs separated by lag vector, \(\bf{h}\).
Interpret relative to the [ill - we need to plot the sill on with the experimental variogram to know the degree of correlation.
the sill is the variance, \(\sigma^2\), given stationarity of the variance and variogram, \gamma_z(\bf{h})):
\(\quad\) and given a standardized feature, \(\sigma_z^2 = 1.0\),
\(\quad\) the distance from the sill to the experimental variogram is the correlation coefficient over the specific lag distance.
Range - the lag distance at which the variogram reaches the sill is know as the range.
at the range, knowing the data value at the tail provides no information about a value at the head.
Nugget effect - sometimes there is a discontinuity in the variogram at distances less than the minimum data spacing. This is known as nugget effect.
the ratio of nugget divided by sill, is known as relative nugget effect, reported in percentage, e.g., 10% relative nugget effect
we model the nugget effect as a no correlation structure over all lags greater than an infinitesimal distance, \(\bf{h} \gt \epsilon\)
measurement error, causes an apparent nugget effect, if this is suspected do not add nugget effect to the variogram model
Also see:
Venn Diagram#
A visual tool for communicating probability relationships using set notation and the probability of events.
A Venn diagram contains:
a box labelled as \(\Omega\) representing the sample space, including all possible outcomes
enclosed labelled shapes representing events, which are subsets of the sample space
What do we learn from a Venn diagram?
the size of regions is proportional to the probability of occurrence
the entire sample space, \(\Omega\), represents all possible outcomes and therefore has probability:
individual regions represent marginal probabilities, for example:
overlapping regions represent joint probabilities, for example:
overlapping regions relative to a conditioning event represent conditional probabilities, for example:
Venn diagrams are an excellent tool to visualize marginal probability, joint probability, and conditional probability relationships and are especially useful for understanding probability operators.
Used in:
Volume of Interest#
The 3D spatial domain that is being characterized, modeled, and evaluated to support subsurface decision making. In general, the volume of interest,
is the subsurface reservoir for oil and gas, the ore body for mining, or the aquifer for hydrogeological applications
may include volume away from the reservoir or ore body to support data integration and extraction modeling
may be further subdivided into local regions or facies and modeled separately
is represented by a grid with features populated from data, estimation, or simulation
in 2D modeling is commonly called the area of interest
the extent and grid cell size are selected based on a trade-off between model accuracy and computational complexity
Used in:
Volume Support#
The spatial volume over which a feature or variable is measured or averaged. Volume support defines the physical extent of a measurement and directly influences variability, smoothing, and scale dependence. In practice, it is often related to (but not identical with) the concept of scale.
Examples include:
Core volume support is:
where \(r_{core}\) is core radius and \(l_{core}\) is core length.
Well Log volume support is:
where \(r_{log}\) is the logging tool radius (or effective radius of investigation) and \(l_{log}\) is the vertical resolution or sampling interval.
Seismic volume support is:
where \(\delta x_{\text{inline}}\) is inline resolution, \(\delta y_{\text{crossline}}\) is crossline resolution, and \(\delta z_{\text{vertical}}\) is vertical resolution.
It is critical to explicitly state volume support when describing data or models:
Volume support strongly influences measured statistics and spatial variability.
Consistent comparison between datasets requires accounting for differences in support through change-of-support or upscaling methods.
Used in:
Also see:
Volume-Variance Relations#
The relationship between volume support and variance. In general, as the volume support increases, the variance of a feature decreases because larger volumes average over more spatial variability.
Predicting volume-variance relations is central to integrating data collected at different scales and building subsurface models that represent the appropriate level of heterogeneity.
General observations and assumptions:
Under linear averaging and stationary conditions, the mean does not change with volume support; only the variance changes.
The distribution shape may change with volume support. This should be evaluated empirically. Common approaches include assuming no shape change with an affine correction or applying a distribution-specific correction such as an indirect lognormal correction.
Variance reduction is controlled by spatial continuity. Features with shorter correlation ranges experience faster variance reduction as volume support increases, while features with longer ranges retain variability over larger volumes.
Over common changes in subsurface modeling scale, the impact may be significant. Therefore, volume-variance relations should not be ignored.
Perfect scale-up accounting is rarely achieved because sufficient data are generally unavailable to fully characterize variability across all scales. This is commonly referred to as the missing scale problem.
A model is required to predict how variance changes with volume support.
Common methods to model and apply volume-variance relations include:
Empirical - build a high-resolution model and numerically upscale to the larger volume support. For example,
calculate a fine-scale permeability model
apply flow simulation to estimate effective permeability over larger block volumes
Power Law Average - a flexible averaging approach for changing support,
\(\quad\) where \(\omega\) is the power of averaging:
\(\omega = 1\) is arithmetic averaging
\(\omega = -1\) is harmonic averaging
\(\omega = 0\) is geometric averaging, obtained as the limit as \(\omega \rightarrow 0\)
\(\quad\) The appropriate \(\omega\) may be determined from:
theoretical understanding, for example, harmonic averaging of permeability for flow perpendicular to beds
numerical upscaling with flow simulation followed by calibration of an effective averaging exponent
Statistical Model - directly adjust statistical properties for the change in volume support. For linear averaging with a stationary variogram model, the Variance Reduction Factor is:
\(\quad\) where \(f\) is the ratio of variance at larger volume support to variance at the original data support:
\(\quad\) The variance reduction factor is calculated from,
the variogram model representing spatial continuity
the original data support
the target model volume support
Also see:
Weak Learner#
A predictive machine learning model that performs only slightly better than random prediction.
A weak learner is represented as,
where \(\hat{f}_k\) is the \(k^{th}\) weak learner, \(X_1,\ldots,X_m\) are the predictor features, and \(\hat{Y}\) is the predicted response feature.
Characteristics of weak learners include,
simple model structure with limited flexibility
high model bias and low model variance
prediction performance only marginally better than random
Examples include,
decision stump - a decision tree with only one split and two terminal regions
simple linear models for complex nonlinear natural systems
Weak learners are the foundation of ensemble methods, especially boosting,
boosting sequentially combines many weak learners, where each new learner focuses on correcting the errors of previous learners
the combined ensemble can produce a strong predictive model with improved accuracy and generalization
The terms used include,
weak predictor - general term for a weak predictive model
weak classifier - weak learner applied specifically to classification problems
Used in:
Weight#
A trainable model parameter associated with a neural network connection that controls the influence of information flowing into a node.
Information flows forward through the network,
each node forms a weighted sum of the incoming information, adds a bias, and applies an activation function
where \(w_{ij}\) is the weight associated with the connection from node \(i\) to node \(j\).
The magnitude and sign of a weight determine how information influences the next node,
positive weights reinforce the incoming signal
negative weights oppose the incoming signal
larger magnitude weights have greater influence on the node output
During training,
the connection weights are updated by backpropagation and gradient-based optimization to minimize the loss function
together, the collection of weights stores the information learned from the training data
While weights are commonly used for neural networks, there are other important uses of the term weights, including,
generalized as a trainable model parameter that scales the influence of an predictor feature, for example multiple linear regression slope parameters
declustering weights to improve sample representativity
kriging weights that account for data closeness, redundancy and spatial continuity
Used in:
Contrast with:
Also see:
Well Image Log#
A special case of well logs where the well logs are repeated at various azimuthal intervals within the well bore resulting in a 2D (unwrapped) image instead of a 1D line along the well bore. For example, Fullbore formation MicroImager (FMI) with:
with 80% bore hole coverage
0.2 inch (0.5 cm) resolution vertical and horizontal
30 inch (79 cm) depth of investigation
can be applied to observe lithology change, bed dips and sedimentary structures.
Used in: TBD
Also see:
Well Log#
Geostatistical Concepts: as a much cheaper method to sample wells that does not interrupt drilling operations, well logs are very common over the wells. Often all wells have various well logs available. For example,
gamma ray on pilot vertical wells to assess the locations and quality of shales for targetting (landing) horizontal wells
neutron porosity to assess location high porosity reservoir sands
gamma ray in drill holes to map thorium mineralization
Well log data are critical to support subsurface resource interpretations. Once anchored by core data they provide the essential coverage and resolution to model the entire reservoir concept / framework for prediction, for example,
well log data calibrated by core data collocated with well log data are used to map the critical stratigraphic layers, including reservoir and seal units
well logs are applied to depth correct features inverted from seismic data that have location imprecision due to uncertainty in the rock velocity over the volume of interest
Used in: TBD
Also see:
Want to Work Together?#
I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.
Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I’d be happy to drop by and work with you!
Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!
I can be reached at mpyrcz@austin.utexas.edu.
I’m always happy to discuss,
Michael
Michael Pyrcz, Ph.D., P.Eng. Professor, Cockrell School of Engineering and The Jackson School of Geosciences, The University of Texas at Austin
More Resources Available at: Twitter | GitHub | Website | GoogleScholar | Geostatistics Book | YouTube | Applied Geostats in Python e-book | Applied Machine Learning in Python e-book | LinkedIn
Comments#
This was a basic introduction to geostatistics. If you would like more on these fundamental concepts I recommend the Introduction, Modeling Principles and Modeling Prerequisites chapters from my text book, Geostatistical Reservoir Modeling{cite}`pyrcz2014’.
I hope this is helpful,
Michael