Machine Learning Glossary D to F#
Michael J. Pyrcz, Professor, The University of Texas at Austin
Twitter | GitHub | Website | GoogleScholar | Geostatistics Book | YouTube | Applied Geostats in Python e-book | Applied Machine Learning in Python e-book | LinkedIn
Chapter of e-book “Applied Machine Learning in Python: a Hands-on Guide with Code”.
Cite this e-Book as:
Pyrcz, M.J., 2024, Applied Machine Learning in Python: A Hands-on Guide with Code [e-book]. Zenodo. doi:10.5281/zenodo.15169139 
The workflows in this book and more are available here:
Cite the MachineLearningDemos GitHub Repository as:
Pyrcz, M.J., 2024, MachineLearningDemos: Python Machine Learning Demonstration Workflows Repository (0.0.3) [Software]. Zenodo. DOI: 10.5281/zenodo.13835312. GitHub repository: GeostatsGuy/MachineLearningDemos 
By Michael J. Pyrcz
© Copyright 2024.
This chapter is a summary of essential Machine Learning Terminology.
Motivation for this Glossary#
Firstly, why create this glossary?
I received a request for a course glossary from students in my Subsurface Machine Learning graduate course. While I usually dedicate a definition slide in my lecture slide decks to important terms, various students requested a consolidated glossary of terminology as part of their course review materials. The e-book provides an excellent vehicle and motivation for creating this resource.
Let me begin with a confession. There is a Machine Learning Glossary written by Google developers. For those seeking the in depth, comprehensive list of machine learning terms please use this book! For those seeking a comprehensive and in-depth reference of geostatistical terminology, this book remains an excellent resource.
So why create another glossary?
Scope - By writing my own glossary, I can limit the scope and descriptions to the concepts covered in this course. I believe many students would be overwhelmed by the size, breadth, and mathematical notation of a comprehensive geostatistics glossary.
Statistical Linkages - At the same time in my opinion machine learning is statistical learning and I have taken liberty to include many standard statistical terms as a foundation for all things machine learing.
Workflow Integration - By integrating the glossary directly into the e-book, I can link terminology to detailed chapter discussions, demonstrations, and examples. The goal is to eventually populate all chapters with hyperlinks to glossary entries, enabling students to move easily between concepts and applications.
Evergreen Resource - Finally, like the rest of this e-book, I want the glossary to be an evergreen living document. It will continue to evolve with new concepts, improved explanations, and feedback from students and readers.
I put quite a bit of time into this project during summer 2026 and I am happy with the way that is has evolved,
More than a glossary, it has become an evergreen network of machine learning concepts.
Data#
Data are observations collected to characterize a population or process. In spatial data analytics and geostatistics, three fundamental aspects determine the value of a dataset:
Data coverage - what proportion of the population has been sampled? In general, hard data have high resolution (small volume support), but poor spatial coverage. For example,
Core coverage in deepwater oil and gas may sample only one five hundred millionth to one five billionth of a reservoir, assuming 3-inch diameter cores with 10% core recovery in vertical wells spaced 500–1,500 m apart.
Core coverage for mining grade control may sample approximately one eight thousandth to one thirty thousandth of an ore body, assuming HQ (63.5 mm) cores with complete recovery in drill holes spaced 5–10 m apart.
\(\quad\) In contrast, soft data often provide excellent (sometimes complete) spatial coverage, but lower resolution, for example,
Seismic and other remote sensing measurements commonly cover the entire area of interest but have substantially lower spatial resolution, generally decreasing with depth.
Volume support (data scale) - what volume or scale is represented by each measurement? Examples include,
core tomography imaging at the pore scale (approximately 1–50 \(\mu\)m)
gamma ray well log sampled every 0.3 m with approximately 1 m radial investigation
ground-based gravity gradiometry with an effective resolution of approximately 20 m × 20 m × 100 m
Information content - what does the dataset tell us about the subsurface? Examples include,
grain size distributions used to calibrate permeability and saturation
fluid contacts used to identify oil-water contacts
structural dip and continuity used to infer reservoir connectivity
mineral grades used to delineate ore shells for mine planning
Used in: TBD
Data Analytics#
In this book, data analytics is used synonymously with:
Data Convexity#
A subset, \(A\), of Euclidean feature space is convex if for any two points \(x_1\) and \(x_2\) within \(A\), the entire line segment connecting these points is also contained within \(A\).
Mathematically,
where \(\lambda\) defines any location along the line segment between \(x_1\) and \(x_2\).
Considering data samples, \(x_i^k\), where \(i\) represents the sample index and \(k\) represents the predictor feature, the data occupy a region in predictor feature space where each axis corresponds to a predictor feature.
if for any two data samples \(x_1\) and \(x_2\) the entire line segment connecting them lies within the region occupied by the data, then the data distribution is convex
if portions of the connecting line segment fall outside the data region, the data distribution is non-convex
Data convexity is important in machine learning because many algorithms rely on assumptions about the structure of the feature space,
non-convex data structures may require nonlinear models or methods capable of capturing complex decision boundaries.
For example,
Used in:
Also see:
Data Preparation#
The workflow of transforming raw data into a reliable, consistent, and model-ready dataset for analysis and machine learning.
In many applied studies, especially in subsurface applications,
the majority of project effort is often dedicated to data preparation, quality control, and integration
data preparation requires domain knowledge combined with robust application of statistics and data analytics
Data preparation is essential due to common challenges with real-world data, including,
data curation – establishing format standards, version control, storage, transmission, security, and documentation practices
large data volumes – challenges with visualization, accessibility, computational management, and data exploration
large volumes of metadata – lack of consistent platforms, standards, and formats for describing data
engineering integration – combining diverse data sources with different scales, interpretations, measurement uncertainties, and levels of quality
data quality issues – missing values, inconsistent formats, measurement errors, duplicates, and incorrect records
Clean and well-documented databases are a prerequisite for successful data analytics and machine learning.
machine learning performance is limited by the quality and representativeness of the input data
garbage in, garbage out
Used in:
DataFrame#
A convenient Pandas data structure for working with tabular data. A DataFrame is a two-dimensional labeled data structure with rows representing samples or observations and columns representing features or variables.
DataFrames provide a convenient structure to,
store, access, and manipulate tabular data
load data from a variety of sources, including files, Python objects, databases, and Excel spreadsheets
calculate summary statistics and visualize data
perform data queries, sorting, filtering, and selection operations
complete data manipulation tasks, including cleaning, transformation, merging, and reformatting
store metadata and information about the dataset, such as dimensions, column names, data types, and missing values
Used in:
Compare with:
DBSCAN#
A density-based clustering algorithm that discovers groups in feature space by identifying regions with sufficient sample density.
DBSCAN is an acronym for Density-Based Spatial Clustering of Applications with Noise (Ester et al., 1996).
Clusters are expanded from dense regions using hyperparameters that define the neighborhood scale and minimum required density.
The primary DBSCAN hyperparameters are,
\(\epsilon\) – the radius of the local neighborhood measured in normalized feature space. This defines the scale or resolution of the clusters. If \(\epsilon\) is too small, many samples remain unassigned as outliers and clusters may be fragmented. If \(\epsilon\) is too large, distinct clusters may merge.
\(minPts\) – the minimum number of samples required within an \(\epsilon\) neighborhood to define a core point. Core points initialize and expand cluster groups.
Density is evaluated by counting the number of samples within an \(\epsilon\) neighborhood. In high-dimensional feature spaces, this neighborhood is defined using a distance metric applied across all dimensions.
An automated or guided approach for estimating \(\epsilon\) is available using a k-distance plot,
calculate the \(k\) nearest neighbor distance for every sample in normalized feature space
sort the distances in ascending order
select \(\epsilon\) near the point of maximum curvature (the elbow), representing a transition between dense clusters and sparse outliers
Salient aspects of DBSCAN clustering include,
Advantages – requires minimal prior knowledge of the number of clusters, can identify clusters with arbitrary shapes, and can efficiently identify outliers
Density-based cluster growth – samples are initially unassigned and clusters are iteratively expanded by connecting density-reachable core points and their neighboring samples
Mutually exclusive – like k-Means clustering, each assigned sample belongs to only one cluster group,
Non-exhaustive – some samples may remain unassigned and are classified as noise or outliers,
Used in:
Contrast with:
Also see:
Debiasing with Secondary Data#
When the full range of a primary feature is not sampled, declustering alone cannot remove sampling bias because parts of the feature distribution are completely missing. Instead, we use,
a secondary data feature that provides coverage over the entire area of interest
the relationship between the Primary Data feature and the secondary feature
to infer the unsampled portion of the primary feature distribution.
The relationship between the primary and secondary features may be established using,
a statistical model that extrapolates the primary feature into the unsampled range
a physical model based on scientific or engineering understanding
expert knowledge of the underlying process
Unlike declustering, which corrects for clustered spatial sampling, debiasing with secondary data addresses situations where part of the primary feature distribution has not been sampled at all.
Mentioned but not demonstrated (to be added later) in: TBD
Compare with: TBA
Decision Criteria#
An engineered feature or metric calculated from one or more subsurface models to support decision making. Decision criteria quantify the consequences of alternative decisions and may represent economic value, technical performance, environmental impact, health and safety, or combinations of these objectives. For example,
contaminant recovery rate to support the design of a pump-and-treat soil remediation project
oil in place to determine whether a reservoir should be developed
Lorenz coefficient as a heterogeneity measure to classify a reservoir and identify appropriate analogs
recovery factor or production rate to schedule production and optimize facilities
recovered mineral grade and tonnage to determine the economic ultimate pit shell
In quantitative decision workflows, the decision criterion is used to rank competing alternatives. Common approaches include,
maximizing a profit metric
minimizing a loss function
Best practice is to define decision criteria that directly represent project value. For example,
rather than stopping at hydrocarbon in place, continue through engineering and economics to estimate project profit (currency).
Used in:
Also see:
Decision Making#
The ultimate objective of data science, including machine-learning, geostatistics and data analytics is to support better decisions. Estimation, prediction, uncertainty modeling , and machine learning are intermediate steps whose value is realized only when they improve decisions.
Decision making involves selecting the best,
estimate
choice
from a set of alternatives.
In quantitative decision workflows, a decision criteria is used to rank competing alternatives. Common approaches include,
maximizing a profit metric
minimizing a loss function
to identify the optimum estimate or decision.
when accounting for uncertainty, this optimization is performed over an ensemble of subsurface realizations and scenarios.
Used in:
Also see:
Decision Tree#
An intuitive supervised regression and classification machine learning model that divides the predictor feature space, \(X_1,\ldots,X_m\), into \(J\) mutually exclusive and exhaustive regions, \(R_j\).
mutually exclusive – any combination of predictor values belongs to only one region, \(R_j\)
exhaustive – all possible combinations of predictor values belong to one of the regions, meaning the regions cover the entire feature space
For regression, the prediction within each region is the mean of the training responses in that region,
where \(n_j\) is the number of training samples within region \(R_j\).
For classification, the prediction within each region is the most common class, determined by the mode or argmax operator,
where \(k\) represents the possible response classes.
Other salient points about decision trees,
supervised learning – the response feature label, \(Y\), is available for the training data and used to construct the model
hierarchical, binary segmentation – the predictor feature space begins as a single region and is sequentially divided into smaller regions through binary splits
compact, interpretable model – because each split is based on a single predictor feature, the model can be represented as a tree structure with binary branches. The resulting model can be implemented as nested if statements, for example,
if porosity > 0.15:
if brittleness < 20:
initial_production = 1000
else:
initial_production = 7000
else:
if brittleness < 40:
initial_production = 500
else:
initial_production = 3000
The decision tree is constructed from the top down. We begin with a single region that covers the entire feature space and then proceed with a sequence of splits,
scan all possible splits over all regions and all predictor features
greedy optimization – select the split that provides the greatest improvement in prediction accuracy. For regression trees, this is achieved by minimizing the residual sum of squares (RSS),
where \(\hat{y}_{R_j}\) is the mean response prediction within region \(R_j\).
for classification trees, splits are selected by minimizing a class impurity measure, such as Gini impurity,
where \(p_k\) is the proportion of samples belonging to class \(k\) within region \(R_j\).
A pure classification region contains only one class,
while higher values indicate greater mixing of classes.
Each split is optimized locally using a greedy algorithm without considering future splits.
Hyperparameters include,
number of regions – controls the complexity of the tree and directly determines the number of terminal nodes
minimum reduction in RSS or impurity – prevents splits that provide insufficient improvement; however, stopping too early may prevent later beneficial splits
minimum number of training samples in each region – controls the reliability of regional predictions and helps prevent an overfit model
maximum tree depth – limits the number of sequential splits and controls model complexity
Used in:
Declustered Statistics#
Once declustering weights are calculated for a spatial dataset, the unweighted (also called naive) statistics are replaced with weighted statistics that account for the declustering weights. These corrected statistics are then used as input for all subsequent analysis and machine learning modeling to mitigate sampling bias. For example,
the declustered mean is used as the stationary global mean for simple kriging.
the weighted cumulative distribution function is used in sequential Gaussian simulation so that the back-transformed realizations reproduce the declustered distribution.
Any sample statistic can be computed using declustering weights, including the entire cumulative distribution function (CDF). Examples include:
weighted sample mean,
where \(n\) is the number of data.
weighted variance,
where \(\overline{x}_{wt}\) is the declustered mean.
weighted Covariance,
where \(\overline{x}_{wt}\) and \(\overline{y}_{wt}\) are the declustered means for features \(X\) and \(Y\).
weighted cumulative distribution function (CDF),
This expression represents the empirical weighted CDF evaluated at the observed data values. Between observations, the CDF is obtained by interpolation.
No declustering method can guarantee improved estimates of the population parameters for every dataset.
however, when preferential sampling is present, declustering methods generally reduce sampling bias and provide improved statistical estimates in expectation.
Used in:
Also see:
Declustering#
A family of methods that assign weights to spatial samples based on local sampling density so that weighted statistics are more representative of the inaccessible population. Data weights are assigned so that,
samples in densely sampled areas receive less weight
samples in sparsely sampled areas receive more weight
There are various declustering methods:
It is important to note that no declustering method can prove that for every data set the resulting weighted statistics will improve the prediction of the population parameters, but in expectation these methods tend to reduce the bias.
these data weights may be applied to improve machine learning model prediction accuracy
Used in:
Also see:
Degree Matrix#
A diagonal matrix representing the number of connections (degree) for each node in a graph.
For a graph with \(n\) nodes, the degree matrix \(D\) is defined as,
the diagonal elements contain the degree of each node, representing the number of connections to other nodes
all off-diagonal elements are zero
where \(A\) is the adjacency matrix and \(D_{ii}\) is the degree of node \(i\).
The entries of the degree matrix are integers,
\(0\) indicates a node with no connections
larger values indicate nodes with more connections
The degree matrix is used in graph-based machine learning methods, including spectral clustering, where it contributes to the calculation of the graph Laplacian,
where \(L\) is the graph Laplacian, \(D\) is the degree matrix and \(A\) is the affinity matrix.
Used in:
Also see:
Density-Based Cluster#
A nonempty set of points where every pair of points is density-connected with respect to the DBSCAN parameters \(\epsilon\) and \(minPts\).
Important aspects,
clusters are formed by groups of density-connected points
density-based clusters can have arbitrary shapes because cluster membership is determined by density connectivity rather than distance from a central point
points that are not density-connected to any cluster are classified as noise or outliers
Used in:
Also see:
Contrast with:
Density-Based Clustering#
A family of clustering methods that identifies groups as contiguous regions of high sample density in predictor feature space, separated by regions of low sample density.
Rather than assigning every sample to a cluster,
dense regions are identified as clusters
isolated observations are often classified as outliers or noise
Compared with k-means clustering, density-based clustering works well when populations,
overlap in predictor feature space
have irregular or non-spherical shapes
contain noise or outliers that should not belong to any cluster
have an unknown number of groups
General characteristics of density-based clustering,
cluster shape - clusters may take arbitrary shapes rather than being approximately spherical
automatic group count - the number of clusters is determined from the data rather than specified in advance
outlier detection - isolated observations are naturally identified as noise
One limitation of density-based clustering,
performance may decrease when clusters have substantially different densities, making it difficult to identify a single density threshold appropriate for all groups
Examples of density-based clustering methods include,
DBSCAN - clusters are formed from connected dense neighborhoods using minimum density criteria
OPTICS - extension of DBSCAN that accommodates varying cluster densities
Used in:
Also see:
Contrast with:
Density-Connected#
Two points \(A\) and \(B\) are density-connected if there exists a point \(Z\) such that both \(A\) and \(B\) are density-reachable from \(Z\).
Important aspects,
density-connectedness is a symmetric relationship, meaning if \(A\) is density-connected to \(B\), then \(B\) is density-connected to \(A\)
density-connected points belong to the same density-based cluster because they share a common density-reachable path through core points
Used in:
Contrast with:
Also see:
Density-Reachable#
A point \(Y\) is density-reachabl* from a point \(A\) if there exists a sequence of points connecting \(A\) to \(Y\) such that each point in the sequence is within the \(\epsilon\) neighborhood of the previous core point.
A density-reachable path requires a chain of core points where each core point is density-connected to the next point, and the final point \(Y\) may be a core point or a border point.
Important aspects,
density-reachability depends on the direction of the starting point, meaning if \(Y\) is density-reachable from \(A\), \(A\) may not be density-reachable from \(Y\)
applied by DBSCAN to grow clusters from core points through regions of sufficient density
Used in:
Contrast with:
Also see:
Deterministic Model#
A model that assumes a system or process is completely specified such that the same inputs always produce the same outputs. Deterministic models do not explicitly represent uncertainty in the system or process; therefore,
uncertainty is neglected and the system is treated as known or certain.
Deterministic models may be based on,
engineering and geoscience physics
expert interpretation and knowledge
data-driven estimation methods
Examples include,
numerical flow simulation for a specified set of reservoir properties
stratigraphic bounding surfaces interpreted from seismic data
kriging estimates
machine learning prediction models that return a single prediction
Advantages:
integrates physics, expert knowledge, and available data
integrates multiple information sources
often straightforward to interpret and apply
Disadvantages:
provides a single model or prediction without explicitly representing uncertainty
may underestimate decision risk when uncertainty is significant
often time consuming to construct, calibrate, and validate
Used in: TBS
Contrast with:
Dimensionality Reduction#
Methods to reduce the number of predictor features within a data science workflow. There are 2 primary methods,
features Selection – find the subset of original features that are most important for the problem
feature projection – transform the data from a higher to lower dimensional space
Known as dimension reduction or dimensionality reduction,
motivated by the curse of dimensionality and multicollinearity
applied in statistics, machine learning and information theory
Used in:
Also see:
Directly Density-Reachable#
A point \(X\) is directly density-reachable from point \(A\) if \(A\) is a core point and \(X\) belongs to the \(\epsilon\)-neighborhood of \(A\).
Mathematically,
and,
where \(N_{\epsilon}(A)\) is the set of samples within distance \(\epsilon\) of point \(A\), and \(minPts\) is the minimum number of samples required for \(A\) to be a core point.
Important aspects,
direct density reachability is the fundamental connection used to grow DBSCAN clusters
the relationship is directional; if \(X\) is directly density-reachable from \(A\), \(A\) may not be directly density-reachable from \(X\) unless \(X\) is also a core point
the starting point must be a core point, but \(X\) may be either a core point or a border point
chains of directly density-reachable points create density-reachable paths used for cluster growth
Used in:
Also see:
Discrete Feature#
A feature that can only take one of a countable set of distinct values. Discrete features may be naturally discrete (categorical feature) or created by grouping (or binning) a continuous feature. For example,
binned continuous feature – porosity between 0% and 20% assigned to 10 bins:
\(\quad\) represented by the bin centroids,
categorical feature – facies:
ordinal feature – Mohs hardness:
Used in:
Contrast with:
Distribution Transformation#
A mapping from one probability distribution to another through corresponding percentile values, also called a quantile transformation. The transformation preserves the rank ordering of the data while changing the distributional shape, resulting in new,
Distribution transformations are commonly applied in geostatistical methods and workflows because,
inference - to transform a sample distribution toward an expected distribution when data are sparse, biased, or insufficient to characterize the full distribution
theory - to satisfy a distributional assumption required by a workflow step, for example, a Gaussian distribution with mean 0.0 and variance 1.0 is required for sequential Gaussian simulation
data preparation - to reduce the influence of extreme values by mapping them into the target distribution while preserving their rank relationship
improve model performance - to improve sensitivity of the machine learning model, for example, sensitivity of an activation function.
How do we perform distribution transformations?
Values are transformed from the original cumulative distribution function, \(F_X\), to a target CDF, \(G_Y\), using percentile matching. This quantile transformation is applied to all sample values:
Forward transform:
Reverse transform:
This approach may be applied to any distribution, including parametric and nonparametric distributions, as long as percentile values can be mapped between the distributions.
The key property is:
rank preserving transform - the percentile position of a value is maintained, for example, P25 remains P25 after transformation
Contrast with affine correction, which only adjusts distribution location and scale (mean and variance),
distribution transformation modifies the complete distribution, including higher-order statistics and distribution shape
Used in:
Compare with:
Dot-product#
A linear algebra operation that measures the alignment between two vectors.
For two vectors \(a\) and \(b\),
The dot product can also be interpreted geometrically,
where \(\theta\) is the angle between the two vectors.
The dot product answers the question,
how much do two vectors align?
Interpretation,
large positive value \(\rightarrow\) strong alignment, vectors point in similar directions
zero \(\rightarrow\) orthogonal vectors, directions are perpendicular
negative value \(\rightarrow\) opposing directions
For similarity measures, the dot product can be interpreted as,
similar vectors \(\rightarrow\) large positive dot product
unrelated vectors \(\rightarrow\) small dot product
opposing vectors \(\rightarrow\) negative dot product
Applications in machine learning include,
attention mechanism – the dot product between Query and Key vectors is used to measure the relevance or similarity between elements
linear models – the prediction is often calculated as a weighted dot product between feature vectors and model parameters,
similarity search – vector representations are compared using dot products to identify similar patterns or observations
Used in:
Compare with:
Drill Cuttings#
Direct samples of subsurface material generated during drilling operations.
drill cuttings are fragments of rock produced by the drill bit and continuously transported to the surface, where they are collected, described, and logged during drilling.
Drill cuttings provide broader spatial coverage than core data because they are commonly recovered along much of the well or borehole trajectory during routine drilling operations. However, compared with core data, drill cuttings,
represent small, irregular, and mixed rock fragments rather than a continuous sample volume. Individual fragments may range approximately from 0.1 mm to 5 cm, although larger fragments (cavings) may occur due to mechanical failure along the borehole or wellbore.
lose orientation and large-scale structural information during recovery and transport because fragments are mixed and disrupted during pneumatic or hydraulic lifting from the borehole or well.
provide lithological and compositional information but generally cannot preserve continuous sedimentary structures, fracture orientations, or fine-scale spatial relationships.
Drill cuttings represent a trade-off between core and indirect measurements; they provide extensive direct sampling coverage but with reduced spatial resolution and geological context.
Used in: TBD
Also see:
Eager Learning#
A machine learning approach where a generalized model is constructed during a training phase before prediction queries are made.
after model parameter training and model hyperparameter tuning, the model is independent of the original training data and can calculate new predictions without accessing the training dataset
the computational effort is concentrated during the training phase, while prediction is typically fast
Examples include,
Logistic Regression - TBA
Used in:
Linear Regression Analytical Solution Description and Demonstration
LASSO Regression Interative Solution Description and Demonstration
Bayesian Linear Regression MCMC for Sampling Posterior Model Parameters
Also see:
Contrast with:
Eigenvalue#
A scalar that quantifies the amount of scaling associated with an eigenvector during a matrix transformation. Eigen value is the,
amount that the eigenvector is stretched or squished by during that transformation
An eigenvalue, \(\lambda\), satisfies
where \(\mathbf{A}\) is a matrix and \(\mathbf{v}\) is the corresponding eigenvector.
In Principal Component Analysis,
the eigenvalues of the covariance matrix quantify the variance explained by the corresponding principal components.
Also see:
Eigenvector#
A nonzero vector, special direction that remains unchanged when a matrix transformation is applied,
including stretching, rotating, or shearing a grid)
An eigenvector, \(\mathbf{v}\), satisfies
where \(\mathbf{A}\) is a matrix and \(\lambda\) is the corresponding eigenvalue.
In Principal Component Analysis,
the eigenvectors of the covariance matrix define the principal component directions, directions with most variance explained.
Also see:
Ergodic Fluctuations#
Statistical fluctuations observed when calculating statistics from finite simulated realizations of an ergodic random function. The statistics calculated from an individual realization are expected to vary around the input model statistics. For example,
the histogram of an individual realization may not exactly reproduce the input histogram
the variogram of an individual realization may not exactly reproduce the input variogram
the correlation coefficient between primary and secondary paired realizations may not exactly reproduce the input correlation coefficient
Some general observations about ergodic fluctuations,
part of the uncertainty model - fluctuations in statistical reproduction, along with scenarios, are an important part of the uncertainty model because they represent natural variability among possible realizations
magnitude - controlled by the ratio of spatial continuity range to the size of the model domain
minimized - when the model domain is large relative to the spatial continuity range, providing many effective independent spatial samples
maximized - when the model domain is small relative to the spatial continuity range, providing fewer effective independent spatial samples
When checking simulated realizations, some fluctuation in the histogram, variogram, and correlation coefficients should be expected.
best practice is to evaluate the expectation of these statistics over many realizations and compare the ensemble statistics with the input model statistics
Used in:
Estimation#
The paradigm and process of obtaining a single best value to represent a feature or variable at an unsampled location or time.
the “best” estimate is determined by an objective criterion, such as minimizing estimation error.
Some additional estimation concepts,
local accuracy - estimation methods prioritize honoring local data and minimizing local uncertainty, often at the expense of reproducing the full range of global spatial variability
deterministic model - the same inputs always produce the same outputs
smoothness - estimation methods commonly produce values that are smoother than the true variability because local averaging reduces variance
nonlinear response - smooth estimates may not be appropriate when applying transforms or decision criteria that are sensitive to heterogeneity, such as flow response, connectivity, recovery, or economic metrics
examples - inverse distance weighting and kriging
many predictive machine learning models focus on estimation, including k-nearest neighbours, decision trees, and random forests
Used in: TBA - estimation vs. simulation in concepts
Contrast with the simulation paradigm:
Evidence#
In Bayes’ Theorem, the evidence term represents the overall probability of observing the data. It provides the normalization required to ensure probability closure of the updated posterior probability.
where:
\(P(A)\) is the prior probability representing uncertainty before observing new information,
\(P(B|A)\) is the likelihood function describing the compatibility of observations \(B\) with state or parameter \(A\),
\(P(B)\) is the evidence term representing the total probability of observing data \(B\) and normalizing the posterior probability,
\(P(A|B)\) is the posterior probability representing updated uncertainty after incorporating observations.
Used in:
Also see:
Expectation#
The expected value is the probability-weighted average outcome of a random variable. It is a measure of central tendency that represents the average value accounting for the likelihood of all possible outcomes.
for a discrete random variable, the expectation is the sum of all possible outcomes weighted by their probabilities,
for a continuous random variable, the expectation is the integral of all possible values weighted by the probability density function,
Expectation is also the mathematical foundation for the average when all realizations are considered equiprobable.
Expectation is extremely useful for doing mathematics with random variables,
expectation of a constant,
expectation of a random variable plus a constant,
expectation of a constant multiplied by a random variable,
expectation of the addition of two random variables,
Expectation is widely used in data science,
for optimum decision making in the presence of uncertainty, i.e., selecting the choice that maximizes expected profit.
for the decomposition of expected test mean scquare error into model variance, model bias and irreducible error components
Used in:
Also see:
Expected Test Mean Square Error#
The expected test mean square error measures the expected prediction error of a model for observations that were not used during training,
where,
\(y_0\) is the true response for a new observation not included in the training data
\(\hat{f}(x_{0,1}^{(i)},\ldots,x_{0,m}^{(i)})\) is the model prediction for that observation
Because the training data are considered a random sample, the model predictions vary from one training dataset to another. The expectation is therefore taken over all possible training datasets.
Under standard assumptions, the expected test mean square error can be decomposed into three additive components,
These components may be summarized as,
Model Bias - systematic prediction error resulting from restrictive model assumptions
Model Variance - prediction variability caused by changes in the training data
Irreducible Error - uncertainty that cannot be explained by the available predictor features
This decomposition demonstrates the model bias–variance trade-off,
increasing model complexity generally decreases model bias but increases model variance
decreasing model complexity generally decreases model variance but increases model bias
The objective of model hyperparameter tuning is to select a model complexity that minimizes the expected test mean square error.
Used in:
Also see:
F1-score#
A categorical classification prediction model metric that summarizes the balance between precision and recall as a single metric calculated from the confusion matrix.
The F1-score is the harmonic mean of precision and recall,
or equivalently,
where \(k\) represents the category or class.
For binary classification, the confusion matrix is composed of,
True Positive (TP) – positive observations correctly classified as positive
False Positive (FP) – negative observations incorrectly classified as positive
True Negative (TN) – negative observations correctly classified as negative
False Negative (FN) – positive observations incorrectly classified as negative
Common summary metrics from the confusion matrix include,
a summarization over the columns and rows in a confusion matrix (truth on y-axis and predicticted categories on x-axis)
classification precision – the proportion of predicted positive observations that are actually positive,
classification recall (sensitivity) – the proportion of actual positive observations that are correctly classified,
F1-score – the harmonic mean of precision and recall,
The F1-score balances precision and recall and is especially useful when classes are imbalanced. Unlike accuracy, the F1-score does not allow the majority class to dominate the performance metric.
For multiclass classification, the F1-score may be summarized over all \(k\) categories using,
macro F1-score – average F1-score over all categories, giving equal importance to each class
weighted F1-score – average F1-score weighted by the number of samples in each category
micro F1-score – calculates precision and recall globally over all samples
Used in:
Also see:
Facies#
A method of grouping rock into discrete categories, creating a new categorical feature. Facies are used to represent geological variability in a manner that improves,
characterization through statistics, e.g., distributions and variograms
prediction of subsurface features, e.g., porosity and permeability away from wells
For oil and gas, the term facies is commonly used, while mining commonly uses terms such as rock types or zones. In subsurface modeling, multiple types of facies may be considered,
lithofacies - based on rock-related characteristics, including lithology, sedimentary structures, and small-scale geological features that influence porosity and permeability, for example, shale, sandstone, dolomite, limestone, laminated sandstone, hummocky cross-stratification, etc.
depofacies - integrate multiple lithofacies with depositional geometry and reservoir-scale architecture that impacts flow behavior and well connectivity, for example, channel axis, channel margin, outer sheet, etc.
seismic facies - large-scale classifications based on acoustic and elastic properties and seismic geomorphological expressions that define the reservoir framework, for example, parallel continuous high amplitude, chaotic amplitudes, mounded discontinuous low amplitudes, truncation, onlap, offlap, etc.
Here are some important considerations for determining facies,
facies or rock type is an important decision for subsurface modeling. Facies determination should remain a collaborative decision integrating expertise from the entire project team (Geologists, Reservoir Modelers, Reservoir Engineers, Petro- and Geophysicists).
facies or rock types must improve subsurface prediction away from the data or they do not add value.
the number of facies is a balancing act between geological realism, statistical inference, and modeling effort.
reservoir modeling is often hierarchical, for example, geological elements contain multiple depofacies, depofacies contain multiple lithofacies, and lithofacies have specific porosity and permeability distributions.
often 80-90% of reservoir-scale heterogeneity may be captured by the facies model.
Here is a summary of criteria for facies, rock types, or any discrete grouping used in a subsurface model,
Separation of rock properties - facies must be separable based on features that impact subsurface environmental and economic performance, for example, grade, porosity, permeability, etc.
Identifiable in data - facies must be identifiable from the most commonly available data. For example, facies identifiable only from cores are not useful if most wells only have well logs.
Map-able away from data - facies must be easier to predict away from data than the rock properties of interest directly; otherwise, facies do not improve prediction.
Sufficient sampling - there must be enough data to infer reliable statistics within each facies, i.e., by-facies statistics.
Used in:
Also see:
Feature#
A property measured, observed, or calculated (i.e., engineered feature) for analysis in a study.
also known as a variable
Features represent the information used to characterize, model, or predict a system. Examples include,
porosity, permeability, mineral concentrations, saturations, contaminant concentration, etc.
derived properties such as seismic attributes, ratios, transformations, or other calculated quantities.
Different fields use different terminology,
in data mining and machine learning this is commonly called a feature
in statistics this is commonly called a variable
in geoscience this is often called a property or attribute
Feature values may require significant measurement, processing, interpretation, and analysis before they are suitable for modeling.
When features are modified, combined, or transformed to improve model performance, this is called feature engineering.
Used in:
Also see:
Feature Engineering#
The process of creating, modifying, combining, transforming, or selecting features to improve model performance, interpretation, or statistical inference. Feature engineering may incorporate domain knowledge, physical understanding, and data analysis. Examples include,
adjusting total porosity to effective porosity
combining porosity and permeability into a single rock quality index measure
transforming data to account for volume support differences using a volume-variance relations model
weighting data samples for improved spatial representativity using declustering
transforming a data distribution to standard normal in geostatistical sequential Gaussian simulation
Feature engineering is commonly applied before modeling to create inputs that better represent the controlling processes and improve prediction or estimation
Used in:
Feature Importance#
Some machine learning methods provide convenient measures of feature importance, i.e., the impact of each predictor feature on the prediction(s). For example,
linear regression applied to standardized features - the standardized model coefficients,
\(\quad\) where the value of each coefficient indicates how much the target feature changes for a 1-unit increase in the corresponding standardized predictor feature.
decision trees - the cumulative reduction in mean square error attributable to splits on each feature, summarized as,
\(\quad\) where \(T_f\) is the set of all nodes that split on feature \(x\), \(N_t\) is the number of training samples reaching node \(t\), \(N\) is the total number of training samples, and \(\Delta_{MSE_t}\) is the reduction in MSE produced by the split at node \(t\).
\(\quad\) Note, feature importance is calculated similarly for classification decision trees by replacing MSE with Gini impurity (or another classification impurity measure).
Feature importance may be used for model-based feature ranking, but remember,
the reliability of feature importance depends on the predictive accuracy of the model, i.e., an inaccurate model will likely produce misleading feature importance estimates.
Used in:
Also see:
Feature Imputation#
Replacing missing feature values in a data table with plausible values for several reasons,
enable statistical calculations and machine learning methods that require complete data tables, i.e., cannot work with missing feature values
maximize model accuracy by increasing the number of reliable samples available for training and testing
mitigate model bias that may occur with listwise deletion when feature values are not missing at random
Feature imputation methods include,
constant value imputation - replace missing feature values with a constant statistic, such as the feature mean, median, or mode
model-based imputation - replace missing feature values with predictions from a model trained using the available feature values for the same sample
There are also iterative methods that depend on convergence,
Multiple Imputation by Chained Equations (MICE) - initialize missing values, then iteratively update them by predicting each missing feature from the remaining available and previously imputed feature values
The goal of feature imputation is to obtain reasonable values that preserve the relationships among the features while minimizing the bias and uncertainty introduced by missing data.
Used in:
Also see:
Feature Map#
A multidimensional representation of learned features produced by applying a convolution filter or other neural network operation to input data, including 2D images and three-dimensional or higher-dimensional models.
Feature maps preserve the spatial arrangement of the input while transforming the information into increasingly useful representations,
each feature map emphasizes a particular learned pattern or characteristic
spatial locations in the feature map correspond to spatial locations in the input
multiple feature maps allow different features to be learned simultaneously
Multiple feature maps arise through,
channels - multiple feature maps produced within the same layer, each generated by a different convolution filter
layers - successive feature maps at increasing levels of abstraction, where early layers commonly learn edges and textures, intermediate layers learn shapes and objects, and deeper layers learn arrangements and higher-level structures
As information flows through a convolutional neural network,
early feature maps commonly represent simple features such as edges, orientations, and textures
deeper feature maps combine simpler features into increasingly complex shapes, objects, and spatial structures
Feature maps are transformed by convolution, activation, and pooling layers before being used for final prediction or classification.
Used in:
Also see:
Feature Projection#
Methods that transform the original \(m\) features into \(p\) projected features, where \(p \ll m\), for dimensionality reduction and to reduce or remove predictor feature redundancy.
For example,
given \(m\) features, \(X_1,\ldots,X_m\), we require \(\binom{m}{2}=\frac{m(m-1)}{2}\) two-dimensional scatter plot s to visualize all pairwise feature relationships
these visualizations do not capture structures in more than two dimensions
once we have four or more features, understanding the relationships within the data becomes very difficult. Recall the curse of dimensionality.
Example machine learning methods for feature projection include linear methods,
principal component analysis - maximize the variance explained
factor analysis - explain variability with a smaller set of latent features
random projection - project data onto randomly generated directions; most effective for very high-dimensional datasets
non-linear methods, also known as manifold learning,
multidimensional scaling (classical or non-metric MDS) - preserve inter-sample distances
t-distributed stochastic neighbor embedding (t-SNE) - preserve local structure and data clusters
Uniform Manifold Approximation and Projection (UMAP) - preserve local and global structure
kernel principal component analysis (Kernel PCA) - project data into a higher-dimensional feature space where linear separation may be possible
and deep learning,
autoencoder - compress data through a neural network bottleneck
Alternative methods for dimensionality reduction include,
feature selection - retain the most relevant features while minimizing redundancy
feature aggregation - combine redundant or highly correlated features
Used in:
Also see:
Feature Ranking#
A collection of methods that quantify the relative importance of predictor features by measuring their contribution to predicting a response feature,
feature ranking is primarily motivated by the curse of dimensionality, seeking the smallest set of predictor features that retains the maximum predictive information while reducing redundancy, improving model interpretability, and often enhancing predictive performance.
As part of feature engineering, feature ranking assigns an importance score to each predictor feature based on its,
high relevance - the amount of useful information the feature provides for predicting the response feature
low redundancy - the extent to which this information is unique and not duplicated by other predictor features
The general classes of feature ranking methods considered in this book include,
Visual Inspection - examination of data distributions, scatter plot s, violin plots, and other graphical summaries
Statistical Summaries - quantitative measures of feature-response relationships, including correlation analysis, partial correlation coefficient, mutual information and maximum relevance minimum redundancy
Model-based Methods - feature importance scores derived from predictive models, including global Shapley values
Exploration of Feature Subsets - systematic evaluation of multiple feature subsets using trial-and-error search with cross-validation on withheld testing data to identify an optimal predictor feature set, for example recursive feature elimination
Used in:
Also see:
Feature Selection#
A dimensionality reduction method that improves model performance, and reduces model complexity while improving model interpretability by selecting a subset of the original predictor features.
Feature selection,
is motivated by the curse of dimensionality
retains a subset of the original features rather than creating new features
commonly uses feature ranking methods to identify the most informative features
Contrast with:
Also see:
Feature Space#
The multiple variate space represented by the ranges and possible combinations of all features for our problem. Commonly feature space only refers to the predictor features and does not include the response feature(s); therefore, it is,
all possible combinations of predictor features for which we need to make predictions
may be referred to as predictor feature space.
Typically, we train and test our machines’ predictions over the predictor feature space,
the space is typically a hypercuboid with each axis representing a predictor feature and extending from the minimum to maximum, over the range of each predictor feature
more complicated shapes of predictor feature space are possible, e.g., we could mask or remove subsets with poor data coverage.
Used in:
Also see:
Feature Transformation#
A feature engineering step involving mathematical operations applied to predictor feature(s) to create a representation that is more suitable for a machine learning workflow. For example,
feature truncation
feature normalization or standardization
feature distribution transformation
There are many reasons that we may perform feature transformations,
to make features consistent in scale and representation for visualization, comparison, and interpretation
to avoid bias or impose feature weighting for methods that rely on distances calculated in predictor feature space, for example k-nearest neighbours regression
to satisfy assumptions or requirements of specific methods, for example, artificial neural networks may perform better when features are normalized to a common range such as \([-1,1]\), and statistical methods based on correlation may benefit from approximately Gaussian distributed features
Feature transformation changes the representation of the predictor features while preserving the underlying information content, allowing machine learning methods to more effectively identify patterns and relationships.
Used in:
Also see:
Fourth Paradigm#
The data-driven paradigm for scientific discovery that builds upon the previous scientific paradigms,
First Paradigm - empirical science - experiments, observations, and the collection of measured data
Second Paradigm - theoretical science - analytical expressions, mathematical models, and fundamental principles
Third Paradigm - computational science - numerical simulation and computational experimentation
The Fourth Paradigm augments, rather than replaces, the previous paradigms. Each scientific paradigm builds upon and is supported by the earlier paradigms, for example,
theoretical science builds upon empirical observations to develop and validate mathematical relationships
computational science integrates analytical expressions, physical models, and calibrated equations derived from experimental observations
data-driven science combines large-scale datasets with computational methods to discover patterns, relationships, and predictive models that may be difficult to identify through traditional approaches
The Fourth Paradigm represents a shift from primarily hypothesis-driven discovery toward data-intensive discovery, while continuing to rely on empirical evidence, theory, and computation.
Used in:
Frequentist Probability#
A measure of the probability that an event occurs based on the long-run relative frequency observed from repeated experiments or repeated sampling. For random experiments and well-defined settings (such as coin tosses),
where:
\(n(A)\) = number of times event \(A\) occurred
\(n\) = number of trials
The frequentist interpretation assumes that probability represents an objective property of a repeatable process. Examples include,
probability of drilling a dry hole for the next well
probability of encountering sandstone at a location (\(\bf{u}_{\alpha}\))
probability of exceeding a rock porosity of \(15\%\) at a location (\(\bf{u}_{\alpha}\))
In geoscience, many processes cannot be repeated exactly; therefore, frequentist probabilities are often estimated from available samples under assumptions of representativity and stationarity.
Used in:
Contrast with:
Want to Work Together?#
I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.
Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I’d be happy to drop by and work with you!
Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!
I can be reached at mpyrcz@austin.utexas.edu.
I’m always happy to discuss,
Michael
Michael Pyrcz, Ph.D., P.Eng. Professor, Cockrell School of Engineering and The Jackson School of Geosciences, The University of Texas at Austin
More Resources Available at: Twitter | GitHub | Website | GoogleScholar | Geostatistics Book | YouTube | Applied Geostats in Python e-book | Applied Machine Learning in Python e-book | LinkedIn
Comments#
This was a basic introduction to geostatistics. If you would like more on these fundamental concepts I recommend the Introduction, Modeling Principles and Modeling Prerequisites chapters from my text book, Geostatistical Reservoir Modeling{cite}`pyrcz2014’.
I hope this is helpful,
Michael