21. Nonlinear Regression
When Straight Lines Are Not Enough
1 Introduction
Some relationships are curved because the biology is curved. Growth saturates, enzyme kinetics asymptote, seasonal cycles oscillate, and responses may level off, bend, or change slope across the range of the predictor. In such cases, a straight-line model is often too rigid.
This chapter focuses specifically on mechanistic nonlinear regression: models where the response is described by a named nonlinear function whose parameters have scientific meaning. Examples include saturating uptake curves, logistic growth, seasonal sine curves, and other biologically motivated responses.
Polynomial regression and GAMs are covered in their own chapters because they answer a different modelling need. Here the emphasis is narrower and stronger: what to do when the biology suggests the form of the curve itself.
2 Key Concepts
- Nonlinearity often reflects real biological structure rather than unwanted curvature.
- Flexible regression can model patterns that a straight line misses.
- Mechanistic nonlinear models are appropriate when the curve has a biologically motivated form.
- Greater flexibility increases the risk of overfitting if used carelessly.
- Model choice should follow biological form, not only convenience or visual preference.
3 When This Method Is Appropriate
You should move beyond a simple linear model when:
- the fitted residuals show clear curvature;
- the biology suggests an asymptote, threshold, saturation, or oscillation;
- the response changes differently across the predictor range;
- the question concerns parts of the response distribution other than the mean.
The diagnostic chapters earlier in the sequence already suggested this possibility. This chapter is about what to do next.
4 Nature of the Data and Assumptions
Mechanistic nonlinear regression is still regression, so the familiar concerns about independence, residual spread, and overall model adequacy do not disappear. In many introductory applications, the response variable is continuous and the residuals are assumed to be approximately normally distributed with constant variance.
The key distinction is that the mean structure is explicitly nonlinear and tied to a named function. These models are usually fitted by nonlinear least squares with nls(), or by mixed-effects extensions when grouped or repeated-measures data are present.
The practical point is that you do not choose a curved model just because it draws a pleasing bendy line through the points. You choose it because the straight-line model is inadequate and because the curved alternative matches the biological process or the inferential question more closely.
5 R Functions
Some of the most useful R functions are:
-
nls()for nonlinear least-squares models; -
nlme::nlme()when nonlinear relationships occur in grouped or repeated-measures data.
Examples:
One practical difference from ordinary regression is that nls() usually needs sensible starting values. Unlike lm(), it uses an iterative search for the best-fitting parameter values, and poor starting values can prevent convergence.
6 Example 1: Why Use a Mechanistic Nonlinear Model?
The first decision is whether the biology gives you a reason to prefer a specific nonlinear function rather than a descriptive curve.
Figure 1 shows three broad situations:
- A polynomial model can describe modest curvature, but it is mainly descriptive.
- A GAM is very useful when the relationship is clearly curved but its exact form is unknown or not informative.
- A mechanistic nonlinear model is most attractive when the biology suggests a specific function, because its parameters can often be interpreted directly.
That last point is important. If we know the process should saturate, as in nutrient uptake kinetics, a mechanistic model tells us more than a flexible smoother. It gives us interpretable parameters such as an asymptote or a half-saturation constant.
7 Example 2: Algal Nutrient Uptake Kinetics
We can measure algal nutrient uptake rates using a multiple flask experiment. We prepare a series of flasks, each containing a different initial concentration of the substrate nutrient, and then estimate nutrient uptake rate over a fixed time interval. The result is a set of uptake rates, \(V\), paired with substrate concentrations, \([S]\).
Applied to algae, the Michaelis-Menten model assumes an irreversible uptake process that saturates at high substrate concentrations. It effectively quantifies key characteristics of the nutrient uptake system, including the maximum uptake rate and the alga’s affinity for the nutrient:
\[V_i = \frac{V_{max} \cdot [S_i]}{K_m + [S_i]} + \epsilon_i\]
where:
- \(V_i\) is the uptake rate at the \(i\)-th observation;
- \(V_{max}\) is the maximum nutrient uptake rate achieved;
- \([S_i]\) is the substrate concentration at the \(i\)-th observation;
- \(K_m\) is the substrate concentration at which uptake reaches half of \(V_{max}\); and
- \(\epsilon_i\) is the residual error.
The two parameters are biologically meaningful. \(V_{max}\) represents the maximum capacity of the alga to utilise the nutrient, while \(K_m\) describes the affinity of the uptake system for that nutrient. Lower values of \(K_m\) indicate higher affinity.
7.1 Do an Exploratory Data Analysis (EDA)
To demonstrate fitting a nonlinear model to \(V\) versus \([S]\) data from a multiple flask experiment, I simulate data across a range of substrate concentrations. The dataset consists of five replicate flasks for each of 13 substrate concentrations.
conc_vec <- c(0, 0.1, 0.5, 2, 5, 7.5, 10, 12.5, 15, 17.5, 20, 25, 30)
n_rep <- 5
Km_vec <- c(10)
Vmax_vec <- c(50)
Km_vec_sd <- c(1.2)
Vmax_vec_sd <- c(0.7)
mf_data <- generate_data(
n_trt = 1,
n_rep = n_rep,
conc_vec = conc_vec,
Km_vec = Km_vec,
Vmax_vec = Vmax_vec,
Km_vec_sd = Km_vec_sd,
Vmax_vec_sd = Vmax_vec_sd
) |>
select(rep, S, V)
mf_data# A tibble: 65 × 3
rep S V
<fct> <dbl> <dbl>
1 1 0 0
2 2 0 0
3 3 0 0
4 4 0 0
5 5 0 0
6 1 0.1 0.511
7 2 0.1 0.588
8 3 0.1 0.480
9 4 0.1 0.496
10 5 0.1 0.453
# ℹ 55 more rows
The plot in Figure 2 already suggests that a straight line is not the right model. Uptake increases quickly at low substrate concentrations and then begins to level off. That is exactly the sort of pattern for which a Michaelis-Menten model was designed.
7.2 State the Model Question
At this point, two related questions arise.
The first is descriptive and mechanistic: can we estimate the uptake parameters \(V_{max}\) and \(K_m\) from these data?
The second is comparative: does a Michaelis-Menten model fit these data better than a simple linear model?
Those are not the same question. The first is about parameter estimation within a nonlinear biological model. The second is about choosing between two competing mean structures.
7.3 Fit the Model
The Michaelis-Menten model is fit with nls():
Formula: V ~ mm_fun(S, Vmax, Km)
Parameters:
Estimate Std. Error t value Pr(>|t|)
Vmax 51.4417 1.1770 43.70 <2e-16 ***
Km 10.5507 0.5975 17.66 <2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 1.312 on 63 degrees of freedom
Number of iterations to convergence: 3
Achieved convergence tolerance: 8.864e-06
The starting values do not have to be exact, but they should be sensible. Here we know from the plot that the asymptote is probably somewhere around 40 to 50, and that the half-saturation constant is likely to be in the single digits.
7.4 Check Diagnostics
Because this is a least-squares nonlinear model, residual diagnostics are interpreted much as they are for ordinary regression.
Shapiro-Wilk normality test
data: mf_data$resid
W = 0.97225, p-value = 0.1517
These diagnostics do not suggest any serious problem. The residual spread is fairly even, and the histogram is approximately symmetric. As always, you should not rely on a single formal test, but the residual behaviour is consistent with a usable fit.
7.5 Interpret the Results
df AIC
lm_mod 3 395.8491
nls_mod 3 223.6862
df BIC
lm_mod 3 402.3723
nls_mod 3 230.2094
Analysis of Variance Table
Response: V
Df Sum Sq Mean Sq F value Pr(>F)
S 1 10274.8 10274.8 422.59 < 2.2e-16 ***
Residuals 63 1531.8 24.3
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
The fitted nonlinear model gives estimated values for \(V_{max}\) and \(K_m\), and these are the biologically meaningful outputs of the analysis. The estimated \(V_{max}\) is the asymptotic uptake rate, while \(K_m\) indicates the substrate concentration at which the uptake rate reaches half of that asymptote.
The model comparison also matters. If the linear and Michaelis-Menten models are both fitted to the same data, the nonlinear model has markedly lower information criteria and a much smaller residual error. That supports the biological interpretation already suggested by the plot; that is, uptake increases with substrate concentration, but it does so in a saturating rather than indefinitely linear way.
Nutrient uptake increased rapidly with substrate concentration at low concentrations and then approached an asymptote, consistent with Michaelis-Menten kinetics (Figure 3). The fitted nonlinear model estimated a maximum uptake rate of 51.44 and a half-saturation constant of 10.55. Relative to a simple linear regression, the Michaelis-Menten model provided the better description of the data, with lower AIC and BIC values and a significant improvement in fit (F = NA, p < 0.05). These results indicate that nutrient uptake in this example is saturating rather than linear across the observed substrate range.
7.6 What If We Add Treatments?
Experiments are seldom as simple as the one above. Consider an experiment designed to assess whether an experimental treatment, such as light intensity or seawater temperature, affects nutrient uptake kinetics. It is biologically plausible to expect that each treatment will result in unique \(V_{max}\) and \(K_m\) values.
At that point, the modelling problem becomes a bit more challenging and exciting. A separate nls() fit can be used for each treatment, or a nonlinear mixed-effects model can be used when dependence or grouping must be handled explicitly. That is the nonlinear counterpart of the fixed-effects and mixed-effects logic introduced in Chapter 18. If the response is also non-normal, you are moving towards the broader territory covered in Chapter 19.
The important point is that nonlinear modelling “draws a curve” and it chooses a mean structure that matches the process and then extending that structure carefully when treatments, dependence, or more complicated designs are present.
8 Choosing Among Flexible Models
There are a few practical considerations to keep in mind when choosing a mechanistic nonlinear model. Sometimes different curved models can provide similar fits to the same data, but they have very different implications for interpretation. The reason to prefer a mechanistic nonlinear model is precisely that its parameters mean something biologically. If that interpretability is weak or artificial, a descriptive alternative such as a polynomial regression or GAM may be more honest.
9 Practical Caution
Flexibility is valuable, but it comes with a cost.
- More flexible models are often harder to interpret.
- They can overfit small datasets.
- They may fit noise rather than structure if the biological logic is weak.
- Nonlinear least-squares models can fail to converge if the functional form or starting values are poor.
This is why flexible regression should be a response to a clear diagnostic or biological need, not just a default preference for curved lines.
10 Summary
- Not all biological relationships are well described by straight lines.
- Mechanistic nonlinear models serve a distinct purpose within the broader family of curved-response models.
- Model choice should follow the biology and the inferential goal.
- Mechanistic nonlinear models are especially valuable when their parameters have clear biological meaning.
- Greater flexibility can improve fit, but it also increases interpretive demands and the risk of overfitting.
The next chapter extends the regression sequence in a different direction again by asking what happens when the scientific target is not the conditional mean, but another part of the response distribution.
Reuse
Citation
@online{smit,_a._j.2026,
author = {Smit, A. J., and J. Smit, A.},
title = {21. {Nonlinear} {Regression}},
date = {2026-03-19},
url = {http://tangledbank.netlify.app/BCB744/basic_stats/21-nonlinear-regression.html},
langid = {en}
}
