Duan smearing estimator when your target is log-transformed

I use log targets a lot in regression problems.

Prices, revenue, rents, salaries: they usually have a long right tail. Modeling log(y) instead of y often makes the problem calmer.

But there is a small trap:

pred_price = np.exp(pred_log_price)

That line is simple, common, and usually biased low.

The fix is also simple. It is called Duan’s smearing estimator.

The setup

I used an old OLX apartment listing dataset I collected from Belo Horizonte. The point here is not to build a serious real estate model. The point is only to show the correction with something concrete.

The target is listing price in BRL.

The raw prices are skewed. After logging the target, the distribution is easier for a plain regression model to deal with.

I fit a boring model on purpose:

• features: area, rooms, bathrooms, parking spots, condo fee, tax, region
• model: Ridge regression
• target: log(price)

Something like:

model.fit(X_train, np.log(y_train))
pred_log_price = model.predict(X_test)

Now we need to come back from log-price to price.

The wrong-but-common back transform

The naive version is:

pred_price_naive = np.exp(pred_log_price)

This feels right, because log and exp are inverse functions.

The catch: the model is estimating the expected value on the log scale, not the expected value on the original price scale.

In general:

exp(E[log(y)]) is not the same thing as E[y]

For right-skewed targets, exp(pred_log_price) is closer to a conditional median. If what you want is an expected price, it tends to undershoot.

Duan’s smearing estimator

After fitting the model, compute residuals on the log scale using the training set:

train_log_pred = model.predict(X_train)
residuals = np.log(y_train) - train_log_pred

Then compute the average exponentiated residual:

smearing_factor = np.exp(residuals).mean()

Finally, multiply the naive prediction by this factor:

pred_price_smeared = np.exp(pred_log_price) * smearing_factor

That is the whole trick.

In this run, the smearing factor was:

1.0522

So the correction increased the back-transformed predictions by about 5.2%.

Why it works

Think of the log model as:

log(y) = prediction + error

Then:

y = exp(prediction + error)
y = exp(prediction) * exp(error)

The naive version keeps exp(prediction) and silently acts like the error part averages to 1.

Duan’s estimator says: do not assume that. Estimate it from the residuals:

average(exp(error))

That average is the smearing factor.

Tiny result

On the holdout set from this toy model:

Notice something important: smearing fixed the average scale, but it did not magically improve every metric. The MAPE got slightly worse here.

That is not a contradiction. Duan’s smearing estimator is about correcting the back-transformation when you want predictions on the original mean scale. It is not a free accuracy cheat code.

One listing

For one holdout apartment:

area:    92 m²
rooms:   3
baths:   3
parking: 2
region:  zona_centro_sul

actual:  R$ 1,365,000
naive:   R$   973,053
smeared: R$ 1,023,821

Still wrong, because the model is intentionally simple. But the corrected version moves in the direction implied by the log-scale residuals.

The rule of thumb

If you train a regression model on log(y) and need predictions back in the original unit, do not stop at:

np.exp(pred)

At least try:

smearing_factor = np.exp(log_residuals_train).mean()
pred_y = np.exp(pred_log_y) * smearing_factor

It is one line, model-agnostic, and usually the first correction I want before interpreting log-target predictions as money again.