Convergence Diagnostics for Synthetic Difference-in-Differences

Introduction

The synthdid package uses iterative Frank-Wolfe optimization to estimate time weights (lambda) and unit weights (omega). Like all iterative algorithms, convergence is not guaranteed within a fixed number of iterations, especially for difficult problems.

This vignette demonstrates how to:

Check convergence using built-in diagnostics
Interpret convergence warnings and diagnostics
Fix convergence issues when they occur
Understand different convergence scenarios

The Convergence Monitoring System

Starting with version 1.0.0, synthdid automatically monitors optimization convergence with zero computational overhead. Every estimate now includes convergence diagnostics that tell you whether the optimization succeeded.

Quick Convergence Check

# Generate synthetic data
setup <- random.low.rank()

# Estimate treatment effect
estimate <- synthdid_estimate(setup$Y, setup$N0, setup$T0, max.iter = 100)

# Quick check: Did it converge?
synthdid_converged(estimate)
#> [1] FALSE

If this returns FALSE, the optimization hit the iteration limit before converging.

Detailed Diagnostics

# Get detailed convergence information
convergence_info <- synthdid_convergence_info(estimate)
print(convergence_info)
#> Synthdid Convergence Diagnostics
#> =================================
#> 
#> Overall Status: NOT CONVERGED 
#> 
#> Lambda weights optimization:
#>  converged iterations max_iter utilization
#>      FALSE        100      100      100.0%
#> 
#> Omega weights optimization:
#>  converged iterations max_iter utilization
#>      FALSE        100      100      100.0%
#> 
#> Recommendation: Consider increasing max.iter or relaxing min.decrease threshold.

This shows you:

Overall status: Did everything converge?
Lambda optimization: Status, iterations used, and utilization
Omega optimization: Status, iterations used, and utilization
Recommendations: What to do if convergence failed

Scenario 1: Full Convergence

Situation: Both lambda and omega weights converge successfully.

When it happens: - Well-conditioned problems - Sufficient iterations provided - Reasonable regularization parameters

# Generate a well-behaved synthetic dataset
set.seed(123)
n_units <- 40
n_periods <- 30
n_treated <- 5
n_post <- 10

# Create low-rank outcome matrix with clear treatment effect
Y <- matrix(rnorm(n_units * n_periods, mean = 50, sd = 5), n_units, n_periods)

# Add treatment effect
treatment_effect <- 10
Y[(n_units - n_treated + 1):n_units, (n_periods - n_post + 1):n_periods] <-
  Y[(n_units - n_treated + 1):n_units, (n_periods - n_post + 1):n_periods] + treatment_effect

# Estimate with adequate iterations
estimate_good <- synthdid_estimate(Y,
                                   N0 = n_units - n_treated,
                                   T0 = n_periods - n_post,
                                   max.iter = 10000)

# Check convergence
cat("Did it converge?", synthdid_converged(estimate_good), "\n\n")
#> Did it converge? TRUE

# Show diagnostics
print(synthdid_convergence_info(estimate_good))
#> Synthdid Convergence Diagnostics
#> =================================
#> 
#> Overall Status: CONVERGED 
#> 
#> Lambda weights optimization:
#>  converged iterations max_iter utilization
#>       TRUE        788    10000        7.9%
#> 
#> Omega weights optimization:
#>  converged iterations max_iter utilization
#>       TRUE       8160    10000       81.6%

✅ What to look for: - overall_converged: TRUE - Iterations < max_iter (not hitting the limit) - Utilization < 100% (stopped early due to convergence)

Interpretation: The optimization worked correctly. The estimate is reliable.

Scenario 2: Partial Convergence (Lambda Fails)

Situation: Omega converges but lambda doesn’t.

When it happens: - Complex temporal patterns - High noise in pre-treatment period - Insufficient iterations for lambda - Too restrictive lambda regularization

# Create data where lambda optimization is harder
set.seed(456)
n_units <- 50
n_periods <- 100  # Many time periods makes lambda harder
n_treated <- 3
n_post <- 20

# Create outcome with complex temporal patterns
time_trend <- 1:n_periods
seasonal <- 10 * sin(2 * pi * (1:n_periods) / 12)  # Seasonal component
Y_complex <- outer(rnorm(n_units), time_trend / 10) +
             matrix(seasonal, n_units, n_periods, byrow = TRUE) +
             matrix(rnorm(n_units * n_periods, sd = 3), n_units, n_periods)

# Add treatment
Y_complex[(n_units - n_treated + 1):n_units, (n_periods - n_post + 1):n_periods] <-
  Y_complex[(n_units - n_treated + 1):n_units, (n_periods - n_post + 1):n_periods] + 15

# Estimate with limited iterations
estimate_partial <- synthdid_estimate(Y_complex,
                                     N0 = n_units - n_treated,
                                     T0 = n_periods - n_post,
                                     max.iter = 500)  # Not enough for complex lambda

# Check convergence
cat("Overall convergence:", synthdid_converged(estimate_partial), "\n\n")
#> Overall convergence: FALSE

# Detailed diagnostics
conv_info <- synthdid_convergence_info(estimate_partial)
print(conv_info)
#> Synthdid Convergence Diagnostics
#> =================================
#> 
#> Overall Status: NOT CONVERGED 
#> 
#> Lambda weights optimization:
#>  converged iterations max_iter utilization
#>      FALSE        500      500      100.0%
#> 
#> Omega weights optimization:
#>  converged iterations max_iter utilization
#>      FALSE        500      500      100.0%
#> 
#> Recommendation: Consider increasing max.iter or relaxing min.decrease threshold.

⚠️ What to look for: - overall_converged: FALSE - Lambda: converged = FALSE, utilization = 100.0% - Omega: converged = TRUE, utilization < 100%

How to fix:

# Solution 1: Increase max.iter
estimate_fixed <- synthdid_estimate(Y_complex,
                                   N0 = n_units - n_treated,
                                   T0 = n_periods - n_post,
                                   max.iter = 5000)  # More iterations

cat("Fixed convergence:", synthdid_converged(estimate_fixed), "\n\n")
#> Fixed convergence: FALSE
print(synthdid_convergence_info(estimate_fixed))
#> Synthdid Convergence Diagnostics
#> =================================
#> 
#> Overall Status: NOT CONVERGED 
#> 
#> Lambda weights optimization:
#>  converged iterations max_iter utilization
#>      FALSE       5000     5000      100.0%
#> 
#> Omega weights optimization:
#>  converged iterations max_iter utilization
#>      FALSE       5000     5000      100.0%
#> 
#> Recommendation: Consider increasing max.iter or relaxing min.decrease threshold.

Alternative fixes:

# Solution 2: Relax convergence threshold
estimate_relaxed <- synthdid_estimate(Y_complex,
                                     N0 = n_units - n_treated,
                                     T0 = n_periods - n_post,
                                     max.iter = 500,
                                     min.decrease = 1e-2)  # Less strict

# Solution 3: Increase regularization (makes problem easier)
estimate_regularized <- synthdid_estimate(Y_complex,
                                         N0 = n_units - n_treated,
                                         T0 = n_periods - n_post,
                                         eta.lambda = 1e-4)  # More regularization

Scenario 3: Partial Convergence (Omega Fails)

Situation: Lambda converges but omega doesn’t.

When it happens: - Large number of control units - Heterogeneous control units - Few pre-treatment periods - Insufficient iterations for omega

# Create data where omega optimization is harder
set.seed(789)
n_units <- 200  # Many control units makes omega harder
n_periods <- 20  # Few periods
n_treated <- 5
n_post <- 5

# Create heterogeneous units
Y_hetero <- matrix(NA, n_units, n_periods)
for (i in 1:n_units) {
  unit_level <- rnorm(1, mean = 50, sd = 20)  # High heterogeneity
  Y_hetero[i, ] <- unit_level + cumsum(rnorm(n_periods, sd = 2))
}

# Add treatment
Y_hetero[(n_units - n_treated + 1):n_units, (n_periods - n_post + 1):n_periods] <-
  Y_hetero[(n_units - n_treated + 1):n_units, (n_periods - n_post + 1):n_periods] + 20

# Estimate with limited iterations
estimate_omega_issue <- synthdid_estimate(Y_hetero,
                                         N0 = n_units - n_treated,
                                         T0 = n_periods - n_post,
                                         max.iter = 1000)

cat("Overall convergence:", synthdid_converged(estimate_omega_issue), "\n\n")
#> Overall convergence: FALSE
print(synthdid_convergence_info(estimate_omega_issue))
#> Synthdid Convergence Diagnostics
#> =================================
#> 
#> Overall Status: NOT CONVERGED 
#> 
#> Lambda weights optimization:
#>  converged iterations max_iter utilization
#>       TRUE          2     1000        0.2%
#> 
#> Omega weights optimization:
#>  converged iterations max_iter utilization
#>      FALSE       1000     1000      100.0%
#> 
#> Recommendation: Consider increasing max.iter or relaxing min.decrease threshold.

⚠️ What to look for: - overall_converged: FALSE - Lambda: converged = TRUE, utilization < 100% - Omega: converged = FALSE, utilization = 100.0%

How to fix:

# Increase max.iter to allow omega to converge
estimate_omega_fixed <- synthdid_estimate(Y_hetero,
                                         N0 = n_units - n_treated,
                                         T0 = n_periods - n_post,
                                         max.iter = 10000)

cat("Fixed convergence:", synthdid_converged(estimate_omega_fixed), "\n\n")
#> Fixed convergence: FALSE
print(synthdid_convergence_info(estimate_omega_fixed))
#> Synthdid Convergence Diagnostics
#> =================================
#> 
#> Overall Status: NOT CONVERGED 
#> 
#> Lambda weights optimization:
#>  converged iterations max_iter utilization
#>       TRUE          2    10000        0.0%
#> 
#> Omega weights optimization:
#>  converged iterations max_iter utilization
#>      FALSE      10000    10000      100.0%
#> 
#> Recommendation: Consider increasing max.iter or relaxing min.decrease threshold.

Scenario 4: Joint Non-Convergence (With Covariates)

Situation: When using covariates, the joint optimization (lambda + omega + beta) doesn’t converge.

When it happens: - Many covariates - Covariates poorly predict outcome - Complex covariate interactions - Insufficient iterations

# Create data with covariates
set.seed(101112)
n_units <- 60
n_periods <- 40
n_treated <- 5
n_post <- 10
n_covariates <- 5

# Generate outcome
Y_cov <- matrix(rnorm(n_units * n_periods, mean = 100, sd = 10),
                n_units, n_periods)

# Generate covariates (somewhat correlated with outcome)
X <- array(NA, dim = c(n_units, n_periods, n_covariates))
for (k in 1:n_covariates) {
  X[, , k] <- matrix(rnorm(n_units * n_periods, mean = 50, sd = 5),
                     n_units, n_periods)
  # Add some correlation with Y
  Y_cov <- Y_cov + 0.2 * X[, , k]
}

# Add treatment effect
Y_cov[(n_units - n_treated + 1):n_units, (n_periods - n_post + 1):n_periods] <-
  Y_cov[(n_units - n_treated + 1):n_units, (n_periods - n_post + 1):n_periods] + 25

# Estimate with covariates and limited iterations
estimate_joint <- synthdid_estimate(Y_cov,
                                   N0 = n_units - n_treated,
                                   T0 = n_periods - n_post,
                                   X = X,
                                   max.iter = 2000)

cat("Overall convergence:", synthdid_converged(estimate_joint), "\n\n")
#> Overall convergence: FALSE
print(synthdid_convergence_info(estimate_joint))
#> Synthdid Convergence Diagnostics
#> =================================
#> 
#> Overall Status: NOT CONVERGED 
#> 
#> Joint optimization (with covariates):
#>  converged iterations max_iter utilization
#>      FALSE       2000     2000      100.0%
#> 
#> Recommendation: Consider increasing max.iter or relaxing min.decrease threshold.

⚠️ What to look for: - overall_converged: FALSE - Joint optimization: converged = FALSE, utilization = 100.0% - Note: With covariates, lambda and omega are optimized jointly, so you see “joint” instead

How to fix:

# Solution 1: Increase max.iter significantly for joint optimization
estimate_joint_fixed <- synthdid_estimate(Y_cov,
                                         N0 = n_units - n_treated,
                                         T0 = n_periods - n_post,
                                         X = X,
                                         max.iter = 10000)

cat("Fixed convergence:", synthdid_converged(estimate_joint_fixed), "\n\n")
#> Fixed convergence: FALSE
print(synthdid_convergence_info(estimate_joint_fixed))
#> Synthdid Convergence Diagnostics
#> =================================
#> 
#> Overall Status: NOT CONVERGED 
#> 
#> Joint optimization (with covariates):
#>  converged iterations max_iter utilization
#>      FALSE      10000    10000      100.0%
#> 
#> Recommendation: Consider increasing max.iter or relaxing min.decrease threshold.

# Compare estimates
cat("\nNon-converged estimate:", as.numeric(estimate_joint), "\n")
#> 
#> Non-converged estimate: 26.31604
cat("Converged estimate:    ", as.numeric(estimate_joint_fixed), "\n")
#> Converged estimate:     26.27549
cat("Difference:            ", abs(as.numeric(estimate_joint) - as.numeric(estimate_joint_fixed)), "\n")
#> Difference:             0.04054887

Alternative approaches:

# Solution 2: Reduce number of covariates (feature selection)
# Keep only most important covariates
X_reduced <- X[, , 1:2]
estimate_fewer_cov <- synthdid_estimate(Y_cov, N0, T0, X = X_reduced)

# Solution 3: Relax convergence threshold
estimate_relaxed_joint <- synthdid_estimate(Y_cov, N0, T0, X = X,
                                           min.decrease = 1e-2)

# Solution 4: Don't use covariates if not needed
estimate_no_cov <- synthdid_estimate(Y_cov, N0, T0)

Scenario 5: Complete Non-Convergence

Situation: Both lambda and omega fail to converge.

When it happens: - Very difficult problem - max.iter way too small - Numerical issues with data - Poorly scaled data

# Deliberately create a hard problem with too few iterations
set.seed(131415)
n_units <- 100
n_periods <- 80
n_treated <- 3
n_post <- 15

# High-dimensional, noisy data
Y_hard <- matrix(rnorm(n_units * n_periods, mean = 1000, sd = 500),
                n_units, n_periods)

# Add complex patterns
for (i in 1:n_units) {
  Y_hard[i, ] <- Y_hard[i, ] + cumsum(rnorm(n_periods, sd = 50))
}

Y_hard[(n_units - n_treated + 1):n_units, (n_periods - n_post + 1):n_periods] <-
  Y_hard[(n_units - n_treated + 1):n_units, (n_periods - n_post + 1):n_periods] + 300

# Estimate with very limited iterations
estimate_fail <- synthdid_estimate(Y_hard,
                                  N0 = n_units - n_treated,
                                  T0 = n_periods - n_post,
                                  max.iter = 50)  # Way too few!

cat("Overall convergence:", synthdid_converged(estimate_fail), "\n\n")
#> Overall convergence: FALSE
print(synthdid_convergence_info(estimate_fail))
#> Synthdid Convergence Diagnostics
#> =================================
#> 
#> Overall Status: NOT CONVERGED 
#> 
#> Lambda weights optimization:
#>  converged iterations max_iter utilization
#>      FALSE         50       50      100.0%
#> 
#> Omega weights optimization:
#>  converged iterations max_iter utilization
#>      FALSE         50       50      100.0%
#> 
#> Recommendation: Consider increasing max.iter or relaxing min.decrease threshold.

🚨 What to look for: - overall_converged: FALSE - Lambda: converged = FALSE, utilization = 100.0% - Omega: converged = FALSE, utilization = 100.0% - Both hitting max_iter

How to fix:

# Major increase in max.iter
estimate_fixed_both <- synthdid_estimate(Y_hard,
                                        N0 = n_units - n_treated,
                                        T0 = n_periods - n_post,
                                        max.iter = 10000)

cat("Fixed convergence:", synthdid_converged(estimate_fixed_both), "\n\n")
#> Fixed convergence: FALSE
print(synthdid_convergence_info(estimate_fixed_both))
#> Synthdid Convergence Diagnostics
#> =================================
#> 
#> Overall Status: NOT CONVERGED 
#> 
#> Lambda weights optimization:
#>  converged iterations max_iter utilization
#>       TRUE       1322    10000       13.2%
#> 
#> Omega weights optimization:
#>  converged iterations max_iter utilization
#>      FALSE      10000    10000      100.0%
#> 
#> Recommendation: Consider increasing max.iter or relaxing min.decrease threshold.

# Compare estimates (likely very different!)
cat("\nNon-converged estimate:", as.numeric(estimate_fail), "\n")
#> 
#> Non-converged estimate: 385.5473
cat("Converged estimate:    ", as.numeric(estimate_fixed_both), "\n")
#> Converged estimate:     392.2319
cat("Difference:            ", abs(as.numeric(estimate_fail) - as.numeric(estimate_fixed_both)), "\n")
#> Difference:             6.684582

Best Practices

1. Always Check Convergence

# GOOD: Check convergence
estimate <- synthdid_estimate(Y, N0, T0)
if (!synthdid_converged(estimate)) {
  warning("Estimate may be unreliable - optimization did not converge")
  print(synthdid_convergence_info(estimate))
}

# BAD: Ignore convergence
estimate <- synthdid_estimate(Y, N0, T0)
# Just use estimate without checking...

2. Start with Reasonable max.iter

# GOOD: Adequate iterations for most problems
estimate <- synthdid_estimate(Y, N0, T0, max.iter = 10000)

# ACCEPTABLE: Test run with fewer iterations
estimate <- synthdid_estimate(Y, N0, T0, max.iter = 1000)
if (!synthdid_converged(estimate)) {
  # Rerun with more iterations
  estimate <- synthdid_estimate(Y, N0, T0, max.iter = 10000)
}

# BAD: Too few iterations
estimate <- synthdid_estimate(Y, N0, T0, max.iter = 10)

3. Understand Iteration Requirements

Rules of thumb:

Small problems (N0 < 50, T0 < 30): max.iter = 1000-5000
Medium problems (N0 < 200, T0 < 100): max.iter = 5000-10000
Large problems (N0 > 200 or T0 > 100): max.iter = 10000-50000
With covariates: Add 50-100% to these numbers

4. When to Worry About Non-Convergence

# CHECK 1: How close to converging?
info <- synthdid_convergence_info(estimate)
if (!info$overall_converged) {
  if (info$lambda$utilization > 95 || info$omega$utilization > 95) {
    # Very close to limit - likely needs more iterations
    cat("RERUN with more iterations\n")
  }
}

# CHECK 2: Does the estimate change much?
estimate1 <- synthdid_estimate(Y, N0, T0, max.iter = 1000)
estimate2 <- synthdid_estimate(Y, N0, T0, max.iter = 10000)

relative_change <- abs(estimate2 - estimate1) / abs(estimate1)
if (relative_change > 0.05) {  # More than 5% change
  cat("IMPORTANT: Estimate changed significantly with more iterations\n")
  cat("Use the estimate with max.iter = 10000\n")
}

Computational Cost of Convergence Monitoring

The convergence monitoring system adds virtually zero overhead:

library(bench)

# Without convergence monitoring (hypothetical old version)
# vs. With convergence monitoring (current version)
benchmark <- bench::mark(
  synthdid_estimate(setup$Y, setup$N0, setup$T0, max.iter = 1000),
  iterations = 10,
  check = FALSE
)

print(benchmark)
# Overhead: < 0.01% (essentially unmeasurable)

The system uses only information already computed during optimization: - Iteration counter (already exists) - Convergence flag (single boolean check) - No extra objective evaluations - No extra memory allocations

Real-World Example: California Prop 99

Let’s apply convergence diagnostics to the canonical California Proposition 99 example:

# Load the California Prop 99 dataset
data("california_prop99")

# Convert to matrix format
setup_ca <- panel.matrices(california_prop99)

# Estimate treatment effect
estimate_ca <- synthdid_estimate(setup_ca$Y, setup_ca$N0, setup_ca$T0)

# Check convergence
cat("California Prop 99 Convergence:\n")
#> California Prop 99 Convergence:
cat("================================\n")
#> ================================
cat("Converged:", synthdid_converged(estimate_ca), "\n\n")
#> Converged: FALSE

print(synthdid_convergence_info(estimate_ca))
#> Synthdid Convergence Diagnostics
#> =================================
#> 
#> Overall Status: NOT CONVERGED 
#> 
#> Lambda weights optimization:
#>  converged iterations max_iter utilization
#>      FALSE      10000    10000      100.0%
#> 
#> Omega weights optimization:
#>  converged iterations max_iter utilization
#>      FALSE      10000    10000      100.0%
#> 
#> Recommendation: Consider increasing max.iter or relaxing min.decrease threshold.

# Show the estimate
cat("\nTreatment Effect Estimate:", as.numeric(estimate_ca), "\n")
#> 
#> Treatment Effect Estimate: -15.60379

Interpretation: - If converged: The estimate is reliable - If not converged: Rerun with increased max.iter before using

Summary

Quick Reference Table

Scenario	Lambda	Omega	Joint	Action
✅ Full convergence	✅	✅	-	Use estimate
⚠️ Lambda fails	❌	✅	-	Increase max.iter or relax tolerance
⚠️ Omega fails	✅	❌	-	Increase max.iter or increase eta.omega
⚠️ Joint fails	-	-	❌	Increase max.iter significantly
🚨 Both fail	❌	❌	-	Major increase in max.iter or check data

Key Functions

synthdid_converged(estimate) - Boolean: did it converge?
synthdid_convergence_info(estimate) - Detailed diagnostics
attr(estimate, "convergence") - Raw convergence data structure

Recommendations

Always check convergence before trusting estimates
Start with max.iter = 10000 for most problems
Increase iterations if convergence fails
Compare estimates with different max.iter if uncertain
Use diagnostics to understand which component needs more iterations

Appendix: Understanding the Frank-Wolfe Algorithm

The Frank-Wolfe algorithm solves:

$\min_{\lambda \geq 0, \sum \lambda = 1} \frac{1}{N_0} \|Y \lambda - b\|^2 + \zeta^2 \|\lambda\|^2$

Key properties:

Each iteration moves toward a simplex vertex
Step size is optimized via line search
Convergence criterion: objective decrease < threshold
Guaranteed to converge eventually, but may take many iterations

Why some problems are harder:

High dimensionality: More time periods → harder lambda
Many units: More control units → harder omega
Noise: High variance → needs more iterations
Complex patterns: Non-smooth data → slower convergence

Questions?

If you encounter convergence issues not covered here, please file an issue at: https://github.com/ZhenyaKosovan/synthdid/issues

synthdid package

2026-02-14

Introduction

The Convergence Monitoring System

Quick Convergence Check

Detailed Diagnostics

Scenario 1: Full Convergence

Scenario 2: Partial Convergence (Lambda Fails)

Scenario 3: Partial Convergence (Omega Fails)

Scenario 4: Joint Non-Convergence (With Covariates)

Scenario 5: Complete Non-Convergence

Best Practices

1. Always Check Convergence

2. Start with Reasonable max.iter

3. Understand Iteration Requirements

4. When to Worry About Non-Convergence

Computational Cost of Convergence Monitoring

Real-World Example: California Prop 99

Summary

Quick Reference Table

Key Functions

Recommendations

Appendix: Understanding the Frank-Wolfe Algorithm

Further Reading

Questions?