UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma

Try It: How Does Clipping Throttle Exploration?

Figure 1 below is an interactive map of the update budget a token receives before its gradient is clipped, drawn over the plane of the current policy probability \(\pi_\theta\) (x axis) and the behavior probability \(\pi_{\text{old}}\) (y axis). Switch Method between DAPO and UP-DAPO for positive advantages (\(\hat{A} > 0\)), inspect the shared non-positive regime (\(\hat{A} \le 0\)), drag \(\epsilon_{\text{high}}\), \(\epsilon_{\text{low}}\), and \(c\), and hover anywhere to read the exact capacity. The key contrast: DAPO ties the budget to \(\pi_{\text{old}}\) through a thin band, while UP-DAPO frees it to the full \(1 - \pi_\theta\), independent of \(\pi_{\text{old}}\).

Figure 1. Interactive Probability Capacity map (this paper's Figure 1). The color encodes the update budget Cap over the plane of the current policy probability π_θ (x axis) and the behavior probability π_old (y axis). For positive advantages, DAPO confines Cap to a thin band bounded by π_θ = (1 + ε_high)π_old, while UP-DAPO frees it to the full 1 − π_θ independent of π_old; for non-positive advantages both methods share the clipped wedge.

Background

From REINFORCE to GRPO, DAPO, and GSPO

RL has become the standard paradigm for eliciting complex reasoning from LLMs. The vanilla REINFORCE objective is strictly on-policy and yields a mathematically stable gradient:

\[\nabla_\theta J_{\text{REINFORCE}}(\theta) = \mathbb{E}_{q \sim Q,\, o \sim \pi_\theta}\!\left[ \sum_{t=1}^{|o|} \hat{A}\, \nabla_\theta \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) \right].\]

REINFORCE is stable but sample inefficient, since each trajectory is discarded after a single step. To reuse sampled data across multiple steps, GRPO and DAPO introduce importance sampling (IS), estimating the current policy \(\pi_\theta\) from samples drawn by a historical policy \(\pi_{\text{old}}\) through the ratio

\[r_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\pi_{\text{old}}(o_{i,t} \mid q, o_{i,<t})}.\]

To stop this ratio from exploding, GRPO bounds it inside a trust region \([1 - \epsilon, 1 + \epsilon]\) via clipping. DAPO decouples the bounds into \((\epsilon_{\text{low}}, \epsilon_{\text{high}})\) and drops the KL penalty, so it relies exclusively on clipping to trade off stability against exploration. GSPO instead moves to a sequence-level ratio, using the length-normalized likelihood of the whole response:

\[s_i(\theta) = \left( \frac{\pi_\theta(o_i \mid q)}{\pi_{\text{old}}(o_i \mid q)} \right)^{\frac{1}{|o_i|}} = \exp\!\left( \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \big( \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) - \log \pi_{\text{old}}(o_{i,t} \mid q, o_{i,<t}) \big) \right).\]

Every one of these methods keeps \(\pi_{\text{old}}\) in the denominator, and every one leans on clipping. That shared structure is exactly what creates the dilemma below.

Probability Capacity (Cap)

To make the cost of clipping precise, we define the Probability Capacity (Cap) as the maximum allowable increase in \(\pi_\theta\) for a token with positive advantage (\(\hat{A} > 0\)), or the maximum allowable decrease for a token with non-positive advantage (\(\hat{A} \le 0\)), before the optimization gradient is truncated. Cap is the effective budget a token has for policy exploration relative to \(\pi_{\text{old}}\). Figure 1 plots this budget directly, and the rest of this post is a story about how UP enlarges it.

Motivation

IS-based RL is caught between two failure modes. Pushing updates too aggressively (pure IS) triggers instability; constraining them too conservatively (clipping) stifles exploration. We examine each in turn.

Dilemma 1: Importance Sampling Induces Instability

The unclipped IS objective maximizes the advantage weighted by the ratio, \(J_{\text{IS}}(\theta) = \mathbb{E}[\hat{A}\, r_{i,t}(\theta)]\), whose gradient is

\[\nabla_\theta J_{\text{IS}}(\theta) = \mathbb{E}_{q \sim Q,\, o \sim \pi_{\text{old}}}\!\left[ \hat{A}\, r_{i,t}(\theta)\, \nabla_\theta \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) \right].\]

Comparing this to the REINFORCE gradient, the flaw is explicit: the IS gradient is scaled by \(r_{i,t}(\theta)\). For a rare, high-reward reasoning path where the behavior probability \(\pi_{\text{old}}\) is infinitesimally small, this ratio explodes as the active policy marginally improves. The pathological scaling destroys stable representations and drives training to diverge, which is precisely why clipping is treated as indispensable.

Dilemma 2: Clipping Stifles Exploration

For positive advantages, the clipped objective is \(J_{\text{IS+CLIP}}(\theta) = \min\!\big[r_{i,t}(\theta), 1 + \epsilon\big]\hat{A}\), and its gradient vanishes the moment the ratio exceeds the trust-region boundary. Mapping this to absolute probability space, the policy is upper-bounded by \(\pi_\theta \le (1 + \epsilon_{\text{high}})\pi_{\text{old}}\), so the remaining budget follows a piecewise Cap:

\[\mathrm{Cap}(\pi_\theta, \pi_{\text{old}}) = \begin{cases} \min\!\big(1,\, (1 + \epsilon_{\text{high}})\pi_{\text{old}}\big) - \pi_\theta & \text{if } \pi_\theta < (1 + \epsilon_{\text{high}})\pi_{\text{old}}, \\[4pt] 0 & \text{if } \pi_\theta \ge (1 + \epsilon_{\text{high}})\pi_{\text{old}}. \end{cases}\]

This exposes a structural bottleneck: the budget is linearly dependent on \(\pi_{\text{old}}\). For a low-likelihood reasoning token with \(\pi_{\text{old}} = 0.01\), a standard \(\epsilon_{\text{high}} = 0.28\) caps the absolute probability increase at a mere \(0.0028\). Once \(\pi_\theta\) reaches \(0.0128\), the capacity drops to zero, the token enters the Upper Clip region, and its gradient is nullified even when the advantage is exceptionally high. Set the method to DAPO in Figure 1 and read along the bottom of the map, where \(\pi_{\text{old}}\) is small, to see this sliver of budget for yourself. Structurally, this stifles exploration and leaves promising long-tail trajectories undiscovered.

Method

UP dismantles the dilemma with two mathematical moves: replace \(\pi_{\text{old}}\) with the current policy \(\pi_\theta\) via a stop-gradient, and route the gradient asymmetrically by the sign of the advantage.

An Unbounded Objective for Positive Advantages

For correct rollouts (\(\hat{A} > 0\)), we replace the IS ratio with a self-anchored ratio \(\tilde{r}_{i,t}(\theta)\) built from the stop-gradient operator \(\mathrm{sg}(\cdot)\) (Eq. 12):

\[J^{+}_{\text{UP}}(\theta) = \mathbb{E}_{q \sim Q,\, o \sim \pi_{\text{old}}}\!\left[ \sum_{t=1}^{|o|} \hat{A}\, \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})}{\mathrm{sg}\!\big(\pi_\theta(o_{i,t} \mid q, o_{i,<t})\big)} \right], \qquad \hat{A} > 0.\]

During backpropagation \(\mathrm{sg}(\pi_\theta)\) is a constant scalar, yet its forward value equals \(\pi_\theta\). Applying the log-derivative trick \(\nabla x / x = \nabla \log x\) collapses the objective to (Eq. 13):

\[\nabla_\theta J^{+}_{\text{UP}}(\theta) = \mathbb{E}_{q \sim Q,\, o \sim \pi_{\text{old}}}\!\left[ \sum_{t=1}^{|o|} \hat{A}\, \nabla_\theta \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) \right].\]

This is a profound simplification: optimizing the self-anchored ratio is mathematically equivalent to maximizing the REINFORCE objective. By anchoring the policy to \(\pi_\theta\) instead of \(\pi_{\text{old}}\), UP eradicates the root cause of IS instability from Dilemma 1, and its Cap becomes fully unconstrained by history, \(\mathrm{Cap} = 1 - \pi_\theta\). Switch the method to UP-DAPO with \(\hat{A} > 0\) in Figure 1: the thin clipped band expands into a full budget that depends only on \(\pi_\theta\).

Why Asymmetric: Keep Clipping for Negative Advantages

The unbounded mechanism is deliberately restricted to correct rollouts. For wrong rollouts the advantage is negative, which flips the gradient direction; applying an unbounded update there performs aggressive gradient ascent and destroys the original representation. UP therefore retains standard clipping for \(\hat{A} \le 0\), where DAPO and UP-DAPO share an identical Cap governed by the lower-clip and dual-clip boundaries (see the \(\hat{A} \le 0\) view in Figure 1):

\[\mathrm{Cap}(\pi_\theta, \pi_{\text{old}}) = \begin{cases} \pi_\theta - (1 - \epsilon_{\text{low}})\pi_{\text{old}} & \text{if } (1 - \epsilon_{\text{low}})\pi_{\text{old}} \le \pi_\theta \le c\,\pi_{\text{old}}, \\[4pt] 0 & \text{if } \pi_\theta < (1 - \epsilon_{\text{low}})\pi_{\text{old}} \quad (\text{Lower Clip}), \\[4pt] 0 & \text{if } \pi_\theta > c\,\pi_{\text{old}} \quad (\text{Dual Clip}). \end{cases}\]

This is the crux of UP: exploration for correct rollouts, safety for wrong ones.

UP-GxPO: One Plugin, Three Instantiations

Because it only rewrites the positive branch, UP plugs into any group-based policy optimizer. The token-level UP-DAPO combines the unbounded positive objective with the DAPO clipped objective for negatives (Eq. 14):

\[J_{\text{UP-DAPO}}(\theta) = \mathbb{E}\!\left[ \frac{1}{\sum_i |o_i|} \sum_{i}\sum_{t} \begin{cases} \hat{A}_{i,t}\, \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) & \text{if } \hat{A}_{i,t} > 0, \\[4pt] \min\!\big( r_{i,t}(\theta)\hat{A}_{i,t},\, \mathrm{clip}(r_{i,t}(\theta), 1 - \epsilon_{\text{low}}, 1 + \epsilon_{\text{high}})\hat{A}_{i,t} \big) & \text{if } \hat{A}_{i,t} \le 0. \end{cases} \right].\]

The same principle gives UP-GRPO, which keeps GRPO’s symmetric clipping and KL penalty on the negative branch (Eq. 15):

\[J_{\text{UP-GRPO}}(\theta) = \mathbb{E}\!\left[ \frac{1}{G} \sum_{i} \frac{1}{|o_i|} \sum_{t} \begin{cases} \hat{A}_i\, \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) - \beta D_{\text{KL}}(\pi_\theta \,\|\, \pi_{\text{ref}}) & \text{if } \hat{A}_i > 0, \\[4pt] \min\!\big( r_{i,t}(\theta)\hat{A}_i,\, \mathrm{clip}(r_{i,t}(\theta), 1 - \epsilon, 1 + \epsilon)\hat{A}_i \big) - \beta D_{\text{KL}}(\pi_\theta \,\|\, \pi_{\text{ref}}) & \text{if } \hat{A}_i \le 0. \end{cases} \right].\]

Finally, UP extends to the sequence level as UP-GSPO, whose positive branch provably reduces to a length-normalized REINFORCE gradient (Eq. 16):

\[J_{\text{UP-GSPO}}(\theta) = \mathbb{E}\!\left[ \frac{1}{G} \sum_{i} \begin{cases} \hat{A}_i \left( \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \log \pi_\theta(o_{i,t} \mid q, o_{i,<t}) \right) & \text{if } \hat{A}_i > 0, \\[4pt] \min\!\big( s_i(\theta)\hat{A}_i,\, \mathrm{clip}(s_i(\theta), 1 - \epsilon, 1 + \epsilon)\hat{A}_i \big) & \text{if } \hat{A}_i \le 0. \end{cases} \right].\]

Experiments

We evaluate UP on dense LLMs (Qwen3-14B-Base, Qwen3-8B), an MoE model (Qwen3-30B-A3B-Base), and a vision-language model (Qwen3-VL-8B-Instruct), using the verl framework. Across every setting UP raises accuracy and exploration while keeping training stable.

Performance, Exploration, and Stability Analysis of UP-DAPO

On Qwen3-14B-Base trained on DAPO-17K-MATH and evaluated on AIME24, UP-DAPO comprehensively outperforms DAPO. As shown in Figure 2, it lifts peak Avg@32 from \(47.71\) to \(51.15\) and Maj@32 from \(58.36\) to \(60.88\). Figure 3 traces the source of these gains: a consistently higher generation entropy alongside a higher Best@32 (from \(80.49\) to \(81.79\)) confirms that removing the \(\pi_{\text{old}}\) anchor genuinely enlarges exploration capacity. Crucially, Figure 4 shows the gradient norm and the KL divergence of UP-DAPO stay comparable to, or slightly below, DAPO’s, so this added exploration does not cost stability.

Figure 2. Performance comparison of DAPO and UP-DAPO on Qwen3-14B-Base during training: (a) Avg@32 and (b) Maj@32 accuracy on AIME24. UP-DAPO reaches a peak Avg@32 of 51.15 against DAPO's 47.71, and a peak Maj@32 of 60.88 against 58.36.

Figure 3. Exploration capacity of UP-DAPO: (a) entropy of generation probabilities on DAPO-17K-MATH and (b) Best@32 accuracy on AIME24. UP-DAPO sustains a higher entropy and lifts peak Best@32 to 81.79 against DAPO's 80.49.

Figure 4. Training stability of UP-DAPO: (a) gradient norm and (b) KL divergence between the active policy and the reference model on DAPO-17K-MATH. Both quantities track DAPO closely, so the extra exploration does not destabilize training.

Ablation Studies of UP-DAPO

Two ablations pin down why both design choices are necessary. As Figure 5 shows, simply relaxing DAPO’s upper clip to infinity while keeping the \(\pi_{\text{old}}\) anchor stays stable at first but explodes beyond 80 steps, with the gradient norm surging to \(10^{13}\); this shows the self-anchored ratio, not merely a loosened bound, is what enables safe unbounded updates. Applying the unbounded update symmetrically to both positive and negative advantages collapses within 25 steps, confirming that clipping on the negative branch is an indispensable safeguard.

Figure 5. Training stability of DAPO, DAPO with \(\epsilon_{\text{high}} = \infty\), REINFORCE (off-policy), and UP-DAPO on Qwen3-14B-Base: (a) gradient norm and (b) KL divergence, both on a log scale. DAPO with \(\epsilon_{\text{high}} = \infty\) explodes to \(10^{13}\) after about 80 steps, and the symmetric unbounded variant collapses within 25 steps, while UP-DAPO stays stable.

Comparison with Other RL Baselines

Under the unified protocol of Liu et al., we train Qwen3-8B on MATH (Levels 3 to 5) and evaluate Pass@1 across five reasoning benchmarks. As reported in Table 1, UP-GRPO attains the best average accuracy of \(61.31\%\), surpassing the strongest prior baseline GSPO (\(60.15\%\)) by an absolute \(1.16\%\). UP-GRPO ranks first or ties for first on four of the five individual benchmarks, and remains highly competitive on OlympiadBench, second only to ASPO. Consistent gains across benchmarks of varying difficulty show that the unbounded positive update is a broadly effective, rather than benchmark-specific, improvement over the GxPO family. Figure 6 explains these downstream gains: UP-GRPO’s policy entropy keeps rising throughout training while most baselines suffer entropy collapse, and its KL divergence stays among the lowest of the twelve methods.

Table 1. UP-GRPO against eleven RL baselines on AIME24, AMC23, MATH500, Minerva, and OlympiadBench (Pass@1, %), trained on MATH (Levels 3 to 5) with Qwen3-8B. Best per column in bold.

Figure 6. Exploration capacity and training stability of UP-GRPO against eleven RL baselines on Qwen3-8B during training: (a) entropy and (b) KL divergence between the active policy and the reference model on MATH (Levels 3 to 5). Most baselines suffer entropy collapse, while UP-GRPO keeps a rising entropy and holds its KL divergence among the lowest.

Universality of UP across Algorithms, Architectures, and Modalities

UP is a universal plugin, not a token-level trick. On the MoE model Qwen3-30B-A3B-Base, Figure 7 shows UP-GSPO raises peak Avg@32 on AIME24 from GSPO’s \(52.71\%\) to \(55.73\%\), an absolute \(3.02\%\), while keeping KL divergence nearly identical to the baseline. On the vision-language model Qwen3-VL-8B-Instruct trained on Geometry3K, Figure 8 shows UP-GRPO lifts accuracy from GRPO’s \(59.30\%\) to \(62.60\%\), an absolute \(3.30\%\), again without any added instability. Together these results show UP resolves the exploration-stability dilemma regardless of optimization granularity (token-level to sequence-level), model architecture (dense to MoE), or modality (language to multimodal).

Figure 7. Performance and stability comparison of GSPO and UP-GSPO on Qwen3-30B-A3B-Base (MoE): (a) Avg@32 accuracy on AIME24 and (b) KL divergence on DAPO-17K-MATH. UP-GSPO reaches a peak Avg@32 of 55.73 against GSPO's 52.71, an absolute gain of 3.02.

Figure 8. Performance and stability comparison of GRPO and UP-GRPO on Qwen3-VL-8B-Instruct: (a) accuracy on the Geometry3K test set and (b) KL divergence. UP-GRPO reaches 62.60 against GRPO's 59.30, an absolute gain of 3.30.

BibTeX

@misc{fan2026up,
      title={UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma},
      author={Chongyu Fan and Pengfei Liu and Jingjia Huang and Sijia Liu and Yi Lin},
      year={2026},
      eprint={2607.XXXXX},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2607.XXXXX},
}

@misc{fan2026up,
      title={UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma},
      author={Chongyu Fan and Pengfei Liu and Jingjia Huang and Sijia Liu and Yi Lin},
      year={2026},
      eprint={2607.XXXXX},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2607.XXXXX},
}