# CombEval: A Framework for Evaluating Combinatorial Counting in Large Language Models
> Ficha editorial pública de Research IA. Estado: Lectura primaria completa. La interpretación editorial no sustituye la fuente primaria.

- Página canónica: https://luiseduardodemiguel.com/research-ia/papers/combeval-a-framework-for-evaluating-combinatorial-counting-in-large-lang
- Fuente primaria: https://arxiv.org/abs/2606.19788
- Versión leída: v1
- Fuente comprobada: 2026-08-19 · lectura primaria completa; extracción editorial automatizada, revisión humana pendiente
- Autores: Yuxu Zhou, Ondřej Kuželka, Yuyi Wang, Yuanhong Wang, Yi Chang
- Fecha del corte: 18 JUNIO 2026.
- Área: EVALUACIÓN

## Tesis y contexto

Framework para evaluar conteo combinatorio en LLMs.

- Problema: Muchos modelos fallan en razonamiento discreto aunque acierten tareas lingüísticas.
- Por qué importa: Conteo y combinatoria son base de planificación, análisis de casos, testing y optimización.

## Evidencia reportada

- **reported-result**: Can the generated combinatorial counting problems in CombEval faithfully reflect the capability hierarchy of current LLMs, producing performance trends consistent with existing benchmarks where stronger models achieve higher accuracy? [localizador](https://arxiv.org/html/2606.19788#S5)
- **reported-result**: Stronger models with larger parameter scales or advanced reasoning mechanisms, such as gpt-5.5 and gemini-3-flash-preview-thinking , achieve substantial performance advantages in both settings, whereas smaller models show clear limitations. [localizador](https://arxiv.org/html/2606.19788#S5)
- **reported-result**: As shown in Figure 3 , both factors consistently increase problem difficulty. [localizador](https://arxiv.org/html/2606.19788#S5)
- **reported-result**: Adding more constraints leads to a clear accuracy drop for most models, indicating that membership, positional, and relative-order constraints effectively disrupt simple counting patterns. [localizador](https://arxiv.org/html/2606.19788#S5)

## Lectura y límite

- Método: La lectura de 2 Related Work describe la intervención y su construcción: Mathematical reasoning benchmarks such as GSM8K 5 and MATH 16 have established a foundation for evaluating arithmetic and algebraic capabilities including combinatorial counting in LLMs, but they also face significant limitations. Primarily, these static datasets are highly susceptible to data contamination. As pre-training corpora expand, test questions are increasingly memorized rather than solved, distorting evaluation results 3 ; 15 ; 51 . While researchers have attempted to mitigate this through dynamic generation in other reasoning fields 30 ; 28 ; 41 ; 24 ; 42 ; 2 , constructing benchmarks specifically…
- Límite: La lectura primaria permite comprobar método y resultados en el HTML, pero no convierte sus conclusiones en validación independiente. La ficha no demuestra transferencia fuera de los datasets, modelos, herramientas y condiciones descritos en 5 Experiments.
- Confianza editorial: Media
- Limitación: El cierre de la fuente señala: We present CombEval, an framework for dynamically generating diverse and difficulty-controllable combinatorial counting problems, used to evaluate the combinatorial counting capabilities of large language models. Experimental results on CombEval reveal that current LLMs, including state-of-the-art closed-source models, still struggle with various combinatorial counting challenges, particularly those involving indistinguishable elements, complex sequencing…
- Limitación: La ficha no demuestra transferencia fuera de los datasets, modelos, herramientas y condiciones descritos en 5 Experiments.

## Localizadores de evidencia
- [Fuente primaria · canonical](https://arxiv.org/abs/2606.19788): tipo abstract
- [HTML · lectura completa](https://arxiv.org/html/2606.19788): tipo abstract
- [Método · 2 Related Work](https://arxiv.org/html/2606.19788#S2): tipo section
- [Evaluación · 5 Experiments](https://arxiv.org/html/2606.19788#S5): tipo section
- [Cierre · 6 Conclusion](https://arxiv.org/html/2606.19788#S6): tipo section

## Próxima prueba

- ¿La propuesta mejora evaluación de razonamiento frente a la línea base actual?
- Métrica: Comparar la métrica principal de la fuente junto con calidad, coste, latencia y tasa de errores.
- Regla de parada: Parar si no aparece una mejora reproducible o si aumenta el riesgo, la complejidad o el coste sin compensación.

## Recursos reproducibles
- [https://github.com/YuxuZhou-CN/combination-problem-generation](https://github.com/YuxuZhou-CN/combination-problem-generation)
- [Link](https://www.kaggle.com/datasets/hemishveeraboina/aime-problem-set-1983-2024)
- [the following issues](https://github.com/arXiv/html_feedback/issues)
- [list of packages that need conversion](https://github.com/brucemiller/LaTeXML/wiki/Porting-LaTeX-packages-for-LaTeXML)

## Enlaces relacionados

- [VAKRA](https://luiseduardodemiguel.com/research-ia/markdown/papers/vakra)
- [VibeLifeBench](https://luiseduardodemiguel.com/research-ia/markdown/papers/vibelifebench)
- [KnowHal](https://luiseduardodemiguel.com/research-ia/markdown/papers/knowhal)