Durgesh Kalwar

I am a PhD student in Computer Science at Arizona State University, advised by Subbarao Kambhampati. My research focuses on understanding the strengths and limitations of Large Reasoning Models (LRMs): the mathematical frameworks behind RL-based post-training of LLMs, the effect of RL post-training on the semantics of intermediate reasoning traces, and post-training methods for more efficient inference. I also work on agentic systems, evaluating how LLM agents coordinate on collaborative tasks under information asymmetry and developing co-training methods to improve that coordination.

Before my PhD, I was a researcher at TCS Research in Mumbai (2021–2024), working with Harshad Khadilkar on sample-efficient and safe reinforcement learning for real-world decision-making, including directed exploration with General Value Functions, offline RL guided by a safety expert, and interpretable inventory control policies for supply chains.

I did my M.Tech. in Machine Learning and Computing from Indian Institute of Space Science and Technology. My thesis was on Safe Exploration in Reinforcement Learning and it mainly focused on safe sequential optimization in switching environments.

I pursued my undergraduate in Electrical Engineering from S. V. National Institute of Technology, India. My thesis was on State Estimation and Control Experiments on Quadrotor, in which I designed and implemented software on an STM32 microcontroller, which incorporated a Kalman filter and enhanced PID control algorithm. I then worked as an electrical maintenance engineer at Cairn Oil and Gas in Barmer, India.

Email  /  CV  /  GitHub  /  Google Scholar  /  LinkedIn  /  X  /  Transcript

Research

I'm interested in large reasoning models, RL-based post-training of LLMs, efficient inference and coordination between LLM agents, as well as reinforcement learning, optimization and distributed learning more broadly. Selected publications are below (* denotes equal contribution); the complete list is on Google Scholar.

Masked Self-Distillation: Internalizing the Chain-of-Thought in Language Models

In this paper, we propose masked self-distillation, a knowledge-distillation based post-training framework in which copies of the same model act as teacher and student, and the student learns to internalize all or part of the intermediate trace. Post-training Qwen3-4B and 8B models on math and graph coloring this way improves task performance while increasing inference efficiency.

Correct Answers, Invalid Traces: What Verifiable Grade-School Math Reveals About Chain-of-Thought Traces

In this paper, we use iGSM, a synthetic grade-school math benchmark whose traces can be checked programmatically step by step, to test whether correct answers are reliably accompanied by valid chain-of-thought traces. Answer correctness and trace validity nearly coincide in distribution but decouple out of distribution: on the hardest instances, 31.6% of correct answers have invalid traces.

Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!

In this position paper, we present evidence that calling intermediate tokens “reasoning” or “thinking” traces is not a harmless metaphor: it confuses the nature of these models and how to use them effectively, and leads to questionable research. We call on the community to avoid such anthropomorphization.

NestRL: A Nested Training Regime for Mutual Adaptation in Human-AI Teaming

In this paper, we formulate human-AI teaming as an Interactive POMDP and propose NestRL, a nested training regime that trains agents at each level against adaptive agents from the level below. In the Overcooked domain, NestRL achieves higher task performance with both unseen adaptive agents and real human teammates, while showing significantly greater adaptability over the course of interaction.

RL in Name Only? Analyzing the Structural Assumptions in RL post-training for LLMs

In this paper, we critically examine the structural assumptions made when modeling LLM post-training as an MDP, and show that they lead to a degenerate MDP under which the GRPO objective reduces to Filtered Iterative SFT, an on-policy variant of supervised fine-tuning. On benchmarks including GSM8K and Countdown, Filtered Iterative SFT achieves performance comparable to GRPO-based training.

Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains

In this paper, we study the effect of RL post-training on intermediate tokens by introducing trace coherence, a First-Order Logic based measure of the consistency of reasoning steps. We find that RL post-training improves local coherence without necessarily producing valid or correct solutions, so claims of improved reasoning via RL must be examined with care.

Efficiency Boost in Decentralized Optimization: Reimagining Neighborhood Aggregation with Minimal Overhead

In this paper, we introduce DynaWeight, a framework for information aggregation in decentralized learning that dynamically allocates weights to neighboring servers based on their relative losses on local datasets, instead of using static schemes such as Metropolis weights. It offers substantial acceleration with minimal additional communication and memory overhead, and works with any underlying server-level optimization algorithm.

Extracting Heuristics from Large Language Models for Reward Shaping in Reinforcement Learning

In this paper, we use off-the-shelf LLMs to generate a plan for an abstraction of the underlying MDP, and use that plan as a heuristic to construct a reward shaping signal for the downstream RL agent. Experiments on the BabyAI, Household, Mario and Minecraft domains show a significant improvement in the sample efficiency of PPO, A2C and Q-learning.

Guiding Offline Reinforcement Learning Using a Safety Expert

In this paper, we proposed an approach called GORL (Guided Offline-RL) that leverages a safety expert to guide the offline RL agent in choosing safe actions in uncertain states. This approach can be applied to existing offline RL algorithms and has been shown to outperform current methods in terms of safety without sacrificing performance.

Using General Value Functions to Learn Domain-Backed Inventory Management Policies

In this paper, we utilize GVFs to learn domain-backed inventory management policies. GVFs are trained on domain-critical characteristics such as prediction of stock-out probability and wastage quantity. Using this domain expertise for more effective exploration, we train an RL agent to compute the inventory replenishment quantities for a large range of products (up to 6000), which share aggregate constraints such as the total weight/volume per delivery. Additionally, we showed that the GVF predictions can be used to provide additional domain-backed insights into the decisions proposed by the RL agent.

Follow your Nose: Using General Value Functions for Directed Exploration in Reinforcement Learning

In this paper, we proposed a General Value functions (GVFs) based exploration strategy to effectively explore in a directed manner in hard exploration tasks.

Safe Sequential Optimization in Switching Environments

In this paper, we considered the problem of designing a sequential decision-making agent to maximize an unknown time-varying function that switches with time. The agent is also constrained to make safe decisions with a high probability during the exploration phase. For this safe sequential optimization problem, we proposed an algorithm called Adaptive-SafeOpt, which incorporates Bayesian optimization and change point detection.

Safe Exploration in Reinforcement Learning

Awards

• Fulton Fellowship (2024), School of Computing and AI, Arizona State University
• Top 10% rank in JEE Advanced 2014 among 150,000 candidates
• 99.97 percentile in JEE Main 2014 among 1.2 million students

Design and source code from Jon Barron's website