Value-Based Methods – Q-Learning, SARSA, Deep Q-Networks (DQN)

• Q-Learning and SARSA (on-policy vs. off-policy)
• DQN: replay buffer, target networks, instability challenges

Policy-Based Methods – REINFORCE and Actor-Critic

• Policy gradients methods
• Actor-Critic architecture and advantages in continuous spaces

Deep RL and Function Approximation

• Neural network regression for Q functions
• Stability, convergence, and error sources (bias vs. variance)

Advanced Algorithms – PPO, DDPG, TD3, SAC

• Proximal Policy Optimization (PPO) and Trust Region concepts
• Deep Deterministic Policy Gradient (DDPG) and its challenges
• Twin Delayed DDPG (TD3) improvements
• Soft Actor-Critic (SAC) for robust, sample-efficient learning

Hierarchical RL and Transfer Learning

• Multi-level policy design for complex tasks
• Transfer learning techniques for RL agents

Multi-Agent Reinforcement Learning

• Fundamentals of cooperation and competition
• Communication, coordination, and scalability challenges

Hyperparameter Tuning & Best Practices

• Neural network architecture, learning rate schedules
• Reward shaping and exploration–exploitation trade-offs

Capstone Project: Real-World Simulation Case and Final Pitch

• Definition of a control/simulation problem
• Collaborative solution development (pairs or teams)
• Final pitch: performance metrics, results, and lessons learned