Reinforcement learning for robot assembly

Abstract

Robotic systems are traditionally employed in manufacturing to automate repetitive tasks such as welding, painting, and pick-and-place. Despite tremendous progress in robotics research, the classical assembly skill remains a challenge. In most cases, the difficult assembly skills still rely heavily on the engineer’s expertise [1]. In addition, the skills are prone to failure in the face of new tasks or variations, such as the shape or size of objects. This is particularly important as customer demand for greater product variety has recently increased. Learning approaches will become prominent in this context since learning shifts the burden from humans to the robot. Instead of attempting to obtain an accurate model of the surrounding environments or to program the controller, the robot can acquire a dynamics model or directly learn optimal control policies from experience. Reinforcement Learning endows a robot with the ability to find optimal behavior autonomously by interacting with its surrounding environment. The integration of deep learning models into RL, known as deep reinforcement learning, has gained significant traction and demonstrated remarkable achievements across various domains. However, contemporary deep reinforcement learning algorithms still encounter numerous challenges when applied in real-world robot manipulation. First, samples on a robotics system are expensive and tedious to obtain. Adding to this problem, model-free deep reinforcement learning algorithms are known to be sample inefficient, i.e., they require a large number of samples. Second, real-world training raises safety concerns. The environment or the engineer might impose several constraints that the robot must satisfy at all times to ensure safety. These constraints are difficult to maintain during the exploration phase, which often involves random action sampling. The two mentioned challenges are among the fundamental issues that prevent integrating deep reinforcement learning into robotics control systems. This thesis demonstrates how we can possibly improve sample efficiency and enable safe learning, making RL more practical for realistic robot tasks. Firstly, it demonstrates substantial improvement in sample efficiency by using manipulation primitives as actions. Manipulation primitives are simple yet generic enough to generalize across various tasks. Secondly, incorporating low-level feedback controllers into RL provides prior knowledge, which can increase learning speed and improve policy performance. A key message in this work is that a robust and high-performance low-level controller can further improve the robustness and performance of policies. Finally, this thesis examines methods to narrow the reality gap - the fundamental problem in sim-to-real reinforcement learning. This work proposes a novel contact reduction method to improve simulation accuracy, facilitating sim-to-real transfer for complex assembly tasks.