Preprint
Reinforcement Learning

Sigmoid-weighted linear units for neural network function approximation in reinforcement learning

Stefan Elfwing(Advanced Telecommunications Research Institute International), Eiji Uchibe(Okinawa Institute of Science and Technology Graduate University), Kenji Doya(Okinawa Institute of Science and Technology Graduate University)
January 11, 2018Neural Networks2,626 citations
arXivPDFDOI

2.6k

Citations

102

Influential Citations

Neural Networks

Venue

2018

Year

Abstract

In recent years, neural networks have enjoyed a renaissance as function approximators in reinforcement learning. Two decades after Tesauro's TD-Gammon achieved near top-level human performance in backgammon, the deep reinforcement learning algorithm DQN achieved human-level performance in many Atari 2600 games. The purpose of this study is twofold. First, we propose two activation functions for neural network function approximation in reinforcement learning: the sigmoid-weighted linear unit (SiLU) and its derivative function (dSiLU). The activation of the SiLU is computed by the sigmoid function multiplied by its input. Second, we suggest that the more traditional approach of using on-policy learning with eligibility traces, instead of experience replay, and softmax action selection can be competitive with DQN, without the need for a separate target network. We validate our proposed approach by, first, achieving new state-of-the-art results in both stochastic SZ-Tetris and Tetris with a small 10 × 10 board, using TD(λ) learning and shallow dSiLU network agents, and, then, by outperforming DQN in the Atari 2600 domain by using a deep Sarsa(λ) agent with SiLU and dSiLU hidden units.

Analysis

Why This Paper Matters

This paper, published in 2018, addresses two critical aspects of deep reinforcement learning: activation functions and the algorithmic framework. At the time, DQN had set a high bar with experience replay and target networks, but the authors challenged the necessity of these components. By proposing SiLU (later popularized as Swish) and its derivative dSiLU, they introduced a smooth, non-monotonic activation that improved gradient flow and representation learning. This was a significant departure from the standard ReLU and its variants, and it opened the door for more research into activation functions tailored for RL.

The paper also demonstrated that on-policy methods with eligibility traces (Sarsa(λ)) could match or exceed DQN's performance without the complexity of experience replay and target networks. This was a surprising result that encouraged the community to revisit on-policy algorithms and simpler architectures. The success in Tetris and Atari showed that these methods are not just theoretical but practically competitive, influencing later work on sample-efficient and stable RL algorithms.

Technical Contributions

  • SiLU activation: Defined as f(x) = x * sigmoid(x), it is smooth, non-monotonic, and has a self-stabilizing property. Its derivative, dSiLU, is also smooth and can be used directly as an activation function.
  • Integration with TD(λ) and Sarsa(λ): The authors showed that these activations work well with on-policy learning and eligibility traces, achieving strong performance without replay buffers or target networks.
  • Shallow and deep architectures: They validated the approach on both shallow networks for Tetris and deep networks for Atari, demonstrating scalability.
  • Softmax action selection: Used instead of epsilon-greedy, which helped in exploration and policy improvement.

Results

The paper reports state-of-the-art results in stochastic SZ-Tetris and Tetris (10x10) using TD(λ) with shallow dSiLU networks. In the Atari 2600 domain, the deep Sarsa(λ) agent with SiLU and dSiLU hidden units outperformed DQN on the games tested. While the abstract does not provide specific scores, the claim of outperforming DQN is significant given DQN's strong baseline. The results suggest that the combination of SiLU/dSiLU and on-policy learning can be more effective than the DQN framework in certain settings.

Significance

The introduction of SiLU has had a lasting impact beyond RL; it was later rediscovered as Swish and became a popular activation function in supervised learning due to its simplicity and effectiveness. The paper also contributed to the ongoing debate about the necessity of experience replay and target networks, showing that simpler on-policy methods can be competitive. This has inspired research into more efficient and stable RL algorithms, such as those using eligibility traces and softmax policies. The work remains highly cited (over 2600 citations) and is a key reference for both activation function design and on-policy deep RL.

reinforcement-learningactivation-functionssiludsilusarsatd-lambdaataritetris
Reinforcement LearningNeural Networks

Related Papers