Assignment #16
Provide a short discussion of each of the assigned papers (listed under Course Materials). Below are some questions to think about.
Karl Sims 2
Questions
- Compare and contrast this with the first Karl Sims paper we read (link). What's fundamentally different about the fitness function in the latter paper? What's an advantage of using competition as the fitness function? A disadvantage?
- Sims evaluates different design choices based on subjective judgment (how "interesting" is the resulting behavior). Can you think of more objective metrics? How could we decide if one design choice (e.g., tournament pairings) is better than another (e.g., random pairings)?
Emergent complexity
Questions
- We previously saw the idea of experience replay (e.g., here). Compare and contrast this with the idea of historical opponent sampling.
- Section 4.1 discusses the problem of sparse rewards in asymmetric games (e.g., kick-and-defend). Is this a problem in symmetric games as well (e.g., sumo)?
- The games considered in this paper lead to rather narrow and task-specific skills, such as kicking a ball. What do you think would be a good competitive game that would lead to the emergence of more general skills or intelligence?
Deep neuroevolution
Note: the random search method described in this paper is
not the same as the
"random search" referred to in the optimization community. This paper matches the colloqiual usage of the term: try a bunch of random parameter settings and pick the one that performs best. In the optimization community,
"random search" refers to a hill-climbing algorithm that approximates the gradient with finite differences (evolution strategies is one example of this kind of "random search").
Questions
- Why do you think random search (as defined in this paper) works so well? Are the tasks too easy? Are RL algorithms really bad? Is random search actually smart?
- Suppose you use algorithm 1 to optimize a Q function, rather than directly optimizing for a policy. What might be some advantages and disadvantages?
Upload a single PDF file through Stellar by Apr 11 at 10 am.