Assignment #11

Provide a short discussion of each of the assigned papers (listed under Course Materials). Below are some questions to think about.

Visual Foresight
(feel free to skip the details of the meta-learning in Section 5.3, as we will cover that topic next class)
Questions

• The paper discusses the problem of unsupervised data collection, where random actions are used to collect the data. What would be some strategies for doing better than random? Try to draw on ideas from papers we saw earlier in the class.
• Section 4 describes the DNA and SNA models for predicting pixel motion. What is the advantage of the SNA model? What is a case where the SNA model would still fail, and how could you augment the method to handle your case?
• Compare and contrast the different cost functions based on: (1) pixel-distance, (2) image-to-image registration, (3) classifier-based. Which do you think is most promising? (Section 5.4 gives some commentary, but try to come up with your own opinion here.)
• This paper argues that learning a visual prediction model, then doing MPC, allows for better generalization than directly learning a control policy. What's the intuition behind that? Do you agree with the argument?

World Models

Questions

For the following three questions, please see the interative demos here:

• Try playing with the z sliders. What are these sliders doing?
• Try changing the Tau parameter (second set of demos). What happens and why? The paper says that high Tau makes the model harder to exploit. Why is that?
• In the VizDoom interactive demo, you can play the game entirely inside the network's hallucination. Can you beat the policy reported in Table 2 of the paper? What's your highest score?

Additional questions:

• Compare and contrast this method with Dyna.
• Do you think this is what our brains are doing when we dream? Notice that the model needed thousands of mental rollouts to play a really simple version of Doom. I feel like I only have a couple dreams per night. Are our brains really that much more sample efficient at learning from mental simulation? Or is dreaming not related to learning? Also, are our dreams so strange because Tau is ramped up?

Upload a single PDF file through Stellar by Mar 31 at 10 am.