Course Overview

Malware reverse engineering involves deep analysis of the code, structure, and functionality of malicious software. The goal of this course is to provide a solid foundation in reverse engineering, which is crucial in understanding modern malware and crafting solutions for the remediation and prevention of cyber attacks.

This course exposes students to an immersive, hands-on experience in malware analysis and examines a wide range of software security topics relating to operating systems, debugging, and software protection.

Why take this course?: You are interested in learning the fundamental principles of dissecting malware and cyber attack triage. Practicing software reverse engineering is also useful for creating interoperable software, for verifying that software and software patches function as promised, and for the simple joy of understanding how software executes at a deep level.

Basic Information

Class location: Van Leer C456
Class day/time: Tues. & Thurs. 12:00 pm to 1:15 pm

Instructor: Professor Brendan Saltaformaggio
Office: Klaus 2314
Email: brendan@ece.gatech.edu
Office hours: Tues. & Thurs. 1:15 pm to 2:15 pm in Klaus 2314, or by appointment

TA: Yotam Mosinzon
Email: yotam@gatech.edu
Office hours: Mon. & Wed. 1:30 pm to 3:00 pm in Van Leer 449 cubicle B, or by appointment

Canvas (Lecture Slides, Assignments, Grades): https://canvas.gatech.edu
Piazza (Discussion, Questions, Announcements): https://piazza.com/gatech/spring2018/ece4894a

Materials

There is no required textbook for this course. The course will be driven by a sequence of hands-on reverse engineering exercises which emphasize the discovery, understanding, and mitigation of common malware tactics --- the devil really is in the details.

The following books are recommended for additional background or more in-depth understanding of the topics discussed in class. Read these books only if you want to learn more! They will not be covered in lectures or on exams!

• Practical Tools/Techniques For Malware Reverse Engineering:
  Michael Sikorski, Andrew Honig. Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software. No Starch Press, 2012. ISBN: 978-1593272906
• A Great Reference For IDA Pro:
  Chris Eagle. The IDA Pro Book. No Starch Press (2nd Edition), 2011. ISBN: 978-1593272890
  
• Background On Low-Level Computer Systems Programming:
  Randal E. Bryant, David R. O'Hallaron. Computer Systems: A Programmer's Perspective. Pearson (3rd Edition), 2015. Online: http://csapp.cs.cmu.edu/. ISBN: 978-0134092669

You may also need a copy of the Intel Developer’s manuals. These are free and available via this link: http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html
It's large, but the best PDF to get is the combined set, downloadable via the first link on that page. If you have an iPad or other tablet, drop this PDF on it and read it whenever you have spare time.

Course Outline

1. Reverse Engineering Principles
   1. Intro. to Malware and Assembly Language
      1. Under what circumstances is reverse engineering useful or breaking contracts?
      2. Why is reverse engineering necessary?
         1. Interoperability/Competition
         2. Auditing
         3. DRM
         4. Analysis of Malware
   2. Background on Malware
      1. Current and Next-Generation Malicious Software
         1. Viruses
         2. Worms
         3. Trojans
         4. Botnets
         5. Polymorphic and Metamorphic Malware
         6. Advanced Persistent Threats
      2. Intro to Defensive Strategies Against Malware
         1. Worm Fingerprinting/Signature Generation
         2. Behavioral Approaches to Detection of Malware
         3. Hardware Agents for System Integrity Checking
2. Low level Software
   1. Overview of Intel Assembly Language
   2. Virtual Machines for Interpreted High-Level Languages
   3. Representation of Compiled High Level Language Structures in Assembly
   4. Operating Systems Background
      1. MS-DOS Internals Related to Malware Case Studies
      2. Modern Windows Execution Environment
   5. Executable File Formats
      1. PE Files
         1. Import Address Table
3. Analysis of Malicious Software
   1. System Monitoring Tools
   2. Dynamic Tracing: System Calls, Filesystem, and Registry
   3. Compiler Issues
   4. Debuggers
      1. OllyDbg
      2. WinDbg
   5. Disassemblers
      1. IDA Pro
      2. Decompilers
   6. Memory Analysis to Support Reverse Engineering
      1. RAM Acquisition
      2. Extraction of Malware
4. Advanced Reverse Engineering Techniques
   1. Encrypted/Packed Executables
      1. Unpacking Case Studies
   2. Anti-Debugging Techniques
   3. Anti-VM Techniques
   4. Code Obfuscation
5. Remediation of Advanced Persistent Threats
   1. Determination of Malicious Behaviors
   2. Analysis of Decompiled Source Code
   3. Revelation of Command and Control Functionalities

Assignments & Grading

There will be 14 lab assignments which are designed to progressively introduce new and important features that a reverse engineer might encounter when analyzing modern malware. Labs will be conducted in teams of 2. Most labs will focus on static analysis with IDA Pro. Each assignment will require very careful time allocation to complete by the deadline (often 1 or 2 weeks each).

Lab Topics:

1. Software Disassembly
2. Malware Case Study (1) - Viruses
3. Malware Case Study (2) - Simple Obfuscation
4. Rapid Analysis of Suspicious Executables (In Class)
5. Malware Case Study (3) - Code Injection
6. Identification of High Level Language Structures in Assembly
7. PE File Analysis (In Class)
8. Intro. to Encrypted/Packed Malware
9. Unpacking Encrypted/Packed Malware
10. Anti-Debugging/Polymorphic Techniques
11. Code Deobfuscation
12. Malware in Embedded Devices
13. Advanced Persistent Threats (1) - Persistence
14. Advanced Persistent Threats (2) - Command & Control

The midterm and final exams will focus exclusively on the malware analyses conducted in the labs. For example, a typical question will target a particularly tricky section of the disassembly of a familiar malware sample. If you actively participated in the labs, then the questions should be answerable within the allowed time based on your previous analyses.

Grading Breakdown:

• 50% for the 14 labs
• 20% for the midterm exam
• 30% for the final exam

Grading Scheme:

• A 100% to 90.0%
• B 90.0% to 80.0%
• C 80.0% to 70.0%
• D 70.0% to 60.0%
• F below 60.0%

Learning Objectives

• Develop a mastery of low-level software primitives and executable inspection
• Learn and apply the fundamental principles of dissecting malware
• Become aware of limitations of existing reverse engineering mechanisms and how to overcome them
• Study the behaviors, tricks, and tactics employed by modern malware
• Engage in critical discussion around key topics in software security and malware prevention

Honor Code

Students are expected to abide by the Georgia Tech Academic Honor Code. Honest and ethical behavior is expected at all times. All incidents of suspected dishonesty will be reported to and handled by the Office of Student Integrity. You will have to do all assignments individually unless explicitly told otherwise. You may discuss with classmates but you may not copy any solution (or any part of a solution).

Learning Accommodations

Whenever needed, the instructor will make accommodations for students with documented disabilities. These accommodations must be arranged in advance and in accordance with the Office of Disability Services.