Introduction to Malware Reverse Engineering

Course Overview

Malware reverse engineering involves deep analysis of the code, structure, and functionality of malicious software. The goal of this course is to provide a solid foundation in reverse engineering, which is crucial in understanding modern malware and crafting solutions for the remediation and prevention of cyber attacks.

This course exposes students to an immersive, hands-on experience in malware analysis and examines a wide range of software security topics relating to operating systems, debugging, and software protection.

Why take this course?: You are interested in learning the fundamental principles of dissecting malware and cyber attack triage. Practicing software reverse engineering is also useful for creating interoperable software, for verifying that software and software patches function as promised, and for the simple joy of understanding how software executes at a deep level.

Basic Information

Class location: Scheller College of Business 200 (map)
Class day/time: Wed. and Fri. 3:30 pm to 4:45 pm

Instructor: Professor Brendan Saltaformaggio
Office: CODA S0923
Email: brendan@ece.gatech.edu
Office hours: Wed. and Fri. 4:45 pm to 6:00 pm in CODA S0923 (or on the walk over), or any time by appointment

TA: Anirudh Gattu
Email: anirudh.gattu@gatech.edu
Office hours: TBD

Canvas (Lecture Slides, Assignments, Grades): https://canvas.gatech.edu
Piazza (Questions, Announcements, Chat): Link In Canvas.

Materials

There is no required textbook for this course. The course will be driven by a sequence of hands-on reverse engineering exercises which emphasize the discovery, understanding, and mitigation of common malware tactics --- the devil really is in the details.

The following books are recommended for additional background or more in-depth understanding of the topics discussed in class. Read these books only if you want to learn more! They will not be covered in lectures or on exams!

Practical Tools/Techniques For Malware Reverse Engineering:
Michael Sikorski, Andrew Honig. Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software. No Starch Press, 2012. ISBN: 978-1593272906
A Great Reference For IDA Pro:
Chris Eagle. The IDA Pro Book. No Starch Press (2nd Edition), 2011. ISBN: 978-1593272890
Background On Low-Level Computer Systems Programming:
Randal E. Bryant, David R. O'Hallaron. Computer Systems: A Programmer's Perspective. Pearson (3rd Edition), 2015. Online: http://csapp.cs.cmu.edu/. ISBN: 978-0134092669

You may also need a copy of the Intel Developer’s manuals. These are free and available via this link: http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html
It's large, but the best PDF to get is the combined set, downloadable via the first link on that page. If you have an iPad or other tablet, drop this PDF on it and read it whenever you have spare time.

Course Outline

Reverse Engineering Principles
1. Intro. to Malware and Assembly Language
  1. Under what circumstances is reverse engineering useful or breaking contracts?
  2. Why is reverse engineering necessary?
    1. Interoperability/Competition
    2. Auditing
    3. DRM
    4. Analysis of Malware
2. Background on Malware
  1. Current and Next-Generation Malicious Software
    1. Viruses
    2. Worms
    3. Trojans
    4. Botnets
    5. Polymorphic and Metamorphic Malware
    6. Advanced Persistent Threats
  2. Intro to Defensive Strategies Against Malware
    1. Worm Fingerprinting/Signature Generation
    2. Behavioral Approaches to Detection of Malware
    3. Hardware Agents for System Integrity Checking
Low-Level Software
1. Overview of Intel Assembly Language
2. Virtual Machines for Interpreted High-Level Languages
3. Representation of Compiled High-Level Language Structures in Assembly
4. Operating Systems Background
  1. MS-DOS Internals Related to Malware Case Studies
  2. Modern Windows Execution Environment
5. Executable File Formats
  1. PE Files
    1. Import Address Table
Analysis of Malicious Software
1. System Monitoring Tools
2. Dynamic Tracing: System Calls, Filesystem, and Registry
3. Compiler Issues
4. Debuggers
  1. OllyDbg
  2. WinDbg
5. Disassemblers
  1. IDA Pro
  2. Decompilers
6. Memory Analysis to Support Reverse Engineering
  1. RAM Acquisition
  2. Extraction of Malware
Advanced Reverse Engineering Techniques
1. Encrypted/Packed Executables
  1. Unpacking Case Studies
2. Anti-Debugging Techniques
3. Anti-VM Techniques
4. Code Obfuscation
Remediation of Advanced Persistent Threats
1. Determination of Malicious Behaviors
2. Analysis of Decompiled Source Code
3. Revelation of Command and Control Functionalities

Assignments & Grading

There will be 10 lab assignments which are designed to progressively introduce new and important features that a reverse engineer might encounter when analyzing modern malware. Labs will be conducted in teams of 2. Most labs will focus on static analysis with IDA Pro. Each assignment will require very careful time allocation to complete by the deadline (often 1 or 2 weeks each). Lab due dates may flex throughout the semester, but labs will always be due at the time class begins on a class day.

Lab Topics:

Software Disassembly
Malware Case Study (1) - Viruses
Malware Case Study (2) - Simple Obfuscation
Malware Case Study (3) - Code Injection
Rapid Analysis of Suspicious Executables (In Class)
Intro. to Encrypted/Packed Malware
Anti-Debugging/Polymorphic Techniques
Unpacking Encrypted/Packed Malware (In Class)
Malware in IoT Devices
Advanced Persistent Threats

The midterm and final exams will focus exclusively on the malware analyses conducted in the labs. For example, a typical question will target a particularly tricky section of the disassembly of a familiar malware sample. If you actively participated in the labs, then the questions should be answerable within the allowed time based on your previous analyses.

Grading Breakdown:

50% for the 10 labs
20% for the midterm exam
20% for the final exam
10% for class/piazza participation

Grading Scheme:

A 100% to 90.0%
B 90.0% to 80.0%
C 80.0% to 70.0%
D 70.0% to 60.0%
F below 60.0%

Educational Objectives

Develop a mastery of low-level software primitives and executable inspection
Learn and apply the fundamental principles of dissecting malware
Become aware of the limitations of existing reverse engineering mechanisms and how to overcome them
Study the behaviors, tricks, and tactics employed by modern malware
Engage in critical discussion around key topics in software security and malware prevention

Course Outcomes

After successfully completing this course, students should be able to:

Identify and disarm common anti-analysis behaviors in malware samples
Statically reverse engineer malware samples in a disassembler
Decide upon and employ appropriate reverse engineering tools for a range of malware analysis cases
Reverse engineer exploit inputs for benign program binaries
Dynamically unpack malware in a debugger and extract clean disassemblies

Honor Code

Students are expected to abide by the Georgia Tech Academic Honor Code. Honest and ethical behavior is expected at all times. All incidents of suspected dishonesty will be reported to and handled by the Office of Student Integrity. You will have to do all assignments individually unless explicitly told otherwise. You may discuss with classmates but you may not copy any solution (or any part of a solution).

Learning Accommodations

Whenever needed, the instructor will make accommodations for students with documented disabilities. These accommodations must be arranged in advance and in accordance with the Office of Disability Services.

Class Attendance

Class attendance is mandatory, and past offerings have shown that students who are actively involved in class discussions have the best experience conquering this challenging subject matter. Course deadlines and assignments can be modified for students with documented absences. These accommodations must be arranged in advance and in accordance with the Georgia Tech Attendance Policy.