ECE 4117 / CS 4854: Introduction to Malware Reverse Engineering

Course Overview

Malware reverse engineering involves deep analysis of the code, structure, and functionality of malicious software. The goal of this course is to provide a solid foundation in reverse engineering, which is crucial in understanding modern malware and crafting solutions for the remediation and prevention of cyber attacks.

This course exposes students to an immersive, hands-on experience in malware analysis and examines a wide range of software security topics relating to operating systems, debugging, and software protection.

Why take this course?: You are interested in learning the fundamental principles of dissecting malware and cyber attack triage. Practicing software reverse engineering is also useful for creating interoperable software, for verifying that software and software patches function as promised, and for the simple joy of understanding how software executes at a deep level.

Basic Information

Class location: Scheller College of Business 103 (map)
Class day/time: Tues. and Thur. 5:00 pm to 6:15 pm

Instructor: Professor Brendan Saltaformaggio
Office: CODA S0923
Email: brendan@ece.gatech.edu
Office hours: Tues. and Thur. 6:15 pm to 7:00 pm in CODA S0923 (or on the walk over), or any time by appointment

TA: Anirudh Gattu
Email: anirudh.gattu@gatech.edu
Office hours: Tues. 11:15 am to 12:15 pm in Kendeda 152 (map), or any time by appointment

Canvas (Lecture Slides, Assignments, Grades): https://canvas.gatech.edu
Piazza (Questions, Announcements, Chat): Link In Canvas.

Materials

There is no required textbook for this course. The course will be driven by a sequence of hands-on reverse engineering exercises which emphasize the discovery, understanding, and mitigation of common malware tactics --- the devil really is in the details.

The following books are recommended for additional background or more in-depth understanding of the topics discussed in class. Read these books only if you want to learn more! They will not be covered in lectures or on exams!

You may also need a copy of the Intel Developer’s manuals. These are free and available via this link: http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html
It's large, but the best PDF to get is the combined set, downloadable via the first link on that page. If you have an iPad or other tablet, drop this PDF on it and read it whenever you have spare time.

Course Outline

  1. Reverse Engineering Principles
    1. Intro. to Malware and Assembly Language
      1. Under what circumstances is reverse engineering useful or breaking contracts?
      2. Why is reverse engineering necessary?
        1. Interoperability/Competition
        2. Auditing
        3. DRM
        4. Analysis of Malware
    2. Background on Malware
      1. Current and Next-Generation Malicious Software
        1. Viruses
        2. Worms
        3. Trojans
        4. Botnets
        5. Polymorphic and Metamorphic Malware
        6. Advanced Persistent Threats
      2. Intro to Defensive Strategies Against Malware
        1. Worm Fingerprinting/Signature Generation
        2. Behavioral Approaches to Detection of Malware
        3. Hardware Agents for System Integrity Checking
  2. Low-Level Software
    1. Overview of Intel Assembly Language
    2. Virtual Machines for Interpreted High-Level Languages
    3. Representation of Compiled High-Level Language Structures in Assembly
    4. Operating Systems Background
      1. MS-DOS Internals Related to Malware Case Studies
      2. Modern Windows Execution Environment
    5. Executable File Formats
      1. PE Files
        1. Import Address Table
  3. Analysis of Malicious Software
    1. System Monitoring Tools
    2. Dynamic Tracing: System Calls, Filesystem, and Registry
    3. Compiler Issues
    4. Debuggers
      1. OllyDbg
      2. WinDbg
    5. Disassemblers
      1. IDA Pro
      2. Decompilers
    6. Memory Analysis to Support Reverse Engineering
      1. RAM Acquisition
      2. Extraction of Malware
  4. Advanced Reverse Engineering Techniques
    1. Encrypted/Packed Executables
      1. Unpacking Case Studies
    2. Anti-Debugging Techniques
    3. Anti-VM Techniques
    4. Code Obfuscation
  5. Remediation of Advanced Persistent Threats
    1. Determination of Malicious Behaviors
    2. Analysis of Decompiled Source Code
    3. Revelation of Command and Control Functionalities

Assignments & Grading

There will be 10 lab assignments which are designed to progressively introduce new and important features that a reverse engineer might encounter when analyzing modern malware. Labs will be conducted in teams of 2. Most labs will focus on static analysis with IDA Pro. Each assignment will require very careful time allocation to complete by the deadline (often 1 or 2 weeks each). Lab due dates may flex throughout the semester, but labs will always be due at the time class begins on a class day.

Lab Topics:

  1. Software Disassembly
  2. Malware Case Study (1) - Viruses
  3. Malware Case Study (2) - Simple Obfuscation
  4. Malware Case Study (3) - Code Injection
  5. Rapid Analysis of Suspicious Executables (In Class)
  6. Intro. to Encrypted/Packed Malware
  7. Anti-Debugging/Polymorphic Techniques
  8. Unpacking Encrypted/Packed Malware (In Class)
  9. Malware in IoT Devices
  10. Advanced Persistent Threats

The midterm and final exams will focus exclusively on the malware analyses conducted in the labs. For example, a typical question will target a particularly tricky section of the disassembly of a familiar malware sample. If you actively participated in the labs, then the questions should be answerable within the allowed time based on your previous analyses.

Grading Breakdown:

Grading Scheme:

Educational Objectives

Course Outcomes

After successfully completing this course, students should be able to:

Honor Code

Students are expected to abide by the Georgia Tech Academic Honor Code. Honest and ethical behavior is expected at all times. All incidents of suspected dishonesty will be reported to and handled by the Office of Student Integrity. You will have to do all assignments individually unless explicitly told otherwise. You may discuss with classmates but you may not copy any solution (or any part of a solution).

Learning Accommodations

Whenever needed, the instructor will make accommodations for students with documented disabilities. These accommodations must be arranged in advance and in accordance with the Office of Disability Services.

Class Attendance

Class attendance is mandatory, and past offerings have shown that students who are actively involved in class discussions have the best experience conquering this challenging subject matter. Course deadlines and assignments can be modified for students with documented absences. These accommodations must be arranged in advance and in accordance with the Georgia Tech Attendance Policy.