Course Overview
Malware reverse engineering involves deep analysis of the code, structure, and functionality of malicious software. The goal of this course is to provide a solid foundation in reverse engineering, which is crucial in understanding modern malware and crafting solutions for the remediation and prevention of cyber attacks.
This course exposes students to an immersive, hands-on experience in malware analysis and examines a wide range of software security topics relating to operating systems, debugging, and software protection.
Why take this course?: You are interested in learning the fundamental principles of dissecting malware and cyber attack triage. Practicing software reverse engineering is also useful for creating interoperable software, for verifying that software and software patches function as promised, and for the simple joy of understanding how software executes at a deep level.
Basic Information
Class location: Scheller College of Business 103 (map)
Class day/time: Tues. and Thur. 5:00 pm to 6:15 pm
Instructor: Professor Brendan Saltaformaggio
Office: CODA S0923
Email: brendan@ece.gatech.edu
Office hours: Tues. and Thur. 6:15 pm to 7:00 pm in CODA S0923 (or on the walk over), or any time by appointment
TA: Anirudh Gattu
Email: anirudh.gattu@gatech.edu
Office hours: Tues. 11:15 am to 12:15 pm in Kendeda 152 (map), or any time by appointment
Canvas (Lecture Slides, Assignments, Grades): https://canvas.gatech.edu
Piazza (Questions, Announcements, Chat): Link In Canvas.
Materials
There is no required textbook for this course. The course will be driven by a sequence of hands-on reverse engineering exercises which emphasize the discovery, understanding, and mitigation of common malware tactics --- the devil really is in the details.
The following books are recommended for additional background or more in-depth understanding of the topics discussed in class. Read these books only if you want to learn more! They will not be covered in lectures or on exams!
-
Practical Tools/Techniques For Malware Reverse Engineering:
Michael Sikorski, Andrew Honig. Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software. No Starch Press, 2012. ISBN: 978-1593272906 -
A Great Reference For IDA Pro:
Chris Eagle. The IDA Pro Book. No Starch Press (2nd Edition), 2011. ISBN: 978-1593272890
-
Background On Low-Level Computer Systems Programming:
Randal E. Bryant, David R. O'Hallaron. Computer Systems: A Programmer's Perspective. Pearson (3rd Edition), 2015. Online: http://csapp.cs.cmu.edu/. ISBN: 978-0134092669
You may also need a copy of the Intel Developer’s manuals. These are free and available via this link:
http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html
It's large, but the best PDF to get is the combined set, downloadable via the first link on that page. If you have an iPad or other tablet, drop this PDF on it and read it whenever you have spare time.
Course Outline
- Reverse Engineering Principles
- Intro. to Malware and Assembly Language
- Under what circumstances is reverse engineering useful or breaking contracts?
- Why is reverse engineering necessary?
- Interoperability/Competition
- Auditing
- DRM
- Analysis of Malware
- Background on Malware
- Current and Next-Generation Malicious Software
- Viruses
- Worms
- Trojans
- Botnets
- Polymorphic and Metamorphic Malware
- Advanced Persistent Threats
- Intro to Defensive Strategies Against Malware
- Worm Fingerprinting/Signature Generation
- Behavioral Approaches to Detection of Malware
- Hardware Agents for System Integrity Checking
- Current and Next-Generation Malicious Software
- Intro. to Malware and Assembly Language
- Low-Level Software
- Overview of Intel Assembly Language
- Virtual Machines for Interpreted High-Level Languages
- Representation of Compiled High-Level Language Structures in Assembly
- Operating Systems Background
- MS-DOS Internals Related to Malware Case Studies
- Modern Windows Execution Environment
- Executable File Formats
- PE Files
- Import Address Table
- PE Files
- Analysis of Malicious Software
- System Monitoring Tools
- Dynamic Tracing: System Calls, Filesystem, and Registry
- Compiler Issues
- Debuggers
- OllyDbg
- WinDbg
- Disassemblers
- IDA Pro
- Decompilers
- Memory Analysis to Support Reverse Engineering
- RAM Acquisition
- Extraction of Malware
- Advanced Reverse Engineering Techniques
- Encrypted/Packed Executables
- Unpacking Case Studies
- Anti-Debugging Techniques
- Anti-VM Techniques
- Code Obfuscation
- Encrypted/Packed Executables
- Remediation of Advanced Persistent Threats
- Determination of Malicious Behaviors
- Analysis of Decompiled Source Code
- Revelation of Command and Control Functionalities
Assignments & Grading
There will be 10 lab assignments which are designed to progressively introduce new and important features that a reverse engineer might encounter when analyzing modern malware. Labs will be conducted in teams of 2. Most labs will focus on static analysis with IDA Pro. Each assignment will require very careful time allocation to complete by the deadline (often 1 or 2 weeks each). Lab due dates may flex throughout the semester, but labs will always be due at the time class begins on a class day.
Lab Topics:
- Software Disassembly
- Malware Case Study (1) - Viruses
- Malware Case Study (2) - Simple Obfuscation
- Malware Case Study (3) - Code Injection
- Rapid Analysis of Suspicious Executables (In Class)
- Intro. to Encrypted/Packed Malware
- Anti-Debugging/Polymorphic Techniques
- Unpacking Encrypted/Packed Malware (In Class)
- Malware in IoT Devices
- Advanced Persistent Threats
The midterm and final exams will focus exclusively on the malware analyses conducted in the labs. For example, a typical question will target a particularly tricky section of the disassembly of a familiar malware sample. If you actively participated in the labs, then the questions should be answerable within the allowed time based on your previous analyses.
Grading Breakdown:
- 50% for the 10 labs
- 20% for the midterm exam
- 20% for the final exam
- 10% for class/piazza participation
Grading Scheme:
- A 100% to 90.0%
- B 90.0% to 80.0%
- C 80.0% to 70.0%
- D 70.0% to 60.0%
- F below 60.0%
Educational Objectives
- Develop a mastery of low-level software primitives and executable inspection
- Learn and apply the fundamental principles of dissecting malware
- Become aware of the limitations of existing reverse engineering mechanisms and how to overcome them
- Study the behaviors, tricks, and tactics employed by modern malware
- Engage in critical discussion around key topics in software security and malware prevention
Course Outcomes
After successfully completing this course, students should be able to:
- Identify and disarm common anti-analysis behaviors in malware samples
- Statically reverse engineer malware samples in a disassembler
- Decide upon and employ appropriate reverse engineering tools for a range of malware analysis cases
- Reverse engineer exploit inputs for benign program binaries
- Dynamically unpack malware in a debugger and extract clean disassemblies
Honor Code
Students are expected to abide by the Georgia Tech Academic Honor Code. Honest and ethical behavior is expected at all times. All incidents of suspected dishonesty will be reported to and handled by the Office of Student Integrity. You will have to do all assignments individually unless explicitly told otherwise. You may discuss with classmates but you may not copy any solution (or any part of a solution).
Learning Accommodations
Whenever needed, the instructor will make accommodations for students with documented disabilities. These accommodations must be arranged in advance and in accordance with the Office of Disability Services.
Class Attendance
Class attendance is mandatory, and past offerings have shown that students who are actively involved in class discussions have the best experience conquering this challenging subject matter. Course deadlines and assignments can be modified for students with documented absences. These accommodations must be arranged in advance and in accordance with the Georgia Tech Attendance Policy.