Advanced Topics in Malware Analysis · Brendan Saltaformaggio

This page is for GT students not enrolled in the Georgia Tech Professional Education Online Masters (OMS) Degree Program. If you are an OMS student, please select an OMS section from here.

Course Overview

This course covers advanced approaches for detecting the presence of vulnerabilities in binary software, the analysis of malicious software, and explores recent research and unsolved problems in software protection and forensics.

The goal of this course is to engage in critical discussion around key research topics in software security and forensics. This course will cover: Binary Program Analysis Principles, Binary Software Security, Software Forensics and Cyber Attack Response. Students will be required to study published research papers from the top-tier academic venues in computer security and cyber forensics.

Students will conduct a half-semester long research project which will focus on solving open-ended research problems and can lead to publications in reputed conferences/journals/workshops. Projects aligned with a student's own graduate research (MS/PhD) will be encouraged if they have an interesting security/forensics component.

Why take this course?: You are interested in learning the fundamental principles of dissecting malware, vulnerability finding/defense, and cyber attack triage. You want to read cutting-edge research publications on these topics. There is ample scope to publish in this area: If the results from your course project look promising, we can write a paper on it and I will fund your travel to go present it.

Basic Information

Class location: Mason 3133 (map)
Class day/time: Mon. and Wed. 11:00 am to 12:15 pm

Instructor: Professor Brendan Saltaformaggio
Office: CODA E1068B
Email: brendan@ece.gatech.edu
Office hours: Mon. and Wed. 12:15 pm to 1:00 pm in CODA E1068B (or on the walk over), or any time by appointment

TA: Chow Eu-Fung
Email: chow.eufung@gatech.edu
Office hours: Fri. 3:30 pm to 5:00 pm in the CODA 10th Floor Lobby, or any time by appointment

Canvas (Lecture Slides, Assignments, Grades): https://canvas.gatech.edu
Ed Discussion (Questions, Announcements, Chat): Link In Canvas.

Materials

There is no required textbook for this course. Instead we will study published research papers from the top-tier academic venues in computer security and cyber forensics.

The following books are recommended for additional background or more in-depth understanding of the topics discussed in class. Read these books only if you want to learn more! They will not be covered in lectures or on exams!

Practical Tools/Techniques For Malware Reverse Engineering:
Michael Sikorski, Andrew Honig. Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software. No Starch Press, 2012. ISBN: 978-1593272906
Practical Tools/Techniques For Memory Forensics:
Michael Hale Ligh, Andrew Case, Jamie Levy, AAron Walters. The Art of Memory Forensics: Detecting Malware and Threats in Windows, Linux, and Mac Memory. Wiley, 2014. ISBN: 978-1118825099
Background On Low-Level Computer Systems Programming:
Randal E. Bryant, David R. O'Hallaron. Computer Systems: A Programmer's Perspective. Pearson (3rd Edition), 2015. Online: http://csapp.cs.cmu.edu/. ISBN: 978-0134092669
A Great Reference For The IDA Pro Projects:
Chris Eagle. The IDA Pro Book. No Starch Press (2nd Edition), 2011. ISBN: 978-1593272890
You may be able to find this one online.

You may also need a copy of the Intel Developer’s manuals. These are free and available via this link: http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html
It's large, but the best PDF to get is the combined set, downloadable via the first link on that page. If you have an iPad or other tablet, drop this PDF on it and read it whenever you have spare time.

Course Outline

Binary Analysis Principles
1. Static Analysis
  1. Static Binary Code Analysis Techniques/Tools
  2. Reverse Engineering
    1. Intro to Malware Classification and Triage
  3. Program Representations
  4. Pointer Analysis and Points-To
  5. Binary Code Control Flow Analysis
    1. Intro to Control Flow Integrity
2. Dynamic Analysis
  1. Dynamic Program Tracing Techniques/Tools
  2. Program Profiling
  3. Dynamic Slicing
  4. Data Flow Tracking
    1. Practical Data Flow Integrity (e.g., libdtf)
3. Symbolic Execution
  1. Deep Software Vulnerabilities
  2. Trigger Input Generation
  3. Automated Exploit Generation
Binary Software Security
1. Introduction to Software Security and Access Control
2. Software Vulnerabilities
  1. Static Protection through Software Bug Finding
  2. Dynamic Vulnerability Discovery
3. Malware Analysis
  1. Return of Malware Classification and Triage
4. Android/iOS Malware
5. Input Generator for Malware Triggering
6. Software Defense
  1. Dynamic Defense Mechanisms
  2. Detecting Malicious Logic in Binaries
  3. Large-Scale Software Vetting
  4. Binary Program Hardening
    1. Return of Control Flow Integrity
Software Forensics and Incident Response
1. Memory Forensics
  1. Data Structure Reverse Engineering
    1. Value-Invariant Discovery
    2. Structural-Invariant Discovery
  2. Program-Analysis-Driven Evidence Recovery
2. Execution Recreation
  1. Postmortem Execution Analysis
  2. Relationships to Debugging

Assigments & Grading

There will be 6 mini-projects during the Binary Analysis Principles portion of the class. 4 of the projects will be static analysis with IDA Pro and 2 will be dynamic analysis with Pin. Each project will require careful time allocation to complete on time (1 or 2 week deadlines). Grades will be based on the results produced by your tool. For some mini-projects, we may schedule demos during office hours.

The mini-projects will cover the following topics:

Intro. to Software Disassembly
Manual Static Malware Reverse Engineering
Automated Static Malware Analysis
Static Data Dependence Detection
Dynamic Control Flow Analysis
Dynamic Control Dependence Detection

For the remainder of the course, students will conduct a large research project (team projects are fine) which will focus on solving open problems in the areas of software security and cyber forensics. I expect that the best among these will lead to publications in reputable conferences & journals. Grading for these research projects will be based on each team's understanding of the problem and the success of their final prototype.

Grading Breakdown:

60% for the 6 mini-projects
30% for the large research project
10% for paper presentations & class participation
In lieu of a final exam, each student will need to prepare a final presentation and report on the outcomes of their research project to be due during the final instruction days of class.

Grading Scheme for ECE 6747 & CS 6747:

A 100% to 90%
B 90% to 80%
C 80% to 70%
D 70% to 60%
F below 60%

Grading Scheme for ECE 4147:

A 100% to 85%
B 84% to 70%
C 69% to 55%
D 54% to 40%
F below 40%

Educational Objectives

Learn and apply the fundamental principles of dissecting malware, vulnerability finding/defense, and cyber attack triage
Become aware of limitations of existing defense mechanisms and how to avoid them
Study cutting-edge research publications on these topics
Engage in critical discussion around key research topics in software security and forensics
Propose solutions to open-ended research problems and implement novel prototype solutions

Course Outcomes

After successfully completing this course, students should be able to:

Identify and disarm common malware tactics
Statically reverse engineer malware samples in a disassembler
Build static analysis tools to automate control flow recovery and identify intractable indirect jumps
Design and implement static analysis routines to perform automated data dependency tracking
Instrument binary programs and malware to collect dynamic instruction traces
Implement dynamic analysis tools to perform online control dependence tracking
Read and present cutting-edge research publications relating to malware analysis, vulnerability finding/defense, and cyber attack triage

Honor Code

Students are expected to abide by the Georgia Tech Academic Honor Code. Honest and ethical behavior is expected at all times. All incidents of suspected dishonesty will be reported to and handled by the Office of Student Integrity. You will have to do all assignments individually unless explicitly told otherwise. You may discuss with classmates but you may not copy any solution (or any part of a solution).

Learning Accommodations

Whenever needed, the instructor will make accommodations for students with documented disabilities. These accommodations must be arranged in advance and in accordance with the Office of Disability Services.

Class Attendance

Class attendance is mandatory, and past offerings have shown that students who are actively involved in class discussions have the best experience conquering this challenging subject matter. Course deadlines and assignments can be modified for students with documented absences. These accommodations must be arranged in advance and in accordance with the Georgia Tech Attendance Policy.