OMS ECE 6747 / OMS CS 6747: Advanced Topics in Malware Analysis

This page is for the Online Masters Degree Program through Georgia Tech Professional Education.

Course Overview

This course covers advanced approaches for detecting the presence of vulnerabilities in binary software, the analysis of malicious software, and explores recent research and unsolved problems in software protection and forensics.

The goal of this course is to engage in critical discussion around key research topics in software security and forensics. This course will cover: Binary Program Analysis Principles, Binary Software Security, Software Forensics and Cyber Attack Response. Students will be required to study published research papers from the top-tier academic venues in computer security and cyber forensics.

Why take this course?: You are interested in learning the fundamental principles of dissecting malware, vulnerability finding/defense, and cyber attack triage. You want to read cutting-edge research publications on these topics. There is ample scope to publish in this area: This course can prepare you to conduct research in cyber attack forensics and malware analysis.

Basic Information

Instructor: Professor Brendan Saltaformaggio
Office hours: Every Tuesday 12:30-1:30pm EST. Zoom Link In Canvas.

TA: Chow Eu-Fung
Office hours: Every Wednesday 4pm EST - 6pm EST. Zoom Link In Canvas.

Canvas (Lecture Slides, Assignments, Grades):
Ed Discussion (Questions, Announcements, Chat): Link In Canvas.


There is no required textbook for this course. Instead we will study published research papers from the top-tier academic venues in computer security and cyber forensics.

The following books are recommended for additional background or more in-depth understanding of the topics discussed in class. Read these books only if you want to learn more! They will not be covered in lectures or on exams!

You may also need a copy of the Intel Developer’s manuals. These are free and available via this link:
It's large, but the best PDF to get is the combined set, downloadable via the first link on that page. If you have an iPad or other tablet, drop this PDF on it and read it whenever you have spare time.

Course Outline

  1. Binary Analysis Principles
    1. Static Analysis
      1. Static Binary Code Analysis Techniques/Tools
      2. Reverse Engineering
        1. Intro to Malware Classification and Triage
      3. Program Representations
      4. Pointer Analysis and Points-To
      5. Binary Code Control Flow Analysis
        1. Intro to Control Flow Integrity
    2. Dynamic Analysis
      1. Dynamic Program Tracing Techniques/Tools
      2. Program Profiling
      3. Dynamic Slicing
      4. Data Flow Tracking
        1. Practical Data Flow Integrity (e.g., libdtf)
    3. Symbolic Execution
      1. Deep Software Vulnerabilities
      2. Trigger Input Generation
      3. Automated Exploit Generation
  2. Binary Software Security
    1. Introduction to Software Security and Access Control
    2. Software Vulnerabilities
      1. Static Protection through Software Bug Finding
      2. Dynamic Vulnerability Discovery
    3. Malware Analysis
      1. Return of Malware Classification and Triage
    4. Android/iOS Malware
    5. Input Generator for Malware Triggering
    6. Software Defense
      1. Dynamic Defense Mechanisms
      2. Detecting Malicious Logic in Binaries
      3. Large-Scale Software Vetting
      4. Binary Program Hardening
        1. Return of Control Flow Integrity
  3. Software Forensics and Incident Response
    1. Memory Forensics
      1. Data Structure Reverse Engineering
        1. Value-Invariant Discovery
        2. Structural-Invariant Discovery
      2. Program-Analysis-Driven Evidence Recovery
    2. Execution Recreation
      1. Postmortem Execution Analysis
      2. Relationships to Debugging

Assigments & Grading

There will be 6 mini-projects during the Binary Analysis Principles portion of the class. 4 of the projects will be static analysis with Ghidra and 2 will be dynamic analysis with Pin. Each project will require careful time allocation to complete on time (1 or 2 week deadlines). Grades will be based on the results produced by your tool. For some mini-projects, we may schedule demos during office hours.

The mini-projects will cover the following topics:

  1. Intro. to Software Disassembly
  2. Manual Static Malware Reverse Engineering
  3. Automated Static Malware Analysis
  4. Static Data Dependence Detection
  5. Dynamic Control Flow Analysis
  6. Dynamic Control Dependence Detection

Grading Breakdown:

Grading Scheme:

Educational Objectives

Course Outcomes

After successfully completing this course, students should be able to:

Honor Code

Students are expected to abide by the Georgia Tech Academic Honor Code. Honest and ethical behavior is expected at all times. All incidents of suspected dishonesty will be reported to and handled by the Office of Student Integrity. You will have to do all assignments individually unless explicitly told otherwise. You may discuss with classmates but you may not copy any solution (or any part of a solution).

Learning Accommodations

Whenever needed, the instructor will make accommodations for students with documented disabilities. These accommodations must be arranged in advance and in accordance with the Office of Disability Services.