Programming for Performance

CS 610, Semester 2024-25-I, IIT Kanpur

Class hours: MF 8-9 AM Tu 9-10 AM in KD 103

Instructor Information

Name	Swarnendu Biswas
Email	swarnendu AT cse.iitk.ac.in

TA Information

Name	Email
Srinjoy Sarkar	srinjoy@cse.iitk.ac.in
Puspesh Kumar Srivastava	puspeshk24@iitk.ac.in

Course Description

To obtain good performance, one needs to write correct but scalable parallel programs using programming language abstractions like threads. In addition, the developer needs to be aware of and utilize many architecture-specific features like vectorization to extract the full performance potential. This course will discuss programming language abstractions with architecture-aware development to learn to write scalable parallel programs. This is not a “programming tips and tricks” course.

We will have 4-6 assignments to use the concepts learned in class and appreciate the challenges in extracting performance.

Prerequisites

Exposure to the following courses (or equivalent) is desirable: CS220 (Computer Organization), and CS330 (Operating Systems).
Programming maturity with popular programming languages like C, C++, and Java.

Course Syllabus and Policies | Academic Integrity | Evaluation Scheme | Resources | References

Course Syllabus and Policies

Syllabus

The course will focus on a subset of the following topics.

Challenges in parallel programming, correctness, and performance errors, understanding performance, and performance models
Exploiting spatial and temporal locality with caches, analytical cache miss analysis
Performance bottleneck analysis: PAPI counters, performance analysis tools
Dependence Testing, Loop Transformations
Shared-memory programming and Pthreads
Compiler vectorization: vector ISA, auto-vectorizing compiler, vector intrinsics
Core OpenMP, Advanced OpenMP, Heterogeneous programming with OpenMP
Parallel Programming Models and Patterns
GPU Architecture and CUDA Programming
Shared-memory Synchronization
Concurrent Data Structures

We may add new, drop existing, or reorder topics depending on progress and class feedback. The course may also involve reading and critiquing related research papers.

Policies

Please be on time for class.
Please try to avoid using laptops and/or mobile devices in class, they are distracting for both the instructor and other students who want to concentrate.
You are allowed up to two late days to submit your assignment, with a 25% penalty for each day.

Feedback

I am open to feedback about the course content and presentation. Feel free to provide suggestions for improvements.

Academic Integrity

You may discuss concepts with classmates, but all submissions must be your own or your team's work when teamwork is permitted.
You should not search online for existing solutions related to the assignments, even as a reference.
Students caught cheating or plagiarizing will receive penalties, automatically fail the course, and will be reported to the institute.

Evaluation Scheme

Assignments	40%
Midsem	30%
Endsem	30%

This is a tentative allocation and might change slightly depending on the strength of the class
Grading will be relative

Resources

Date	Topic	Resources	Recommended Reading
		First Course Handout
30/07, 02/08	Compiler Challenges for Parallel Architectures	Slides	AK 1.1-1.6
05/08, 06/08, 09/08	Cache Memory	Slides	HP APP B.1-B.4, 2.1--2.3 CSAPP 6.2-6.4
12/08, 13/08	Write Cache-Friendly Code	Slides	CSAPP 6.5-6.6 DRAG 11.1-11.2
16/08	PAPI Library	Slides
19/08, 20/08, 23/08, 27/08, 30/08	Cache Coherence and False Sharing	Slides	MCM Chapters 2, 6, 8 (IITK has subscribed to the ebook)
02/09, 03/09, 06/09	Dependence Testing	Slides	AK Chap 2, 3 DRAG 11.6
07/09, 09/09, 10/09, 13/09	Loop Transformations	Slides	AK 5.2-5.4, 5.7.2, 5.9, 6.2.1, 6.2.2, 6.2.5, 6.3.1-6.3.4 AP 4.1, 4.2, 4.5, 5.1-5.6 HP 4.5 Compiler Transformations for High-Performance Computing
23/09, 24/09, 27/09	Vectorization	Slides	HP 4.1-4.2 Program Optimization Through Loop Vectorization Topics in Loop Vectorization
30/09, 01/10, 04/10, 14/10	OpenMP	Slides	PP Chapter 5 (IITK has subscribed to the ebook) LLNL OpenMP Tutorial Introduction to OpenMP - Tim Mattson (Intel) OpenMP Application Programming Interface v5.2 OpenMP Application Programming Interface Examples v5.2
15/10, 18/10, 21/10, 22/10, 25/10, 26/10, 28/10, 29/10, 01/11	GPU Architecture and CUDA Programming	Slides	NVIDIA CUDA C Programming Guide NVIDIA CUDA C Best Practices Guide KH Chapters 1-5,13,20 (IITK has subscribed to the ebook) HP 4.4
02/11, 04/11, 05/11, 08/11, 11/11, 12/11	Concurrent Data Structures	Slides	MP Chapters 3, 9, 10, 13 (IITK has subscribed to the ebook)

References

I have listed (NOT in any particular order) a few popular references.

[CSAPP] Computer Systems: A Programmer's Perspective, 3rd edition - R. Bryant and D. O'Hallaron
[DRAG] Compilers: Principles, Techniques, and Tools - A. Aho, M. Lam, R. Sethi, and J. Ullman
[HP] Computer Architecture: A Quantitative Approach, 6th edition - J. Hennessy and D. Patterson
[AK] Optimizing Compilers for Modern Architectures - R. Allen and K. Kennedy
[AP] Automatic Parallelization: An Overview of Fundamental Compiler Techniques - Samuel P. Midkiff
[PP] An Introduction to Parallel Programming - Peter S. Pacheco
[KH] Programming Massively Parallel Processors: A Hands-on Approach, 3rd edition - David B. Kirk and Wen-mei W. Hwu
[MCM] A Primer on Memory Consistency and Cache Coherence, 2nd edition - Vijay Nagarajan, Daniel J. Sorin, Mark D. Hill and David A. Wood
[MP] The Art of Multiprocessor Programming, 1st edition - Maurice Herlihy and Nir Shavit

We may read and discuss related materials and research papers, which we will announce in class.