Neural Networks and Deep Learning ECBM E4040 - Fall 2019


Announcements

2019/09/15 New! In regards to the submission of your homeowork for assignment 0, name the *.zip file as follows. Assignment0.2019.UNI (Example: Assignment0.2019.zz9987).
2019/09/12 Two (same) recitation sessions will be held on Friday 9/13 13:10-15:00 in MUD 633, and 16:10-18:00 in MUD 633.
2019/09/09 Assignment 0 due date and method of submission:: It is available as a *.zip file reachable from https://ecbm4040.bitbucket.io/2019_fall/homework.html (link to liondrive); Due Monday 9/16 at 11:59pm; It needs to contain a student's copy of the modified Jupyter notebook and all other files in subdirectories.; It needs to be uploaded into the courseworks as a zip file with the similar content as the assignment.
2019/09/09 New lecture material uploaded into the google drive :: Topics: Machine Learning. This lecture material illustrates well what type of linear algebra and probability/statistics is needed to understand deep learning well.
2019/09/08 Course assistant office hours (only for week of 9/9/2019) :: Arjun DCunha: Wednesday 1:00-2:30pm; Aijia Gao & Zhen Wang: Tuesday 8am-9:30am; Chengrui Wu: Wednesday 1:00-2:30pm; Huixiang Zhuang: Thursday 2:30-4:00pm; Deepak Ravishankar: Thursday 8:00-9:30am; Room: Mudd 1300 student lounge.

Zoran Kostic, PhD

Associate Professor of Professional Practice, Electrical Engineering Dept., Columbia University.
zk2172@columbia.edu, instructor webpage, Office Hours: Th. 11:30am-1pm, Shapiro 813, (for email communication use subject "E4040 deep learning from student")

Course Assistants:

Deepak Ravishankar, Huixiang Zhuang, TODO add other TAs (common email e4040TAs@columbia.edu).

Office Hours: Mudd 1301; Regular times after Sep. 14: Arjun DCunha: TBD; Aijia Gao: TBD; Zhen Wang: TBD; Chengrui Wu: TBD; Huixiang Zhuang: TBD; Deepak Ravishankar: TBD;


Course Description

The course covers theoretical underpinnings, architecture and performance, datasets, and applications of neural networks and deep learning. The course will use Python coding language, TensorFlow deep learning framework and Google Cloud computational platform with graphics processing units (GPUs). The main resource for the course is the book entitled "Deep Learning" by Ian Goodfellow, Yoshua Bengio, and Aaron Courville, MIT Press, http://www.deeplearningbook.org, 2016.

Students will contribute to the course by:

Prerequisites and Expectations: There will be 4 assignments (homeworks). All assignments will be graded. The homework/assignment 0, which should be done before the add/drop deadline, will help you ascertain the programming know-how needed to work on the following homeworks. If you have difficulties with assignment 0, please carefully evaluate whether you will have enough time to brush up your programming knowledge or acquire the needed skills in "real-time" as the course progresses. The course requires rigorous theoretical background in probability and statistics, linear algebra and calculus - the course will contain material on mathematical/theoretical concepts. A formal machine learning (ML) course is a must-take course for complementing this deep-learning course, but is not a prerequisite - if you have no prior ML knowledge, you will need to learn some of the concepts on your own during this course. The exam will test your knowledge of deep learning concepts, tools, and theory, as well as your knowledge across the prerequisite subjects. You are strongly advised to drop the class if you cannot complete assignment 0 by the add/drop date, or if you judge that the expectations for your mastery of linear algebra, probability and statistics and calculus may be overly demanding.

2019 Course link (Lectures, Syllabus, Assignments - accessible with CU email login until Oct. 2019) link


(Tentative) Grading policy

HW(40%), (one) Exam(25%), Project(35%).

Late homeworks (assignments): A student is entitled to 4 late days without penalty. For all homeworks together, a student can divide those four days in any fashion needed. Examples: (i) Homework 3 is late 4 days, in which case no other homework can be late for any amount of time; (ii) Homework 1 is late 1 day, homework 2 is late 2 days, in which case the student still has one more late-day for future assignments. The unit of delay can not be divided into less than a full day (like hours). Requests for additional extensions will not be granted: if the budget of 4 days is blown, the student will be given 0 credit for homework(s) for which their submission is late.


Computational Environment(s) and Prerequisites

Introduction

Computational tools are essential for learning about, designing, and experimenting with deep learning models. At the highest level, deep learning developers use one of the deep learning frameworks to build and run models, which rely on a myriad of either generic or custom software libraries. Course E4040 will use a particular version of the TensorFlow framework. The underlying computing language is Python, and the course will rely on Python code/libraries and Jupyter Notebooks for developing and experimenting with code. The students will be asked to run the tools and deep learning models on their own (local) computers, as well as in the Google cloud. Versioning of tools is complex, and students need to follow the precise tool version instructions in order to have functional computing environments. The course relies on Conda and PIP for tool versioning.


Local Environment Setup -> link

Students will be asked to use their personal/local computers to run python code, jupyter notebooks, TensorFlow framework and deep learning models. This is possible both with computers which have graphics processing units (GPUs) and with computers which do not have GPUs, where the execution will be faster on computers with GPUs. The begining assignments in this course are runnable on single processor (non-GPU) computers. Later assignments and projects will require that students run them on machines with GPUs; in this case the assignments can be run on personal computers or on the Google Cloud GPU machines. We will provide coupons for Google cloud access. Clicking on this section leads to instructions how to install the required tools on personal computers.

Python Tutorial -> link

There are many online Python tutorials. This section is linked to one such tutorial. For majority of this course, we request that students use python 3.6 (compatible with the prescribed version of Tensorflow 1.13) when working on programming assignments. In the shared Google cloud image, we only installed Python 3.6 version. Therefore, you should strive to use exactly the same version on your personal/local computers.

Google Compute Engine (GCE) Setup on the Google Cloud Platform (GCP) -> link

For more complex assignments, students will be able to run deep learning code on Google Compute Engine (GCE) virtual machine (VM) instances, with access to GPU resources. The instructors will provide coupons (codes) which will enable students to use Google cloud resources for free. This tutorial provides instructions how to setup a VM instance using the VM image created for the E4040 course.

TensorFlow Tutorial -> link

TensorFlow is an open-source deep learning framework developed by Google. It is essential that you learn to program in TensorFlow for this course. Note that the 2019 version of the E4040 course uses version 1.13 of TensorFlow, whereas the most recent TensorFlow homepage may be refering to a more recent version (such as 2.0). For 2019 assignments, students have to use the course-prescribed versions of TensorFlow and Python. If students wish to experiment with other versions of tools, they are advised to create additional (conda) virtual environments to prevent conflicts with tool versions used for the assignments.

Linux Introduction -> link

The GCP VM instances which we use are Linux-based. If you haven't used Linux system before, this tutorial offers hands-on instructions to basic Linux commands that you will be using throughout the course.

Git Commands -> link

Your homework assignments will be based on Github classroom repositories ocassionally we use Bitbucket as well), which can be manipulated by Git commands or visual clients such as GitKraken. If you don't know what Git is, it is strongly recommended that you read the first 3 chapters of this website. Additional instructions will be provided for use of the assignment-related github directories.

Google Colab -> link

Colab provides a free cloud service based on Jupyter Notebooks. It supports free GPUs with popular deep learning libraries with an execution time limit. It is a good platform for developing and testing the deep learning code, which can avoid the issues of tool installation. However, it has limitations. We do not recommend that you do the assignments on Colab since everything except Jupyter Notebooks would be automatically deleted every 12 hours. Moreover, we request that you first set up your own local computing environment and google cloud resources (GCE) since both are a necessary skill for a practicing deep learning professional.

FAQ -> link

Solutions to some common problems


ECBM E4040 Neural Networks and Deep Learning, 2019.

Columbia University