Going Inside Machine Learning and Deep Learning Algorithms
From Machine Learning Algorithms in Depth by Vadim Smolyakov
If you want to excel in ML and deep learning, you need to know more than how to implement the algorithms — you need to know them inside-out. This book delves into selected algorithms and teaches you how to build your own from scratch.
Read on to learn more.
Why Learn Algorithms from Scratch?
This book dives into the design of machine learning (ML) algorithms from scratch. It builds up algorithms from first principles and takes the reader on a journey from mathematical derivation to software implementation of some of the most exciting algorithms in the field of ML. Understanding ML algorithms from scratch has a number of advantages.
First, it builds the reader’s depth of knowledge of the field. This can be useful when you need to modify an existing algorithm to meet the needs of your application. Second, it allows the reader to choose the right algorithm for the task. By knowing the inner workings of an algorithm, it’s easier to understand the design tradeoffs and choices made in developing the algorithm. Third, the reader will be able to interpret and explain the results of the algorithm to the stakeholders in an industrial or an academic setting. Fourth, the reader will be able to use intuition developed by reading this book to troubleshoot advanced ML problems.
Breaking down a complex problem into smaller pieces and understanding where things went wrong often requires a strong sense of fundamentals and algorithmic intuition. This book will allow the reader to construct minimum working examples and build upon existing algorithms to develop and be able to debug more complex models. Finally, we are often interested in improving the performance of existing models. The principles discussed in this book will enable the reader to accomplish that. To summarize, understanding ML algorithms from scratch will help you choose the right algorithm for the task, explain the results, troubleshoot advanced problems, extend an algorithm to a new situation and improve performance of existing algorithms.
Implementing Algorithms
A key part of learning algorithms from scratch is software implementation. It’s important to write good code that is both efficient in terms of its use of data structures and low algorithmic complexity. While the intention of this book is to write all code from scratch (without reliance on third party libraries), we can still use ML libraries (such as scikit-learn: https://scikit-learn.org/stable/) to check the results of our implementation if available. We’ll be using Python language throughout this book.
Data Structures
A data structure is a way of storing and organizing data. Each data structure offers different performance trade-offs and some are more suitable for the task than others. We’ll be using a lot of linear data structures such as fixed size arrays in our implementation of ML algorithms since the time to access an element in the array is constant O(1). We’ll also frequently use dynamically resizable arrays (such as lists in Python) to keep track of data over multiple iterations.
Throughout the book, we’ll be using non-linear data structures, such as map (dictionary) and set. The reason is that ordered dictionary and ordered set are built upon self-balanced binary search trees (BSTs) that guarantee O(nlogn) insertion, search and deletion operations. Finally, a hash table or unordered map is another commonly used, efficient data structure with O(1) access time assuming no collisions.
Problem-Solving Paradigms
Many ML algorithms that are explored in this book can be grouped into four main problem-solving paradigms: complete search, greedy, divide and conquer, and dynamic programming. Complete search is a method for solving a problem by traversing the entire search space in search of a solution. A machine learning example where complete search takes place is an exact inference by complete enumeration. During exact inference, we must completely specify a number of probability tables to carry out our calculations. A greedy algorithm takes a locally optimum choice at each step with the hope of eventually reaching a globally optimum solution. Greedy algorithms often rely on a greedy heuristic. A machine learning example of a greedy algorithm consists of sensor placement. For example, given a room and several temperature sensors, we would like to place the sensors in a way that maximizes room coverage. Divide and conquer is a technique that divides the problem into smaller, independent sub-problems and then combines the solutions to each of the sub-problems. A machine learning example that uses divide and conquer paradigm can be found in CART decision tree algorithm. As we’ll see in a future chapter, in CART algorithm an optimum threshold for splitting a decision tree is found by optimizing a classification objective (such as Gini index). The same procedure is applied to a tree of depth one greater resulting in a recursive algorithm. Finally, Dynamic Programming (DP) is a technique that divides a problem into smaller, overlapping sub-problems, computes a solution for each sub-problem and stores it in a DP table. A machine learning example that uses dynamic programming occurs in Reinforcement Learning (RL) in finding a solution to Bellman equations. For a small number of states, we can compute the Q-function in tabular way using dynamic programming.
What you will learn in this book
This book is about learning algorithms from the inside out by building them from scratch. You can expect to learn the following:
- Develop mathematical intuition for classic and modern ML algorithms
- Learn the fundamentals of Bayesian inference and deep learning
- Learn about data structures and algorithmic paradigms in ML
- Learn about supervised, unsupervised and deep learning algorithms
- Apply algorithms developed in this book to the areas of core ML, natural language processing, computer vision, optimization, computational biology and finance
- and much more!
Who this book is for
This book is for anyone who wants to learn the inner workings of machine learning and deep learning algorithms and those who want to learn how to design their own algorithms, e.g. data scientists, and ML and deep learning engineers. Knowledge of Python is assumed and experience with ML and deep learning algorithms, tools, and libraries is helpful.
Learn more about the book here.
