Machine Learning Yearning In Short

gaunthan Posted on May 23 2018

> Machine Learning Yearning is a deeplearning.ai project.

## Why Machine Learning Strategy
> Machine Learning Strategy helps you with choosing the best first step for improving the performance of your learning algorithm when it performs poorly.

## Scale drives machine learning progress
> One of the most reliable ways to improve an algorithm's performance today is still to **train a bigger network** and **get more data**.

## Your development and test sets
> Choose dev and test sets to reflect data you expect to get in the future and want to do well on.

## Your dev and test sets should come from the same distribution
> Having mismatched dev and test sets makes it harder to
figure out what is and isn’t working, and thus makes it harder to prioritize what to work on.

## How large do the dev/test sets need to be?
> With enough examples, you will have a good chance of detecting an improvement between different algorithms (10,000 examples for 0.1% improvement, e.g.)

> No need to have excessively large dev/test sets beyond what is needed to evaluate the performance of your algorithms.

## Establish a single-number evaluation metric
> Having a single-number evaluation metric speeds up your ability to make a decision when you are selecting among a large number of classifiers.

>  Taking an average or weighted average is one of the most common ways to combine multiple metrics into one.

## Optimizing and satisficing metrics
> Another way to combine multiple evaluation metrics, is to optimize one metric and satisfy the others.

## Having a dev set and metric speeds up iterations
> Building a machine learning system is an iterative process, the faster you can go round this loop, the faster you will make progress.

> Each time you try an idea, measuring your idea’s performance on the dev set lets you quickly decide if you’re heading in the right direction.

## When to change dev/test sets and metrics
> 1. The actual distribution you need to do well on is different from the dev/test sets.
> 2. You have overfit to the dev set
> 3. The metric is measuring something other than what the project needs to optimize

## Error analysis: Look at dev set examples to evaluate ideas
> The process of looking at misclassified examples is called error analysis which can often help you figure out how promising different directions are.

## Evaluating multiple ideas in parallel during error analysis
>

## Cleaning up mislabeled dev and test set examples
> If the fraction of the dev set that is mislabeled impedes your ability to make judgments of error analysis, then it is worth spending time to fix the mislabeled dev set labels.

## If you have a large dev set, split it into two subsets, only one of which you look at
> Explicitly splitting your dev set into Eyeball and Blackbox dev sets allows you to tell when your manual error analysis process is causing you to overfit the Eyeball portion of your data.

## How big should the Eyeball and Blackbox dev sets be?
> Your Eyeball dev set should be large enough to give you a sense of your algorithm’s major error categories.

> To refine that statement, a Blackbox dev set of 1,000-10,000 examples will often give you enough data to tune hyperparameters and select among models, though there is little harm in having even more data.