I came across a paper called A Few Useful Things to Know about Machine Learning
which summarized some black arts in machine learning.
- Learning = Representation + Evaluation + Optimization
- It's generalization that counts
- Data alone is not enough (we also need knowledge)
- Overfitting has many faces
- Intuition fails in high dimensions
- Theoretical guarantee are not what they seem
- Feature engineering is the key
- More data beats a clever algorithm
- Learn many models, not just one (model ensembles)
- Simplicity does not imply accuracy
- Representable does not imply learnable
- Correlation does not imply causation
The author also recommended his book The Master Algorithm and his ML course.