Below are the learnings from eMasters Data Science for Decision Making – IIT Gandhinagar – Part 1. This needs to be an article in itself. You can find more details here about eDSDM here: e-Masters | IITGNX
- There is Mathematics for your AIML problem in feature engineering, preprocessing, evaluation metrics, models, errors, etc. and so on or you can build some relationship between Mathematics and your problem either by breaking the problem into pieces or transforming data and so on.
- Human intuition is still invaluable like in situations of imbalanced datasets, regression and more.
- Visualization and EDA almost always help for your problems. For higher dimension problems you can do PCA, T-SNE, shadowing on lower dimensions, etc. more approaches to bring it to lower dimensions with or without transformation like basis transformation. In this situation, you should also feed data after this transformation for testing / using the model.
- There are problems beyond model selection and preprocessing, feature engineering like data quality, overfitting, evaluation, errors, hyper parameter tuning and generalization which we need to think about.
- Situations like classification of medical diagnosis issues have very high value repercussions for wrong classification whereas some models with overfitting won’t generalize and cause problems like in normal situations with regression.
- Probability concepts like equally likely for randomness, distributions, fairness, probability tree, joint probability, permutations and combinations, etc. have lot of value and are worth learning.
- There is no learning like learning by experience, concepts and application.
- Your assumptions maybe wrong as well so it’s good to verify.
- Accuracy alone may not be a good measure, please add recall & precision as well in your analysis especially in imbalanced datasets.
- Domain knowledge matters. Don’t ignore this.
- Email me: Neil@HarwaniSytems.in
- Website: www.HarwaniSystems.in
- Blog: www.TechAndTrain.com/blog
- LinkedIn: Neil Harwani | LinkedIn