Traits of a good Information Technology company – Part 1

  1. Promote merit and people with soft and hard skills both not just technology or soft skills alone
  2. Get them to attend trainings from HBR, MIT Sloan & various business schools from across the world regularly
  3. Setup a culture of knowledge sharing, collaboration & team work
  4. Plan & design all projects / initiatives in advance with spirit not lip service
  5. Setup accountability & responsibility
  6. For top management, managers, architects & leaders “Practice what you preach”
  7. Inculcate culture of cyber security, discussion & respect
  8. Keep people busy with genuine work
  9. Respond to people when they reach you, ignoring and tolerating problems for long will not solve the problems and snowball into bigger problems
  10. Few bad leaders will destroy culture across the organization rapidly
  11. Transparency & explaining WHY helps when you are a leader or manager
  12. Operational level problems over time convert into strategy level problems, fix them when they start. In the same way, strategic problems convert into operational problems quickly. Cater to both of these regularly
  13. Think design & systems thinking
  14. Plan and evaluate the future regularly – if you don’t invest in the future now, your path will derail from future when it should have come to you
  15. Have tie-ups with top institutes for further education / training of employees
  16. Give time off, leaves & hobby time to employees – they are not robots. People have people problems, robots too have breakdowns
  17. Actions have consequences, repeated violations have more consequences. Everybody needs to understand this
  18. Don’t judge the book by it’s cover – take 3-6 months of regular interactions to evaluate an employee. Instant judgements don’t help
  19. Awards, limelight & likes don’t always determine the worth of a person
  20. Anything can be achieved anyhow is a dangerous concept, companies that promote this beyond the legal or fair boundaries are not the right places to work – We are not at war, this is corporate world not World War 1/2
  21. Culture eats strategy for breakfast – this is true
  22. Promote, remove & reskill right set of people who have all stakeholders’ interest in mind, stakeholders should include employees & customers both
  23. Give regular & timely feedback at-least once a quarter, be open to discussions & feedback yourself as manager
  24. In internet and social media world, you cant hide your problems as an organization, they will bounce back with time most of the times. These can only be solved by discussion & transparency
  25. Shallowness, spam, power play, sycophancy & such can be detected easily, don’t promote it. Authenticity & transparency helps
  26. Sincerity, transparency & authenticity most of the times gives you the same in return, remember that
  27. Revenue is not everything, other things like strategy, culture, stakeholders matter equally
  28. Practice inclusivity and enablement of all sections of employees rather than only one set based on any bias
  29. Understand why and how biases are formed and correct them with right intervention

Email me: Neil@TechAndTrain.com

What all has changed post COVID-19?

In the last one year, many things have changed for us. Here is a list of areas in business & life that have changed as observed by me & students who were attending the class for subject of BigData in MBA.

1. Digital payments now have wide acceptance 

2. Work & study from home is now a reality, locations matter a lot less now

3. Cyber security is now at the fore-front

4. Digitization of everything – Though we need to be careful of what, how and when we digitize

5. Mental & overall health for people is now a priority 

6. OTT platforms are now providing opportunities to various type of actors, film makers and others beyond the traditional ecosystems of cinemas & multiplexes

7. True democratization of knowledge / information & opportunity is happening, if you want to showcase your talent – internet & digital gives you that space 

8. Healthcare services are now online / tele-medicine is a reality 

9. Pharma companies are now collaborating in a big way 

10. Public health is now a priority for governments 

11. Digital innovation in physical businesses is now at the fore-front 

12. Online learning is widely accepted not just by certificate providers but by universities as well

13. Hiring & business models are changing 

14. RegTech is now mainstream

15. Local shops / stores / super-markets are now converging and getting a voice on various online app ecosystems / platforms – should the local stores act as small warehouses and deliver via app ecosystems, this could be the new model 

16. Newspapers / magazines are going for online subscription based models and shifting away from paper based publishing

17. App ecosystems have changed rapidly to accommodate alternative business models of deliver anything rather than specific things

18. Hotels, airlines & restaurants are looking at alternative solutions to problems that they faced in pandemics

This is a short compilation of changes that we see around us. It is not the strongest who survive, it is the ones who evolve and change that survive

What else do you think will change in the near term?

Email me: Neil@TechAndTrain.com

Kubernetes topics to Search & Learn

  • Pods
  • Containers
  • Registry
  • Master node
  • Secondary nodes
  • Control plane
  • Namespace
  • Ingress
  • Jaeger
  • Prometheus
  • Grafana
  • Kubectl
  • Docker
  • Kubernetes API
  • MiniKube
  • Logging
  • Kubeadm
  • Stateful vs. Stateless pods
  • Configurations
  • Yamls
  • Helm
  • Ansible
  • Operator
  • Metrics & Monitoring
  • Objects
  • Persistent volume
  • Persistent volume claim
  • Clusters
  • Auto cleanup
  • Kube proxy
  • Scheduler
  • Kubelet
  • Controller manager
  • api-server
  • Administration
  • Extensions & plugins
  • Patterns for Kubernetes
  • Authentication
  • Authorization
  • https://kubernetes.io/docs/home/

#kubernetes #clusters #orchestration #cloud

Email me: Neil@TechAndTrain.com

Visit my creations:

  • www.TechAndTrain.com
  • www.QandA.in
  • www.TechTower.in

Best practices for low code development

Best practices for low code development:

  • Finalize the domain / entity model early
  • Check the capabilities of platform before writing code for any feature
  • Learn XPath
  • Try to have daily builds, daily demos, daily development goals with short sprints – comes from agile world
  • Learn to use the CI/CD and Code management tools of the platform
  • Avoid deep linking URLs, avoid page URL altogether if possible
  • Check security / roles / entity access settings regularly
  • Integrate using in-built tools rather than custom code
  • For features like audit, associations, validations try to work at the entity level to start with. Incase you are unable to manage them their then move to workflow level validations
  • Understand the limitations of platform and then make appropriate suggestions
  • Old components, modules, etc. building up in apps should be cleared regularly
  • Check the app stores of the platform for modules that are already available before you build one
  • Check added libraries regularly for unused or deprecated / insecure versions
  • Scan uploaded files before using
  • Try to use architecture / design principles like SOLID in terms of singular responsibility and so on
  • Think of SSL, containerization, cloud, kubernetes, automation testing, logging, user interface & licensing in advance and not as a reaction later

Low code has arrived !!!

What can Jenkins do for you?

Topic: “What can Jenkins do for you?” might sound a bit old fashioned and cliched as Jenkins has been around for a while but it has very varied capabilities via plugins & build pipelines to manage many things. Brief list of capabilities which in no way are exhaustive are given below:

  1. Continuous build management
  2. Continuous deployment
  3. Continuous testing
  4. Continuous quality checks and code scans
  5. Continuous security testing
  6. Continous license checks
  7. Continous Kubernetes, cloud & docker deployment / monitoring
  8. Continuous email notifications for events
  9. Integration with JIRA
  10. Integration with notification systems
  11. Continuous monitoring
  12. Continuous reports & test results analysis

Key concepts, documentation & keywords in Kafka – Part 1

Here are some important concepts, documentation and keywords of Kafka that you can refer and learn. There are two major flavors of Kafka – Apache Kafka & Confluent Kafka, I have listed major keywords, documentation and concepts from both here:

  • Broker
  • Zookeeper
  • kSQL
  • REST-Proxy
  • Schema-Registry
  • Connectors
  • Operator
  • Control Center
  • Streams
  • Topics
  • Consumers
  • Producers
  • Partitions
  • Offset
  • Log
  • Node
  • Replica
  • Message
  • Leader
  • Follower
  • Replicator
  • Schema management
  • Confluent Hub
  • Events
  • Associated keywords in today’s cloud deployments: Docker containers, Kubernetes, Ansible, Security

Associated documentation:

Building data models that everyone can understand and more importantly believe

Building data models that everyone can understand and more importantly believe. Faculty Article – Author: Mr. Balakrishnan Unny & Mr. Neil Harwani. Thank you Sapience – IMNU’s (Nirma University) Alumni Newsletter for publishing our article in Changing Times 2.0 (A Special Edition).

Productivity hacks for Architects / Designers / Tech Leads

As per my experience, the biggest productivity hacks for Architects / Designers / Tech Leads are not to decide the variables / class names / loops / scope / data types / exception handling / object relational mapping & so on – they definitely are important and should be done, but so are the below points:

1. Design patterns

2. What is the code for?

3. Functional to technical mapping

4. Solution creation

5. Pseudocode & logic steps

6. Logic of solution for design / programming problems

7. Co-ordination with stakeholders & communication

8. Code review

9. Logic review of programmed modules

10. Architecture / Design thoughts

11. Knowledge updation around tools / products / frameworks usage

12. Time management of developers

13. Task management of developers

14. Solving problems in design

15. Programming standards management

16. Technical best practices management

17. New technology exploration

18. Helping sales, presales & practice

19. Working on POCs, solutions, products and accelerators

20. Updating oneself with the current happening in industry and domain

21. Automation, Security, Testing, Deployment, Continuous integration / deployment, Integrations, Logging, User Interface / User Experience, Application monitoring, Support structure, Clustering / Auto-scaling, Non functional requirements and other such important areas

22. Establish collaboration / teamwork among technical staff working with them

23. Right documentation and knowledge sharing practices

Many get stuck in only programming, that is definitely something we all love and do, but you should be dividing your time as an Architect / Tech Lead / Designer between programming and above tasks equally. Current enterprise softwares are complex and you can’t achieve much without collaboration and above form an important link for productivity in complex, large team projects.

#architecture #design #technicallead #solutionsarchitect

Email me: Neil@TechAndTrain.com

Visit my creations:

  • www.TechAndTrain.com
  • www.QandA.in
  • www.TechTower.in

Data Analysis Process in Analytics / Data Science

This article is based on understanding from Wikipedia article on Data Analysis & my experiences in Data Science / Analytics / AI / ML – https://en.wikipedia.org/wiki/Data_analysis

Various areas like Data Mining, Predictive Analysis, Exploratory Data Analysis, Text Analytics, Business Intelligence, Confirmatory Data Analysis and Data Visualization overlap with this area

Before starting your journey on solving an industry or academic or research problem in Data Science / Analytics / AI / ML / Decision Science, a fundamental step where many students & professionals struggle is Data Analysis. In this article, I am providing a step by step approach on analyzing your data. Directly starting with programming of various algorithms or neural network on your data could at times be counterproductive and should be avoided. Initial stage should involve robust data analysis via steps given below followed by model building which can include custom or already proven algorithms or a derivative of some popular models. For each of the points discussed below, I have added additional information on top of interpretation of Wikipedia information based on my experience in industry towards the end of each of the points or I have added new points post the interpretations.

Your steps for data analysis should generally be:

  1. Setup your data analysis process at a high level with your objectives – inspecting data, cleaning it, processing (could include dimensionality reduction / feature engineering), transformation, modelling and communicating it. Many forget the functional and feedback loop in this process setup to improve data quality – that must be included too.
  2. Next step is in understanding the data in terms of what is it telling us. Data could be quantitative style numbers or textual or a mix of it. Treatment for all three is different. For quantitative / numerical data, we try to understand whether it is time-series, ranking, part to whole, deviation, frequency distribution, correlation, nominal or geographical or geospatial data. For textual or mixed type of data we need to use approaches of text mining, sentiment analysis, natural language processing to get insights around frequency of words, influential words & sentences by weight, trends, categories, clusters and more. Most of this article revolves around quantitative or numerical data perse and not textual data. I have provided a very brief idea on textual data analysis here in this point.
  3. Next step would be to have the quantitative techniques being applied on the data in terms of sanity, audit / reconciliation of totals via formulas, relationships between data, checking things like whether variables in data are related in terms of correlation / sufficiency / necessity / etc. I would suggest using R Studio or similar tool for this step.
  4. Post this we want to actually perform actions like filtering, sorting, checking range and classes, summary, clusters, relationships, context, extremes, etc. At this stage, exploratory data analysis techniques come in very handy where we use various libraries which provide graphical representation. Excel & Tableau come in handy here.
  5. Our next step will be to check for biases, deciphering facts & opinions, deciphering any numerical incorrect / irrelevant inferences which are being projected and need correction / improvement. This needs detailed study of data from domain / functional perspective and applying statistical analysis on it. Working with a business / functional consultant in this phase is especially useful.
  6. Some areas which we need to take care of include quality of data, quality of measurements, transformations various variables / observations into log scale or others like what we have on richter scale for earthquakes, mapping to objectives and characteristics. This is an intuitive step where visualizing data through various transformations in R / Python / etc. using libraries like Ggplot2, Plotly, Matplotlib, etc. helps.
  7. Next comes checking outliers, missing values, randomness, analysis & plotting various of charts based on type of data whether categorical or continuous. This is statistical analysis & visualization where I find R to be most suited.
  8. Building models around our data analysis steps could involve linear, non-linear models and checking values via hypothesis testing and mapping to algorithms to process, predict, cluster, find trends and so on. Products / tools like R / Python with libraries like Scikit learn, Numpy, Pandas, MLR, Caret, Keras, TensorFlow, etc. help here
  9. While running the models take care of cross-validation of data & sensitivity analysis – This can generally be done using some options in model training & testing phase for supervised learning.
  10. Feedback loop to circle and improve data & results, accuracy analysis and improvement, pipeline building, interpretation of results & functional mapping to domain are additional things that we need to consider on top of the basics given in Wikipedia article. Also, things like dimensionality reduction techniques like PCA, SVD and such need to be explored in detail as they are helpful in this analysis.

Additional information on top of what is in Wikipedia article:

  1. Explainable AI / ML – https://en.wikipedia.org/wiki/Explainable_artificial_intelligence
  2. Interpretable ML – https://statmodeling.stat.columbia.edu/2018/10/30/explainable-ml-versus-interpretable-ml/
  3. Tools / languages / products to use: R, Python, Pandas, Numpy, Tableau and so on
  4. EDA – https://en.wikipedia.org/wiki/Exploratory_data_analysis
  5. Which chart to use – https://www.tableau.com/learn/whitepapers/which-chart-or-graph-is-right-for-you
  6. List of charts – https://python-graph-gallery.com/all-charts/
  7. Confirmatory data analysis – https://en.wikipedia.org/wiki/Statistical_hypothesis_testing
  8. Singular Value Decomposition – https://en.wikipedia.org/wiki/Singular_value_decomposition
  9. Dimensionality Reduction – https://en.wikipedia.org/wiki/Dimensionality_reduction

Email me: Neil@TechAndTrain.com

Visit my creations:

  • www.TechAndTrain.com
  • www.QandA.in
  • www.TechTower.in

What are we doing in AI / ML / Data Science / Decision Science / Analytics World? – Glossary

Over the last few years I have explored, programmed, worked in, researched and taught Data Science / AI / ML / Analytics / Decision Science to multiple students and with many software professionals. I have collected many keywords that you can google and explore. This will help you to keep pace and learn about things happening is these areas. It’s like a glossary of words to search over internet. It’s a mix and match of technologies, algorithms, concepts, AI / ML / Information Technology terms, BigData words and so on in no particular order. I will keep expanding this till it’s a relatively exhaustive list.

  • Automatic Machine Learning
  • Transfer Learning
  • Explainable Machine Learning
  • Keras
  • PyTorch
  • MLR
  • R
  • Python
  • Ggplot2
  • MathplotLib
  • MLib
  • Spark
  • Hadoop
  • Tableau
  • Chatbots
  • Talend
  • MongoDB
  • Neo4j
  • Kafka
  • ELK
  • NoSQL
  • Cassandra
  • AWS SageMaker
  • SVM
  • Decision Trees
  • Regression: Logistic, Multiple, Simple Linear, Polynomial
  • Scikit Learn
  • KNIME
  • BERT
  • NLG
  • NLP
  • Random Forest
  • Hyper parameters
  • Boosting
  • Association rules / mining – Apriori, FP-Growth
  • Data mining
  • OpenCV
  • Self driving cars
  • AI / Memory embedded SOCs, GPUs, TPUs
  • Neural engine chipsets
  • Neural Networks
  • Deep Learning
  • EDA
  • Statistical & Algorithmic modelling
  • Sampling
  • Probability distributions
  • Hypothesis testing
  • Intervals, extrapolation, interpolation
  • Scaling
  • Normalization
  • Agents, search, constraint satisfaction
  • Rules based systems
  • Semantic net
  • Propositional logic
  • Fuzzy reasoning
  • Probabilistic learning
  • First order logic
  • Game theory
  • Pipeline building
  • Ludwig
  • Bayesian belief networks
  • Anaconda Navigator
  • Jupyter
  • Synthetic data
  • Google dataset search
  • Kaggle
  • CNN / RNN / Feed forward / Back propagation / Multi-layer
  • Tensorflow
  • Deepfakes
  • KNN
  • K means clustering
  • Naive Bayes
  • Dimensionality reduction
  • Feature engineering
  • Supervised, unsupervised & reinforcement learning
  • Markov model
  • Time series
  • Categorical & Continuous data
  • Imputation
  • Data analysis
  • Classification / Clustering / Trees / Hyperplane
  • Differential calculus
  • Testing & training data
  • Visualization
  • Missing data treatment
  • Scipy
  • Pandas
  • LightGBM
  • Numpy
  • Dplyr
  • Google Collaboratory
  • PyCharm
  • Plotly
  • Shiny
  • Caret
  • NLTK, Stanford NLP, OpenNLP
  • Artificial intelligence
  • SQL / PLSQL
  • Data warehousing
  • Cognitive computing
  • Coral
  • Arduino
  • Raspberry Pi
  • RTOS
  • DARPA Spectrum Challenge
  • 100 page ML book
  • Equations, Functions, and Graphs
  • Differentiation and Optimization
  • Vectors and Matrices
  • Statistics and Probability
  • Operations management & research
  • Unstructured, semi-structured & structured data
  • Five Vs
  • Descriptive, Predictive & Prescriptive analytics
  • Model accuracy
  • IoT / IIoT
  • Recommendation Systems
  • Real Time Analytics
  • Google Analytics

If you are learning something by googling these topics, feel free to provide suggestions for adding more words here. You are welcome to discuss / suggest on top of this article as well. Thank you for reading.

Email me: Neil@TechAndTrain.com

Visit my creations:

  • www.TechAndTrain.com
  • www.QandA.in
  • www.TechTower.in

Ideas on Innovation around Software. We Thrive On Ideas. We are Learner Centered, Open Source & Digital Focused.