Data Science - Interview Questions

Data Science Inte rvie w Questions (30 day s of Interview Pre pa ra ti on) INEURON.AI Page 2 Q1. What is the difference bet ween A I, Data Science, ML , and DL? Ans 1 : Artificial Intelligen ce : AI is purely math and sc ientific exercis e, but when it becam e computational, it started to solve hum an problems f ormalized into a subset of com puter science. Artif icial intelligence has changed the origina l computational statistics p aradigm to the m odern idea that machines could m imic actual human capa bilities, such as decisi on making and perfo rming more “human” task s. Modern AI into two categories 1. General AI - Planning, decision m aking, identifying objects , recognizing sounds, social & business transactions 2. Applied AI - driverless/ Autonom ous car or mac hine sm artly trade st ocks Machine Learning : I nstead of engineers “teaching” or programm ing computers to have what the y need to carry out task s, that perhaps computers could teach themselves – learn som ething without b eing explicitly programm ed to do so. ML is a f orm of AI where base d on more data, and they can change actions and response, which will make m ore efficient, adaptab le and scalable. e.g., na vigation apps and recommendation engi nes. Classified into:- 1. Superv ised 2. Unsuper vised 3. Reinforc ement learning Data Science : D ata science has m any tools, techniques, and algorithm s called from these f ields, plus others – to handle big data The goal of data sci ence, somewhat sim ilar to machine learning, is to m ake accurate predictions and to automate and perf orm transactions in real - ti me, such as purc hasing internet traffic or autom atically generating content. INEURON.AI Page 3 Data science relies less on math and coding and m ore on data and building n ew systems to proc ess the data. Relying on the fie lds of data integration, distribut ed architecture, aut omated machine learning, data visualization, data en gineering, and autom ated data-driven decis ions, data science can cover an entire spectrum o f data processi ng, not only the algorithm s or statistics related to data. Deep Learning : It is a technique for im plementing ML. ML provides the d esired output from a given input, but DL reads t he input and applies it to another data. I n ML , we can ea sil y classif y the flo w er bas ed upon th e features. Suppose you want a machine to look at an image and determ ine what it represents to the human e y e, whet her a face, flo wer, landscape, truck, building, etc. Machine learning is not sufficient for this task because machine learning can on ly produce an output from a data set – w hether ac cording to a known algorithm or based on the inherent stru cture of the data. Y ou might be able to us e machine learning to determ ine whether an image w as of an “ X” – a flower, say – and it would learn and get m ore accurate. But that output is binary (yes/no) and is d ependent on the algorithm, not the data. In the image recognition ca se, the outcom e is not binary and not depe ndent on the algorithm. The neural network performs MICRO calculations with computational on m any lay ers . Ne ural networks also support weight ing data for ‘confidence . Th ese results in a probabilistic s y stem , vs. deterministic, and can handle task s that we think of as requiring m ore ‘human - like’ judg ment. Q2. What is the d ifference bet ween Supervised learning , Unsupervis ed learning and Reinforcement learning? A ns 2 : Machine Learning Machine learning is the sc ientific stud y of algor ithms and statistical m odels that computer s y stems use to effectively perform a specific task without using ex plicit instructions, relying on patt erns and inference instead. Building a model by learning the patterns of histor ical data wit h some relationship betw een data to make a data-driven predicti on. Types of M achine Learning • Supervised Learning • Unsupervised Learn ing • Reinforcem ent Lea rning Supervised learning In a supervised learni ng model, the algorithm learns o n a labeled d ataset, to generate reasona ble predictions for the res ponse to new data. (Forecasting outcom e of new data) • Regression • Classification INEURON.AI Page 4 Unsupervised learning An unsupervised m odel, in contrast, pro vides unlabelled data that t he algorithm tries to m a ke sens e of by extracting features, co -occur re nce and under ly ing patt erns on its own. W e use unsupervised learning for • Clustering • Anomal y detection • Association • Autoencoders Reinforcement Lea rning Reinforcem ent lea rning is less supervised and depend s on the l earning agent in determ ini ng the output solutions by arriving at different possible wa y s to achieve the best p ossible solution. Q3. D es cri be t he general architecture of Machine learning. Business understand ing: Understand the gi ve use cas e, and also, it 's good to k now more about the domain for which the us e cases are built. Data Acquisition and Understanding: Da ta gathering from dif ferent sources and understanding the data. Cleaning the dat a, handling the m issing data if any, dat a wrangli ng , and EDA( Explorator y data analysis). INEURON.AI Page 5 Modeling: F eature Engineering - scaling the d ata, feature selection - not all features are important. W e use the back w ard elim ination method, corre lation factors, PCA and dom ain knowledge to select the features. Model Training bas ed on trial and error m ethod or by experience , we select the al go rithm and train with the selected features. Model evaluation Accuracy of the m odel , confusion matrix and cross -validation. If accurac y is not high, t o achieve higher accurac y, we tune the m odel...either by changing the al gorith m used or by feature sel ection or by gather ing more data, etc. Deployment - Once the model has good acc uracy, we de ploy the model either i n the cloud or R as berry py or any other place. O nce we deploy, we monitor the perform ance of the m odel.i f its good...we g o live with the model or re iterate the all proc ess until our model perf ormance is good. It 's not done yet!!! W h at if, after a f ew days, our m odel perform s bad ly because of new data . In that case, we do all th e process again b y collectin g new data and redeplo y the m odel. Q4. What is Linear Regression? Ans 4: Linear Regression ten ds to establish a relationship b etween a depende nt variable(Y) and one or m ore independent vari able(X) by finding the best f it of the straight l ine. The equation for the Linear m odel i s Y = m X+c, where m is the slope and c is the intercept In the above diagram , the blue dots we see are the distribution of ' y ' w.r.t 'x .' There is no straight line that runs through all the dat a points. So, the objective here is to fit th e be st fit of a strai ght line that will tr y to minimize the error b etween the expected and actu al value. INEURON.AI Page 6 Q5. OLS Sta ts Model (Ordinary Least Square) A ns 5: OLS is a stats m odel, which will help us in ident ifying the more significan t feature s that can has an influence on the out put. OLS model in p y thon is executed as: lm = smf.ols(f ormula = 'Sales ~ am+constant', da ta = data).fit() lm.conf _int() lm.summary() And we get the outp ut as below, The higher the t -value for the feature, the more significant the feature is to the out put variable . And also, the p-value p lay s a ru le in rejecting the Null h y pothesis(Nul l hypothesis stating the f eatures has zero significance on the targe t variable.). If the p -value is less than 0.05( 95% confidence interval) for a feature, then we can consider the feature to be significant. Q6. What is L1 R egularization (L1 = lasso) ? A ns 6: The main objective of creating a model(tra ining data) is m aking sure it fits the data properl y and reduce the loss. Sometim es the model that is trained which will fit the data b ut it may fail and give a poor performance during a nalyzing of data (test data) . T his leads to o verfitting. Regularization cam e to overcome overfitt ing. Lasso Regression ( L east Absolute Shrinkage and Selection Operator ) adds “ A bsolute value of magnit ude” of coef ficient , as penalt y te rm to the loss function. INEURON.AI Page 7 Lasso shrinks the less important feature’s coefficient t o zero ; th us, removing some f eature altogether. So, this works well for f eature selection in case we ha ve a huge number of f eatures. Methods like C ross-validation, Stepwise Regression are there to handle overfitting and perform feature selection work well with a small set of features . These techniques are g ood when we are dealing with a large set of features. Along with shrink ing coefficients, the lasso perfor ms feature selection , as well. (Rem ember the ‘selection‘ in the lasso f ull -form ?) Because some of the coefficients becom e exactly zero, which is equivalent to the part icular feature being exclu ded from the model. Q7 . L2 Regula rization(L2 = Ridge Regression ) A ns 7: Overfitting happens when the model learns si gnal as well as noise in the tr aining data and wouldn’t perform well on ne w/unseen data on which m odel wasn’t trained on. To avoid overfitting your model on training data like cross-v alidation sampling , reducing the number of features, pruning , regularization , etc . So to avoid ov erfitting , we perform Regularization . INEURON.AI Page 8 T he Regression model that u ses L2 regularization is called Ridge Regressio n. The formula fo r Ridge Regression :- Regularization add s the penalty as model comp lexity increases. The regulariz ation parameter (lambda) penalizes all the parameters except int ercept so that the model generalize s the data and won’t overfit. Ridge regression adds “squared magnitude of the coefficient " as penalty term t o the loss function. Here the box pa rt in the above image repr esents the L 2 regularization element/te rm. Lambda is a hyperparam eter. INEURON.AI Page 9 If lambda is z ero, then it is equivalent to OLS. B ut if la mbda is very large, then i t will add too much weight , and it will lead to under-fitt ing . Ridge regularizat ion forces the w eights to be small but does not make them zero and does not give the sparse solution . Rid ge is not robust to outliers as square term s blow up the error d ifferences of the outlie rs , and the regularization term tries to fix it by penalizing the weights Ridge regression performs better when all the input f eatures influence the out put, and all with weights are of roughly equal size . L2 regularization ca n learn complex data patt erns . Q8. What i s R square(where to use and w here not)? A ns 8. R-squared is a stat istical measure of how close the data are to the fitted r egression line. It is also known as the coefficient of determination, or th e coefficient of multiple dete rmination for multiple regression. The definition of R -squared is the percentag e of the response variable variation that is explained b y a linear model. R-squared = Explain ed variation / Total variat ion R-squared is alwa y s betwe en 0 and 100%. 0% indicates that the m odel explains none of the variability of the response data around its mean. 100% indicates that the model explains all the variabi lity of the response data aro und its mean. In general, the higher th e R-squared, the better the model fits y our data. INEURON.AI Page 10 There is a problem with the R-Square. T he problem arises when we ask this question to oursel ves.** Is it good to help as m any independent variables as possible?** The answer is No because we understood th at each independent variab le should have a meaningf ul impact. But, even** if we add independent var iables which are not m eaningful**, will it improve R -Square value? Yes, this is the basic pro blem with R-Squ are. How many junk indepen dent variables or im portant independent variable or impactful independent varia ble you add to y our model, the R- Squared value will always increase. It w ill nev er decrease with the addition of a newl y independent variable , whether it could be an impac tful, non-impactful, or ba d variable, so we need an other way to measure eq uivalent R- Square, which pen alizes our model with an y junk independent variable. So, we calculate the Adjusted R- Square with a better adjustm ent in the form ula o f generic R -square. Q9 . What is Mean Squ are Error? The mean squared error tells you how close a regr ession line is to a set of po ints. It does this by taking the distances f rom the points to the reg ression line (these distance s are the “errors”) and squaring them. Giving an intuit ion The line equation is y=Mx+B . We want to find M (slope) and B (y-intercept) that minimizes the squared error.

Data Science - Interview Questions

Engenharias

Ciência de Dados

Continue navegando

Materiais relacionados

Perguntas relacionadas

What data is possible to extract from an interview? a. Only what was planned, with direct and objective questions. b. No subjective information s...

What is the concept of Statistics? I- Statistics is the science of obtaining conclusions from data. II- Statistics is the science of obtaining dat...

What was the data collection instrument used in the research? The data collection instrument was a questionnaire composed of 40 open and closed qu...

A junção de Big Data e Data Science veio para ficar, sendo somente possível o tratamento dos grandes volumes de dados disponíveis na Internet atrav...

Outros materiais