Tag: University of Mumbai BMS Notes

The least square method is the process of finding the best-fitting curve or line of best fit for a set of data points by reducing the sum of the squares of the offsets (residual part) of the points from the curve. During the process of finding the relation between two variables, the trend of outcomes are estimated quantitatively. This process is termed as regression analysis. The method of curve fitting is an approach to regression analysis. This method of fitting equations which approximates the curves to given raw data is the least square.

It is quite obvious that the fitting of curves for a particular data set are not always unique. Thus, it is required to find a curve having a minimal deviation from all the measured data points. This is known as the best-fitting curve and is found by using the least-squares method.

Least Square Method

The least-squares method is a crucial statistical method that is practised to find a regression line or a best-fit line for the given pattern. This method is described by an equation with specific parameters. The method of least squares is generously used in evaluation and regression. In regression analysis, this method is said to be a standard approach for the approximation of sets of equations having more equations than the number of unknowns.

The method of least squares actually defines the solution for the minimization of the sum of squares of deviations or the errors in the result of each equation. Find the formula for sum of squares of errors, which help to find the variation in observed data.

The least-squares method is often applied in data fitting. The best fit result is assumed to reduce the sum of squared errors or residuals which are stated to be the differences between the observed or experimental value and corresponding fitted value given in the model.

There are two basic categories of least-squares problems:

• Ordinary or linear least squares
• Nonlinear least squares

These depend upon linearity or nonlinearity of the residuals. The linear problems are often seen in regression analysis in statistics. On the other hand, the non-linear problems generally used in the iterative method of refinement in which the model is approximated to the linear one with each iteration.

Least Square Method Graph

In linear regression, the line of best fit is a straight line as shown in the following diagram:

The given data points are to be minimized by the method of reducing residuals or offsets of each point from the line. The vertical offsets are generally used in surface, polynomial and hyperplane problems, while perpendicular offsets are utilized in common practice.

Least Square Method Formula

The least-square method states that the curve that best fits a given set of observations, is said to be a curve having a minimum sum of the squared residuals (or deviations or errors) from the given data points. Let us assume that the given points of data are (x₁,y₁), (x₂,y₂), (x₃,y₃), …, (x_n,y_n) in which all x’s are independent variables, while all y’s are dependent ones. Also, suppose that f(x) be the fitting curve and d represents error or deviation from each given point.

Now, we can write:

d₁ = y₁ − f(x₁)

d₂ = y₂ − f(x₂)

d₃ = y₃ − f(x₃)

…..

d_n = y_n – f(x_n)

The least-squares explain that the curve that best fits is represented by the property that the sum of squares of all the deviations from given values must be minimum. i.e:

Sum = Minimum Quantity

Limitations for Least-Square Method

The least-squares method is a very beneficial method of curve fitting. Despite many benefits, it has a few shortcomings too. One of the main limitations is discussed here.

In the process of regression analysis, which utilizes the least-square method for curve fitting, it is inevitably assumed that the errors in the independent variable are negligible or zero. In such cases, when independent variable errors are non-negligible, the models are subjected to measurement errors. Therefore, here, the least square method may even lead to hypothesis testing, where parameter estimates and confidence intervals are taken into consideration due to the presence of errors occurring in the independent variables.

Secondary data is the data that have been already collected by and readily available from other sources. Such data are cheaper and more quickly obtainable than the primary data and also may be available when primary data can not be obtained at all.

Advantages of Secondary data

1. It is economical. It saves efforts and expenses.
2. It is time saving.
3. It helps to make primary data collection more specific since with the help of secondary data, we are able to make out what are the gaps and deficiencies and what additional information needs to be collected.
4. It helps to improve the understanding of the problem.
5. It provides a basis for comparison for the data that is collected by the researcher.

Disadvantages of Secondary Data

1. Secondary data is something that seldom fits in the framework of the marketing research factors. Reasons for its non-fitting are:

• Unit of secondary data collection: Suppose you want information on disposable income, but the data is available on gross income. The information may not be same as we require.
• Class Boundaries may be different when units are same.

1. Thus the data collected earlier is of no use to you.

2. Accuracy of secondary data is not known.
3. Data may be outdated.

Evaluation of Secondary Data

Because of the above mentioned disadvantages of secondary data, we will lead to evaluation of secondary data. Evaluation means the following four requirements must be satisfied:

1. Availability: It has to be seen that the kind of data you want is available or not. If it is not available then you have to go for primary data.
2. Relevance: It should be meeting the requirements of the problem. For this we have two criteria:
   1. Units of measurement should be the same.
   2. Concepts used must be same and currency of data should not be outdated.
3. Accuracy: In order to find how accurate, the data is, the following points must be considered: –

• Specification and methodology used
• Margin of error should be examined
• The dependability of the source must be seen.

4. Sufficiency: Adequate data should be available.

Robert W Joselyn has classified the above discussion into eight steps. These eight steps are sub classified into three categories. He has given a detailed procedure for evaluating secondary data.

• Applicability of research objective.
• Cost of acquisition.
• Accuracy of data.

This data comes from everywhere: sensors used to gather climate information, posts to social media sites, digital pictures and videos, purchase transaction records, and cell phone GPS signals to name a few. This data is big data. What has also changed in the last decade is that we now have the means to sift through these 2.5 quintillion bytes of data in a reasonable amount of time. All these changes have major implications for organizations today.

In organizations, analytics enables professionals to convert extensive data and statistical and quantitative analysis into powerful insights that can drive efficient decisions.

Therefore with analytics, organizations can now base their decisions and strategies on data rather than on gut feelings. Moreover, with the rate at which this data can be analyzed, organizations are able to keep tabs on the customer trends in near real time. As a result effectiveness of a strategy can be determined almost immediately. Thus with powerful insights, analytics promises reduced costs and increased profits.
The analytics Industry is one of the fastest growing in modern times with it poised to become a $50 billion market by 2017. With this sudden surge in the analytics industry, there is a tremendous increase in the demand for analytics expertise across all domains, throughout all major organizations across the globe. It has been predicted that by 2018, the United States alone could face a shortage of 140,000 to 190,000 people with deep analytical skills as well as 1.5 million managers and analysts with the know-how to use the analysis of big data to make effective decisions.
IBM’s recent study revealed that “83% of Business Leaders listed Business Analytics as the top priority in their business priority list.”
Deloitte has mentioned in its study that: Decision makers who can leverage everyday data & information into actionable insights for the growth of their organization by taking reliable decisions, will find themselves in a much better position to achieve strategic growth in their career.

There is an information overload in today’s world and data analytics helps to cut out the clutter to help businesses make safe and smart choices.

A recent report by Nucleus Research found that companies realize a return of USD10.66 for every dollar they invest in analytics.

In the developed economies of Europe, government administrators could save more than €100 billion ($149 billion) in operational efficiency improvements alone by using big data, not including using big data to reduce fraud and errors and boost the collection of tax revenues. Thus big data courses in India are going to be essential in a few years.

There is a saying “Today Data is the new Oil”. Data in today’s business & technology world is absolutely crucial. The Big Data technologies and initiatives are rising to analyze this data for gaining insights that can help in making strategic decisions. The concept evolved at the beginning of 21st century, and every technology giant is now making use of Big Data technologies. Big Data refers to vast data sets that may be structured or unstructured. There is a massive amount of data which has been produced everyday by businesses & users alike. Big data analytics is the process of examining large data sets to find the underlying insights & patterns. Data analytics field is absolutely vast.

The Big Data Analytics is indeed a revolution in the field of information technology. The use of Data analytics by the companies is increasing day by day. The primary focus of the companies is on the customers. Hence this field is flourishing in the area of B2C applications. There are 3 divisions of Big data analytics: Prescriptive Analytics, Predictive Analytics, Descriptive Analytics. There are four different perspectives to explain why big data analytics is so important. They are

• Data Science Perspective
• Business Perspective
•  Real Time Usability Perspective
• Job Market Perspective

Big Data Analytics & Data Science

The analytics involves the use of advanced techniques & tools of analytics on a data obtained via different sources & different sizes. Big Data has the properties of high variety, volume & velocity. The data sets are basically retrieved from various online networks, web pages, audio & video devices, social media, logs & many other sources.

It involves the use of techniques like machine learning, data mining, natural language processing & statistics. The data is extracted, prepared & blended to provide analysis for the businesses.

Benefits of Big Data Analytics

Due to enormous growth in the field of Big Data Analytics it is extensively used in multiple industries like

• Banking
• Healthcare
• Energy
• Technology
• Consumer
• Manufacturing

The importance of big data analytics leads to intense competition and increased demand for big data professionals. Data Science and Analytics is an evolving field with huge potential. Data analytics help in analyzing the value chain of business and gain insights. The use of analytics can enhance the industry knowledge of the analysts. Data analytics experts provide the organizations a chance to learn about the opportunities for the business.

Primary data

Primary data are original observations collected by the researcher or his agent for the first time for any investigation and used by them in the statistical analysis.

The primary data is the one type of important data. It is collection of data from first-hand information.

This information published by one organization for some purposes. This type of primary data is mostly pure and original data.

The primary data collection is having three different data collection methods are:

• Data Collection through Investigation:

In this method, trained investigators are working as employees for collecting the data. The researchers will use the tools like interview and collect the information from the individual persons.

• Personal Investigation Methods:

The researchers or the data collectors will conduct the survey and hence they collect the data. In this method we have to collect more accurate data and original data. This method is useful for small data collection only not big collection of data projects.

• Data Collection through Telephones:

The data researcher uses the tools like telephones, mobile phones to collect the information or data. This is accurate and very quick process for data collection. But information collected is not accurate and true.

(2) secondary data

The secondary data is the other type of data, which is collection of data from second hand information. This information is known as, given data is already collected from any one person for some purpose, and it has available for the present issues. And mostly these secondary data are not relevant and pure or original data.

In Statistics, the basis of all statistical calculations or interpretation lies in the collection of data. There are numerous methods of data collection. In this lesson, we shall focus on two primary methods and understand the difference between them. Both are suitable in different cases and the knowledge of these methods is important to understand when to apply which method. These two methods are the Census method and Sampling method.

Census Method

Census method is the method of statistical enumeration where all members of the population are studied. A population refers to the set of all observations under concern. For example, if you want to carry out a survey to find out student’s feedback about the facilities of your school, all the students of your school would form a part of the ‘population’ for your study.

At a more realistic level, a country wants to maintain information and records about all households. It can collect this information by surveying all households in the country using the census method.

In our country, the Government conducts the Census of India every ten years. The Census appropriates information from households regarding their incomes, the earning members, the total number of children, members of the family, etc. This method must take into account all the units. It cannot leave out anyone in collecting data. Once collected, the Census of India reveals demographic information such as birth rates, death rates, total population, population growth rate of our country, etc. The last census was conducted in the year 2011.

Sampling Method

Like we have studied, the population contains units with some similar characteristics on the basis of which they are grouped together for the study. In the case of the Census of India, for example, the common characteristic was that all units are Indian nationals. But it is not always practical to collect information from all the units of the population.

It is a time-consuming and costly method. Thus, an easy way out would be to collect information from some representative group from the population and then make observations accordingly. This representative group which contains some units from the whole population is called the sample.

The first most important step in selecting a sample is to determine the population. Once the population is identified, a sample must be selected. A good sample is one which is:

• Small in size.
• It provides adequate information about the whole population.
• It takes less time to collect and is less costly.

In the case of our previous example, you could choose students from your class to be the representative sample out of the population (all students in the school). However, there must be some rationale behind choosing the sample. If you think your class comprises a set of students who will give unbiased opinions/feedback or if you think your class contains students from different backgrounds and their responses would be relevant to your student, you must choose them as your sample. Otherwise, it is ideal to choose another sample which might be more relevant.

Again, realistically, the government wants estimates on the average income of the Indian household. It is difficult and time-consuming to study all households. The government can simply choose, say, 50 households from each state of the country and calculate the average of that to arrive at an estimate. This estimate is not necessarily the actual figure that would be arrived at if all units of the population underwent study. But it approximately gives an idea of what the figure might look like.

Difference between Census and Sample Surveys

• It is the process of arranging data into homogeneous (similar) groups according to their common characteristics.
• Raw data cannot be easily understood and it is not fit for further analysis and interpretation. This arrangement of data helps users in comparison and analysis.
• For example, the Population of town can be grouped according to sex, age, marital status etc.

Classification of data

The method of arranging data into homogeneous classes according to some common features present in the data is called classification.

A planned data analysis system makes fundamental data easy to find and recover. This can be of particular interest for legal discovery, risk management and compliance. Written methods and set of guidelines for data classification should determine what levels and measures the company will use to organise data and define the roles of employees within the business regarding input stewardship. Once a data-classification scheme has been designed, security standards that stipulate proper approaching practices for each division and storage criteria that determine the data’s lifecycle demands should be discussed.

Objectives of Data Classification

The primary objectives of data classification are:

• To consolidate the volume of data in such a way that similarities and differences can be quickly understood. Figures can consequently be ordered in a few sections holding common traits.
• To aid comparison.
• To point out the important characteristics of the data at a flash.
• To give importance to the prominent data collected while separating the optional elements.
• To allow a statistical method of the material gathered.

Methods of Classification

An important branch of mathematics that deals with gathering, organizing, estimating and interpreting the vast numerical data for a survey or a research, is known as statistics. There may be one or more numbers of statistical data that are used more than once. The number of times a particular data item is utilized, is known as its frequency.

When the distribution of frequencies is listed in a table OR tabular presentation of frequency distribution, known as frequency table. It is used to list out one or more variables taken in a sample. Each sample contains an individual frequency and each frequency is distributed with an interval between each frequency. It is also of two types that is univariate and joint. Frequency distribution can be defined as a summary presentation of the number of observations of an attribute or values of a variable arranged according to their magnitudes either individually in the case of discrete series or in a range or class interval in the case of both discrete and continuing series.

Frequency Table

Frequency Distribution Table is a way to organize data. A frequency distribution table is an organized tabulation of the number of individual events located in each category. It contains at least two columns, one for the score categories (X) and another for the frequencies (f). Below we have explained briefly for you to understand the concept of frequency table better and workout frequency table example:

Solved Example

Question: Here is the list of marks obtained for the students in the examination. Find the number of students who got more than 85 marks, More than 95, Less than 80 more than 76.

Solution:

From the table we can conclude that:

Students who got more than 85 = 8 + 5 + 1 = 14

Students who got more than 95 = 1

Students who got less than 80 more than 76 = 14.

Construction of Frequency Distribution

The following steps are involved in the construction of a frequency distribution.

(1) Find the range of the data: The range is the difference between the largest and the smallest values.

(2) Decide the approximate number of classes in which the data are to be grouped. There are no hard and first rules for number of classes. In most cases we have 5 to 20 classes. H.A. Sturges provides a formula for determining the approximation number of classes.

K=1+3.322logN

where K= Number of classes

and logN = Logarithm of the total number of observations.

Example: If the total number of observations is 50, the number of classes would be

K=1+3.322logN

K=1+3.322log50

K=1+3.322(1.69897)

K=1+5.644

K=6.644

7 classes, approximately.

(3) Determine the approximate class interval size: The size of class interval is obtained by dividing the range of data by the number of classes and is denoted by h class interval size

h = Range Number of Classes

In the case of fractional results, the next higher whole number is taken as the size of the class interval.

(4) Decide the starting point: The lower class limit or class boundary should cover the smallest value in the raw data. It is a multiple of class intervals.

Example: 0,5,10,15,20, etc. are commonly used.

(5) Determine the remaining class limits (boundary): When the lowest class boundary has been decided, by adding the class interval size to the lower class boundary you can compute the upper class boundary. The remaining lower and upper class limits may be determined by adding the class interval size repeatedly till the largest value of the data is observed in the class.

(6) Distribute the data into respective classes: All the observations are divided into respective classes by using the tally bar (tally mark) method, which is suitable for tabulating the observations into respective classes. The number of tally bars is counted to get the frequency against each class. The frequency of all the classes is noted to get the grouped data or frequency distribution of the data. The total of the frequency columns must be equal to the number of observations.

Data can be presented in the form of organized information, combined in tables or even graphically represented. Imagine seeing a set of data in the written form or in tabular form versus a graph that gives you the same information. Isn’t it simpler and quicker to comprehend data if we can visually see it?

It is for this purpose that data can be organized graphically for interpretation in a single glance in Statistics. The two forms of graphical representation that we shall cover in this lesson are bar diagram and histogram.

Bar Diagram

Also known as a column graph, a bar graph or a bar diagram is a pictorial representation of data. It is shown in the form of rectangles spaced out with equal spaces between them and having equal width. The equal width and equal space criteria are important characteristics of a bar graph.

Note that the height (or length) of each bar corresponds to the frequency of a particular observation. You can draw bar graphs both, vertically or horizontally depending on whether you take the frequency along the vertical or horizontal axes respectively. Let us take an example to understand how a bar graph is drawn.

The above table depicts the number of students of a class engaged in any one of the three sports given. Note that the number of students is actually the frequency. So, if we take frequency to be represented on the y-axis and the sports on the x-axis, taking each unit on the y-axis to be equal to 5 students, we would get a graph that resembles the one below.

The blue rectangles here are called bars. Note that the bars have equal width and are equally spaced, as mentioned above. This is a simple bar diagram.

Histogram

A bar diagram easy to understand but what is a histogram? Unlike a bar graph that depicts discrete data, histograms depict continuous data. The continuous data takes the form of class intervals. Thus, a histogram is a graphical representation of a frequency distribution with class intervals or attributes as the base and frequency as the height.

The key difference is that histograms have bars without any spaces between them and the rectangles need not be of equal width. So, we will understand histograms using an example.

In this case, see that we are considering class intervals such as 0-5, 5-10, 10-15 and 15-20. These are continuous data. In case, the class intervals given to you are not continuous, you must make it continuous first.

Here, you can interpret the histogram using the information that the graph gives. Consider the frequency to be as given on the left vertical axis and ignore the values on the right vertical axis. Thus, for the class interval 0-5, the corresponding frequency is 3. Again, for 5-10, the frequency is 7, and so on.

Note that we have taken the simple case of a histogram with bars of equal width. But as mentioned, it might not be the case if the class intervals are not even in size. In that case, you will get a histogram with bars stuck to each other (without any space between them) but with different widths. It could look something like this, but exactly how it will look depends on the data:

A pie chart (or a pie graph) is a circular statistical graphical chart, which is divided into slices in order to explain or illustrate numerical proportions. In a pie chart, centeral angle, area and an arc length of each slice is proportional to the quantity or percentages it represents. Total percentages should be 100 and total of the arc measures should be 360° Following illustration of pie graph depicts the cost of construction of a house.

From this graph, one can compare the sum spent on cement, steel and so on. One can also compute the actual sum spent on each individual expense. Consider an example, where we want to know how much more is the labour cost when compared to cost of steel.

Amount spent on labor =9060×600000=$ 150000

Sum spent on steel =54/360×600000=$ 90000

Excess=150000−90000=$ 60000

Let 60000=x% of 600000

⟹x/100×600000=$ 60000

⟹x=10% of total expense.