Data Analysis for Bad DataWhen I read and listen to the stories written by people in various fields and positions, I learned that there are various types of "data analysis jobs." I have been involved in data analysis in my work for about 20 years, and I myself have done various forms of data analysis.
We use "data" in a broad sense, but we also use numerical data in a database, and we also use information in our heads as data. Even with numerical data, there are various purposes, goals, and things to do to achieve them.
I'm trying to figure out what kind it is.
This page is based on my memory so far. There may be other types of data analysis jobs out there that aren't on this page, and if we find any in the future, we'll add them. In addition, the expressions "many and few" in the text are only based on the author's experience.
In the explanation of data analysis, the story often starts from the state where there is table data.
On the other hand, when analyzing data in practice, it is normal to start from where there is no data to be analyzed.
Data analysis is rarely started by someone asking for it.
For example, people who work in quality control, production management, marketing, etc., analyze data in their main business. In such a case, data analysis begins in the flow of advancing the main business.
In that case, you need to think about what kind of data analysis you want to do.
When analyzing data, the data for data analysis is not always prepared in advance. If it's in a database or somewhere on the internet, you can search for it and use it, but sometimes you need data that doesn't exist anywhere in the world.
If the data used for data analysis can be created, measured , and prepared by the person who conducts data analysis, the number of things that can be done in data analysis will increase dramatically. Also, if you have knowledge of data analysis in the measurement, good quality data will be collected.
The order shown in the diagram is the preferred order. For example, there is a long way to go to "build a system", but depending on the time and situation, we may skip a step and start building the system.
There are people out there whose job it is to collect and report data according to the needs of their clients.
In that kind of work, being able to freely extract and process data is useful to someone.
In the explanation of data analysis, the expression "data is a treasure trove, and finding treasure is the job of data analysis" is sometimes used.
However, even if data analysis reveals things like, "This and this are correlated, and this and this are not", or "Such patterns occur from time to time," it is not possible to do work that involves the data on a regular basis. It is often said that "it is common sense" for those who are This often happens when you just apply data analysis techniques to the data and summarize the results.
Going back to the previous stage, it is sometimes meaningful to simply summarize the data and visualize the actual situation in a graph or the like. For example, "I thought this pattern was rare, but it's surprisingly common." These things are also useful. It seems that useful information can be derived most efficiently when people who know the background of the data well and people who know various methods of data aggregation and visualization work together.
For example, as a result of data analysis, if we find the cause of defective products, we can solve the problem by preventing the cause from occurring. In such a case, there is no need to think about continuous use of data.
In the explanation of data analysis, the solution is also about data, and the continuous use of data, such as creating a system or creating a business model, comes out behind data analysis. However, in reality, there are many cases where this is not the case.
As you can see on the Data Science Jobs page, data analysis that is meaningful for creating a machine learning model is relatively rare among all data analysis cases.
Depending on the business model , even if you create a system, you may not use a machine learning model, and even if so, it may be a digital transformation (DX) .
Since data analysis is a job here, the end is "report analysis results". The other party of the report can be the boss, another department, the client's company, etc.
There are various routes that lead here. For example, "finish by aggregating data" and "finish by creating a system" are quite different, but whether or not it is good depends on the purpose of the data analysis. If there are people who want the aggregated data, it will be the end if they aggregate it.
NEXT Data Analysis for Bad Data
