Data mining. Textbook

Оглавление

Vadim Shmal. Data mining. Textbook

Data mining

Agent mining

Anomaly detection

Association Rule Learning

Clustering

Classification

Summing

Data personalization in forecasting

State Space Search

Knowledge representation and reasoning

Expert systems

Learning Classifier Systems

Intelligent Process Analysis

Multi – agent system

Adaptive control

Knowledge engineering

Machine learning

Neural networks

List of used literature

Отрывок из книги

Data mining is the process of extracting and discovering patterns in large datasets using methods at the interface of machine learning, statistics, and database systems, especially databases containing large numerical values. This includes searching large amounts of information for statistically significant patterns using complex mathematical algorithms. Collected variables include the value of the input data, the confidence level and frequency of the hypothesis, and the probability of finding a random sample. It also includes optimizing the parameters to get the best pattern or result, adjusting the input based on some facts to improve the final result. These parameters include parameters for statistical means such as sample sizes, as well as statistical measures such as error rate and statistical significance.

The ideal scenario for data mining is that the parameters are in order, which provides the best statistical results with the most likely success values. In this ideal scenario, data mining takes place within a closed mathematical system that collects all inputs to the system and produces the most likely outcome. In fact, the ideal scenario is rarely found in real systems. For example, in real life this does not happen when engineering estimates for a real design project are received. Instead, many factors are used to calculate the best measure of success, such as project parameters and the current difficulty of bringing the project to the project specifications, and these parameters are constantly changing as the project progresses. While they may be useful in certain situations, such as the development of specific products, their values should be subject to constant re-evaluation depending on the current conditions of the project. In fact, the best data analysis happens in a complex mathematical structure of problems with many variables and many constraints, and not in a closed mathematical system with only a few variables and a closed mathematical structure.

.....

Revealing the Data Anomalies Significance

In the context of evaluating data anomalies, it is useful to identify the relevant circumstances. For example, if there is an anomaly in the number of delayed flights, it may happen that the deviation is quite small. If many flights are delayed, it is more likely that the number of delays is very close to the natural probability. If there are several flights that are delayed, it is unlikely that the deviation is much greater than the natural probability. Therefore, this will not indicate a significantly higher deviation. This suggests that the data anomaly is not a big deal.

.....