Читать книгу Data mining. Textbook - Vadim Shmal - Страница 7

Classification is the task of generalizing a known structure to be applied to new data. For example, an email program might try to classify an email as «legitimate», or «spam», or maybe «deleted by the administrator», and if it does this correctly, it can mark the email as relevant to the user.

However, for servers, the classification is more complex because storage and transmission are far away from users. When servers consume huge amounts of data, the problem is different. The job of the server is to create a store and pass that store around so that servers can access it. Thus, servers can often avoid disclosing particularly sensitive data if they can understand the meaning of the data as it arrives, unlike the vast pools of data often used for email. The problem of classification is different and needs to be approached differently, and current classification systems for servers do not provide an intuitive mechanism for users to have confidence that servers are classifying their data correctly.

This simple algorithm is useful for classifying data in databases containing millions or billions of records. The algorithm works well, provided that all relationships in the data are sufficiently different from each other and that the data is relatively small in both columns and rows. This makes data classification useful in systems with relatively little memory and little computation, and therefore the classification of large datasets remains a major unsolved problem.

The simplest classification algorithm for classifying data is the total correlation method, also known as the correlation method. In full correlation, you have two sets of data and you are comparing data from one set to data from another set. This is easy to do for individual pieces of data. The next step is to calculate the correlation between the two datasets. This correlation of two sets of data tells you what percentage of the data is in each set. Thus, using this correlation, you can classify data as either one set or the other, indicating the parts of the data set that come from one set or the other.

This simple method often works well for data stored in simple databases with a small amount of data and slow data access speeds. For example, a database system may use a tree structure to store data, with the columns of a record representing fields in the structure. This structure did not allow data to be ranked because the data would be in two separate rows of the tree structure. This makes it impossible to make sense of the data if the data fits in only one tree structure. If the database has two data trees, you will need to compare each of the two trees. If there were a large number of trees, the comparison could be computationally expensive.

Therefore, full correlation is a poor classification method. Data correlation does not distinguish between relevant parts of the data, and the data is relatively small in both columns and rows. These problems make full correlation unsuitable for simple data classification systems and data storage systems. However, if the data is relatively large, full correlation can be applied. This example is useful for storage systems with a relatively high computational load.

Combining a data classification method with a data storage system improves both performance and usability. In particular, the size of the resulting classification algorithm is largely independent of the size of the data store. The detailed classification algorithm does not require a lot of memory to store data at all. It is often small enough to be buffered, and many organizations store their classification systems this way. Also, the performance characteristics of the storage system do not depend on the classifier. The storage system can handle data with a high degree of variability.

Why are classification systems not so good?

Most storage systems do not have a good classifier, and the data classification system is unlikely to get better over time. If your storage system does not have a good classifier, your classification system will have problems.

Most companies don’t think this way about their storage systems. Instead, they assume that the system can be fixed. They see it as something that can be improved over time based on future maintenance efforts. This belief also makes it easy to fix some of the problems that come from bad storage systems. For example, a storage system that doesn’t accept overly short or jumbled data can be improved over time if more people are involved in fixing it.