Читать книгу Statistical Significance Testing for Natural Language Processing - Rotem Dror - Страница 7

Contents

Preface

Acknowledgments

1 Introduction

2 Statistical Hypothesis Testing

2.1 Hypothesis Testing

2.2 P-Value in the World of NLP

3 Statistical Significance Tests

3.1 Preliminaries

3.2 Parametric Tests

3.3 Nonparametric Tests

4 Statistical Significance in NLP

4.1 NLP Tasks and Evaluation Measures

4.2 Decision Tree for Significance Test Selection

4.3 Matching Between Evaluation Measures and Statistical Significance Tests

4.4 Significance with Large Test Samples

5 Deep Significance

5.1 Performance Variance in Deep Neural Network Models

5.2 A Deep Neural Network Comparison Framework

5.3 Existing Methods for Deep Neural Network Comparison

5.4 Almost Stochastic Dominance

5.5 Empirical Analysis

5.6 Error Rate Analysis

5.7 Summary

6 Replicability Analysis

6.1 The Multiplicity Problem

6.2 A Multiple Hypothesis Testing Framework for Algorithm Comparison

6.3 Replicability Analysis with Partial Conjunction Testing

6.4 Replicability Analysis: Counting

6.5 Replicability Analysis: Identification

6.6 Synthetic Experiments

6.7 Real-World Data Applications

6.7.1 Applications and Data

6.7.2 Statistical Significance Testing

6.7.3 Results

6.7.4 Results Summary and Overview

7 Open Questions and Challenges

8 Conclusions

Bibliography

Authors’ Biographies