“Notes on a New Philosophy of Empirical Science”, (2011-04-28):
This book presents a methodology and philosophy of empirical science based on large scale lossless data compression. In this view a theory is scientific if it can be used to build a data compression program, and it is valuable if it can compress a standard benchmark database to a small size, taking into account the length of the compressor itself. This methodology therefore includes an Occam principle as well as a solution to the problem of demarcation. Because of the fundamental difficulty of lossless compression, this type of research must be empirical in nature: compression can only be achieved by discovering and characterizing empirical regularities in the data. Because of this, the philosophy provides a way to reformulate fields such as computer vision and computational linguistics as empirical sciences: the former by attempting to compress databases of natural images, the latter by attempting to compress large text databases. The book argues that the rigor and objectivity of the compression principle should set the stage for systematic progress in these fields. The argument is especially strong in the context of computer vision, which is plagued by chronic problems of evaluation.
The book also considers the field of machine learning. Here the traditional approach requires that the models proposed to solve learning problems be extremely simple, in order to avoid overfitting. However, the world may contain intrinsically complex phenomena, which would require complex models to understand. The compression philosophy can justify complex models because of the large quantity of data being modeled (if the target database is 100 Gb, it is easy to justify a 10 Mb model). The complex models and abstractions learned on the basis of the raw data (images, language, etc) can then be reused to solve any specific learning problem, such as face recognition or machine translation.
I’m compressing ~1.3 GB folders each filled with 1440 JSON files and find that there’s a 15-fold difference between using the
tarcommand on macOS or Raspbian 10 (Buster) and using Python’s built-in
tarfilelibrary…The output is:
- zsh tar filesize: 23.7 MB
- py tar filesize: 1.49 MB
…The zsh archive uses an unknown order, and the Python archive orders the file by modification date. I am not sure if that matters…EDIT: Ok, I think I found the issue: BSD
tarwithout any sort options put the files in the archive in an undefined order… I think the reason sorting has such an impact is as follows:
My JSON files contain measurements from hundreds of sensors. Every minute I read out all sensors, but only a few of these sensors have a different value from minute to minute. By sorting the files by name (which has the creation Unix time at the beginning of it), two subsequent files have very little different characters between them. Apparently this is very favourable for the compression efficiency.