# Questions tagged [compression]

Data compression is a process used to reduce the number of bits used to store a "message". Compression can be lossless or lossy. Lossy compression is an option for audio and visual data, whereas many other applications require lossless compression.

50 questions
Filter by
Sorted by
Tagged with
19 views

### Is there a natural extension of the ACF/PACF to more general measures of dependency?

Assume that we have a time series that we want to model as a stationary real-valued stochastic process: $$X_t , t \in \{0, 1, \dots \}.$$ Two complementary measures of linear dependence between the ...
27 views

### Optimal variable-time sampling of a real-time data stream

Here's a signal processing/information theory problem that I've encountered in a software engineering context: Say I have a logging utility in my application that I use for recording timestamped ...
1 vote
96 views

### Linear Distance in Latent Feature Space of an AutoEncoder

I would like to perform a cluster analysis on a mixed data set containing continuous, categorical and binary data. As I have 93 features in total, I thought it might help to use an AutoEncoder to ...
35 views

### Is there a Hidden Markov Model compression scheme for time series?

Hidden Markov Models (HMMs) are very useful for time series analysis and inference. At the same time, probability distributions over a data type are used in finding compression schemes for data of ...
114 views

### Reverse engineering device RGB data

I'm working with a device that maps certain RGB colors to a 7 bit value (0-127): I want to reverse the process, i.e. given any RGB triplet, what is the (closest) corresponding color index (0-127)? ...
48 views

### Rotation-sensitivity of SVD

Suppose I perform a truncated SVD on a symmetric, PSD matrix $A \in R^{N \times d}$ (lowering the dimensionality from $d$ to $k$). Further suppose that there is a rotation matrix $Q$ such that some of ...
1 vote
23 views

### Evaluate two lossy compression algorithms

I am trying to evaluate several methods to compress some 2D data points. The algorithm itself is not relevant, but from the output, I can compute the MSE and the number of points (which can be used to ...
309 views

### Understanding Minimum Description Length for Time Series

I am trying to reproduce (in Python) the minimum description length work found near Figure 6 from this paper along with their sample Matlab code. ...
96 views

201 views

### Iterative PCA in C++

I have a time-series data set (10,000 samples/sensors over ~2,000 time-series samples). I would like to keep the covariance structure in the data (sensor-sensor covariance) up to some error, while ...
1 vote
279 views

115 views

### Local redundancy - Product Quantization and global redundancy - Residual Quantization connection

I am currently reading Compressing Deep Convolutional Networks using Vector Quantization paper. The paper states in section 3.2.5 that Product Quantization explores some local redundancy and Residual ...
443 views

### Why low rank expansions can exploit the redundancy that exist between different feature channels and filters?

I read Jaderberg et al., 2014 paper about Speeding up Convolutional Neural Network with Low Rank Expansions. In the introduction, it is written in bold font: Our key insight is to exploit the ...
303 views

### Should we pre-train neural networks on compressed data?

I believe this is actually related to the topic of "auto-encoding" but my question is trying to express some basic gaps in my knowledge that I have not seen directly addressed anywhere. You can train ...
120 views

### Detecting if a File is Compressed using Machine Learning?

Let's say that I want a 'general' way using some machine learning model to classify if a file is possibly compressed or not. I don't need a 100% success rate. How would you approach this? I ...
1 vote
99 views

### Clustering technique and validation for distance based on file compression

I have a distance matrix based on a normalized compression distance between files: $$d(x, y) = \frac{ C(xy) - \min \{ C(x), C(y) \} } {\max \{ C(x), C(y) \}}$$ Here, $C(xy)$ is the concatenation ...
16 views

### Removing minimally informative bits

Each sample in my data set is an $N$-bit bit-vector (say $N$=200). The bits in each sample are not uncorreleated within the sample. I built a matrix $S_{N \times N}$ with each $s_{ij}$ being the ...
196 views

### Analyzing 3D data: What can be done?

I am new to this kind of analysis, and I want to know what values I can look at in 3D data. The data itself is a 3D volume $(x,y,z)$ with a floating point value in every coordinate. It is a ...
10k views

### Comparison of entropy and distribution of bytes in compressed/encrypted data

I have some question which occupies myself for a while. The entropy test is often used to identify encrypted data. The entropy reaches its maximum when the bytes of the analyzed data are distributed ...
437 views

### Distance independent approximation of Nearest Neighbor/k-NN.

Nearest neighbor/k-NN for use with Normalized Compression Distance. I wonder if there exist any approximation of NN/k-NN algorithm which work for all distance measures ? I would like to test ...
1 vote
727 views

### Dense Data Compression (Delta Encoding?)

Let $X$ be a dense integer set such that its elements are closely knit in value. For instance: 1 1 1 1 2 2 2 2 2 3 3 5 5 6 6 6 6 6 7 7 7 7 8 8 8 8 8 8 ... I am ... 138 views

### Weights of random sets of random 32-bit strings

I have random sets of $N$ random 32-bit strings, where all bits are i.i.d. with $\mathbb{P}(0) = \mathbb{P}(1) = 1/2$. Define $\ \ \ \$weight( 32-bit x ) = number of 1 bits in x, i.e. Hamming ...
1k views

### How to compress sets of integer series?

I have a set of integer series $S_1$, $S_2$, ... $S_n$. Each series has 3600 data points. Each data point is a positive integer. Each data point is stored as an ...
1k views

### Ultimate compression algorithm

I was not sure where to put this question, so I put it here. Feel free to move it to another stack exchange site moderators. Lets say I have a 10 gigs of pictures (or for that matter any type of data,...
280 views

### Compression theory, practice, for time series with values in a space of distributions (say of a real random variable)

Example of problem: Part of our research team is working on providing operationally wind power forecast. Usually, since there are different time scalse that interest forecast user, a forecast is ...