Random feature

Random features (RF) are a technique used in machine learning to approximate kernel methods, introduced by Ali Rahimi and Ben Recht in their 2007 paper "Random Features for Large-Scale Kernel Machines",^[1] and extended by ^[2][3]. RF uses a Monte Carlo approximation to kernel functions by randomly sampled feature maps. It is used for datasets that are too large for traditional kernel methods like support vector machine, kernel ridge regression, and gaussian process.

Given a feature map ${\textstyle \phi :\mathbb {R} ^{d}\to V}$, where ${\textstyle V}$ is a Hilbert space (more specifically, a reproducing kernel Hilbert space), the kernel trick replaces inner products in feature space $${\displaystyle \langle \phi (x_{i}),\phi (x_{j})\rangle _{V}}$$ by a kernel function$${\displaystyle k(x_{i},x_{j}):\mathbb {R} ^{d}\times \mathbb {R} ^{d}\to \mathbb {R} }$$Kernel methods replaces linear operations in high-dimensional space by operations on the kernel matrix: $${\displaystyle K_{X}:=[k(x_{i},x_{j})]_{i,j\in 1:N}}$$ where ${\textstyle N}$ is the number of data points.

The problem with kernel methods is that the kernel matrix ${\textstyle K_{X}}$ has size ${\textstyle N\times N}$. This becomes computationally infeasible when ${\textstyle N}$ reaches the order of a million. The random kernel method replaces the kernel function ${\textstyle k}$ by an inner product in low-dimensional feature space ${\textstyle \mathbb {R} ^{D}}$:$${\displaystyle k(x,y)\approx \langle z(x),z(y)\rangle }$$ where ${\textstyle z}$ is a randomly sampled feature map ${\textstyle z:\mathbb {R} ^{d}\to \mathbb {R} ^{D}}$.

This converts kernel linear regression into linear regression in feature space, kernel SVM into SVM in feature space, etc. Since we have ${\displaystyle K_{X}\approx Z_{X}^{T}Z_{X}}$ where ${\displaystyle Z_{X}=[z(x_{1}),\dots ,z(x_{N})]}$, these methods no longer involve matrices of size ${\textstyle O(N^{2})}$, but only random feature matrices of size ${\textstyle O(DN)}$.

The radial basis function (RBF) kernel on two samples ${\displaystyle x_{i},x_{j}\in \mathbb {R} ^{d}}$ is defined as^[4]

${\displaystyle k(x_{i},x_{j})=\exp \left(-{\frac {\|x_{i}-x_{j}\|^{2}}{2\sigma ^{2}}}\right)}$

where ${\displaystyle \|x_{i}-x_{j}\|^{2}}$ is the squared Euclidean distance and ${\displaystyle \sigma }$ is a free parameter defining the shape of the kernel. It can be approximated by a random Fourier feature map ${\displaystyle z:\mathbb {R} ^{d}\to \mathbb {R} ^{2D}}$:$${\displaystyle z(x):={\frac {1}{\sqrt {D}}}[\cos \langle \omega _{1},x\rangle ,\sin \langle \omega _{1},x\rangle ,\ldots ,\cos \langle \omega _{D},x\rangle ,\sin \langle \omega _{D},x\rangle ]^{T}}$$where ${\displaystyle \omega _{1},...,\omega _{D}}$ are IID samples from the multidimensional normal distribution ${\displaystyle N(0,\sigma ^{-2}I)}$.

Since ${\displaystyle \cos ,\sin }$ are bounded, there is a stronger convergence guarantee by Hoeffding's inequality.^{[1]: Claim 1}

By Bochner's theorem, the above construction can be generalized to arbitrary positive definite shift-invariant kernel ${\displaystyle k(x,y)=k(x-y)}$.

Define its Fourier transform$${\displaystyle p(\omega )={\frac {1}{2\pi }}\int _{\mathbb {R} ^{d}}e^{-j\langle \omega ,\Delta \rangle }k(\Delta )d\Delta }$$then ${\displaystyle \omega _{1},...,\omega _{D}}$ are sampled IID from the probability distribution with probability density ${\displaystyle p}$. This applies for other kernels like the Laplace kernel and the Cauchy kernel.

Given a random Fourier feature map ${\displaystyle z}$, training the feature on a dataset by featurized linear regression is equivalent to fitting complex parameters ${\displaystyle \theta _{1},\dots ,\theta _{D}\in \mathbb {C} }$ such that$${\displaystyle f_{\theta }(x)=\mathrm {Re} \left(\sum _{k}\theta _{k}e^{i\langle \omega _{k},x\rangle }\right)}$$which is a neural network with a single hidden layer, with activation function ${\displaystyle t\mapsto e^{it}}$, zero bias, and the parameters in the first layer frozen.

In the overparameterized case, when ${\displaystyle 2D\geq N}$, the network linearly interpolates the dataset ${\displaystyle \{(x_{i},y_{i})\}_{i\in 1:N}}$, and the network parameters is the least-norm solution:$${\displaystyle {\hat {\theta }}=\arg \min _{\theta \in \mathbb {C} ^{D},f_{\theta }(x_{k})=y_{k}\forall k\in 1:N}\|\theta \|}$$At the limit of ${\displaystyle D\to \infty }$, the L2 norm ${\displaystyle \|{\hat {\theta }}\|\to \|f_{K}\|_{H}}$ where ${\displaystyle f_{K}}$ is the interpolating function obtained by the kernel regression with the original kernel, and ${\displaystyle \|\cdot \|_{H}}$ is the norm in the reproducing kernel Hilbert space for the kernel.^[5]

A random binning features map partitions the input space using randomly shifted grids at randomly chosen resolutions and assigns to an input point a binary bit string that corresponds to the bins in which it falls. The grids are constructed so that the probability that two points ${\displaystyle x_{i},x_{j}\in \mathbb {R} ^{d}}$ are assigned to the same bin is proportional to ${\displaystyle K(x_{i},x_{j})}$. The inner product between a pair of transformed points is proportional to the number of times the two points are binned together, and is therefore an unbiased estimate of ${\displaystyle K(x_{i},x_{j})}$. Since this mapping is not smooth and uses the proximity between input points, Random Binning Features works well for approximating kernels that depend only on the ${\displaystyle L_{1}}$ distance between datapoints.

Orthogonal random features^[6] uses a random orthogonal matrix instead of a random Fourier matrix.

In NIPS 2006, deep learning had just become competitive with linear models like PCA and linear SVMs for large datasets, and people speculated about whether it could compete with kernel SVMs. However, there was no way to train kernel SVM on large datasets. The two authors developed the random feature method to train those.

It was then found that the ${\displaystyle O(1/D)}$ variance bound did not match practice: The variance bound predicts that approximation to within ${\displaystyle 0.01}$ requires ${\displaystyle D\sim 10^{4}}$, but in practice required only ${\displaystyle \sim 10^{2}}$. Attempting to discover what caused this led to the subsequent two papers ^[2][3].^[7]

• Kernel method
• Support vector machine
• Fourier transform
• Monte Carlo method

• Random Walks - Random Fourier features