A multimodal differential privacy framework based on fusion representation learning

Abstract

Differential privacy mechanisms vary in modalities, and there have been many methods implementing differential privacy on unimodal data. Few studies focus on unifying them to protect multimodal data, though privacy protection of multimodal data is of great significance. In our work, we propose a multimodal differential privacy protection framework. Firstly, we use multimodal representation learning to fuse different modalities and map them to the same subspace. Then based on this representation, we use the Local Differential Privacy (LDP) mechanism to protect data. We propose two protection methods for low-dimensional and high-dimensional fusion tensors respectively. The former is based on Binary Encoding, and the latter is based on multi-dimensional Fourier Transform. To the best of our knowledge, we are the first to propose LDP-based methods for the representation learning of multimodal fusion. Experimental results demonstrate the flexibility of our framework where both approaches show efficient performance as well as high data utility.

Keywords:

• Differential privacy
• Fourier transform
• binary encoding
• multimodal fusion
• low-rank decomposition
• representation learning

1. Introduction

The ways humans perceive the world vary with a direct display of different modalities, such as images, text, voice and other sensory data and structured data like demographic data. These multimodal data can give humans comprehensive information to make better judgments (Baltrušaitis et al., 2018). During the past decade, the ways and the amount of data collected greatly increase to bring us an unprecedented opportunity to perceive some phenomena behind them. Naturally enough, there is commonly used information in tabular data, symptom description in text data, CT image data for one patient, comments in text data, posted pictures in image data and demographic data for one netizen. It is natural for any individual to have multimodal information.

Every unimodal information has its advantages. For instance, when identifying humans crying, audio information contributes more than video information (Atrey et al., 2010). It has been found that multimodal data classifiers perform better than unimodal data classifiers (Ngiam et al., 2011). This indicates that the integration of multimodal data can often make better judgements, which is consistent with the way of human understanding the world.

Though methods vary in learning information of inter- and intra-modality, multimodal fusion has gained popularity in recent years due to its flexibility in fusing at different levels as well as capturing cross information among modalities (Atrey et al., 2010). Traditional multimodal fusion methods are difficult to capture nonlinear relationships. The development of deep learning has brought new ideas to multimodal fusion, in which multimodal fusion representation learning has gradually become the mainstream.

Based on information theory, differential privacy can resist the arbitrary computational power of adversary (Dwork, 2008). It balances the problem of protecting personal privacy while preserving the distribution, which coincides with the urgent need for data analysis nowadays. Further, it provides rigorous privacy protection rules. When the input of DP is a database, it's named Centralized Differential Privacy (CDP) which requires a trusted third party to collect raw data. Unlike CDP, Local Differential Privacy (LDP) takes each individual as input which offers a stronger privacy protection level. Since LDP does not need to be constrained by trusted third parties, it is widely used by Apple, Google, Microsoft and other organisations (Cormode et al., 2018). However, differential privacy is a mechanism and needs to be elaborated for specific scenes to gain specific algorithms. The diversity of modality generates different definitions of differential privacy algorithms, which leads to difficulties in unifying them into one algorithm.

Further, academia and industry are exploring the multimodal field after the progress of unimodal techniques. For example, multimodal data were used in recommendation systems to obtain representations of items and users (Deldjoo et al., 2022). The traditional differential privacy methods for unimodal data have been difficult to meet the demands of data utility and privacy protection because of the correlation between different modalities.

This brings us to the purpose of our work. We propose a multimodal Local Differential Privacy (LDP) framework for deep learning. We also propose two LDP approaches for different representation dimensions. The first is based on low-rank decomposition which obtains the low-dimensional vector representation, and the second approach can handle high-dimensional fusion representation situations. Specifically, our contributions are as follows:

• To the best of our knowledge, we are the first to propose a general multimodal differential privacy framework in deep representation learning to deal with various kinds of modalities. The framework is composed of three modules including Encoding, Perturbation and Aggregation. The models inside Encoding and Aggregation modules can be replaced according to specific tasks.

• The first approach we propose is suitable for low-dimensional representation. The coding module adopts the low-rank decomposition based on Canonical Polyadic (CP) decomposition, which does not need to directly obtain the high-dimensional fusion tensor but to explicitly obtain the low-dimensional fusion representation. The Binary Encoding-based LDP mechanism is then conducted to output protected representations, which can be aggregated by an untrusted third party.

• Since the low-rank decomposition is only approximate and will lose some information inevitably, we propose the second approach which firstly adopts Tensor Fusion Network (TFN) to obtain a high-dimensional tensor, then based on the multi-dimensional Discrete Fourier Transform, we perturb the data in the form of complex tensor according to $(\alpha ,\text{β})-DP$, a version of approximate differential privacy.

• Experimental results on several commonly used multimodal datasets show that data processed by our privacy protection method remain high utility even in a low privacy budget.

The remainder of this paper is structured as follows: Section 2 and Section 3 introduce related works and preliminary concepts respectively. Section 4 presents our proposed framework in detail, including Encoding, Perturbation and Aggregation modules. Section 5 introduces our Binary Encoding-based LDP approach and Fourier transform based LDP approach for different representation dimensions. Section 6 describes datasets carried on in our work and experimental settings. In Section 7, experimental results are illustrated as well as further discussion. Section 8 concludes the paper.

2. Related work

Differential Privacy was proposed by Dwork (2008). It has become a de facto standard in the data privacy protection field. Many works have been done for unimodal data with differential privacy but few works on multimodal fusion data. A differential privacy mechanism named $E_{d_{\chi }}$-privacy was proposed in Fernandes et al. (2019), to hide the clues about the author's writing style while keeping the text content remains unchanged. The privacy protection level is defined at the bag of words. Yue et al. (2021), first divided different tokens of documents into high and low sensitivity ones, according to token frequency, then used token similarity to define the sensitivity and selection method of differential privacy. Lyu et al. (2020) proposed Optimized Multiple Encoding (OME) algorithm, an LDP protocol, to perturb text embeddings output by a pretrained model.

When it comes to the image data, Fan (2019) employed Single Value Decomposition (SVD) technique to extract a feature vector from an image, and then employ sampling methods on the vector to achieve privacy guarantees under $E_{d_{\chi }}$-privacy. Fan (2018) defined the “m-neighborhood” notion to employ a differential privacy mechanism which allows protecting sensitive information under at most m pixels. For audio data, Han et al. (2020) extracted x-vector that represents voiceprint and then computed the similarity between x-vector pairs. Finally, voiceprint was replaced and merged with speech content to get synthesised audio.

In addition to the above works, Rastogi and Nath (2010) proposed algorithm PASTE based on differential privacy to protect distributed time-series data. PASTE uses $FP{A}_k$ algorithm to perturb k discrete Fourier transform (DFT) coefficients that can be used to reconstruct n query answers. To improve $FP{A}_k$, Acs et al. (2012) proposed Enhanced Fourier Perturbation (EFPA) under the histogram scenario to select k with a certain probability. Bao et al. (2021) observed the strong autocorrelations of streaming data consisting of regular updates and proposed Correlated Gaussian Mechanism (CGM) to realise local differential privacy. Yang et al. (2021) provided a privacy guarantee for tensor-valued queries by Tensor Variate Gaussian (TVG). The added noise gained from a tensor-valued random variable in a standard normal distribution is conducted with an n-mode matrix product with n different matrices.

The part of our work on tensor LDP perturbation is different from all these works. The data we should perturb is tensor, but not time-series. It is generated by multi-dimensional DFT, but not one-dimensional. The tensor entries are complex, but not just real. These existing techniques can not be directly applied to such kinds of data. In order to compare these works with our work clearly, we also summarise them in Table 1.

Table 1. Summary of related differential privacy work.

Work	Modality	Methods and pros	Cons
SVD-Privacy (Fan, 2019)	image	Extract a feature vector from an image by SVD, and then sample the vector to achieve better data utility under $E_{d_{\chi }}$-privacy.	Unimodal (image modality only).
Pixel DP (Fan, n.d.)		Employ differential privacy mechanism which allows protecting sensitive information under at most m pixels (“m-neighborhood” notion).
$E_{d_{\chi }}$-privacy (Fernandes et al., 2019)	text	Hide the clues about the author's writing style by $E_{d_{\chi }}$-privacy with higher utility where protection level is defined at the bag of words.	Unimodal (text modality only).
UMLDP (Yue et al., 2021)		Divide sensitive and nonsensitive tokens in document. Employ DP by similarity between tokens.
OME (Lyu et al., 2020)		Perturb text embeddings output by pretrained model. A randomisation factor dynamically adjust the flip probability of 0 and 1 to make better data utility.	Unimodal (text modality only) and indirect process on encoded data rather than raw data.
Voice Indistinguishability (Han et al., 2020)	audio	Measure the similarity of x-vector pairs that represent voiceprint and then replace the voiceprint with DP mechanism to gain a better synthesised audio.	Unimodal (audio modality only).
PASTE (Rastogi & Nath, 2010)	tabular data	Use $FP{A}_k$ algorithm to perturb k discrete Fourier transform coefficients that can be used to reconstruct n query answers.	Unimodal (structured data only).
EFPA (Acs et al., 2012)		Select k with a certain probability under the histograms scenario to impove the accuracy.
CGM (Bao et al., 2021)		Correlate the noise between two query functions of streaming data consisting of regular updates and achieve better data utility.
TVG (Yang et al., 2021)		Add noise gained from tensor-valued random variable in standard normal distribution which is conducted with n-mode matrix product with n different matrices. The DP approach improves the data utility.
Our work: multimodal DP framework	image, text, audio, etc	Fuse sub-representations of different modalities from sub-encoders to gain a fusion representation by a three- module framework and propose two approaches based on the framework for vector-valued representations and tensor-valued representations.	Indirect process on encoded data rather than raw data.

3. Preliminaries

3.1. Multimodal representation fusion

Different representations of information will affect the ways of processing the data. A good representation will make the subsequent processing of information more convenient. However, how to remove redundant information and represent the input data without losing much accuracy is the main problem. Deep learning provides us with a new idea, which allows us to map the input data to a low-dimensional representation. Because the low-dimensional representation will be restored to the original data label through the subsequent network of neural networks, as long as the recovery is accurate enough, the low-dimensional vector can be used as the representation of the input data. Furthermore, the connection ways of neural network can combine input features to extract high-order abstract features. The convergence of network parameters ensures that the representation is sufficiently representative.

Zadeh et al. (2016) were the first to use a deep learning network to get fusion representations by fully connecting sub-representations from sub-networks. The disadvantage of this model is that the complementary information among different modalities is difficult to be learned by the model. In response to this problem, Zadeh et al. (2017) modelled sub-representations as different dimensions of n-fold Cartesian space. Therefore, the fusion process between different modalities can be carried out by tensor outer product. However, this also brings the curse of dimensionality. Liu et al. (2018) extended previous work by applying the low-rank decomposition to gradient weight. By simplifying the mathematical formula, the representation vector can be gained directly instead of calculating the outer product and thereby reducing the complexity.

3.2. Differential privacy

If two databases differ in only one record, we call them adjacent databases and record them as D and $D^{\mathrm{\prime }}$ respectively. Given a protection mechanism M which takes database D as input and output $M\left(D\right)$, we say M satisfies ϵ-differential privacy (Dwork & Roth, 2014), if for any pair of $(D,D^{\prime })$, and all possible subsets $S\subseteq Range\left(M\right)$, (1) $\begin{array}{r}Pr\left[M\right(D)\in S]\le e^{ϵ}Pr\left[M\right(D^{\mathrm{\prime }})\in S]\end{array}$(1) We call ϵ the privacy budget which means the protection level of data. A lower budget implies better protection of privacy, but usually lower data utility. Based on the definition of DP, there are two common ways to employ the DP mechanism for data analysis. One is Centralized Differential Privacy (CDP) where the input data is a database. It implies a third party collects the raw data from each individual and is responsible for answering any query without disclosing the existence of any individual's data. According to Dwork and Roth (2014), it perplexes the views of attacker on whether individuals data is or not part of the database, thus protecting them from any harm that they may face due to their data being in the database.

Obviously, CDP needs a trusted third party, otherwise, privacy will be leaked. However, in most real settings, it is difficult to meet the demand. Therefore, the other approach, Local Differential Privacy (LDP), taking a record as input has also been widely adopted.

3.2.1. Local differential privacy

In the local setting, each user corresponds to one record. Algorithm M inputs a record and outputs a privacy-protected record. Intuitively, according to an output result of privacy algorithm M, it is almost impossible to infer which record its input data is. As in Equation (2(2) $\begin{array}{r}Pr\left[M\right(x_1)=y]\le e^{ϵ}Pr\left[M\right(x_2)=y]\end{array}$(2) ), suppose $x_1$ is the true data, $x_2$ is fake data from some domain, it will be difficult for an attacker to infer the output of M is generated from $x_1$ or $x_2$. Since the algorithm M can be implemented by the local user himself, the constraint of a trusted third party as in CDP no longer exists. (2) $\begin{array}{r}Pr\left[M\right(x_1)=y]\le e^{ϵ}Pr\left[M\right(x_2)=y]\end{array}$(2) According to Xiong et al. (2020), LDP possesses sequential composition, parallel composition, post-processing properties.

3.2.2. $(\alpha ,\text{β})-DP$

$(\alpha ,\text{β})-DP$ mechanism (Hall et al., 2013) is an approximation of differential privacy when the randomised algorithm in dataset D outputs a vector $v_D$. The output is: (3) $\begin{array}{rl}& \widetilde{v_D}=v_D+\frac{c\left(\text{β}\right)\mathrm{\Delta }}{\alpha }Z,Z\sim {\mathcal{N}}_d(0,M),\end{array}$(3) (3a) $\begin{array}{rl}& \mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}\underset{D\sim D^{\prime }}{sup}\Vert M^{-1/2}(v_D-v_{D^{\prime }}){\Vert }_2\le \mathrm{\Delta },\end{array}$(3a) (3b) $\begin{array}{rl}& c\left(\text{β}\right)\ge \sqrt{2log\frac{2}{\text{β}}},\end{array}$(3b) in which $D^{\prime }$ is the adjacent database and positive definite symmetric matrix $M\in {\mathbb{R}}^{d\times d}$. The case when $M=I_d$ is equivalent to Euclidean distance (Dwork et al., 2006) where $I_d\in {\mathbb{R}}^{d\times d}$ is the identity matrix.

$(\alpha ,\text{β})-DP$ is actually an alias of $(ϵ,\delta )-DP$, which was originally proposed for CDP. However, according to Dwork and Roth (2014), an ϵ-CDP algorithm that takes as input a database of size n = 1 can be an ϵ-local algorithm for the local DP. If the local user has a database with a size larger than 1, the parallel composition can be utilised. It was pointed out that $(ϵ,\delta )-DP$ satisfies parallel and sequential compositions, as well as the post-processing property (Near & Abuah, 2021). In this paper, we will implement $(\alpha ,\text{β})-DP$ of Hall et al. (2013), in the LDP model.

Parallel Composition: Specifically, when mechanism $M_1$ satisfying $({ϵ}_1,{\delta }_1)-DP$ and $M_2$ satisfying $({ϵ}_2,{\delta }_2)-DP$ are applied respectively on two disjoint databases $D_1$ and $D_2$, then $D_1\cup D_2$ satisfies $({ϵ}^{\prime },{\delta }^{\prime })-DP$ in which ${ϵ}^{\prime }=max({ϵ}_1,{ϵ}_2)$, ${\delta }^{\prime }=max({\delta }_1,{\delta }_2)$.

Sequential Composition: When mechanism $M_1$ satisfying $({ϵ}_1,{\delta }_1)-DP$ and $M_2$ satisfying $({ϵ}_2,{\delta }_2)-DP$ are applied sequentially on the same database D, then D satisfies $({ϵ}_1+{ϵ}_2,{\delta }_1+{\delta }_2)-DP$.

Post-processing: When M satisfying $(ϵ,\delta )-DP$ is applied to database D, the input of function G is output of M, then G also satisfies $(ϵ,\delta )-DP$.

3.3. Binary encoding methods

Based on randomised response algorithms, we can protect privacy by flipping bits in binary vectors. How to choose the flip probability under the guarantee of LDP is the main problem. When it comes to a real vector instead of a binary vector, we can transmit the vector to a binary matrix. The following sections describe different LDP methods based on binary encoding.

3.3.1. Unary encoding (UE)

For a binary vector $\overrightarrow{z}=(z_1,z_2,\dots ,z_n),z_i\in \{0,1\}$, UE encodes $\overrightarrow{z}$ to $\overrightarrow{z^{\prime }}$ by the rules: (4) $\begin{array}{r}Pr(z_i^{\prime }=1)=\{\begin{array}{ll}p,& \mathrm{i}\mathrm{f}{z}_i=1\\ q,& \mathrm{i}\mathrm{f}{z}_i=0\end{array}\end{array}$(4) The unary encoding satisfies ϵ-LDP for (5) $\begin{array}{r}ϵ=\ln \frac{p(1-q)}{(1-p)q}\end{array}$(5) Among different choices for p and q, there are Symmetric Unary Encoding, Optimized Unary Encoding (Wang et al., 2017) and Optimized Multiple Encoding algorithms (Lyu et al., 2020).

3.3.2. Symmetric unary encoding (SUE)

RAPPOR's implementation (Erlingsson et al., 2014) chooses p = 0.75 and q = 0.25 such that p + q = 1, making the treatment of 1 and 0 symmetric. (6) $\begin{array}{r}p=\frac{e^{ϵ/2}}{e^{ϵ/2}+1},q=\frac{1}{e^{ϵ/2}+1}\end{array}$(6) Changing one bit of the vector has the greatest impact on 2 bits difference between the two vectors, so the sensitivity is 2 here. In Equation (6(6) $\begin{array}{r}p=\frac{e^{ϵ/2}}{e^{ϵ/2}+1},q=\frac{1}{e^{ϵ/2}+1}\end{array}$(6) ), $ϵ/2$ can be replaced by $ϵ/\mathrm{\Delta }f$.

3.3.3. Optimized unary encoding (OUE)

When the relative number of either 0 or 1 in the binary vector is obviously large, we can use the ratio of unchanged bits to flipped bits to allocate different privacy budgets respectively. For example, when there are more 1s than 0s, we can allocate more privacy budgets to 1 to maintain utility: (7) $\begin{array}{r}\frac{p}{1-p}=e^{{ϵ}_1},\frac{1-q}{q}=e^{{ϵ}_2},\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}ϵ={ϵ}_1+{ϵ}_2\end{array}$(7)

3.3.4. Optimized multiple encoding (OME)

Suppose the length of fusion representation is r, then for OUE and SUE, the sensitivity $\mathrm{\Delta }f$ are both $2r$ because they are binary vectors. When the representation value is real, we can extract the vector into a binary matrix (Lyu et al., 2020), that is, for $\overrightarrow{x}=\{x_1,x_2,\dots ,x_r\}$, each $x_i$ can be transformed into a length-l binary vector $\overrightarrow{x_{b_i}}$ to obtain a binary matrix $X=\{\overrightarrow{x_{b_1}},\overrightarrow{x_{b_2}},\dots ,\overrightarrow{x_{b_r}}\}$. In this case, the sensitivity $\mathrm{\Delta }f$ is rl. The choices of p and q are in Equations (8(8) $\begin{array}{rl}& p=Pr\{1\to 1\}=\{\begin{array}{ll}{\displaystyle \frac{\lambda }{1+\lambda }},& \mathrm{f}\mathrm{o}\mathrm{r}i\in 2n\\ {\displaystyle \frac{1}{1+{\lambda }^3}},& \mathrm{f}\mathrm{o}\mathrm{r}i\in 2n+1\end{array}\end{array}$(8) ) and (9(9) $\begin{array}{rl}& q=Pr\{0\to 1\}=\frac{1}{1+\lambda e^{\frac{ϵ}{rl}}}\end{array}$(9) ). (8) $\begin{array}{rl}& p=Pr\{1\to 1\}=\{\begin{array}{ll}{\displaystyle \frac{\lambda }{1+\lambda }},& \mathrm{f}\mathrm{o}\mathrm{r}i\in 2n\\ {\displaystyle \frac{1}{1+{\lambda }^3}},& \mathrm{f}\mathrm{o}\mathrm{r}i\in 2n+1\end{array}\end{array}$(8) (9) $\begin{array}{rl}& q=Pr\{0\to 1\}=\frac{1}{1+\lambda e^{\frac{ϵ}{rl}}}\end{array}$(9) According to Lyu et al. (2020), λ here is a randomisation factor, which is used to balance the utility and privacy. When there are more 0s than 1s in the data, increasing λ will make q smaller, then one bit “0” will not be flipped to “1” with higher probability. If there are more 1s than 0s, p can also be calibrated to an increase by adjusting λ, so that “1” will not also be flipped with higher probability.

3.4. Fourier transform

A signal can be represented in terms of the linear combination of orthogonal functions. It is known as the trigonometric Fourier series when these orthogonal functions are trigonometric functions. As the limit form of Fourier series, the Fourier transform (or FT) decomposes functions from time domain to frequency domain in the form of linear transformation. The Fourier transform of a function $f\left(\overrightarrow{r}\right)$ in n-dimensional space is $\mathfrak{F}f\left(\overrightarrow{\omega }\right)$, defined as follows (Equation (10(10) $\begin{array}{rl}\mathfrak{F}f\left(\overrightarrow{\omega }\right)& ={\int }_{R^n}f\left(\overrightarrow{r}\right)e^{-2\pi j\overrightarrow{\omega }\cdot \overrightarrow{r}}\mathrm{d}\overrightarrow{r}\end{array}$(10) )): (10) $\begin{array}{rl}\mathfrak{F}f\left(\overrightarrow{\omega }\right)& ={\int }_{R^n}f\left(\overrightarrow{r}\right)e^{-2\pi j\overrightarrow{\omega }\cdot \overrightarrow{r}}\mathrm{d}\overrightarrow{r}\end{array}$(10) (11) $\begin{array}{rl}{\mathfrak{F}}^{-1}g\left(\overrightarrow{\omega }\right)& ={\int }_{R^n}g\left(\overrightarrow{r}\right)e^{2\pi j\overrightarrow{\omega }\cdot \overrightarrow{r}}\mathrm{d}\overrightarrow{r}\end{array}$(11) where $\overrightarrow{r}=(r_1,r_2,\dots ,r_n),\overrightarrow{\omega }=({\omega }_1,{\omega }_2,\dots ,{\omega }_n)\in {\mathbb{R}}^n$. Equation (11(11) $\begin{array}{rl}{\mathfrak{F}}^{-1}g\left(\overrightarrow{\omega }\right)& ={\int }_{R^n}g\left(\overrightarrow{r}\right)e^{2\pi j\overrightarrow{\omega }\cdot \overrightarrow{r}}\mathrm{d}\overrightarrow{r}\end{array}$(11) ) is the inverse Fourier transform (or IFT) of a function $g\left(\overrightarrow{r}\right)$. Note that the n-fold multiple integral goes from $-\mathrm{\infty }$ to ∞ and the output shape of FT and IFT is the same as input.

3.4.1. Multi-dimensional discrete Fourier transform

Since computers can only process discrete signals, Discrete Fourier transform (DFT) can be employed to decompose discrete signals obtained by interval sampling. For multi-dimensional discrete signals which can be represented by tensors, multi-dimensional DFT can be conducted (Diaz & Lutoborski, 2017). As in Equation (12(12) $\begin{array}{r}F\left(\overrightarrow{\omega }\right)=\sum _{r_1=0}^{N_1-1}\sum _{r_2=0}^{N_2-1}\dots \sum _{r_n=0}^{N_n-1}f\left(\overrightarrow{r}\right)e(\overrightarrow{w},\overrightarrow{r})\end{array}$(12) ), f is a tensor of order n, and its i-th index has $N_i$ dimensions. Note that $\overrightarrow{r}=(r_1,r_2,\dots ,r_n)$, $\overrightarrow{w}=(w_1,w_2,\dots ,w_n)$. (12) $\begin{array}{r}F\left(\overrightarrow{\omega }\right)=\sum _{r_1=0}^{N_1-1}\sum _{r_2=0}^{N_2-1}\dots \sum _{r_n=0}^{N_n-1}f\left(\overrightarrow{r}\right)e(\overrightarrow{w},\overrightarrow{r})\end{array}$(12) Let $\overrightarrow{r/N}=\{r_1/N_1,r_2/N_2,\dots ,r_n/N_n\}$, then $e(\overrightarrow{w},\overrightarrow{r})$ is: (13) $\begin{array}{r}e(\overrightarrow{w},\overrightarrow{r})=\prod _{i=1}^n{e}^{\frac{-j2\pi r_i{w}_i}{N_i}}=e^{-j2\pi \overrightarrow{w}\cdot \overrightarrow{r/N}}\end{array}$(13) The time consumption of directly calculating DFT of N-point signals is $O\left(N^2\right)$, while the Fast Fourier transform (or FFT) proposed by Cooley and Tukey (1965) utilises the symmetry of root of unity to break the DFT into smaller DFTs, reducing the computation time to $O(N\log N)$. In our work, we use FFT to calculate DFT on discrete signals.

4. Multimodal fusion representation learning framework

Our framework consists of three modules including Encoding, Perturbation and Aggregation, which are explained in detail in the following.

4.1. Encoding module

Encoding module takes the raw multimodal data as input and outputs fusion representations. Firstly, sub-encoders encode raw multimodal data into sub-representations. There are many effective pretrained models that can output high-quality representations, and we can choose the one that well suits the data type. Models can also be compressed by techniques like knowledge distillation, to speed up the processing of data. Therefore the computation cost of sub-encoders will not be a great burden for local users.

Secondly, sub-representations are fused to achieve a fusion representation. What should be considered here is how to reserve the intra-modality information in the fusion for purpose of further learning. In addition to directly concatenating sub-representations, the fusion representation can be gained by outer product (Zadeh et al., 2017), as Equation (14(14) $\begin{array}{r}\mathbb{Z}=\underset{m=1}{\overset{M}{⨂}}{\mathbf{Z}}_m,{\mathbf{Z}}_m\in {\mathbb{R}}^{d_m}\end{array}$(14) ) shows. Here ${\mathbf{Z}}_m$ is a sub-representation of the m-th modality, $d_m$ is the representation size of each modality, and $⨂$ is the outer product for tensors. (14) $\begin{array}{r}\mathbb{Z}=\underset{m=1}{\overset{M}{⨂}}{\mathbf{Z}}_m,{\mathbf{Z}}_m\in {\mathbb{R}}^{d_m}\end{array}$(14)

4.2. Perturbation module

The Perturbation module will apply privacy-preserving methods like the LDP mechanism given some privacy budget to convert fusion representations to protected representations. The shape of converted representation may be different from the original, depending on how the third party specifies the data usage. An example here is our proposed Binary Encoding-based approach (Figure 2), where the representation is a 1-D tensor and the output is a binary matrix. The other case when representations share the same shape will also bring some convenience. For example, some methods can be defined to mix the original data with the protected data after perturbation to make a better utility-protection balance. We use the same shape for these representations in our Fourier transform based LDP approach.

Figure 1. The encoding module based on low-rank multimodal fusion.

Figure 2. The perturbation and aggregation modules based on binary encoding.

4.3. Aggregation module

In the Aggregation module, the untrusted third party can use its well-designed network to analyse the representation data obtained by original data and train the model. The fusion representations are abstract, which do not necessarily have specific semantics. However, supervised learning or unsupervised learning can still be carried out from these fusion representations. This is as similar as we can judge whether an object is an aircraft because we have abstract impressions of aircraft, even if we cannot draw the appearance and structure of the aircraft concretely. This also provides an advantage to resist the third party to infer the privacy information of a specific sample. Besides, the network structure can be flexible to adapt to different privacy budgets for specific problem requirements.

5. LDP approaches for fusion representation

5.1. Binary encoding-based LDP for fusion vector

As shown in Figure 1, we take multimodal datasets as input of the Encoding module to get one-dimensional fusion representation by low-rank decomposition. In this figure, a typical dataset consisting of video ($X_v$), audio ($X_a$) and textual ($X_l$) data is input into encoders, and different encoders can be used for the corresponding modalities to gain sub-representations. Then they are fused implicitly to a cube tensor by decomposing the fusion tensor weights into CP decomposition forms. This avoids the exponential increase of computation cost in direct calculations with sub-representations (Liu et al., 2018).

We utilise a network architecture with weight $\mathbb{W}$ to transform a tensor $\mathbb{Z}$ into a vector $z_{fusion}$, i.e. $z_{fusion}=\mathbb{W}\cdot \mathbb{Z}$.

The CP decomposition (Hitchcock, 1928) decomposes an Nth-order tensor into a linear combination of rank-1 tensors, as illustrated in Equation (15(15) $\begin{array}{rl}\mathbf{B}& \cong \sum _{r=1}^R{\lambda }_r{{\mathbf{b}}_{\mathbf{r}}}^{\left(1\right)}\circ {{\mathbf{b}}_{\mathbf{r}}}^{\left(2\right)}\circ \cdots {{\mathbf{b}}_{\mathbf{r}}}^{\left(N\right)},\mathbf{B}\in {\mathbb{R}}^{I_1\times I_2\times \cdots \times I_N}\end{array}$(15) ). $\mathbf{B}$ is the N-th order tensor of size $I_1\times I_2\times \cdots \times I_N$. ${{\mathbf{b}}_{\mathbf{1}}}^{\left(1\right)}\circ {{\mathbf{b}}_{\mathbf{2}}}^{\left(2\right)}\circ \cdots {{\mathbf{b}}_{\mathbf{r}}}^{\left(N\right)}$ is outer product among vectors ${{\mathbf{b}}_{\mathbf{r}}}^{\left(i\right)}$ from dimension i until rank R. We apply CP decomposition to the network weight $\mathbb{W}$, and adopt the method proposed by Liu et al. (2018), to decompose $\mathbb{W}$ into matrix ${\mathbf{W}}_m^{\left(i\right)}$ made up of rank-1 vectors, i.e. $\mathbb{W}=\sum _{i=1}^r\underset{m=1}{\overset{M}{⨂}}{\mathbf{W}}_m^{\left(i\right)}$. (15) $\begin{array}{rl}\mathbf{B}& \cong \sum _{r=1}^R{\lambda }_r{{\mathbf{b}}_{\mathbf{r}}}^{\left(1\right)}\circ {{\mathbf{b}}_{\mathbf{r}}}^{\left(2\right)}\circ \cdots {{\mathbf{b}}_{\mathbf{r}}}^{\left(N\right)},\mathbf{B}\in {\mathbb{R}}^{I_1\times I_2\times \cdots \times I_N}\end{array}$(15) (16) $\begin{array}{rl}{z}_{fusion}& =\left(\sum _{i=1}^R\underset{m=1}{\overset{M}{⨂}}{\mathbf{W}}_m^{\left(i\right)}\right)\cdot \mathbb{Z}=\underset{m=1}{\overset{M}{\bigwedge }}[\sum _{i=1}^R{\mathbf{W}}_m^{\left(i\right)}\cdot {\mathbf{Z}}_m]\end{array}$(16) According to Zadeh et al. (2017), the fusion tensor $\mathbb{Z}$ created by outer product can be decomposed together with the weight in the network. As Equation (16(16) $\begin{array}{rl}{z}_{fusion}& =\left(\sum _{i=1}^R\underset{m=1}{\overset{M}{⨂}}{\mathbf{W}}_m^{\left(i\right)}\right)\cdot \mathbb{Z}=\underset{m=1}{\overset{M}{\bigwedge }}[\sum _{i=1}^R{\mathbf{W}}_m^{\left(i\right)}\cdot {\mathbf{Z}}_m]\end{array}$(16) ) shows, $⨂$ is outer product between matrices, $\bigwedge$ means the element-wise product, · is the dot product with the effect of tensor contractions, and $z_{fusion}$ is a vector obtained. Thus we can gain the fusion representation $z_{fusion}$ without explicit calculation of the tensors. This low-rank trick remains high efficiency and performance. Concretely, this derivation reduces the computation complexity from $\mathit{O}\mathit{\left(}{\mathit{d}}_{\mathit{y}}\prod _{\mathit{m}\mathit{=}\mathit{1}}^{\mathit{M}}{\mathit{d}}_{\mathit{m}}\mathit{\right)}$ to $\mathit{O}\mathit{(}{\mathit{d}}_{\mathit{y}}\cdot \mathit{R}\cdot \sum _{\mathit{m}\mathit{=}\mathit{1}}^{\mathit{M}}{\mathit{d}}_{\mathit{m}}\mathit{)}$, where $d_y$ is the size of vector $z_{fusion}$.

After gaining the fusion representation $z_{fusion}$ in the form of vector, we apply OME on $z_{fusion}$ since it scales well even if the domain size is large. We firstly extract an l-length binary vector on each element in $z_{fusion}$, and transform $z_{fusion}$ into a binary matrix. Then each bit in the matrix is perturbed according to Equations (8(8) $\begin{array}{rl}& p=Pr\{1\to 1\}=\{\begin{array}{ll}{\displaystyle \frac{\lambda }{1+\lambda }},& \mathrm{f}\mathrm{o}\mathrm{r}i\in 2n\\ {\displaystyle \frac{1}{1+{\lambda }^3}},& \mathrm{f}\mathrm{o}\mathrm{r}i\in 2n+1\end{array}\end{array}$(8) ) and (9(9) $\begin{array}{rl}& q=Pr\{0\to 1\}=\frac{1}{1+\lambda e^{\frac{ϵ}{rl}}}\end{array}$(9) ). We follow the intuition that 0s and 1s happen at different ratios and each ratio owns its privacy budget, as Equation (7(7) $\begin{array}{r}\frac{p}{1-p}=e^{{ϵ}_1},\frac{1-q}{q}=e^{{ϵ}_2},\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}ϵ={ϵ}_1+{ϵ}_2\end{array}$(7) ) shows. Further, the odds of 0 and 1 bits flipping should satisfy Equations (8(8) $\begin{array}{rl}& p=Pr\{1\to 1\}=\{\begin{array}{ll}{\displaystyle \frac{\lambda }{1+\lambda }},& \mathrm{f}\mathrm{o}\mathrm{r}i\in 2n\\ {\displaystyle \frac{1}{1+{\lambda }^3}},& \mathrm{f}\mathrm{o}\mathrm{r}i\in 2n+1\end{array}\end{array}$(8) ) and (9(9) $\begin{array}{rl}& q=Pr\{0\to 1\}=\frac{1}{1+\lambda e^{\frac{ϵ}{rl}}}\end{array}$(9) ). When given fixed representation size and vector extracting size, if randomisation factor λ is constant, lower ϵ values and higher rl will result in higher q value.

If the local user has more than one $z_{fusion}$, the above perturbation can be implemented on each of them. Suppose the same privacy budget is used for each perturbation, by parallel composition, the union of their outputs satisfies ϵ-LDP.

Finally, the Aggregation module trains the perturbed binary matrices. By the post-processing property, the output of aggregation also satisfies ϵ-LDP. The diagram of perturbation and aggregation is shown in Figure 2. The whole process is shown as Algorithm 1.

5.2. Fourier transform based LDP for fusion tensor

Since the low-rank decomposition will inevitably lead to the loss of accuracy, as an upstream task, sometimes we may want to trade a slight increase in the time cost for more accurate and more informative high-dimensional representations. Although the rise in accuracy will lead to the growth of exponential complexity, considering the real situations where a limited number of modalities are encountered, we can still use the high dimensional but accurate representations. The outer product fusion schemes, such as TFN (as Figure 3 shows), were proposed for these scenarios. Therefore, we further explore how to carry out the differential privacy method when the Encoding module outputs tensor representation.

Figure 3. The encoding module based on tensor fusion.

For a reason that representation may have abstract semantics in higher dimensional semantic space, it can be regarded as a signal. Based on this signal, we exploit the semantic information about time in different dimensions.

Instead of directly adding DP noise to data, we will add noise to the DFT tensor of the data. The multi-dimensional DFT is employed on fusion tensor ${\mathbb{Z}}_{fusion}$ to get the DFT tensor. Both the DFT tensor and ${\mathbb{Z}}_{fusion}$ are in the order of M. In the m-th dimension of the DFT tensor, $k_m$ points are kept, and we choose $k_m<<d_m$ to alleviate the problem of exponential growth. Let $\overrightarrow{k}=(k_1,k_2,\dots ,k_M)$, we denote the DFT tensor of ${\mathbb{Z}}_{fusion}$ by ${\mathcal{F}}^{\overrightarrow{k}}\left({\mathbb{Z}}_{fusion}\right)$. The whole process of Perturbation module is illustrated in Figure 4.

Figure 4. Fourier transform based LDP perturbation module.

It is worth mentioning that in ${\mathcal{F}}^{\overrightarrow{k}}\left({\mathbb{Z}}_{fusion}\right)$ each element is a complex number. To perturb the Fourier transform coefficients means to perturb each complex number. Instead of perturbing only the real part of a complex number as used in the exiting works, we treat each complex number as a two-dimensional vector composed of its real and imaginary parts. We use the approximate differential privacy for vector-valued output, i.e. $(\alpha ,\text{β})-DP$ introduced in Section 3.2.2, to perturb each complex number.

We define two fusion tensors which differ in only one entry as adjacent tensors, and denote them by ${\mathbb{Z}}_{fusion}$ and ${\mathbb{Z}}_{fusion}^{\prime }$. Let $\hat{f}\left(\overrightarrow{s}\right)$ and $\widehat{f^{\prime }}\left(\overrightarrow{s}\right)$ be entry at position $\overrightarrow{s}$ of the DFT tensor ${\mathcal{F}}^{\overrightarrow{k}}\left({\mathbb{Z}}_{fusion}\right)$ and ${\mathcal{F}}^{\overrightarrow{k}}\left({\mathbb{Z}}_{fusion}^{\prime }\right)$, respectively, in which $\overrightarrow{s}=(s_1,s_2,\dots ,s_M)$, and each $s_i\in \{1,\dots ,k_i\}$ for $i=1,\dots ,M$. We treat both $\hat{f}\left(\overrightarrow{s}\right)$ and $\widehat{f^{\prime }}\left(\overrightarrow{s}\right)$ as two-dimensional vectors. In Equation (17(17) $\begin{array}{rl}\mathrm{\Delta }& =\underset{{\mathbb{Z}}_{fusion}\sim {\mathbb{Z}}_{fusion}^{\prime }}{sup}\Vert M^{-1/2}\left(\hat{f}\right(\overrightarrow{s})-\widehat{f^{\prime }}(\overrightarrow{s}\left)\right){\Vert }_2\\ & =\underset{{\mathbb{Z}}_{fusion}\sim {\mathbb{Z}}_{fusion}^{\prime }}{sup}\Vert I_2\left(\hat{f}\right(\overrightarrow{s})-\widehat{f^{\prime }}(\overrightarrow{s}\left)\right){\Vert }_2\\ & =\underset{{\mathbb{Z}}_{fusion}\sim {\mathbb{Z}}_{fusion}^{\prime }}{sup}\Vert \hat{f}\left(\overrightarrow{s}\right)-\widehat{f^{\prime }}\left(\overrightarrow{s}\right){\Vert }_2\end{array}$(17) ), we choose $M=I_2$, the identity matrix of $2\times 2$. Among all pairs of vectors $\hat{f}\left(\overrightarrow{s}\right)$ and $\widehat{f^{\prime }}\left(\overrightarrow{s}\right)$, we calculate the supremum of $\Vert \hat{f}\left(\overrightarrow{s}\right)-\widehat{f^{\prime }}\left(\overrightarrow{s}\right){\Vert }_2$ as Δ. However, a brute-force searching could be too time-consuming. Below we give a method to efficiently compute Δ. (17) $\begin{array}{rl}\mathrm{\Delta }& =\underset{{\mathbb{Z}}_{fusion}\sim {\mathbb{Z}}_{fusion}^{\prime }}{sup}\Vert M^{-1/2}\left(\hat{f}\right(\overrightarrow{s})-\widehat{f^{\prime }}(\overrightarrow{s}\left)\right){\Vert }_2\\ & =\underset{{\mathbb{Z}}_{fusion}\sim {\mathbb{Z}}_{fusion}^{\prime }}{sup}\Vert I_2\left(\hat{f}\right(\overrightarrow{s})-\widehat{f^{\prime }}(\overrightarrow{s}\left)\right){\Vert }_2\\ & =\underset{{\mathbb{Z}}_{fusion}\sim {\mathbb{Z}}_{fusion}^{\prime }}{sup}\Vert \hat{f}\left(\overrightarrow{s}\right)-\widehat{f^{\prime }}\left(\overrightarrow{s}\right){\Vert }_2\end{array}$(17) An efficient method to compute Δ: Since we are using LDP for perturbation, we can only assume the local range of tensor ${\mathbb{Z}}_{fusion}$ is known. Let $\mathrm{\Delta }z\left(\overrightarrow{r}\right)$ be the local sensitivity of $z\left(\overrightarrow{r}\right)$, $\overrightarrow{r}=(r_1,r_2,\dots ,r_M)$, i.e. the entry at position $\overrightarrow{r}$ of tensor ${\mathbb{Z}}_{fusion}$ (each $r_i\in \{1,\dots ,d_i\}$ for $i=1,\dots ,M$). That is, $\mathrm{\Delta }z\left(\overrightarrow{r}\right)=\underset{a,b}{max}|a-b|$, a and b belong to the local range of $z\left(\overrightarrow{r}\right)$. By the following theorem, Δ in Equation (17(17) $\begin{array}{rl}\mathrm{\Delta }& =\underset{{\mathbb{Z}}_{fusion}\sim {\mathbb{Z}}_{fusion}^{\prime }}{sup}\Vert M^{-1/2}\left(\hat{f}\right(\overrightarrow{s})-\widehat{f^{\prime }}(\overrightarrow{s}\left)\right){\Vert }_2\\ & =\underset{{\mathbb{Z}}_{fusion}\sim {\mathbb{Z}}_{fusion}^{\prime }}{sup}\Vert I_2\left(\hat{f}\right(\overrightarrow{s})-\widehat{f^{\prime }}(\overrightarrow{s}\left)\right){\Vert }_2\\ & =\underset{{\mathbb{Z}}_{fusion}\sim {\mathbb{Z}}_{fusion}^{\prime }}{sup}\Vert \hat{f}\left(\overrightarrow{s}\right)-\widehat{f^{\prime }}\left(\overrightarrow{s}\right){\Vert }_2\end{array}$(17) ) is actually the maximum of all local sensitivities at all entries of ${\mathbb{Z}}_{fusion}$.

Theorem 5.1

Let $\mathrm{\Delta }z\left(\overrightarrow{r}\right)$ be the local sensitivity of $z\left(\overrightarrow{r}\right)$, the entry at position $\overrightarrow{r}$ of tensor ${\mathbb{Z}}_{fusion}$, $\overrightarrow{r}=(r_1,r_2,\dots ,r_M)$, each $r_i\in \{1,\dots ,d_i\}$ for $i=1,\dots ,M$. Then $\mathrm{\Delta }=\underset{r_i,i}{max}\mathrm{\Delta }z\left(\overrightarrow{r}\right)$.

Proof.

Let $\overrightarrow{s}=(s_1,s_2,\dots ,s_M)$, in which each $s_i\in \{1,\dots ,k_i\}$ for $i=1,\dots ,M$. Let $d=max(d_1,d_2,\dots ,d_M)$. By Equation (12(12) $\begin{array}{r}F\left(\overrightarrow{\omega }\right)=\sum _{r_1=0}^{N_1-1}\sum _{r_2=0}^{N_2-1}\dots \sum _{r_n=0}^{N_n-1}f\left(\overrightarrow{r}\right)e(\overrightarrow{w},\overrightarrow{r})\end{array}$(12) ), the corresponding multiple DFT of ${\mathbb{Z}}_{fusion}$ is given by $\hat{f}\left(\overrightarrow{s}\right)=\sum _{r_1=0}^{d-1}\sum _{r_2=0}^{d-1}\dots \sum _{r_M=0}^{d-1}z\left(\overrightarrow{r}\right)e(\overrightarrow{s},\overrightarrow{r}),$in which $z\left(\overrightarrow{r}\right)=0$ when any $r_i>d_i$.

Suppose ${\mathbb{Z}}_{fusion}^{\prime }$ and ${\mathbb{Z}}_{fusion}$ differ in only one entry at position $\overrightarrow{\mathbf{r}}$. Let $z^{\prime }\left(\overrightarrow{r}\right)$ be the entry at position $\overrightarrow{r}$ of tensor ${\mathbb{Z}}_{fusion}^{\prime }$. Then $\begin{array}{rl}& {\hat{f}}^{\prime }\left(\overrightarrow{s}\right)=\sum _{r_1=0}^{d-1}\sum _{r_2=0}^{d-1}\dots \sum _{r_M=0}^{d-1}{z}^{\prime }\left(\overrightarrow{r}\right)e(\overrightarrow{s},\overrightarrow{r}).\\ & \hat{f}\left(\overrightarrow{s}\right)-{\hat{f}}^{\prime }\left(\overrightarrow{s}\right)=\left(z\right(\overrightarrow{\mathbf{r}})-z^{\prime }(\overrightarrow{\mathbf{r}}\left)\right)e(\overrightarrow{s},\overrightarrow{\mathbf{r}})\\ & =\left(z\right(\overrightarrow{\mathbf{r}})-z^{\prime }(\overrightarrow{\mathbf{r}}\left)\right)\cdot \exp (-\frac{j2\pi {\mathrm{\Sigma }}_{i=1}^M{s}_i{\mathbf{r}}_i}{d}).\end{array}$By Equation (17(17) $\begin{array}{rl}\mathrm{\Delta }& =\underset{{\mathbb{Z}}_{fusion}\sim {\mathbb{Z}}_{fusion}^{\prime }}{sup}\Vert M^{-1/2}\left(\hat{f}\right(\overrightarrow{s})-\widehat{f^{\prime }}(\overrightarrow{s}\left)\right){\Vert }_2\\ & =\underset{{\mathbb{Z}}_{fusion}\sim {\mathbb{Z}}_{fusion}^{\prime }}{sup}\Vert I_2\left(\hat{f}\right(\overrightarrow{s})-\widehat{f^{\prime }}(\overrightarrow{s}\left)\right){\Vert }_2\\ & =\underset{{\mathbb{Z}}_{fusion}\sim {\mathbb{Z}}_{fusion}^{\prime }}{sup}\Vert \hat{f}\left(\overrightarrow{s}\right)-\widehat{f^{\prime }}\left(\overrightarrow{s}\right){\Vert }_2\end{array}$(17) ), $\begin{array}{rl}\mathrm{\Delta }& =\underset{{\mathbb{Z}}_{fusion}^{\prime },{\mathbb{Z}}_{fusion}}{sup}\Vert \hat{f}\left(\overrightarrow{s}\right)-{\hat{f}}^{\prime }\left(\overrightarrow{s}\right){\Vert }_2\\ & =\underset{\overrightarrow{\mathbf{r}}}{max}\mathrm{\Delta }z\left(\overrightarrow{\mathbf{r}}\right),\end{array}$and this concludes the proof.

After Δ is computed, by $(\alpha ,\text{β})$-DP, $\hat{f}\left(\overrightarrow{s}\right)$ is perturbed as follows, $\tilde{f}\left(\overrightarrow{s}\right)=\hat{f}\left(\overrightarrow{s}\right)+\frac{c\left(\text{β}\right)\mathrm{\Delta }}{\alpha }Z,Z\sim {\mathcal{N}}_2(0,I_2).$Let ${\tilde{\mathcal{F}}}^{\overrightarrow{k}}\left({\mathbb{Z}}_{fusion}\right)$ be the perturbed DFT tensor. Then by the sequential composition of $(\alpha ,\text{β})$-DP, this perturbed tensor satisfies $(\prod _{i=1}^M{k}_i\alpha ,\prod _{i=1}^M{k}_i\text{β})$-DP.

If the user has more than one fusion tensor, the above mechanism can be applied independently to each fusion tensor. For simplification, we just assume on each tensor entry, mechanism of $(\alpha ,\text{β})$-DP is applied. Since these tensors are disjoint, by parallel composition, the union of these tensors can have a mechanism satisfying $(\prod _{i=1}^M{k}_i\alpha ,\prod _{i=1}^M{k}_i\text{β})$-DP.

Let ${\mathcal{F}}^{-1}\left({\tilde{\mathcal{F}}}^{\overrightarrow{k}}\right({\mathbb{Z}}_{fusion}\left)\right)$ represents the Inverse Discrete Fourier Transform (or IDFT) of ${\tilde{\mathcal{F}}}^{\overrightarrow{k}}\left({\mathbb{Z}}_{fusion}\right)$. By the post-processing property of $(\alpha ,\text{β})$-DP, the result of IDFT also satisfies $(\prod _{i=1}^M{k}_i\alpha ,\prod _{i=1}^M{k}_i\text{β})$-DP. The whole process is shown as Algorithm 2.

6. Experiments

6.1. Datasets

We evaluate our framework on three commonly used multimodal datasets (Table 2) including the Multimodal Corpus of Sentiment Intensity (CMU-MOSI) (Zadeh et al., 2016), the Persuasive Opinion Multimedia (POM) (Park et al., 2014) and Interactive Emotional Dyadic Motion Capture Database (IEMOCAP) (Busso et al., 2008).

Table 2. Overview of experimental datasets.

Dataset	Content	Modalities	Label
CMU_MOSI	2199 opinion-expressing short videos	audio, text, video	[−3,3] (strongly negative to strongly positive)
POM	1000 movie review videos (half of the videos got 5-star ratings and the rest of them got 1 or 2-star rating)	audio, text, video	16 sentimental labels with each value ranges in [−3,3]
IEMOCAP	10 actors performed relevant emotions according to scripts written before and flexible improvisation in two-person dialogue scenes	audio, text, video, facial expressions, hand movements	0–1 label of 10 themes (happiness, anger, sadness, frustration, neutral state, etc.)

The datasets are split into training, validation, and test sets following the proportion and settings by Liu et al. (2018). Considering the generalisation performance of the model, we train the data in the training set, find the parameters that make the validation set perform best during the training process, and finally evaluate the performance on the test set. The maximum epoch and patience are set to 1000 and 50 respectively. For the regression task of CMU-MOSI and POM dataset, we use L1 Loss. For the classification task of IEMOCAP, we use CrossEntropy Loss. For audio and video data, we use fully connected network as the encoder. For text data, we apply Long Short-Term Memory (LSTM) network as the encoder. The composition structure of these sub-networks is shown in Figure 1. It is worth mentioning that for training multimodal data, we need to align the data of different modalities firstly, and we use the word-level alignment here.

We first train model on specific network structures of Figures 1 and 3, and then export representation. DP approaches are conducted on these representations. Finally, we train a three-layer fully connected network as our Aggregation module which takes privacy-protected representations as input and evaluates different metrics on test set.

6.2. Binary encoding-based LDP

We evaluate Mean Absolute Error (MAE) and Multi-Acc (average accuracy of output classes) on POM, F1 score and accuracy on IEMOCAP, and MAE on CMU-MOSI.

We compare SUE, OUE and OME methods which have different combinations of p and q (Equations (6(6) $\begin{array}{r}p=\frac{e^{ϵ/2}}{e^{ϵ/2}+1},q=\frac{1}{e^{ϵ/2}+1}\end{array}$(6) ), (7(7) $\begin{array}{r}\frac{p}{1-p}=e^{{ϵ}_1},\frac{1-q}{q}=e^{{ϵ}_2},\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}ϵ={ϵ}_1+{ϵ}_2\end{array}$(7) ), (8(8) $\begin{array}{rl}& p=Pr\{1\to 1\}=\{\begin{array}{ll}{\displaystyle \frac{\lambda }{1+\lambda }},& \mathrm{f}\mathrm{o}\mathrm{r}i\in 2n\\ {\displaystyle \frac{1}{1+{\lambda }^3}},& \mathrm{f}\mathrm{o}\mathrm{r}i\in 2n+1\end{array}\end{array}$(8) ) and (9(9) $\begin{array}{rl}& q=Pr\{0\to 1\}=\frac{1}{1+\lambda e^{\frac{ϵ}{rl}}}\end{array}$(9) )). As illustrated in Figure 5, different choices of privacy budgets ϵ ranging from 0.5 to 10 show the influence on $F1$ score and accuracy. The last row of heat map shows the results of raw data without privacy protection.

Figure 5. Experimental results shown in heat map on IEMOCAP dataset. Figure (a) shows the F1 score result. The bottom row shows the results of raw data with no perturbation. Figure (b) shows the accuracy.

In our implementation, the Aggregation module is composed of three fully connected layers, which is simple but effective enough to evaluate the effectiveness of our framework and approaches. In real applications, a better fine-tuning network may be needed to solve problems such as overfitting.

In addition, we also employ the Laplace mechanism directly to raw multimodal data. The results of this experiment can reflect the difference between the method of employing DP on representation and employing DP on raw multimodal data directly.

6.3. Fourier transform based $(\alpha ,\text{β})-DP$

We evaluate our approach by three metrics consisting of Binary Accuracy, MAE and F1 score to provide diverse perspectives of the effect on data utility under the compositions of different α and β. The preprocessing and alignment of different modalities data, as well as the sub-networks, are the same as those experiments for Binary Encoding-based LDP. The main difference is the fusion method in the Encoding module. Here we use Tensor Fusion Network (Zadeh et al., 2017). We compare the results by setting different combinations of α and β. For each pair of α and β, we conduct experiments for 10 runs and take the arithmetic mean value. Note that $\alpha \le 1$ as per the requirement of Hall et al. (2013). In our experiments α ranges from 0.15 to 0.9, and β ranges from 0.001 to 2.

7. Results and discussion

As heat maps shown in Figure 5, both $F1$ score and accuracy increase when more privacy budgets ϵ are given. The last row is the result of raw data without privacy protection method. From the darkness of the map, we can see that the OME mechanism remains stable under different ϵ, the reason for which is mainly attributed to the large sensitivity of representation defined in OME. We can also see that the OME mechanism performs well even on a tight privacy budget. This shows the high utility of data under OME because the randomisation factor λ in OME dynamically adjusts the flip probability of 0 and 1.

Interestingly, we can see that on the happiness emotion, when a relatively small privacy budget is given, there are still fairly accurate results in all algorithms. In contrast, all algorithms perform with lower accuracy on the anger emotion. Further analysis of the cause will be explored in the future work.

Figure 7 shows the MAE loss of our Binary Encoding-based LDP approaches on the CMU-MOSI dataset. When given more privacy budgets ϵ, the MAE loss decreases on all three binary encoding methods. Among them, OME remains stable and highest utility. When given a small privacy budget, the score will shake because of the low utility of data. We can also find that all these methods perform better than the direct employment of Laplace DP mechanism on the raw multimodal data. Thus for multimodal data with high-dimensional features, processing the original data directly may not be the best choice for the reason that noises may also grow exponentially.

Figure 6. Experimental results on POM dataset given different privacy budgets ϵ ranging from 0.5 to 10. Figure (a) shows the results of average accuracy of output classes (Multi-Acc). Figure (b) shows the MAE Loss results.

Figure 7. MAE loss on dataset CMU-MOSI, where the horizontal axis denotes privacy budget ϵ. The line “lap” demonstrates the results of applying the Laplace DP mechanism to raw data.

In experiments of employing Binary Encoding-based LDP on POM dataset, we can find accuracy gets higher when more ϵ are given, as Figure 6(a) shows. Compared with the other two mechanisms, OME remains the highest accuracy as well as the lowest MAE loss. When the privacy budget approaches 0, there will be of great shake because the noise is large relatively and the data utility is extremely poor.

The result of experiments on implementing Fourier transform based LDP accord with the theoretical results. As Figure 8(b) shows, when α is fixed, the MAE will decrease with the increase of β. When given more privacy budgets, the data will gain better utility. When β remains unchanged, we can see that the MAE decreases with the increase of α. In the real scenarios, the user just needs to select acceptable α and β.

Figure 8. Experimental results on MOSI dataset carried out Fourier transform based LDP mechanism. (a) Binary Accuracy of test set on CMU-MOSI. (b) Mean Absolute Error (MAE) of test set on CMU-MOSI. (c) F1 score of test set on CMU-MOSI.

Complexity Analysis. Since the user does not need to train data locally, the complexity (user's perspective) consists of two parts, the first part is the Encoding module, and the second part is the Perturbation module. The Encoding module is determined by different encoding structures, which depend on the specific task. If it is the neural network like our experiments, the complexity is $max\left(len\right(l^i)\ast len(l^{i+1}\left)\right)$, where $len\left(l^i\right)$ means the number of nodes in the ith layer of neural network. The complexity of Perturbation module depends on the specific approach. For Binary Encoding-based approach, the fusion complexity is $O(d_y\times r\times \sum _{m=1}^M{d}_m)$ where r denoted the rank, $d_y$ denotes the output size and $d_m$ denotes the size of sub-representations of each modality. Perturbation complexity is $O\left(rl\right)$ where r and l are the dimension of representation and the expansion length of real numbers respectively. The complexity of Fourier transform based LDP approach is $O\left(d_y\prod _{m=1}^M{d}_m\right)$ where m is the number of modalities. Therefore, the total complexity is $O(max(len\left(l^i\right)\ast len\left(l^{i+1}\right))+d_y\times r\times \sum _{m=1}^M{d}_m+rl)$ for Binary Encoding-based LDP approach, and $O(max(len\left(l^i\right)\ast len\left(l^{i+1}\right))+d_y\prod _{m=1}^M{d}_m)$ for Fourier transform based LDP approach.

Discussions. Under the definition of local differential privacy, our approaches can provide local users with plausible deniability, which means from the outputs of perturbation mechanisms, it is impossible to infer which value the input is (Xiong et al., 2020). The approaches can resist attackers with arbitrary background knowledge (Dwork & Roth, 2014). They can also prevent possible privacy attacks by untrusted data collectors because the data is processed locally. Furthermore, it is difficult to reconstruct high-dimensional multimodal data by attackers given the low-dimensional representation, which provides an additional protection on privacy.

One limitation of our framework is that when high-dimensional raw multimodal data is encoded into representations, some original information may be lost inevitably. Since multimodal data is inherently high-dimensional and extremely sparse, utilising an appropriate representation can well balance data storage and accuracy. In addition, the accuracy (or data utility) of specific task is also affected by the models in Encoding and Aggregation modules, not just the data itself. Thus utilising suitable models is also important for performance advancement.

8. Conclusions

We propose a framework for multimodal differential privacy based on fusion representation learning. The framework is designed for general purposes of multimodal fusion representation learning, and consists of Encoding, Perturbation and Aggregation modules. Based on the framework, we propose two approaches for vector-valued and tensor-valued representation respectively. Both our approaches reach great data utility while maintaining a privacy guarantee. We validate our framework and approaches on three representative multimodal datasets. Experimental results show the high performance on data utility and privacy protection, and also demonstrate the flexibility and practicability of the framework. In the future, we will improve the general framework by making full use of the advantages of deep learning models and reducing information loss. We will also improve the differential privacy mechanisms on representations, such as assigning different privacy budgets to different modalities.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Key-Area Research and Development Program of Guangdong Province [grant number 2020B010164003], and the Science and Technology Program of Guangzhou, China [grant number 201904010209].