Prospects and applications of photonic neural networks

Figures & data

Figure 1. Artificial neuron model

Table 1. Efficiency and speed of various index modulation techniques on silicon photonics. Options for phase modulation of silicon waveguides. a) thermal tuning with TiN filament; b) thermal tuning with embedded photoconductive heater; c) PN/PIN junction across the waveguide for injection and/or depletion modulation; d) III–V/Si hybrid waveguide; e) metal-oxide-semiconductor (MOS), where the ‘metal’ is actually an active semiconductor; f) lithium niobate cladding adds a strong electrooptic effect; g) 2 single-layer-graphene (SLG) ‘capacitor’; h) non-volatile phase change material. ${ }^1$This bandwidth was not yet shown experimentally. A big challenge is to reduce the contact resistance with Graphene, reducing RC-loading effect. ${ }^2$Not experimentally shown at high-speed. ${ }^3$Demonstrated up to 20 MHz

Modulation Effect	Speed	Efficiency	Ref.
Thermo-optic TiN [a)	5.6$\mu$s	$P_{\pi }L=6.8\mathrm{m}\mathrm{W}\mathrm{m}\mathrm{m}$	99,
Thermo-optic N+/N/N+ Si (b]	$\mu$s	$P_{\pi }L=0.8\mathrm{m}\mathrm{W}\mathrm{m}\mathrm{m}$	Jayatilleka et al. (2015)
Reverse-biased PN [c)	41 GHz	$V_{\pi }L=46\mathrm{V}\mathrm{m}\mathrm{m}$	100,
Graphene SLG [g]	30 GHz${ }^1$	$V_{\pi }L=28\mathrm{V}\mathrm{m}\mathrm{m}$	101,
LiNbO3/Si Hybrid [f]	70 GHz	$V_{\pi }L=22\mathrm{V}\mathrm{m}\mathrm{m}$	102,
III–V MQW/Si Hybrid (d]	27 GHz	$V_{\pi }L=2.4\mathrm{V}\mathrm{m}\mathrm{m}$	H.-W. Chen et al. (2011)
III–V/Si MOS [e)	2.2 GHz	$V_{\pi }L=0.9\mathrm{V}\mathrm{m}\mathrm{m}$	103,
ITO MOS [e]	GHz${ }^2$	$V_{\pi }L=0.52\mathrm{V}\mathrm{m}\mathrm{m}$	104,
Forward-biased PIN [c]	0.5 GHz	$V_{\pi }L=0.36\mathrm{V}\mathrm{m}\mathrm{m}$	105,
PCM (h]	0.8 GHz${ }^3$	$E_{\pi }=400\mathrm{p}\mathrm{J}$	Ríos et al. (2015)

Figure 2. Generic system schematic of an analog photonic neural network or photonic tensor core accelerator. The digital-to-analog domain crossings are power costly and ideally a photonic DAC without leaving the optical domain, e.g. Ref [112] would be used instead, thus reducing system complexity and hence allowing for extended scaling laws. The optical processor itself can perform different operations depending on the desired application; is the processor a tensor core accelerator, then it will perform MAC operations such as used directly in VMMs or convolutions, e.g. Ref. 138. Is the processor a neural network, then nonlinearity and summations are needed in addition to linear operations. For higher biological plausibility such as for temporal processing of signals at each node, then mapping of partial differential equations onto photonic hardware is needed. This includes spiking neuron approaches where event-driven scheme play a role) [72,85,113–116]

Figure 3. Schematic illustration of photonic neural networks training through in situ backpropagation. Colored squares represent phase shifters. (a) Forward propagation: send in ${\mathbf{X}}_{l-1}$ and record the intensity of the electric field ${\left|{\mathbf{e}}_{og}\right|}^2$ at each phase shifter. (b) Backpropagation: send in error signal ${\delta }_l$ and record the electric field intensity ${\left|{\mathbf{e}}_{aj}\right|}^2$ at each phase shifter and the complex conjugate output ${\mathbf{X}}_{TR}^{\ast }$. (c) Interference measurement: send in ${\mathbf{X}}_{l-1}+{\mathbf{X}}_{TR}$ to get gradient of the loss function $\mathcal{L}$ for all phase shifters simultaneously by extracting $k_0^2\mathcal{R}\{{\mathbf{e}}_{og}{\mathbf{e}}_{aj}\}$. (Tyler et al., 2018) Hughes et al. (2018)

Figure 4. (a) Illustration of pre-synaptic and post-synaptic spikes from two neurons N1 and N2. (b) Illustration of a STDP response. Right purple region: long-term potentiation; Left brown region: long-term depression. tpost-tpre: time difference between the firing of the post- and pre-synaptic spikes. (c) Experimentally measured STDP curves from the photonic based STDP circuit

Figure 5. (left) Concept of training and implementing photonic neural networks. Inset shows the NLAF of the photonic neural network measured with real-time signals. The NLAF is realized by a fast O/E/O neuron using a SiGe photodetector connected to a microring modulator with a reverse-biased PN junction embedded in the microring. (right) Constellations of X-polarization of a 32 Gbaud PM-16QAM, with the ANN-NLC gain of 0.57 dB in Q-factor and with the PNN-NLC gain of 0.51 dB in Q-factor [22]

Figure 6. (a) Generic model of a reservoir computer [b) Illustration of the RC equalization scheme in 162,based on a single nonlinear node with time delayed feedback. The input ${\mathbf{a}}^{\mathbf{i}}$ is a vector of N samples representing a single bit or symbol. The mask vector $\mathbf{m}$, which defines the input weights, is multiplied by the elements in ${\mathbf{a}}^{\mathbf{i}}$ and injected sequentially into the reservoir. The output is a linear combination of the virtual nodes in the reservoir with weights optimized to produce an estimate ${\mathbf{b}}^{\mathbf{i}}$ of the bit or symbol value

Figure 7. (a) Illustration of the JAR design and the four functional units. (b) Spectral waterfall measurement of the photonic JAR in action with sinusoidal reference signal fR and jamming signals fJ = 150 MHz. (i) fJ is approaching fR from the low frequency side and triggers the JAR, (ii) fJ is approaching fR from the low frequency side and triggers the JAR, and then is moved away

Figure 8. Concept of blind source separation

Figure 9. (Comparison of multi-antenna radio front-ends followed by dimensionality reduction. a) Dimensionality reduction with electronic DSP in which each antenna requires an ADC. b) Dimensionality reduction in the analog domain [a.k.a physical layer) in which only one ADC is required. A photonic implementation of weighted addition is pictured, consisting of electrooptic modulation, WDM filtering, and photodetection. From .177

Figure 10. (Experimental waveforms of the received mixtures (left column) and corresponding estimated sources (right column) when performing FastICA (black) and photonic BSS (red) at three frequency bands whose central frequencies are at: (a) 781 MHz, (b) 880 MHz, and [c) 1000 MHz. From .24

Figure 11. Photonic Tensor Core [PTC) featuring Peta-OPS throughput and 10s ps-short latency to process VMM in PICs 138. (left] schematic of a modular exemplary PTC composed of 4$\times$4 photonic dot-product engines (i.e. 1$\times$4 vector-multiplications) using ring resonators enabling WDM-based parallelism. Data entered via (exemplary) EO modulators at 50 Gbps are MUXed then dropped at passive ring resonators. Being ‘passive’ for the ring resonator is importantly beneficial since it allows for reduced complexity, less real-estate used on the chip, and does not consume power unlike thermally-tuned approaches to perform the optical multiplication. (right) The latter can be elegantly executed using nonvolatile photonic random access memories (P-RAM) based on electrically-programmable phase change materials (PCM). The image shows a prototype of a 3-bit (8-states) binary-written P-RAM element on a Silicon PIC. Using non-GST PCMs enables a rather low (<0.02 dB/state) insertion loss [135]

Figure 12. Example of an Optical Convolutional Neural Network (CNN) accelerator exploiting the massive (10⁶ parallel channel) parallelism of free-space optics. The convolutional filtering is executed as point-wise dot-product multiplication in the Fourier domain. The conversion into and out-of the Fourier domain is performed elegantly and completely passively at zero power While SLM-based systems can perform such Fourier filtering in the frequency domain, the slow update rates does not allow them to outperform GPUs. However, replacing SLMs with fast 10s kHz programmable digital micromirror display [DMD) units, gives such optician CNNs an edge over the top-performing GPUs. Interestingly, the lack-of-phase information of these amplitude-only DMD-based optical CNNs can be accounted for during the NN training process. For details refer to .117

Figure 13. Schematic illustration of a convolution. An input image $\mathbb{A}$ with dimensionality $H\times W\times D$ (where $H$, $W$ and $D$ are the height, width and depth of the image, respectively) is convolved with a kernel $\mathbb{F}$ with dimensionality $R\times R\times D$, to produce an image $\mathbb{O}$. Assuming $H=W$, the overall output dimensionality is $(H-R+1{)}^2$. The bottom of the figure shows how a convolution operation generalized into a single matrix-matrix multiplication. where the kernel $\mathbb{F}$ is transformed into a vector with $D{R}^2$ elements, and the image is transformed into a matrix of dimensionality $D{R}^2\times (H-R+1{)}^2$. Therefore, the output is represented by a vector with $(H-R+1)$ elements

Figure 14. Photonic architecture for producing a single convolved pixel. Input images are encoded in intensities $A_{l,h}$, where the pixel inputs $A_{m,n,k}$ with $m\in [i,i+R_m],n\in [j,j+R_m],k\in [1,D_m]$ are represented as $A_{l,h}$, $l=1,\dots ,D$ and $h=1,\dots ,R^2$. Considering the boundary parameters, we set $D=D_m$ and $R=R_m$. Likewise, the filter values $F_{m,n,k}$ are represented as are represented as $F_{l,h}$ under the same conditions. We use an array of $R^2$ lasers with different wavelengths ${\lambda }_h$ to feed the MRRs. The input and kernel values, $\mathrm{W}\mathrm{C}\mathrm{I}\mathrm{M}\mathrm{\_}\mathrm{A}\mathrm{\_}1985028$ and $F_{l,h}$ modulate the MRRs via electrical currents proportional to those values. Then, the photonic weight banks will perform the dot products on these modulated signals in parallel. Finally, the voltage adder with resistance $\mathbf{R}$ adds all signals from the weight banks, resulting in the convolved feature

Figure 15. Fully parallel and passive convolutions integrated in 3D. (a) 9 convolutional Haar filters and their 3D integration topology using photonic waveguides. (b) SEM micrograph of the 3D printed convolutional filter unit, fabricated with direct laser writing. (c) The filter’s optical transfer function agrees well with the target defined in [a). Figure reproduced from .186

Figure 16. (a] Illustration of a recurrent neural network at time step $t$. (b) Illustration of the recurrent representation of the physical wave system. [c) A vowel classification task on an optical analog recurrent neural network. $x(t)$ is the waveform input signal at the source location. $P_x(t)$ is the output signal at different probe locations. Figure reproduced from .189

Figure 17. Schematic figure of the procedure to implement the MPC algorithm on a neuromorphic photonic processor. Firstly, map the MPC problem to QP. Then, construct a QP solver with continuous-time recurrent neural networks (CT-RNN) [194]. Finally, build a neuromorphic photonic processor to implement the CT-RNN. The details of how to map MPC to QP, and how to construct a QP solver with CT-RNN are given in De Lima et al. (2019). Adapted from De Lima et al. [27]

Figure 18. Photonic neural network benchmarking against a CPU. (a,b) Phase diagrams of the Lorenz attractor simulated by a conventional CPU (a) and a photonic neural network (b). (c,d) Time traces of simulation variables (blue, green, red) for a conventional CPU (c) and a photonic CTRNN [d). The horizontal axes are labeled in physical real time, and cover equal intervals of virtual simulation time, as benchmarked by $\gamma$CPU and $\gamma$Pho. The ratio of real-time values of $\gamma$‘s indicates a 294-fold acceleration. From .118

Figure 19. A) Side view (i) no camouflage – fish is visible (ii) silvering – fish is destructively interfered at colors that could indicate the presence of the fish; (b) Bottom view (i) no camouflage – fish appears darker against the bright water surface when seen from below (ii) counterillumination – fish illuminates itself to the same color and intensity as the background. (c) Silvering (i) photonic RF FIR creates destructive interference condition at the stealth signal frequency (f_s); (ii) Transmission in optical fiber will only push the constructive interference condition to a much higher frequency (f_c+); (iii) Dispersion compensation fiber at the last section of the transmission will move the constructive interference condition back to f_c; (iv) Correct dispersion at the stealth receiver allows constructive interference condition to occur at the stealth signal frequency f_s. (d) Measured RF spectra and constellation diagrams (i) during transmission without a correct stealth receiver (ii) at the designated stealth receiver with correct location and dispersion

Figure 20. (left) Photonic machine-learning accelerators such as photonic tensor unit (PTU) [.138] enabled a higher level of data security with reduced overhead (e.g. time and power consumption). This opens possibilities for not only securing ‘secure’ and ‘confidential’ data, but also meta-data. [right) Flow-chart for detection of anomalies and misuses exploiting rapidly updating photonic neural network and trained photonic neural network includes programmable nonvolatile photonic memory on-chip for zero-static power consumption processing, once the kernel is written 138

Figure 21. Photonic tensor core and neural networks enable intelligent prefiltering and correlation for scientific discovery, such as between electromagnetic signals collected by Very-Large-Array telescope systems performing intelligent pre-filtering, thus reducing computation load in supercomputers

Jacques M, Samani A, El-Fiky E, et al. Optimization of thermo-optic phase-shifter design and mitigation of thermal crosstalk on the SOI platform. Opt Express. 2019;27:10456.

 PubMed Web of Science ®Google Scholar

Patel D, Ghosh S, Chagnon M, et al. Design, analysis, and transmission system performance of a 41 GHz silicon photonic modulator. Opt Express. 2015;23:14263.

 PubMed Web of Science ®Google Scholar

Sorianello V, Midrio M, Contestabile G, et al. Graphene-silicon phase modulat- ors with gigahertz bandwidth. Nat Photonics. 2018;12:40–44. Retrieved from.

 Web of Science ®Google Scholar

He M, Xu M, Ren Y, et al. High-performance hybrid silicon and lithium niobate Mach–Zehnder modulators for 100 Gbit s −1 and beyond. Nat Photonics. 2019;13:359–364. Retrieved from.

 Web of Science ®Google Scholar

Hiraki T, Aihara T, Hasebe K, et al. Heterogeneously integrated III-V/Si MOS capacitor Mach- Zehnder modulator. Nat Photonics. 2017;11:482–485. Retrieved from.

 Web of Science ®Google Scholar

Amin R, Maiti R, Carfano C, et al. (2018b, aug). 0.52 V-mm ITO-based mach-Zehnder modulator in silicon photonics. APL Photonics, 3, 126104. Retrieved from http://aip.scitation.org/doi/10.1063/1.5052635http://arxivorg/abs/180903544

 Web of Science ®Google Scholar

Green WM, Rooks MJ, Sekaric L, et al. (2007). Ultra-compact, low RF power, 10 Gb/s silicon Mach-Zehnder modulator. Opt Express, 15, 17106. Retrieved from https://www.osapublishing.org/oe/abstract.cfm?uri=oe-15-25-17106

 PubMed Web of Science ®Google Scholar

Meng J, Miscuglio M, George JK, et al. Electronic bot- tleneck suppression in next-generation networks with integrated photonic digital-to-analog converters. In: Advanced photonics research. 2019. p. 2000033.

 Google Scholar

Miscuglio M, Sorger VJ. Photonic tensor cores for machine learning. Appl Phys Rev. 2020;7:031404.

 Web of Science ®Google Scholar

Tait AN, De Lima TF, Nahmias MA, et al. Silicon photonic modulator neuron. Phys Rev Appl. 2019;11:064043.

 Web of Science ®Google Scholar

Jha A, Huang C, Prucnal PR. Reconfigurable all-optical nonlinear activation functions for neuromorphic photonics. Opt Lett. 2020;45:4819–4822.

 PubMed Web of Science ®Google Scholar

Huang C, Fujisawa S, De Lima TF, et al., (2020). Demonstration of photonic neural network for fiber nonlinearity compensation in long-haul transmission systems. In 2020 optical fiber communications conference and exhibition (ofc) (pp. 1–3), San Diego, California, USA.

 Google Scholar

Argyris A, Bueno J, Fischer I. Photonic machine learning implementation for signal recovery in optical communications. Sci Rep. 2018;8:1–13.

 PubMed Web of Science ®Google Scholar

Tait AN, De Lima TF, Ma PY, et al. (2018). Blind source separation in the physical layer. In 2018 52nd annual conference on information sciences and systems (ciss) pp. (1–6), Princeton, NJ, USA.

 Google Scholar

Ma PY, Tait AN, Zhang W, et al. Blind source separation with integrated photonics and reduced dimensional statistics. Opt Lett. 2020;45:6494–6497.

 PubMed Web of Science ®Google Scholar

Miscuglio M, Meng J, Yesiliurt O, et al. (2020). Artificial synapse with mnemonic functionality using gsst-based photonic integrated memory. In 2020 international applied computational electromagnetics society symposium (aces) pp. (1–3), Monterey, CA, USA.

 Google Scholar

Miscuglio M, Hu Z, Li S, et al. Massively parallel amplitude-only Fourier neural network. Optica. 2020;7:1812–1819.

 Web of Science ®Google Scholar

Moughames J, Porte X, Thiel M, et al. Three-dimensional waveguide interconnects for scalable integration of photonic neural networks. Optica. 2020;7:640–646.

 Web of Science ®Google Scholar

Hughes TW, Williamson IAD, Minkov M, et al. (2019). Wave phys- ics as an analog recurrent neural network. Sci Adv, 5 . Retrieved from https://advances.sciencemag.org/content/5/12/eaay6946

 Web of Science ®Google Scholar

Cichocki A, Unbehauen R, Swiniarski RW. Neural networks for optimization and signal processing. Vol. 253. wiley New York; 1993.

 Google Scholar

De Lima TF, Peng H-T, Tait AN, et al. Machine learning with neuromorphic photonics. J Lightwave Technol. 2019;37:1515–1534.

 Web of Science ®Google Scholar

Tait AN, De Lima TF, Zhou E, et al. Neuromorphic photonic networks using silicon photonic weight banks. Sci Rep. 2017;7:1–10.

 PubMed Web of Science ®Google Scholar