Artificial intelligence: reflecting on the past and looking towards the next paradigm shift

Figures & data

Figure 1. The logic theorist (Newell & Simon, 1956).

Figure 2. The Dartmouth Workshop. In the back row from left to right are Oliver Selfridge, Nathaniel Rochester, Marvin Minsky, and John McCarthy. In front on the left is Ray Solomonoff; on the right, Claude Shannon. The identity of the person between Solomonoff and Shannon remained a mystery for some time. (photo: Margaret Minsky found in (Solomonoff, 2023)).

Figure 3. Chapter 12 of perceptrons (Minsky & Papert, 2017).

Figure 4. ImageNet classification with deep convolutional neural networks (Krizhevsky et al., 2012, 2017).

Figure 5. DeepFace: Closing the gap to human-level performance in face verification’ (Taigman et al., 2014).

Table 1. Comparison between BERT and BART in a summary table.

Feature	BERT (Bidirectional Encoder Representations from Transformers)	BART (Bidirectional and Auto-Regressive Transformers)
Architecture	Transformer-based, predominantly Encoder	Transformer-based, includes both Encoder and Decoder
Training Direction	Bidirectional	Bidirectional, but also auto-regressive in the decoding phase
Training Tasks	Masked Language Model (MLM) and Next Sentence Prediction (NSP)	Multiple tasks like text generation, summarisation, etc.
Contextual Understanding	Yes, both left and right context	Yes, with enhanced sequence-to-sequence tasks
Fine-tuning Required?	Required for specific tasks	Required for specific tasks
Token Handling	Masks certain tokens for prediction	Entire sequences can be masked
Applications	Text Classification, Named Entity Recognition, Sentiment Analysis	Text Summarisation, Translation, Question-Answering
Developer	Google	Facebook AI
Year of Introduction	2018	2019
Notable Derivatives	RoBERTa, DistillBERT	mBART, BART-large
Impact on NLP	Pioneering, set new standards for NLP tasks	Extended capabilities, especially in sequence-to-sequence tasks

Newell, A., & Simon, H. A. (1956). The logic theory machine a complex information processing system. IRE Transactions on Information Theory, 2(3), 61–79. https://doi.org/10.1109/TIT.1956.1056797

 Google Scholar

Solomonoff, G. (2023). The meeting of the minds that launched AI there’s more to this group photo from a 1956 AI workshop than you’d think. IEEE Spectrum, Guest Article(History Of Technology). https://spectrum.ieee.org/dartmouth-ai-workshop.

 Google Scholar

Minsky, M., & Papert, S. A. (2017, January). Perceptrons: An introduction to computational geometry. Perceptrons. https://doi.org/10.7551/MITPRESS/11301.001.0001

 Google Scholar

Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Proceedings.Neurips.cc. Retrived October 20, 2023, from https://proceedings.neurips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html

 Google Scholar

Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017, June). ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6), 84–90. https://doi.org/10.1145/3065386

 Web of Science ®Google Scholar

Taigman, Y., Yang, M., Ranzato, M. A., & Wolf, L. (2014). Deepface: Closing the gap to human-level performance in face verification. Proceedings of the IEEE conference on computer vision and, 2014•openaccess.thecvf.com. October 20, 2023, from http://openaccess.thecvf.com/content_cvpr_2014/html/Taigman_DeepFace_Closing_the_2014_CVPR_paper.html

 Google Scholar