8 Google employees invented modern artificial intelligence! Here’s the story.
It's a story of ambition, passion, dedication, and collaboration embodied by eight of the world's best AI engineers, whose names are not only etched in the annals of history but also intricately tied to the essence of AI's inception. The AI revolution didn’t make any noise until a research paper titled "Attention is All You Need" was published, authored by these eight engineers who are the focus of this article.
These individuals, who were once Google employees, embarked on a journey to redefine the world, heralding the dawn of an era where machines learn, adapt, and evolve. As we approach the seventh anniversary of this paper's publication, it’s undeniable that this work has achieved legendary status in the field of modern AI.
In this article, we unveil how these eight individuals reshaped the course of technology and ushered in the dawn of modern AI.
Read also: What is Artificial Intelligence? Everything You Need to Know About AI.
The authors began with a flourishing and advanced technology—neural networks—and transformed it into something else, a digital system so powerful that its outputs could seem like the product of alien intelligence! This architecture is called transformers, and it’s the secret behind all the amazing AI products we now know, including ChatGPT and image generators like DALL-E and Midjourney.
The authors had no idea how famous this research paper would become, and all of them are now celebrities because of it! Geoffrey Hinton—who isn’t one of the authors but is perhaps the most prominent AI scientist in the world—said, "Without this brilliant paper on transformers, I don't think we would be where we are today in the world of AI." All eight authors have since left Google and have started working on a system that gathers the essence of human minds to create a machine that might save humanity.
The whole story begins with the fourth name among the eight engineers: Jakob Uszkoreit, the renowned computational linguist. When Jakob was a high school student in the late 1960s, he was imprisoned for 15 months in his native East Germany for protesting the Soviet invasion of Czechoslovakia. After his release, he fled to West Germany and studied computer science and linguistics in Berlin. He then made his way to the United States and worked at the AI lab at SRI Research Institute in Menlo Park, California.
1- Jakob Uszkoreit:
Jakob didn't initially intend to focus on natural language processing, but as he began his graduate studies, he secured an internship at Google in their Mountain View office, where he joined the company’s translation team. Uszkoreit abandoned his PhD plans to join Google, and in 2012, he decided to become part of a team at Google working on a system that could answer users’ questions directly on the search page without redirecting them to other websites.Meanwhile, Apple had just announced Siri, the virtual assistant that conversationally promised quick answers, and Google executives felt a huge competitive threat. As a result, they began paying more attention to Uszkoreit’s new team.
Uszkoreit recalls, "It was a false panic," as Siri never truly posed a threat to Google. However, he welcomed the opportunity to delve into systems where computers could engage in a sort of dialogue with humans. At the time, recurrent neural networks—once academically uncommon—suddenly began outperforming other AI engineering methods. These networks consist of multiple layers through which information is passed and re-passed to determine the best responses. Neural networks were achieving tremendous victories in fields like image recognition, sparking a sudden renaissance in modern AI.
Google aimed to produce systems capable of generating human-like responses, such as auto-completing sentences in emails or creating relatively simple chatbots for customer service. In 2014, Jakob Uszkoreit began developing a different approach, which he referred to as "self-attention." This type of network could translate a word by referencing any other part of a passage, with these other parts clarifying the word’s purpose and helping the system produce a good translation.
Although modern AI scientists were careful not to confuse the metaphor of neural networks with how the biological brain functions, Uszkoreit seemed to believe that the self-attention technique somewhat resembled the way humans process language. Uszkoreit then managed to persuade a few of his colleagues to experiment with self-attention. Their work showed promising results, and in 2016, they published a paper on the subject. Rather than conducting further research at that point, the researchers preferred to apply the lessons they had learned, eager to reap the rewards and disseminate them across various parts of Google.
One day in 2016, Uszkoreit was having lunch at a Google cafeteria with a scientist named Ilya Polosukhin, one of the eight protagonists of this story. Born in Ukraine, Polosukhin had been working at Google for nearly three years, and things weren’t going well for him at the time. Polosukhin said, "To answer something on Google.com, you need something very cheap and high-performance because you only have milliseconds to respond." When Polosukhin voiced his concerns, Uszkoreit had no issue suggesting his magic solution: "Why not use self-attention?"
2- Ilya Polosukhin:
Polosukhin occasionally collaborated with a colleague named Ashish Vaswani, one of the eight engineers. Vaswani was born in India and raised in the Middle East, before attending the University of Southern California to pursue a PhD in the university’s elite machine translation group. After that, he moved to Mountain View to join Google, specifically a new organization called Google Brain. However, he was still searching for a significant project to work on. His team 1965 was working in a building next to the language team Polosukhin was in in 1945, and he heard about the self-attention idea from Polosukhin. He immediately showed interest in the project and agreed to work on it.The three researchers together prepared a document titled "Transformers: Self-Attention Recurrent and Processing for Various Tasks." By then, new collaborators were joining the effort. Among them was an Indian engineer named Niki Parmar, who had been working at an American software company in India before moving to the United States. Niki earned her master’s degree from the University of Southern California in 2015 and received job offers from all the major tech companies, but she chose Google. When she started, she joined Uszkoreit’s team and worked on various models to improve Google’s search engine.
Read also: Is it wrong for people to use AI-generated art?
3. Ashish Vaswani:
Another new member who joined the team was Llion Jones, who was born and raised in Wales. At the University of Birmingham, he took a course in artificial intelligence, and neural networks piqued his curiosity. At the time, they weren't explained in detail as they are now but rather presented as a historical development. He earned his master’s degree in July 2009, but couldn't find a job during the recession, so he lived on welfare for several months. Eventually, he found a job at a local company, then applied to Google and got hired. He ended up in one of the research and development teams, where Polosukhin was his manager. One day, Jones heard about the concept of self-attention from a colleague named Matt Kelsey and later joined Uszkoreit’s team.4. Niki Parmar:
Over time, this transformative project attracted other researchers at Google Brain who were also trying to improve large language models. This third wave included Lukasz Kaiser, a Polish-born theoretical computer scientist, and his intern Aidan Gomez. Gomez grew up in a small farming village in Ontario, Canada, and when he was a freshman at the University of Toronto, he became fascinated with AI and joined the machine learning group (Geoffrey Hinton’s lab). A testament to Gomez’s brilliance is that he enrolled in several courses, not realizing until months later that they were meant for PhD students, not undergraduates like himself.5. Llion Jones:
Uszkoreit’s team began building the "self-attention" model to translate text from one language to another. They measured its performance using a metric called BLEU, which compares machine output to human translator work. From the start, their new model performed remarkably well. In 2017, Naftali Shazeer—one of the eight engineers—heard about their project by chance. Shazeer was a veteran at Google, having joined the company in 2000, and was something of a legend within the firm. He had been working on deep learning for five years and had recently become interested in large language models. But these models weren’t close to producing the seamless conversations he believed were possible between humans and machines.Shazeer’s joining the group was crucial. Uszkoreit says, "These theoretical mechanisms like self-attention always require extremely precise implementation, often carried out by a select few genius engineers with experience," and Shazeer was one of those genius engineers.
6. Noam Shazeer:
The team was highly motivated and also wanted to finish their work before May 19, the deadline for submitting papers to the largest AI event, the Conference on Neural Information Processing Systems in December. As winter in Silicon Valley turned to spring, the pace of experiments accelerated. They tested two versions of the transformers: one produced after 12 hours of training, and a stronger version called Big, which was trained over three and a half days. They tasked them with translating from English to German.The basic model outperformed all competitors, and Big achieved a high BLEU score that decisively broke previous records while also being more computationally efficient. And that was just the beginning, as the model continued to improve. The team then focused their research attention on applying transformer models to essentially all forms of human expression, planning to expand the model’s work to include exploring images, audio, and video.
7. Aidan Gomez:
Google, like nearly all tech companies, quickly filed provisional patents for this work. The purpose wasn’t to prevent others from using the ideas but rather to build its patent portfolio for defensive purposes to protect the company’s intellectual property. By December 2017, the paper they published, "Attention is All You Need," was generating a lot of buzz. Their four-hour session on December 6th was packed with scientists eager to learn more. By 10:30 PM, when the session ended, there was still a crowd, and security had to ask them to leave. Perhaps the most gratifying moment for the team was when computer scientist Sepp Hochreiter praised their work, considering that Hochreiter was the co-inventor of Long Short-Term Memory (LSTM), which the team's paper had just displaced as the go-to tool in the AI toolkit.A startup called OpenAI was much faster to pounce on this new technology than Google. Shortly after the research was published, Ilya Sutskever, OpenAI's chief scientist who knew Uszkoreit’s team from his time at Google, suggested that one of OpenAI's AI scientists work on the idea. This led to the first GPT products. Sam Altman, CEO of OpenAI, said last year, "When the Attention paper came out, I don’t think anyone at Google realized what it meant." He implied that Google didn’t give the paper the attention it deserved, which caused it to lag behind OpenAI.
8. Lukasz Kaiser:
Google began integrating the transformer concept covered in the research paper into its products in 2018, starting with its translation tool. That same year, it introduced a new language model based on transformers called BERT, which it began applying to search the following year. But these changes by Google seem modest compared to the quantum leap made by OpenAI and the bold integration Microsoft carried out with transformer-based systems in its product line.It’s an undeniable fact that all eight authors of the paper have since left Google to forge their paths. Polosukhin founded Near, a company that built a blockchain with market tokens valued at around $4 billion. Ashish Vaswani and Niki Parmar teamed up as business partners in 2021 to start Adept, valued at an estimated $1 billion, and are now on their second company, Essential AI, with $8 million in funding.
Shazeer co-founded Character AI, valued at around $5 billion. Aidan Gomez co-founded Cohere in Toronto in 2019, valued at $2.2 billion. Jakob Uszkoreit’s biotech company, Inceptive, is worth about $300 million. Notably, all these companies—except Near—rely on transformer technology.
In conclusion, the story of the eight Google employees who propelled modern AI into existence is not just a historical milestone in technological advancement but a testament to human curiosity and potential. Their journey serves as a reminder that the most extraordinary achievements aren’t made by the typical individuals who just want to do their jobs, but by creative people driven by a passion for what they do. As we stand on the brink of a future shaped by AI, let’s draw inspiration from the pioneering spirit of these eight individuals and find within ourselves the motivation to innovate.