Elon Musk has echoed recent sentiments shared by AI experts about the potential scarcity of real-world data for training artificial intelligence models? Speaking during a livestream with Stagwell chairman Mark Penn, Musk remarked, “We�ve now exhausted basically the cumulative sum of human knowledge �? in AI training?” He noted that this phenomenon occurred around last year?
Shift to Synthetic Data
The sentiment Musk shared aligns with former OpenAI chief scientist Ilya Sutskever’s views expressed at NeurIPS, a prominent machine learning conference? Sutskever referred to this situation as “peak data” and predicted a shift from current model development techniques due to the lack of new data?
Musk suggested that synthetic data, which is data generated by AI models themselves, could be a viable alternative? He emphasized, �The only way to supplement [real-world data] is with synthetic data, where the AI creates [training data]? With synthetic data � [AI] will sort of grade itself and go through this process of self-learning?�
Adoption by Industry Leaders
Various tech giants, including Microsoft, Meta, OpenAI, and Anthropic, are already harnessing synthetic data for training their flagship AI models? According to Gartner, 60% of the data for AI and analytics projects by 2024 is expected to be synthesized?
For instance, Microsoft�s Phi-4 model, which was open-sourced recently, uses synthetic data alongside traditional real-world data? Similarly, Google�s Gemma models and Anthropic�s Claude 3?5 Sonnet system have integrated synthetic data components? Meta has refined its latest Llama series of models utilizing AI-generated data?
Advantages and Concerns
Despite its benefits, such as cost-effectiveness, synthetic data has its challenges? AI startup Writer, for instance, reported that its Palmyra X 004 model was developed with mostly synthetic data at a significantly lower cost of $700,000, compared to the estimated $4?6 million for a similar-sized model from OpenAI? However, the use of synthetic data also presents risks?
Some studies suggest that reliance on synthetic data might lead to what is known as model collapse? This issue arises when a model becomes less innovative and more prone to bias, eventually impairing its functionality? If the original datasets used to generate synthetic data contain biases, this issue might be perpetuated in the outputs?