Contractors working to improve Google's Gemini AI are comparing results against those produced by Anthropic's rival model, Claude, according to internal correspondence seen by TechCrunch.
Google would not say when TechCrunch reached for comment if it had received permission to use Claude in testing Gemini.
As tech companies race to build better AI models; The performance of these models is typically evaluated against competitors by running them. Own models with industry standards Rather than having contractors carefully evaluate their opponents' AI responses.
Contractors working on Gemini are tasked with rating the accuracy of the model's outputs, scoring every response they see according to several criteria, such as accuracy and validity. His response to the correspondence, seen by TechCrunch, gave contractors up to 30 minutes to decide whether Gemini or Claude's was better.
The Journal reports that contractors have recently noticed references to Anthropic's Claude appearing on the internal Google platform they use to compare it to Gemini. At least one of the exits submitted to Gemini contractors, seen by TechCrunch, explicitly stated, “I am Claude created by Anthropic.”
In an internal chat, the contractors showed that Claude's reactions showed a greater emphasis on safety than Gemini. “Claude's security settings are the strongest among AI models,” wrote one contractor. In some cases, Claude will not respond to what is considered safe, such as acting as a different AI assistant. In another case, Claude avoids answering the alarm, and Gemini's response is flagged as a “major safety breach” including “barefoot and enslavement”.
Humanity Commercial Terms of Service Customers are not allowed access to “build a competing product or service” or “train competing AI models” without Anthropic's permission. Google is key. Investor Humanity.
When TechCrunch asked Shira McNamara, a spokeswoman for Google DeepMind — which runs Gemini — whether Google had obtained Anthropic's permission to use Claude, she said: Will not say no. A spokesperson for Manusa did not comment at a press conference before the news was released.
McNamara said that while DeepMind “compares model results” for evaluations, Gemini does not train Anthropic models.
“Of course, in accordance with standard industry practice, in some cases we compare model results as part of our evaluation process,” McNamara said. “However, the suggestion that Anthropic models were used to train Gemini is incorrect.”
last week, TechCrunch reports exclusively. Google contractors working on the company's AI products now have to rate Gemini's AI responses in areas outside their expertise. Contractors expressed concern in internal correspondence that Gemini could produce inaccurate information on highly sensitive topics such as health care.
Tips can be safely sent to this reporter at Signal at +1 628-282-2811.
TechCrunch has an AI-focused newsletter. Register here. Get it in your inbox every Wednesday.