When Rishub Jain worked on safety at Google DeepMind, he held one of the most coveted jobs in tech. He was among a small group of researchers working to mitigate the most consequential risks of AI.

But he left to cofound a nonprofit, Sampura Research. Jain traded the prestige and access that come with a Google job because he feels he can have a sharper, more profound impact on AI safety from the outside. Jain says he wants the autonomy to pursue research on a specific issue in AI alignment, which he felt had been deprioritized.

"All the top AI companies do care a lot about safety, and they invest in it, but they could be investing a lot more, and that's kind of what was inhibiting me," Jain says. "If they invested more in safety, I would still be at the lab and working on what I'm interested in with a large team." Google did not respond to a request for comment about whether it maintains a focus on this specific area of research, but DeepMind's principles state that it maintains "a holistic approach to responsibility, safety and security."

Jain's founder story echoes concerns raised by safety researchers who have quit OpenAI and Anthropic, including dire warnings that these companies are "gambling with our lives," and could develop technology that ends humanity. Once dismissed as fringe and doomsday-obsessed, the AI safety profession is growing as hacks by rogue agents and a string of departures have vaulted the issue into mainstream concern.

Open roles in AI safety jumped by 91% on LinkedIn over the past year. The industry expects a flood of cash from much-anticipated IPOs at Anthropic and, potentially, OpenAI, as those made absurdly wealthy by the equity returns may funnel money into potential clean-up crews for the messes created by the technology's rapid development.

The AI safety problem is existentially big, but the people who have spent time honing AI expertise are few. These are people who test and train AI systems to operate ethically. Some focus on technical research, others on policy and governance. And they're increasingly sounding alarms.

"The people who are most worried about AI are often the people who have been closest to the technology," Laura González Salmerón, advising manager at 80,000 Hours, a nonprofit focused on transitioning people into careers with high social impact. The scarcity issue is less about funds and more about people. "Talent is the main bottleneck."

Startups and nonprofits in AI safety must fight for talent in a shallow pool.

Jain's Sampura lists a salary range of $273,000 to $395,000 for an experienced team leader, which would put that worker in the top 3% of earners in the US, but likely pales in comparison to offers from larger companies. Meta reportedly offered $100 million in signing bonuses last year to poach the world's top experts. OpenAI gives employees an average of $1.5 million in stock-based compensation, and technical staff at Anthropic can earn more than $1 million annually.

Money aside, researchers are divided over whether they're more effective when working inside tech companies to mitigate risks as the latest models develop, or stepping back and attempting to steer the ship toward safer outcomes from the outside, like Jain did.

"I'm a little bit concerned about the pace of AI, and if we're able to actually come up with something useful before the rug gets pulled beneath us," Jain says. He's worried that AI's advancement will outpace the work to keep it in line with human values. "That's kind of what's been happening already."

David Robinson, who led transparency work for OpenAI's safety team, wrote an essay for The Atlantic explaining his departure. He said that AI companies should have more layers of checks and safety measures, akin to those in airports and nuclear energy, rather than the steamrolling culture of Big Tech.

"Perhaps I should have stayed and fought for fundamental shifts in our staffing and culture, but in practice, my colleagues and I were so busy sprinting that we seldom had the chance to consider big changes, much less to actually make them," he wrote. Robinson has said he plans to work on these issues from outside of the company.

As the issue grows in prominence, so has a niche of organizations helping train and recruit workers — and the opportunities aren't just for AI experts. Neav Topaz, co-director of Kairos, a nonprofit that upskills people to work in AI safety and recently raised $50 million from Coefficient Giving, says he speaks with about 10 to 20 people each month who want to move to AI safety, and interest is growing. He tells me that people are asking him more frequently about what the work entails.

"People are watching AI get good at their own jobs and starting to ask themselves what comes next," he says. The emerging jobs in AI safety "pay well, the field is growing fast, and you get to work on what might be the defining problem of our time."

Kairos this summer launched its Generator Residency, a cohort of 27 people who spent three months training for career transitions into AI safety. It's less about the research and technical work and more about merging other expertise, like in consulting, with AI safety principles.

He says that of the 27 residents, a dozen have already landed roles in the AI safety space, and many of the rest have offers to do so.

"Most of the people who will do this work are doing something else today," Topaz says. "The biggest talent gap in the field isn't researchers. It's people who grasp how high the stakes are and can translate their varied expertise into things that could make a difference in this space."

The major AI companies maintain that safety is a top priority. Last month, Anthropic CEO Dario Amodei wrote an essay calling for a slowdown in AI development, and Sam Altman and Elon Musk have agreed with him. Amodei and Altman have both said recently that they will bring third-party evaluators into their companies to bolster safety protocols.

These companies have their own safety teams, but safety work competes with business priorities to build the most advanced models and win in the market. The tension, along with safety issues like OpenAI's rogue agents hacking Hugging Face, has led to more scrutiny of internal safety work. Within days of Robinson's departure, OpenAI fired three researchers, claiming they shared sensitive internal data improperly with third-party AI safety researchers. The ousted researchers wrote a letter, reported by The Wall Street Journal, to OpenAI's board, arguing that their engagement with safety researchers fell within the scope of their roles, and that their terminations were "chilling those who remain at OpenAI." OpenAI said that the decisions "were not about raising safety concerns or speaking out," and that the company would not fire people for doing so.

While some have obsessed over AI risks for years, 2026's breakthrough moment has accelerated funding for and interest in the industry. 80,000 Hours has had calls with some 8,000 people since its founding in 2011, the majority of which have focused on AI in some way, González Salmerón says. Philanthropic groups like Coefficient Giving offer massive grants to startups in the AI safety space. METR, a nonprofit AI safety group that investigated the OpenAI agentic hack on Hugging Face, says it raised $71 million in six months this year.

Much of the funding flowing into AI safety filters through effective altruism, the Silicon Valley-tied philanthropic ideal focused on using evidence and logic to donate money where it will achieve the most good. Critics say the philosophy can place too much emphasis on avoiding future problems rather than addressing imminent needs. With more alarms sounding around the existential threat of AI, more people view those theoretical, distant fears as urgent causes.

AI safety groups and individual donors have spent nearly $50 million in this year's election on candidates focused on regulating AI, Axios reported. Jaan Tallinn, a billionaire who invested early in Anthropic and could soon have even more billions as a result, has donated $170 million to nonprofits focused on AI safety. Tallinn's view, as he recently told The Wall Street Journal, is that he can have the biggest impact on the future of AI and its risks by investing in AI companies, and then donating returns to nonprofits working in the AI safety space.

Coefficient Giving, the philanthropic venture from Facebook cofounder Dustin Moskovitz rooted in the effective altruism movement, announced its AI-safety-focused arm, Project Tailwind, in September. It says it's on track to give more than $1 billion this year to help companies of all sizes scale up in preparation for what it believes will be a deluge of philanthropic capital. Sampura Research has received an $11 million grant from the group. If startups don't begin hiring now, the logic goes, they won't move quickly enough when massive wealth hits the industry.

The dizzying development of AI has created safety risks, and there are "gaps in the ecosystem that aren't being filled and that aren't being owned by people," says Rossa O'Keeffe-O'Donovan, the program director in technical AI safety at Coefficient Giving. There are few people, he says, who sit at the intersection of technical expertise, risk analysis, a solutions-oriented vision, and the capacity to hustle as a startup founder.

New roles will emerge not just for technical talent, but also for consultants, policy wonks, communications experts, and founders who can run a business alongside technical experts. Working from the outside, they'll face hurdles in buying compute power, a lack of direct access to the latest AI models, reduced leverage, and smaller paydays.

The world feels unmoored, and the future perhaps more malleable than ever. The appeal of doing work that does good could stave off feelings of uncertainty, dread, and anxiety that people increasingly feel toward AI. In one doomer camp, AI will obliterate white-collar jobs. The optimistic view is that AI drastically reshapes work and spurs the creation of jobs we haven't yet thought of. AI safety is one of the first.

"There are lots of people beyond AI researchers who can help, like cybersecurity experts, like biosecurity experts, like economists," says Mantas Mazeika, a researcher with the Center for AI Safety, a nonprofit focused on AI research and advocacy. "They're making very significant contributions that the pure AI researchers, who were most of the people focusing on the problem beforehand, would be completely unable to do on their own."

• Anthropic is looking to shape AI regulation by pitching an AI safety bill of its own.
• POLITICO obtained a confidential draft of the proposed legislation.
• The bill would let the government block an AI model only if it poses a foreseeable, imminent risk of more than $1 billion in damages, or of killing or seriously injuring more than 50 people.

Representatives for the artificial intelligence giant Anthropic have quietly pitched Capitol Hill lawmakers on a draft bill to regulate frontier AI safety and security, according to three people with knowledge of the efforts who were granted anonymity to discuss private deliberations.

The confidential draft, obtained by POLITICO and not previously reported, marks a notable effort by the frontier AI lab to shape the rules governing its own technology as Congress weighs how to respond to the fast-moving industry.

"This bill represents the proposals outlined in Anthropic's Advanced AI Framework, which we have been advocating for with federal and state lawmakers since June," an Anthropic spokesperson said. "These are sensible policies we believe Congress should act on."

The bill would let the government block a model's release or use only if it poses a foreseeable, imminent risk of more than $1 billion in damages, or of killing or seriously injuring more than 50 people. The risk must stem from the creation or release of a biological weapon, the model evading its developer's controls or a cyberattack. The language mirrors that found in California's AI laws including SB 53, which was signed into law last year.

Leading AI labs disclosed a series of incidents this summer, in which AI agents hacked or probed websites and platforms. None reported so far appear to have caused widespread deaths or over $1 billion in damages.

Anthropic has yet to publicly back particular legislative efforts in Congress, but they have backed some state AI safety laws, including two in California. Their head of U.S. state and local government relations previously told POLITICO that they hope states will continue to pass ever-stronger AI safety laws that raise the standard nationwide.

Under the bill, independent evaluators, which would register with the Commerce Department, would be tasked with periodically evaluating the lab and its frontier models. The evaluators may be paid for their work by the lab itself so long as the payment is not contingent on the evaluations' results.

If the evaluators determine there is "imminent enumerated risk" for a company's frontier models, they must report it to Commerce within 72 hours of completing their assessment.

While the Commerce Department could bring a request to the judge to halt the release of a model after determining that it poses a risk, the language explicitly prohibits any other U.S. "department, agency, or officer" from suspending the model's development, deployment or internal use. A federal judge for the District Court for the District of Columbia could order the suspension of a model.

Over the summer, Anthropic was subject to sweeping export controls that shut down two of its most advanced models. The decision was made in a whirlwind series of phone calls between administration officials and the company's executives, worrying some in the tech industry that access to their technology may be restricted on the whims of government officials.

Anthropic's bill would also preempt some state AI laws related to public disclosure of risk, third-party evaluation and reporting of critical incidents. Preemption has been a political flashpoint on Capitol Hill, with Democrats rejecting some proposals they feared would neuter regulations passed by the likes of California, Illinois and New York.

Many tech industry groups have argued that a state-by-state patchwork of laws is difficult to comply with compared to a unified national standard.

The legislation would also require frontier labs to create safety frameworks, publish transparency reports for new frontier models, implement protections for whistleblowers and report critical safety incidents to Commerce.

Capitol Hill has seen a flurry of activity on AI safety legislation this summer. In the House, Reps. Lori Trahan (D-Mass.) and Jay Obernolte (R-Calif.) spearheaded the FRONTIER Act, which would give the government the ability to stop the deployment of risky models and preempt some state AI laws.

In the Senate, Majority Leader John Thune and Sens. Amy Klobuchar (D-Minn.) and Ted Cruz (R-Texas) have been negotiating an AI safety bill intended to mitigate catastrophic risks. But it has faced headwinds from some safety-minded groups and Sen. Maria Cantwell (D-Wash.), the top Democrat on Cruz's Commerce Committee, who want the government to have a larger role in reviewing AI models.

Other progressives, including Rep. Greg Casar (D-Texas) and Sen. Bernie Sanders (I-Vt.), have gone well beyond their colleagues in calling for a pause in advanced AI development and a ban on superintelligence, a type of theoretical AI that far surpasses humans' cognitive capabilities.

This story originally appeared on POLITICO and is courtesy of the Axel Springer Global Reporters Network, which harnesses the resources of the company's newsrooms to publish ambitious scoops, investigations, interviews, opinion pieces, and analysis. It allows journalists — including those from POLITICO, Business Insider, WELT, BILD, Onet, and Fakt — to collaborate on major stories for an international audience of hundreds of millions across platforms.