This article is part of The D.C. Brief, TIME’s politics newsletter. Sign up to get stories like this sent to your inbox.

House Speaker Mike Johnson’s position may be precarious.

From nearly every faction of his broad Republican Party, the congenial Louisiana congressman is confronting furious members, a significant number of whom see the Speaker as a component of the issue. As they prepare for Obamacare costs to surge shortly, and agricultural-state lawmakers attempt to justify support for soybean farmers affected by President Donald Trump’s tariffs, Johnson is the Republican leader in the best position to make House members feel their work has purpose. However, after serving as Trump’s Speaker for under a year, few consider him capable of the task. Even the most pro-Trump legislators are beginning to express skepticism, grumbling that Johnson has silenced their input and diminished the chamber’s authority as a result. And it’s best not to inquire about the feelings of the members in Johnson’s caucus unless you are prepared for a lengthy—and justified—complaint session.

“Women will never be taken seriously until leadership decides to take us seriously, and I’m no longer holding my breath,” Rep. Nancy Mace wrote in a New York Times opinion piece published Monday titled “What’s the Point of Congress?” Mace, who has chosen not to seek re-election and is instead campaigning for Governor of South Carolina, offered tepid approval of Johnson as “better than his predecessor” but was clearly pessimistic about her role, and his capacity to improve it.

As they approach what appears to be a period of legislative inactivity, House Republicans are having difficulty articulating their achievements despite their party’s complete control. A mere 46 bills have been signed into law by Trump, setting this Congress on a path to be labeled the least productive in a generation. (For comparison, President Joe Biden signed 274 bills during his last two years in office, and 365 in his first two, as reported by GovTrack.)

For Mike Johnson—or any Republican candidate on next year’s ballot—it becomes increasingly difficult to request votes when they have so little to demonstrate for their two-year term. While they assisted Trump in passing his major spending package, the majority of its benefits will not be evident before Election Day, and the more difficult measures to finance it will only take effect after voting concludes.

Stated simply: Johnson is leading the House with a fragile hold on power. The contemporary GOP is a strange and disorderly coalition of conflicting agendas, where proponents of hawkish foreign policy coexist with America First isolationists, and free-market advocates share space with protectionist tariff supporters. Trump’s blend of intimidation and support has largely contained dissent, but that control is weakening weekly.

Republicans required significant time to agree on Johnson in 2023, and even then only after persuasion and prodding from Trump. Should Trump withdraw his support, the Johnson era could conclude rapidly—a prospect Johnson might not mind, as he recently confided to a podcaster that the role was a major burden, likening it to constant crisis management.

However, Johnson might be saved by the fact that no other Republican is waiting to replace him, and few appear eager for the ungrateful task of catering to Trump’s demands. Johnson is nearing a term three times longer than that of his predecessor, the ousted Speaker Kevin McCarthy, who served 269 days. This is small comfort for Johnson, who has spent the initial ten months of Trump’s Washington comeback under attack from all directions.

Supporters of Johnson point out that he has demonstrated strong fundraising abilities; in the last cycle alone, he and his allied political committee raised $23 million to maintain a slim majority. He began this year by collecting $32 million in the first quarter, followed by $19 million in the second, and $13 million in the third. Associated super PACs and advocacy groups are also generating substantial funds, buoyed by the possibility of full Republican control of Congress and the presidency. If a primary responsibility of the Speaker is to raise money and maintain party satisfaction, he has at least met one of those criteria.

Nevertheless, financial success cannot obscure numerical realities. Among the 39 House members who have decided to retire, 23 are Republicans. As Puck noted, this places the current wave of congressional departures on course to exceed the previous one, which saw 21 Republicans among the 45 members who left. In private conversations, some estimate an additional two dozen representatives are prepared to leave Washington, particularly given the Democrats’ stronger-than-expected performance in special and state-level elections throughout the year.

This does not mean all these Republican seats will shift to Democratic control. One prominent member who is resigning a year early, Rep. Marjorie Taylor Greene of Georgia, represents a district with a built-in Republican advantage of approximately 19 points. In a 60 Minutes interview broadcast Sunday, Greene declined to identify as MAGA and portrayed her former ally Trump as a directionless figure. Meanwhile, Rep. Don Bacon, a disgruntled Republican and the sole announced retiree from a competitive district, has stated he will complete his term.

Even so, this calculation provides Johnson—and Trump—with a margin of error of just one seat, at least until a special election is held in Georgia to fill Greene’s vacancy. (A date for this election has not been scheduled.)

But even if there were motivation for the House to accomplish something significant, there remains the reality that Mace highlighted in her essay: “The obstacles to achieving almost anything are enough to make any member who came to Washington with noble intentions ask: Why am I even here?”

This is the reason many have chosen to leave. And for those who remain, there is no assurance that Johnson will continue as their leader. As Rep. Elise Stefanik, a key Trump ally, told The Wall Street Journal last week, the prevailing opinion is that Johnson has shown he is not fit for the position: “He certainly wouldn’t have the votes to be Speaker if there was a roll-call vote tomorrow.”

Make sense of what matters in Washington. .