For economists, "prime-age" refers to the ages between 25-54, which is post-school and pre-retirement for most workers. Didem Tüzemen asks "Why Are Prime-Age Men Vanishing from the Labor Force?" in the Economic Review of the Federal Reserve Bank of Kansas City (First Quarter 2018, pp. 5-28). She begins: "The labor force participation rate for prime-age men (age 25 to 54) in the United States has declined dramatically since the 1960s, but the decline has accelerated more recently. From 1996 to 2016, the share of prime-age men either working or actively looking for work decreased from 91.8 percent to 88.6 percent. In 1996, 4.6 million prime-age men did not participate in the labor force. By 2016, this number had risen to 7.1 million."

As Tüzemen shows, this rise in nonparticipation rates of prime-age males is broad-based. If you break down prime-age male labor force participation by education levels (less than high school, only high school, some college, college or more), the nonparticipation level is higher for those with less education, but it's up in every education category.  If break down the prime-age group into decades (25-34, 35-44, 45-54), then nonpartication is higher in the 45-54 age group, but it's been rising in every age category, too. 

Perhaps more of a clue comes from the employment survey data tiself.  As Tüzemen reports:
Those who report their status as "not in the labor force” also respond to another question, which asks, “what best describes your situation at this time? For example, are you disabled, ill, in school, taking care of house or family, in retirement, or something else?”
The answers to this survey question suggest that between 1996 and 2016, the share of nonparticipating men who give "disability" as an answer has declined, while the share who refer to family responsibilities, taking care of family, and in retirement have all increased.

Of course, these decisions about not being in the labor market are not made in a vacuum, but are presumably also affected by the reality of labor market opportunities. That's just a long way of saying that the rise in nonparticipation can involve both decisions about labor supply and realities of labor demand. Tüzemen makes a case that evolving labor demand is probably more important for rising male nonparticipation than choices about labor supply. In particular, she focuses on the "polarization" of the labor market--the overall pattern in which low-skill workers do OK, because they are providing personal services that are (at least so far) hard to replace with automation or software, and high-skilled workers do OK, because they are well-positioned to make gains from the use of automation and software, but those in the ranks of the middle-skilled can find themselves at risk. 

You can read the article to sort through the details of this argument, but here are a couple of points that caught my eye. One is that while nonparticipation of prime-age males has risen in every education group, the biggest rise is not in the lowest-skill or highest-skill groups, but rather in the middle.

The other point is that many of those currently out of the labor force are not looking to return. When the Great Recession hit from 2007-2009, the share of nonparticipating prime-age men who said they still wanted a job rose sharply. But now, the share of that group that says they want a job has declined back to levels from the early 2000s. This pattern suggests to me that some of the labor market nonparticipants who wanted a job have now returned to the labor market, while others have given up on employment.