Why do you think most harem anime and manga have lame male protagonists that would be considered losers and pathetic by most people?

Most anime viewers in Japan (and anime and manga producers care foremost about their domestic audience because foreign audience still make up only a very small percentage of their profits despite the anime and manga boom across the world) are the so called otaku and hikikomori, shy teens with socializing issues, so they relate to characters that act like they would. It’s sad to say but most Japanese fans of manga and anime would behave exactly like they do.

Not only does Japan have the lowest sexual frequency in the entire world, it is also the only country where the percentage of people who are not happy with their sex life is higher than that of those who are. It's a mix of "Japanese people are simply just too shy to make moves" (which reflects the personalities of anime and manga characters) and "Japanese working culture is oppressive enough to leave little room for literally anything else".

In that sense it was a sort of cynical commentary on the genre, deconstructing many anime tropes but also showing how flawed it is to assume that someone with problems like Shinji has (depression, social anxiety, abandonment issue, paranoia, etc.) can be "fixed" by finding the right partner. But that subtlety was lost on the imitators, and we can trace a lot of the beta-males to other shows trying to catch the perceived market for Eva-like shows. ("Wow, Eva is so successful, maybe we should make our character more like Shinji!").

What's your love story?

Writing a character whose main trait is just being awkward with the opposite sex is easier than writing a more realistic character that has dynamic relationships and personality.

A few reasons.

The author might just be lazy, so he copies a cliche.

Hello,hope y'all doin good, i came to Quora to share my strange story , a very weird one , a story when luck smiled at me ,maybe u will enjoy it , let's begin,have fun... A year ago ,I was a real porn addicted(btw I was 18) ,but never had sex before, I don't have a gf I didn't try to find one even ,always thinking to go to find a sex worker but then I just don't , everyday watching different bodies getting fucked and everyday enjoying. One day, I was watching porn, a big ass lady with big boobs ,just after seeing her the image of my female cousin poped in my mind, (let's introduce her : she's 35 years old , very big ass , nice boobs ,not very big but nice,always wearing tight clothes , she's divorced ) and I thought of me fucking her ,I never had sexual desires for her but now I do days went by and when I met her I was so horny ,I couldn't stay with the family cz my penis was clearly erected , I realized this is my first time I get horny for one of my family ,it not illegal in my country.well to make a long story short( if u want details just text me I will tell u 😊),I decided to give her signs that I want to fuck her,finally I decided to have sex and with my cousin , I thought it is the best beggining for me, i started touching her when I came across her in a narrow place , make her feel my hard cock when we hug , I thought it will hard and I will be ashamed but no , I felt nothing and she said nothing , probably she thought it was by mistake,anyways, I decided then to talk with her about sex, waited for her to be alone in a room and talk with her, I confessed everything about me watching porn and addicted..etc,she said it's normal and u are growing up and u must have sex,well at that time I was like whaaat????? Well I didn't control myself and asked her for sex ( horny like I Ve never been before) she said that she will think Abt it ,2 weeka went by then she called me ,telling that she reserved a room in a hotel and we meet tonight ,we met,and bruuhh, sex is great , I mean, I had to find a pirstitue ,what I was waiting for to have such a feeling ????, I will never forget that night, I started kissing her she was kissing hard ,she misses sex so bad , she sucked my dick and swallowed my semen ,I felt I'm in a dream , then when fucked ,her ass was very big and the anus was open ,didn't struggle to get my hard cock inside it , she was obviously missing sex , she was shouting ,fuck me yh fuck me , I go fast after every word until I cum , we did that 3 times , then we went to her pussy , using condoms I fucked her so hard the moans were higher , everything was perfect ,in the end I asked her to lick her body , licked pussy ,ass, boobs,then she sucked my cock until we sleeped ,all I know that she was dirty ,well before even having sex with her I knew she is an open minded woman , and a woman that looks that she donesnt know anything , but she knows everything, but never expected having sex with her ,well she was horny and that helped...but no one of us regretted that sex ever.. We still have sex from time to time ,and I started having sex with sex workers , joining threesomes..etc If u want pics of her text me.

I can't really give you a straight anwser, because it is highly dependent on the anime and character. But if i had to guess, it's prominency is because of a mix of number 1, 4 and 5.

Yeah, why do most anime and manga protagonists, especially when it comes to romance, are spineless and whiny, this guy probably being the quintessential example:

It might be part of the character, so he can develop later on in the series (but it’s rare sadly).

The Hemi V-8 Is Back: ‘We Screwed Up,' Says Ram CEO - Motor1.com

Shinji Ikari himself was a special case, because he is a deconstruction of the harem anime protagonist archetype. He can't form a relationship with anyone, even if he wants to, because of how broken he is, how broken the "waifus" he meets are, who would normally be willing sex objects for his fantasies, and how flawed the world is in general. He starts with Rei. He cannot have a real relationship with her. She's not a kuudere, in a cutesy harem anime sense, but a soulless clone with many battle scars. And she's a clone of his mother, which puts him off. Then there's Asuka. She acts like a harem anime tsundere. But she has no sweeter side. She just acts superior and haughty because of her psychological scars. And she doesn’t become cuter and kinder as the story goes, she only gets crazier. Then there's Misato. Normally in anime, an older woman who drinks and acts sexy is hot, a teen boy's fantasy. And living with such a woman is a sexual fantasy. But Misato spirals into depression after losing Kaji and seeks to uncover the truth behind the SEELE conspiracy he was trying to unveil when he was murdered. No time to be the cute, fun 'bottle fairy' she was introduced as. This show is saying people exist for their own reasons, not for your (the viewer's) sexual fantasies. Then Kaworu's story is a breaking down of the yaoi idea that a boy who has no success with women will be uplifted by a Manic Pixie Dream Man. That realizing he was gay was the reason he had no luck.

But also this guy:

The author might be drawing inspiration from his own experience with the opposite sex. See number 1 and number 2.

JoJo Siwa Says She Is 'Head Over Heels' as She Confirms Relationship with Chris Hughes: 'It's Not Platonic Anymore' - People.com

Also, like you mentioned this problem is mainly associated with various harem shows. And the reason the main character behaves like that is quite simple actually (at least for me). He cannot choose or engage with any of the girls because if he were to pick one, then she would essentially "win" in a fandom, and the merchandise of the other characters would go down. So studios in these shows make sure that every character gets equal screen time and treatment etc. to ensure that they always sell.

I mean hell even in the Sword Art Online series, despite the protagonist being canonically and openly in love with one girl and one girl alone, every single arc of the story has him paired up and getting in touch with a new female character so that the fandom can ship them and fall in love with a new “waifu” (and buy her merchandise).