Longbow is the inherent counter to Shinobi; Japanese can counterplay with Horsemen into eventual Yumi.

Mounted Samurai need to do something in early Castle Age because they start falling off once walls are built and you're looking at a large clump of crossbows and Spearmen; especially when you consider the attack speed buff of the English gives them favorability in those later Castle Age engagements.

Japanese can't be overly greedy and go straight to Imperial like English can; you need to do something once you are in Castle Age.

Longbows can hit from either side, so typically against English you have to build an outpost on each side of the gold vein to be safe.

When you dive with a huge trash heap of feudal Yumi, their health pool is so small that TCs and crossbowmen will overkill, so they are effective.