In a medium bowl, beat the pudding mix with the cold milk for about 2 minutes and then mix in the peanut butter. Let mixture set for a couple of minutes and then gently fold in the Cool Whip.
Layer the bottom of an 8-inch square pan with graham crackers. For me this took 9 graham cracker squares. Break or overlap graham crackers to cover the bottom.
Spread half of the pudding mixture on top of the graham cracker. Thinly slice half of the bananas over the pudding mixture.
Make another layer of graham crackers, top with the remaining pudding mixture, and slice the remaining bananas on top.
Finish the cake with a final layer of graham crackers. Frost with chocolate frosting (see note; recipe below). Cover cake and chill for at least 4 hours or overnight. Slice and garnish with chopped Reese's Peanut Butter Cups if desired. As with most éclair cakes, this isn't the easiest to slice pretty pieces but it tastes great.
For the Chocolate Frosting: Melt the chocolate in a microwave safe bowl in 30 second intervals until melted, stirring in between. Cool slightly before adding the remaining ingredients.
Add the powdered sugar, butter, vanilla, and milk. Beat until mixture is creamy. Add more milk as desired to make thinner if wanted.
Notes
You can either chill the cake and then frost it or frost it and then chill it. The frosting does harden making it a little harder to slice because it's a delicate cake but we didn't mind at all. I frosted mine, then chilled it. Cake: TGWAE Original