I think the frustration lies elsewhere than harbaugh.
Im more frustrated in misses in high draft picks that have been pretty much complete misses.
Elam,upshaw,pierce,Gilmore (becoming one but too early to tell yet), brooks, Arthur brown, maxx Williams, Carl Davis
All drafted in rounds 1-3 from 2012-2015 and have given little to no production comparable to draft position or "what they were supposed to be"
(maybe upshaw shouldn't be here but I don't think he lived up to 35 pick and his hype)
I also believe the ray rice debacle hurt the team badly as well with his contract fallout beyond all the media fallout