Yeah that would definitely be a problem. I was talking with one of the big dudes at the Y and I loved what he said about dieting and working out to build muscle mass. "Working out without a proper diet is kind of like going upstream in a row boat, you'll get somewhere but eventually remain the same."
Recipe for success:
Skinless chicken breasts
Fish (I prefer Tilapia myself)
Brown Rice/ or a Sweet potato (After your workout)
Baby Spinach/can substitute Green Beans/ Brocolli / Asparagus
Almonds
Oats in the morning. I'm not talking about the little packets of individually sized brown sugar oatmeal, I'm talking about the regular non sweetened oatmeal that comes in that big Silo like container. They taste awful but are super good for you. (Can add Cinnamon and sweet and low if you wish/Splenda)
Drink: Water (Can use mixes like crystal light/Mio)
This is a small example of a cutting diet, You need to eat around 6 to 8 times a day with these foods. It boosts matabolism, gives you high protein for your muscles, while still being low fat to help you lose weight. I;ve been on the diet before, it's insane! It works wonders, the first week you'll feel like you're stuffing yourself silly, but then your body adapts to all the food, and you feel hungry all the time (Sign of increased metabolism) and you continue your workouts while maintaining muscle mass and losing fat. (Another rule of thumb for a cutting diet is to never mix a carb and a fat with your protein, this way your body will only be able to use one energy source, either the fat or the carb, eating both at the same time will cause your body to use one as energy and store the other.)
There's a lot of rules for this diet and I can go on and on about it, but tbh, this diet can drive you mad, especially since you're restricted to eating such a limited amount of food. Especially if you like Ice Cream

lol ...or pizza...etc (Every 30 days you get a cheat meal though, notice how I said cheat meal, not a cheat day)
Chicken itself can start to taste utterly disgusting if you use this diet too, but some 'Frank's Red hot Buffalo sauce is AMAZING!!"
Bulking diets are much different and a lot easier to do because you get to eat a lot more carbs =D (HUZZAH!)