This can happen for various reasons. A few pics would be cool, if you're able.
Most likely they just stalled. If a large portion of the jar is colonized, you're probably good.
Are you planning on growing straight from the cake, or using the cake to inoculate your final medium? Either way, it's probably best to cut off the uncolonized portion before proceeding to the next step. Be sure to use a sterilized knife. I have done this in the past with no issues, but I also was not growing straight off the cake.
Be an adult only when necessary.