When the bite gets tough try lightening up. Get rid of any terminal tackle and weight. Going to just a hook and worm may do the trick.
Cast out and let the worm freefall to the bottom. The action of the worm falling is usually when the fish will take the bait. You must watch your line for the slightly movement and be ready to set the hook.
Adding a bobber up the line a ways is ok if you need the extra weight to make an effective cast but go as small as you can.