In the past, I've used index cards to track my submissions, keeping a set of cards for publications and a set for my work, but for the past few years, I've used a spreadsheet much like the one above. Across the top, I list all of the stories or poems I think are ready to get published. Down the side, I list the publications I'd like to see publish my work. The dates indicate when I mailed the submission, and the second column lists responses. This makes it easy to go down and see which works are circulating at the moment, which journals I haven't submitted to in a while, and which jounals have had something for more than a year. (After a year, you may wish to write and query the editor or withdraw your submission.)
However you choose to do it, you should be able to see at a glance where each story has been sent in order to avoid submitting to a journal more than twice in one year, or re-sending the same story accidentally. This will also help you track simultaneous submissions.