The big difference between bacteria and all the other kingdoms is that bacteria are prokaryotic and the others are all eukaryotic.
Note that some members of Kingdom Protista and Kingdom Fungi are unicellular, so that isn't strictly a characteristic of bacteria.
A prokaryote is a single-celled organism that lacks a membrane-bound nucleus (karyon), mitochondria, or any other membrane-bound organelle.
Hope this helps :D