I should have studied more. This is the correct sentence.
The speaker is expressing a regret; now it is too late to change the result of a past action / decision. The speaker is referring to the past; he / she did not study enough ,probably, for a test and he / she could not have passed the exam or could have got a low grade. Now, at the moment of the message, the speaker thinks that if he/ she had studied more, he / she would have got a different result. Therefore, the speaker regrets, now, not having studied enough. This regret over his / her past actions is expressed with " should " + " perfect infinitive".
- I should of study more. ( This option is wrong. The prepositon " of" never follows a modal verb and the verb " study" is in the infinitive).