This question is a bit controversial because many people have different opinions on it.
In my opinion, I would say the most difficult to perform would be hajj - pilgrimage. This is due to the fact that in order to perform hajj you have to book a flight, hotel, etc. to the Islamic sacred city - Mecca, Saudi Arabia. Some people would be unable to pay for all the expenses of the journey; thus, I believe some would not be able to fulfill this pillar of Islam.
Hope my answer helps :)