Okay so here's what happened: The battery got completely drained right after I started that PsOne title. When I tried to charge it, the charger couldn't have proper contact (not the first time, because I'm using a EU-to-AU power adapter to use for my european charger) but this time it just happened to be so it was almost having contact, that way making it look like it's flashing every few seconds, and then stopped. Now I happened to try again and it could actually work in a specific position if you're careful to not move it. So it actually turned on this time.
I will eventually look for the multi-purpose charger I mentioned earlier.