With comments marked as spam you can empty the folder (though if there are too many this might be less of load on the server), but there is no easy way to get rid of moderated comments that are spam. And if you get 10,000+, this could take hours to delete manually. Below is a quick trick using an SQL command in phpMyAdmin.
1. Make sure you know the proper database for your WordPress blog if you have several installed. You can find this in databases or the wp-config file in the blog directory (use File Manager for this). Once you know the proper database, click on phpMyAdmin in the cPanel dashboard.
2. Find the database of the site that has all the moderated spam.
3. Look for the wp_comments table in the database, and click on that table.
4. You should now be in the wp_comments table. From there, click on the SQL tab.
DELETE FROM 'wp_comments' WHERE 'comment_approved' = 0
You’ll get a dialog box confirming the command.
After that, you should have gotten rid of every commented that has not been approved.