You’d need to restore the invoice, change the invoice number to a value you won’t use and then delete it.
We don’t permanently delete data in order to ensure referential integrity, we don’t allow duplicating deleted invoice numbers as it could cause a problem if the deleted invoice is later restored.
Dropping the Invoices table would likely do “Very Bad Things” to the overall system. There’s a lot of functionality that relies on the table to exist, even if empty.