Webhooks not firing

Hosted • invoiceninja.com | v4/Legacy
1

Hello,

we are using webhooks extensively (no integration like Zapier, we created our own), and since a few hours, webhooks are not firing any more.

To check if the error is on our site, we also tried external services such as https://webhook.site (create a new event subscription e.g “Created Client”, create a new client, no webhook triggering happens).

As our service heavily relies on the webhooks, we are wondering what is hapening here. We tried with two accounts, both seem to have this problem…

Thanks for any hints!

2

Hi,

As mentioned by email, we aren’t aware of any problems. If anyone else is seeing this issue please comment here.

You may want to consider upgrading to v5, the option is available on Settings > Account Management.

3

Thanks for the suggestion. We tried creating a new test account on v5, and can confirm that webhooks are firing there. Although they fire 3 times with 10 seconds apart - but I guess this is a different topic.

Not sure how quickly we can upgrade to v5 as we need to do a lot of customization and testing with the new system - it’s just worrysome that v4 suddenly stopped working.

So just to confirm, upgrading to v5 means having an additional instance, and v4 will continue working? So we could give it a try and upgrade on of our two company accounts and start migrating stuff.

4

By the way, there was a change on root certificates from lets-encrypt yesterday which may affect clients:

Can you by any chance see something related to this in the webhook logs?

5

Hi,

we observed several strange issues with webhooks (gitlab → slack,…) yesterday evening also related to this lets-encrypt change, most of them got resolved after restarts (caching issue)

there is a good summary here which clients are permanently affected based on old versions
Let's Encrypt's Root Certificate is expiring! → Affected Clients

we definitely plan to update to v5 soon but is there a way you can restart the webhook part of invoice ninja platform to see if problem get resolved that way

Thanks
Stefan

6

Upgrading to v5 is a non-destructive migration, you can keep using v4 while you try it out.

@david any thoughts on the Let’s Encrypt issue?

7

Our webhook handler does 3 attempts 10 seconds apart, so it sounds like it isn’t getting a 200 response back.

You may want to inpsect your failed_jobs table for more information on why the jobs are not completing successfully.

8

Thanks for the reply about the 3 attempts, but this is a different topic as I wrote - we were just testing if anything gets to the webhook. In v4, nothing gets sent there at all!

9

@david

we are customers of your hosted offering, so we don’t know how we can check failed_jobs

just to sum up: since yesterday our platform API doesn’t receive webhooks, our assumption is that the hosted Invoice Ninja offering has a problem with the Lets Encrypt root certificate change yesterday

side note @hillel: we are already preparing the upgrade to v5 but this is not as seamless as your wrote as we use your id’s (client, contact, invoice,…) in our system and your change from number to string (especially as you didn’t change 123 to “123”) requires several implementation changes and an explicit migration step on our side

10

@Stefan,

I’ve made a change. Can you please retest now.

11

unfortunately still not working yet

also tried different domain name (again with a LetsEncrypt certificate) but no webhook received there

what I can confirm that webhooks against request bins (like https://requestbin.com/) are working - what we will try is to use another domain with different certificate issuer

12

So to sum up: after @david said he changed something, at least something is being sent to https://requestbin.com, still not to https://webhook.site. Since both our application and webhook.site are using a Let’s Encrypt certificate, the problem may be connected with that.

We managed to get a work-around running by additionally using AWS application loadbalancer, but this is just temporary. Since invoiceninja v4 is using a PHP version with EOL end of 2021, you probably will not spend further efforts to make v4 compatible with the new Let’s Encrypt certificates; so our only way forward is upgrading to v5 (which means quite some effort, since all the client IDs were replaced during migration, plus some other API changes).

I guess we can close this topic for now, thanks for your assistance!