Error during forced Migration v4 -> v5

ecomsilio · January 14, 2021, 11:05am

I have updated to v4.5.29 and v5.0.47-C38 and I have tried the migration again. I now get the following error.

v5

[2021-01-14 10:56:36] production.INFO: Importing invoices  
[2021-01-14 10:56:36] production.INFO: Missing resource key: client_contacts_5  
[2021-01-14 10:56:37] production.ERROR: Missing resource key: client_contacts_5 {"userId":1,"exception":"[object] (Exception(code: 0): Missing resource key: client_contacts_5 at /var/www/invoiceninja/app/Jobs/Util/Import.php:1324)
[stacktrace]

v4

[2020-09-14 20:09:55] production.ERROR: ***InvalidArgumentException*** [0] : /var/www/html/invoiceninja/vendor/laravel/framework/src/Illuminate/Support/Manager.php [Line 91] => Driver [.env] not supported. {"context":"PHP","user_id":0,"account_id":0,"user_name":"","method":"GET","user_agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:79.0) Gecko/20100101 Firefox/79.0","locale":"en","ip":"185.234.218.42","count":1,"is_console":"no","is_api":"no","db_server":"mysql","url":"auth/.env"} []

ecomsilio · January 14, 2021, 1:16pm

Because of the “client_contacts_5” in the error message, I decided to delete client with ID 5 including all connected invoices etc. (Made a snapshot first to be able to go back again).

I removed the migration files from the v5 server and used php artisan migrate:fresh --seedto get a fresh setup. I then went through the migration process again. Still the same error.

I then looked up the contact with ID 5 instead of the client with ID 5 and was able to find it using the REST Call GET /api/v1/contacts/5
The contact was not connected to a client anymore. When I tried to delete the contact using DEL /api/v1/contacts/5 (not sure if this route exists) and I got a 302 Response redirecting me to my InvoiceNinja URL.

I then tried to update an existing client using PUT /api/v1/clients/[Client ID] which gave me a 404 Not Found response saying “record does not exist”.

Payload:

{
  "account_key": "y.....gfy",
  "contacts": [
    {
      "account_key": "y.....gfy",
      "is_owner": true,
      "id": 40,
      "first_name": "John",
      "last_name": "Smith",
      "email": "john@smith.com",
      "contact_key": "b.....prb",
      "updated_at": 1581880342,
      "archived_at": null,
      "is_primary": true,
      "phone": "876543567890",
      "last_login": "",
      "send_invoice": true,
      "custom_value1": "BA9876543567",
      "custom_value2": "12BA"
    },
    {
      "account_key": "y.....gfy",
      "is_owner": true,
      "id": 5,
      "first_name": "Max",
      "last_name": "Mustermann",
      "email": "max@wefef.com",
      "contact_key": "a.....bsh",
      "updated_at": 1581880342,
      "archived_at": null,
      "is_primary": false,
      "phone": "87654345",
      "last_login": "",
      "send_invoice": true,
      "custom_value1": "AB3456789",
      "custom_value2": "AB12344"
    }
  ]
}

I am not sure why this client.contact ID 5 is causing this error.
Anything I can do to get this migration to a success?

Thanks,
Gijs

david · January 14, 2021, 10:57pm

Hi Gijs,

This is a new one for me.

To start debugging this, I would first confirm that the contact was actually migrated to V5. Then we’ll be able to debug further.

Ubi · January 15, 2021, 7:40am

Hi @david, just to confirm that i let it run the migration all night long and at ceratain point it starts. I don’t know how and why it took some time but this morning i wake up with the company correctly imported. I just not get the final email as written like confirmation on V4 migration tool.
So, i chek the logs and i can see :

[2021-01-14 00:01:02] production.INFO: latest version = 5.0.46  
[2021-01-14 00:01:02] production.INFO: updating currencies  
[2021-01-15 03:51:02] production.INFO: Importing account  
[2021-01-15 03:51:02] production.INFO: Importing company  
[2021-01-15 03:51:02] production.INFO: the path /storage/5jynr....K9.png  
[2021-01-15 03:51:02] production.INFO: Importing users  
[2021-01-15 03:51:02] production.INFO: Importing payment_terms  
[2021-01-15 03:51:02] production.INFO: Importing tax_rates  
[2021-01-15 03:51:02] production.INFO: Importing clients  
[2021-01-15 03:51:07] production.INFO: Importing company_gateways  
[2021-01-15 03:51:07] production.INFO: Importing client_gateway_tokens  
[2021-01-15 03:51:07] production.INFO: Importing vendors  
[2021-01-15 03:51:07] production.INFO: Importing projects  
[2021-01-15 03:51:07] production.INFO: Importing products  
[2021-01-15 03:51:07] production.INFO: Importing credits  
[2021-01-15 03:51:07] production.INFO: Importing invoices  
[2021-01-15 03:51:12] production.INFO: Importing recurring_invoices  
[2021-01-15 03:51:12] production.INFO: Importing quotes  
[2021-01-15 03:51:13] production.INFO: Importing payments  
[2021-01-15 03:51:18] production.INFO: Importing expense_categories  
[2021-01-15 03:51:18] production.INFO: Importing task_statuses  
[2021-01-15 03:51:18] production.INFO: in task statuses  
[2021-01-15 03:51:18] production.INFO: finished task statuses  
[2021-01-15 03:51:18] production.INFO: Importing expenses  
[2021-01-15 03:51:18] production.INFO: Importing tasks  
[2021-01-15 03:51:18] production.INFO: Importing documents  
[2021-01-15 03:51:18] production.INFO: Address in mailbox given [] does not comply with RFC 2822, 3.6.2.  
[2021-01-15 03:51:19] production.ERROR: Address in mailbox given [] does not comply with RFC 2822, 3.6.2. {"userId":1,"exception":"[object] (Swift_RfcComplianceException(code: 0): Address in mailbox given [] does not comply with RFC 2822, 3.6.2. at /home/.../public_html/vendor/swiftmailer/swiftmailer/lib/classes/Swift/Mime/Headers/MailboxHeader.php:355)

And this is explaining why i didn’t get the email

SO very happily i start to check how to re-set invoices layout and flavour, but when i press on V5 “customoize & preview” i got a 500 error modal pop up without deatils, just ERROR 500 :

I check the logs again and this is the outcome

[2021-01-15 03:51:19] production.INFO: latest version = 5.0.47  
[2021-01-15 03:51:19] production.INFO: updating currencies  
[2021-01-15 06:41:03] production.ERROR: Downloaded Chromium binary is not executable. Make sure to set correct permissions (0755) {"userId":1,"exception":"[object] (Beganovich\\Snappdf\\Exception\\BinaryNotExecutable(code: 0): Downloaded Chromium binary is not executable. Make sure to set correct permissions (0755) at /home/......mmm//public_html/vendor/beganovich/snappdf/src/Snappdf.php:57)

Thank you

david · January 15, 2021, 8:54am

So this is a easy one, you need to chmod +x the chromium binary

@ben can you help with the location of the binary?

ecomsilio · January 15, 2021, 9:03am

Hi @david

How do I find this contact in v5?
In v4 the contact is not searchable and is not connected anymore to a client. I only can find it via the REST API (GET /api/v1/contacts/5).

How do I check for this contact in v5 via the API? I have found a route for GET /api/v1/clients which contains the contacts and a route for GET /api/v1/companies but that doesn’t contain clients or contacts. I can’t find a route specific for contacts.

Thanks,
Gijs

@Ubi
I had the same error and this helped me:

and

Ubi · January 15, 2021, 9:51am

Hi @ecomsilio, thank you, i tried but still… same error. The difference there is that I am on Debian 10, my library is named just chromium, not chromium.brower and I am not sure i did everything correctly.

Saiyng that, yes david and @ben, where is located the library ? I have this path once installed :

Thank you again guys.

ben · January 15, 2021, 10:21am

If you use vendor/bin/snappdf download, the binary will be located in vendor/beganovich/snappdf/versions. However, it is auto-wired for you so no need for additional actions.

Ubi · January 15, 2021, 10:31am

Thank you @ben ! It worked right after i ran

and no new rows on laravel.log

Seems all ok now, awesome.

ben · January 15, 2021, 10:32am

Great, happy you got it working. It’s weird cause chmod() is part of the download script so it should do it for you.

david · January 15, 2021, 10:34am

@ecomsilio can you look directly in the v4 database for the contact with a id of 5.

ecomsilio · January 15, 2021, 11:01am

Hi @david

Does this help?

david · January 15, 2021, 11:27am

Yes it does.

So it looks like that contact has been deleted in V4, however there are still references to that contact elsewhere in the system, ie attached to an invitation etc etc.

I’ll need to modify the V4 code as we don’t migrate the deleted contacts at the moment.

I’ll need to migrate the deleted contacts so all the other references are able to be linked back in.

david · January 15, 2021, 11:37am

@ecomsilio

I’ve checked in a fix to the master branch of v4, if you can pull in those changes and retry that would be great!.

ecomsilio · January 15, 2021, 11:49am

Thanks for the quick update. I installed IN via curl -L https://download.invoiceninja.com/ -o invoiceninja.zip and not via Github and I normally use the update script that checks the updated version from https://www.invoiceninja.org/

I don’t want to break things and I unsure on how to update using github. Si I will wait for the new release and then try it again.

Thanks,
Gijs

rpbx · January 17, 2021, 11:58pm

Would it break structure if we delete all deleted items from the db directly?

david · January 18, 2021, 2:14am

When you do a force update, the old V5 company is deleted so there is no point in manually editting the DB. In fact, we strong recommend not modifying any data directly in the DB.

ecomsilio · January 18, 2021, 1:08pm

Hi @david

I have just updated tp v4.5.30 and tried the migration again. The target system had not receicev a migration file ever before so it was the first mirgation for this system.

All clients are available now but I am missing invoices and products. The log contains the following error:

[2021-01-18 00:01:03] production.INFO: latest version = 5.0.47  
[2021-01-18 00:01:03] production.INFO: updating currencies  
[2021-01-18 11:47:08] production.INFO: Importing account  
[2021-01-18 11:47:08] production.INFO: Importing company  
[2021-01-18 11:47:08] production.INFO: the path /storage/y33feijmdzo41u51a4dlp61cpphdngfy/vwJ2gSWC8I707Q0x89RWFiI5tmtfgfo5rfiMKKsB.png  
[2021-01-18 11:47:08] production.INFO: Importing users  
[2021-01-18 11:47:08] production.INFO: Importing payment_terms  
[2021-01-18 11:47:08] production.INFO: Importing tax_rates  
[2021-01-18 11:47:08] production.INFO: Importing clients  
[2021-01-18 11:47:09] production.INFO: Importing company_gateways  
[2021-01-18 11:47:09] production.INFO: Importing client_gateway_tokens  
[2021-01-18 11:47:09] production.INFO: Importing vendors  
[2021-01-18 11:47:09] production.INFO: Importing projects  
[2021-01-18 11:47:09] production.INFO: Importing products  
[2021-01-18 11:47:09] production.INFO: Importing credits  
[2021-01-18 11:47:09] production.INFO: Importing invoices  
[2021-01-18 11:47:09] production.INFO: client_contacts  
[2021-01-18 11:47:09] production.INFO: Resource client_contacts not available.  
[2021-01-18 11:47:10] production.ERROR: Resource client_contacts not available. {"userId":1,"exception":"[object] (Exception(code: 0): Resource client_contacts not available. at /var/www/invoiceninja/app/Jobs/Util/Import.php:1320)
[stacktrace]
#0 /var/www/invoiceninja/app/Jobs/Util/Import.php(701): App\\Jobs\\Util\\Import->transformId()
#1 /var/www/invoiceninja/app/Jobs/Util/Import.php(202): App\\Jobs\\Util\\Import->processInvoices()
#2 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): App\\Jobs\\Util\\Import->handle()
#3 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Container/Util.php(40): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#4 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure()
#5 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\\Container\\BoundMethod::callBoundMethod()
#6 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Container/Container.php(610): Illuminate\\Container\\BoundMethod::call()
#7 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(128): Illuminate\\Container\\Container->call()
#8 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(128): Illuminate\\Bus\\Dispatcher->Illuminate\\Bus\\{closure}()
#9 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(103): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#10 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(132): Illuminate\\Pipeline\\Pipeline->then()
#11 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Foundation/Bus/Dispatchable.php(67): Illuminate\\Bus\\Dispatcher->dispatchNow()
#12 /var/www/invoiceninja/app/Jobs/Util/StartMigration.php(111): App\\Jobs\\Util\\Import::dispatchNow()
#13 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): App\\Jobs\\Util\\StartMigration->handle()
#14 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Container/Util.php(40): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#15 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure()
#16 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\\Container\\BoundMethod::callBoundMethod()
#17 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Container/Container.php(610): Illuminate\\Container\\BoundMethod::call()
#18 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(128): Illuminate\\Container\\Container->call()
#19 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(128): Illuminate\\Bus\\Dispatcher->Illuminate\\Bus\\{closure}()
#20 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(103): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#21 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Bus/Dispatcher.php(132): Illuminate\\Pipeline\\Pipeline->then()
#22 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(118): Illuminate\\Bus\\Dispatcher->dispatchNow()
#23 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(128): Illuminate\\Queue\\CallQueuedHandler->Illuminate\\Queue\\{closure}()
#24 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Pipeline/Pipeline.php(103): Illuminate\\Pipeline\\Pipeline->Illuminate\\Pipeline\\{closure}()
#25 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(120): Illuminate\\Pipeline\\Pipeline->then()
#26 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Queue/CallQueuedHandler.php(70): Illuminate\\Queue\\CallQueuedHandler->dispatchThroughMiddleware()
#27 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Queue/Jobs/Job.php(98): Illuminate\\Queue\\CallQueuedHandler->call()
#28 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(406): Illuminate\\Queue\\Jobs\\Job->fire()
#29 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(356): Illuminate\\Queue\\Worker->process()
#30 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Queue/Worker.php(158): Illuminate\\Queue\\Worker->runJob()
#31 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(116): Illuminate\\Queue\\Worker->daemon()
#32 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Queue/Console/WorkCommand.php(100): Illuminate\\Queue\\Console\\WorkCommand->runWorker()
#33 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(36): Illuminate\\Queue\\Console\\WorkCommand->handle()
#34 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Container/Util.php(40): Illuminate\\Container\\BoundMethod::Illuminate\\Container\\{closure}()
#35 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(93): Illuminate\\Container\\Util::unwrapIfClosure()
#36 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Container/BoundMethod.php(37): Illuminate\\Container\\BoundMethod::callBoundMethod()
#37 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Container/Container.php(610): Illuminate\\Container\\BoundMethod::call()
#38 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Console/Command.php(136): Illuminate\\Container\\Container->call()
#39 /var/www/invoiceninja/vendor/symfony/console/Command/Command.php(255): Illuminate\\Console\\Command->execute()
#40 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Console/Command.php(121): Symfony\\Component\\Console\\Command\\Command->run()
#41 /var/www/invoiceninja/vendor/symfony/console/Application.php(971): Illuminate\\Console\\Command->run()
#42 /var/www/invoiceninja/vendor/symfony/console/Application.php(290): Symfony\\Component\\Console\\Application->doRunCommand()
#43 /var/www/invoiceninja/vendor/symfony/console/Application.php(166): Symfony\\Component\\Console\\Application->doRun()
#44 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Console/Application.php(93): Symfony\\Component\\Console\\Application->run()
#45 /var/www/invoiceninja/vendor/laravel/framework/src/Illuminate/Foundation/Console/Kernel.php(129): Illuminate\\Console\\Application->run()
#46 /var/www/invoiceninja/artisan(37): Illuminate\\Foundation\\Console\\Kernel->handle()
#47 {main}
"}

Any idea what to do? I you need more information, please let me know. If you want, we can also send you a sql dump or we can use screensharing.

Thanks,
Gijs

rpbx · January 19, 2021, 1:42am

All we did was delete the deleted invoices, contacts, quotes, and payments from the old v4 prior to migrating. I can however revert back to where all the deleted items are there as 4.30 fixed that. Do you recommend reverting back?

david · January 19, 2021, 2:02am

@rpbx

You could potentially wipe out a bunch of foreign key dependencies doing this so we strong recommend not touching the DB. I would suggest rolling back any manual DB changed.