ISSUE: Can't Import Data from v4 to v5

nickmeche · August 24, 2021, 2:47am

I cannot import my data using the JSON file method. I am trying to export from v4.5.16 to a JSON file to import into the new v5 installation. (v5.3.0-C58)

I have even migrated my self-hosted v4 to hosted v4 and then used the migration tool in the hosted v4. I then downloaded the backup from the v5 hosted on invoicing.co and tried to restore from that backup on my self-hosted v5 but nothing happens.

Any insight on what may be going wrong?

I am running invoiceninja v5.3.0-C58 on cloudron (ubuntu 20.42)

When I check the logs, this is what I am seeing

Aug 23 19:28:03 [2021-08-24 00:28:03][82] Processing: App\Jobs\Company\CompanyImport
Aug 23 19:28:03 [2021-08-24 00:28:03][82] Failed: App\Jobs\Company\CompanyImport

More logs

Aug 23 21:56:00 [2021-08-24T02:56:00+00:00] Execution #139 output:
Aug 23 21:56:00 [2021-08-24T02:56:00+00:00] Running scheduled command: '/usr/bin/php7.4' 'artisan' queue:work > '/dev/null' 2>&1
Aug 23 21:56:10 172.18.0.1 - - [24/Aug/2021:02:56:10 +0000] "GET / HTTP/1.1" 302 1977 "-" "Mozilla (CloudronHealth)"
Aug 23 21:56:16 172.18.0.1 - - [24/Aug/2021:02:56:16 +0000] "POST /api/v1/refresh?current_company=true&updated_at=1629771975&first_load=true HTTP/1.1" 200 155045 "<APP URL>" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"
Aug 23 21:56:18 172.18.0.1 - - [24/Aug/2021:02:56:18 +0000] "POST /api/v1/import_json?&import_settings=true&import_data=true HTTP/1.1" 200 733 "<APP URL>" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"
Aug 23 21:56:20 172.18.0.1 - - [24/Aug/2021:02:56:20 +0000] "GET / HTTP/1.1" 302 1977 "-" "Mozilla (CloudronHealth)"
Aug 23 21:56:30 172.18.0.1 - - [24/Aug/2021:02:56:30 +0000] "GET / HTTP/1.1" 302 1977 "-" "Mozilla (CloudronHealth)"
Aug 23 21:56:40 172.18.0.1 - - [24/Aug/2021:02:56:40 +0000] "GET / HTTP/1.1" 302 1977 "-" "Mozilla (CloudronHealth)"
Aug 23 21:56:50 172.18.0.1 - - [24/Aug/2021:02:56:50 +0000] "GET / HTTP/1.1" 302 1977 "-" "Mozilla (CloudronHealth)"
Aug 23 21:56:58 172.18.0.1 - - [24/Aug/2021:02:56:58 +0000] "POST /api/v1/refresh?current_company=true&updated_at=1629772187&first_load=true HTTP/1.1" 200 155045 "<APP URL>" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"
Aug 23 21:57:00 172.18.0.1 - - [24/Aug/2021:02:57:00 +0000] "GET / HTTP/1.1" 302 1977 "-" "Mozilla (CloudronHealth)"

hillel · August 24, 2021, 3:10am

Hi,

It isn’t supported, the v4/v5 JSON formats are different.

You need to use the migration tool in the latest v4.

nickmeche · August 24, 2021, 3:15am

Thanks for the reply.

Should I be able to migrate from a v5 to a v5?

I used the migration tool in the hosted platform to migrate from v4 to v5 and that worked, but when I tried to export from the v5 hosted and import to the v5 self hosted, this wasn’t working either.

I also tried to update my v4 install, but cannot remember how to do it. I’m not that tech savvy when it comes to linux and running commands. When I tried updating my v4 install, I kept getting “the server cannot handle your request at this time”. It was like that for over 2 hours when I finally decided to revert to my previous snapshot.

hillel · August 24, 2021, 3:21am

Yes, that should work. @david any ideas to debug the error?

You can download the latest v4 here: https://download.invoiceninja.com

https://docs.invoiceninja.com/update.html

nickmeche · August 24, 2021, 3:34am

Thanks, I will try to update again.

david · August 24, 2021, 4:41am

@nickmeche

I have a fix for the company export on hosted going up shortly.

nickmeche · August 24, 2021, 4:52am

Aweseome, thanks.

I still cannot figure out how to update my old install.

I have tried using the auto script, but it tells me that I don’t have a version.txt file, so therefore it thinks the code is up to date.

I also tried coping the invoice-ninja directory to a temp directory and then unzipping the new version into “invoice-ninja” and copying over the .env file from the temp directory, and still kept getting the HTTP error 500. so i just gave up.

I will try to export the hosted copy again in a bit and see if that works.

Thanks for all of your help.

nickmeche · August 24, 2021, 10:04pm

@david Just wondering if you were able to get the fix uploaded by any chance? I’m still waiting to import my data.

Thanks!!

david · August 24, 2021, 10:09pm

@nickmeche

If you aren’t receiving the email, you can inspect /public/backups folder that will hold the .zip file.

nickmeche · August 24, 2021, 10:21pm

@david I get the email, but when I try to import that into v5 it doesn’t do anything.

david · August 24, 2021, 11:10pm

@nickmeche

It sounds like your queues are not running? Have these been configured?

You may want to change QUEUE_CONNECTION=sync in the .env file

The other thing to check is your max_upload parameters for your webserver and PHP are > than the file size you are attempting to import.

nickmeche · August 24, 2021, 11:24pm

@david
Ok thanks, I’ll double check it, but im not sure if I’ll be able to modify anything but the env file. My install is running on cloudron.

nickmeche · August 25, 2021, 1:15am

@david I have updated the QUEUE_CONNECTION as you stated and restarted. Even after restarting I still am not able to import data. Anything else that could be causing this?

nickmeche · August 25, 2021, 2:20am

@david After more investigating, I am also noticing that exporting from hosted v5 and importing to another hosted v5 is not working. So it’s possible that it may not be an issue with my installation, but rather the export/import process on v5?

Just to be clear, these are the steps that I have taken so far:

Exported my self-hosted install from v4.5.16 to the hosted version of v4.
Used the hosted migration tool to migrate the v4 hosted to v5 hosted. (This works fine all data is available in hosted v5).
Used the export tool to export a copy of my data from the hosted v5.
Downloaded the zip file and used the import JSON file option in my self-hosted v5 install.
It’s at this point that I get a message saying “Successfully started import”
Once I get that “Successfully started import” message nothing happens. None of the data is imported.

I again have tried all of the above steps except switched out my self-hosted v5 install for another hosted v5 install under a different account… this still produces the same result – no data is ever imported.

david · August 25, 2021, 2:38am

I think the issue is the job is not starting to perform the import.

If you inspect the jobs table, are there any records in there?

nickmeche · August 25, 2021, 2:40am

@david I’m not exactly sure where to find the jobs table, but I just noticed it imports my company name and logo… but that’s it. No clients, invoices, quotes, payments, etc.

david · August 25, 2021, 4:01am

@nickmeche

Ok that is progress. Can you advise if there are any errors in storage/logs/laravel.log

nickmeche · August 25, 2021, 5:08am

@david This is what I see in the logs right after I try importing the JSON file. For now, I was able to just import some of the data by CSV but that obviously doesn’t migrate my quotes and I have A LOT of quotes that I would like to get moved over without having to manually re-create them.

[2021-08-25 05:04:44] production.INFO: SQLSTATE[23000]: Integrity constraint violation: 1452 Cannot add or update a child row: a foreign key constraint fails (`5ace283fc9196240`.`company_user`, CONSTRAINT `compa
ny_user_company_id_foreign` FOREIGN KEY (`company_id`) REFERENCES `companies` (`id`) ON DELETE CASCADE) (SQL: insert into `company_user` (`user_id`, `company_id`, `account_id`, `permissions`, `notifications`, `s
ettings`, `slack_webhook_url`, `is_owner`, `is_admin`, `is_locked`, `deleted_at`, `created_at`, `updated_at`, `permissions_updated_at`, `ninja_portal_url`) values (GRb4Vo1eBL, 9wdLGjjbjP, 1, , {"email":[]}, ?, ,
 1, 1, 0, ?, 2021-08-24 00:09:56.000000, 2021-08-24 00:12:12.000000, 1629763796, https://invoiceninja.invoicing.co/client/key_login/<key-goes-here>?subscription=true))

david · August 25, 2021, 6:57am

@nickmeche

It looks like a bug, i am checking in version 5.3.1 shortly, this will resolve the issue you are seeing.

nickmeche · August 26, 2021, 1:41am

@david Yes. that seems to have fixed it. Thank you so much!

Just wanted to report another bug that I found, and maybe it’s already been said and being worked on, but I figured I’d just double-check.

When you go to delete a company in v5 and it asks for your password, I accidentally typed the wrong password and it took it anyways. I was able to replicate this multiple times as well.