Deserialization Error and Migration to v5 woes

Edited to add last paragraph…

Self-hosted v4.5.19 on Ubuntu 16.04 with PHP 7.3 7.2

I’m not sure if these 2 things are related or not, but for several weeks now the iOS App has been having issues. These are the symptoms:

  • Dashboard Overview screen just shows spinning circle (no data)
  • Dashboard Activity correctly displays
  • Data entered into any of the fields of a Quote, Invoice, Product, etc. is not shown on screen, but if I just pretend like I can “see” it, and enter all fields then save, it does create correct invoice and PDF shows all data. Example: click calendar icon to select Invoice/Due date and hit ok, field appears blank, even though it actually is saving my entries as evidenced by viewing the PDF.
  • Overview screen of any client’s “Paid to Date” and “Balance Due” is blank
  • Clicking Refresh Data in Settings produces the following Deserialization error:

Everything appears to be working from a browser on computer, with the exception of Migration, which brings me to next issue.

I installed a fresh copy of v5 on a separate VPS instance running Ubuntu 20.04 and PHP 7.4. I created a DB and made sure to use the same email to register account. I setup SSL cert, changing my URL from https://invoice.mydomain.com to https://myinvoice.mydomain.com, using the new Nginx template. Upon reaching the Setup screen I entered all information, receiving a “Success” message after clicking each “Test” button (ie. Database and SMTP).

Then I went back to v4, logged in and went to start Migration. Proceeded through steps, which seemed pretty normal, except for the screen which asks which company you are migrating because it only gave me what appears to be a random string of characters rather than actual company name, but assume this is just the ID for company “under the hood” so to speak.

Ultimately, the migration ends with “Whoops, looks like something went wrong” error (below):

1. So, are these two issues related?
2. How can I resolve them?

EDIT: So while troubleshooting these errors I found that some of the files in the invoiceninja directory of my v4 installation, which is at /var/www/html/invoiceninja had incorrect permissions (ownership). Some were owned by root, others by www-data, which is the correct owner (I think). Thus, I ran the following commands and tried to migrate again, though it still resulted in the same “Whoops, looks like something went wrong error”:

sudo chown -R www-data:www-data /var/www/html/invoiceninja
sudo chmod -R 775 /var/www/html/invoiceninja/storage
sudo chmod -R g+s /var/www/html/invoiceninja/storage

Lastly, this error is repeated several times in the laravel-error.log file:

[2020-10-27 22:01:52] production.ERROR: ***ErrorException*** [0] : /var/www/html/invoiceninja/app/Traits/GenerateMigrationResources.php [Line 770] => Undefined offset: 1  {"context":"PHP","user_id":1,"account_id":1,"user_name":"Josh Madrone","method":"POST","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15","locale":"en","ip":"47.208.10.222","count":3,"is_console":"no","is_api":"no","db_server":"mysql","url":"migration/companies"} []

Any help would be much appreciated!!!

Hi,

For the v4 mobile app have you tried logging out and then back in?

Someone else will reply to help with the v5 migration.

@jmadrone have you pulled in the latest changes from the github master branch for your V4 installation?

No, I haven’t. I’'m pretty sure I did the originall install with the .zip file. Would I just do a git pull?
Any other commands afterwards?

I noticed that iOS app on my iPad Pro was working fine FYI. So I had tried to logout and back in, but this time I logged out and logged back in without populating the “secret” field, and everything seems to be back to normal so thank you. Still need help with the v4 -> v5 Migration though.

a git pull should just work!

@david, here is the output from a git pull:

ubuntu@invoice:/var/www/html/invoiceninja$ sudo git pull
remote: Enumerating objects: 4976, done.
remote: Counting objects: 100% (4976/4976), done.
remote: Compressing objects: 100% (742/742), done.
remote: Total 43074 (delta 4514), reused 4459 (delta 4206), pack-reused 38098
Receiving objects: 100% (43074/43074), 171.87 MiB | 9.24 MiB/s, done.
Resolving deltas: 100% (33832/33832), completed with 754 local objects.
From https://github.com/invoiceninja/invoiceninja
 + 7ef523c...0962e0e master     -> origin/master  (forced update)
 + f908b34...9acfb40 develop    -> origin/develop  (forced update)
 * [new branch]      v5-develop -> origin/v5-develop
 * [new branch]      v5-stable  -> origin/v5-stable
 * [new tag]         v4.5.10    -> v4.5.10
 * [new tag]         v4.5.11    -> v4.5.11
 * [new tag]         v4.5.12    -> v4.5.12
 * [new tag]         v4.5.13    -> v4.5.13
 * [new tag]         v4.5.14    -> v4.5.14
 * [new tag]         v4.5.15    -> v4.5.15
 * [new tag]         v4.5.16    -> v4.5.16
 * [new tag]         v4.5.17    -> v4.5.17
 * [new tag]         v4.5.18    -> v4.5.18
 * [new tag]         v4.5.19    -> v4.5.19
 * [new tag]         v5.0       -> v5.0
 * [new tag]         v5.0-release -> v5.0-release
 * [new tag]         v5.0.1     -> v5.0.1
 * [new tag]         v5.0.1-release -> v5.0.1-release
 * [new tag]         v5.0.10    -> v5.0.10
 * [new tag]         v5.0.10-release -> v5.0.10-release
 * [new tag]         v5.0.11    -> v5.0.11
 * [new tag]         v5.0.12    -> v5.0.12
 * [new tag]         v5.0.12-release -> v5.0.12-release
 * [new tag]         v5.0.13    -> v5.0.13
 * [new tag]         v5.0.13-release -> v5.0.13-release
 * [new tag]         v5.0.16    -> v5.0.16
 * [new tag]         v5.0.16-release -> v5.0.16-release
 * [new tag]         v5.0.17    -> v5.0.17
 * [new tag]         v5.0.17-release -> v5.0.17-release
 * [new tag]         v5.0.18    -> v5.0.18
 * [new tag]         v5.0.18-release -> v5.0.18-release
 * [new tag]         v5.0.19    -> v5.0.19
 * [new tag]         v5.0.19-release -> v5.0.19-release
 * [new tag]         v5.0.2     -> v5.0.2
 * [new tag]         v5.0.2-release -> v5.0.2-release
 * [new tag]         v5.0.20    -> v5.0.20
 * [new tag]         v5.0.20-release -> v5.0.20-release
 * [new tag]         v5.0.21    -> v5.0.21
 * [new tag]         v5.0.21-release -> v5.0.21-release
 * [new tag]         v5.0.22    -> v5.0.22
 * [new tag]         v5.0.3     -> v5.0.3
 * [new tag]         v5.0.3-release -> v5.0.3-release
 * [new tag]         v5.0.4     -> v5.0.4
 * [new tag]         v5.0.4-release -> v5.0.4-release
 * [new tag]         v5.0.5     -> v5.0.5
 * [new tag]         v5.0.5-release -> v5.0.5-release
 * [new tag]         v5.0.6     -> v5.0.6
 * [new tag]         v5.0.6-release -> v5.0.6-release
 * [new tag]         v5.0.7     -> v5.0.7
 * [new tag]         v5.0.7-release -> v5.0.7-release
 * [new tag]         v5.0.8     -> v5.0.8
 * [new tag]         v5.0.8-release -> v5.0.8-release
 * [new tag]         v5.0.9     -> v5.0.9
 * [new tag]         v5.0.9-release -> v5.0.9-release
error: Your local changes to the following files would be overwritten by merge:
	.gitignore
	.travis.yml
	README.md
	app/Console/Commands/CalculatePayouts.php
	app/Console/Commands/ChargeRenewalInvoices.php
	app/Console/Commands/CheckData.php
	app/Console/Commands/CreateLuisData.php
	app/Console/Commands/CreateTestData.php
	app/Console/Commands/PruneData.php
	app/Console/Commands/RemoveOrphanedDocuments.php
	app/Console/Commands/ResetData.php
	app/Console/Commands/SendRecurringInvoices.php
	app/Console/Commands/SendReminders.php
	app/Console/Commands/SendRenewalInvoices.php
	app/Console/Commands/TestOFX.php
	app/Console/Commands/UpdateKey.php
	app/Console/Commands/stubs/command.stub
	app/Console/Kernel.php
	app/Constants.php
	app/Http/Controllers/AccountApiController.php
	app/Http/Controllers/AppController.php
	app/Http/Controllers/Auth/AuthController.php
	app/Http/Controllers/CalendarController.php
	app/Http/Controllers/ClientController.php
	app/Http/Controllers/ClientPortalController.php
	app/Http/Controllers/ClientPortalProposalController.php
	app/Http/Controllers/DocumentAPIController.php
	app/Http/Controllers/InvoiceApiController.php
	app/Http/Controllers/InvoiceController.php
	app/Http/Controllers/OnlinePaymentController.php
	app/Http/Controllers/ProjectController.php
	app/Http/Controllers/ProposalController.php
	app/Http/Controllers/ReportController.php
	app/Http/Controllers/TaskApiController.php
	app/Http/Controllers/TaskKanbanController.php
	app/Http/Kernel.php
	app/Http/Middleware/Authenticate.php
	app/Http/Middleware/StartupCheck.php
	app/Http/Requests/CreateOnlinePaymentRequest.php
	app/Http/Requests/RegisterRequest.php
	app/Http/Requests/Request.php
	app/Jobs/Client/GenerateStatementData.php
	app/Jobs/ConvertProposalToPdf.php
	app/Jobs/LoadPostmarkHistory.php
	app/Libraries/Utils.php
	app/Listeners/SubscriptionListener.php
	app/Models/Account.php
	app/Models/AccountGatewaySettings.php
	app/Models/Activity.php
	app/Models/Company.php
	app/Models/EntityModel.php
	app/Models/Gateway.php
	app/Models/Invoice.php
	app/Models/InvoiceItem.php
	app/Models/PaymentMethod.php
	app/Models/Task.php
	app/Models/Traits/HasLogo.php
	app/Models/Traits/HasRecurrence.php
	app/Models/User.php
	app/Ninja/Datatables/ActivityDatatable.php
	app/Ninja/Datatables/RecurringInvoiceDatatable.php
	app/Ninja/Import/CSV/PaymentTransformer.php
	app/Ninja/Mailers/ContactMailer.php
	app/Ninja/PaymentDrivers/BasePaymentDriver.php
	app/Ninja/PaymentDrivers/CheckoutComPaymentDriver.php
	app/Ninja/PaymentDrivers/GoCardlessV2RedirectPaymentDriver.php
	app/Ninja/PaymentDrivers/PayPalExpressPaymentDriver.php
	app/Ninja/PaymentDrivers/StripePaymentDriver.php
	app/Ninja/Presenters/ClientPresenter.php
	app/Ninja/Reports/ExpenseReport.php
	app/Ninja/Reports/InvoiceReport.php
	app/Ninja/Reports/QuoteReport.php
	app/Ninja/Repositories/AccountRepository.php
	app/Ninja/Repositories/ActivityRepository.php
	app/Ninja/Repositories/ClientRepository.php
	app/Ninja/Repositories/DashboardRepository.php
	app/Ninja/Repositories/InvoiceRepository.php
	app/Ninja/Repositories/RecurringExpenseRepository.php
	app/Ninja/Transformers/DocumentTransformer.php
	app/Ninja/Transformers/ExpenseTransformer.php
	app/Ninja/Transformers/InvoiceTransformer.php
	app/Ninja/Transformers/TaskTransformer.php
	app/Ninja/Transformers/UserAccountTransformer.php
	app/Services/InvoiceService.php
	app/Services/PaymentService.php
	bower.json
	composer.json
	composer.lock
	config/services.php
	database/seeds/ConstantsSeeder.php
	database/seeds/CountriesSeeder.php
	database/seeds/CurrenciesSeeder.php
	database/seeds/LanguageSeeder.php
	database/seeds/PaymentLibrariesSeeder.php
	database/seeds/PaymentTypesSeeder.php
	docs/conf.py
	docs/configure.rst
	docs/index.rst
	docs/install.rst
	docs/mobile_apps.rst
	docs/update.rst
	public/built.js
	public/built.js.map
	public/css/built.css
	public/css/built.public.css
	public/css/fullcalendar.css
	public/css/grapesjs.css
	public/css/grapesjs.css.map
	public/js/Chart.min.js
	public/js/Chart.min.js.map
	public/js/card.min.js
	public/js/card.min.js.map
	public/js/grapes.min.js
	public/js/grapes.min.js.map
	public/js/grapesjs-blocks-basic.min.js
	public/js/grapesjs-preset-newsletter.min.js
	public/js/grapesjs.min.js
	public/js/grapesjs.min.js.map
	public/js/select2.min.js
	public/js/select2.min.js.map
	public/js/tablesorter.min.js
	public/js/tablesorter.min.js.map
	public/pdf.built.js
	public/pdf.built.js.map
	resources/assets/js/pdf.pdfmake.js
	resources/assets/js/script.js
	resources/lang/ca/texts.php
	resources/lang/cs/texts.php
	resources/lang/da/texts.php
	resources/lang/de/texts.php
	resources/lang/el/texts.php
	resources/lang/en/texts.php
	resources/lang/en_AU/texts.php
	resources/lang/en_GB/texts.php
	resources/lang/es/texts.php
	resources/lang/es_ES/texts.php
	resources/lang/fi/texts.php
	resources/lang/fr/texts.php
	resources/lang/fr_CA/texts.php
	resources/lang/hr/texts.php
	resources/lang/it/texts.php
	resources/lang/ja/texts.php
	resources/lang/lt/texts.php
	resources/lang/mk_MK/texts.php
	resources/lang/nb_NO/texts.php
	resources/lang/nl/texts.php
	resources/lang/pl/texts.php
	resources/lang/pt_BR/texts.php
	resources/lang/pt_PT/texts.php
	resources/lang/ro/texts.php
	resources/lang/sl/texts.php
	resources/lang/sq/texts.php
	resources/lang/sr_RS/texts.php
	resources/lang/sv/texts.php
	resources/lang/th/texts.php
	resources/lang/tr_TR/texts.php
	resources/lang/zh_TW/texts.php
	resources/views/accounts/account_gateway.blade.php
	resources/views/accounts/management.blade.php
	resources/views/auth/login.blade.php
	resources/views/clients/edit.blade.php
	resources/views/clients/show.blade.php
	resources/views/dashboard.blade.php
	resources/views/emails/design2_html.blade.php
	resources/views/emails/design3_html.blade.php
	resources/views/export/invoices.blade.php
	resources/views/header.blade.php
	resources/views/invoices/edit.blade.php
	resources/views/invoices/view_borderless.blade.php
	resources/views/list.blade.php
	resources/views/partials/sign_up.blade.php
	resources/views/payments/checkoutcom/partial.blade.php
	resources/views/payments/credit_card.blade.php
	resources/views/payments/edit.blade.php
	resources/views/proposals/edit.blade.php
	resources/views/reports/report_builder.blade.php
	resources/views/tasks/kanban.blade.php
	resources/views/tasks/time_tracker.blade.php
	resources/views/users/account_management.blade.php
	resources/views/vendors/edit.blade.php
	routes/api.php
	routes/web.php
Please, commit your changes or stash them before you can merge.
error: The following untracked working tree files would be overwritten by merge:
	app/Console/Commands/ExportMigrations.php
	app/Console/Commands/MobileLocalization.php
	app/Http/Controllers/Migration/StepsController.php
	app/Http/Middleware/EligibleForMigration.php
	app/Http/Requests/MigrationAuthRequest.php
	app/Http/Requests/MigrationCompaniesRequest.php
	app/Http/Requests/MigrationEndpointRequest.php
	app/Http/Requests/MigrationTypeRequest.php
	app/Ninja/PaymentDrivers/PaymentActionRequiredException.php
	app/Ninja/PaymentDrivers/PaymentExpressPxPayPaymentDriver.php
	app/Ninja/PaymentDrivers/PaymentExpressPxPostPaymentDriver.php
	app/Services/Migration/AuthService.php
	app/Services/Migration/CompanyService.php
	app/Services/Migration/CompleteService.php
	app/Traits/GenerateMigrationResources.php
	database/migrations/2019_08_27_074711_add_app_store_order_id.php
	docs/digital_ocean.rst
	public/images/social/signin/btn_facebook_signin.png
	public/images/social/signin/btn_github_signin.png
	public/images/social/signin/btn_google_signin_dark_normal_web@2x.png
	public/images/social/signin/btn_linkedin_signin.png
	resources/lang/bg/pagination.php
	resources/lang/bg/passwords.php
	resources/lang/bg/texts.php
	resources/lang/bg/validation.php
	resources/views/migration/auth.blade.php
	resources/views/migration/companies.blade.php
	resources/views/migration/completed.blade.php
	resources/views/migration/download.blade.php
	resources/views/migration/endpoint.blade.php
	resources/views/migration/import.blade.php
	resources/views/migration/includes/errors.blade.php
	resources/views/migration/start.blade.php
	resources/views/payments/stripe/step2.blade.php
	storage/framework/.DS_Store
	storage/migrations/.gitignore
Please move or remove them before you can merge.
Aborting

Just to confirm, this git pull was done on the v4 branch directory, yes?

git status should show on branch master

Also important to confirm you are at least on the latest version v4.5.19 prior to doing the pull.

@david, yes the git pull was performed on my v4 installation directory, which is on a separate AWS instance than my v5. This v4 is 4.5.19, with PHP 7.2, on Ubuntu 16.04. Here is the output of git status, which does show on branch master:

ubuntu@invoice:/var/www/html/invoiceninja$ git status
On branch master
Your branch and 'origin/master' have diverged,
and have 90 and 242 different commits each, respectively.
  (use "git pull" to merge the remote branch into yours)
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

...

There were too many lines of code to put in my reply but that is what’s at the top of the output from git pull.

If you are brave, i’d suggest the following.

  1. Take a backup of your AWS instance.
  2. git fetch origin
  3. git reset --hard origin/master

This will bring your instance up to the latest and should work fine. If it all goes pear shaped, just restore your backup.

@david I took a snapshot, then ran the commands you suggested. I expected to see whizzing lines of code but this was the only output, so I hope that’s expected?

ubuntu@invoice:~$ sudo -i
root@invoice:~# cd /var/www/html/invoiceninja
root@invoice:/var/www/html/invoiceninja# git fetch origin
root@invoice:/var/www/html/invoiceninja# git reset --hard origin/master
HEAD is now at 0962e0e Update lang file

Should I try to migrate again now? Do I need to run any other commands first?

EDIT: Migration resulted in the same “Whoops, something went wrong” message. Here is the last line from laravel-error.log:

[2020-10-30 03:33:08] production.ERROR: ***ErrorException*** [0] : /var/www/html/invoiceninja/app/Traits/GenerateMigrationResources.php [Line 777] => Undefined offset: 1  {"context":"PHP","user_id":1,"account_id":1,"user_name":"Josh Madrone","method":"POST","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Safari/605.1.15","locale":"en","ip":"47.208.10.222","count":2,"is_console":"no","is_api":"no","db_server":"mysql","url":"migration/companies"} []

One change since I first posted about this error is that I previously had 2 companies setup and have since deleted one of them, leaving only the primary/main company. I’m sure I’m using same username/password. What is the “Force Migration” option? It says it will result in wiping of all company data…

I’m a stumped on this one.

The error is failing when attempting to migrate your quotes over and is failing on the quote date for some reason. What is bizarre is the error indicates a reference to a nonexistent key of an array when it shouldn’t even be an array.

What would be helpful would be if you could insert the following at line 764 in that file.

info(print_r($quotes,1));

Don’t post the contents here. If you are on our Slack channel https://slack.invoiceninja.com you can send me a DM (David Bomba) and we can figure out what is tripping things up.