Problems when preparing migration from v4.5.39 to v5.2.3-C51

Hi there,
I’m running 4.5.39 on a Debian 10 machine. I’m trying to start a migration to v5. Since the migration doesn’t want to succeed, I tried to find out the reason for this. So I run php artisan ninja:check-data to find out where the problem lies.

I see about 100 errors like this:
2021-06-22 08:21:29 === Company: 1 Account:1 Client:17 Balance:394.05 Actual Balance:393.32 ===

as well as 35 versions of
2021-06-22 08:21:29 sr_RS is invalid: Cross account migration is not allowed. Please read more about it here: <a href="https://invoiceninja.github.io/docs/migration/#troubleshooting">https://invoiceninja.github.io/docs/migration/#troubleshooting</a>

I saw in another post that the [lang] is invalid errors can be ignored, but it seems that this is also preventing the --fix option, because I try to fix the incorrect balance/activities entries with --fix=true but the script doesn’t fix anything.

So, how can I proceed here? Well, besides manually fixing the balance values in the database …

P.S. The wrong balance values are a result of the required modifications last year due to the temporary tax reduction in germany, where we had 6 months with 16% tax instead of 19% tax. It seems that there was something wrong in the first month, and since then the client balance is off 3% of the first invoice.

Update - just tried to understand invoiceninja/app/Console/Commands/CheckData.php and detected that lines 682 to 845 are commented out and this is where the balance fixing code lies…

Is this intended?

The check data command can be used to check for problems, but we don’t support using it to correct issues (hence the code being commented out).

we don't support using it to correct issues sounds confusing when noticing the “–fix” option and reading the help text in the header of the script:

Since the application was released a number of bugs have inevitably been found.
Although the bugs have always been fixed in some cases they've caused the client's
balance, paid to date and/or activity records to become inaccurate. This script will
check for errors and correct the data.

But never mind, I fixed the incorrect balance values already. Still confused by the language text errors, though. Looks like the migration added additional texts but the check script doesn’t know about it:

root@host:/var/www/html/invoiceninja# grep -r 'Cross account migration is not allowed' . |grep -v 'storage/logs'
./resources/lang/nl/texts.php:    'cross_migration_message' => 'Cross account migration is not allowed. Please read more about it here: <a href="https://invoiceninja.github.io/docs/migration/#troubleshooting">https://invoiceninja.github.io/docs/migration/#troubleshooting</a>',
./resources/lang/bg/texts.php:    'cross_migration_message' => 'Cross account migration is not allowed. Please read more about it here: <a href="https://invoiceninja.github.io/docs/migration/#troubleshooting">https://invoiceninja.github.io/docs/migration/#troubleshooting</a>',
[...]

since ninja:check-data still exits with FAILURE because of them, I’m not sure if that’s the blocker I’m having with my migration attempts.

I’m familiar with the code, I wrote it (and disabled it).

The cross account error can be from using different email addresses on v4 and v5.

I double checked the setup on both v4 and v5 systems. They both have the same email address and the same password, as the migration guide requires. ninja:check-data gives no errors besides the language text ones, but still the migration fails. Besides that - it’s no error because of different email addresses, it says that there are entries in the language files named ‘cross_migration_message’ that check-data doesn’t recognize. I’ve seen your reply on V4 to v5 - "check-data" and migration error about that.

The frontend says Oops, something went wrong. Migration can't be processed at the moment. Please checks the logs.
and the log file says production.INFO: {"message":"Invalid token","errors":{}} [] []

and I don’t know what to check here. I followed all the info from the migration guide and checked all items on the troubleshooting topic, but it still has an issue somewhere.

Thanks for the details! @david do you have any suggestions?

So, any news on that topic?

Are you still seeing this error?

cc @david

I think so. I didn’t check it since then, because it doesn’t make sense to run the same migration again and again and again unless I get any hint on what to change to resolve the error. And since @david seems to be on vacation for at least 2,5 months: Is there anyone else that has enough knowledge of the software to know what to do?

I wouldn’t call lockdown vacation, but I can at least dream that it is a vacation.

So my suggestion to debug this is to use the curl command below to test your login credentials to V5, swap your v5 url + username/password into the CURL statement below. Let us know if it returns a valid response or an error response.

curl -X POST 'http://ninja.test:8000/api/v1/login' \
-H "Content-Type:application/json" \
-d '{"email":"small@example.com","password":"password"}' \
-H "X-API-SECRET:password" \
-H "X-Requested-With: XMLHttpRequest";

This returns nothing at all. No matter if the credentials are correct or not. The laravel.log also doesn’t contain any new errors, but I found this in the apache error log:

[Tue Aug 31 10:58:55.613308 2021] [php7:error] [pid 25512] [client 192.76.134.67:37756] PHP Fatal error: Uncaught Error: Call to a member function connection() on null in /var/www/naruto5/vendor/laravel/framework/src/Illuminate/Support/Facades/Schema.php:45\nStack trace:\n#0 /var/www/naruto5/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(166): Illuminate\Support\Facades\Schema::getFacadeAccessor()\n#1 /var/www/naruto5/vendor/laravel/framework/src/Illuminate/Support/Facades/Facade.php(255): Illuminate\Support\Facades\Facade::getFacadeRoot()\n#2 /var/www/naruto5/app/Exceptions/Handler.php(75): Illuminate\Support\Facades\Facade::__callStatic()\n#3 /var/www/naruto5/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(414): App\Exceptions\Handler->report()\n#4 /var/www/naruto5/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(112): Illuminate\Foundation\Http\Kernel->reportException()\n#5 /var/www/naruto5/public/index.php(57): Illuminate\Foundation\Http\Kernel->handle()\n#6 {main}\n thrown in /var/www/naruto5/vendor/laravel/framework/src/Illuminate/Support/Facades/Schema.php on line 45

Looks like I screwed it completely while testing around.
I’m going to do a complete reinstall of v5 and report then.

Ok, I removed the folder and the database and reinstalled v5.3.1 from scratch, followed the migration guide on Free Source Available Invoicing, Expenses & Time-Tracking | Invoice Ninja and got the same error:

Oops, something went wrong. Migration can't be processed at the moment. Please checks the logs.

the logfile on the v4 system tails with this:

[2021-08-31 09:52:00] production.INFO: get quotes [] []
[2021-08-31 09:52:00] production.INFO: get payments [] []
[2021-08-31 09:52:00] production.INFO: get credits [] []
[2021-08-31 09:52:00] production.INFO: get documents [] []
[2021-08-31 09:52:00] production.INFO: get expense categories [] []
[2021-08-31 09:52:00] production.INFO: get task statuses [] []
[2021-08-31 09:52:00] production.INFO: get expenses [] []
[2021-08-31 09:52:00] production.INFO: get tasks [] []
[2021-08-31 09:52:00] production.INFO: get documents [] []
[2021-08-31 09:52:04] production.INFO:  [] []

The logfile on the v5 system only contains this:

[2021-08-31 09:50:25] production.INFO: latest version = 5.3.1  

The mentioned curl test replies with a quite large json construct containing user data, company data, designs, etc.but no error.

Sorry to ask again, but… any news on that, @david ?
I have several problems with v4 and would really like to migrate to v5, but there’s already two years worth of data in my v4 installation which prevents me from just ditching it and starting with a fresh v5, so being able to migrate is essential.

when we see these errors typically it involves an incorrect URL being provided. Does your URL Path include /public or a subdirectory by chance?

No subdirectory or /public. But this was a good hint. while gathering data to provide details to prove your assumption wrong, I detected this in the apache error log:

[Wed Sep 15 12:11:44.175422 2021] [php7:error] [pid 28616] [client 192.76.134.67:49318] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/www/html/app/Jobs/Util/Import.php on line 205
[Wed Sep 15 12:11:44.175838 2021] [php7:error] [pid 28616] [client 192.76.134.67:49318] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 32768 bytes) in /var/www/html/vendor/symfony/error-handler/Error/FatalError.php on line 1

I’m checking this and report again. Meanwhile, it would be helpful if such errors could be catched by the migration script instead of the generic “something went wrong” message.

PHP OOMing happens at a much lower level and we can’t catch that in the app as such. However as you note, the logging does dump the exception in the logs.

First of all, sorry for bugging around… it seems that I now got all things right for successfully importing my v4 data. I now got a v5 instance with a fully migrated data store, where I can login with my admin user and do things.
Unfortunately, my colleagues can’t login with their accounts, and since there’s no “change password” field for admins to reset them, only the “forgot password” link remains, but there seems to be a problem with the email configuration.

I put this into the .env file according to the .env.example:

MAIL_MAILER=smtp
MAIL_HOST=localhost
MAIL_PORT=25
MAIL_USERNAME=null
MAIL_PASSWORD=null
MAIL_ENCRYPTION=none
MAIL_FROM_ADDRESS='invoice@company.com'
MAIL_FROM_NAME='My Company'

Unfortunately, there’s no email generated. The laravel.log contains nothing, even though APP_DEBUG is enabled, the local postfix mailer doesn’t log anything either, only the apache access log shows the "POST /api/v1/reset_password?first_load=true&include_static=true HTTP/1.1" command. Postfix is up and running on 127.0.0.1:25.

I also tried to re-send an invoice, but that also does not generate any smtp connection. And it makes no difference if company is enabled or not.

Here’s a method to set the password manually:

  • From the CLI run: php artisan tinker
  • Then run: bcrypt('new_password');
  • Update the password in the users table in the database with the output

ah, thx. Yep, that works. And while doing this I noticed that:

MariaDB [ninja]> select id, password from users;
+----+--------------------------------------------------------------+
| id | password                                                     |
+----+--------------------------------------------------------------+
|  1 | $2y$10$AUkNQQ72fDfLo699eOxxgu8jjY/sWcFZzNEnwwcqERbHupNbD9.Z2 |
|  2 |                                                              |
|  3 |                                                              |
|  4 |                                                              |
|  5 |                                                              |
|  6 |                                                              |
+----+--------------------------------------------------------------+
6 rows in set (0.001 sec)

Seems like the migration didn’t transfer the password row at all.