Migration failing miserably


as I have quite a huge Database which I need to migrate I have used the “migrations:export” command to export it. The first issue is there: It does not export the ninja-tokens data during export and then fails during import. Anyways, not a huge problem and easy fix in the code (there’s a difference between the cli export and the web export).

Once that is fixed the next issue appears: I created a user with the same email and password, and after a 25 Minute migration process (seriously??? How can it take so long?) I end up with a database filled with data. There are some useless companies and users created (what are those for? Why do they need to be there?) and once I delete the “new company” that was created during setup before my migration from the account management tab it just nukes my account and leaves everything in a broken state.

Additionally I sometimes can’t even sign in as the performance of v5 is just too bad (timeout after 30 seconds…).

So for me I am staying on v4 for now as v5 is just completely broken on so many levels (you can try to setup a smtp connection to localhost:25 without username and password during initial setup… Should be the easiest thing but just fails and needs manually editing the config file afterwards…).

How are you deleting this company? Through the app, or directly in the database? If you are directly modifying the database you’ve probably bricked the app. You should not modify the database directly.

If you are migrating a massive dataset, it makes sense that is may take some time to migrate the data. The schema between v4 and v5 is completely different and can take time.

If you have any specific questions, I’m happy to help.

As described I’m deleting it from the account management tab from the App. Instead of only deleting that company it deletes the entire user (I’ve checked that in the database).

Why are there “dummy users” created during import and how can that stuff be deleted again?

The missing ninja-tokens when cli export is used are caused by migration hosted tokens · invoiceninja/invoiceninja@51e111f · GitHub only adding it to the HTTP export. It’s probably a good idea to get that fixed.

Also it seems like when I do the migration the old users passwords don’t get migrated? I have multiple users for the same company which get created but apparently when I check in the database the password is empty.

Will these issues be adressed at some point? Or do I just accept that v5 is not completely unusable right now and I have to continue with v4?

I assume you can you create new tokens and update the passwords, what issues are you seeing in v5 after the migration?

The issue isn’t creating new tokens, that just breaks the migration completely as it doesn’t like when those are missing. Anyways, I fixed that so it’s exported properly.

The issues once v5 is setup are the ones mentioned above:

@david any thoughts?

Some ideas…

  • I see this post started a while ago, have you tried with the latest versions of v4 and v5?

  • If you’re seeing a timeout logging in I suggest checking that is_large is set to true for all of the records in the companies table.

Passwords cannot be migrated as they are not stored anywhere. We only store the hashes. This is one part of the migration where user action cannot be avoided. Each user will need to do a password reset.

If you have a very large dataset, then it can take some time for the migration to process.

What’s wrong with migrating the hashes? There is absolutely no reason those can’t be exported and imported again.

I can retry with the latest version (v4 is automatically updated for me, so I would just have to update v5 and then try again) but at this point I don’t really see any advantage in upgrading, especially if I need to reset all users passwords.