V5 Test Migration results and bugs

strider27 · July 27, 2022, 3:26am

Hello, we ran a test migration from V4.5.50 to V5.4.11 a few days ago,

The test migration went mostly great, (had to increase max_allowed_packet field in /etc/my.cnf) However we would appreciate some feedback on below.

We have a few noticeable issues relating to migrated data:

Imported Recurring invoices are marked as pending except for one which seems to be the first one imported as it is numbered 1 (this one is marked active).
Imported Recurring invoices have no invoice history, i.e. no invoices generated prior to the migration are listed in the “invoices” section of the Recurring invoice. Schedule section of the recurring invoice has no entries for previously generated invoices only dates for upcoming ones.
Imported Clients have no contacts listed in the “table view” unless we open the clients click edit then save. Additionally after the migration completed the email mentioned that All our “clients without a single primary contact”.
Markdown parser for product descriptions in v4 allowed for no space after “#” and no requirement for double space prior to new line. In v5 those 2 are required and of course we used the v4 way… meaning that all Descriptions in migrated Products and Tasks as well as Invoice, Recurring invoice and Quotation line items, now don’t look good enough to be presented to the client.

Can the migration script be updated so it can append a single space to hashtags or groups of hashtags and prepend 2 spaces to carriage returns? Or whatever the equivalent is?

While Product descriptions can be manually fixed, the line items for existing invoices, quotes etc would be impossible to fix by hand due to the sheer volume. We are willing to try messing with MySQL directly either in v4 or v5, but would need a bit of guidance from devs as am not sure what that query would even look like

No errors were detected in any of the usual logs.

The upgrade path was Production instance > Cloned To another VM > Cloned VM Updated including PHP 7.4 > Invoice Ninja 4.5.16 dev branch updated to Invoice Ninja v4.5.50 zip archive.

V5 instance was Ubuntu 20.04 + Nginx + PHP7.4 + INinja v5.4.11

hillel · July 27, 2022, 6:06am

Hi,

Thanks for the detailed feedback!

cc @david

david · July 27, 2022, 6:35am

The migration doesn’t transfer any history, so the recurring invoices are brought over, but are considering pending until they send from the first time from v5.

This can sometimes happen, running

php artisan ninja:check-data --fix=true

should resolve this

v4 had partial markdown support, in v5 it is full markdown support so this is the reason you are seeing this particular issue. However from my tests, the markdown appears to have come across ok. Can you confirm in Settings > Account Managment you have markdown in PDF support turned on?

If you do, please share a screenshot of how the invoice is appearing post migration.

strider27 · July 27, 2022, 7:22am

After runing sudo -u www-data php artisan ninja:check-data --fix=true
We get this:

2022-07-27 07:14:25 2022-07-27 07:14:25 Running CheckData... on Connected to Default DB Fix Status = Fixing Issues
2022-07-27 07:14:25 0 clients with incorrect balances
2022-07-27 07:14:25 0 clients with incorrect paid to dates
2022-07-27 07:14:25 0 contacts without a contact_key
2022-07-27 07:14:25 0 clients without any contacts
2022-07-27 07:14:25 0 contacts without a contact_key
2022-07-27 07:14:25 0 vendors without any contacts
2022-07-27 07:14:25 0 invoices without any invitations
2022-07-27 07:14:25 0 quotes without any invitations
2022-07-27 07:14:25 0 credits without any invitations
2022-07-27 07:14:25 0 recurring_invoices without any invitations
2022-07-27 07:14:25 0 wrong invoices with bad balance state
2022-07-27 07:14:25 0 users with duplicate oauth ids
2022-07-27 07:14:25 Done: SUCCESS
2022-07-27 07:14:25 Total execution time in seconds: 0.087177991867065

In Address.php line 54:

  Email "" does not comply with addr-spec of RFC 2822.

Not all our clients have emails or even contacts or contact details but most of them do.

Cleared Cache, Logged Off and on ran php artisan optimise, same issue, no contact names, emails etc in the Clients table view.

david · July 27, 2022, 7:36am

@hillel is the blank contact screen a symptom of no primary contact being set?

@strider27 , the address.php error is due to no email address being set for the .env var

ERROR_EMAIL=

strider27 · July 27, 2022, 7:38am

V5 requires a space between hashtags and text, as well as 2 trailing spaces and return for a new line, in V4 that was not a requirement hence the below mess:

V4

V5

That’s why I’m hoping to run a mysql query,
The query would append a space to hashtags that are not immediately followed by another hashtag or a space.
And another query that would prepend every new line with a double space?
I’m hoping that would fix it.

hillel · July 27, 2022, 7:39am

@david yes, that could explain it

strider27 · July 27, 2022, 7:44am

Oh that’s ok, this is a test migration, there is live data and the company is set to active but the instance is not able to email, process payments etc.

david · July 27, 2022, 7:51am

@strider27

Do you use all the markdown options in v4? ie.

or only ###

strider27 · July 27, 2022, 8:06am

@david
As far as I remember single, double and triple hashtag.

david · July 27, 2022, 9:50am

@strider27

That is going to be tough to split instance of #, ## and ### apart due to partial matching… You could try the following ensuring you do a full DB backup in case this does not work

php artisan tinker


Invoice::query()->cursor()->each(function ($invoice){

  $line_items = $invoice->line_items;

  foreach($line_items as $item)
  {
      $item->notes = str_replace("\n", "  ", $item->notes);
      
      if(preg_match("/^###$/", $item->notes)){

        $item->notes = str_replace("###", "### ", $item->notes);

      }

      if(preg_match("/^##$/", $item->notes)){

        $item->notes = str_replace("##", "## ", $item->notes);

      }

      if(preg_match("/^#$/", $item->notes)){

        $item->notes = str_replace("#", "# ", $item->notes);

      }      

  }

  $invoice->line_items = $line_items;

  $invoice->save();

});

strider27 · July 28, 2022, 2:06am

@david

Nice, Could you please help with code for Recurring, Quotes and Products as well?

Had to play around with Regex but got it eventually with negative lookahead and negative lookbehind . Also had to do newlines in 2 steps, doing it in a single step just prepended the lines with 2 spaces instead of appending them.

Tried running it against Recurring Invoice myself by changing the query and a few other bits but ended up nuking all items in Recurring Inv haha.

Invoice::query()->cursor()->each(function ($invoice){
  $line_items = $invoice->line_items;
  foreach($line_items as $item)
  {
      if(preg_match("/(?<!#)###(?!#)(?! )/", $item->notes)){
        $item->notes = str_replace("###", "### ", $item->notes);
      }
      if(preg_match("/(?<!#)##(?!#)(?! )/", $item->notes)){
        $item->notes = str_replace("##", "## ", $item->notes);
      }
      if(preg_match("/(?<!#)#(?!#)(?! )/", $item->notes)){
        $item->notes = str_replace("#", "# ", $item->notes);
      }

// running '$item->notes = str_replace("\n", "  \n", $item->notes);'

	$item->notes = str_replace("\n", "  ", $item->notes);  
	$item->notes = str_replace("  ", "  \n", $item->notes);
  }
  $invoice->line_items = $line_items;
  $invoice->save();

});

david · July 28, 2022, 3:25am

@strider27

For recurring invoices / quotes etc etc, just replace Invoice:: with

RecurringInvoice::
Quote::

Product::query()->cursor()->each(function ($product) {

$product->notes = xxxxx;
$product->save();

});

strider27 · July 28, 2022, 4:33am

@david
Thanks, I could swear that’s what I did when I nuked the Recurring but seems to have worked this time, except Products

I get

PHP Error:  Call to undefined method Imdhemy\Purchases\Product::query() in 
/usr/share/nginx/invoiceninja/vendor/laravel/framework/src/Illuminate/Support/
Facades/Facade.php on line 337

When I run

Product::query()->cursor()->each(function ($product){
  $notes = $product->notes;
  foreach($notes as $item)
  {
      if(preg_match("/(?<!#)###(?!#)(?! )/", $item->notes)){
        $item->notes = str_replace("###", "### ", $item->notes);
      }
      if(preg_match("/(?<!#)##(?!#)(?! )/", $item->notes)){
        $item->notes = str_replace("##", "## ", $item->notes);
      }
      if(preg_match("/(?<!#)#(?!#)(?! )/", $item->notes)){
        $item->notes = str_replace("#", "# ", $item->notes);
      }
        $item->notes = str_replace("•	", "", $item->notes);
        $item->notes = str_replace("	", "", $item->notes);
  }
  $product->notes = $notes;
  $product->save();
});

david · July 28, 2022, 5:09am

instead of Product::

try

App\Models\Product::

strider27 · July 28, 2022, 5:27am

Get a lot of PHP Warning: foreach() argument must be of type array|object, string given in Psy Shell code on line 3

david · July 28, 2022, 11:35pm

for products, you don’t need the foreach loop at all.

strider27 · July 28, 2022, 11:49pm

Of course, only one instance of notes per product, my bad.

Thanks for your help! These issues are now resolved and we are one step closer to a full migration.

Code that worked was if anybody is trying this themselves:

App\Models\Product::query()->cursor()->each(function ($product){
  $notes = $product->notes;
      if(preg_match("/(?<!#)###(?!#)(?! )/", $notes)){
        $notes = str_replace("###", "### ", $notes);
      }
      if(preg_match("/(?<!#)##(?!#)(?! )/", $notes)){
        $notes = str_replace("##", "## ", $notes);
      }
      if(preg_match("/(?<!#)#(?!#)(?! )/", $notes)){
        $notes = str_replace("#", "# ", $notes);
      } 
  $product->notes = $notes;
  $product->save();
});

strider27 · July 29, 2022, 12:18am

Any suggestion re: contacts not appearing in the client list table, until contact is edited and saved again?

We only have about 100 contract clients whose details need to be up to date for recuring invoices, email notifications etc so in our case it can be fixed manually.

However if someone is migrating a large amount of regular clients with recuring invoices + email reminders enabled etc, would not having a default contact assigned not affect them majorly?

david · July 29, 2022, 12:22am

@strider27

Can you check the contacts as see if the is_primary flag is set for these contacts that are not appearing?