Git pull automation: how to deal with modified files?

Hello,

I am working on a script to automate keeping IN up to date using git and I keep adding stuff to it as I encounter issues.

On my last run through a cron job, it did not update the app, so I ran it manually to see what happened, and when I ran git pull, I got told that a couple of files (composer.lock, composer,json) had been modified and that the process would abort. I don’t know how how that happened exactly, but I ran the git checkout composer.json composer.lock command and then I could procede with the pull.

That made me think about automating dealing with modified files by backing them up instead of deleting them, but it was way over my competence level. I heard that GPT Chat was good at coding, so I tried it out. Wow! It’s a beast (at least compared to me), it understands exactly everything I ask and puts it into code, it modifies the code accordingly when I ask for a modification, it answers to my questions perfectly… I’m impressed. All in all it’s a work in progress (Both GPT chat and my script), but I think I’m on the right path. I’ll first start by letting GPT describe that part of the script:

Q. Can you do a recap of the whole script for now and tell me what it does in detail?

A. Sure, here’s a summary of the script:

  1. The script starts by using the git status --porcelain command to get a machine-readable output of the status of the files in your repository.
  2. The grep '^[AM]' and grep '^UU' filters the files that are in conflict and modified, the awk '{print $2}' command is used to extract the name of the file.
  3. It then uses an if statement to check if there are any modified or conflicting files, if there are, it proceeds to the next step, if not, it skips the following steps and continues to the git pull command.
  4. The script then uses an if statement to check if the gitbackup folder already exists, if it doesn’t, the script will create it using the mkdir gitbackup command. If the folder already exists, it will skip the mkdir statement and continue with the rest of the script.
  5. For each file in modified_files and conflict_files, it runs the git stash push -m "stash-conflict" $file command, this command saves the current changes in a new stash and the -m flag is used to provide a name to the stash.
  6. It then runs a second loop, this time for each file in modified_files and conflict_files, using an if statement to check if the file is already in the gitbackup folder, if it is, it will add an incremental number to the file name and move it to the gitbackup folder. If the file is not in the gitbackup folder, it will move it to the gitbackup folder.
  7. Finally, it uses the echo command to send an email notification to the email address specified, informing that there were conflicts that need to be checked, the mail command is used to send the email.

This script is designed to be run after a git pull command that is blocked by conflicts, it will backup the modified and conflicting files in the gitbackup folder, it will add an incremental number to the file name to avoid overwriting the files, it will then send an email notification to the provided email address, informing that there were conflicts that need to be checked.

Please make sure that the script is being run in the correct directory and that the path to the /gitbackup folder is correct, also, check that the script has the necessary permissions to create the folder and move the files, and that the git stash, mv and mail commands are properly configured in your server.

Please let me know if you have any other question or if you need any help.

Now the code:

#!/bin/bash

# check for modified and conflicting files
modified_files=($(git status --porcelain | grep '^[AM]' | awk '{print $2}'))
conflict_files=($(git status --porcelain | grep '^UU' | awk '{print $2}'))

# stash the modified and conflicting files
if [ ${#modified_files[@]} -gt 0 ] || [ ${#conflict_files[@]} -gt 0 ]; then
    if [ ! -d "gitbackup" ]; then
        mkdir gitbackup
    fi
    for file in "${modified_files[@]}" "${conflict_files[@]}"; do
        git stash push -m "stash-conflict" $file
    done

    for file in "${modified_files[@]}" "${conflict_files[@]}"; do
        if [ -f gitbackup/$file ]; then
            i=1
            while [ -f gitbackup/${file}_$i ]; do
                let i++
            done
            mv gitbackup/$file gitbackup/${file}_$i
        else
            mv gitbackup/$file gitbackup/
        fi
    done

    # Send email notification
    echo "Conflicts found while trying to pull the changes from the remote repository" | mail -s "Conflicts found" your@email.com
else
    git pull
fi

Comments are welcome!

Thanks

Hi,

@david any thoughts?

I can’t comment on whether this is good or not, however assuming your have cloned the repo from

and are on the v5-stable branch, these git commands are all you need to sync the repo’s

git fetch origin
git reset --hard origin/v5-stable

It won’t matter what you have done to the repo, this will reset your repo to the HEAD of the origin.

OK, thanks!

I did clone the repo from the link you posted and I am running the stable branch. Here is what I used to install:

git clone -b v5-stable --single-branch https://github.com/invoiceninja/invoiceninja.git

I tried your fetch and reset commands followed by a git pull:

charleso@server:~/public_html/admin$ git fetch origin
charleso@server:~/public_html/admin$ git reset --hard origin/v5-stable
HEAD is now at 0f028d519 Merge pull request #8212 from turbo124/v5-stable
charleso@server:~/public_html/admin$ git pull
Already up to date.

Git tells me it’s up to date, yet IN tells me that there is an update available:

image

What am I missing?

We do a daily cron job to check the latest version from here:

https://pdf.invoicing.co/api/version

so that would explain this. it will update.

I see, thanks!

It’s indeed reporting 5.5.53:

image

But that’s also the latest version on GitHub:

So any idea why git pull tells me my installation is up to date when I’m running .62?

image

I’m confused :melting_face:

@hillel can you advise how the system checks these versions?

@david the app uses the account current_version/latest_version properties

@charles

You may want to try refreshing the cache

Clearing the cache through the app menu did not help.

The git pull command (and the following install and cleanup) worked fine to update my install for the last couple of versions, but it stopped on .62 working after I ran

git fetch origin
git reset --hard origin/v5-stable

Coincidence or possibly related?

Edit: I tried git rebase and I got this:

error: cannot rebase: You have unstaged changes.
error: Please commit or stash them.

So I’m back to square one with modified files (I have no idea how that happened). Should I just force checkout them, or stash them in backups like suggested in my first post?

Edit2: It looks like cPanel added some lines to the .htaccess fie last night regarding the PHP version to use. I then used checkout on the file and tried git pull again… but it still tells me I’m up to date.

Edit3: I erased the .git folder and redid a complete git clone of the whole repository, then moved the files to overwrite my current installation. It worked to update my app to .63. I still have no clue as of why the rest of the methods failed…

If you keep running into issues with git, I made a bash script (with the help of ChatGTP) for updates that doesn’t use git or composer - GitHub - CoryTrevor/invoice-ninja-updates: Bash script for updates

1 Like

@josh Sounds great, I’ll try that! Thanks!

Edit: Any reason why you use the full release (invoiceninja.zip) vs the smaller, updated source code (version.zip)?

@charles Cool, let me know how it goes. You might need to make a couple of tweaks to it for your server as mentioned in the readme. Just let me know if there’s any issues.

1 Like

@charles I used the full zip because I can’t use composer due to php-cli version being 7.4 so needed to download the full version that has all the vendor files in it.

I guess it could be recoded to just use the source code zip and extract to the main directory and then run composer.

1 Like

@josh

Ok seriously, it’s perfect.

I updated the path to point to my install, of course, and I also renamed the generic sounding “update” to “invoiceninja_temp”, just in case the script crashes eventually for some reason and I’m left with a generic folder. By referencing Invoice Ninja, I’ll know what it is right off the bat.

I also changed the php lines to just execute php without a path.

For testing I created a backup, then I just changed the version in version.txt to something prior to force the update. I tried again after the update to see if it would exit properly.

It works beautifully, and takes just a couple of seconds to do the whole thing! Thanks a lot for this! I’m adding this to my nightly cron jobs right now, problem solved :slight_smile:

@charles Awesome, glad to hear it worked.

Good idea with changing the update folder name. It appears quite a few times in the script so just make sure to replace all instances of it.

After completing a test update, if you double check that that the /public/storage folder still has your logo image and PDFs in there then that should mean everything is working with the update script.

Yup, pretty much everything is there and working as intended!

That being said, I noticed that the script erases everything not included in the backup and not provided by the latest release. I had to add a line to make it keep my Laravel workers log file. On my system, that’s:
cp public_html/admin/laravel_worker.log invoiceninja_temp/
Maybe you could add a commented out line that could be edited for such additional files?

Also, is there a reason why you wouldn’t want to backup /storage/logs? That might prove useful for debugging at some point I think, but it’s getting cleaned at every update.

@charles Good to hear its working.

Yes, it uses a clean slate approach where anything that it hasn’t specifically instructed to copy to the update directory and isn’t included in the latest release is removed. That is by design to avoid the issues that can arise when just using the overwrite method.

There was a previous issue where the files in /vendor/beganovich/snappdf/versions/ had a different name for each version so they kept piling up with each update and taking up a lot of hard drive space. The built-in self-updater is now coded to remove those files. There’s also some cache files that need to be removed as well when just doing overwrite. So I personally prefer to tell it what to keep rather than having it keep everything by default and then having to tell it exactly what to remove as it’s not always possible to know what needs to be removed.

Years ago in WordPress there was a security vulnerability in a script called timthumb.php which led to theme and plugin developers removing it from their themes and plugins. But even after updating some people were still getting hacked through the script. Turns out it was because they were using the FTP overwrite method to update so the exploitable files weren’t removed.

I didn’t get it to keep /storage/logs as whenever I’m in the log file it’s to look at for an error message that’s just occurred which I find easier with a fresh log file rather than one that’s full with lots of older messages but I can appreciate that having older logs would be useful in some situations for debugging. That’s a good idea having a commented out line that can be used for including additional files. I’ve added that now.

I haven’t got a public_html/admin folder on my install. What sort of system are you on?

1 Like

Good thinking!

I indeed noticed after my last message that all downloaded/installed versions of Chrome are deleted in the Snappdf folder; that makes the script a bit less “set and forget” than it could be, since it would effectively break Snappdf. It could be a good idea to keep at least the latest version that’s installed (and in use)? The script would just have to check the contents of /vendor/beganovich/snappdf/versions/revision.txt (for something looking like 1099031-Linux_x64), then check if a folder of that name exists in the same folder as revision.txt, then if so, add it to the exclude list and back it up. I think the code within your folder structure would look something like this:

# Get the Snappdf Chrome installation version from the revision file, if available
chromerevision=$(cat /vendor/beganovich/snappdf/versions/revision.txt)

# Check if the Snappdf Chrome installation folder exists
if [ -d "public_html/vendor/beganovich/snappdf/versions/$chromerevision" ]; then
	# Copy the Snappdf Chrome installation to the update directory
	cp -r public_html/vendor/beganovich/snappdf/versions/$chromerevision update/vendor/beganovich/snappdf/versions
fi

Or more simply, at the expanse of some time/computing power/bandwidth, the latest revision of Chrome could just be re-downloaded at the end of the Invoice Ninja update and cleanup process by simply running /vendor/bin/snappdf download

I get your point, but thanks for adding the option!

I am running Invoice Ninja on a cPanel account on Ubuntu 20.04. “admin” is just the custom folder I installed Invoice Ninja in (with a subdomain pointing to it), since my public_html web root is occupied by a Wordpress installation.

Good point about snappdf. I wasn’t aware that the chromium binaries for it were no longer included by default. Since I’m guessing it relies on composer for updates, I like your idea of having a line for /vendor/bin/snappdf download that can be uncommented for people using snappdf so I’ve added that now. That should always get the latest version otherwise if it just copies the current version each time then it would need to be updated separately.

That makes sense. I don’t seem to have a laravel_worker.log file in my installation’s root directory but I guess that’s being added by the server rather than the app.