Git pull automation: how to deal with modified files?

Hello,

I am working on a script to automate keeping IN up to date using git and I keep adding stuff to it as I encounter issues.

On my last run through a cron job, it did not update the app, so I ran it manually to see what happened, and when I ran git pull, I got told that a couple of files (composer.lock, composer,json) had been modified and that the process would abort. I don’t know how how that happened exactly, but I ran the git checkout composer.json composer.lock command and then I could procede with the pull.

That made me think about automating dealing with modified files by backing them up instead of deleting them, but it was way over my competence level. I heard that GPT Chat was good at coding, so I tried it out. Wow! It’s a beast (at least compared to me), it understands exactly everything I ask and puts it into code, it modifies the code accordingly when I ask for a modification, it answers to my questions perfectly… I’m impressed. All in all it’s a work in progress (Both GPT chat and my script), but I think I’m on the right path. I’ll first start by letting GPT describe that part of the script:

Q. Can you do a recap of the whole script for now and tell me what it does in detail?

A. Sure, here’s a summary of the script:

  1. The script starts by using the git status --porcelain command to get a machine-readable output of the status of the files in your repository.
  2. The grep '^[AM]' and grep '^UU' filters the files that are in conflict and modified, the awk '{print $2}' command is used to extract the name of the file.
  3. It then uses an if statement to check if there are any modified or conflicting files, if there are, it proceeds to the next step, if not, it skips the following steps and continues to the git pull command.
  4. The script then uses an if statement to check if the gitbackup folder already exists, if it doesn’t, the script will create it using the mkdir gitbackup command. If the folder already exists, it will skip the mkdir statement and continue with the rest of the script.
  5. For each file in modified_files and conflict_files, it runs the git stash push -m "stash-conflict" $file command, this command saves the current changes in a new stash and the -m flag is used to provide a name to the stash.
  6. It then runs a second loop, this time for each file in modified_files and conflict_files, using an if statement to check if the file is already in the gitbackup folder, if it is, it will add an incremental number to the file name and move it to the gitbackup folder. If the file is not in the gitbackup folder, it will move it to the gitbackup folder.
  7. Finally, it uses the echo command to send an email notification to the email address specified, informing that there were conflicts that need to be checked, the mail command is used to send the email.

This script is designed to be run after a git pull command that is blocked by conflicts, it will backup the modified and conflicting files in the gitbackup folder, it will add an incremental number to the file name to avoid overwriting the files, it will then send an email notification to the provided email address, informing that there were conflicts that need to be checked.

Please make sure that the script is being run in the correct directory and that the path to the /gitbackup folder is correct, also, check that the script has the necessary permissions to create the folder and move the files, and that the git stash, mv and mail commands are properly configured in your server.

Please let me know if you have any other question or if you need any help.

Now the code:

#!/bin/bash

# check for modified and conflicting files
modified_files=($(git status --porcelain | grep '^[AM]' | awk '{print $2}'))
conflict_files=($(git status --porcelain | grep '^UU' | awk '{print $2}'))

# stash the modified and conflicting files
if [ ${#modified_files[@]} -gt 0 ] || [ ${#conflict_files[@]} -gt 0 ]; then
    if [ ! -d "gitbackup" ]; then
        mkdir gitbackup
    fi
    for file in "${modified_files[@]}" "${conflict_files[@]}"; do
        git stash push -m "stash-conflict" $file
    done

    for file in "${modified_files[@]}" "${conflict_files[@]}"; do
        if [ -f gitbackup/$file ]; then
            i=1
            while [ -f gitbackup/${file}_$i ]; do
                let i++
            done
            mv gitbackup/$file gitbackup/${file}_$i
        else
            mv gitbackup/$file gitbackup/
        fi
    done

    # Send email notification
    echo "Conflicts found while trying to pull the changes from the remote repository" | mail -s "Conflicts found" your@email.com
else
    git pull
fi

Comments are welcome!

Thanks

Hi,

@david any thoughts?

I can’t comment on whether this is good or not, however assuming your have cloned the repo from

and are on the v5-stable branch, these git commands are all you need to sync the repo’s

git fetch origin
git reset --hard origin/v5-stable

It won’t matter what you have done to the repo, this will reset your repo to the HEAD of the origin.

OK, thanks!

I did clone the repo from the link you posted and I am running the stable branch. Here is what I used to install:

git clone -b v5-stable --single-branch https://github.com/invoiceninja/invoiceninja.git

I tried your fetch and reset commands followed by a git pull:

charleso@server:~/public_html/admin$ git fetch origin
charleso@server:~/public_html/admin$ git reset --hard origin/v5-stable
HEAD is now at 0f028d519 Merge pull request #8212 from turbo124/v5-stable
charleso@server:~/public_html/admin$ git pull
Already up to date.

Git tells me it’s up to date, yet IN tells me that there is an update available:

image

What am I missing?

We do a daily cron job to check the latest version from here:

https://pdf.invoicing.co/api/version

so that would explain this. it will update.

I see, thanks!

It’s indeed reporting 5.5.53:

image

But that’s also the latest version on GitHub:

So any idea why git pull tells me my installation is up to date when I’m running .62?

image

I’m confused :melting_face:

@hillel can you advise how the system checks these versions?

@david the app uses the account current_version/latest_version properties

@charles

You may want to try refreshing the cache

Clearing the cache through the app menu did not help.

The git pull command (and the following install and cleanup) worked fine to update my install for the last couple of versions, but it stopped on .62 working after I ran

git fetch origin
git reset --hard origin/v5-stable

Coincidence or possibly related?

Edit: I tried git rebase and I got this:

error: cannot rebase: You have unstaged changes.
error: Please commit or stash them.

So I’m back to square one with modified files (I have no idea how that happened). Should I just force checkout them, or stash them in backups like suggested in my first post?

Edit2: It looks like cPanel added some lines to the .htaccess fie last night regarding the PHP version to use. I then used checkout on the file and tried git pull again… but it still tells me I’m up to date.

Edit3: I erased the .git folder and redid a complete git clone of the whole repository, then moved the files to overwrite my current installation. It worked to update my app to .63. I still have no clue as of why the rest of the methods failed…

If you keep running into issues with git, I made a bash script (with the help of ChatGTP) for updates that doesn’t use git or composer - GitHub - CoryTrevor/invoice-ninja-updates: Bash script for updates

1 Like

@josh Sounds great, I’ll try that! Thanks!

Edit: Any reason why you use the full release (invoiceninja.zip) vs the smaller, updated source code (version.zip)?

@charles Cool, let me know how it goes. You might need to make a couple of tweaks to it for your server as mentioned in the readme. Just let me know if there’s any issues.

1 Like

@charles I used the full zip because I can’t use composer due to php-cli version being 7.4 so needed to download the full version that has all the vendor files in it.

I guess it could be recoded to just use the source code zip and extract to the main directory and then run composer.

1 Like

@josh

Ok seriously, it’s perfect.

I updated the path to point to my install, of course, and I also renamed the generic sounding “update” to “invoiceninja_temp”, just in case the script crashes eventually for some reason and I’m left with a generic folder. By referencing Invoice Ninja, I’ll know what it is right off the bat.

I also changed the php lines to just execute php without a path.

For testing I created a backup, then I just changed the version in version.txt to something prior to force the update. I tried again after the update to see if it would exit properly.

It works beautifully, and takes just a couple of seconds to do the whole thing! Thanks a lot for this! I’m adding this to my nightly cron jobs right now, problem solved :slight_smile: