Multi Document Block Oriented Search and Replace script

Share |

Download Multi Document Block Oriented Search and Replace script version 1.5, 2010-04-16.

Web Masters, and Programmers! Are you in need of a way to make large, and small text block changes to all of your pages, or a selection of files of one type all at once? This GUI enabled script does search and replace of large block elements across as many files as you select for it to work on.

UPDATE: With version 1.5 I have added a multi level undo menu entry, and the ability to save working file set descriptions, so you do not have to reselect every file in your large file sets every time you decide to work on them. Each file set description contains the file names of a list of files of one file extension only, for one file folder only, or one file folder and it's sub folders only.

In future versions I will be adding the ability to specify multiple file extensions for each file set, and multiple parent folder paths as well.

I had been in need of a tool to do large block element search and replace for quite some time to help me administer my web sites. Sometimes there are changes I want to make to all of the pages on my site, and I could not find an editor that would accomplish the feat of large text block search and replace operations across multiple files at once.

This script depends on the Xdialog package for the GUI widgets it uses to interact with it's users. To install Xdialog issue the following command in Terminal.

sudo apt-get install xdialog

To use the script, simply decompress it from the MultiDocBlockSearchReplace.sh.zip file into any folder on your system. You can then add the following command to a menu entry to start it, or you can issue the command from Terminal as well.

/home/username/folder/path/MultiDocBlockSearchReplace.sh

Replace "username", "folder", and "path" with the actual folder path as it exists on your computer, unless you place this script in a folder within the system's executable file search path, in which case you need only call the script name without a folder path

Here is the source code for the MultiDocBlockSearchReplace.sh script file.

#!/usr/bin/env bash
# MultiDocBlockSearchReplace.sh version 1.5, 2010-04-16.
#
# This script is a GUI enabled, multi-document, block oriented search, and replace
# program. It can save working file sets for each folder you work in, for differing
# file types. These file sets definitions are saved in the folder -
#
#     /home/$USERNAME/.MDBOSearchandReplace/
#
# in files named as follows -
#
#     folderName.fileType.set (Where folderName is the last portion af any folder 
#                                             path that leads to the file set, and fileType is the
#                                             extension of the files in this file set.)
#          Contains three lines as follows -
#                False (whether or not the file search is set to recursive through folders.)
#                /home/ray/links/ (the path to the working directory for this file set.)
#                html (The file extension that this file set consists of.)
#
#     folderName.fileType.set.FileSet (Where folderName is the last portion af any 
#                                                          folder path that leads to the file set, and
#                                                          fileType is the extension of the files in this
#                                                          file set.)
#          Contains a list of the paths, and filenames of every file in this saved file set
#          list, one file path per line of the file.
#
# Usage: Add the following line to a menu entry, or run it from the command line,
#
#       /folderPath/MultiDocBlockSearchReplace.sh
#
#           Where folderPath is the actual path to this script's location, unless you
#           place this script in a folder within the system's executable file search path,
#           in which case you need only call the script name without a folder path.
#
#
## Copyright:
##      Copyright 2009 by Christopher Ray Parrish 
##      All Rights Reserved
##
## License:
##      Released under the GNU General Public License (GPL)
##      http://www.gnu.org/copyleft/gpl.html
##
##      This program is free software; you can redistribute it and/or
##      modify it under the terms of the GNU General Public License
##      as published by the Free Software Foundation; either version 3
##      of the License, or (at your option) any later version.
##
##      This program is distributed in the hope that it will be useful,
##      but WITHOUT ANY WARRANTY; without even the implied warranty of
##      MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
##      GNU General Public License for more details.
##
# this function does the actual replacement of one block element with another.
function DoReplace {
     # Four arguments are passed in, FileContents, SearchString, ReplaceString, and ReplaceAll
     FileContent="$1"
     SearchString="$2"
     ReplaceString="$3"
     ReplaceAll="$4"
     if [ "$ReplaceAll" == "True" ]
        then
            # Do the actual replacement of all occurrences of SearchString with ReplaceString.
            FileContent=${FileContent//"$SearchString"/"$ReplaceString"}
        else
            # Replace only the first occurrence of SearchString with ReplaceString.
            FileContent=${FileContent/"$SearchString"/"$ReplaceString"}
     fi
     # Return the new value of FileContent variable to the calling assignment.
     echo "$FileContent"
}
# This function allows choosing, or typing in a file extension to work with.
function SelectFileType {
     # Present user with a drop down list of possible file types, and allow typing in a different one as well.
     FileType=`Xdialog --auto-placement --stdout --title "MDBOSR - Select File Type" --editable --combobox "Select a file type from the drop down list, or type one into the list." 0 0 "html" "htm" "css" "sh" "txt" "js"`
     # test for Cancel, or X button click.
     ReturnValue=$?
     case $ReturnValue in
            1) # Cancel clicked, do nothing, and return to the menu
                     return;;
            255) # X button clicked, do nothing, and return to the menu
                     return;;
      esac
      # return value of FileType to calling assignment statement.
      echo "$FileType"
}
# This function inventories the specified folder for files of FileType as passed in.
function MakeFileList {
     # Load variables with passed in values
     Directory=$1
     echo "$Directory"
     cd $Directory
     FileType=$2
     Recursive=$3 # True or False, whether or not to search subfolders for files of FileType
     if [ "$Recursive" != "True" ]
        then
            ls -x -1 *.$FileType > "$Directory/FileList.txt"
            FileList=""
            # loop through FileList.txt adding the path to the front of each filename.
            while read EachFile; do
                  ThisPathandName="$Directory/$EachFile"
                  ThisPathandName=${ThisPathandName// /%20}
                  if [ "$ThisPathandName" != "$Directory/" ]
                    then
                         # Force a line feed in the file list.
                         FileList="$FileList
$ThisPathandName"
                  fi
             done < "$Directory/FileList.txt"
        else
             FileList=`find -type f -name \*.$FileType`
             # URL encode the filenames so they work with the comboboxes.
             FileList=${FileList// /%20}
             # Get rid of ./ at front of filename and path.
             FileList=${FileList//.\//$Directory\/}
     fi
     # Return value of FileList to calling assignment statement.
     echo "$FileList"
}
# This function allows the user to specify the folder he wishes to work in.
function SelectFolder {
     # Prompt user to select a folder to work in.
     SelectedFolder=`Xdialog --auto-placement --stdout --title "MDBOSR - Select Working Folder" --no-buttons --dselect "$DirectoryName" 0 0`
     echo "$SelectedFolder"
}
function SelectFileSet {
     # Assign passed in values to local variables
      FileList="$1"
     # Clean up old file.
     rm -f "/home/$USERNAME/.MDBOSearchandReplace/FileList.txt"
     # Write FileList to a temporary file.
     echo "$FileList" >  "/home/$USERNAME/.MDBOSearchandReplace/FileList.txt"
     # Generate an array of values to pass to the  check list widget.
     LoopCount=0
     ArgumentCount=1
     # Loop through the file, adding values to the argument array
     while read EachFile; do
            if [ "$EachFile" != "/home/$USERNAME/.MDBOSearchandReplace" ] && [ "$EachFile" != "" ]
               then
                    # Write $EachFile to tag argument position in array
                    Arguments[$ArgumentCount]="$EachFile"
                    ArgumentCount=$(( $ArgumentCount + 1 ))
                    # Write value of LoopCount to the item argument position in the Arguments array.
                    Arguments[$ArgumentCount]=$LoopCount
                    ArgumentCount=$(( $ArgumentCount + 1 ))
                    # Write "off" to the Status arguments array position.
                    Arguments[$ArgumentCount]="off"
                    ArgumentCount=$(( $ArgumentCount + 1 ))
                    LoopCount=$(( $LoopCount + 1 ))
            fi
     done < "/home/$USERNAME/.MDBOSearchandReplace/FileList.txt"
     # Present user with a check list of files to select from.
     SelectedFile=`Xdialog --auto-placement --stdout --separator "\n" --title "MDBOSR - Select Working File Set" --radiolist "Select the file set you want to broadcast changes through." 20 80 16 "${Arguments[@]}"`
     echo "$SelectedFile"
}
# This function allows the user to select the files they want changes applied to from a checklist
function SelectFiles {
     # Assign passed in values to local variables
      FileList="$1"
     # Clean up old file.
     rm -f "$DirectoryName/FileList.txt"
     # Write FileList to a temporary file.
     echo "$FileList" > "$DirectoryName/FileList.txt"
     # Generate an array of values to pass to the  check list widget.
     LoopCount=0
     ArgumentCount=1
     # Loop through the file, adding values to the argument array
     while read EachFile; do
            if [ "$EachFile" != "" ] && [ "$EachFile" != "$Folder" ]
                 then 
                       # Write $EachFile to tag argument position in array
                       Arguments[$ArgumentCount]="$EachFile"
                       ArgumentCount=$(( $ArgumentCount + 1 ))
                       # Write value of LoopCount to the item argument position in the Arguments array.
                       Arguments[$ArgumentCount]=$LoopCount
                       ArgumentCount=$(( $ArgumentCount + 1 ))
                       # Write "off" to the Status arguments array position.
                       Arguments[$ArgumentCount]="off"
                       ArgumentCount=$(( $ArgumentCount + 1 ))
                       LoopCount=$(( $LoopCount + 1 ))
            fi
     done < "$DirectoryName/FileList.txt"
     # Present user with a check list of files to select from.
     SelectedFiles=`Xdialog --auto-placement --stdout --separator "\n" --title "MDBOSR - Select Working Files" --checklist "Select all of the files you want to broadcast changes through." 20 80 16 "${Arguments[@]}"`
     echo "$SelectedFiles"
}
# This function allows the user to specify the Search Term to be replaced by Replace Term.
function DefineSearchTerm {
     touch "$DirectoryName/dummy.txt"
     SearchTerm=`Xdialog --auto-placement --stdout --title "MDBOSR - Paste, or Type in a Search Term Block" --editbox "$DirectoryName/dummy.txt" 30 80`
     rm -f "$DirectoryName/dummy.txt"
     echo "$SearchTerm"
}
# This function allows the user to specify the Replace Term to replace Search Term.
function DefineReplaceTerm {
     touch "$DirectoryName/dummy.txt"
     ReplaceTerm=`Xdialog --auto-placement --stdout --title "MDBOSR - Paste, or Type in a Replace Term Block" --editbox "$DirectoryName/dummy.txt" 30 80`
     rm -f "$DirectoryName/dummy.txt"
     echo "$ReplaceTerm"
}
# Main program starts here.
# Initialize the directory name of the current working folder as a global variable.
DirectoryName="/home/$USERNAME/.MDBOSearchandReplace"
if [ ! -d "/home/$USERNAME/.MDBOSearchandReplace" ]
     then
          mkdir "/home/$USERNAME/.MDBOSearchandReplace"
fi
UndoLevel=0
ReplaceTerms=[]
SearchTerms=[]
UndoTypes=[]
# Start the Main Menu loop.
while [ "$MenuChoice" != "Exit" ]; do
     # Pop up the menu.
     MenuChoice=`Xdialog --auto-placement --stdout --title "Multi Document Block Oriented Search Replace" --no-tags --menubox "Please make a selection below." 20 60 18 "FileType" "Select type of files to work on" "Folder" "Select Working Folder" "SetRecursive" "Set Folder Search to Recursive." "SetNoRecurse" "Set Folder Search to Non Recursive" "SelectFiles" "Select Files to Make Changes to" "SaveFileList" "Save File List and Settings Currently Selected" "LoadFileList" "Load Saved File List and Settings" "SetSearch" "Set Search Term" "SetReplace" "Set Replace Term" "Replace" "Replace Single Occurrence of Search Term in All Files" "ReplaceAll" "Replace All Occurrences of Search Term in All Files" "Undo" "Undo Last Search and Replace Operation" "Exit" "Exit this program."`
# test for Cancel, or X button click.
ReturnValue=$?
case $ReturnValue in
     1) # Cancel clicked, do nothing, and exit program
       exit;;
 255) # X button clicked, do nothing, and exit program.
       exit;;
esac
# Test Menu Choice, and branch accordingly.
case $MenuChoice in
       "FileType")
                       FileType=`SelectFileType`;;
          "Folder")
                       Folder=`SelectFolder`;;
        "SetRecursive")
                       Recursive="True";;
        "SetNoRecurse")
                       Recursive="False";;
         "SelectFiles")
                       FileList=`MakeFileList $Folder $FileType $Recursive`
                       FileList=`SelectFiles "$FileList"`;;
      "SaveFileList")
                       FolderName=${Folder/\/home\/$USERNAME\//}
                       FolderName=${FolderName//\//}
                       echo $Recursive > /home/$USERNAME/.MDBOSearchandReplace/$FolderName.$FileType.set
                       echo $Folder >> /home/$USERNAME/.MDBOSearchandReplace/$FolderName.$FileType.set
                       echo $FileType >> /home/$USERNAME/.MDBOSearchandReplace/$FolderName.$FileType.set
                       echo "$FileList" > /home/$USERNAME/.MDBOSearchandReplace/$FolderName.$FileType.set.FileList;;
      "LoadFileList")
                       Directory="/home/$USERNAME/.MDBOSearchandReplace"
                       FileSetLists=`MakeFileList $Directory "set" "False"`
                       FileSet=`SelectFileSet "$FileSetLists"`
                       FileList=`cat "$FileSet.FileList"`
                       LoopCount=1
                       while read ThisLine; do
                              if [ $LoopCount == 1 ]
                                   then
                                        Recursive=$ThisLine
                              fi
                              if [ $LoopCount == 2 ]
                                   then
                                        Folder=$ThisLine
                             fi
                             if [ $LoopCount == 3 ]
                                   then
                                        FileType=$ThisLine
                             fi
                             LoopCount=$(( $LoopCount + 1 ))
                      done <"$FileSet"
                      ReplaceTerms=[]
                      SearchTerms=[]
                      UndoTypes=[];;
         "SetSearch")
                       SearchTerm=`DefineSearchTerm "$FileList"`;;
         "SetReplace")
                       ReplaceTerm=`DefineReplaceTerm "$FileList"`;;
         "Replace")
                       echo "$FileList" > "$DirectoryName/FileList.txt"
                       while read ThisFile; do
                              ThisFile=${ThisFile//%20/ }
                              FileContents=`cat "$ThisFile"`
                              FileContents=`DoReplace "$FileContents" "$SearchTerm" "$ReplaceTerm" "False"`
                              echo "$FileContents" > "$ThisFile"
                       done < "$DirectoryName/FileList.txt"
                       ReplaceTerms[$UndoLevel]="$ReplaceTerm"
                       SearchTerms[$UndoLevel]="$SearchTerm"
                       UndoTypes[$UndoLevel]="Single"
                       UndoLevel=$(( $UndoLevel + 1 ));;
         "ReplaceAll")
                       echo "$FileList" > "$DirectoryName/FileList.txt"
                       while read ThisFile; do
                              ThisFile=${ThisFile//%20/ }
                              FileContents=`cat "$ThisFile"`
                              FileContents=`DoReplace "$FileContents" "$SearchTerm" "$ReplaceTerm" "True"`
                              echo "$FileContents" > "$ThisFile"
                       done < "$DirectoryName/FileList.txt"
                       ReplaceTerms[$UndoLevel]="$ReplaceTerm"
                       SearchTerms[$UndoLevel]="$SearchTerm"
                       UndoTypes[$UndoLevel]="True"
                       UndoLevel=$(( $UndoLevel + 1 ));;
            "Undo")
                       echo "$FileList" > "$DirectoryName/FileList.txt"
                       UndoLevel=$(( $UndoLevel - 1 ))
                       if [[ $UndoLevel -gt -1 ]]
                            then
                                 while read ThisFile; do
                                        ThisFile=${ThisFile//%20/ }
                                        FileContents=`cat "$ThisFile"`
                                        FileContents=`DoReplace "$FileContents" "${ReplaceTerms[$UndoLevel]}" "${SearchTerms[$UndoLevel]}" "${UndoTypes[$UndoLevel]}"`
                                        echo "$FileContents" > "$ThisFile"
                                 done < "$DirectoryName/FileList.txt"
                            else
                                 UndoLevel=$(( $UndoLevel + 1 ))
                                 xmessage "The maximum depth of undo levels has been exceeded, nothing to undo."
                       fi;;
              "Exit")
                       rm -f "$DirectoryName/FileList.txt"
                       exit;;
esac
done

Download Multi Document Block Oriented Search and Replace script, version 1.5, 2010-04-16.

To use the program you need to do several things in a certain order to set up the file set you want to work on. Here is a screen shot of the main menu of Multi Document Block Oriented Search Replace.

The first thing you need to do is set the type of files you would like to work on, so select the first menu item "Select type of files to work on", and click OK. You will be presented with a drop down list of many popular file types.

The drop down list is editable, so if the file extension you want to work on is not in the list, simply click in the box, and type in the extension of the type of files you would like to work on.

Next you will need to set the working folder. This is the folder where the files you want to work on reside. There are two settings for searching for files you want to work on. One setting is recursive, which means to search sub folders of the working folder for files as well as searching the working folder for files.

The other setting is non recursive which means to only search the set working folder for files to work on. There are two menu entries for changing this setting. To set the working folder select the menu entry "Select Working Folder".

You will get the dialog shown at the right. Navigate your way to the folder that contains the files you want to work on, and click OK on the folder selection dialog.

Once you have all of those settings made it is time to "Select Files to Make Changes to". This menu entry selection presents you with a check list of all of the files found that match the extension type setting, in the folders you have selected. Click each file that you want to add to the working set in the list, then click OK.

Once you have selected all of the files you want in this working set, you may now select the "Save File List and Settings Currently Selected" menu entry to create a new working file set description.

The next time you want to work on the same file set, you only need to use the "Load Saved File List and Settings" menu entry to work on one of your previously saved working file sets, then immediately be able to define your search, and replace terms for your changes.

Now it is time to define the block of text you would like to search for and replace. Select "Set Search Term" from the main menu, and click OK. You will be presented with a text input dialog to paste, or type your search term into.

You will have to press CTRL-V to paste your blocks of text into the search term, and replace term dialogs, as Xdialog widgets do not support context menus for your mouse.

The same procedure applies to the menu entry "Set Replace Term", with it presenting you with a dialog to paste the replace term into. Paste, or type in your replace term, and click the OK button.

Now you need to select the appropriate menu entry from the two "Replace Single Occurrence of Search Term in All Files", or "Replace All Occurrences of Search Term in All Files". Either of these menu entries will begin the actual process of doing the search and replace operation across all of the files you have selected for a working set.

Have some patience as the menu disappears, and the script works on replacing the search term in all of the files with the replace term, as it can take a long time for this operation to complete on a large file set. One it is done, the main menu will appear again, and you will know that the operation is complete.

If you make a mistake, there is a multiple level undo menu entry, which will take you backwards through your previous replacements for this run of the program on the current file set, one at a time, undoing each replacement in turn, until all undo levels have been exhausted.

You may make changes to as many files as you select for the working set. You can select as few as one file to work on, or as many as you think this script can handle. I've successfully made changes to around 60 files at once with it, which took around five minutes to complete with a large block of text being searched for and replaced. That's a lot faster than the hours it used to take me to make a change of this magnitude.

I have no way to test it for a larger file set than that, as I do not have a convenient large set of files for the script to work on, other than the web pages for my web sites, so I have no idea what the limits are as far as how many files can be worked on at once. If you find a situation where you have selected more files than the script will handle please email me, and let me know how many files you tried when it failed, and how large the files were.

I hereby release MultiDocBlockSearchReplace.sh as open source software for Linux computers. I have thoroughly tested it under Ubuntu 8.04 Hardy Heron, and have no idea if it will work on other Linux distributions. Give it a try, and let me know if it works in your flavor of Linux.

Download Multi Document Block Oriented Search and Replace script, version 1.5, 2010-04-16.

Help me keep writing code for others to use. Please consider making a donation of whatever you can afford if you use my scripts, even something as small as one dime would help a lot. Thanks.

Last modified on:


Hover over buttons to preview skins, click to set them permanently.