Web masters, are you sick and tired of editing your Google site map file every time you write a new web page?
This Linux script automates the process of adding new urls to your google_sitemap.xml file. You will no longer have to open your site map in an editor to manually add URLs to your site map. It is GUI enabled, so it's very easy to use.
This script depends on the Xdialog package for the GUI widgets it uses to interact with it's users. To install Xdialog issue the following command in Terminal.
sudo apt-get install xdialog
To use the script, simply decompress it from the .zip file into any folder in your home folder. You can then add the following command to a menu entry to start it, or you can issue the command from Terminal as well.
/home/username/folder/path/AddURLtoSitemap.sh
Replace "username", "folder", and "path" with the actual folder path as it exists on your computer.
Here is the source code for the AddURLtoSitemap.sh script file.
#!/usr/bin/env bash
# This script automates the process of adding new urls to your google_sitemap.xml file. The script generates a configuration
# file from user input to keep track of your domain names, and local site map file locations, for each of your web sites.
#
# Using this script is as easy as typing, or pasting in a new web page file name, and selecting the web site you will be adding it to. The rest is done for you!
#
# Usage - /home/username/foldername/AddURLtoSitemap.sh
#
# This script takes no command line parameters, as all data entry is accomplished via GUI dialog boxes provided by the
# Xdialog package.
#
#
function ReplaceSpaces {
# loop through command line parameters, concatenating them with %20
ThisLine="$1"
shift
while [ "$1" != "" ]
do
ThisLine="$ThisLine%20$1"
shift
done
return
}
# This function removes individual URLs from the site map file the user selects.
function RemoveURL {
# Prompt for which web site's map to remove URL from
# build list of web sites from configuration file to feed to combo box for prompting the user
# which site to remove from the configuration file.
SiteList=""
LineCount=0
while read ThisLine; do
(( LineCount +=1 ))
Remainder=$(( $LineCount % 2 ))
# if this line number is an odd number
if [ $Remainder -ne 0 ]
then # concatenate this line to site list for presentation to combo box widget
SiteList="$SiteList $ThisLine"
fi
done < "WebSites.conf"
# Prompt user to select web site to remove with a drop down list
WebSite=`Xdialog --stdout --title "AddURL to Sitemap - Remove URL" --combobox "Select web site to remove URL from." 6 70 $SiteList`
# test for Cancel, or X button click.
ReturnValue=$?
case $ReturnValue in
1) # Cancel clicked, do nothing
return;;
255) # X button clicked, do nothing
return;;
esac
# match up web site with site map filename
# build array of lines from WebSites.conf
LoopCount=0
while read ThisLine; do
Parameters[$LoopCount]=$ThisLine
(( LoopCount +=1 ))
done < WebSites.conf
# call GetMapName to match up web site with site map name. Entire Parameters array is being fed to the function as
# command line parameters.
MapName=`GetMapName $WebSite ${Parameters[@]}`
# Loop through site map picking out the lines with urls in them.
URLs=""
while read ThisLine; do
case "$ThisLine" in
*\*) # if line starts with concatenate it into string variable URLs.
# remove spaces from the url, and replace them with %20
ReplaceSpaces $ThisLine
URLs="$URLs $ThisLine";;
esac
done < "$MapName"
# Prompt user with drop down list of urls to select one to remove from site map.
SelectedLine=`Xdialog --stdout --title "Add URL to Site Map - Remove URL" --combobox "Select web page to remove from site map" 10 70 $URLs`
# test for Cancel, or X button click.
ReturnValue=$?
case $ReturnValue in
1) # Cancel clicked, do nothing
return;;
255) # X button clicked, do nothing
return;;
esac
# Read site map file into arrays for processing.
ReadSiteMap $MapName
# write new site map file without the offending url within it.
HeaderLines=${Header[0]}
LoopCount=1
# write the beginning of the file...
echo "${Header[$LoopCount]}" > $MapName
while [ $LoopCount -ne $HeaderLines ]; do
# increment LoopCount
LoopCount=$(( $LoopCount + 1 ))
# write all header lines to file
echo "${Header[$LoopCount]}" >> $MapName
done
# Write main body of file without offending entry.
# Loop through arrays, writing the body of the file
LoopCount=0
LoopLimit=${Location[0]}
while [ $LoopCount -lt $LoopLimit ]; do
LoopCount=$(( $LoopCount + 1 ))
if [ "${Location[$LoopCount]}" == "$SelectedLine" ]
then LoopCount=$(( $LoopCount + 1 ))
fi
if [ "${Location[$LoopCount]}" != "" ] && [ "${Location[$LoopCount]}" != "$SelectedLine" ]
then
echo "" >> $MapName
echo "${Location[$LoopCount]}" >> $MapName
if [ "${LastMod[$LoopCount]}" != "" ]
then echo "${LastMod[$LoopCount]}" >> $MapName
fi
if [ "${ChangeFrequency[$LoopCount]}" != "" ]
then echo "${ChangeFrequency[$LoopCount]}" >> $MapName
fi
if [ "${Priority[$LoopCount]}" != "" ]
then echo "${Priority[$LoopCount]}" >> $MapName
fi
echo " " >> $MapName
fi
done
# finish off the file with the closing urlset tag.
echo "" >> $MapName
}
# This function allows user to select a web site from the configuration file to remove from the configuration file
# and removes that web site's information from the WebSites.conf file.
function RemoveSiteFromConfig {
# Check to make sure the WebSites.conf file is not empty, and return to the main menu if it is.
FileCheck=`cat WebSites.conf`
if [ "$FileCheck" == "" ]
then
Xdialog --title "Configuration File Empty" --msgbox "There are no web sites currently configured in WebSites.conf, please add some before attempting to remove any again." 0 0
return
fi
# build list of web sites from configuration file to feed to combo box for prompting the user
# which site to remove from the configuration file.
SiteList=""
LineCount=0
while read ThisLine; do
(( LineCount +=1 ))
Remainder=$(( $LineCount % 2 ))
# if this line number is an odd number
if [ $Remainder -ne 0 ]
then # concatenate this line to site list for presentation to combo box widget
SiteList="$ThisLine $SiteList"
fi
done < "WebSites.conf"
# Prompt user to select web site to remove with a drop down list
WebSite=`Xdialog --stdout --title "AddURL to Sitemap - Remove Web Site" --combobox "Select web site to remove from configuration file." 0 0 $SiteList`
# test for Cancel, or X button click.
ReturnValue=$?
case $ReturnValue in
1) # Cancel clicked, do nothing
return;;
255) # X button clicked, do nothing
return;;
esac
# get directory script started from
DirectoryName=`dirname $0`
# remove the web site selected from the configuration file
Skip="No"
touch "$DirectoryName/temp.txt"
while read Line; do
# if this line from the configuration file isn't the domain name to be removed, or the next line after it
if [ "$Line" != "$WebSite" ] && [ "$Skip" != "Yes" ]
then
echo "$Line" >> "$DirectoryName/temp.txt"
else if [ "$Line" == "$WebSite" ]
then # set up to skip the next line because it is a site map name, not a domain name.
Skip="Yes"
else # we previously skipped the appropriate line, so reset Skip variable.
Skip="No"
fi
fi
done < "$DirectoryName/WebSites.conf"
# over write old WebSite.conf with newly created dile
cat temp.txt > WebSites.conf
# remove temporary file
rm -f temp.txt
}
# The following function add a new web site's information to the configuration file.
function AddSiteToConfig {
# prompt user to supply the domain name of the new web site.
Response=`Xdialog --stdout --title "Add URL to Sitemap - Add Web Site to Configuration" --inputbox "Please type, or paste in the full domain name of this new web site." 0 0`
# test for Cancel, or X button click.
ReturnValue=$?
case $ReturnValue in
1) # Cancel clicked, do nothing
return;;
255) # X button clicked, do nothing
return;;
esac
# place the user's response in the configuration file.
echo "$Response" >> WebSites.conf
DirectoryName=`dirname $0`
# prompt the user to supply the location, and name of their local google_sitemap.xml file.
Xdialog --title "Select your google_sitemap.xml file for this site" --no-buttons --fselect "$DirectoryName/*" 0 0 2> response.txt
# place the user's response into the configuration file
cat response.txt >> WebSites.conf
# remove temporary file
rm -f "response.txt"
return;
}
# The following function gets the correct site map file name and path for the web site that has been selected.
function GetMapName {
# WebSite domain name is passed in as $1
WebSite=$1
# shift the domain name out of $1
shift
# loop until the domain name value matches the most recently shifted parameter
while [ "$WebSite" != "$1" ]
do
shift
done
# shift once more to access the site map name from the passed in parameters..
shift
# assign the site map file name to a variable.
MapName=$1
echo "$MapName"
return;
}
function ReadSiteMap {
# Read Site map into arrays, site map name is passed in as $1.
MapName="$1"
URLCount=0
HeaderLines=0
while read ThisLine; do
# Load different arrays depending on what each line is
case $ThisLine in
*\*)
echo "nothing" >/dev/null;;
*\*)
URLCount=$(( $URLCount + 1 ));;
*\ *)
echo "nothing" >/dev/null;;
*\*)
Location[$URLCount]="$ThisLine";;
*\*)
LastMod[$URLCount]="$ThisLine";;
*\*)
ChangeFrequency[$URLCount]="$ThisLine";;
*\*)
Priority[$URLCount]="$ThisLine";;
*\*)
echo "nothing" >/dev/null;;
*)
HeaderLines=$(( $HeaderLines + 1 ))
Header[$HeaderLines]="$ThisLine";;
esac
done <"$MapName"
Location[0]=$URLCount
Header[0]="$HeaderLines"
}
# The following function adds a new web page url to the Google site map for the proper site.
function AddNewURL {
# initialize loop counter.
LoopCount=0
# read configuration file to the end of file.
while read EachLine
do
# check to see if 3 divides evenly into the loop counter plus one.
Remainder=$(( ( $LoopCount + 1 ) % 3 ))
# if loop counter is evenly divisible by three, and loop counter is not equal to zero
if [ "$Remainder" -eq "0" ] && [ "$LoopCount" -ne "0" ]
then
# add every third parameter of "off" to the concatenation array
ListItems[$LoopCount]="off"
# increment loop counter
LoopCount=$(( $LoopCount + 1 ))
# add this loops read in line to the concatenation array.
ListItems[$LoopCount]=$EachLine
# increment loop counter.
LoopCount=$(( $LoopCount + 1 ))
else
# just add this read in line to the concatenation array.
ListItems[$LoopCount]=$EachLine
# increment loop counter.
LoopCount=$(( $LoopCount + 1 ))
fi
# read one line at a time from the website configuration file.
done < WebSites.conf
# write last parameter to the ListItems array variable which is used in the following dialog call to pass in it's parameters.
ListItems[$LoopCount]="off"
# prompt user to select which web site the new web page is being added to.
WebSite=`Xdialog --stdout --title "AddURL to Sitemap - Web Site Selection" --radiolist "Please select the web site this new link is for." 10 70 0 "${ListItems[@]}"`
# test for Cancel, or X button click.
ReturnValue=$?
case $ReturnValue in
1) # Cancel clicked, do nothing
return;;
255) # X button clicked, do nothing
return;;
esac
# prompt the user for the name of a new web page to add to a site map.
WebPage=`Xdialog --stdout --title "AddURL to Sitemap - Enter New File Name" --inputbox "Please type, or paste in \"folder/path/FileName.ext\" for your new web page." 0 0`
# test for Cancel, or X button click.
ReturnValue=$?
case $ReturnValue in
1) # Cancel clicked, do nothing
return;;
255) # X button clicked, do nothing
return;;
esac # place user's response in a variable.
# test for proper input
case $WebPage in
/*) # started with a slash, not correct
Xdialog --title "Incorrect input" --msgbox "Path must not begin with a / character."
return;;
*)
# match the WebSite name up with the correct site map name.
MapName=`GetMapName $WebSite ${ListItems[@]}`
# Read in the site map into arrays
ReadSiteMap $MapName
# set array variable to last item in the array plus one
ArrayPosition=$(( ${Location[0]} + 1 ))
# Assign the new page's value to the last position in the locations array
Location[ArrayPosition]="http://$WebSite/$WebPage "
# Prompt user for change frequency
ChangeFreq=""
ChangeFreq=`Xdialog --stdout --title "Add URL to Sitemap - Select Change Frequency" --combobox "Select change frequency from the drop down list." 0 0 "always" "hourly" "daily" "weekly" "monthly" "yearly" "never"`
if [ "$ChangeFreq" != "" ]
then
ChangeFrequency[$ArrayPosition]="$ChangeFreq "
fi
LastModified=""
LastModified=`Xdialog --stdout --title "Add URL to Sitemap - Enter Last Modified" --inputbox "Input the last modified date in the form YYYY-MM-DD." 0 0`
if [ "$LastModified" != "" ]
then LastMod[$ArrayPosition]="$LastModified "
fi
PrioritySelected=""
PrioritySelected=`Xdialog --stdout --title "Add URL to Sitemap - Select Priority" --combobox "Select page priority from the drop down list." 0 0 "0.0" "0.1" "0.2" "0.3" "0.4" "0.5" "0.6" "0.7" "0.8" "0.9" "1.0"`
if [ "$PrioritySelected" != "" ]
then
Priority[$ArrayPosition]="$PrioritySelected "
fi
Location[0]=$ArrayPosition
# loop through the site map arrays, writing the new site map as we go.
LoopCount=1
HeaderLines=${Header[0]}
echo "${Header[$LoopCount]}" > $MapName
while [ $LoopCount -ne $HeaderLines ]; do
# increment LoopCount
LoopCount=$(( $LoopCount + 1 ))
# write all header lines to file
echo "${Header[$LoopCount]}" >> $MapName
done
# Loop through arrays, writing the body of the file
LoopCount=0
while [ $LoopCount -ne ${Location[0]} ]; do
LoopCount=$(( $LoopCount + 1 ))
echo "" >> $MapName
echo "${Location[$LoopCount]}" >> $MapName
if [ "${LastMod[$LoopCount]}" != "" ]
then echo "${LastMod[$LoopCount]}" >> $MapName
fi
if [ "${ChangeFrequency[$LoopCount]}" != "" ]
then echo "${ChangeFrequency[$LoopCount]}" >> $MapName
fi
if [ "${Priority[$LoopCount]}" != "" ]
then echo "${Priority[$LoopCount]}" >> $MapName
fi
echo " " >> $MapName
done
echo "" >> $MapName
return;;
esac
}
# Start the main program here.
# change directory to the one this script was executed from.
cd `dirname $0`
# Declare, and initialize some global arrays.
Location[0]=0
LastMod[0]=0
ChangeFrequency[0]=0
Priority[0]=0
Header[0]=0
MenuChoice=" "
#
while [ "$MenuChoice" != "Exit" ]
do
# call the first dialog, which is the main menu
MenuChoice=`Xdialog --stdout --title "Add URL to Google Site Map" --menubox "Please select any menu entry." 12 60 0 "AddSite" "Add a new web site to the configuration file." "RemoveSite" "Remove web site from configuration file" "NewURL" "Add new web page url to one of the site maps." "RemoveURL" "Remove web page URL from a site map" "Exit" "Shut this program down."`
# test for Cancel, or X button click.
ReturnValue=$?
case $ReturnValue in
1) # Cancel clicked, do nothing
exit;;
255) # X button clicked, do nothing
exit;;
esac
# branch to the proper routine for the selected menu choice.
case "$MenuChoice" in
"AddSite" )
AddSiteToConfig;;
"RemoveSite")
RemoveSiteFromConfig;;
"NewURL" )
AddNewURL;;
"RemoveURL")
RemoveURL;;
"Exit" )
clear
exit;;
esac
done
exit
Download Add URL to Google Site Map script.
Here is a screen capture of the main menu of Add URL to Google Site Map. There are only five menu entries.
The first menu entry is "Add a new web site to the configuration file". This menu entry saves a file called WebSite.conf in which it keeps two lines for each configured web site. Line one is the domain name of each web site, and line two of each entry is the path to, and name of a google_sitemap.xml file for that web site on your local machine.
It is possible that this script will work with site maps that have a different file name than google_sitemap.xml, but before you try, make sure that the site map you want to use is a standard .xml file with <urlset>, and <url> tags as in the following sample single entry site map file.
<?xml version='1.0' encoding='UTF-8'?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd"> <url> <loc>http://rayslinks.com/index.html</loc> <lastmod>2009-11-01</lastmod> <changefreq>weekly</changefreq> <priority>1.0</priority> </url> </urlset>
When you use the first menu entry, you will be first presented with a dialog that asks for the new web site's domain name. After typing, or pasting in the new domain name, and clicking OK, you will be presented with a file selection dialog to select the site map file for that web site.
You may click the Cancel button, or the title bar X button on any dialog in Add URL to Google Site Map to return to the main menu without completing the action started by the menu entry you chose.
Once you have added web sites to the configuration file, you may remove them from it with the second menu entry "Remove Web site from configuration file".
You may of course edit WebSite.conf manually to add and remove web sites, but it is easier with the script. If you do manually edit WebSite.conf, make sure that you do not save it with one empty line at the top of it, or this script will not function properly.
If you wish to manually empty WebSite.conf completely, the best manual method is to simply delete the file, and the script will rebuild it the next time it is ran.
After selecting the second menu entry, you will be prompted with a drop down list of domain names that you may select from for removal from the configuration file.
Select the web site you wish to remove from the drop down list, and click the OK button to effect the removal.
The third menu entry "Add new web page URL to one of the site maps", provides the way to add a new web page's URL to one of your site maps.
After selecting the third menu entry, you will be presented with a radio button list of web sites to choose from for the new page to be added to.
After selecting a web site from the radio button list, and clicking the OK button, you will be presented with a text entry dialog to enter the new web page's folder path, and file name into.
Make sure that you do not put a leading "/" [forward slash] in the path, and file name entry dialog. Entries should be of the form -
Folder/Path/Filename.ext
The Add URL to Google Sitemap script will add the "http://www.somesite.com/" portion of the URL as it adds it to the site map file.
After entering the new file's name, you will be prompted for the change frequency of the page, the priority of the page, and the last modified date of the page in turn. To leave any of these values blank, simply click the "Cancel" button on the dialog that prompts you for them.
The fourth menu entry "Remove web page URL from a site map" allows you to specify URLs to be removed from one of your site map files.
When you select this menu entry, you are first presented with a drop down list of web sites to choose from. Select the one you are retiring the web page on, and click OK.
You will then be presented with another drop down list of possible URLs to remove from that site map. Select the one you want to remove, and click OK.
I have this script set up to replace any space characters in your site map URL's filenames with their equivalent %20. I had to do this to make the script run correctly, and it is better practice to use the %20 designation for a space character in your site map files anyway.
The last menu entry, 'Shut this program down" of course just exits Add URL to Google Site Map.
I hereby release AddURLtoSitemap.sh as open source software for Linux computers. I have thoroughly tested it under Ubuntu 8.04 Hardy Heron, and have no idea if it will work on other Linux distributions. Give it a try, and let me know if it works in your flavor of Linux.
Download Add URL to Google Site Map script.
Help me keep writing code for others to use. Please consider making a donation of whatever you can afford if you use my scripts, even something as small as one dime would help a lot. Thanks.