Having the ability to get information from a web site and incorporate it into the function of an app can open up a completely new world of app posibilities for you.
This article offers an introduction to retrieving information from a web site and analyzing the retrieved information so that it can be used by a Window-Eyes app. This article will use VBScript as the scripting language, and XML as the form in which the web site will use to pass the retrieved information to your app. (Note that often such information is available in the JSON format (in addition to, or instead of, XML).
While it is possible to examine just the text of a web page (or its HTML) to try and retrieve the information you require, your app will be easier to write, and much more reliable, if you find a web site which provides the information you want in a specialized format: one designed specifically for communicating the information to other computer programs. While various schemes have been used, currently one known as REST (Representational state transfer), which uses the HTTP protocol to download information in either the XML or JSON format, is by far the most popular, and therefore the most likely to be documented and supported. The first problem you will then have to face is to find a web site which does offer the needed information in such a specialized format. Once you have the source of the information you need, you can learn to utilize the HTTP protocol and the XML format to get the information into your app directly from the web site.
Contents |
Using HTTP
Windows provides a built-in object which makes it very easy for you to use the HTTP protocol in order to perform such tasks as communicating with a web site, so that you can request to download the information that you need. The requests this object lets you perform (known as the "verbs" which you can use) are generally either of two: an HTTP "get" and an HTTP "post".
Either type of request starts with the basic URL of the web site (the address you would enter into a browser in order to bring up a web page). Either type can then add additional parameters to the end of the URL, after a special character (usually the question mark) which separates the URL from the parameters that you are adding. The basic difference between these two types of HTTP commands is that a "post" typicly also includes information stored inside of the body of the HTTP request object (often as an XML document) (the HTTP request object takes care of transmitting this additional information to the web server). A "get" command consists only of the URL and the parameters being sent to the web server; therefore, all the information the web server may need in order to fullfill your request will need to be contained in the list of parameters, while in a "post" command such information can be split between both parameters and an internal XML document, or contained entirely in the internal XML document. One reason for placing information inside of an XML document is that the URL along with its optional parameters is limitted in length to approximately 2048 characters (some web servers may document a much lower maximum length).
The parameters you will need to specify, just like the URL, are specific to the web site that you are requesting the information from. You obviously will need some type of documentation from this web site on the parameters and values it may expect to be added to the end of the URL. Such parameters are usually separated from the URL by a question mark, and separated from each other with the ampursand (the "and" sign). They most often take the form of a parameter name, an equals sign, and then a value (although you can also have just the parameter name, without any following equals sign and value).
If the web site you are working with expects a "get" command, you can test out your understanding of the proper URL and parameters by typing them into your browser as the address of the page you wish it to display. The web server will then interpret the URL and parameters you used, and cause your browser to display the XML (or HTML) results that it would return to your app as the answer to your command. For example, the URL and parameters below will cause the information which script central holds for an app named "UIDesign" to be returned as an XML document; you can enter this URL into your browser and it should show you the XML document that it would return:
https://www.gwmicro.com/App_Central/scriptTools.php?scriptid=1326&showXML&showChanges
Note that this command specifies 3 parameters: scriptid with a value of 1326, showXML, and showChanges (the last two parameters specify conditions to be used simply by their presence).
I found this URL and its parameters by going to the web page for the UIDesign app, and choosing a link on that page for its "XML feed"; then I observed the URL of the page which was displayed by my browser. As far as I know, this feature of App Central is not documented anywhere. This is often the case when you are looking for specific XML data, and you may have to play "detective" in tracking down the URL and parameters you will need to use in order to perform your query.
Handling Special Characters
You can see from the example above that the question mark, the equals sign, the comma, and the ampursand all are in use when specifying the parameters of an HTTP request. An immediate problem is presented if you needed to use a parameter which required you to specify a string value and not just a number, and this string value needed to contain one of these characters already in use. In order to solve this problem, you must not use any of the special characters in your parameter values; instead, you must convert them to some other representation which the web server will understand, and convert back to the original form. This is refered to as "URL encoding" or "escaping" these parameter values. This is necessary even for a space character.
Luckily VBScript has a function named "escape(<input string>)" which takes as input the string of a parameter value, and provides as output a string where every special character is encoded into a numeric format with a preceeding percent sign. It's this "escaped" string which you then need to specify as a parameter value on the HTTP request command line. For instance, if you needed to specify a parameter called "name", and the value you wished to use was "GW Micro", you would have to use the form:
&name=GW%20Micro
as your parameter. Also, it is permitted to use a plus sign to replace the space character as in the example below:
&name=GW+Micro
Automating the use of HTTP
Windows provides a flexible object which allows you to interact with a web site via the HTTP protocol named the WinHTTPRequest object. To make use of it, you first instantiate it as shown below:
Set loHTTPRequest = CreateObject("WinHttp.WinHttpRequest.5.1")
The object is now ready for use for either a "get" or a "post" command. It can be used synchronously (it pauses after the command is sent, waiting for the answer from the web server), or asynchronously (it sends the command to the web server and your script continues executing; in this case you will need an event handler or other arrangements to handle the answer when it's available).
Before you can send the command you will need to specify the URL and the parameters:
lcRequestURL = "https://www.gwmicro.com/App_Central/scriptTools.php" lcRequestParams = "scriptid=" & escape("1326") & "&showXML&showChanges" lcHTTPCommand = lcRequestURL & "?" & lcRequestParams
The call to the method below uses the HTTP request object to open a connection to the web server for a "get" command; specifies the commandline (the URL along with its parameters), and specifies that the script is not using an asynchronous interface:
loHTTPRequest.Open "GET", lcHTTPCommand, False
And finally the call to the method of the HTTP request object below actually "sends" the command to the web server, and since we specified no asynchronous interface, it begins waiting until the answer is received back from the web server (I believe the waiting will time out after 30 seconds if no answer is received):
loHTTPRequest.send
Any answer received is stored in the HTTP request object's .responseText property (if one was received; this is indicated by the value of the .status property). HTTP has a series of pre-defined status codes with various meanings; a value of 200 indicates that an answer was received from the web server, and so is what we are always looking for; any other value indicates some sort of problem. Even if an answer is received however (and a status of 200 is returned), it does not mean that there was no problem; the answer from the web server could be the equivalent of "no such parameter", or "no such name found", etc. The next issue therefore (when a status of 200 is received) is that of interpreting the response; which as was mentioned at the top of the article, is assumed to be in XML format.
If you only have access to information returned in JSON format (and there is a definite shift in web applications towards using only JSON), and you wish to program in VBScript, see the WE shared object library app named "HomerJax" (written by Jamal Mazrui) which is designed to convert JSON objects for you into a COM object usable by a VBScript app. Alternatively, you can search the web for any of a number of JSON parsing applications written in VBScript such as this JSON VBScript parser which you could incorporate into your app.
Examining an XML response
When you receive a response from a web server designed to return XML, you will find an entire XML document in the ".responseText" property of the winHTTPRequest object. An XML document is a stream of text where a tree structure of information is represented in the text by using a series of "tags" which surround the information, identifying where the information is in the tree, and assigning it a name and optionally some attributes. Windows has several built-in objects which make it easy to read and write XML documents; one of the most commonly used can be created with the command below:
set loXMLDoc = CreateObject("MSXML2.DOMDocument.3.0")
You can then load the XML document (which was returned as the response) into this new XML document object with the following commands:
XMLDoc.async = false XMLDoc.loadXML loHTTPRequest.responseText
You now can use this object (which is also called the XML Document Object Model) to easily access the information contained in the XML document. It allows you to analyze the tree of information, finding specific "elements" (or "nodes") and there-by obtaining access to their "values" and "attributes", and also to change their contents and the tree structure. See this XML DOM tutorial for more detailed information. (Note that there is a technical difference in XML between a "node" and an "element", but these terms are often used interchangeably).
Understanding XML
You will need to know which elements will be in the XML document, and how the information will be structured, in order for your app to be able to retrieve the desired information. Often you won't have sufficient documentation, so you'll have to determine this by examining the raw XML.
If you enter the URL and parameters for the example web site mentioned above into your browser, it will show you the raw XML. When examining it, you will see a series of tags which consist of a less than sign, followed by the name of the tag, possibly followed by some optional attributes, and a greater than sign. After the tag may optionally come its value, and/or any nodes which are children of this node (much as a window in the windows tree can contain both text and child windows). Each of the child nodes will begin with their own tags just as the current node did. Each will terminate with a closing version of its tag which is a less than sign, a slash, the tag name, and a greater than sign. And finally the current node will terminate with its own closing tag.
When you find nodes declared one after the other at the same level, they are known as sibling nodes (much like members of a collection). Any time you find one node declared inside of another, then the inner node is a child node of the parent node (much like a property which belongs to the parent object).
In looking at this example the first line you will see is a special header tag (present in most XML documents) which begins with a question mark and XML; you can just ignore this header tag. The first node declared is a "wescriptupdate" node, and if you skip to the bottom of the page you will see a closing tag for this node. If you search the page for this tag, you will find it only appears at the top and the bottom of the page; so the entire document is a single node of type "wescriptupdate". If you were drawing it out, you would draw a single node at the top, of type "wescriptupdate", and everything else would be drawn as children under this single node. (note that the case of XML element names is important, unlike the case of VBScript variable names).
Examining the children of this node you will find it has a single element of the "script" type; again, if we were drawing it we would draw a single element (type "wescriptupdate") with a single child (type "script").
Looking to the children of the "script" element we finally see a list of sibling elements which are of differing types and which contain values; not only child nodes. The first one we see is an element of type "scriptid" and after its tag, instead of further tags, we actually see some text. This text is its value; that is, if it were a property of an object, this is the value the property would hold. In this example it's a string of "1326". Some of the other sibling elements to this element which we also see are "scriptname" and "version". Here are the opening tags for this example:
<wescriptupdate> <script> <scriptid>1326</scriptid> <scriptname>UI Design</scriptname> ...
Using the XML DOM
Once you understand how a particular XML document is structured, you then need to write your app to pull the needed information from the document by making use of your understanding and the XML DOM. You could do this by looping through all the elements of the XML document, testing on the name and type of each element, looking for the ones which are important to your app. However, another way you can do this is to request the specific elements which you want to extract data from (by specifying their names and the names of any parent elements), and this is more often the preferred technique. This is especially easy to do when there is only one element present for any given name (as is the case in this example). The lines below show you how you could extract specific items from our example XML:
set loNodeList = loXMLDoc.getElementsByTagName("scriptid")
if loNodeList.length > 0 then lcScriptID = loNodeList(0).text
The above uses the ..getElementsByTagName method to search all of the child nodes of the object (in this case the entire document object) for tags of a specific name; it returns a list of those which it found. Since we are expecting only one to be found, if it finds any then we know it found what we wanted, and so we take the first item from the list (important note here: this method returns a list where the first index is zero). Once we have the element, we use its .text property to extract the text which was specified between the opening tag for the "scriptid" and the closing tag ("1326" in this example).
It becomes more challenging however if the XML document was a list of <script> ... </script> elements, one after the other, each having its own <scriptid> tags. In such a case you would want to process each <script> tag separately, and for each one you would want to search only its child elements looking for its <scriptid> tag. You actually already have been introduced to all that you need to accomplish this, as shown in the example below:
set loScriptList = loXMLDoc.getElementsByTagName("script")
for each x in loScriptList
' get the script id for this script
set loNodeList = x.getElementsByTagName("scriptid")
if loNodeList.length > 0 then lcScriptID = loNodeList(0).text
speak lcScriptID
The above example gets a list of <script> elements from the document object (by using the .getElementsByTagName method), and then in a loop it gets a list of <scriptid> elements for each <script> (and there should only be one <scriptid> for each). The use of .getElementsByTagName inside of the loop doesn't search the entire document, but only searches the portion of the tree for the current <script> tag (thus assuring you that the returned <scriptid> element is the one for the current <script> element). This isn't the only way to retrieve data from an XML document, but it's certainly one of the easiest to understand. For more advanced searching options, see the .SelectNodes() example here and the related topic "XPath".
You can simplify the above search inside of the loop however. Because you know that a <scriptid> element is a direct child of any <script> element, you don't need to search down through all of the elements under <script> looking for it (which is how the .getElementsByTagName method does it). Instead, you only need to search the immediate children once you have a <script> element. Since there can only be one element with a <scriptid> tag for any <script> element, we don't actually need a list of all of them returned to us. When you know these two conditions are true (you only need to search the direct
children, and there can only be one result), then you can use a simpler searching method as shown in the example below:
set loScriptList = loXMLDoc.getElementsByTagName("script")
for each x in loScriptList
' get the script id for this script
lcScriptID = x.selectSingleNode("scriptid").text
speak lcScriptID
The use of the .selectSingleNode method can be much faster in a very large tree, where the portion you are searching has many possible nodes at the grandchild or deeper level, because it only searches the direct child nodes.
Resources
To see a list of sites which have an XML API for their services, check out This API database filtered for XML. It can also show you over 8000 APIs which use a variety of data protocols.
Full Example
Below is the full example for this article, showing how to retrieve the script id for the UIDesign app from its XML script info page. For some example apps which make use of retrieving XML info via HTTP, see the "Weather or Not" and the "Install Packages" apps.
lcHTTPCommand = "https://www.gwmicro.com/App_Central/scriptTools.php?scriptid=1326&showXML&showChanges" Set loHTTPRequest = CreateObject("WinHttp.WinHttpRequest.5.1")
loHTTPRequest.Open "GET", lcHTTPCommand, False
loHTTPRequest.send
if loHTTPRequest.status = 200 then
set loXMLDoc = CreateObject("MSXML2.DOMDocument.3.0")
loXMLDoc.async = false
loXMLDoc.loadXML loHTTPRequest.responseText
set loNodeList = loXMLDoc.getElementsByTagName("scriptid")
if loNodeList.length > 0 then
lcScriptID = loNodeList(0).text
speak "the script ID found was " & lcScriptID
ELSE
SPEAK "FAILED TO FIND THE SCRIPT id TAG ELEMENT"
end if ' loNodeList.length > 0
ELSE ' loHTTPRequest.status = 200 SPEAK "Received error " & loHTTPRequest.statusText & " from the web server" end if ' loHTTPRequest.status = 200
