|
|
|
|
Key Research Activities
The research activities described below are six complementary methods for evaluating existing websites disseminating development research.
What follows is an overview of each research activity, as undertaken during the evaluation of eight websites disseminating urban development research.
For each research method there is an explanation of the type of information sought and the tools necessary to collect it. For details of the specific questions asked see the Evaluation Framework.
Each activity seeks to answer different types of questions relating to six website components. The relationship between 'website component', 'question type' and 'research method' is shown in in the Evaluation Framework, which should be used in conjunction with this section to guide your own website evaluation.
by
Jon Taylor
Website Evaluation Framework
File size: 35KB
For examples of how the resulting data can be analysed, readers are invited to see the
four example evaluations
following this section of the module.
Observation
A high proportion of the questions shown in the Evaluation Framework can be objectively answered by the evaluator whilst browsing the website concerned.
During the evaluation of the eight websites assessed during this research, up to three hours was spent browsing each site in an attempt to answer the series of questions shown.
Each question is based upon a set of normative assumptions about what a 'quality' website should contain, but nevertheless their interpretation is corroborated by other information gathered.
In most cases the questions asked are simple and unambiguous, but those concerning 'directory structure', 'document downloads' and 'meta-tags'warrant further explanation.
In computer file systems, a directory is a named group of related files that are separated from other groups of files by the naming convention. The question in the Evaluation Framework concerning a website's directory structure seeks to find out whether the different groups of related files - or webpages - we see within the website are uniquely identified within the sites directory. By this we effectively mean: does each webpage have a unique URL?
Websites that uniquely identify each webpage can be navigated by using the browser's Address field. More importantly, users wanting to direct other people (or computer programmes) to a specific webpage within a domain may do so by passing on the unique URL. This will direct new users precisely to the desired page, rather than to the root URL for that website.
By asking the question: in what format can documents be downloaded? We attempt to assess whether the format used is likely to be appropriate for the documents intended audience.
The large file size of PDF files or the fact that not all users have the latest version of Microsoft Word, for example, is likely to prevent a significant proportion of users from accessing documents of this type
(note 12).
Meta-tags are keywords or search terms contained within a websites definition document. Used appropriately, they can help promote your website in search engines. An evaluator can check for their existence by using the View: Source function on their browser.
Automated Tests
For certain mechanical aspects of website performance web-based analysis tools can be used. Automated website analysis methods have been criticised by writers on the topic, since they tell us nothing about the actual behaviour and experience of real users
(note 13).
Automated tests are useful, however, to evaluate purely mechanical aspects of a website that would be tedious or impossible to measure by eye.
Tests for spelling errors, link-rot and browser compatibility were undertaken using the online website analysis tool Dr HTML. You can make use of this service by visiting
http://www2.imagiware.com/
(note 14).
Dr HTML is a proprietary software programme, and the project team are unaware of any comparable Open Source Software website analysis packages. You can, however, run an analysis of a single webpage for free at the company's website.
Tests for compatibility errors between your website definition document and particular browsers may require a browser compatibility table, if the information is to be interpreted successfully.
A detailed table showing browser type against coding standards for HTML, JavaScript and Frames, for example, can be found at
http://hotwired.lycos.com/webmonkey/browserkit/
The HTML validator available from the World Wide Web Consortium
(note 15)
- available from http://validator.w3.org/ - can be used for free to test the validity of the HTML contained within your website's definition document. By typing in the URL for a single webpage the service will check your websites HTML against recognised standards for HTML version 3.2 and version 4.01.
Additional evaluation and repair software for web accessibility is available from
http://www.w3.org/WAI/ER/existingtools.html
To measure the download times for the home pages of participating websites the free website analysis tool Bobby was used, http://www.cast.org
The software will record the download time for a specified webpage using a 28,000 baud modem.
The response times measured are only a rough indicator of the time experienced by actual website users. Times are dependent not only upon the type of modem used, but on the throughput of the server, the type and quality of the connection to the Internet, and the rendering speed of the user's browser and processor.
This means that actual download times are likely to be greater than those recorded. A benchmark of 10 seconds is used to assess the excessiveness of a pages download time, since we know from research conducted in the USA that one-third of web-users will stop viewing a site if they have to wait longer
(note 16).
Usability Test
Testing of actual users, undertaking real world tasks, is regarded to be one of the best methods for evaluating websites
(note 17).
Given the scope of this research and the geography of its participants the project team were unable to conduct use-case testing on a large scale.
During the eight evaluations undertaken as part of this study, five INTRAC colleagues volunteered to complete a task-based questionnaire concerning website usability. Each user considered themselves to be frequent Internet users, although none had previously seen the websites in question.
The five users were asked to find one specified piece of information for each of the websites under evaluation. The objective of the task was to find out how many clicks it took users, on average, to locate the required information.
Upon searching each site the user manually noted down the number of clicks taken. This figure was then compared to the minimum number of clicks necessary to find that information.
The quantitative data collected using this test is a measure of website navigability, in that if the actual number of clicks taken by users is greater than the minimum number of clicks possible, it could indicate that information is difficult to find on that website.
We know from previous website usability tests that testing five users is a sufficient number to gather meaningful data
(note 18).
You can conduct a usability test in the following way. Simply pick out some memorable text from within the website under study and, starting at the home page, note the minimum number of clicks it takes to find that information.
Then construct a question that effectively leads people to find that information by encouraging them to make associations between the thing requested and the categories that might be used to describe it.
For example, when evaluating the FHDC website, users were asked to find the name of the barrio in the Barrio Design Participation Project. A logical user, following a logical website structure, might look for this information by browsing the Projects section and checking the Design Participation sub-section.
The test allows us to test for any significant discrepancies between where a user might think a piece of information is held, and where it actually exists according to the website's Architecture and the categories used to describe it.
User Survey
Writers on website usability have been critical of the use of online surveys as part of a methodology for evaluating websites
(note 19).
The basic problem is that there is often little relationship between what a user says they do in a survey and their actual behaviour online. Despite this evidence, the project team used an online survey to ask questions of their users for two important reasons: first, the alternative to undertaking a user survey, namely live usability tests, was unfeasible for a multi-national project of this nature; second, a survey is not an inappropriate research method if it seeks subjective and selective opinions by design
(note 20).
A user survey can be used to ask both specific and factual or subjective questions of your users. It is the most appropriate means of gathering information about the profile and experiences of users located internationally.
The following guidelines offer a description of how to make use of the survey shown and are not an explanation of how to conduct online surveys in general. Details of the survey questions asked can be found in the Evaluation Framework.
The user survey used during this evaluation is a variation of the questionnaire you can see on the project website at
http://www.urbandevelopmentforum.org/WebsiteEvaluation/YourComments.html
To conduct an online survey successfully, questions would need to be coded into a form, so that responses can be submitted to the evaluator or another computer programme.
The survey mentioned above was automatically rendered as a webpage following its definition in an XML document. Submissions are transported as an XML file to an online survey aggregator that produces basic descriptive statistics for each data set. This service is available for use by CSOs undertaking an evaluation of their website
(note 21).
The websites evaluated during this project placed a link from their site to the survey on the projects domain. The questionnaire was posted for six weeks from the end of June to the first week in August 2001, and a total of just over 100 valid responses were returned.
Although the information collected tells us little about the user's actual behaviour online, the questionnaire was an excellent tool for constructing an understanding of the user's unique experience of website architecture, style and content. The subjective opinions of users were triangulated with other, more objective data, so that more general inferences about each website could be made.
It is recommended that a user survey should be a permanent feature of your website, but you should keep it posted for at least two months before attempting to analyse the results.
Surveys receive the largest response if they are prominent or offer an incentive. The websites receiving the highest number of returns during this project were those which created a pop-up hyperlink to the survey and which notified their members of its existence in an email.
Webmaster Survey
Participating webmasters were asked to complete a short questionnaire for their website concerning their site's log-file data and a number of internal organisational issues. Details of the questions asked are shown in the Evaluation Framework.
The questionnaire assumed that each webmaster had access to log-file data for their website over the six-month period from December 2000 to May 2001. It also assumed that each website's directory structure enabled webmasters to disaggregate this data according to unique URLs within each website.
The survey was designed for websites held on a single server. The webmaster survey designed for this evaluation was emailed to a representative of each participating website.
Several commentators have warned website evaluators about the pitfalls of log-file data analysis
(note 22).
If misinterpreted, log-file statistics can be a very misleading measure of site performance. Taken alone, the data tells us little about the exact number, identity and experiences of website users. A fundamental rule of log-file analysis is to be precise and consistent when making use of any log-file data terms.
Terms commonly used in the analysis of log-file data are page requests and unique visitors. A page request is defined as a series of hits that successfully retrieve the collection of files constituting a single webpage. This statistic can be useful in that it is an indicator of the level of activity around a website.
A visit refers to the number of consecutive page requests made from a client to a server in a single session. We use this information to make an estimate of the number of people who have viewed the website. Some log-file software disaggregates this information according to the number of unique visitors in a 24-hour period.
In all cases an interpretation of log-file data regarding visitor numbers should be made cautiously, since to most log-file analysis software a visitor will include other software programmes or the same people more than once.
A common mistake is for the number of hits to be taken as a measure of site activity. Given that one hit refers to one recorded request for a single file, and that a single webpage containing numerous files will return several hits, the unit is clearly a very imprecise measure of the number of unique visitors.
For those readers whose website does not currently utilise log-file analysis software, webalizer is a free Open Source Software tool available at
http://www.mrunix.net/webalizer/
An availability requirement refers to the extent to which a webmaster expects to be able to guarantee the availability of their website. Different website strategies will require different guarantees of availability.
The purpose of the evaluation is to make explicit any discrepancy between the level of availability required and the level actually offered by the websites host.
This term is not a feature of a log-file, data, if available, will come from the service level agreement between the website and the Internet service provider.
Telephone Interviews
Five minute telephone interviews were conducted with a representative from each website. Details of the questions asked can be found in the Evaluation Framework.
The conversations were semi-structured and covered issues around the organisational processes by which textual-content is selected, edited and reviewed.
The qualitative data gathered says little about website usability and technology, but can be usefully used to develop an understanding of the internal organisational processes by which textual content is selected and edited for the website.
Evaluators should seek to make explicit these processes by asking such questions of their own work.
|
|
|
|
