|
|
|
|
Example Evaluations
[ ENSURE Euro. Network for Sustainable Urban and Regional Development ]
[ FHDC Forum Habitat in Developing Countries ]
[ NAERUS Network Assoc. of Euro. Researchers on Urbanisation in the South ]
[ RUDI Resource for Urban Design Information ]
Here are four 'example evaluations' undertaken as part of this project. This section can be used in conjunction with the
Evaluation Guidelines pages
to show how collected data can be analysed.
The text provided is based upon the analysis of the Data Collection Tables for each website. Each table summarises the information collected for each of the six website components.
For details of the four websites additionally evaluated please send a short message to jon.taylor@appropriatesoftware.net
ENSURE Euro. Network for Sustainable Urban and Regional Development
http://www.european-association.org/ensure
by
JT
ENSURE Data Collection Tables
File size: 73KB
Architecture
There is some evidence to suggest that ENSURE is a well-structured, easily navigable website. All respondents to the user survey stated that the website was either very easy or fairly easy to browse, whilst all participants in the usability test were able to find the requested piece of information in the minimum number of clicks necessary. Visitors can also take advantage of a detailed site map to orientate themselves.
One notable navigational problem is that visitors to one section of ENSURE must first navigate 'home' before they can browse to a new section of the website.
This appears to be because individual sections within ENSURE do not have a unique URL and are each rendered within the same Frame set. This, in turn, is likely to be a reflection of the fact that ENSURE is itself a section within the European Association website, and does not have its own domain.
Whilst visitors navigating through the Frame can reach the 'home' page from any page on the site, its use means that different sections of the website are not uniquely identified within the sites directory structure.
This not only problematises navigation between sections, but effectively means that any hyperlinks directed to ENSURE will lead to the initial Frame set, rather than a specific page within the website. This can make finding specific information difficult.
Technology
Several technical concerns may impede site access for an international audience attempting to view the ENSURE website. ENSURE's site definition document fails to validate as either HTML version 3.2 or version 4.0, according to W3C standards. This can cause both rendering and navigation problems in certain browsers.
An analysis of the 'home' page revealed four browser support conflicts that impede access to users with a variety of browsers such as Netscape Navigator 3.0, 4.0 and 6.0. Moreover, documents can only be downloaded in PDF and Word for Windows 98 format. PDF files can be slow to download and problematic to manipulate once accessed. Given that in 1998 only 17 of users globally possessed Word for Windows 98, it is likely that a significant proportion of users will still be unable to access these files.
Using Frames without an all text alternative, the site becomes un-navigable when images are disabled in the user's browser.
Frames can also be slow to download, but the download time for ENSURE's 'home' page, at six seconds using a 28,800 baud modem, did not seem to be badly affected.
Style
One benefit of using Frames is that the site's layout and appearance are consistent between pages. The majority of respondents felt that ENSURE was an attractive website.
Content
There is strong evidence to suggest that whilst the content of the ENSURE website is of a high standard, its audience and the intensity of interaction with its readership are limited.
All users described the standard of writing on the ENSURE website as 'excellent.' Users also stated that the content was fairly useful and relevant to their work. One user stated that the content had helped improve his understanding of urban issues 'a lot', compared to three respondents for whom the site's content still managed to help them 'a little'. Moreover, only one user stated that the site's links were 'not very useful'.
No spelling mistakes or grammatical errors were found, and all text is referenced where appropriate.
The majority of the site's content is contributed by a small number of participants, and ENSURE's lack of interactivity is perhaps a reflection of the site's high degree of specialisation.
As with some of the other websites reviewed during this study, much of ENSURE's substantive textual content, such as articles and papers, are a by-product of the organisation's traditional selection and editing processes, carried out during the proceedings of a conference.
Given the favourable evidence above, it would perhaps be beneficial if users could make more use of this content by being able to interact more fully with the site. No online facilities are available for users to submit either content or comment upon the work of others, although, where appropriate, the contact details of authors are provided.
Strategy
The target audience for the site is defined broadly as "scientists of different backgrounds that are active or interested in the field of sustainable regional development research". From the small number of responses, users do indeed appear to be researchers from both NGOs and academic institutions, based in both the North and South.
Given this potentially broad audience and the quality of ENSURE's content, it is surprising that the number of recorded visits over a six-month period is a mere 276; especially since the site is advertised through search engines, directories and cross-linking.
This may be an inevitable consequence of the website's specialist content, but one additional explanation is that the number of visitors is low because the site does not have an independent domain name. Prospective viewers using a search engine to find the site will be unable to locate it simply by typing in the acronym ENSURE, or the details of any of its associated activities. Viewers arriving at the site as a result of ENSURE's marketing must first navigate through the European Association domain to locate the ENSURE content they are interested in.
Management
ENSURE's limited budget of US1,250 per annum, and a total staff input time of less than eight hours per week, are a clear limiting factor in terms of the development of the site and any decisions to increase the site's functionality and strategic scope.
Conclusions
Analysis of the ENSURE website shows that despite the site's excellent content and pleasing appearance, visitor numbers remain low. Whilst this is in part an inevitable result of the site's specialisation, a number of technical and usability problems have been identified that could exclude users.
Recommendations
- Create an independent domain
- Tidy up the HTML
- Consider providing an alternative to Frames
- Ensure that documents can be downloaded as either HTML, plain text or an earlier version of word
- Insert a date of last update stamp
FHDC Forum Habitat in Developing Countries
Site expired
by
JT
FHDC Data Collection Tables
File size: 74KB
Architecture
There is mixed evidence to suggest that FHDC is a well-structured, easily navigable website. The vast majority of respondents found the site either 'fairly easy' or 'very easy' to browse, there is a detailed table of contents, and users can reach the 'home' page from any page on the website.
However, while the structural essentials for good navigability are in place, the usage of best-practice was not found to be consistent, and three out of five users during the usability test were unable to locate a specified piece of information in the minimum number of clicks necessary.
On most pages within the website the changing colour of selected section links is used as a visual cue, orientating users around the site, and - again, from most pages - users can navigate to a different section of the website without returning 'home'. But these features are, for the most part, only available for the first two levels of the system's hierarchy or architecture, which may disorientate users.
For example, when navigating from the 'home' page to within the Design Participation section, within two clicks the viewer is taken through a consistent 'Frameset' into a section that lacks many of the features of good navigability previously present.
Given that the system's directory structure is well formed and facilitates navigation, it is possible, however, to update these sections without having to make fundamental updates to the website's overall architecture.
Technology
Some technical concerns may impede site access for an international audience attempting to view the FHDC website. FHDC's site definition document does not validate as either HTML version 3.2 or version 4.0, according to W3C standards. This can cause both rendering and navigation problems in certain browsers.
Moreover, an analysis of the 'home' page revealed six browser support conflicts that impede access to users with Netscape browsers, whilst available documents can only be downloaded as PDF files.
The download time for FHDC's 'home' page, at 17 seconds using a 28,800 baud modem, may be marginally excessive for some users.
Style
A majority of FHDC users found the website to be 'fairly attractive', and two-thirds stated that they did not find the graphics distracting at all. Only one in seven users, however, said that the site looked 'very attractive'.
Browsing the site one can see that whilst layout and formatting are largely consistent, the website's look does differ in older areas of the site, such as the Training Manuals and Design Participation sections. The layout also differs within the Links section, where a Frame is used to structure the large amount of data present.
Content
The evidence clearly suggests that the content of FHDC is a highly valued resource for its viewers, although more could be done if the website is to support interactive processes such as networking and debate.
The majority of respondents stated that the site's content had helped improve their understanding of urban issue 'a lot', whilst the same proportion answered similarly to the question, 'How useful do you find the content of this website for your work?'
There was slightly less enthusiasm for the standard of writing on the website, with the majority of respondents claiming that this was adequate rather than excellent, but the majority of respondents to the user survey did find FHDCs links to be useful.
Where enabled by FHDC, users seem to make good use of the websites content by either passing information on to a colleague, or by contacting a contributor. However, these activities will be constrained without more referencing of substantial textual content, and whilst few author contact details are provided.
With the exception of the Design Participation Forum, there is no facility for users to submit content other than a hyperlink.
Strategy
The target audience for the FHDC website is defined as "researchers and professionals working for the improvement of the built environment in developing countries", we can see from the profiles of respondents to the user survey that FHDC has been successful in capturing its intended audience.
Respondents to the survey were researchers, project leaders, students and architects from mainly academic, independent and governmental institutions. There was also a near 50
split between respondents from the South and those from the North.
Searching under the acronym 'FHDC' returns the website in first position using Google, and the site is also listed 'number one' when searching under the phrases 'Built Environment in Developing Countries' and 'Architecture and Planning in Developing Countries'.
However, the website's prominence on major search engines may not be the only reason for the fact that it receives around 1,000 visits each month. Two-thirds of all user survey respondents found out about FHDC after following a hyperlink from within another website.
FHDC is clearly a well-promoted website that is recognised as a valued resource by other websites in the field of urban development. The respondents themselves clearly value the site since they visit at least once each month for networking or research purposes.
Management
FHDC receives no regular funding for its activities and the total staff input time of 9 to 16 hours per week is voluntary. Given these limitations, the website has been highly successful attracting a large number of visitors, who clearly value the webmasters work.
Conclusions
FHDC is a frequently visited website serving a large international community of users. A number of technical concerns should, therefore, be addressed if the site is to improve this service to its international audience.
The financial and human resource constraints currently faced by FHDC continues to be a constraint on these improvements.
Recommendations
- Tidy-up the HTML
- Reduce the number, scale-down or compress image files
- Consider improving the consistency of the websites architecture and style
- Ensure that documents can be downloaded as either HTML, plain text or an earlier version of word
- Where possible include full contact details of contributors
- Insert a date of last update stamp
NAERUS Network Assoc. of Euro. Researchers on Urbanisation in the South
Site expired
by
JT
NAERUS Data Collection Tables
File size: 73KB
Architecture
When asked how easy they found NAERUS to browse all but one respondent stated that this was either 'fairly easy' or 'very easy'. However, since visitors in one section of the site cannot move to another without passing through the 'home' page, it is possible that this positive perception has something to do with the relatively small amount of content on the site.
Visitors benefit from a table of contents and a hyperlinked logo used on each page to navigate them 'home', but with no relational navigational cue - to indicate where within the overall architecture of the site users are - NAERUS may become increasingly difficult to navigate should more content be added.
In the non-Frames version of the website the architectural foundations for such a development are already in place, since the site's directory structure uniquely identifies each page.
Technology
Although NAERUS's site definition document fails to validate as either HTML version 3.2 or version 4.0, according to W3C standards, an analysis of the 'home' page showed that this had not caused any detectable browser support conflicts.
Although an all-text alternative to the site is not offered, an alternative to the default version with Frames is provided. In addition, as an alternative to PDF, documents can be downloaded in WinWord and RTF, to increase accessibility for those users whose systems do not support PDF.
One potential inconvenience to users identified is the lengthy download time for the 'home' page, which even for the no Frames version of the site was 14 seconds, using a 28,800 baud modem. Whilst not necessarily spoiling the appearance of the site, this can be attributed to the large number of image files (in GIF format) on the page.
Style
The majority of respondents to the user survey stated that NAERUS was a 'fairly attractive' website. However, browsing the site reveals some lack of consistency in the formatting of text on older pages and in the site's overall look (this can be seen by browsing through the workshop sub-sections under the activity section, for example.)
Only three respondents stated that the site was 'very attractive', whilst 45 said that the graphics offered some form of distraction.
Content
Evidence suggests that in addition to the high quality of NAERUS's content, its readership has made use of the site's interactivity to participate in a small 'NAERUS community'. The majority of respondents to the user survey said that the standard of writing on the website was either 'excellent' or 'adequate' for its purpose.
Furthermore, half of all respondents stated that the site had improved their understanding of urban issues 'a lot', whilst a similar set of responses was returned for the questions: 'How useful do you find the website's content for your work', and 'How useful do you find the site's links?'
All content on the NAERUS site is well referenced and includes the author's contact details, an opportunity for networking that half of all respondents to the user survey had taken advantage of. Two thirds of all respondents had also passed information from NAERUS on to their colleagues, and all users are allowed to subscribe to a newsletter.
Fewer respondents had submitted content to the site, but research dissemination is only an option for NAERUS members concerned with the association's conference.
As with several of the website's evaluated for this project, much of NAERUS's textual content is produced, selected and edited as part of the off-line activities carried out during the proceedings of a conference.
Given the quality of NAERUS's content and the facilities already present on the site, an attempt to increase the readership and open the 'NAERUS community' to members beyond those immediately concerned with the conference may be an option to consider.
Such users could also be given an opportunity to comment on NAERUS's content publicly, without having to be physically involved with the association's conference.
Strategy
The target audience for NAERUS is defined broadly as "a pluridisciplinary network of researchers and experts working on urban issues in developing countries", and this is broadly reflected in the user survey. Just under a quarter of all responses were from a Southern readership, whilst half of all respondents were researchers, project leaders and consultants, working either within academic institutions or independently.
In view of this potentially broad audience and the quality of NAERUS's content, it is surprising that the number of recorded visits over a six-month period is a modest 786, especially given that the site is advertised through open means, such as search engines, directories and cross-linking.
This may be an inevitable consequence of the website's specialist content, but there are good reasons to suppose that the readership could be increased, should the website opt to broaden its appeal beyond the confines of the NAERUS conference.
Current NAERUS users tend to be regular visitors, benefiting from the opportunity to network, conduct research and keep informed on urban issues, and it is likely that this information could be of benefit to others.
Management
Given its resources, the NAERUS site is undoubtedly value for money. NAERUS has an occasional budget of US1,400, but this is not regularly received.
Moreover a total staff input time of less than eight hours per week is a clear limiting factor in terms of any decision to increase the site's strategic scope.
Conclusions
Analysis of the NAERUS website shows that with good content and an active membership base the number of visitors to the site could be significantly increased.
A small number of minor architectural and technical issues could also be altered to improve the site for an international audience.
Recommendations
- Include a navigable site index within sections
- Reduce the number, scale down or compress image files
- Insert a date of last update stamp
RUDI Resource for Urban Design Information
http://www.rudi.net (New Site)
by
JT
RUDI Data Collection Tables
File size: 74KB
Architecture
The evidence strongly suggests that within the RUDI domain all content is well structured and easily navigable. When asked how easy they found RUDI to browse, the majority of respondents stated that was either 'fairly easy' or 'very easy'.
The site contains a search feature and a site index, and users can click on the site logo to navigate 'home' from any page. Moreover, when asked to locate a specified piece of information on the site during the usability test, all five users took the minimum number of clicks necessary.
Almost all of the usability problems concerning system architecture that appeared during the evaluation of RUDI occurred outside of the RUDI domain, and are not technically a characteristic of the RUDI website.
The navigational problem described here is mentioned only because the webpages refered to are referenced as content within the RUDI website and sport the RUDI logo, and are likely, therefore, to be associated with RUDI by the visitor.
Navigating RUDI to browse associated content outside of the RUDI domain, visitors are unable to browse to another section within RUDI by using the drop-down menu that normally appears on each 'RUDI' webpage.
Consequently, without a navigational sign, viewers have little clue where within the current section they are, in relation to the rest of the website.
This problem is, in part, a consequence of RUDI's role as a portal for Urban Design information. But the practice of encouraging users to browse between different websites (with an entirely different look and feel) as part of the same session may explain why two respondents found RUDI difficult to browse.
A secondary usability problem is that the unique URLs attributed to each section of the website are not consistently the same as the title for that section. The URL for the section 'Talk', for example, is */feedback.html
This inconsistency may increase the chances of error if users attempt to use the directory structure to navigate the site or post a hyperlink to that section.
Technology
Several technical concerns are likely to impede site access for an international audience attempting to view the RUDI website. RUDI's site definition document fails to validate as either HTML version 3.2 or version 4.0, according to W3C standards. This can cause both rendering and navigation problems in certain browsers.
An analysis of the 'home' page revealed 14 browser support conflicts that impede access to users with older browsers such as Netscape Navigator 3.0.
Moreover, a marginally excessive download time for RUDI's 'home' page of 14 seconds, using a 28,800 baud modem, may put off certain users.
Style
Browsing shows that within the RUDI domain the site's look and formatting are consistent. Close to half of all respondents to the user survey, however, stated that RUDI was an 'unattractive' website.
Given that users did not attribute this to distracting graphics, it may be that users are indeed put off by the experience of browsing between websites with different looks as part of the same visitor session. Given the low number of total responses, this finding may also, of course, be an anomaly.
Content
RUDI's textual content goes through a minimal selection and editing process, but the outputs were considered to be of a high standard by respondents of the user survey.
The majority of users said that the site had helped improve their understanding of urban issues and had been useful for their work, whilst the standard of writing was generally thought to be 'adequate' for its purpose.
There is also evidence that RUDI has been partially successful in developing a community of RUDI users. Over half of all respondents to the user survey had passed information on to a colleague, despite the inconsistent provision of author contact details, one third of all respondents had contacted a contributor to the website.
There is, however, no online facility for the submission of content; a facility that could be the starting point of a system to solve the possible usability problem relating to the site's architecture, referred to above.
One minor failing is that although RUDI's links were generally thought to be 'very useful', 14 dead-links out of a total of 213 were identified.
Strategy
The target audience for the site is defined broadly as being teachers, researchers and professionals in the field of urban design and its related disciplines. According to the small number of user survey responses, users appear to be project leaders, planners, students and researchers, visiting RUDI on a regular basis to conduct research and keep informed on urban design issues.
From the user survey we can also surmise that the majority of users are commercial or academic clients from more developed countries.
The actual number of visitors to the site is unknown. Log-file data to estimate the number of visitors was unavailable, and the number of page requests, at 434,025 over a six-month period, tells us little.
Given the range of marketing strategies employed by RUDI, it is surprising that the majority of reported users discovered RUDI through word of mouth. This may be testimony to RUDI's popularity, but could reflect the effectiveness of other marketing strategies.
Such evidence may also help explain why so few visitors from Southern countries were reported. RUDI is, however, readily locatable. Searching using Google with the term 'RUDI' returns the website in number one position, as does the term 'Urban Design.'
Management
RUDI is a large website with a total staff input time of over 81 hrs per week. This should justify funding to increase its dedicated budget to more than US 250 per annum, and warrants consideration of the opportunity to provide content management support to authors currently using ad hoc website production tools to disseminate their work via RUDI.
Conclusions
RUDI is a large frequently used website which has adopted a clear strategy for its online activities. Given its profile, content management support to authors - who use RUDI to disseminate their research - could improve the users overall experience and enhance RUDI as an Internet resource for the Urban Design community. The site exhibits a small number of minor usability problems.
Recommendations
- Ensure that the syntax of section URLs is consistent with the section titles on the webpage
- Tidy up the HTML
- Ensure that author contact details are consistently provided
- Remove or update dead-links
- Reduce the number, scale down or compress image files
- Utilise log-file data that provides estimates of visitor numbers
|
|
|
|
