IBUSS Beta Test Site Report

IBUSS Beta Test Site Report

September 1, 2003

The ND Center for Persons with Disabilities at Minot State University developed a unique software program called Internet BUsiness Support Software or IBUSS. The IBUSS program was designed to help voucher sites in remote rural areas to coordinate and share information among separate sites using an integrated, data-base system. IBUSS may be used by a community service group, a city, a disability support group or any legitimate non-profit organization.

Five agencies in Montana, North Dakota and Wyoming have completed the GRIT online course and are currently beta-testing the IBUSS software. No sites in South Dakota were able to participate. Four of the five sites will receive funds through the ND Center for Persons with Disabilities to cover some of the expenses related to serving as a beta-test site. The current sites are:

ND Association for the Disabled: NDAD provides assistance with transportation costs on a statewide basis. Consumers may live in a large community such as Grand Forks or a smaller community (e.g. Dunseith). Transportation is often provided to enable consumers to reach clinics or hospitals for health care appointments. Transportation is also provided for more routine purposes (e.g. shopping). NDAD is testing the software through rides offered in Bismarck and Grand Forks and plans to expand the test to rural communities in the future. The funding source is located in Grand Forks and the voucher site is located in Bismarck. NDAD is a private-non profit organization that relies on gaming to meet transportation costs.

Dakota Center for Independent Living: DCIL serves the greater Bismarck/Mandan area. DCIL also provides remote services in Dickinson, ND a community of about 3,500 people. Dickinson is located in southwestern ND, just off the Interstate near the North Dakota and Montana border. DCIL provides vouchers to local cab-companies to transport consumers during evenings and weekends when buses are not available. Services include transportation for consumers living in outlying rural areas as well.

Wyoming Independent Living Rehabilitation: WIRL currently runs a voucher program serving in 5 remote rural counties (i.e. Platte, Black, Goshen, Niobrara, Converse) in southeast Wyoming. Populations run from approximately 2500 to about 10,000 depending on the county. Larger communities have limited bus service from senior centers and smaller communities rely solely on voucher systems. Voucher site operators work from home offices and the funding source office is located in Casper, Wyoming. WIRL has secured funding through the DD council, Wyoming’s University Center of Excellence and VR to fund voucher programs. Ken has been operating his voucher program for just over a year and follows the checkbook model developed by Tom Seekins of Montana.

Sanders County Commission on Aging: SCCA operates in Sanders County, a remote area of northwestern Montana. SCCA serves a 10,000 population county that is 100 miles long. No transportation systems are available. This is a new voucher program.

Confederated Salish &Kootenai Tribes: CSKT also serves a remote area of Montana on the Flathead Indian Reservation. CSKT is a pilot site for APRIL. They are also relatively new to voucher provision.

Beta-test Requirements: IBUSS is set up to coordinate communication and information exchange at three levels; (1) a funding source sets up ride program description and criterion and decides which agencies (if any) and voucher sites may coordinate rides or implement the ride program. (2) A program manager that works for a state agency such as Medicaid, VR, determines if individual consumers meet eligibility requirements set up by the funding source and (3) a voucher site, prints and mails vouchers, manages invoices and keeps related records.

Sites participating in the beta-test were required to provide partners who could test each level of IBUSS. Participants could assign employees at central and remote sites to take different roles or find an outside agency to serve as a site manager or funding source. All of the participants chose the first option. Staff were assigned to specific roles and required to complete an online course for their specific level. Participants were provided with online, phone and site support. Coursework was completed between June 1, 2003 and August 1, 2003. Summer vacations and turnover delayed the ability of some agencies to start. One agency, Sanders County started late, after no agency in South Dakota was willing to participate.

Data Results: NDCPD was able to review progress within the course online. Each participant was asked to complete an online survey as they finished their final lesson. Many participants elected to complete the course for all three levels. Supervisors generally wanted to know what their employees would be doing and how using IBUSS might impact their work load. Newer sites found that using IBUSS made them think about many aspects of voucher delivery in a new way. The following data is currently available from the online survey and participant comments at three levels.

Funding Source

Program Manager

Voucher Site Managers

Funding Source Respondents:

Estimated length of time to complete the course:

Hypothesis: The course could be completed in a reasonable amount of time

Results: 100% of 6 respondents indicated it took 4-8 hours to complete

Comment: Participants were able to return to the course when convenient. The time was usually parsed out over several days.

Time needed to complete the course was:

Hypothesis: Participants would rate time spent as acceptable

Results: 40% of 5 respondents indicated a 3 on a 1-5 scale (as expected)

60% of 5 respondents indicated a 4 on a 1-5 scale (as expected)

Comments: It was also expected that participants at the funding source level would complete the course the fastest and that those using the voucher site level would take longest at there are more chapters at that level and more tasks. The opposite proved true. Most people assigned to manage voucher sites had accounting or book keeping backgrounds and sailed through the material with ease. People had the most difficulty at the funding source level and often delegated the work or had difficulty with directions.

Course design made the course:

Hypothesis: Participants would find the course design easy to use

Results: 60% of 5 respondents rate the course as a 3 acceptable

20% of 5 respondents rated the course as a 2 less than acceptable

10% of 5 respondents rated the course as a 4 more than acceptable

Comments: None of the respondents found the course design easy to use. A log, based on participant phone feedback, indicated several problems with course design that confused participants.

1. Directions were not clear. Participants were unsure of when/where to start and to proceed.

2. Clicking done when finished seemed foolish and this step was skipped. Also, if people took more than 20 minutes to complete the lesson, the DONE feature timed them out. This was annoying and confusing. It was corrected after most people had completed the course.

3. Instructors were almost never emailed, participants preferred regular voice contact.

4. The level of passwords needed was too confusing. Participants prefer one password they can use throughout the course, for all levels and in the beta-test program. Currently people log-in to the course at each level, sign into the IBUSS data base and long in again to a new level.

5. The manual was dry and too technical. In some cases it was redundant and in other cases the screens did not match last minutes changes in the program.

6. Having three open screens or more at one time was confusing for most students. Sometimes they could not find their way back.

7. Students should be informed up front that IBUSS would time them out if they left the screen for more than 20 minutes. Some students came back to find their work gone.

When anything went wrong, students had a tendency to assume it was their fault. In spite of these challenges participant comments tended to be supportive and are listed as follows:

· Funding level training was some what confusing.

· We received wonderful assistance every time needed.

· Manual should follow and give instructions on each menu item in order.

· Much better after I read the directions and followed them.

· Should be very easy to use

Course features that participants liked:

Hypothesis: Participants would like most/all of the course features highlighted

Results: Participants were asked to select from the following list. Data next to each item indicates the number & percentage of respondents who selected the item in order of frequency from most to least.

Comments: All of the participants appreciated the anytime, any where component to training. The majority of the features were liked when selected as a whole. Comments passed on through the log and course email indicated that participants liked features if they were easy to use and expected the components to work effectively. The grid and directions for completing tasks was confusing.

Indicate (any/all) course features that you disliked:

Hypothesis: Participants would dislike few/none of the course features highlighted

Online comments regarding features disliked were recorded as follows:

I followed instructions of the manual and submitted at every opportunity, but somehow missed getting back to the table where you click done and get a questionnaire, are allowed to make comments, etc. Either I did something wrong or it was not clear how to get back to that chart to do that process.

Comments: These findings echo the data showing features that were liked. No one selected availability, table of contents, data base, site map or technical assistance as a problem area. Only one person had difficulty with the lesson page and training manual content. Phone feedback indicated that students wanted someone available to help them whenever they were online. Much of the phone feedback helped students with places in which the course was confusing.

Rate how well you believe the course prepared you to use IBUSS:

Hypothesis: Students would feel well prepared to actually use IBUSS.

Results: 3 of the 5 respondents (60%) rated at level 3 – prepared.

2 of the 5 respondents (40%) rated at level 5 – very prepared

Comments: Much of the discussion about the course, tended to focus on problems with both the course and IBUSS design. The log indicates that many students were complementary over-all and couldn’t wait to start the actual beta-test in which real people would be given real rides. That displays a level of confidence that is highly desirable in spite of problems with course design.

IBUSS software features that you liked

Hypothesis: Participants would like most/all of the IBUSS features highlighted

Comments: Overall participant response to IBUSS was much better than the course. Respondents told project staff that IBUSS was easy to figure out with one exception. Several participants did not realize when ride programs were added. No evidence that the program “took” appears on the main menu. Students did not know they needed to enter Manage Programs and click on Modify to see a list of programs. Consequently, several students entered ride programs multiple times and became frustrated.

Indicate IBUSS software features that you disliked:

Hypothesis: Participants would dislike few/none of the IBUSS features highlighted

Comments: Participants disliked very few features of IBUSS. One participant was determined to use only his own pre-printed checks and not the generic voucher icon feature. Subsequently his staff indicated that the pre-printed checks confused consumers and decided to use IBUSS vouchers when checks ran out.

Estimate length of time to set up ride program using IBUSS

Hypothesis: Participants would estimate using IBUSS to set up a ride program wouldn’t take very long.

Results: 3 of 5 participants (60%) estimated they could set up a ride program in less than 2 hours. 2 of 5 participants (40%) estimated they could set up a ride program in 2-4 hours.

Comments: Participants who learned about IBUSS believe they can set up a ride program in a short period of time. Setting up a ride program probably caused the most difficulty during the course because participants needed help to respond to error messages when they had too many windows open or were timed out, misunderstood choices (i.e. driver reimbursement requirements were selected when the site didn’t really need them) or couldn’t see ride programs they had entered (design flaw). In spite of these issues, participants were confident that the time to set up a ride program would be minimal.

Future use of IBUSS: Participants were asked to respond to variables related to the future use of IBUSS.

Would recommend IBUSS to other agencies or organizations in the state:

Plan to use IBUSS after the beta-test if more funds are available:

Plan to use IBUSS after the beta-test using own/partner resources:

Results: The results were an overwhelming maybe. Although 60% of respondents said they would recommend IBUSS to another organization after completing the course, none of the respondents would commit to continued use of IBUSS if more funds were available or to an extended beta-test period. 100% of the respondents indicated maybe to the second two questions.

Comments: This question may be too naïve for the participants who are appropriately cautious. However it should be noted that some respondents were not in a decision making position. Site visits led some respondents to think more broadly about using IBUSS on a continued basis.

Rate estimate of consumer satisfaction with ride programs coordinated using IBUSS

Hypothesis: Participants will identify features that consumers may like and predict consumer satisfaction.

Results: 2/5 respondents predicted consumers would be satisfied at level 3. The remaining 60% (n=3) of the respondents indicated no comment and were unwilling to commit.

Comment: This question is probably only useful as a post data indicator.

IBUSS meets quality indicators: Participants were asked to indicate if IBUSS matched any of the following criterion:

Hypothesis: After completing the course, software design specialists hoped that users would rate IBUSS as cost effective, user friendly, helpful with rural economic development and transportation.

Results: 3 of 5 participants (60%) rated IBUSS as user friendly, helpful with rural economic development and transportation after completing the online class.

Comments: None of the participants were willing to indicate that using IBUSS was cost effective. Apparently from other data, they did not believe the cost had been beyond their capabilities but they were perhaps at this point in time, unaware of hidden costs or future challenges and unwilling to comment. Survey comments read as follows:

· I think it will be very easy for the rider to use. I also think it will be fairly easy to use.

· Not sure if it would be cost effective?

· We won't know this until after we have beta tested the program in a real life situation.

PROGRAM MANAGER RESPONSES

It should be noted that only a few individuals beta-tested the course at this level. In general, program managers found the course easier to use than did persons at the funding source level. Also, due to a clerical error, the number of survey questions was shorter. The results are listed for you to view.

Estimated length of time to complete the course:

Time needed to complete the course was:

Course design made the course:

Indicate (any/all) course features that you liked:

Indicate (any/all) course features that you disliked:

Rate how well you believe the course prepared you to use IBUSS:

About using IBUSS

Indicate (any/all) IBUSS software features that you liked:

- Much easier that funding level.

- The only lesson I had difficulty with was Lesson One. When I went to IBUSS I wasn't able to access other programs to be able to explore managing those programs. When it came to time to take the quiz, it asked questions on programs I wasn't able to access to be able to find out what the rules were.

Otherwise the other lessons were easy to follow and understand as I went through the training manual.

Voucher Site Manager Participants

About the course: