I am trying to put all benchmarks in one place. If you know of anything else that should be there let me know.
In addition to
Will we be trying to define the "right" way for the benchmark to be using the UI?
I am asking the question: How do you benchmark a user interface? This is much more difficult than running a script because it requires the user to do something and this introduces a certain amount of randomness to the test. Some things in the UI can be benchmarked I believe. Some have complained that the UI seems slower or more sluggish. It might be possible to capture that.
I used to watch a certain TTM run through a SW demo and got to thinking that he was doing it about as fast as humanly possible. I have never seen a normal user construct parts and assemblies that fast. So what he was doing was the limit on the interface.
I also consider the SW challenge at SWW a form of benchmark. There are many variables. It is totally up to the user to determine the solution to the problem and then to implement it. The fact that so few finish, but that some finish in a very short time gives some feel for SW performance.
I have seen macros written in the application that track how far the mouse moves or number of clicks, there is a really cool one I have seen called mouse miles or something like that that just tracks how many miles you travel with your mouse after a while.
This would give you an indication of user interface. Turn such a macro on and give the user a few tasks to do. Get the above information and maybe even an over all time. This would only really be useful to test various users or even different versions of the software as they move things around or create compound functions.
--Matt
T> I am asking the question: How do you benchmark a user interface? This
It's a tricky problem, benchmarking interfaces. I've seen advertising from various CAD systems claiming a more efficient UI based on "user efforts" or some such. I remember SE was into that.
Monitoring the number of clicks and mouse mileage could be used as a user benchmarking tool instead, discovering what is the fastest way to accomplish certain tasks within a system. That might be interesting.
Now that this link has been out a while I would really like to see some results, especially for high and middle Intel stuff. I just have AMD myself so can't really compare unless it be with a 650Mhz laptop.
I think it is a little more complex than mouse miles, although that sounds pretty cool to see. The ease of an interface could be measured in number of steps, keyboard actions, and even mouse mileage. However, there are also the less easily quantified aspects of logical layout, intuitive placement of icons or menues, how large clickable zones are, that enhance workflow and the user experience (note the debate between Windows and Mac OS which has a lot to do with these less easily quantified user experiences).
For example, if you are familiar with Alias StudioTools / Maya marking menu system, you would probably agree that it is a nightmare for the new or casual user, but jockeys that "know" it and use it daily find it significantly faster than using a normal menu system. But on a casual basis I really find it frustrating. When I have a job where I use it for a few days, I enjoy it again... then forget it again.
Now if mouse milage could be converted to calories burned, we could also see which CAD program has better health benefits ;-)
Daniel
I totally agree, I was just trying to give one example. I for one actually like pro/e's marching menus as I could get around fast with those. I am also one that likes quick keys and couldn't get along without them. In terms of the interface one of the huge benefits is the ability to use quick keys and even api. I find as soon as I do things a few times, I script them and tie them to quick keys. How easy is this to do today? But then how easy is it to transfer this to another user, sometimes difficult.
I even know people whom have learned to use the mouse with the left hand as it makes so much more sense in the CAD world to have your right hand on the num-pad. I haven't been able to train myself that way....
But with that said #of clicks, mileage and average velocity for a set number of functions do actually test much of what you describe below. And the goal of most benchmarks is a quick test to show relative numbers. The problem is trying to level the playing field in a way these numbers actually mean something outside of the test.
--Matt
daniel wrote:
The other problem with this metric is actually making the comparison. But then this tends to lead back to the question of what is to be gained by this benchmark. The benchmarks I tend to do has an outcome, replacement or justification of hardware etc. And that tends to lead toward how I do the benchmark. We are currently looking into gig-a-bit to the desktop, as it is the next step as we just moved over to SAN technology for our network storage and have really gotten storage as fast as we can.
--Matt
Matt Feider wrote:
PolyTech Forum website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.