EXPLOITING THE USER INTERACTION CONTEXT FOR AUTOMATIC TASK DETECTION

Figures & data

FIGURE 1 The event aggregation pyramid represents our conceptual view of the user interaction context. (Figure is provided in color online.)

FIGURE 2 The concept hierarchy of the User Interaction Context Ontology (UICO), visualized with the Protégé tool. The left area contains the action dimension, the right area contains the resource dimension, and the bottom area contains the user dimension and the information need dimension. (Figure is provided in color online.)

TABLE 1 List of Our Application and System Sensors, and Data Recorded by These Sensors

Sensor	Observed data and metadata
Application Sensors
Microsoft Word	document title, document url, language, text encoding, content of visible area, file name, folder, user name
Microsoft PowerPoint	document title, document url, document template name, current slide number, language, content, file name
Microsoft Excel	spreadsheet title, spreadsheet url, worksheet name, content of the currently viewed cell, authors, language, file name, file uri, folder, user name
Microsoft Internet Explorer	currently viewed url, urls of embedded frames, content as html, content as plain text
Microsoft Explorer	currently viewed folder/drive name, url of folder/drive path
Mozilla Firefox 2.x and 3.x	currently viewed url, urls of embedded frames, content as html
Mozilla Thunderbird	(html/plain text) content of currently viewed or sent email, subject, unique path (uri of email/news message) on server, user's mail action (compose, read, send, forward, reply), received/sent time, email addresses and full names of the email entries
Microsoft Outlook 2003/2007	(create, delete, modify, open and distribute) tasks, notes, calendar entries, contacts, data about email handling
Novell GroupWise email client	(create, delete, modify, and distribute) tasks, notes, calendar entries, todos, data about email handling
System Sensors
File System Sensor	copying from/to, deleting, renaming from/to, moving from/to, modification of files and folders (file/folder url)
Clipboard Sensor	clipboard changes (i.e., text copied to clipboard)
Network Stream Sensor	header and payload content from network layer packets (http, ftp, nttp, smtp, messenger, ICQ, Skype … )
Generic Windows XP System Sensor	mouse movement, mouse click, keyboard input, window title, date and time of occurrence, window id/handle, process id, application name
Accessibility Object Sensor	name, value, description, help text, help text description, tooltip of the accessibility object

TABLE 2 Distribution of the Task Instances with Respect to the Task Class and the Environment

	Task 1	Task 2	Task 3	Task 4	Task 5
Laboratory computer	30	26	26	24	3
Personal workstations	25	19	25	28	14
Sum	55	45	51	52	17

TABLE 3 Overview of the Best Accuracies a Achieved by Various Feature Sets f

	f	l	g	a	p	r
UICO	All Categories	J48	750	86.71	0.96	0.85
	Application Category	J48	50	86.71	0.96	0.84
	Resource Category	NB	3000	66.49	0.89	0.72
	Content Category	NB	4000	65.15	0.87	0.65
	Ontology Structure Category	J48	359	63.38	0.86	0.64
	accessibility object name	J48	50	86.21	0.96	0.86
	window title	J48	75	81.26	0.94	0.81
	accessibility object value	J48	25	71.15	0.90	0.69
	content in focus	J48	1500	64.72	0.87	0.64
	content of event-block	KNN-1	75	63.74	0.87	0.66
	Top 4 features	J48	5000	88.55	0.97	0.87
	Top 5 features	J48	1000	88.53	0.97	0.87
	Top 2 features	KNN-10	25	88.12	0.96	0.89
	Top 20 features	J48	500	88.07	0.96	0.87
	Top 3 features	J48	3500	87.66	0.96	0.88
	Top 7 features	J48	300	87.19	0.96	0.85
	Top 6 features	J48	250	87.16	0.96	0.86
	Top 8 features	J48	3000	87.14	0.96	0.84
	Top 10 features	J48	300	86.73	0.96	0.85
	Top 15 features	J48	500	86.28	0.96	0.85
	Top 9 features	J48	300	85.76	0.96	0.83
Dyonipos	ApplicationName (A)	J48	24	48.20	0.78	0.52
	Content (C)	KNN-1	125	63.38	0.86	0.63
	WindowTitle (W)	NB	100	78.87	0.93	0.80
	A C	J48	150	68.27	0.89	0.69
	A W	J48	100	80.28	0.94	0.80
	C W	NB	750	82.58	0.95	0.85
	A C W	NB	300	83.51	0.95	0.85
SWISH	W	J48	150	79.35	0.93	0.78
TaskPredictor	W & filepath	J48	1387	79.42	0.94	0.81

TABLE 4 Overview of the Best Accuracies a (Ranked Within Each Section)

	f	l	g	a	p	r	R G
Feature categories	Application Category	NB	500	91.75	0.97	0.92	3
	Content Category	NB	500	86.60	0.95	0.87	5
	All Categories	NB	750	82.47	0.93	0.83	8
	Resource Category	NB	1000	68.04	0.86	0.67	14
	Ontology Structure Category	J48	359	65.98	0.86	0.67	15
	Action Category	SVM-2^5	11	59.79	0.81	0.58	17
	Switching Sequence Category	NB	1832	40.21	0.66	0.39	20
Single features	window title	J48	100	85.57	0.95	0.87	6
	content in focus	NB	150	84.54	0.94	0.84	7
	accessibility object name	J48	100	80.41	0.92	0.81	9
	event-block content	NB	200	73.20	0.89	0.76	10
	accessibility object value	J48	150	71.13	0.89	0.72	11
	UICO datatype relations	J48	221	70.10	0.88	0.71	12
	used resources metadata	J48	1000	68.04	0.86	0.68	13
	used resources content	NB	175	62.89	0.84	0.65	16
	content of resources	NB	250	58.76	0.81	0.61	18
	accessibility object role	J48	31	54.64	0.79	0.55	19
Top k	k  = 6	NB	250	94.85	0.98	0.95	1
best single	k  = 5	NB	150	92.78	0.97	0.94	2
features	k = 4	NB	500	89.69	0.96	0.91	4