HelpMeOut Talk - CHI2010

What would other
programmers do?
Suggesting solutions to error messages
Björn Hartmann
University of California, Berkeley
EECS, Computer Science Division
bjoern@eecs.berkeley.edu
Daniel MacDougall, Joel Brandt,

Scott R. Klemmer
Stanford University HCI Group
End-user programmers outnumber
professional software developers.
To write code, end users often find and
modify existing examples.
HelpMeOut - What Would Other Programmers Do? Suggesting Solutions to Error Messages 2
End-user programmers outnumber
professional software developers.
To write code, end users often find and
modify existing examples.
This strategy does not work well for
debugging. (but it should)
Research Idea:
Help programmers fix errors by showing
examples how other programmers have
fixed similar errors in the past.
(In real-time, automatically)
Test Environment: Processing
images: processing.org
photo: d. shiffman
MIT Media Lab, Sensable Cities
Scenario
Architecture
Three Steps
Collecting example fixes
Finding relevant fixes
Presenting found fixes
Three Steps
Collecting Example Fixes
Goal:
Build a database of example bug fixes that
take code from a broken into a fixed state.
(With minimal user intervention.)
Possible approaches:
Web search
Web search
Mining version control repositories
BugMem, Kim, SIGSOFT’06;
Dynamine, Livshits, SIGSOFT’05
Web search
Mining version control repositories
BugMem, Kim, SIGSOFT’06;
Dynamine, Livshits, SIGSOFT’05
Instrumenting IDEs
inGimp, Terry, CHI’08
CommunityCommands, Matejka, UIST’09
Photoshop Tutorials by Demonstration, Grabler, SIGGRAPH’09
Approach:
Instrument programmers’ IDEs to
automatically collect errors and source
code changes that correct those errors.
(Caveat: no logic bugs)
How do we find the right source pairs?
Tracking compile-time errors
Save source and error message at each compilation

Look for transitions from broken to error-free state
Tracking runtime errors
Progress heuristic
If the code makes progress past the
previous point of error,
mark error as resolved
Tracking runtime errors
Implementation
1. At runtime, count how often each line is executed
2. On error, store line’s execution count
3. On next run, if count for error line exceeds previous
count, mark as resolved.
Important assumptions
1. Errors cause uncaught exception
2. Identical input across executions
Three Steps
Finding Relevant Fixes
Goal:
Given a user’s broken code, find most
relevant examples in the fix database.
Finding Relevant Fixes
Approach:
Use both error message and source code
structure to search for relevant fixes.
Determining similarity
patch
User’s code Example Example

(broken) (broken) (fixed)
Determining similarity
similar to ?
User’s code Example
(broken) (broken)
Finding fixes for compiler errors
Naive implementation:
Look at code surrounding error and
compute string edit distance.
Better:
Match on code structure, not text.
Better: match on structure, not text
Replace identifiers with lexical analyzer.

Assumption: Locality (i.e., fix is near error)
Finding fixes for exceptions
Can we use the same approach for code
with runtime errors?
No, because such errors lack locality.
String s = null; root cause

//...
//...
System.out.println(s); exception
Architecture Outline
Present found fixes
Goal:
Present fixes to users in a way that
facilitates reasoning about relevance &
applicability to their own code
Applying found fixes
: : ?
Example Example User’s code
(broken) (fixed) (broken)
Present found fixes
Approach:
1. Show before-after comparison
2. Offer explanations
3. Attempt to patch users’ code
where possible
Where do these explanations
come from?
Crowdsource them
from experts.
How can we best encourage
experts to contribute?
We ensure their efforts have
maximum impact.
Priority list of frequently returned fixes
Applying found fixes
patch patch
?
Example Example User’s code
(broken) (fixed) (broken)
Applying patches automatically
float [] A = new float[];

float [] Z = new float[10];
Applying patches automatically
Feasibility Study
Guiding Questions
What size database is needed to return
useful suggestions?
How useful are suggestions collected
through instrumentation?
Which types of errors are covered?
Which aren’t?
Method
2 workshops on Processing at California College of Art

8 participants; 39 person-hours of usage
Self-Report Data
Database Seeded with Textbook Examples
image: arkadian.eu
Results: Common errors
274 queries (7 per person/hour)
84% returned at least one suggestion
How useful are suggestions?
Manual analysis of queries and returned
suggestions:
Does at least one of the top 3 returned
suggestions lead to a solution?
HelpMeOut gave users a helpful
suggestion half of the time.
Analysis
Can’t smarter editors solve these problems?
Structured & syntax directed editors can

prevent syntax errors...
Cornell Programm Synthesizer, Teitelbaum
Scratch, Resnick, CAMC’09
...but they increase the viscosity of code.

Constant background compilation with
error hinting aids syntax error correction.
Future work
dynamic languages
suggest improvements
other authoring domains
bjoern@eecs.berkeley.edu
bid.berkeley.edu

HelpMeOut Talk - CHI2010

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

HelpMeOut Talk - CHI2010

Uploaded by

Copyright:

Available Formats

What would other

Daniel MacDougall, Joel Brandt,

Save source and error message at each compilation

User’s code Example Example

Better: match on structure, not text

Replace identifiers with lexical analyzer.

String s = null; root cause

float [] A = new float[];

2 workshops on Processing at California College of Art

Structured & syntax directed editors can

...but they increase the viscosity of code.

You might also like