Optimization in Crowdsourcing

Optimization in Crowdsourcing


Thank you, and it’s a nice
follow up on the first part of Collin’s presentation,
which was on crowdsourcing. So I was asked to speak
on optimization for crowdsourcing and education. I’m not gonna speak
on education very much, because I haven’t
done that much work. But I’ve done some work
on crowdsourcing and in particular in the context of
sequential optimization which is what I’m going to
be talking about. It is really nice to hear
the word POMDP in the morning session come in twice. I feel
like I am in an AI setting, so I am going to be talking
more about POMDPs. Just to start this is
joint work with Dan Weld. Most of this work has happened
at the University of Washington even after my coming
back to India. Some of the work that has been
happening in this particular field is with Dan Weld and
two of our students who have are about to get degrees at
University of Washington. So this is the 30,000 feet view. Crowdsourcing has become huge. And this is probably the only
slide I’m gonna connect with social good. So crowdsourcing is social
good from one vantage point. It has, starting 2006, when the
word crowdsourcing was formed, and then Amazon Mechanical Turk
and many, many different platforms have
really made crowdsourcing grow very huge and
has grown really rapidly. And I would make the point that
it democratizes labor, and this is I think also the point
that Collin was trying to make that now we can give labor to
a lot of people in the world. At the same time there
are some challenges in making crowdsourcing as successful
as we want it to be. And it is our thesis that AI,
machine learning, optimization can really make crowdsourcing
achieve its potential, it can reduce work errors,
and in our experiments we have sometimes been able to
reduce errors by up to 85%. So we have really made and
sometimes we have found crowdsourcing to be super
super successful, okay, using the AI techniques. So now, I don’t have to go too
much into how crowdsourcing came up, Wikipedia is one
of the first examples of a crowdsourced encyclopedia,
and look at the viral growth of Wikipedia articles that happen
when people got interested. Citizen Science was being
referred to by Bart earlier and Galaxy Zoo was one such example
which was getting 50 million classifications in a year, which
machine learning algorithms were not able to do at the time. And it was not just that,
the human workers, human citizen scientists
actually ended up discovering special form of galaxies
known as the Pea Galaxies, which the astronomers
had no idea about. Most of my talk is gonna be
about labor marketplaces which are projected to grow
by $5 billion by 2018. These are older
projections and some of the older statistics, I couldn’t find
the new statistics. But basically for example at
Amazon Mechanical Turk we had more than half a million workers
three or four years ago. And oDesk was seeing 35
million hours clocked in their platform in 2012. Now
they have become UpWork. So there’s so much going on. There’s so many very interesting
strengths which got me really excited about crowdsourcing. For example, world becoming
a unified labor force. Like a global
meritocracy of kinds. Whether you are in India whether
you are in Africa whether you are in some Pacific island,
as long as you have internet connectivity and skill to monetize
digitally, then you can probably get work. This is such a cool
thing for mankind in general. And of course it had many other
strengths, not from the social side, social good aspect like it
was perfect for startups, it was cloud
computing for human work. Really interesting new
applications could come about, for example, one of my colleagues thought
about an app for blind people. Where blind people stuck in
a new circumstance can actually ask a question and some crowd
worker can answer their question based on the image
that they are sending. The things that you would
not have expected if such a thing was not available and
so on. But you can find pretty much
every kind of expertise on that crowdsourcing platform. At the same time there
are lots of challenges, which is what got us interested. The most important challenge was
high variance in worker quality. There are challenges
about how to track the quality of output and in
general broad challenges about how do you get high
quality output? Because these are crowdworkers,
some workers are awesome, some workers are not that great,
some workers need training, how do you manage this
large enterprise? And there are many other things
where AI can play a role like, usually you divide a complex
task into small micro tasks and so how do you divide it,
how do you test such work flows? How do you optimize them? How do you figure out who’s the
right worker to be working on my kind of tasks and
so on and so forth? So we have really worked very
hard on demonstrating the value of the AI and
machine learning for crowdsourcing. At a high
level it works like this. So imagine this is an AI agent,
this is your requester who is telling you what tasks
need to be solved, and every AI system is operating
in some environment and so here my environment is
a crowdsourcing platform. And so when a new task comes
along, the AI system figures out which jobs to give out and
when work comes out, you can do your learning and
you can your planning. So you can ask questions like,
what are my workflow parameters? What are the optimal parameters? What are individual
worker’s abilities? What are they good at? What are they may
not be as good at? How difficult is my task,
and so on? So then by learning all these
parameters then you can control the task
better to figure out, if I have multiple workflows,
which workflow do I select? Within a workflow
which job do I post? When do I know that my work has
been done at a high quality? If I have a worker who is good,
but has some hole in their understanding, when
do I teach them? When do I test their quality
and so on and so forth? So we have done a reasonably large
body of work in this space, and let me just give you a couple
of quick examples of the things. For example, everybody starts out with
these simple yes, no questions. Is this bird an Indigo Bunting? Now I can ask several of you and
some of you will say yes, some of you will say no, and
hopefully people who know this thing will give us answers,
but they may make mistakes. So can we do some kind of
consensus on top of them? But not just consensus. When do we know we have
a good quality answer and when do we know that we
need to ask more people? That becomes a decision theory
question and so when using POMDPs in a cost controlled setting
if you did dynamic optimization, dynamic sequential optimization,
we got much higher accuracy than the static controller
for the same cost. So this got us interested, but then we went to
more complex tasks. Suppose this is your doctor, he
wrote this nice thing for you. Now, how do you know
what is going on? So, you can give it
to a crowdworker and in $0.27 in this particular
case, they were able to get the full transcription using
a very interesting workflow. But they didn’t know how to
optimize this workflow, we did. So when we optimize this we
found that we were able to use the same amount of money, but get much higher quality of
image transcription and so on. We have done taxonomization
of all the items. And again, one of the graduate
students who was an HCI graduate student came up with a very
creative workflow to do this taxonomization, but she was
not an optimization person. And so, when you model this for optimization you find that for
13% of work you can get the same quality if
you use optimization carefully. We have many,
many examples of this. Like, if you have lots of workers
with varying abilities and if you have lots of questions
with varying difficulties, we know which particular worker
to give each question to and again red is our thing,
and higher is better. We have thought about
when to test a worker, and when not to test a worker, and again higher is better and
we are the red one. We have thought about when
to train a worker, and what to train the worker on. And again green is what we
could eventually achieve, but we were able to achieve
red in our experiments. And in the latest work, we have done not just
quality-cost optimization, but, we have also done
quality-cost-completion time optimization. If I start paying
each worker more, then more workers will take my
job, my time will reduce, but my budget will exhaust quickly. So I will not be able to
solve the whole task, so my quality will go down. There’s a very interesting
interplay that happens when I start doing this
three-way optimization, how to manage
the whole optimization. So I’m be able to use
my budget the best, with the parameters
that I have in my task. We have a latest
paper coming out. So at a high level we have
looked at intelligent control and optimization in the
context of smart crowdsourcing. Crowdsourcing has a lot
of advantages but there are some challenges
to be resolved. For example,
we can figure out what to ask, how many times to ask,
who to ask, how/when to teach, when to test, when to stop. All these actions can be
taken by our AI agents using POMDPs as its base, and
we can do it for data quality. In other works,
we’ve also done it for classifier quality. If I’m in
an active learning setting, then I can use workers
very effectively. And from our point of view, from an AI-minded person’s point
of view, it’s a small step towards the vision of human
intelligence and machine intelligence coming together
to achieve something bigger. In our more recent work we
are not only thinking about democratization of workers, but also thinking about
democratization of requesters. Suppose you as an individual
want to give out a crowdsourcing task, can I make it easy for
you to crowdsource? There are a lot of
good practices, but it’s not very streamlined, still. You don’t know how
to make the right task, you may be confused
about your own task. You may not know what the workers are getting
confused about. How do you improve
the task design? So we’ve done an HCI interface, where we’ll give you the
interface that will allow you to learn your task better using
interactions with workers. So this is the work that we
have done in the context of crowdsourcing. I also wanted to add that in my
other lives, I do research in natural language processing, and
recently we have started two very interesting projects. One
is on analyzing legal data sets. So lots of court data in Indian
judicial system has been languishing. And we have started analysis to
understand which courts are outperforming other courts, which
districts are doing well, which districts are not doing well,
where are the cases getting stuck, so that we can inform
the legal department someday. And in second project, we have
started a healthcare initiative, where we’re trying to
read MRI images and CAT scans automatically
to see if we can help the radiologists do their
job more effectively. And since we’re talking
about social good, for young people I also advised
a dating company to help you find your right partner. So there is a lot of very, very
interesting stuff that we can do as an AI person in the mix, but
I always feel that until we get the right partner to partner
with, who is the domain expert, it’s a non-starter for us. So in the remaining time as we
are doing these discussions, I would love to hear more about
interesting problems from domain experts, where AI people can
actually help contribute. So I’ll stop here.>>[APPLAUSE]

Leave a Reply

Your email address will not be published. Required fields are marked *