There aren’t any good stock photos of “ten thousand”, so this piece will just have lots of kittens
I
started writing a blog in May 2016, partly because I kept writing rants
on Facebook that apparently were “too good not to be online somewhere”,
and partly because I was bored after my Master’s degree and wanted
something to do with my Sunday mornings.
Sleeping in, of course, was never an option.
This is Luna. Luna is my 6am alarm clock. Every. Single. Day
18
months later, and I’ve written about 100,000 words, been published in
all sorts of places, and am now getting regular offers to pitch to major
publications — more on this in the coming months.
And
most importantly of all, I got to 10,000 followers. This time last
year, it was 100 and about half of them were related to me.
All in all, it’s been a good year.
Pictured: Getting what you always wanted
So
what’s in store for the Health Nerd? You’ll be happy to know that this
year I’ve applied for a PhD with the University of Wollongong, which is
actually super exciting and not scary like it feels to me sometimes. I’m
also going to be — hopefully — releasing some episodes of a podcast
that I’ve started with a brilliant co-host. The topic will be science in
the media and I’m really excited to introduce all of you to my dulcet
tones over the airwaves.
I’m so much less awkward than I am in text.
What
does all of this activity mean to the blog? Nothing! I’ll still be
aiming for my regular one health story a week on Medium, as well as an
extra member’s-only article a month for all you subscribers who love
that extra content.
Pictured: “Extra content”
To
sum up, I’d just like to say thank you to you all. I’d never have made
it here without all you brilliant people following me and making this
all worthwhile. It was a fantastic 2017, and 2018 shows every sign of
being brilliant as well.
Social
media and digital executives in newsrooms already have a tough job
connecting their content to consumers via social media, but Facebook’s proposed changes in the algorithms of its ‘newsfeed’
are going to make it a lot harder. Social networks offer immense
opportunities for reaching vast new audiences and increasing the
engagement of users with journalism. The most important platform in the
world is about to make that more difficult.
Clearly,
this is a blow for news publishers who have spent the last decade or so
fighting a battle for survival in a world where people’s attention and
advertising have shifted to other forms of content and away from news
media brand’s own sites. They are clearly very concerned. Yet, could this be a wake-up call that will mean the better, most adaptive news brands benefit?
I’m
not going to argue that this is good news for news publishers, but
blind panic or cynical abuse of Facebook is not a sufficient response.
The honest answer is that we don’t know exactly what the effect will be
because Facebook, as usual, have not given out the detail and different
newsrooms will be impacted differently.
It’s exactly the kind of issue we are looking at in our LSE Truth, Trust and Technology Commission.
Our first consultation workshop with journalists, and related
practitioners from sectors such as the platforms, is coming up in a few
weeks. This issue matters not just for the news business. It is also
central to the quality and accessibility of vital topical information
for the public.
Here’s my first attempt to unpack some of the issues.
Mark Zuckerberg: making time on Facebook ‘well spent’
Firstly,
this is not about us (journalists). Get real. Facebook is an
advertising revenue generation machine. It is a public company that has a
duty to maximise profits for its shareholders. It seeks people’s
attention so that it can sell it to advertisers. It has a sideline in
charging people to put their content on its platform, too. It is a
social network, not a news-stand. It was set up to connect ‘friends’ not
to inform people about current affairs. Journalism, even where shared
on Facebook, is a relatively small part of its traffic.
Clearly,
as Facebook has grown it has become a vital part of the global (and
local) information infrastructure. Other digital intermediaries such as
Google are vastly important, and other networks such as Twitter are
significant. And never forget that there are some big places such as
China where other similar networks dominate, not Facebook or other
western companies. But in many countries and for many demographics,
Facebook is the Internet, and the web is increasingly where people get their journalism. It’s a mixed and shifting picture but as the Reuters Digital News Report shows, Facebook is a critical source for news.
From Reuters Digital News Report 2017
If you read Zuckerberg’s statement he makes it clear that he is trying to make Facebook a more comfortable place to be:
“recently
we’ve gotten feedback from our community that public content — posts
from businesses, brands and media — is crowding out the personal moments
that lead us to connect more with each other.”
His users are ‘telling him’ (i.e. fewer of them are spending less time on FB) what a plethora of recent studies and books
have shown which is that using Facebook can make you miserable. News
content — which is usually ‘bad’ news — doesn’t cheer people up. The
angry, aggressive and divisive comment that often accompanies news
content doesn’t help with the good vibes. And while the viral spread of
so-called ‘fake news’ proves it is popular, it also contributes to the
sense that Facebook is a place where you can’t trust the news content.
Even when it is credible, it’s often designed to alarm and disturb. Not
nice. And Facebook wants nice.
“We
can’t make money unless you keep telling us things about yourself that
we can sell to advertisers. Please stop talking about news.”
Another
accusation is that Facebook is making these changes because of the
increasing costs it is expending at the behest of governments who are
now demanding it does more to fight misinformation and offensive
content. That might be a side-benefit for Facebook but I don’t think
it’s a key factor. It might even be a good thing for credible news if
the algorithmic changes include ways of promoting reliable content. But
overall the big picture is that journalism is being de-prioritised in
favour of fluffier stuff.
Even Jeff Jarvis, the US pioneer of digital journalism who has always sought to work with the grain of the platforms, admits that this is disturbing:
“I’m
worried that news and media companies — convinced by Facebook (and in
some cases by me) to put their content on Facebook or to pivot to
video — will now see their fears about having the rug pulled out from
under them realized and they will shrink back from taking journalism to
the people where they are having their conversations because there is no
money to be made there.”*
The
Facebook changes are going to be particularly tough on news
organisations that invested heavily in the ‘pivot to video’. These are
often the ‘digital native’ news brands who don’t have the spread of
outlets for their content that ‘legacy’ news organisations enjoy. The
BBC has broadcast. The Financial Times has a newspaper. These
organisations have gone ‘digital first’ but like the Economist they have
a range of social media strategies. And many of them, like the New York
Times, have built a subscription base. Email newsletters provide an
increasingly effective by-pass for journalism to avoid the social media
honey-trap. It all makes them less dependent on ‘organic’ reach through
Facebook.
But
Facebook will remain a major destination for news organisations to
reach people. News media still needs to be part of that. As the
ever-optimistic Jarvis also points out,
if these changes mean that Facebook becomes a more civil place where
people are more engaged, then journalism designed to fit in with that
culture might thrive more:
“journalism
and news clearly do have a place on Facebook. Many people learn what’s
going on in the world in their conversations there and on the other
social platforms. So we need to look how to create conversational news.
The platforms need to help us make money that way. It’s good for
everybody, especially for citizens.”
News
organisations need to do more — not just because of Facebook but also
on other platforms. People are increasingly turning to closed networks
or channels such as Whatsapp. Again, it’s tough, but journalism needs to
find new ways to be on those. I’ve written huge amounts
over the last ten years urging news organisations to be more networked
and to take advantage of the extraordinary connective, communicative
power of platforms such as Facebook. There has been brilliant
innovations by newsrooms over that period to go online, to be social and
to design content to be discovered and shared through the new networks.
But this latest change shows how the media environment continues to
change in radical ways and so the journalism must also be reinvented.
Social media journalist Esra Dogramaci has written an excellent article
on some of the detailed tactics that newsrooms can use to connect their
content to users in the face of technological developments like
Facebook’s algorithmic change:
“if
you focus on building a relationship with your audience and developing
loyalty, it doesn’t matter what the algorithm does. Your audience will
seek you out, and return to you over and over again. That’s how you
‘beat’ Facebook.”
Journalism Must Change
The
journalism must itself change. For example, it is clear that emotion is
going to be an even bigger driver of attention on Facebook after these
changes. The best journalism will continue to be factual and objective
at its core — even when it is campaigning or personal. But as I have written before,
a new kind of subjectivity can not only reach the hearts and minds of
people on places like Facebook, but it can also build trust and
understanding.
This
latest change by Facebook is dramatic, but it is a response to what
people ‘like’. There is a massive appetite for news — and not just
because of Trump or Brexit. Demand for debate and information has never
been greater or more important in people’s everyday lives. But we have
to change the nature of journalism not just the distribution and
discovery methods.
The media landscape is shifting to match people’s real media lives in our digital age. Another less noticed announcement from Facebook
last week suggested they want to create an ecosystem for local
personalised ‘news’. Facebook will use machine learning to surface news
publisher content at a local level. It’s not clear how they will vet
those publishers but clearly this is another opportunity for newsrooms
to engage. Again, dependency on Facebook is problematic, to put it
mildly, but ignoring this development is to ignore reality. The old
model of a local newspaper for a local area doesn’t effectively match
how citizens want their local news anymore.
What Facebook Must Do
Facebook
has to pay attention to the needs of journalism and as it changes its
algorithm to reduce the amount of ‘public content’ it has to work harder
at prioritising quality news content. As the Guardian’s outstanding
digital executive Chris Moran points out, there’s no indication from
Facebook that they have factored this into the latest change:
Fighting
‘fake news’ is not just about blocking the bad stuff, it is ultimately
best achieved by supporting the good content. How you do that is not a
judgement Facebook can be expected or relied upon to do by itself. It
needs to be much more transparent and collaborative with the news
industry as it rolls out these changes in its products.
When
something like Facebook gets this important to society, like any other
public utility, it becomes in the public interest to make policy to
maximise social benefits. This is why governments around the world are
considering and even enacting legislation or regulation regarding the
platforms, like Facebook. Much of this is focused on specific issues
such as the spread of extremist or false and disruptive information.
In
October, Nubank announced our boldest product release since our
revolutionary credit card: NuConta, a single account that allows people
to save, invest and transfer money in real-time. NuConta is a complete
redefinition of how banking accounts should work, and here’s how our
design team arrived at this new concept.
NuConta — Main Screen and Account Screen
Start with why
Nubank’s core mission is to remove complexity and empower people.
The credit card, our first product, has achieved viral growth among our
customers by doing exactly this, but it’s a product accessible only to a
select population due to credit analysis requirements. In fact, we have
received over 13 million requests to get a Nubank card, but had to say
no to a considerable percentage of those people. Our customer base,
which is nearing 3 million accounts, is already representative — we’re
currently Brazil’s no.5 credit card issuer — but still limited when
compared to a market of 120 million Brazilians who are unbanked or have
to deal with expensive and abusive banking services that dominate the
market.
An example of the list of fees from traditional Brazilian banks
The
next logical step in bringing our mission and user experience to a
broader population, regardless of their income, credit scores or other
barriers, was to build a product anyone could have access to: our
version of a banking account, NuConta.
The team
Designing,
building and launching NuConta in 12 months was only possible because
of how the team was structured and governed. The squad,
as we call teams here, was composed from the very beginning of product,
financial, engineering and design talent, none of whom had built
something like this before. It was a group of unbelievably bright, but
most of all humble people, who were completely open to learning and having their opinions challenged.
This
was, of course, a fascinating and unique project to be working on, so
people couldn’t avoid coming in with their premature views and
expectations for what the final product should be. The role of design
was as much about facilitating alignment between these expectations,
bringing clarity to compromises and decisions, and replacing our biased
opinions with real people’s pains and needs, as it was about designing
UI flows, copy, or making prototypes.
Design in an Agile Team
As
much as we designers like to structure and visualize our process, what
happens in our reality is more complex and asynchronous than the
diagrams we’ve grown accustomed to.
Doing
solid foundational research before starting anything else can surely
de-risk product placement, but inevitably delays engineering and thus
time-to-MVP.
What we did then, was to work in parallel threads:
The Design team working to figure out how to shape the product to answer the customers’ needs.
The Engineering team working at full-speed to put the banking infrastructure together.
The Intel and Legal team powering through the sea of legislation, third-party interactions and contracts.
How was our process to get to the MVP
Each
of these threads constantly fed each other with their learnings and
advancements, and we adjusted the product accordingly up to the last
moment before shipping.
“Go
do the best research you can, come back to tell us what you’ve learned,
and we’ll adjust as we go. We’re not afraid of throwing our code
away.” — The best thing a UX person could hear from their dev team.
Foundational Research
We
were committed to challenging our most basic assumptions about how
people understand and use banking services, so our initial conversations
were purposefully broad and naive. We started with a simple list of
questions, but we adjusted frequently along the way.
How do people currently manage their money?
What do they think about the different account types? Do they even understand them?
Do people save money? Why? Where?
Are people aware of product options in the market?
How do they feel about the products on the market? What attracts them? What scares them?
How complicated is the current landscape for a person with no financial background?
If people want to start saving money now, what would be their first step?
The
fastest way for us to start asking (and testing) these questions was to
sneak them into studies already being conducted by other product teams
(namely, credit card and Nubank Rewards). We learned a lot very quickly
by “stealing” a few minutes of our colleagues’ interviewing time, but
their target audience could be skewed to non-target segments.
To
balance that out, we got out of the building to do intercept interviews
in places like public universities and spaces, where we could reach a
wider variety of people who were not already Nubank customers.
Intercept interviews at the University of São Paulo and Av. Paulista
Designing for financial services is about getting the words right as much as it is about getting screens right
These
in-depth early interviews took 40–90 minutes each. We sat down and very
informally let people talk about their financial lives, how they felt
about saving, investing, transferring, and spending their money. We also
experimented with some visual aids such as card-sorting and very cheap
prototypes, but honestly our best feedback tools were our competitor's
websites and products. We learned an incredible amount just by having
people experiment with things other companies have already built.
Analysis of interview transcripts and sense-making phase
Personas
By
combining the data from these interviews with others conducted in our
lab, and also talking to a lot of our own employees about their
finances, we were able to come up with eight personas that represent a
gradient of behavior and demographic patterns.
Personas
can yield mix results, but we’re confident they were essential in this
case. Here’s some of the value personas brought to our process:
The team learned that people represent a gradient of experience and behaviors.
What we thought would add the most value, was often biased as it was based on our financially savvy perspective.
The team learned that it would be impossible to design a product for every type of person, and that we had to choose some group to focus on.
Our discussions evolved from being biased by personal opinions to being based on our chosen target’s jobs-to-be-done, pains and needs.
Everyone in the team now remembers who Diego (one of our personas) is,
what he struggles with, and that our efforts should be focused on
making his financial life less complicated and more empowered, instead
of our own.
Our eight generated personas
Problem definition
With
well-defined personas, we now had a lens through which we could look at
the problem. Our initial mission, which was very broad, could now be
better expressed in sentences that the whole team could agree on:
What NuConta is / does
NuConta
is an evolution of your current and savings accounts, designed for
people looking for a more accessible and easy-to-use bank account. It’s
free of complexity, doesn’t charge abusive fees, and makes your money
grow at a fair interest rate. Differently from big banks and other
fintech pre-paid products, NuConta has zero bureaucracy, less friction
to adopt and a superior user experience.
Likewise, defining what NuConta was not supposed to do, at least for launch, was equally helpful in creating vision alignment and prioritization:
What NuConta is not / doesn't
NuConta
doesn’t serve people looking for advanced investments, home brokerage,
paid wealth management advice. It tries to be accessible and
self-explanatory, but without becoming an educational product. At its
early stage, it also will not support goals or any other kind of gaming
mechanics.
We
learned from this process that defining a problem is sometimes harder
than sitting down to solve it. After dozens of long work days, heated
meetings, lots of head-scratching, word-smithing, and sense-making about
the material collected during research, the team was finally committed
to a unified vision of the product that reflected our customer’s pains
and needs instead of our own. We were now ready to deep dive into
exploring solutions that would eventually become NuConta.
Part
II of this post will tell how the next steps played out: product
ideation, UI explorations, concept & usability testing, onboarding
design, copywriting, and implementation.
Hit Subscribe to be notified when Part II comes out!
This
past summer I managed the largest acquisition campaign in my company’s
history. I work at HubSpot, a marketing software company that
popularized lead-gen campaigns and the whole idea of “inbound
marketing,” so this is no small feat (we’ve run massive campaigns over
the years).
The campaign, Four Days of Facebook, drove 10x the number of average leads of a typical acquisition campaign and 6x the lifetime value of projected customers.
But I didn’t do it alone. This campaign involved 11 teams and 33 people who directly contributed to the work.
Cross-functional
campaigns like this can be big, complicated, and challenging which is
why they so often take a boss or recognized leader to make them happen.
So I wanted to share my experience as a “non-boss.” I hope it encourages
other individual contributors out there to get their co-workers in
other departments excited about working on high-impact, cross-functional
projects.
Pre-planning: create alignment
You won’t have all the answers on day one, but make sure every conversation you’re having at this stage focuses on one thing: impact.
You’ll be asking a lot of people to work hard on something outside of
their normal day-to-day, make it clear that your asks will translate
into business results.
Meet with senior leaders of each team before you ask for their employees commitment on helping. Again, make it clear that you won’t be wasting anyone’s time, you’re out to generate big results.
Have a kickoff meeting with the team who will be responsible for delivering the work. At
a high-level, you want to let everyone know that you have senior
leadership buy-in and the project will be worth their time. On a more
tactical level, you’ll also want to get people up-to-speed on the tools
you’ll be using to manage the project.
Go the extra mile to develop a team culture for your team.
You know how developers name their projects crazy-sounding names? It’s
surprisingly effective! Give your temporary team a name that makes
people feel like they’re a part of something, set up an email alias, and
create a Slack channel. Get people excited!
Throughout
the pre-planning stage, keep your vision front and center. For Four
Days of Facebook we were partnering with Facebook, a fact I repeated
constantly.
If people are excited and engaged with your vision, they’ll put up with the inevitable bumps as you achieve lift-off.
During: maintain momentum
The Progress Principle
is the idea that humans love the satisfaction of wins, even if they’re
small. It’s your best friend as you seek to keep multiple teams and
dozens of people aligned and moving in the right direction–constantly
show (and celebrate) forward progress.
Display it:
I put together a registration goal waterfall chart that was updated
everyday to show progress. It’s motivating to close-in on and cross that
goal line.
Never shut up about it:
I linked to information about this campaign in my email signature,
Slack rooms, wherever I had the attention of my co-workers. And that
information was short, sweet, and up-to-date.
Be a good partner: You’re
not technically the manager of the people on a cross-functional team,
but you should implement some management best practices: give people
autonomy, figure out how they like to work and what kind of support they
need from you.
Ask for feedback:
I asked questions constantly– Is this system or process working for
you? Can I set up these reports in an easier way? At one point during
this campaign I asked the senior manager of a few folks working on the
project if she had thoughts on how I could run it better, she told me
she would love to see weekly updates sent to her and other senior
managers. I was avoiding this as I didn’t want to clutter inboxes, but
it ended up being one of my best tools for building internal momentum
around the campaign.
Don’t overlook the fundamentals of good project management. A framework like DARCI
makes roles & responsibilities super easy so you the project lead
can just say, “This meeting is for people who are Responsible and
Accountable only, we’ll be covering deadlines for next week”, or “This
meeting is for people that need to be Informed, it’ll be a milestone
check-in.”
Find a project management framework, and stick to it.
Wrapping up: close-the-loop
I
run 4–5 acquisition campaigns at HubSpot every quarter and running a
campaign of this size and impact was a complete rush and I can’t wait to
do it again. But before jumping into the next big project, it’s
important to do a clean wrap-up, I want people to be excited to work
with me and my team again in the future.
Say thank you:
Do it publicly via a company announcement or email, and privately. I
wrote handwritten notes to every person who contributed to this
campaign.
Share results soon:
Share the quantitative results, but don’t miss Twitter comments from
attendees, feedback from partners, or the accolades of your co-workers.
This is your chance to make it clear that you promised impact and
delivered it.
Look for improvement opportunities:
Because no matter how successful your campaign was, there are
opportunities to do better — Were any deadlines missed? Why? Did any
team members not work well together? Can this be addressed?
It’s
easy to get stuck in a rut of executing one marketing campaign after
the next, and it’s scary to think about leading a big cross-functional
project that could potentially fail publicly.
But
so often the answer to higher impact is better collaboration. Learning
how to lead across teams 10x’ed the impact I was having at my company, I
hope it does the same for you.
By now you must have heard this buzz word called “ Progressive Web App ”. Let me give you a simple definition about PWAs.
Progressive
Web Apps are Web Apps which combines the best features of Web and
Native Apps. It is progressive because it is constantly progressing.
Why Progressive Web Apps?
Now let’s talk about the “WHY”. Why Frances Berriman and Alex Russell came up with a concept called Progressive Web Apps? Before that let’s understand what problems does it solve.
Problems with Native Apps?
We
all have used Android or iOS apps on our smartphones. We use them for
all kinds of thing. But while installing any Android/iOS apps we go
through these problems :
Is this app worth downloading?
Do I have enough space?
My available data is not sufficient.
One recent survey shows that people are turning away from Android/iOS apps,
because not all app experiences are satisfying or worthwhile. Some
people simply don’t want any more apps on their phone, some even
hesitate to download any app.
If
you take a look at the apps installed on your mobile right now there
might be at least a dozen apps that you do not use regularly. Sometimes
apps only works good when the phone has an active internet connection.
The
irony is that most of the apps have a fully responsive website
performing the same functions. So why waste your precious disk space and
your internet data on your smartphone by installing the native app? The
average size of apps that we install from play store/app stores would
range from 30–200MB. Moreover, these app needs to updated every week!
But Progressive Web Apps are within some
KBs and are automatically updated. Thanks to service worker. 🙂
What if a website can do that and much more than a Native app? This is what Progressive Web Apps (PWA) are trying to accomplish.
In short, Progressive web apps combine everything that is great about a native mobile application with everything that is great about a mobile website.
Some other ways I like to describe them:
“The best of the web, plus the best of native apps”
Progressive — The word progressive means it works for every user, regardless of browser choice because they’re built with progressive enhancement as a core tenet.
Responsive — Automatically adjustable to any form: desktop, mobile, tablet etc.
Load Time — Progressive Web Apps are instantly available
App-like — Feels like a mobile app with app-style interactions since it’s built on the app shell model.
Fresh — Always up-to-date so you do not need to update it again and again like any other Android/iOS apps.
Safe — Served via HTTPS to ensure content is securely delivered
Engaging — Features like push notifications, etc. makes it very engaging.
Installable — Allows users to install the website as an app on their home screen without the taking user to an app store.
Linkable — Easily shared via a URL and do not require complex installation.
Benefits of making a Progressive Web App rather than building a fully functional Android App?
Cost Effective
— For an app publisher, the biggest advantage is the cost saving in
terms of app development and maintenance. Because it is assumed that
making a website is lot more easier than making a Android App.
Cross Platform—
Unlike any other apps, Progressive Web Apps are not restricted to any
specific platform. That means you do not need to develop separate
versions of app for different platforms.
10 REASONS PROGRESSIVE WEB APPS WILL BE THE FUTURE OF APPS
Flipkart,
India’s largest e-commerce site, decided to combine their web presence
and native app into a Progressive Web Application that has resulted in a
70% increase in conversions. Flipkart’s PWA takes merely 100KB to
download, is 100 times smaller than the 10MB Android app and 300 times
smaller than the iOS app. Repeat visits take less than 10KB to download.
How to use a Progressive Web App?
Probably,
you must be wondering how to use a Progressive Web App! Well, you can
just click on any links mentioned above. But provided that you must be
using a smart phone in order to install it your own device.
Now let’s click open Flipkart in chrome. ( You must be using a Mobile Device ). Below are some screenshots which demonstrates the installation of a PWA.
Chrome Dev Summit Announcments:
Progressive Web Apps: Great Experiences Everywhere (Google I/O ‘17)
Microsoft is supporting Google’s Progressive Web Apps platform and that’s great news for everyone.
Microsoft recently announced
that Progressive Web Apps (PWA) will soon be supported by the Microsoft
Edge web browser. Moreover, a few apps hitting the Windows Store — like
Slack and Trello — are now mixing PWA with Electron and elements of the Universal Windows Platform (UWP).
My colleagues and I wanted to create something that would make people go “wow” at our latest hackathon.
Because
imitation is the sincerest form of flattery and IoT is incredibly fun
to work with, we decided to create our own version of Amazon Go.
Before I explain what it took to make this, here’s the 3 minute demo of what we built!
There were four of us. Ruslan,
a great full-stack developer who had experience working with Python.
John, an amazing iOS developer. Soheil, another great full-stack
developer who had experience with Raspberry Pi. And finally, there was
me, on the tail end of an Android developer internship.
I
quickly realized that there were a lot of moving parts to this project.
Amazon Go works on the basis of real-time proximity sensors in
conjunction with a real-time database of customers and their carts.
We
also wanted to take things a step further and make the entry/exit
experience seamless. We wanted to let people enter and exit the store
without needing to tap their phones.
In
order to engage users as a consumer-facing product, our app would need a
well-crafted user interface, like the real Amazon Go.
On
the day before the hackathon, I put together a pseudo-design doc
outlining what we needed to do within the 36 hour deadline. I
incorporated the strengths of our team and the equipment at hand. The
full hastily assembled design doc can be seen below.
There were six main components to EZShop, our version of Amazon Go.
A quick diagram I whipped up visualizing the components of this project
The Kairos Facial Recognition API
The Kairos facial recognition API
was a fundamental component for us. It abstracted the ability to
identify and store unique faces. It had two APIs that we used: /enroll and /verify.
/enroll is described as:
Takes a photo, finds the faces within it, and stores the faces into a gallery you create.
We enrolled all new customers into a single “EZShop” gallery. A unique face_id attribute would be returned and stored with the customer’s registered name in our real-time database.
When we wanted to verify a potential customer’s image, we would POST it to the /verify endpoint. This would return the face_id with the highest probability of a match.
In
a real-world implementation, it probably would have been a better idea
to use a natively implemented facial recognition pipeline with
TensorFlow instead of a network API. But given our time constraints, the
API served us very well.
The Realtime Firebase Database
The
Firebase database was another fundamental piece to our puzzle. Every
other component interacted with it in real time. Firebase allows
customized change listeners to be created upon any data within the
database. That feature, coupled with the easy set-up process, made it a
no brainer to use.
The
schema was incredibly simple. The database stored an array of items and
an array of users. The following is an example JSON skeleton of our
database:
New users would be added to the array of users in our database after registering with the Kairos API. Upon entry or exit, the customer’s boolean in_store attribute would be updated, which would be reflected in the manager and personal app UIs.
Customers picking up an item would result in an updated item stock. Upon recognizing which customer picked up what item, the item’s ID would be added to the customer’s items_picked_up array.
I had planned for a cloud-hosted Node/Flask server that would route all activity from one device to another, but the team decided that it was much more efficient (although more hacky) for everybody to work directly upon the Firebase database.
The Manager and Personal Customer Apps
John, being the iOS wizard that he is, finished these applications in the first 12 hours of the hackathon! He really excelled at designing user-friendly and accessible apps.
The Manager App
This iPad application registered new customers into our Kairos API and Firebase database. It also displayed all customers in the store and the inventory of store items. The ability to interact directly with the Firebase database and observe changes made to it (e.g. when a customer’s in_store attribute changes from true to false) made this a relatively painless process. The app was a great customer-facing addition to our demo.
The Personal Shopping App
Once the customer was registered, we would hand a phone with this app installed to the customer. They would log in with their face (Kairos would recognize and authenticate). Any updates to their cart would be shown on the phone instantly. Upon exiting the store, the customer would also receive a push notification on this phone stating the total amount they spent.
The Item Rack, Sensors, and Camera
Soheil and Ruslan worked tirelessly for hours to perfect the design of the item shelf apparatus and the underlying Pi Python scripts.
The item rack apparatus. Three items positioned in rows, a tower for the security camera, and ultrasonic sensors positioned at the rear
There were three items positioned in rows. At the end of two rows, an ultrasonic proximity sensor was attached. We only had two ultrasonic sensors, so the third row had a light sensor under the items, which did not work as seamlessly. The ultrasonic sensors were connected to the Raspberry Pi that processed the readings of the distance from the next closest object via simple Python scripts (either the closest item or the end of the rack). The light sensor detected a “dark” or “light” state (dark if the item was on top of it, light otherwise).
When an item was lifted, the sensor’s reading would change and trigger an update to the item’s stock in the database. The camera (Android phone) positioned at the top of the tower would detect this change and attempt to recognize the customer picking up the item. The item would then instantly be added to that customer’s cart.
Entrance and Exit Cameras
I opted to use Android phones as our facial recognition cameras, due to my relative expertise with Android and the easy coupling phones provide when taking images and processing them.
The phones were rigged on both sides of a camera tripod, one side at the store’s entrance, and the other at the store exit.
A camera tripod, two phones, and lots of tape
Google has an incredibly useful Face API that implements a native pipeline for detecting human faces and other related useful attributes. I used this API to handle the heavy lifting for facial recognition.
In particular, the API provided an approximate distance of a detected face from the camera. Once a customer’s face was within a close distance, I would take a snapshot of the customer, verify it against the Kairos API to ensure the customer existed in our database, and then update the Firebase database with the customer’s in-store status.
I also added a personalized text-to-speech greeting upon recognizing the customer. That really ended up wowing everybody who used it.
The result of this implementation can be seen here:
Once the customer left the store, the exit-detection state of the Android application was responsible for retrieving the items the customer picked up from the database, calculating the total amount the customer spent, and then sending a push notification to the customer’s personal app via Firebase Cloud Messaging.
Of the 36 hours, we slept for about 6. We spent our entire time confined to a classroom in the middle of downtown Toronto. There were countless frustrating bugs and implementation roadblocks we had to overcome. There were some bugs in our demo that you probably noticed, such as the cameras failing to recognize several people in the same shot.
We would have also liked to implement additional features, such as detecting customers putting items back on the rack and adding a wider variety of items.
Our project ended up winning first place at the hackathon. We set up an interactive booth for an hour (the Chipotle box castle that can be seen in the title picture) and had over a hundred people walk through our shop. People would sign up with a picture, log into the shopping app, walk into the store, pick up an item, walk out, and get notified of their bill instantly. No cashiers, no lines, no receipts, and a very enjoyable user experience.
Walking a customer through our shop
I was proud of the way our team played to each individual’s strengths and created a well put-together full-stack IoT project in the span of a few hours. It was an incredibly rewarding feeling for everybody, and it’s something I hope to replicate in my career in the future.
I hope this gave you some insight into what goes on behind the scenes of a large, rapidly prototyped, and hacky hackathon project such as EZShop.
Hardik Gandhi is Master of Computer science,blogger,developer,SEO provider,Motivator and writes a Gujarati and Programming books and Advicer of career and all type of guidance.
