« Links from my Metavante Client Conference presentation | Main | PayPal markets its credit card to users at logout »

Fallout from Rudder's mishap, will it impact all third-party PFM apps?

By Jim Bruene on May 20, 2009 7:27 PM | Comments (7)

imageYesterday, Rudder suffered an embarrassing email glitch that affected 732 customers. In the pre-Internet days, no one other than those few hundred customers, and a few of their friends, would have heard about it. Even in the days before blogs became common, pre-2007, it's unlikely the story would have made it to the mainstream press.

And even last year, before Twitter, the story might have died without ever crossing over to the mass media. But when it comes to breaking news and company gaffes, it's a whole new ball game. Everyone wants 15 minutes of fame as an investigative reporter, and Twitter is the dream platform.

I'm going to recap the problem, and how the news broke, in excruciating detail, because it illustrates the power of Twitter- and blog-fueled grassroots reporting. If you are a financial services company, think about how you could use social media to help with damage control should something similar happen to you.  

What happened at Rudder
According to the detailed description first published in TechCrunch and then later published by the company on a new blog created specifically for this issue, an email upgrade the night of May 18 caused 732 users to receive dozens of email updates containing balance and transaction information of other users. Only Rudder users with email addresses that begin with "a" or "b" received the erroneous emails because the company stopped the email job at that point after realizing the "upgrade" had gone terribly wrong.

Besides seeing the info in the email updates, the bigger security/privacy problem was that unauthorized users were able to click through email links to access the full aggregated account at Rudder.com (see screenshot in the TechCrunch article). However, at no time could anyone actually log in to anyone's bank account or move money in any way.

Luckily, Rudder, like all account-aggregation companies, does not include account numbers or personal details in the updates. However, the email addresses of each user was displayed, so any of the 732 customers using an email address at Rudder that can be traced back to their real name, had their financial details exposed to hundreds of users. 

How the news broke
At 5:36 AM yesterday (19 May), Twitter user @adambassador tweeted this:

image 

And @adambassador didn't stop at that. He took the time to search and communicate warnings directly to several other users who'd recently mentioned "Rudder.com" on Twitter. Adambassador would go on to tweet 21 times yesterday about the Rudder problem.

One of the people who heard from @adambassador was financial services consultant and blogger, Mike Linskey (@mikelinskey) who'd just Tweeted about several of the PFM companies he'd seen at our FinovateStartup conference, including Rudder.com. Mike then posted the problem to his Fincision blog at 8:04 AM, and at Mike's request, adambassador posted screenshots of the emails to document the problem, which were then published in Mike's blog entry.

image

At 10:05 AM, using Twitter, Mike alerted the blog Mashable about the Rudder problem. A half-hour later, Mashable, the fifth largest blog in the country (see note 1), posted the story citing adambassador's tweets and Mike's blog entry. From the Mashable blog entry (below), the problem was retweeted 115 times (see the retweet button below on left).

image

Then at noon, the second largest blog, TechCrunch, with more than 2 million subscribers, posted the story. And because of high comment activity, it stayed on the top of TechCrunch most of the day (see screenshot below), generating 58 comments.

How Rudder handled it
By almost any standard, Rudder did a good job responding. Although their reply took more than 10 hours since the error was first reported on Twitter, Rudder's CEO posted a detailed comment on the TechCrunch (scroll down to his comment at 4:38 PM here) and Mashable posts, apologizing for the error and explaining in great detail what had happened. 

In addition, Rudder created a special "Rudder Update" blog (see screenshot below) apologizing, explaining the mishap and exactly what info was mistakenly displayed, and detailing the steps they were taking to fix the problem and help affected customers:

  • Turned off the email system entirely
  • Contacted each affected user individually and offered them a complimentary subscription to an identity theft service
  • Engaged an independent security auditor to survey its system and look for weaknesses
  • Published a URL for users to go in and delete their accounts if desired

Analysis
Rudder did a good job considering the situation. It was smart to comment on TechCrunch and Mashable, and the new damage-control blog site was a savvy move. And the company did an exceptionally good job with the tone and wording of its mea culpa.

That said, the company could have used social media better. The company's Twitter page (@userudder) and that of its CEO (@nikhilroy) were silent all day. A short Twitter posting, even "we've stopped all emails and are working on it" would have reassured users and potentially made the Mashable post less alarming. Also, the company didn't have a blog, so there was no place where they could post periodic updates during the day. It was complete silence for 11 hours, other than the interview with TechCrunch's Erick Schonfeld mid-day.

Impact on third-party PFM credibility
While this was embarrassing and violated the privacy of several hundred users, there will likely be no financial loss to anyone. There was no data breach or stolen account numbers. Even a single bank account statement stolen from a mailbox could cause more potential financial damage.

And even though third-party PFM providers have had a relatively spotless record for security/privacy, this mistake, now well-documented in two of the largest online publications in the world, will be cited in the media for years, to cast doubt on the security of online personal finance.

It might cost the industry a point or two in short-term market share, but it would take something much worse to materially slow growth. Even Rudder should be fine. By addressing the issue in a highly professional way on the same day, most customers will be reassured, at least those that weren't directly impacted.

The bigger lesson here is the need for damage-control procedures that take into account the power and speed of new media (note 3). The entire episode could have--prior to Twitter and the blogosphere--been known to just a few hundred customers of a very small company, but instead traveled from a lone tweet to a large splash across the homepage of a major publication, all within a 6-hour period.     

TechCrunch featured the Rudder post on its main page most of the afternoon (19 May 2009)

image

Special damage control blog created by Rudder yesterday
(19 May 2009; link)

image

Note:
1. Ranking by Technorati authority (here)
2. Thanks to Mike Linskey for the tip yesterday morning.
3. Also, account aggregation users should use an email address that is not directly associated with their name.

Comments (7)

Most Recent Posts:

TrackBack

TrackBack URL for this entry:
http://www.netbanker.com/cgi-bin/mt/mt-t.cgi/2126

7 Comments

It is unfortunate for those affected that this occurred, but security and privacy breaches do happen and it's a measure of both the company and industry in how well they and we respond to them.

I think it's important to note that this was a privacy problem and not a security issue and that it does not appear to be a problem with the way their site is implemented.

For some perspective, for sites that do not store FI account data (account numbers and login information), the risk is more of a privacy issue than security and is about as severe as someone finding a grocery store receipt and I would argue it's even less as store receipts often include bank card/credit card information as well as other data than can be mined.

That being said, privacy and security is of utmost importance to all of us and any such problems, no matter how severe or widespread they are should be treated seriously. It could have been much worse.

For what it's worth, from your summary of events, they have dealt with the issue in a reasonable way and we can all learn from the experience.

Vince, it was a bigger problem than you've made it out to be. In each of those emails were details of where these people spend their money and how much. Utilities, restaurants, friends and so on. From that information, a financial criminal can do a lot. Ever hear of blackmail?

I closed my rudder.com and my mint.com accounts after this. No more risk from the PFMs for me.

Adam

Adam, My apologies, it was not my intent to trivialize the issue. Only to point out that for most people that list of transactions is usually quite mundane, I know mine is.

Second, the fact that those transaction details are sent via email makes them insecure from the get go.

Those emails pass through at least one other machine on their way to the user... and unless both ends (ie the PFM and the user) agree to a key pair, it's unecrypted and available to be read by people who have access to those machines as well as any software running on those machines.

There is risk in everything we do (especially online), and we must all judge whether the risks are acceptable. I just think the risks associated with personal finance data (as opposed to banking data) are lower than people think and that I think the utility of these services is worth those risks.

Well I was waiting for this to happen to one of these guys. Who will be next?

At least with my Bank or Credit Union we have some recourse (FDIC, NCUA).

These startup's that have complete financial information and no SAS70 audits or compliance and just as SSL certificate are scary.

Consumers should be made aware of this. Jim maybe you do not rate the vendors on this site but someone should.

@Tommie

You bring up good points...but I'm not sure it's as bad as you think. As unfortunate as this incident was, there have been far worse data breaches at FDIC/NCUA-regulated sites.

And consumer protections would be pretty much the same if someone stole your identify via a mistake at Rudder or Bank of America. For financial gain, a crook would have to get to your actual money or credit, which would typically be held at a regulated financial institution.

Finally, most of the PFMs use aggregation technology from either Yodlee (Mint) or CashEdge (Rudder). Thosw vendors have been very well vetted by large financial institutions going back almost a decade, including, I expect, multiple SAS-70 audits.

That said...trust is ultimately in the eye of the consumer, and yes, they trust their banks and credit unions, more than a web-based startup. So, these kind of dustups provide benefits to the incumbents.

Jim--

Thanks for the analysis and the kind words. We really, seriously appreciate it. The response from our users has been overwhelmingly supportive. We definitely did not expect that.

We do indeed use CashEdge, and you are correct in your understanding of how we get our data. That said, we are going through an intense audit right now that should tell us more about what we can do better. We'll be reporting on the blog with updates as we progress, changes we're making, etc. It's going to be a slow and long process, but we hope to regain users' trust over time.

It's certainly true that banks, for example, experience breaches or potential breaches somewhat frequently. This happened to me just a few weeks ago. I received a new ATM card in the mail, and I didn't even bat an eye. But I think that this new class of PFM apps is different. We're trying new things to add value, but users have a choice as to whether they think that the risk is worth the reward. You have to choose a bank, but you don't have to choose to use an app like Rudder. The trust barrier is set much higher.

We're relatively confident that identity theft won't result from our breach -- that said, we're buying all of our affected users comprehensive identity theft protection.

As you say, the real damage is done in the court of public opinion. It's frightening to think that your data is vulnerable, regardless of whether it is possible to steal your identity, or your money. At Rudder we're going to go above and beyond to make this right. And we hope that we're able to convince people along the way that we are, despite this incident, worthy of their trust.

We're also very much aware of the potential impact on other PFM startups. The higher purpose of helping Americans become more financially healthy matters more than anything.

Thanks again,
Nikunk Somaiya
CEO, Rudder
http://rudderupdate.tumblr.com

I am really concerned by the noise this incident has created, not about the seriousness of the breach and what can be done to prevent it, but the suggestions that only banks and credit unions are the logical provider of PFM services. One of the big reasons the PFM providers have been successful is because the banks and credit unions did not innovative in a decade since account aggregation has been available.

Lets hope we draw the right lessons from this incident and not stifle the entire industry.

Upcoming Events

Recent Research

  • NEW! Mobile Account Opening (MAO): Port online account opening to smartphones to reach the sizable mobile-only segment - Find out more
  • NEW! Mobile Banking Security: The new channel is a boon for improving security, both real and perceived - Find out more
  • NEW! Money 3.0: Payments Go Mobile Apple, Bitcoin, & mobile apps are changing the game. What’s a bank to do? - Find out more
  • NEW! Digital Banking Forecast: Current, future and historical usage: 1994 to 2023 - Find out more

 

   

RSS Subscribe via RSS
RSS Subscribe to Comments



Email:


@NetBanker Twitter Feed



See all @NetBanker tweets