Reg man Darren Pauli hangs with happy hackers-for-hire
Feature Nathaniel Wakelam made US$250,000 last year.
In his second job, finding and reporting bugs to bug bounty programs.
Wakelam’s a 20-year-old high school and university drop-out who has become something of a poster boy for the bug bounty boom, a movement that sees the world’s biggest companies pay guys like him tens of thousands of dollars for reporting vulnerabilities in popular apps, services, and video games.
Wakelam is not unique. Prominent bug bounty hunter Mark Litchfield reckons he has earned half a million US dollars over the last two years.
These bounties are secondary incomes for most bounty hunters. Over a few hours on an ordinary Friday night, Wakelam can be found teasing out remote code execution bugs on some of the world’s most popular consumer platforms and earning himself a quick US$20,000, or around two-thirds of the Australian annual minimum wage, with time left to go out for a drink.
He is one of hundreds of talented hackers around the world who have jumped on the bug bounty boom finding and reporting holes for money in the world’s biggest technology and household companies.
Bounty hunter … Nathaniel Wakelam (Image: Darren Pauli)
Bug bounty programs have become big business because big business sees them as cheap and effective security research.
The likes of General Motors, PayPal, and gaming and media giants are paying millions of dollars to hackers from Australia to Zimbabwe , while other blue chip security slackers debate the return on investment of proper patching.
“It’s been a ride,” Wakelam tells The Register. “We’ve made a lot of money.”
The Netscape Champions
It has been 20 years since Jarrett Neil Ridlinghafer successfully pitched what he and friends would later call a “bug bounty” during a polarised meeting with Netscape executives at the dawn days of the Dot-Com boom.
It was a stroke of genius that tapped into the lifeblood of the Californian open source advocates and also a means of protecting the image of the web as a safe place during its early years.
The idea was striking in its simplicity; ask Netscape’s highly engaged, tech-savvy community, which was already peering under the browser’s hood, to report any security flaws they found.
In return, engineers would send them stuff.
The community would soon go on to not only report vulnerabilities under the Netscape Champions program but help engineers stress test the browser in simulated distributed denial of service attacks.
The Champions program would have never left the launchpad had engineers not universally backed Ridlinghafer’s pitch to tap into Netscape’s hitherto distributed brains trust.
The then-Netscape-employee-number-121 had won over most staff at that meeting. “When I pitched my idea I was nervous,” Ridlinghafer told The Reg says during a phone call from Odessa, Ukraine where he is working on a incubator for next-generation tech. “They didn’t seem to think these guys (the Netscape community) would do a better job finding bugs than our engineers.
I said they already are.”
Rick Schell was Netscape’s vice president of engineering and one of the dissenters at the meeting.
The subsequent years have obscured his recollection as to precisely why he did not immediately leap at the bugs bounty idea, but the former Borland tech says it was not for suspicion of external help.
“I had been at Borland in the late 80s where we had groups of outsiders helping us all the time,” Schell says, now working with a Menlo Park venture capital firm. “Beta programs are about external people helping.
If anything, it (dissent) had to do with paying outsiders bounties in the way it was proposed.”
Schell suggests the bug bounty program may have been perceived as a possible risk to Netscape’s hoped-for rapid release program, by threatening to bog it down in bug triage. “We were moving very fast, and any tax on the organisation that we weren’t set up for was problematic.”
Schell was helped onto the bugs bounty ship by the late Netscape marketing vice president Mike Homer and chief executive Jim Barksdale. Some time after the program would be seen universally as a benefit to security and Netscape’s reputation, notably at a time when Intel made itself a pariah for sweeping the floating point bug under the rug.
Ridlinghafer, who two years earlier was cleaning pools in Arizona for US$9 a pop, found himself in the operations wing working with company engineers. He says the incoming bugs were amazing, and a vibrant community was eager to exchange vulnerabilities for T-shirts and schwag from the Netscape shop.
They were happy too with the boozy parties Netscape would throw for its Champions every few months in the tech mecca.
Sometime around 1996 another meeting was reconvened between executives.
It was decided Netscape vulnerability hunters would get cash alongside T-shirts under what became the world’s first bug bounty as it is currently known.
The bounties ranged up to about $1,000 for the most critical holes.
A tale of two very naughty boys
When Wakelam says “we’ve made a lot of money”, he’s referring to his hacker mate Shubham Shah, 19, whom he met as a 14-year-old on internet relay chat.
Pals … Nathan Wakelam and Shubham Shah (Image: Darren Pauli)
While separated by state boundaries – Wakelam lived in Melbourne and Shah 100km away in Sydney – the pair would follow similar trajectories at school and into their young work lives as security consultants by day and bug bounty hunters by night.
Shah aced his final school exams, ranking fourth in his year and quickly publishing a school exam preparation app that was would be soon touted by the State Government launched by State education minister Adrian Piccoli.
Shah can also walk on the wild side, because his curiosity was only matched by his lack of concern for rules. “I was almost expelled,” Shah says. “They, like, legit despised me for ages.”
As a 13-year-old in his first year of high school, he cloned and sold bus tickets for $10 less than their $25 face value.
About 50 kids bought the fakes each month adding up to hundreds of dollars.
That same year he would break into a teacher’s administration account and change student home addresses to 123 Pirate Street, just for a laugh.
He was busted in CCTV footage and suspended.
In his third year of high school he spun up an authenticated web proxy and granted 60 students across two schools access so they could bypass the Bluecoat firewalls at the Department of Education in the Australian state of New South Wales had put in place. Some kid snitched and he was suspended again.
His final suspension for flipping bulk quantities of iPhones at school came with the threat of expulsion. “It shows how much of a fuck-up you can be,” Shah says.
Wakelam had a similar story; from early years of high school he was getting busted at school and was using his knowledge of tech to legitimately make thousands of dollars without – and he would say in spite of – the help of teachers.
“I didn’t like school and I didn’t fit in,” Wakelam says over beers in a dive bar in Fitzroy, Melbourne. “Not because I was a nerd or anything, you know, I used to enjoy different things – like I’d still go to parties and get f*cked up, but I read a lot and maintained my education.
I just didn’t buy into the idea of standardised learning and didn’t think the skills (from school) would be the things that would help me with my future.”
The son of a nurseryman and daycare nurse, Wakelam’s escape from high school came at the end of his third year. His parent supported that decision, which he now credits as a far-sighted decision.
Bug bounties were a perfect fit for the drop-out both as a cash cow and a means to fill holes in a CV missing what he calls “checkbox bullshit” like a university degrees. Wakelam’s resume now is largely a list of the bugs he has found which he expands on in interviews, telling employers about his methodologies and the relevance of the vulnerabilities to the affected businesses.
Ridlinghafer’s money machine
In September 2013 Wakelam was working the phones in a call centre for about A$500 a week.
It didn’t last.
Two months later he would find and submit a bug under Yahoo!’s then new bounty program and clear AU$15,000 for about 20 hours work. Wakelam quit the call centre the next day and left before lunch.
In the first year he cleared US$75,000 in bug bounty payments for working 20 to 40 hours a week.
In 2015 he scored US$250,000, his second year as a serious bug hunter. He also landed 500,000 United Airlines flight miles for a single bug, enough for a few first class flights to the US.
Shah’s first employer suffered a similar fate.
The then burger-flipper threw in his apron and a wage of A$6.50 an hour pocket to chase bug bounties. “I was sick of it, so I decided to go into bug bounties and I actually made like $500 or $1,000 a time for submitting really simple bugs,” he says.
In 2015 Shah met Wakelam, who shared some bounty-scoring tips. Shah then cleared US$50,000 in his first two months as a full-time hunter, one bug alone bringing him US$30,000 after 30 hours of hacking.
Some readers might at this point wonder if two admitted miscreants might be making up their hauls.
The Register has seen balance sheets and invoices which match Wakelam’s and Shah’s claims.
“I made (US)$17,500 from one bug alone,” Shah told The Register during the Wellington Kiwicon hacker confab in December. “I’ve got another 25 or something owing.” An invoice seen by this reporter shows another dozen ready to be paid, adding up to nearly US$37,000. One critical bug is worth more than US$13,000. “I’ll hack for about 40 hours a month and pick one week where I make most of my money, and then I’ll have a break.”
The duo prefer the money multipliers of private invite-only bug programs of the kind hosted through Hacker One. Researchers cannot apply to play in these higher-value bounties and can only hope HackerOne handpicks them for the job. Selection criteria are secret but as Wakelam ranks in the service’s top 10 most accurate and prolific bug submitters he’s often called on to help. Shah bounces around further down the top 100 list and still gets taps for the lucrative bounties.
It is worth the wait for the call to come from a program like HackerOne, as Wakelam reckons the same bug which attracts $25 on a public bug program can land $7000 in a private bounty. “You’re typically adding two zeros to the reward,” he says. Researchers throwing in a professional report explaining why the bugs are relevant to the business and how it could result in serious loss could earn even more.
The pair’s favourite money-spinning bug bounty program is run by a company seemingly built of blank cheques.
The organisation is a household name to virtually everyone in information technology but did not want to be named for this story. Wakelam and Shah say the company’s bug bounty program is a model on which others should be built with its wide-reaching target scopes and agreeable security team which seeks to reward rather than reprimand hackers who report bugs found outside of predefined boundaries.
They hack for public programs too, for job variety as well as cash. Uber, Yahoo!, and United Airlines are but a few of the companies they’ve considered.
What are you waiting for?
“I would recommend it to anyone,” says Mark Litchfield, a security industry veteran who runs the in-house bounty program Bug Bounty HQ. “It is a fantastic entry point into security.” In a phone call from his home in Las Vegas, Nevada, the former NCC co-founder says he has made about US$500,000 in bug bounties since February 2014.
The leading HackerOne point scorer reckons he has made some US$350,000 on that program alone, with another US$120,000 from Paypal and US$15,000 each from Google and Bug Crowd. “It clearly pays all my bills,” he says.
The money seems easy.
In December, the British ex-pat set a goal “over a couple of Heinekens” to make US$50,000 in a month and finished on New Year’s Day with US$47,000 from Paypal, Yahoo!, and BugCrowd. He was paid US$9,000 four days later, but only because staff responsible for making payments took a holiday.
Litchfield has been in the security game for decades. While in London at the turn of the century he and brother David, now of Google Project Zero, sold their company Cerberus Information Security which they started in 1993 to the burgeoning high profile consultancy @stake, later bought by Symantec.
The following year the brothers would set up NGS Software and run the company until it was bought out in 2008 by NCC.
From stripping binaries to the Vegas strip … Mark Litchfield with his wife
He is now thoroughly invested in bug bounties. Most of the cash he scores in the programs is tipped into Bug Bounty HQ, a platform designed as a just-add-water framework for businesses wanting roll out internal bounty programs.
As he puts it he is “a bug hunter who is using bug bounty money to build a bug bounty platform for the bug hunting community”.
This reporter has heard of hackers who have used the cash to set up instant investments. One US bug seeker has purchased a house just from the winnings.
Another twenty-something has tipped $100,000 into a share portfolio using only bounty money.
Hackers in India, Egypt, and Africa now represent enjoy the lion’s share of Google’s bug bounty payments and have lifted themselves and their families out of poverty, industry sources reckon; in one account Mumbai parents had dreamed the proverbial dream of their son as a lawyer only to find their financial liberation would come from his successes as a bug hunter.
While Litchfield and others largely chase bug bounty coin, most vulnerability disclosures are handed over for free, with researchers receiving only a word of thanks, a reputation boost, or a tee-shirt from companies nursing the old Netscape mindset.
One Brazil-based researcher known by his hacker handle “Brute” submitted thousands of mostly simple but dangerous cross-site scripting bugs through the XSSposed bug platform over the course of last year. While he has since stopped, his efforts scored him a job at consultancy Sucuri.
Brute says the effort helped him learn about the security field while he ran a university tech department, and had benefited small companies receiving the reports which otherwise would lack the resources to identify the vulnerabilities.
“I started to submit XSS to XSSposed to learn about security,” Brute says. “I only used a browser and simple bash scripts to find random targets.” That method made him the most prolific bug reporter on the platform – which housed hundreds of active benevolent researchers.
Bang for buck or bounty bubble?
It’s been five years since Google launched what would become one of the most lucrative public bug bounties.
Two weeks ago the technology deity announced it had paid out US$6m in bounties to researchers from across the world since the Vulnerability Rewards Program began. with an average of US$1.2 million paid out a year.
That stepped up to more than $2m over 2015.
The largest payment was US$37,500.
Facebook last week announced it has paid out US$4.2m in bugs since 2011, or some US$860,000 a year. Microsoft has handed over US$500,000 since September 2014, including top bounties worth US$100,000. Since their inception in 2012 managed bounty programs BugCrowd has paid out about US$1.4m while HackerOne has handed out some US$6m or a whopping US$1.4m a year.
The money fountain may seem to sceptics like signs of an excited tech sector where the big players are more interested in showboating and public relations than extracting the best bang for bug buck: Certainly Google’s $7337 (“l33t”) bug bounty prize seems like a figure more rooted in cultivating hacker credibility than on some boring return on investment model.
Roll your own or buy ready made?
Yet it is difficult to find bug bounty critics.
This reporter asked the companies paying out the most expensive bugs for comment including Google, Yahoo!, Facebook, and Microsoft, along with other lucrative private bounties by major consumer IT sectors, but all either declined or did not respond.
Security analyst James Turner says businesses need to set bounties according to their individual risk appetite.
Turner rates bug bounties an important security tool and a necessity for businesses running high-value internet-facing assets, like money-making apps and websites. “The value of a bug bounty is directly proportional to the importance of the asset,” Turner says. “But businesses will need to determine that value in their own right.”
There are plenty of businesses in the Antipodes and elsewhere that fit Turner’s bug bounty bill; airlines, technology and services, and telecommunications are but a few among those that have millions of customers who would potentially leave for rival firms if a major breach or outage occurred in the absence of security testing. “These businesses need to make sure their interfaces with their customers are resilient and reliable,” he says, quoting BugCrowd founder Casey Ellis that “nothing sobers up an engineer like realising a 14-year-old hacked you”.
Ellis brewed Bug Crowd in sunny Sydney but took the idea to the startup capital of San Francisco for a successful run at venture capital funding.
The organisation has risen quickly to dominate the managed bug bounty space alongside bounty service HackerOne.
For all its success and for all the bounty cash, however, Ellis says having the many eyes of white hat hackers looking for holes is critical to the security online business. “Honestly, we are screwed if we don’t do this,” Ellis says referring to the role of bounties in filling the shortage of security penetration testing talent. “Bug bounties are more than just wanting to run a hacker program to look cool.”
BugCrowd runs programs where hackers compete in public, private, short and longer term bounties vying to be the first to report the most valuable bugs. He says these bounties can attract return-on-investment up to three to five times the total payment in terms of both the number and severity of bugs reported. “Security has gone from a thing that we had to bang our heads on the wall to get people to care about to something where you can talk about it at dinner and not necessarily be the geek,” Ellis says. “And that puts pressure on the execs to invest in it.”
Katie Moussouris says bounty programs have become well-oiled with private bounties gaining better signal to noise ratios for more severe bugs and higher quality reports. “There’s been a boom of bug bounties between 2010 and 2013,” says the HackerOne chief policy officer. “High profiles companies are doing them.” Moussouris has been in the security industry since about 1997 hopping through @stake and Symantec to climb the Microsoft hacking rungs where she launched its first paid bug bounty program, and wrote its vulnerability disclosure policy.
HackerOne … Katie Moussouris (Image: Darren Pauli)
She says organisations should gain an understanding of the worth of vulnerabilities according to their own risk appetites while market rates exist.
They should also build out security wings noting that are bounties do not operate in isolation and require businesses have enough resources and skills to fix the vulnerabilities that come in should return on investment be achieved.
Forming a bounty scheme that resonates with the hacker mindset will further increase submissions, and therefore profit.
This means easy registration processes for bug hunters, straightforward disclosure agreements, and the ability for hackers to retain their intellectual property. “Mature organisations aren’t afraid of bug bounties and of vulnerability disclosure after a patch has been applied,” she says. “It’s a powerful way to demonstrate security.”
Regrets … Jarrett Ridlinghafer
Next month hackers will compete in Pwn2Own, in Vancouver, Canada, showcasing zero-day exploits against the world’s most popular consumer software.
Google and Microsoft are shelling out to be one of a list of big sponsors for an event in which their own browsers are up on the hacking target list.
Bug hunters there will be surely courted by zero-day exploit brokers like Zerodium which offers US$1 million iOS bounties.
The bug bounty boom is one of the biggest changes to the information security business in recent years.
It has been an effective advertisement to penetration testing, both elevating the benefit of having corporate assets hacked to executives while also in attracting new hackers to the field.
Those in the bug game and more broadly across the security sector reckon the impressive payouts are sustainable and will continue as bugs keep falling out of code.
So too does Ridlinghafer. “I mean, what a frickin’ idiot,” Ridlinghafer says of his decision to exit the then infant bug bounty world. “I wish I’d stayed put, maybe I’d be a billionaire by now.”
Security professionals have told this reporter they intend to jump on the bug bounty bandwagon to supplement their paychecks.
It is unsurprising; corporate hackers and penetration testers can access a vast income stream by merely applying their existing skills to after-hours work. “The second they realise how much money we are making, it’s going to get crowded,” hacker Wakelam says. “If I’m making three times the amount of money guys are making with 15 years more experience than I am, there’s a problem there – and they need to adapt.” ®