Screening | Step 4: Calibrate

Welcome to the final part of a 4-part series on screening senior engineers.

Step 1. Before You Do Anything
Step 2. Designing Impactful Challenges
Step 2a. The Challenge Checklist
Step 3. Setting Expectations, Warming Your Candidates 
Step 4. Calibrating After the Screening (This piece)


In steps 1, step 2 and step 3, we strategized on the best types of challenges to tease the engineers you need. But what good is a code challenge screening if you’re testing different candidates with different questions? Consistency can make the data you’re collecting meaningful. HackerRank’s head of challenge curation Dr. Heraldo Memelli finds that all too often, companies change the question after it’s already gone out to some candidates, ruining your data set. Periodically review which questions have been directly correlated with successful candidates.

After the initial screening, it’s crucial to mirror your code challenges with the rest of your interviewing rounds. It’s logical — folks who clear the automated code challenges online are more likely to perform well in-person if the questions are consistent. Automated code challenges are the best tool to predict who will perform well in the in-person interview. So, target your questions to mirror the style of questions asked in the actual interview as much as possible.

Choose questions that you’ve actually asked in on-site interviews so that you can see how candidates might react. This can help you frame and word the question appropriately as well. Of course, this means you’ll have to retire those questions from your in-person interviews. Aim to keep your code challenge banks fresh. Trying out new code challenges on your own time is a great way to not only benchmark your own team against newcomers but also keep your engineering teams’ skills sharp. Investing the time in revamping and optimizing your set of interview code challenges is minuscule compared to the cost of hiring wrong engineers. 

Tieing it All Together

Asking senior engineers to revisit fundamentals in an interview sounds outrageous. And it is if you simply send a cold email without any proper preparation. But automated code challenges are the most objective and successful predictors of hiring we have to filter through candidates at scale.

Gauging not only their intelligence but also how much they value fundamentals through algorithm and data structure questions are strong instruments to find the best engineering talent. If you set expectations and are mindful of senior engineers’ complicated hubris, you’ll have a stronger pool of senior engineers. And you’ll be set up with a replicable system to build and scale your engineering team. A set of well-designed mix of questions that test fundamentals, knowledge and depth of thinking, is a good way to weed out candidates who don’t have the basic skills you need to build revolutionary software.


Please subscribe to our blog to get a quick note when we occasionally write thoughtful, long form posts.


Welcome to the final part of a 4-part series on screening senior engineers.

Step 1. Before You Do Anything
Step 2. Designing Impactful Challenges
Step 2a. The Challenge Checklist
Step 3. Setting Expectations, Warming Your Candidates
Step 4. Calibrating After the Screening  (This piece)

 

Screening | Step 3: Set Expectations

Welcome to the 3rd part of a 4-part series on screening senior engineers.

Step 1. Before You Do Anything
Step 2. Designing Impactful Challenges 
2a. The Challenge Checklist
Step 3. Setting Expectations, Warming Your Candidates (This piece)
Step 4. Calibrating After the Screening


In step 1 and step 2, we figured out what you want and designed the challenges to help surface the right engineers tailored to your needs. Now you can focus on the most important part of executing your new screening process: practicing tactful communication to engage senior engineers.

Senior engineers love a good challenge, but not if it’s time-consuming and futile. The truth is, accomplished engineers will scoff at a cold email prompting them to spend 1-2 hours solving a fundamental code challenge for a company they may never hear from again. It’s understandable why should they? Chances are they’ve already earned their stripes putting in late hours building software that’s reshaping our world one way or another. When designed and delivered haphazardly, automated code challenges will be perceived as insulting.

It’s critical to infuse an element of empathy in the delivery and design of your challenges. Cold emails with no explanation as to what they’ll be asked and why, are a surefire way to lose senior engineers. Those who make an attempt to make their candidates feel valued while scaling their screening systems will be the winners of today’s competitive war for talented engineers.

There are a couple of things you can do to mitigate this:

1. Be very clear about what you intend to ask and what you expect. Give them easy-to-digest prep material.

This is an important step in ensuring that you’re not rejecting great people just because they lack CS fundamental knowledge. Most senior engineers are not going to have CS fundamental knowledge right off the bat. First of all, nearly 60% of working software engineers don’t even have a proper CS background in one study. Second of all, even if they did, it was likely too long ago to remember anything that’s applicable in the real world. Expecting too much with no warning results in this: Screen Shot 2016-01-29 at 11.52.36 AM

Before sending testing engineers on fundamentals, explain that we’re not trying to quiz them on random things they don’t even need to use on the job. Instead, fundamental binary tree questions help gauge problem solving and critical thinking skills, as we explained in Step 0. Although many great candidates get rejected because they failed to adequately prepare, it’s actually not that hard for smart developers to learn or re-learn the fundamentals. It’s part of why companies are fine with requiring them. And, as Kickstart CEO Soham Mehta said, the best senior engineers want to brush up on their fundamentals anyway. 

2. Calculate the amount of time you need to take the challenges (usually between 45 and 90 minutes) and give them more than enough time to actually complete it. Communicate this to your candidates and explain that it shouldn’t take you more than an hour, but you have X hours to take it. It helps to simply articulate that you recognize that they’re extremely busy. Their time is valuable, so there’s no intense time limit to submit.

3. Explain why you’re even asking them to do this. For instance, if the code challenge replaces the resume or initial phone screening, let them know. Explain: “We ask you to do this code challenge so you don’t have to spend more time on the phone with multiple engineers.”  If you’re sending code challenges with no explanation, reasoning or preparation, your screening tool is an arbitrary knowledge test. You’re bound to anger senior engineers.

Bonus Tip: We did a quick analysis of dozens of code challenges and found a consistently lower completion rate when using the word “test.” Avoid this term like the plague. Instead, always use “challenges.”

So after you’ve implemented great code challenges, what can you do to make sure they’re successfully producing the best senior engineering candidates?

Read onto Step 4. Calibrating After the Screening


If you like what you see, please subscribe to our blog to get a quick note when we occasionally write thoughtful, long form posts.


Welcome to the 3rd part of a 4-part series on screening senior engineers.

Step 1. Before You Do Anything
Step 2. Designing Impactful Challenges
Step 2a. The Challenge Checklist
Step 3. Setting Expectations, Warming Your Candidates (This piece)
Step 4. Calibrating After the Screening

 

 

 

Girls Who Code: Airbnb Revenue Analyst to Software Engineer in 1 Year

Many of you have it. That dormant bug that’s hungry for hacking. Even if you don’t professionally pursue it, the spirit of engineering is often ingrained deep in the crevices of our brains.  Common triggers include things that are abysmally clunky, slow and annoying. You’re perpetually compelled to hack together better solutions for the daily details of life.

The world of finance is filled with triggers for hackers. Massive volumes of financial data are often too heavy for the shoulders of Excel VBA or Pivot Tables. It could mean waiting hours just to load a spreadsheet.

And it’s never too late to awaken the inner hacker.

This was the case for Kari Tarr. She was working as a revenue analyst for Airbnb when her itch to hack transformed into urgency to become an engineer. It’s understandable. If you think about it, she was working on analyzing revenue for one of the most high-growth startups of all time. Airbnb earned about $250 million in 2013, and tripled to $850 million this year. Her work involved analyzing massive spreadsheets, a daily arduous and time-consuming battle.

After being exposed to the true power of programming, Tarr made it her mission to achieve what many inner hackers often dream about.  She switched from the finance team to the engineering team in 1 year.

How did she achieve this herculean transformation? We sat down with Tarr to get inside her head on how she–and you–can feasibly change an entire career trajectory by indulging the inner hacker.

So, first off, what was your first entry into programming?

You know the scene in the movie Clueless where she has that dream closet? And she can see all of her outfits on the computer screen? I really wanted that and my dad was a software engineer at the time, so I asked him to make it for me. He said he wouldn’t do it for me but he’d help me learn to do it myself. So, I got into very simple front-end HTML. Again, this is in the 90s, so simple HTML sites were all the rage.

That’s awesome. The Clueless Closet is apparently real by the way. So, if you’ve always been interested in programming, why didn’t you study CS?

As a kid, I was a geek but definitely not thought of myself as a programmer. Especially because my dad was a very typical engineer—very quiet and reserved. And I was very outgoing. I made these associations in my mind that I wasn’t supposed to be an engineer.

I thought about majoring in CS, but I took an AP Calculus class that absolutely destroyed me. I was around people who were very nerdy and mostly guys. It felt intimidating – so I went the business school route instead.

So, practically speaking, how did you start coding?

I think if you’re an engineer at heart, you’ll always gravitate toward it and eventually find your way there. I was always looking for ways to be more technical in my roles. I had a habit of inventing new job descriptions that let me do more and more technical things, Excel macros and VBA being the gateway then teaching myself SQL.

At Airbnb, I became friends with the engineers. Being a startup, we’re dealing with tremendous growth and often have to go way outside of our roles and hack together solutions quickly, building the road as you’re driving on it, so to speak. To get one of my projects done and out the door, I had to learn a little bit of Linux, and I was immediately hooked. I think I kind of felt drunk with power and became obsessed with learning as much as I could. I became a part-time self-taught person at that point.

What kind of project was it?

I was dealing with lots and lots of data, and I started learning how to edit files through the command line instead of opening an Excel spreadsheet.

I was basically doing things like changing files from tab to comma delimited, so I could import them into another program. It was taking me seconds to do something that took hours in Excel. I was really excited and  wanted to see what else I could do.

But how did you convince Airbnb engineers to give you access to the code base?

Two things: first, it was looking for the right opportunities. If something needed to be automated, or there was a weird bug that the engineering team didn’t have the bandwidth for, I’d dig into it. I sought out opportunities that would force me to get experienced with our code base.

Second, I tried to make it really hard for them to say no. For example, I wanted to build a tool for my team so I reverse engineered some existing code and got something up and running. Granted, it was pretty terrible code as I had no experience in writing “good” code, but I got it working.I had a prototype and I showed it to one of the engineering managers and said “hey, I want my team to be able to use this. It’s already working. I did all the work. Can you help me ship it?” It’s hard to say no to that.

Plus, the team at Airbnb is incredibly supportive. I had some awesome mentors who also were wonderful at helping me navigate things and get me unstuck.

And how did the engineering manager react when you first pitched the idea of switching to their department?

At first, they were against the idea. It’s risky to take someone on who doesn’t have experience. They were worried that they’d end up spending too much getting me up to speed. So, I figured okay I’ll keep learning on the side and formally apply in the next 3-5 years when I’m ready. I’m not sure why I wasn’t more discouraged. I guess I kind of expected it. I know the caliber of people who work here and I felt like I had a long way to go before I’d be able to compete.

But then I got some good advice from a friend who told me if I really wanted to learn this stuff, I needed to be living and breathing it every day and I should just jump off into the deep end. So the scariest thing I could think of was to ask them to give me a chance to prove myself.

The best managers help you figure out where you want to go and give you the steps to get there. Once I adamantly said “This is where I want to go” they set up a path for me to get there.

Now that you’ve successfully become a software engineer in 1 year, in hindsight, what are the best tools for self-learning?

I think the best thing to do would be just getting your hands dirty as quickly as possible. I started by reading books, but it left me with so many questions. You have to see things in action early and often.

The thing I really struggled with the most was wanting to deeply understand everything the first time I dug into it. With programming, it doesn’t always work like that. You might have to see or hear it in 6 different contexts before it clicks. That’s why it’s helpful to do a combination of:

  • Reading lots of books
  • Getting a mentor
  • Going to meetups, surround yourself with people doing what you’re doing
  • Working through coding problems

Eventually if you keep at it, things start coming together. One thing that particularly helped me get over that hurdle was pair coding with friends. Gayle Laakmann McDowell is a good friend of mine who helped me through a lot of the coding interview problems. She and I would hop on HackerRank and work through algorithms. It was a beautiful way to learn because she could literally see where I was having trouble. If she could tell I wasn’t solid on something, she’d make up smaller sub problems. Eventually, 36 mini problems later, I’d solved some matrix iterative algorithm. It was slow, but it got me from point A to point B and it was really encouraging.

Now I’ve been working as a full time software engineer for about 6 months I’ll admit, the learning curves never stop. Once you get over one and think you’re in the clear, there’s another ahead that makes you feel like a beginner all over again. If you’re a life-long learner, you’ll love it, because you’ve really got no choice. It’s been a roller coaster but I’ve shipped some awesome projects and I wake up every day really excited to get to build stuff for a living.  

____________________________________________________________

 

Are you ready to become a coder? The hardest part is starting. Join the 30 Days of Coding challenge for coding newbies. 

 

 

Why is Computing History Vanishing?

When future generations of computer scientists look back at the advancements in their field between 1980-2015, they’ll turn blank pages.

Historian Martin Campbell-Kelly points out that “up to the late 1970s, software history was almost exclusively technical.” Since then, traces of the technical history of computing have been vanishing. It’s a very curious case. Relative to its ancestors, math and physics, computer science is still an infant at just 75-years old. Some of the biggest breakthroughs in software technologies happened in the last couple decades, yet historians have shied away from capturing the history of computing through a technical lense.

Think about it: When was the last time you read a detailed technical explanation of a breakthrough that takes you inside the mind of the inventor? Computer science has grown exponentially in the past several decades, but recent critical source codes have gone untouched. Kelly depicts the evolution of software literature below, based on titles he found most useful since 1967. You can see that as the years go by, the emphasis moves away from pure technology to the application of technology.

Screen Shot 2015-11-24 at 10.22.24 PM

Elsewhere, museum board members of the National Cryptologic Museum have been known to criticize historians’ efforts of adequately chronicling the National Security Agency’s work on cryptography. Look at the “lack of historical context given to the recent revelations by Edward Snowden of NSA activities. Historians could be providing useful context to this acrimonious debate, but thus far we have not,” says Paul E. Ceruzzi of the Smithsonian Institution. Unlike onlookers, historians are likely numb to such controversy. After all, it’s not too different from the events in WWII’s Bletchley Park when Alan Turing intercepted communication from the Germans.

What carries even more weight is the great living legend Donald Knuth’s reaction to Kelly’s paper. “I finished reading but only with great difficulty because the tears had made my glasses wet,” he says. Knuth has noticed the trend, but believes it’s horrifying that historians are in favor of prioritizing business history over technical history.  

Then, last year, he did something he hasn’t done in years. Knuth momentarily stepped away from Volume 4 of his epic series The Art of Computer Programming, poked his head out of his hermit shell, and devoted last year’s lecture at Stanford to: Let’s Not Dumb Down Computer Science History.  

Knuth sparks a fascinating debate–one worthy of further exploration. Why are historians overlooking the technicalities of today’s breakthroughs in computer science? And how will this trend impact future generations of computer scientists?

Tracing the Missing Pieces

Since the invention of computers, historians used to be knee-deep in the technical trenches of computing. There’s plenty of analytical literature on the likes of the ENIAC, Mark I and early IBM computers. But come the pivotal 80’s—when the personal computer started proliferating in homes—historians shifted their focus onto software’s broader economic impact. They’re covering things like funding (here) and business models (here). Shelves are filled to the brim with books on how tech giants and unicorns are revolutionizing the world.

But what about its technologies? Have historians looked inside the black boxes of recent breakthroughs, like:

  • [1993] R programming language, which statisticians and data scientists depend on to create reproducible, high-quality analysis
  • [2001] BitTorrent, the peer-to-peer file sharing protocol that mandates about half of all web traffic
  • [2004] MapReduce, which has been invaluable for data processing

Trained historians have yet to place many of these revolutionary inventions under a historical microscope. No one is contextualizing how these advancements came to be and why they matter. So, what happened? The answer is a prism with many faces.

As Knuth notes in his talk, there’s little incentive to study history of computing for scientists. It’s completely respectable to write a historical dissertation in biology, mathematics or physics. But it’s just not the case for computer science. In fact, history departments within computer science departments are rare–if at all in existence— in America. At best, it might be masked under “other” specialty for PhD candidates:

Screen Shot 2015-11-28 at 12.04.38 PM

So, who does that leave us?

ACM published this infographic depicting the state of computer science history today. You can see it’s mostly a secondary interest for a subfield of history or science:

Screen Shot 2015-11-29 at 11.50.25 AM

Historians of science are usually specialists within a broader history department, under the humanities umbrella. So, it follows, the accounts from non-technical historians will always be less technical than that of programmers. The onus is also on computer scientists to write the technical history that lives up to the caliber of Knuth.

Even if you decide to embark on computing history, historians will cast a wider net in reaching audiences by writing about software’s impact on business, society and economics. Naturally, technical articles are only valuable to a tiny slither of scientists, yielding limited financial support.

“When I write a heavily technical article, I am conscious of its narrow scope. but nonetheless it is a permanent brick in the wall of history. when I write a broader book or article, I am aware that it will have a more ethereal value but it’s contributing to shaping our field,” Kelly writes in response to Knuth.

When Kelly wrote technically-heavy pieces, filled with jargon and acronyms, his esteemed colleagues’ told him his view was too narrow-minded. For instance, Kelly wrote about the EDSAC in the 1950s. But critics said he neglected to include its fascinating uses:

  • Generated the world’s highest known prime number
  • Created a stepping stone to the discovery of DNA by Watson and Crick
  • Reduction of radio telescope data, a crucial process in radio astronomy

Studying the byproduct of computing is undeniably valuable. But when it comes to the technical discoveries that lead to these technologies, we have only darkness.

Furthermore, computer science historians’ job prospects are severely limitedit’s either academia or museum work. PhD in other computer science specialties, on the other hand, have high-paying options as researchers in R&D labs of Google, Facebook, Microsoft, etc. You’d be hard-pressed to find someone who built their career on computer science history.

Alternatively, the eclipse of technical history could be a byproduct of the secrecy of government agencies. In the earlier NSA example, for instance, historians have the additional hurdle of declassifying projects. This has been an ongoing larger hurdle since the days of top-secret missions in WWII, when bomb simulations generated many of the major developments in computer science.

Another reason for the lack of technical history of software, could be the volatility of the discipline. It’s hard for historians make definitive claims without arriving at false conclusions when the field changes so fast. Just look at this piece on what’s worked in computer science since 1999.

 

Screen Shot 2015-11-28 at 7.39.14 PM

Concepts that were considered impractical in 1999 are unambiguously essential today. It’s risky for historians to make definitive claims when the field can shift in just a 10-year window. It’s like figuring out where to jump on a moving train.

Finally, the sheer exponential rate of growth doesn’t help either. The train is not only getting faster but also longer. You probably know the widely cited projection from the Bureau of Labor Statistics, which says that computer science is the fastest-growing professional sector for the decade 2006-2016. The percentage increases for network systems analysts, engineers and analysts are 53%, 45% and 29%, while other sciences (like biological, electrical and mechanical engineers) hover around 10%.

To top it off, look at the growth in the total number of open source projects between 1993 and 2007. We’re amidst a paradigm shift in which much of today’s pivotal software is free and open. This research, by Amit Deshpande and Dirk Riehle of SAP Research, verifies that open source software is growing at an exponential rate. Michael Mahoney puts it best when he says: “We pace at the edge, pondering where to cut in.”

Screen Shot 2015-11-28 at 9.15.15 PM

Donald Knuth: This is a ‘Wake Up Call’

But this shift toward a more open, less patented software world is all the more reason for this to be a wake up call for computer scientists and historians alike. As source code becomes more open, we unlock barriers to history.

Knuth sets the example with the mind of a computer scientist and the zeal of a historian. As he entered the field in the 1950s, his detailed history on assemblers and compilers set the bar high. Still today, his Art of Computer Programming is critically acclaimed as the best literature for understanding data structures and algorithms. He practiced what few grasp today: Without technical history, we can never truly understand why things are the way they are.

We need in-depth history to learn how other scientists discovered new ideas. Reading source code is something most legendary programmers do to become a better programmer. As a kid, Bill Gates infamously dumpster dove to find the private source code for TOPS-10 operating system. Being able to see inside a programmer’s mind as he unraveled a complex knot can help teach us how to solve our own problems. It’s how any new breakthrough materializes fast. When Brendan Eich, for instance, set out to create Javascript in 10 days, he needed a strong foundational knowledge of existing languages to built upon. He pulled structs from  C language, patterns from SmallTalk and the symmetry between data and code offered by LISP.

Even more importantly, history uncovers valuable lessons from failures. If all we have is a highlight reel, future generations might arrive to false conclusions about the expectation of success. They won’t be able to see the false starts that lead up to the “aha!” moment.

Likewise, when William Shockley, John Bardeen and Walter Brattain attempted to execute on the theory for a solid-state replacement for a vacuum tube, they went through months of trial and error. There should have been a change in current when placing a strong electrical field next to a semiconductor slab, but it didn’t happen. There was a shield that deemed electrons immobile. They tried several techniques, like shining light, drops of water and specks of wax to charge electricity. Eventually, they successfully amplified the current, and introduced transistors to the world. But learning from the team’s initial failures can teach us more about what exactly transistors are and what they aren’t.

We’ve only just started learning what’s possible in the computer revolution. The sooner we document and analyze these formidable years, the brighter our future will be. The acclaimed computer scientists like Shockley, Brendan Eich and Donald Knuth should be as well known as mathematicians Albert Einstein, Isaac Newton and Rene Descartes. This is not to say that historians’ current efforts in contextualizing the impact of computing has been wasted. Undoubtedly, the new field is infiltrating every industry, and this analysis is important. But, for the sake of future computer scientists, we need both breadth and depth in computing history. Computer scientists and trained historians must work together to leave a holistic imprint of today’s pivotal advancements to truly fill the pages of computer science history for future generations of hackers.

 

Have you noticed that there aren’t as many strong technical historical analysis of computing and computer science? How can we fill this void?

 

 

 

Tech’s Loophole in ‘Years of Experience’

This article originally appeared in Forbes.


Time is arbitrary–a relative school of thought. Ancient Roman civilizations looked at sun and moon cycles and decided that just about 365 days would make up one year. Newton said time is absolute, after which Einstein theorized its relativity. The existence of multiple scientific notions of time itself is proof that time is not real.

The quantification of time was conventionalized centuries ago. Time is not only imperfect but also permutable. The Daylight Savings Act of 1976 exemplifies this perfectly. Benjamin Franklin declared that we should simply move the first hour of light to the evening during the fall to boost productivity. With the mere pass of a law, Americans altered their perception of time forever.

Time and space are modes by which we think and not conditions in which we live.”

— Albert Einstein.

Time is not real and yet the number of years in a particular job or skill often determines your job opportunity and–potentially–your entire career trajectory. When you really scrutinize why organizations filter by years of experience in job descriptions and what this reveals, it’s hard to believe that it’s still a primary factor for hiring today in tech.

Given the prolific concern of a skill gap, the imperfect illusion of time spent on a skill shouldn’t be a top factor in hiring in the tech sector. When it comes to solid experience, it’s not the years that predict great performance but, as Vinod Khosla advised at TechCrunch Disrupt, “it’s the rate of learning.”

The Loophole is Real

Speculation of agism is well-documented in the technology sector of Silicon Valley. Some point to anecdotal “frat bro culture” of startups (here), while others point to the pattern of successful young co-founders (here). While one-off instances of age discrimination lawsuits are undoubtedly overblown in the media (like that of Google and Twitter), the average median age in tech is a stat that’s hard to ignore. PayScale did a study to find the median age of tech workers, and found that just 6 of the 32 companies it looked at had a median age greater than 35 years old.

pic_1

Informationweek did a survey of tech workers and found that a 70% said they’ve witnessed age discrimination. This has even caught the attention of the EEOC, the government agency in charge of discrimination laws:

“Some of our offices have made it a priority in  looking at age discrimination in the tech industry,” EEOC senior counsel Cathy Ventrell-Monsees.

If hiring based on age is illegal, why is there such a homogenous culture when it comes to age range? The loophole is in the years of experience. Regardless of what age you are, if you don’t meet the requirement, your resume might never reach the hands of hiring managers in tech. It’s why lawyers, for instance, have pointed out that requiring people with “Native Digital” experience is teetering on the edge of age discrimination. It implies that you have to have been born in the digital age, Fortune reports.

pic_2

It’s a flawed process. Hiring managers typically write down a ballpark range of minimum, let’s say, 3-5 years of experience in a particular skill. Usually, recruiters run with this range and subscribe to the idea that since Scott has 20 years in COBOL, he probably doesn’t know about cutting-edge tech, like Swift or Block Chain. But it goes both ways. If John spent 5 years working on Java, he’s considered more qualified than Jill, who only spent 1 year learning on the side. And so this arbitrary notion of time spent in a job helps create a hard-to-prove loophole to filter people by years instead of pure skill.

Today ‘years of experience’ is one of the top filters that companies use to cut through high volumes of prospects. Just look at the Premium feature of LinkedIn, the most-used recruiting tool by hiring managers and recruiters. It’s among the first filters that recruiters see on the left module.

Screen Shot 2015-09-28 at 12.40.32 PM

Some recruiters have alternative techniques. A blogger dubbed “Boolean Blackbelt” uses this boolean search to find people who graduated in 2004, for instance:

site:linkedin.com -dir (java | j2ee) -recruiter (engineer | consultant | programmer | developer) “location * Greater Atlanta”  “(“BA” | “B.A.” | “BS” | “B.S.” | “Bachelor” | “Bachelors”) * * * * * * 2004″

To top it off, McGraw Hill textbooks like “Start Your Own Business,” author actually capitalizes “PREVIOUS EXPERIENCE” in its guidelines on how to write a great job description. Really, why all of this emphasis on years of previous experience?

The Flaw is Inadvertently Stuck in the Process

A common perception is that because the nature of technology is fast, younger people are stereotypically more adaptable. But many reports underscore an accusatory connotation against companies. For instance, when Google and Twitter were under fire for age discrimination lawsuits earlier this year, repeated reports surfaced the infamous Mark Zuckerberg quote back in 2007: “Young people are smarter.” With each age discrimination lawsuit in tech, that soundbite gets another jolt of life (here, here and here).

But it’s generally not deliberate. The root of this homogenous range of age is two-fold. First, such classification based on stereotypes (re: the Scott example) is sociologically human nature, as PhD Robert B. Cialdini says in Influence.

“We can’t be expected to recognize and analyze all the aspects in each person, event, and situation we encounter in even one day. We haven’t the time, energy, or capacity for it. Instead, we must very often use our stereotypes, our rules of thumb to classify things according to a few key features and then to respond mindlessly when one or another of these trigger features is present. – Cialdini.

For instance, one common stereotype against older workers is they have slower cognitive abilities than the youth. It’s just not true. Scientific evidence reveals that older workers aren’t necessarily less cognitive nor are they any less creative.

Second, this long-standing qualifications has been used to narrow down job applicants for centuries. According to one 1901 journal, engineer Frederick W. Taylor, one of America’s first management consultants, first crystallized the idea of analyzing job ads to write better, standardized job descriptions in Shop Management. He created a list of the most common attributes for each profession to be able to find similar people who’d succeed in the role. Sounds pretty logical for the 1900s.  

By the end of 1917, most managers in the country adopted this boilerplate. It just so happens that previous experience was included in checklist. Until the Age Discrimination in Employment Act of 1967, people used to actually specified the age in job descriptions.

Changing the Syntax, Closing the Loophole

It’s not the length of time we’re looking for, but the rate of learning and substance of achievements of which they’re proud. The loophole is a matter of poor syntax–a steel oversight that withstood the passage of time and innovation.

Opponents might argue that, for some senior level jobs, you need specific experience that can only come with time. You need to be able to see projects through; see the ramifications of your decisions.

While these are all valid points, it’s still not the correct syntax. Again, the maturity and seniority that comes with time can be evident by what you’ve achieved and learned. Some people are ahead of their years, while others grow to remain emotionally stunted. Years in a job–alone–aren’t enough to prove that you learned something in those years. Dan Parker runs a coding boot camp, Code Fellows, in Seattle puts it well when he says: “Regretfully, 10 years of experience can also mean 1 year of experience done 10 times.”

Pic_4

Take Kari Tarr, for instance, who was dead-set on switching careers from finance to engineering at Airbnb. She put in late nights of hard work to learn coding on the side. She wrangled friends to use CodePair to watch her code and help her progress. But when she first approached the high-growth startup’s engineering team to express her interest, they were all extremely hesitant. It’s understandable. Airbnb is a billion dollar startup with an extremely high bar for hiring. How can they let someone–even one of their own–come aboard with no experience?

“They were worried they might have to spend too much time showing me how to carry out specific tasks,” Tarr says. “I expected them to be worried.”

But the challenge was on. She rolled up her sleeves and made it a targeted goal to prove herself to the Airbnb engineering team. Tarr did this by strategically choosing projects and keeping an eye out for opportunity.

“If something needed to be automated, I’d volunteer for it. I looked for opportunities that would force me to get exposure to our code base,” Tarr says.

And the engineering team watched it all happen. A year later, they could see her progress and were convinced she’d be a great addition to the team. It didn’t matter whether or not she fell into the right bucket of “years of experience.” Her rate of learning was through the roof.

Plus, the effervescent nature of technology means that the skills you learned last month could become irrelevant tomorrow. Compared to other industries, software engineers generally don’t have as much experience because the field is relatively new. StackOverflow, for instance, pointed out that:

40% of doctors have at least 10 years of professional experience in the US while only 25% of developers have at least 10 years of experience.

The most in-demand programming languages today, according to IEEE, were created just 20-30 years ago. This list doesn’t include newer cutting-edge programming favorites, like Ruby on Rails (10 years old), Go (6 years old) and Swift (15 months old).

Pic_5

And it can be cyclical too. COBOL programmers are stereotypically outdated, but there’s a strong case to prove that COBOL will make a comeback within the next few years. Software engineers are serial learners. There’s a study that finds that, in general, 6-months is usually the amount of time it takes to pick up new tools for professional programmers. If you filter by skill and focus on how fast people learn, you’ll open up a flood of talented engineers.

In turn, David Heinemeier Hansson, author of Ruby on Rails, suggests that engineering candidates should actually view qualifications like “3-5 years of experience in Ruby” as a red flag.

It’s really time we reevaluate how we’re measuring experience. Years are not a measure of knowledge because everyone learns at a different rate. Rather than succumbing to overgeneralizations of entire generations, and risking age discrimination, a better filter is the rate of learning–focused purely on skill.

Will you eliminate “Years of Experience” as a primary filter to narrow the field?
Tell us if you agree or disagree in the comments below.

 

 


 

Girls Who Code: You Can’t Live Without Female Software Engineers

We all know the sobering stats: Only 16% of the US’s 3.1 million software engineers are women.

What we don’t emphasize often enough is that even though women make up a dismal fraction of software engineers, their influence is extraordinarily pervasive. They’re pioneering many cutting-edge technologies. SpaceX’s Amanda Stiles, for instance, doesn’t let naysayers keep her from simulating space exploration by building the F9, Dragon and Crew operations. Meanwhile, Microsoft’s Dona Sarkar is busy helping to build the first untethered hologram computer. How can we not mention Mary Lou Jepsen who is leading the next frontier of virtual reality at Facebook-owned Oculus.

The influence of female software engineers is felt all the time. In fact, we bet you can’t even go a day–heck, a few hours–without feeling the influence of female software engineers. From the very moment you wake up to the time you go to sleep, there’s a resilient female engineer who helped create the daily apps and technologies you touch.

Here’s an illustration, showcasing women behind popular, powerful technology we use hour by hour:

you-can-t-live-without-female-engineers (1)

When you take a look at this powerful infographic, it’s actually really disturbing to see sensationalized headlines like: “Why are there so few female leaders in tech?” amplified in the media. (Other similar headlines here, here and here). There seems to be a new article every day that claims to pinpoint the core problem resulting in few women in tech. From amiss gender stereotypes to unconscious bias during interviews, there are a myriad of reasons why there are such few women in engineering. But it’s a nuanced problem, and discussing the many reasons for it–alone–can be futile.

Our dialogue should also emphasize illuminating kickass women achieving amazing things. Female software engineers have immeasurable influence on the world today, prevailing the lack of diversity or hurdles of Boys’ Club that stand in their way. As Sabrina Farmer, a Google engineering manager who heads up Gmail, says:

“I’m not Superwoman, and my job is hard, but the pay is good and the perks are incredible, I have a great career and family, and the world has changed because of technology — and I’ve been a part of it.”

If all we ever talk about are the sobering percentages of few women in tech, our new generations of young girls might grow up believing that women aren’t as influential in building technology we depend on. In reality, with or without public recognition, women have always been the cornerstone of modern technology. Historically speaking, coding runs deep in females’ blood.

Women Have Always Been the Underdog Heroines of Programming

The first programmers were women. Those familiar with female influence in tech most often point to two names: Grace Hopper and Ada Lovelace. Like all female software engineers, both Lovelace and Hopper brought a unique, essential perspective that helped grow and shape modern computing. They offer diverse opinions that encourages open communication, empathy and analytical thinking. In the 1800s, Lovelace, for instance, worked alongside Charles Babbage, the first person to conceptualize a programmable computer. One expert analyzed the letters exchanged between Lovelace and Babbage and found that: “Lovelace and Babbage had a very ‘different qualities of mind’…whereas Babbage focused on the number crunching possibilities of his new designs, Lovelace went beyond number-crunching to see possibilities of wider applications.”

It was Lovelace, for instance, who suggested that the Analytical Engine could be used for more than just numbers. Just look how beautifully she describes the revolutionary punch card mechanism:

“We may say most aptly, that the Analytical Engine weaves algebraical patterns just as the Jacquard-loom weaves flowers and leaves.”

About a century later, after Pearl Harbor, the Navy recruited Hopper to build computers. She was a leader in furthering innovation computer development field. As author Kurt W. Beyer points out, her pivotal mark was her advocation for open source ideology and inter-operational computing languages, versus closed-source protection by intellectual property law:

“‘It is in our earnest plea,’ she wrote to ACM, ‘that we receive comments both mild and violent, suggestions and criticisms, as soon as possible.’ By broadening participation during the development phase, Hopper increased the odds that the computing community would freely adopt the resultant language.’

Today Hopper is lovingly referred to as the mother of COBOL, as the first to push for a programming language that was readable as English instead of computer jargon.

But these two programming heroines are just the tip of the iceberg.

In the 1940s, historian Nathan Ensmenger explains that most people generally thought that software programming was “women’s’ work,” consisting of plugging in numbers and shifting switches (like secretarial filing). Hardware engineering was considered more manly, according to the Smithsonian. But the field was so new. People didn’t realize that these women would rise to the occasion and become the cornerstone of computing.  They’d even become experts on how to improve functionality and solve tough programming tasks.

computer_girls

As the war went on, the demand for smart mathematicians to calculate weapon trajectories using computers grew. The US military recruited women as “Math Rosies” for ballistic research. Some women even went on to work on bigger machines, like the ENIAC.

Jean Jennings Bartik was among six women who helped build the ENIAC, and were mentored by John Mauchly. Men may have built the hardware of such machines, but it was women like Bartik who laboriously debugged every vacuum tube and learned how to make the 6-foot machine work–sans books or even chairs. Plus, the mission was secret, there wasn’t much opportunity for public recognition for their work. It was so secretive that they couldn’t even see the computer they were working on until security clearance came through. Still, even after the ENIAC was finally announced:

“They all went out to dinner at the announcement,” she says in a talk at a computer history museum. “We weren’t invited and there we were. People never recognized, they never acted as though we knew what we were doing. I mean, we were in a lot of pictures.”

Interestingly enough, here’s how renowned Grace Hopper pitched programming to women in Cosmo magazine, circa 1960s:

“[Programming is] just like planning a dinner. You have to plan ahead and schedule everything so that it’s ready when you need it…. Women are ‘naturals’ at computer programming.”

It makes sense — she wanted to appeal to the maternal side of women, since that was most valued at the time. But, in reality, Hopper was aiming to fill a huge demand to work through complex problems by smart women with an aptitude for math and numbers. Unfortunately, these female programmers or “Computer Girls” never really got the credit they deserved through time.

It’s Hard to Be What You Can’t See

Whether or not the world recognizes it, smart, ambitious women have always been and always will be the cornerstone of computing. From its inception to new frontiers, software requires diversity from its engineers. Among immeasurable impact, female software engineers can help open up lines of communication, broaden viewpoints and bring a level of creativity and empathy that’s essential to innovation. Let’s stop focusing only on the bleak diversity numbers, and start highlighting the empowering stories of triumphant women who pioneered innovation–not just for the recognition–but to help shape our tech-driven world today. As our very eloquent HackerRank Ambassador Anjan Kaur says:

“Software engineering is too interesting to leave just to men.”

 

Want to take action to help bring more women to engineering? Get involved.

Women’s Cup is an all-woman online hackathon happening October 10th. Developers will solve coding problems to challenge themselves, develop their skills and win prizes! Companies who sponsor the event will obtain a list of the top female engineers at the end of the contest.

Join the cause. Join Women’s Cup now.

 

 

 

Image sources: [1] [2] [3] 

Legendary Productivity And The Fear Of Modern Programming [TechCrunch]

This article originally appeared on TechCrunch


 

JavaScript master Douglas Crockford once said that software is the most complex thing that humans have ever created. It’s made up of intricate bits and bytes pieced together like virtual puzzles. These insurmountable calculations help us achieve extraordinary feats, like routing human beings to the uncharted craters of Mars. By the very nature of the burgeoning computer science discipline, “we’re dealing with things that are on the edge of what humans can handle and more complicated than ever before,” said renowned computer scientist Donald Knuth.

While no one programming legend can possibly accomplish any big feat solo, there are programmers worthy of fame for their supreme productivity. Every so often, leaders of new revolutionary tools make an explosion in the field that reverberates across generations of new programmers.

But what’s even more interesting is that some of the highest-achieving programmers — who can make sense of such unfathomable complexity — can’t foresee a lucidly bright future ofprogramming. Several accomplished computer scientists share a grave concern of the shift toward our more fragmented web. Tools that act as layers, like frameworks and packages, are created to help programmers be more productive, but some experts fear they’ll actually have the opposite impact long-term.

If the foundation of the modern world is built on software, then deconstructing the toolkit of today’s software leaders can help us not only become better programmers, but develop a better future. Contrary to popular belief, greatness isn’t exclusive to unreal legends. Culture critic Maria Popova puts it most eloquently when she says, “Greatness is consistency driven by a deep love of the work.”

After researching stories on and conducting in-depth interviews regarding seven programmingpioneers, from computer scientist Donald Knuth to Linux’s Linus Torvalds, we uncover productivity patterns common to achieving greatness and pitfalls of which to steer clear: There’s never been a widely used programming tool that was created by just one lone wolf in a cave. Sure, Jeff Dean is the trailblazer of the famed distributed computing infrastructure upon which Google towers. Peter Norvig may be immediately associated with JSchema. David Heinemeier Hannson’s pride and joy is the Ruby on Rails framework. But each of these creators had support for their groundbreaking inventions.

Teamwork is the foundation of an empire that lone wolves simply can’t sustain. It’s why Google, home of world-renowned engineers, doesn’t tolerate lone wolves on campus. It’s the antithesis of “Googliness,” and software development in general, for two core reasons.

First, the mere proximity to other engineers fuels greatness. When Rob Pike worked at Bell Labs, well before making waves on the Unix team, he recalls fond memories of hovering around clunky minicomputers with terminals in a machine room in the Unix Room. “The buzz was palpable; the education unparalleled,: he said. “The Unix Room may be the greatest cultural reason for the success of Unix as a technology.”

Folks like Ken Thompson and Dennis Ritchie (authors of C Programming Language) would code, sip coffee, exchange ideas and just hang out in the Unix Room. It was this necessity of convening in a physical room that helped turn Unix into what it is today. Since the proliferation of PCs, however, proximity to each other isn’t as rigid in modern programming. Going out of your way to meet with smart engineers, however, is a timeless essential contributing to greatness.

Just ask Jeff Dean, the famed Googler who often is referred to as the Chuck Norris of the Internet. As the 20th Googler, Dean has a laundry list of impressive achievements, including spearheading the design and implementation of the advertising serving system. Dean pushed limits by achieving great heights in the unfamiliar domain of deep learning, but he couldn’t have done it without proactively getting a collective total of 20,000 cappuccinos with his colleagues.

“I didn’t know much about neural networks, but I did know a lot about distributed systems, and I just went up to people in the kitchen or wherever and talked to them,” Dean told Slate. “You find you can learn really quickly and solve a lot of big problems just by talking to other experts and working together.”

Second, every great coder, with however pristine a track record, must check in their code for a peer review. Pivotal Labs, the company behind the software-scaling success of Twitter, Groupon and a dozen other high-growth Silicon Valley startups, requires the freedom for any coder to look at any code. It’s simple: If code is too dependent on one person, your business is susceptible to dire wounds.

Even the great Guido van Rossum checks in his Python code, and Thompson checks in C code. Linus Torvalds, author of the biggest collaboration Open Source Software project in history, says he actually judges coders — not by their ability to code — but how they react to other engineers’ code. As a bonus, reading others’ code also can help you become a better coder. You often can see things in the source code that are left unsaid in the office.

Knuth would wholeheartedly agree. “The more you learn to read other people’s stuff,” he said, “the more able you are to invent your own in the future, it seems to me.”

When Collaboration Goes Awry

Collaboration is generally a solid learning tool, but can be catastrophic when injected at the wrong time. Hansson has seen too many Open Source Software (OSS) projects fallen victim to premature collaboration. That’s why, for the first 1.5 years of RoR’s life, Hansson had commit rights to the framework. And it took another 4.5 years before he felt comfortable collaborating with another framework to produce RoR version 3. Even then, it wasn’t easy for him.

“I’ve seen plenty of open source projects being pulled in a thousand different ways because you allow collaboration to happen too early before the culture and vision is established enough so that you can invite enough people safely into the project without them spoiling it,” he says in a Developer’s Life podcast.

Plus, the truth of the matter is, many programmers simply enjoy blasting through code alone. Solo coding is faster, short-term. You don’t have to worry about communication mishaps. And things are generally more uniform. Engineer Ben Collins-Sussman once asked a room full of programmers:

  • How many of you work solo? …Crickets…

  • How many of you LIKE to work solo? Nervous laughter and raised hands spread across the room.

Collaborating is a necessary evil for many of the greats, like Norvig, who is now Director of Research at Google. Norvig makes the observation that too much collaboration is also not as effective. “If you’ve got two good programmers,” he says, “it’s better for them to work independently and debug each other’s work than to say we’ll take 50% hit just for that second set of eyes. 10% of the time it’s good to sit down and have that shared understanding. But I think most of the time, you’re not going to be as effective.”

For collaborating on a project that lacks a clear vision, unlike that of Hansson, it’s good to figure out what the problem is together. Once you have an idea, great programmers divvy up the work; leave them alone to burn through the code and sync up during review. The best collaboration happens by creating a solid feedback loop, where you can catch one another’s errors before you’re too far into the project.

John Carmack, known for creating Doom, was also very wary of adding more programmers to his process of achieving his vision of great games. There was a time when Carmack wrote the majority of the basic program, he says.

Now, we’ve got lots of situations where, if something is not right, it could be like, “Oh that’s Jan Paul’s code, or Jim’s code, or Robert’s code.” It’s not so much a case where one person can just go in and immediately diagnose and fix things. So, there is a level of inefficiency.

Again, there’s a limit to what a lone wolf can do. Doom 3 required a lot more features, which meant more programmers. Carmack found this to be a double-edged sword. So, it’s manageable, he says — and necessary — in achieving a grander vision.

Jack Of All Trades And Master Of One

Most — if not all — of the seven legendary programmers had at least one thing in common: They mastered one well-defined domain, primarily using one language to carry out their work. Jonathon D. Tang pointed this out in a recent Hacker News comment, and it rings true. Knowing your language inside out is key to optimal performance. So, “read the reference, read books that not only show you the mechanics of the language but also debugging and testing,” Norvig advises. Being able to memorize the common calls for your language of choice can help expedite productivity.

Screen Shot 2015-09-14 at 11.22.54 AM
But it can get really comfortable (or boring) really fast if you stick to just one set of tools and domain. So, after mastering one definitive domain, most of them have experience in switching contexts, which helps them open their minds and see things in a different way.

For instance, John Carmack switched from creating video games to Oculus for virtual reality. LIkewise, Rob Pike moved from Plan 9 to Go. Andy Hertzfeld transitioned from writing the original Mac system software in assembly to writing the Google+ Circle Editor in JavaScript. “But rarely do the greats ever try to juggle multiple platform at the same time,” Tang said in a follow-up email. Delving into new domains after mastering one helps you see things in different levels.

Visualizing The Program In Your Brain

Nearly every programming legend points to the importance of visualizing solutions and being able to hold programs in your head. There’s a lot of productivity lost when you dive into code or start testing without first developing a mental model of the solution.

Carmack’s ability to hold gaming concepts is among the most remarkable, considering just how massively complex the virtual world of gaming is. “A video game’s program is actually more complex than that of space shuttles sent to the moon and back,” he said in an interview. Paul Miller helps put his work into context. When manipulating light beams to create a virtual world, he renders an image at least 30 times a second to prevent the game from breaking. This adds up to a trillion calculations per second. Meanwhile, Disney’s Pixar takes 10 hours to render a single frame. In video games, you’ve got just milliseconds to make an impact.

Given the extensiveness of video gameprogramming, Carmack says, “being able to clearly keep a lot of aspects of complex system visualized is valuable. Having a good feel for time and storage that are flexible enough to work over a range of 10 orders of magnitude is valuable.”

Interestingly enough, Knuth, Norvig, Dean, Pike, Torvalds, Thompson and Hansson have all at one point said they’re believers of having a strong mental model, focus and visualization. It’s all about the ability to see the solution before diving into the problem.

The best concrete example comes from Norvig. He once was tasked to write a Sudoku-solver program. From the get go, he knew from his AI knowledge that the combination of field of constraint propagation and recursive search would solve the problem. Meanwhile, another programmer tested all sorts of code on his blog, but never really solved anything. It’s perfectly possible to write correct, tested code without correctly approaching the problem.

Herein lies the key to approaching the problem correctly. “I think it’s useful to imagine the solution, to see if it’s going to work,” said Norvig. “It’s useful to see if it feels comfortable.”

There’s Joy Inside The Black Box

In Coders at Work, Knuth expresses a major concern for the future of programming if young programmers are simply assembling parts without studying them. The neatly packaged boxes might be a good short-term solution for speed, but programmers will lack the grand visualization that’s necessary for true progress in programming. Plus, it’s just not as fun to copy/paste commands without knowing the fundamentals of why and how it’s happening.

“If the real case takes a certain time, t, then the complex case takes time 4t. But if you’re allowed to open the box, then you’ll only need three real matrix multiplications instead of four because there’s an identity that will describe the product of two complex matrices,” Knuth explains.

In fact, the very purpose of Knuth’s most famous work, The Art ofProgramming, helped programmers learn how and why data structures worked. “So my book sort of opened people’s eyes: ‘Oh my gosh, I can understand this and adapt it so I can have elements that are in two lists at once. I can change the data structure.’ It became something that could be mainstream instead of just enclosed in these packages.”

Comparing it to mathematics, it wouldn’t be fun to simply get the right theorem for the right problem each time. The joy comes from trying various theorems, visualizing what might work and getting that thrill when it finally does. Norvig shares this concern and stresses that programmers need to ask more questions before copy/pasting black boxes.

Sure, the code might have worked once, but what are the failure cases? Is it consistent? Are there other scenarios for expanding functionality? How can you understand it better? Simply relying on prepackaged boxes to carry out a command can be tedious and mundane. It’s more fun to problem-solve, think critically and learn the mechanisms of why code strung together is performing in a certain way.

Thompson is downright fearful of modern programming because it’s made up of layers upon layers upon layers. “It confuses me to read a program which you must read top-down. It says ‘do something,’ and you go find ‘something’ and it says ‘do something else’ and it goes back to the top maybe. And nothing gets done. I can’t keep it in my mind — I can’t understand it.”

While frameworks, APIs and structures might make programmers feel more productive, trueproductivity comes from unpacking the containers and truly understanding what’s inside. Only then can you build upon your knowledge and continue to push the limits of what human beings can achieve.

The greatest engineers who have brought us this far in software grew up in an entirely different era of computing — packed inside rooms filled with terminals, carrying mental models of new algorithm-based programs. Newer generations of programmers are born into a more fragmented world. The programs today are far too large to carry in our minds. Plugging in black boxes can get you to a certain point — very quickly — but the greatest programmers will always be bored with redundancy. True greatness comes with consistent drive to seek new problems with like-minded programmers, be able to see the floor before the roof and stay curious about what’s inside the black box.

 

 


Enjoyed this piece? Subscribe to our free newsletter to get new articles in your inbox. 

The Interdependency Of Stanford And Silicon Valley [Tech Crunch]

This article also appeared on Tech Crunch


There was a time when Stanford University was considered a second-rate engineering school. It was the early 1940s, and the Department of Defense was pressed to assemble a top-secret team to understand and attack Germany’s radar system during World War II.

The head of the U.S. scientific research, Vannevar Bush, wanted the country’s finest radio engineer, Stanford’s Frederick Terman, to lead 800 researchers on this secret mission. But instead of basing the team at Terman’s own Stanford lab — a mere attic with a leaky roof — he was sent to the acclaimed Harvard lab to run the mission.

It’s hard to imagine Stanford passed over as an innovation hub today. Stanford has outpaced some of the biggest Ivy League universities in prestige and popularity. It has obliterated the traditional mindset that eliteness is exclusive to the Ivy League. Stanford has lapped top schools by centuries. It ranks in the top 3 in multiple global and national rankings (here, hereand here).

Plus, survey results point to Stanford as the No. 1 choice of most students and parents for the last few years, over Harvard, Princeton and Yale. In fact, even Harvard students haveacknowledged Stanford’s notable rise in popularity.

elite_university

But something a little more intriguing is happening on Stanford’s campus…something that goes beyond these academic rankings. Since the beginning of time, the goal of academia has been not to create companies, but to advance knowledge for the sake of knowledge.

Yet Stanford’s engineering school has had a strong hand in building the tech boom that surrounds it today. It’s not only witnessed, but also notoriously housed, some of the most celebrated innovations in Silicon Valley.

While Stanford faculty and students have made notable achievements across disciplines, their role in shaping the epicenter of The Age of Innovation is perhaps one of the top — if not the most unique — distinguishers. As the world’s eyes fixate on the booming tech scene in Silicon Valley, Stanford’s affiliation shines brightly in the periphery.

In return, its entrepreneurial alumni offer among the most generous endowments to the university, breaking the record as the first university to add more than $1 billion in a single year. Stanford shares a relationship with Silicon Valley unlike any other university on the planet, chartering a self-perpetuating cycle of innovation.

But what’s at the root of this interdependency, and how long can it last in the rapidly shifting space of education technology?

Fred Terman, The Root Of Stanford’s Entrepreneurial Spirit

To truly understand Stanford’s role in building Silicon Valley, let’s revisit WWII and meet Terman. As the leader of the top-secret military mission, Terman was privy to the most cutting-edge, and exclusive, electronics research in his field. While the government was eager to invest more in electronics defense technology, he saw that Stanford was falling behind.

“War research which is [now] secret will be the basis of postwar industrial expansion in electronics…Stanford has a chance to achieve a position in the West somewhat analogous to that of Harvard of the East,” Terman predicted in a letter to a colleague.

After the war, he lured some of the best students and faculty to Stanford in the barren West by securing sponsored projects that helped strengthen Stanford’s reputation in electronics. Here’s a great visualization, thanks to Steve Blank, about how Stanford first fueled its entrepreneurship engine through war funds:

terman_cold_war

This focus on pushing colleagues and students to commercialize their ideas helped jumpstart engineering at Stanford. Eventually, Stanford’s reputation grew to becoming a military technology resource, right up there with Harvard and MIT.

But Terman’s advocacy of technology commercialization went beyond the military. As the Cold War began, Terman pushed to build the Stanford Industrial Park, a place reserved for private, cutting-edge tech companies to lease land. It was the first of its kind, and famously housed early tech pioneers like Lockheed, Fairchild, Xerox and General Electric.

The research park was the perfect recipe for:

  • A new revenue stream for the university
  • Bringing academic and industry minds together in one space
  • Inspiring students to start their own companies

You might say that the Stanford Industrial Park was the original networking hub for some of the brightest minds of technology, merging academia and industry, with the goal of advancing tech knowledge.

They had a harmonic relationship in which industry folks took part-time courses at Stanford. In return, these tech companies offered great job opportunities for Stanford grads.

Since then, Stanford’s bridge from the university to the tech industry has been cast-iron strong, notorious for inspiring an entrepreneurial spirit in many students. The most famous story, of course, is that of Terman and his mentees William Hewlett and David Packard, who patented an innovative audio oscillator. Terman pushed the duo to take their breakthrough commercial.

Eventually, Hewlett-Packard (HP) was born and moved into the research park as the biggest PC manufacturer in the world. To date, he and the late David Packard, together with their family foundations and company, have given more than $300 million to Stanford.

Because of their proximity to top innovations, Stanford academics had the opportunity to spot technological shifts in the industry and capitalize by inventing new research breakthroughs. For instance:

  • Computer Graphics Inc.: Students were enamored by the possibility of integrated circuit technology and VLSI capability. The Geometry Engine, the core innovation behind computer generated graphics, was developed on the Stanford campus.
  • Atheros: Atheros introduced the first wireless network. Teresa Meng built, with government funding, a low-power GPS system with extended battery life for soldiers. This led to the successful low-power wireless network, which eventually became Wi-Fi.

These are just a few of some of the most groundbreaking technological innovations sprouted from Stanford soil: Google, Sun Microsystems, Yahoo!, Cisco, Intuit … and the list goes on — to more than 40,000 companies.

Stanford also has a reputation as a go-to pool for talent. For instance, the first 100 Googlers were also Stanford students. And today, 1 in 20 Googlers hail from Stanford.

Proximity to Silicon Valley Drives its Tech Entrepreneurial Spirit

If you stroll along the 700 acres of Stanford’s Research Park, not only will you see cutting-edge companies like Tesla and Skype, but also world-renowned tech law firms and R&D labs. It’s a sprawling network of innovation in the purest sense of the term — it’s the best place to uproot a nascent idea.

Proximity to Silicon Valley is not the most important thing that distinguishes Stanford, but it’s certainly the most unique. It’s the hotbed of computer science innovators, deep-pocketed venture capital firms and angel investors.

At least today, everyone who wants to “make it” in tech is going to Silicon Valley. And — just like Terman’s early Stanford days — it’s where you can meet the right people with the right resources who can help you turn the American entrepreneurial dream into a reality.

Just look at the increasing number of H-1B visa applicants each year, most of whom work in tech. There were more than 230,000 applicants in 2015, up from 170,000 in 2014. Four out of the top 11 cities that house the most H-1B visa holders are all in Silicon Valley.

Plus, an increasing number of non-tech companies are setting up R&D shops in Silicon Valley. Analyst Brian Solis recently led a survey of more than 200 non-tech companies; 61 percent of those had a presence in Silicon Valley, which helped them “gain access and exposure to the latest technology.”

There’s certainly a robust emphasis on technology entrepreneurship penetrating the campus of Stanford engineers.

Still, opponents often point to media exaggeration that reduce Stanford into a startup-generator. Of course, Stanford’s prestigious curriculum is a draw for top faculty and research across disciplines. But, given the evidence and anecdotes, there’s certainly a robust emphasis on technology entrepreneurship penetrating the campus of Stanford engineers. How can it not?

Michael Harris, a Stanford alumnus, can attest to a general sense of drive and passion. “It’s not quite as dominant as the media makes it seem,” he said, “but there’s some element of truth.”

Stanford students are by and large interested in creating real things that have a real effect in the world. The fact that Silicon Valley is right here and students have fairly good access through friends, professors, the school, etc. to people in the industry is definitely a big bonus. It gets people excited about doing work in the tech industry and feeling motivated and empowered to start something themselves.

This Entrepreneurial Spirit Is Evolving Into A Sense Of Urgency

Terman’s early emphasis on turning the ideas developed in academia into viable products is just as — if not more — rampant today. The most telling evidence is that Stanford’s campus is producing more tech startup founders than any other campus.

But what’s even more curious is that some students, particularly in the graduate department, don’t even finish their degrees. It’s monumental to pay thousands of dollars for a master’s degree in computer science, only to leave to launch a startup. Even at the undergrad level, Harris thought about leaving college after doing one amazing internship the summer after his junior year.

“I will say that working in industry teaches you more things faster about doing good work in industry than school does by a really big margin (order of magnitude maybe),” Harris said, “so I don’t actually think it’s crazy for people not to go back to school other than the fact that some companies seem to think it’s important for someone to have a piece of paper that says they graduated from college.”

Of course, most people do finish their degrees. But this sense of urgency to leave — whether or not the majority follow through — is palpable.

Last year, six Stanford students quitschool to work at the same startup.Another 20 left for similar reasons the year before that. Apparently, Stanford’s coveted StartX program wasn’t enough for them.

StartX is an exclusive 3-month incubator program to help meet the demand for students who want to take their business ideas to the market — complete with renowned mentorship and support from faculty and industry experts to help the brightest Stanfordites turn their ideas into a reality.

In a recent talk, Stanford President John Hennessy proudly spoke about this program as a launchpad for students to scratch their itch for entrepreneurship. But when an audience member asked him about students dropping out of school, he said, “Look, for every one Instagram success, there are another 100 failed photo-sharing sites.” And, he added, “So far, all of the StartX program students have graduated — at least all of the undergrads.”

Generally, Stanford’s graduation rates have dipped somewhat in recent years. Of students who enrolled in 2009, 90 percent had graduated within 5 years, Stanford said, compared with a 5-year graduation rate of 92.2 percent 5 years earlier. And this is not a bad thing for Stanford. Since the very beginning, a core function of Stanford’s excellence is its investment in its students to build great commercial products — starting with the early days of Terman.

The Future: What Will Stanford Be Without Silicon Valley?

But both education and the Valley are shifting. The very nature of innovation frees us from brick-and-mortar walls of elite institutions and companies.

If the best application of technology is to democratize opportunity, then every single person on the planet should have affordable access to Stanford’s world-class education online. The rise of Massive Open Online Courses (MOOC) and online resources are an indication of the future of education.

It’s a future in which ambitious students have the opportunity to educate themselves. At the forefront of technology, educational institutions, including Stanford, are starting to decentralize the model through online course material.

Stanford shares a relationship with Silicon Valley unlike any other university on the planet.

Meanwhile, Silicon Valley may have pioneered the tech boom, but it’s no longer the only tech hub. Bursts of technological hubs are forming all over the world. In a piece on the H-1B visa cap, I found that the top investors in early stage startups have set up shop in India, China and Israel, three of the largest global tech hubs after Silicon Valley.

Realistically, the H-1B visa cap and city infrastructure can’t practically support exponential growth in Silicon Valley. The nucleus of innovation will eventually shift, deeming the proximity to Silicon Valley irrelevant.

Plus — as some students aren’t even finishing their degrees — it’ll be worth re-evaluating if thousands of dollars for a master’s in CS at Stanford is really worth the brand name on a resume or access to coffee with top startup founders who happen to reside in Palo Alto.

But if Stanford’s proximity to Silicon Valley drives its entrepreneurial essence, which helps bolster both the reputation and funding of Stanford, what will happen when the ambitious, startup founders at Stanford start getting their education online?

Will Stanford end up disrupting the very unique factor that distinguishes Stanford from any other university on the planet? Or will Stanford’s alumni continue to fuel its self-perpetuating cycle of innovation and maintain its reputation as an innovation hub?

How Amazon Web Services Surged Out of Nowhere

Few people–if any–saw it coming. Even engineers who helped build the omnipresent Cloud that is Amazon Web Services (AWS) are surprised by its goliath success today.

“You never know how big something will be while you’re working on it, Christopher Brown,” an early AWS engineer told Business Insider.

When Amazon CEO Jeff Bezos first proposed selling infrastructure-as-a-service (IaaS), his board of directors raised an eyebrow. It’s understandable. For the general population, “Cloud-based technology platform” isn’t exactly the first thing that comes to mind when you think of Amazon. AWS is far removed from the core function as an e-commerce site going 21-years-strong. But today AWS zooms so far ahead in the market that its competitors are dwarfed into tiny specs in its rear view mirror. Even its own execs behind its growth strategy say they’re surprised at the speed of AWS’s claim to fame. Here’s a matrix depiction of the market share by Gartner:

 

gartner

For several years, AWS has been a developer’s paradise, a platform where you can build and run software without having to set up your own servers. It’s grown to be so huge that even its IaaS competitors, like DigitalOcean, depend on it. Nearly 40% of traffic runs through its infrastructure, according to an estimation by Jarrod Levitan, Chief Cloud Officer at TriNimbus.  Another more concrete statistic by Deepfield Networks found that 1/3 of Internet users visit an AWS-supported website daily. 

If AWS stopped working tomorrow, much of the Internet would dim. You wouldn’t be able to browse billion dollar social networking sites, like Pinterest. Your DropBox photos and files would be missing in action. You couldn’t stream Netflix. Forget scoring a great deal on Airbnb. Many of these billion dollar startup may have never disrupted their respective industries without AWS to propel them quickly on the scene. Plus, enterprises and even the ever-paranoid governmental agencies, like the CIA, NASA, Department of Defense and the Pentagon are now relying on AWS.

Not only is it massive, but its capabilities are astounding. Of the dozen services, arguably among the most innovative is Amazon Kinesis, a real-time processing service for live events and streaming. It can continuously capture and store terabytes of data each hour from hundreds of thousands of sources at once. It’s great, for instance, for helping advertising agencies make sense of live social chatter.

amazon

The team knew the initial concept was–at most–an interesting idea, but there are so many existing big players in the realm of data storage. How could Amazon, an online shopping site, compete? Having put his neck on the line in front of investors, Bezos must have known he was on to something…but no one could have predicted the scale to which AWS has escalated.

The Canny Rise of ‘the Infrastructure of the World’

There’s an old folklore about how AWS got its legs. The story goes that because Amazon’s technology capacity requirement naturally shoots off the charts during holiday season, it has excess capacity for the rest of the year. So, why not rent the excess storage to other companies that need it?

Amazon CTO Werner Vogels wonders: Why won’t this myth die?

It was never a matter of selling excess capacity, actually within 2 months after launch AWS would have already burned through the excess Amazon.com capacity.

One thing that the fable gets right is that AWS was created out of Amazon’s own need to support high volume of data. But each interface was created with design and intent that outsiders will eventually use it. Former Amazonian Steve Yegge recalls the infamous mandate put forth by Bezos around 2002:

“He issued a mandate that was so out there, so huge and eye-bulgingly ponderous, that it made all of his other mandates look like unsolicited peer bonuses.”

 Essentially, the mandate not only required all developers to expose data through surface interfaces over network but also built to be “externalizable,” or good enough for outsiders to use. So, you see, it was always about deliberate business and innovation from its inception, stemmed by Bezos’ singular focus on obsessively serving their customers. Amazon had been perfecting its infrastructure to meet massive needs for several years. Given its internal success. it only made sense for Bezos to take his team’s invaluable skill and profit from it.

But AWS wasn’t Bezos’ brainchild. 

The initial proposal paper (because Bezos doesn’t like slides) came from head of global infrastructure Chris Pinkham, who first ideated a potential “Infrastructure of the World.”

He worked closely with website engineering manager Benjamin Black:

Chris was always pushing me to change the infrastructure, especially driving better abstraction and uniformity, essential for efficiently scaling. He wanted an all IP network instead of the mess of VLANs Amazon had at the time, so we designed it, built it, and worked with developers so their applications would work with it.

From their own experience, they knew that maintaining servers eats up the majority of time and money. This push for universal infrastructure no doubt came from Bezos’ relentless prioritization of customers. Bezos famously asks every team to leave an empty seat in meetings for their customer, as a demonstration of just how customer-centric the company should be. In this case, the better, more comprehensive infrastructure Amazon offers, the better support its customers would enjoy. And so AWS was born into a world starved of affordable, reliable and all-encompassing IaaS.

Untitled Infographic (6)

But Why is Amazon the Chosen One?

Amazon has a historical lead in an industry so new that few people understand truly understand it. Some call it a Coke without a Pepsi competitor. AWS’s biggest IaaS challengers are Microsoft Azure, Google Compute Engine and IBM Softlayer. To supplement the Gartner matrix above, take a look at the difference in revenue last year–AWS is raking in more than its top competitors, including hybrid servicers, combined:

Screen Shot 2015-08-25 at 8.09.44 AM

Its legendary lead is a result of three core reasons. First, much of this growth is in parallel with the rise of Cloud computing itself. You can see the steady increase of Cloud popularity and interest via Google Search Trends. It starts to rise around 2007, just one year after the release of Amazon’s two most widely acclaimed services: Elastic Compute Cloud (EC2) that maximizes compute capacity and Simple Storage Service (S3), which stores massive data.

 

Screen Shot 2015-08-25 at 8.22.44 AM

 

Second, Amazon’s EC2 and S3 were the very first widely accessible virtual infrastructure services by a number of years. It’s part of cloud computing history. AWS had been working on solving a pain point that’s common to all developers, and perfected the solution long before anyone else in the industry.

Untitled Infographic (1)

Third, when the mindset started shift from apprehension about the Cloud to a necessary gravitation to the Cloud, existing players could only scramble to copy AWS services. Microsoft’s Windows Azure, for instance, started off as a Platform-as-a-Service (PaaS), but kept losing to AWS. Microsoft had to add IaaS capabilities to stay in the Cloud game, becoming a hybrid service and rebranded as “Microsoft Azure.” Then, Google followed the lead and launched its own virtual server service, marking the start of copycats.

As a result of these copycats, there’s been a significant price war to developers’ benefit. Specifically, AWS has dropped its price 49 times in 8 years. But what’s even more impressive is some scrambling competitors rely on AWS to provide IaaS to their own customer base. We mentioned DigitalOcean’s dependence on AWS earlier in this piece.

But Target.com’s story is even more telling. It had been a customer of AWS since 2001, until it decided to go its separate ways and build its own server. A little disheartened, but still amicable, AWS helped Target.com transfer its data. But after one popular promotion, the site crashed. It’s unsettling moments like this that light a fire under any infrastructure to capitalize on AWS’s offerings for additional support.

Moving forward, AWS isn’t settling to dominate the IaaS market. At the end of 2014, it announced a number of new PaaS tools to bolster developer’s paradise. This includes: Code Commit, Code Pipeline and Code Deploy. David Bernstein, CEO of Cloud Strategy Partners, sees AWS becoming any developer’s comprehensive environment.

The Cost of Supreme Excellence

After Pinkham was appointed to execute AWS, it was time to get to work. The AWS might have started with Pinkham and a handful of engineers sent to Africa, where he was based. But it’s grown to a massive organization under Amazon’s umbrella. A former EC2 engineer says there are several teams that are dedicated to each offering (e.g. S3 team).

A recent New York Times Amazon in-depth expose reveals some awful things about Amazon’s culture in general. Referring to it a “bruising workplace,” reporters cite unforgiving examples of lack of empathy by Amazonian leaders who prioritize work above all. Some Amazonians have responded with claims of inaccuracies and false depictions of isolated incidents strung together to depict a hellish culture.

Whether or not these anecdotes are truly representative of Amazon’s culture, one thing is certain: It takes herculean dedication to achieve excellence, especially to that of AWS’s caliber. This kind of market leadership doesn’t come without cost of sweat and tears. It’s always subjective to judge a company’s culture–what may be a joyous challenge to one might be relentlessly poor balance of work and life to another. Such high-performing teams can be self-selecting. After all, every single Amazonian chose to be there and most likely has InMail invitations from top tech companies waiting to be read.

Granted this is not a scientific measure, most reviews of the AWS team are glowing. Several AWS engineers can attest to the grueling workplace, but still express gratification in their role in trailblazing their industry:

I worked for EC2 in Cape Town, South Africa. It was the best!!!! I can’t imagine ever finding a working environment as cool as what I had in my team. That said, there were on-call weeks where I was cursing the company and my job at 4am, being awake for the 3rd consecutive night.  But the org is aware of problems like heavy on-call load. They started a program to improve unnecessary tickets. I found AWS (at least in Cape Town) to be a really well-run place to work.

Bezos himself is very clear that this type of revolutionary environment may not be for everyone. In one meeting, a female engineer asked what Amazon will do to better the work-life balance? His response is clear as daylight and indicative of severely high expectations:

“The reason we are here is to get stuff done, that is the top priority,” he answered bluntly. “That is the DNA of Amazon. If you can’t excel and put everything into it, this might not be the place for you,” he says, according to the book on the history of Amazon.

For better or worse, this is the culture that lead to the success of AWS today. Bezos was almost prophetic when he green-lit Pinkham and Black’s proposal to sell Amazon’s internal virtual service. By focusing his talented engineers’ energy on the infrastructure, and laying out a rigid culture of excellence, AWS is “winning” the market share for now.

Developers, what has been your experience with AWS…is AWS too big to fail or can competitors knock it off the #1 spot?

 


If you enjoyed reading this, please subscribe to our newsletter for an occasional update when we have something new to share!

Also, if you’d like 100 free AWS credits, you can enter the upcoming World Cup CodeSprint (or online hackathon) and complete one challenge.

The Risky Eclipse of Statisticians

If statisticians have historically been leaders of data, why was there a need for a brand new breed of data scientists?  While the world is exploding with bounties of valuable data, statisticians are strangely working quietly in the shadows. Statistics is the science of learning from data, so why aren’t statisticians reigning as kings of today’s Big Data revolution?

In 2009, when Google was still fine tuning its PageRank algorithm based on the statistical innovation Markov Chain, Google’s Chief Economist Hal Varian declared statistician as the sexiest job of the decade. We’re about halfway through, and it seems that Varian missed the target.

“Professional statisticians are milling at the back of the church, mesmerized by the gaudy spectacle of [Big Data] before them.” – David Walker, statistician, Aug 2013.  

Google Trends shows us that while the popularity of Big Data is thriving, statisticians’ popularity has been declining over the years. Back in 2010, predictive modeling and analytics website Kaggle proudly dangled Varian’s prediction as a carrot on their careers page to lure people to join their team. But today the quote curiously vanished–no longer deemed worthy.

Screen Shot 2015-07-15 at 7.06.12 PM

What speaks even louder volumes is that statisticians are often left out of some of the biggest national discussions happening around Big Data today. For instance, UC Berkeley’s Terry Speed observes:

  • US National Science Foundation invited 100 experts to talk about Big Data in 2012. Total number of statisticians present? 0.
  • The US Department of Health and Human Services has a 17-person Big Data committee. Total number of statisticians? You guessed it…0.

Justin Strauss, co-founder at Storyhackers, who previously led data science programs in the healthcare industry, can attest to this more generally. He says he has “seen an underrepresentation” of statisticians at conferences and other events related to Big Data. But statistics is the foundation of understanding Big Data. This was supposed to be their decade–their time to shine in the limelight. So, what changed? As renowned statistician Gerry Hahn once said:

“This is a Golden Age of statistics, but not necessarily for statisticians.”

Instead of crowning statisticians king, the Big Data revolution borrowed the foundational elements of applied statistics, married it with computer science and birthed an entirely new heir: The Data Scientist. But this underrepresentation of statisticians puts the future of Big Data at risk. The accurate evaluation of data that comes from a strong foundation of statistics could be lost in the hype.

Why Didn’t Statisticians Own Big Data?

Plenty has been written about the recent rise of data scientists, but the application of data science to the industry is ancient. In the 1900s, statistician William Gosset studied yeast for the Guinness Brewing Company and invented the t-distribution in the process. Statistician Kaiser Fung points out that one of the most notable examples of a business built upon statistical algorithms came decades before Google. Fair Isaac Company introduced the analytics of credit scoring in the 1950s. Not to mention the US government has been performing census calculations with incredible precision for hundreds of years as well.

There are three plausible reasons why statisticians aren’t leading Big Data today. First, computational statistics of Big Data never flourished in mainstream statistical sciences.

“The area of massive datasets, though currently of great interest to Computational statisticians and to many data analysts, has not yet become part of mainstream statistical science.” – Buja A. Keller-McNulty

This quote was published in 1999. And, a decade later, it never happened. Although early statisticians recognized and discussed Big Data, many of them were ignored. Speed points out that statisticians have published books and papers about the techniques of wrangling large datasets. But they collected dust, evident by the number of citations earned. For instance:

Screen Shot 2015-07-20 at 8.53.36 AM

Second, statistics is a crucial part of data science, but it–alone–is insufficient in making sense of exponential amounts of messy data we are producing daily. It requires computational power that can only be charged by the advanced technology we have today. In 2010, the world stored about 6 exabytes of data, a stat so incomprehensible that it’s borderline meaningless. For a frame of reference, if you converted all words ever spoken by humans into text, that’s about 5 exabytes! Here are some more quick Big Data stats:

Untitled Infographic (13)

Machine learning is deeply rooted in statistics, but few statisticians have the technical skills to manipulate a dataset of 10 billion in which each data point has a dimension of 10,000. But it’s not that statisticians lack computational knowledge. It’s that the field of statistics simply wasn’t equipped with the computing power we have today. For instance, data scientist David Hardtke lead the invention of the Bright Score, an algorithm that assesses your fit for a job, which was acquired by LinkedIn. But he says none of these ideas are really new. Back when he first started in the space, he met a senior researcher at Recruit Holdings, a japanese recruiting firm.

“He told me he’s really interested in what I’m doing because he tried to do the same thing in the 80s. He said, frankly, it was way too expensive back then. You had to buy these massive computers and it wasn’t cost effective,” Hardtke says. 

But now, we’re at this convergence of super cheap, high-speed computing that’s helping data scientists process powerful insights and find answers to questions that remained a mystery 20 years ago. With Big Data booming, pure statistics is fading into the background relative to the demand of data science.

Third, some statisticians simply have no interest in carrying out scientific methods for business-oriented data science. If you look at online discussions, pure statisticians often scoff at the hype surrounding the rise of data scientists in the industry. Some say it’s a buzzword with good marketing (here), other say it’s a made up title (here) and some call them folks who sold out to shareholders (here).

Statisticians’ Absence Could Lead to Misuse of Data

Even without a prominent presence of statisticians, educational institutions are churning out entirely new curriculums devoted to the so-called “new” field of data science in just the last few years.  But when dealing with Big Data, someone on the team needs to have a strong grasp of statistics to avoid reaching inaccurate conclusions.

The elevated hype about data scientists is undeniable. The WSJ reports that these jobs are so in-demand that data scientists with two years of experience are earning between $200,000 and $300,000 annually. It was dubbed the sexiest job of the 21st century in 2012. Universities are having to turn down data science students because of the outpour in popularity. As a result, there are at least a dozen new data science bootcamps that aim to prepare graduates for data science jobs. And universities across the nation are creating brand new courses and programs for data science or business analytics. Here’s a visualization thanks to Columbia Data Science:

columbia

But, as with any new curriculum, space is limited. This is where it gets risky. Ben Reddy, PhD in Statistics, at Columbia University finds that the foundation of statistics often takes a backseat to learning the technical tools of the trade in data science classes. And even if students are carrying out statistical models in classes, doing statistics doesn’t guarantee that you understand statistics. Since learning R or NumPy is usually the gateway to getting your hands on real-world data, understanding statistical analysis is often less interesting comparatively.

“Anyone who can type <t.test(data)> into R (not to mention <lm()>, <knn()>, <gbm()>, etc.) can “do” statistics, even if they misuse those methods in ways that William Sealy Gosset wouldn’t approve on his booziest days at the Guinness brewery.” Reddy writes. 

The worst part is, you can usually get away with carrying out subpar analysis because it’s hard to identify the quality of statistics without examining analysis in detail, he adds. And, usually, there’s not enough transparency to do this in the real-world. So, with the absence of statisticians in Big Data today, how well are the fundamentals of statistics carried over in this new data science boom? Most students haven’t even graduated from these brand new data science courses yet, so it remains to be seen.

But this risk in losing the fundamentals is largely why Hardtke, a physicist himself, is opposed to these new degree programs. He makes a compelling point: It’s better to have someone who’s really passionate about geology, physics or any other science because they’ll pick up the tools of data manipulation as part of a bigger mission.

“I’d rather have someone major to get some answer and learn the tools along the way rather than learn the tools as the terminal point,” Hardtke says.

But the Most Powerful Data Science Teams are Multidimensional

Folks outside of the space often don’t realize that the most astonishing achievements in data science weren’t accomplished by just one superstar, unicorn data scientist. When Hardtke was tasked with building a strong data science team at startup Bright.com several years ago, he couldn’t afford to recruit the best data scientist away from the likes of Google and Facebook. But he knew something most data scientist-crazed recruiters don’t understand: At its core, it’s all about learning how to ingest data using statistical methodology and computational techniques to find an answer.

Most scientific disciplines require this knowledge. So, he hired scientists across disciplines: physicist, mechanical engineer, statistician, astrophysicist–basically anyone who wasn’t a computer scientist or data scientist. The most successful, passionate data science teams in Silicon Valley comprise of a combination of different scientific disciplines that look at one problem from unique angles. It’s the only way to work through seemingly impossible problems in data science.

If you ask Nitin Sharma, for instance, about his data science team at the early days of Google, his eyes instantly light up. With experts from psychology, computer science, statistics and several other disciplines, Sharma’s diverse team offered perspectives from every dimension possible. Google’s head of search Amit Singhal once asked him: “How do you know if people are happy with the search results?” Tracking the simple act of clicks on links can’t determine whether or not the searcher was happy with his result. And so, the challenge was on for Sharma’s team.

“I can’t tell you the details of what Google did, but conceptually, we looked at what sequence do these clicks have? How much time they’re spending? How often do they refine queries? How often do they click on results? How many results? How does it depend on the type of query?” Sharma says. 

And, ultimately, Sharma’s team was able to work together to find a successful plan to monitor a user’s happiness, which offered deeper insight into search behavior and satisfaction with search results. While both data science and statistics share a common goal of extracting meaningful insight from data, the evolution of data science in the last 10 years emphasizes a demand for a combination of interdisciplinary skill.

Data science is making statistics–alone–irrelevant in industry. Hence, eclipsing statisticians, or fathers of data science.

On a scale of 1-10, Sharma says we’ve only inched maybe 1-2 in terms of progress in data science. With the forthcoming revolution of the Internet of Things, there’s infinite possibilities before us. The biggest challenge will be: How do we process and understand this unsurmountable data? The onus can’t be on “rockstar, unicorn” data scientists alone. And it can’t fall onto statisticians either. Although the demand for pure statistics will shrink relative to data science and over time, it’s going to be more important than ever to have interdisciplinary knowledge from a variety of fields. And to ensure quality and foundational understanding of applied statistics, it’s crucial to save a seat for statisticians at the Big Data table.

Have you noticed an underrepresentation of statisticians in Big Data? Tell us what you think in the comments below!


To get occasional notifications when we write blog posts, please sign up for our email list.