We went from 14 to 31 people during 2022, which was Plotline's third year. I made a bunch of hiring decisions, some good, some bad. Here is what I learned. Most of it is stuff I had been told and didn't believe until I saw it myself.
The good resume is often the wrong signal
I hired a guy named Wes in spring 2022 for a senior product role. His resume was, by any standard, very good. Stripe, Lyft, a stint at a startup that exited. References were strong. He interviewed well. He gave me the answers I wanted to hear about how he thought about product.
He lasted seven months. The problem was not that he was bad. The problem was that he had only ever worked at companies where the systems were already in place. He couldn't operate without those systems. At a 20 person company you don't have systems. You have one person who knows the customer and one engineer who knows the codebase, and you build a thing because that's the only option. Wes needed scaffolding. We didn't have any.
I have since hired three people whose resumes I would have dismissed in 2022. All three are still here and all three are doing some of the best work in the company. The signal I was missing was not "where did you work" but "have you ever been the only person responsible for something messy and unclear."
Reference calls are useless unless you make them not useless
Most reference calls go like this. You call a guy. He says nice things. He uses the word "great" three to five times. You hang up. You feel like you did your due diligence.
The thing that changed for me is asking one specific question. "Can you tell me about a time this person was wrong about something and what they did when they realized it." If the reference can answer that question with a real, specific story, you have a real person. If the reference cannot, the candidate has probably built a career on not being wrong out loud, which is a problem at a small company where being wrong out loud is required basically every Wednesday.
Other questions I now ask: "What kind of work did this person not like doing." "Was there ever a time you thought about moving them off a project, and why." If the reference can't or won't answer those, that's information too.
Stop hiring senior people for problems junior people can solve
We made this mistake at least three times in 2022. The pattern is, you have a problem. The problem feels important. You think, "I need someone senior to solve this." You hire a senior person at senior comp. They solve the problem in about three weeks. Then they have nothing to do that matches their seniority. Then they get bored. Then they leave.
I now ask the question "could a smart, motivated person who is two years out of school solve this in two months." If the answer is yes, that's who I hire. The senior person should be hired for problems that take 18 months to solve, not 18 days.
The interview problem you give people matters more than the resume
We used to give a generic case study. Pretend you are PM at a fictional company, here's a problem, walk us through how you'd solve it. The signal was bad. Everyone has prepped for that kind of question.
Now we give them a real problem from our roadmap. Sometimes a problem we're actively working on. We pay them $400 for the take home, which surprises some people, but I think it's the right thing to do. They get a week. They come in and present.
What this surfaces is whether they can ask good questions, whether they think about real constraints (cost, time, engineering), and whether they have an opinion. The people who can't do this part eliminate themselves. The people who can are usually obvious within the first 15 minutes of the readout.
It also surfaces an unexpected thing. About a fifth of the time, the candidate comes back with a solution we hadn't considered, and we actually use parts of it. Whether or not we hire them.
The thing I still don't have a clean answer to
How to evaluate cultural fit without falling into the trap where cultural fit means "this person reminds me of myself." I think we still get this one wrong some percentage of the time. We try to use structured interviews and weighted scoring, which helps. But there's a residual gut feel layer that I am not sure how to debias. I have read the literature on this. I have tried the techniques. I think we are maybe 70 percent of the way there. That last 30 percent is hard.
If anyone has actually solved this, please email me. I will buy you coffee.