For those not in the tech space, or for those who are but haven’t had to interview recently, one of the most recent trends in recruiting is to send candidates a coding challenge. The idea is to screen out those who can’t actually write code (apparently surprisingly common, see https://blog.codinghorror.com/why-cant-programmers-program/) so that you don’t have your engineers interviewing someone who is clearly not qualified.
There are a few companies that have developed platforms that make it easier for companies to do this and by far the most popular is HackerRank. They give you a problem description at the top, a text editor to write and run code in below that and then an input to test on as well as the expected output for it.
After you write code that is able to solve the known input, the server then runs the code on inputs that you are not supposed to be able to see. This is to give companies a gauge on how well you are able to write code that is able to deal with edge cases that are not given to you.
HackerRank lets you see how many hidden test cases there are as well as how many hidden test cases that you have passed. This leads to many frustrating moments when you have written code that gives the correct output on every test case except for one.
About a week ago, [company name omitted], one of the largest and most well known tech companies, sent me one of these and I faced this exact situation. After working on the last problem for a bit, I had a solution that worked for eight out of nine test cases. Sigh.
I read the problem description again and identified four areas in which there could possibly be an edge case with one of them requiring a rewrite of the main data structure that I had used.
Being lazy, I obviously did not want to do this. I really wanted to know what hidden input I was failing on so that I didn’t have to add code for all four of these edge cases when they were just testing for one.
I got up to get another cup of coffee and when I was walking back to my desk, I had a really stupid thought: “Since I know what the edge cases look like, can I just raise a runtime error if I see them?”
I really didn’t expect this to work, but added the two lines of code to test it:
then I scrolled down and I pressed the submit button:
Bingo, I knew which edge case the problematic input was and could skip writing the solutions for the other three.
HackerRank does have functionality to mask the error message, so you aren’t able to write a custom exception to dump the exact input, but this is still a massive bug and something that should be addressed urgently. I’ll be sending this to HackerRank and to the company that gave me the challenge.
Thanks for reading!
HackerRank’s CTO, Hari Karunanidhi, reached out to me:
Harishankaran Karunanidhi (HackerRank)
Nov 7, 03:51 MST
Thanks for sharing the feedback. We have a tough job of finding the right balance between having the test secure and also candidate friendly. For example, hiding the type of error would make it really frustrating for a developer to solve a challenge.
In this case, kudos on tracing out the corner case. For example, your idea of “Since I know what the edge cases look like, can I just raise a runtime error if I see them?” is not something all candidates might get during an online assessment.
Once again, thanks for reporting this. But we’ll display the “Runtime Error” to make sure the product is easy for test solvers to solve. We’ll also work towards making better testcases, so that you are forced to take care of all the corner cases.
His reasoning is understandable, but I still feel that they should just mark test cases with errors and incorrect. Still cool that the CTO himself responded.