Allen Pike 2018-05-30T00:56:35+00:00 http://www.allenpike.com/ Allen Pike http://www.allenpike.com/2018/the-great-bug-hunt The Great Bug Hunt 2018-05-29T16:10:10-08:00 Allen Pike http://www.allenpike.com/ <p>A fun thing about programming is that most days, you make progress. Maybe you fix some issues, maybe you add a feature, maybe you build towards something bigger. Your code moves ever forward.</p> <p>Until it doesn’t.</p> <p>On occasion, you will hit a Bug. Not a mundane bug, some trifle you can fix in an hour, or even a day. This is a true Bug. One that defies reason. One that evokes a “that’s not possible,” a “how could this even happen?”, or most dreadfully, a “could there be a bug in the compiler?” Hold on kids, we’re going hunting.</p> <p>Recently, I reported a regression in an app we’re working on: it had become dreadfully slow on launch. QA had noticed the issue too, which is good, but nobody on the development team had seen it happen – which is bad. Scrolling performance was awful on our CI release builds, but fine when the project was built via Xcode. The build settings seemed to be the same. Standard performance profiling turned up nothing obvious. We had ourselves a Bug.</p> <p>Diagnosing and fixing a Bug requires patience, thoughtfulness, and above all a systematic, scientific mindset. We must eliminate variables one by one. Persistently forming hypotheses and testing each one is the name of the game. While the poor engineer assigned to hunt the Bug already knows this, it is our tradition to ease the pain by sharing the story of a legendary Bug Hunt.</p> <p>Gather ‘round, friends, for the Story of the Crashing Xbox.</p> <p><img style='max-width: 100%' src="http://www.allenpike.com/images/2018/campfire.jpg" /></p> <p>You see, before Steamclock, my co-founder Nigel worked in the games industry. The games industry is very fun, when it’s fun. It’s also very not-fun when it’s not-fun. Games are particularly not-fun when you’re finishing them, so by law that’s when you will discover a Bug.</p> <p>At that time, the team was working on what was to be one of the first games ever for a brand new game console, called the “Xbox”. As final testing ramped up, QA set up three pre-release Xboxen to run automated tests overnight. If the previous day’s build of the game was still running the next morning, it was a sign of a stable build.</p> <p>Unfortunately, by morning one of the consoles had crashed. Crashes are never good, but this was a particularly bad crash: something running on the graphics card had brought the whole system down. Diagnosing a GPU crash meant hard mode: no debuggers, no stack traces, no “printf” debugging. All you could do is read the code and try things, like an animal.</p> <p>And so began the Bug Hunt. Each day the lead engineer would comb through the evidence they had, develop a hypothesis, and work to eliminate possibilities. Each night, QA would get a “random” crash, without a clear cause. “That’s not possible.” “How could this even happen?” “Could there be a bug in the compiler?” All the greatest hits.</p> <p>Naturally, the game ran perfectly well on the engineer’s machine – for multiple days, even. This was little consolation, as the deadline to print and ship the game loomed large.</p> <p>Luckily, a pattern was soon detected – albeit a strange one. The game was only crashing overnight on <em>one</em> of the three Xboxes. A search for differences ensued. It wasn’t the power cables. It wasn’t the controllers. It wasn’t the order they were burning the DVDs. Bring the Xbox back to your desk – no crash. Put it back – crash. It was something about the very specific setup QA was using.</p> <p>Now, the process of elimination requires eliminating all variables. So eventually, in desperation, the engineer tried one more thing: he shuffled which console sat on what table. Boom.</p> <p>It wasn’t <strong>that Xbox</strong> that crashed, it was any Xbox sitting on <strong>that table</strong> that crashed. In the middle of the night.</p> <p><img style='max-width: 100%' src="http://www.allenpike.com/images/2018/brains.jpg" /></p> <p>Now, sometimes you need to do strange things for science, and this was one of those times. Stoically, the engineer set up a chair, gathered the requisite quantity of Red Bull, and the Bug Hunt became a Bug Watch. He vowed to watch the Xbox run automated testing on that cursed table until he saw the problem with his own two eyes.</p> <p>The night passed by slowly, then quickly, and eventually dawn approached. The game still ran. Infuriatingly, it ran. The sun began to rise.</p> <p>As he started to consider calling it for the night, something interesting finally happened: The Table was struck by a ray of light from the rising sun. Minute by minute, a sunbeam crept across the table towards the Xbox, its warm glow quietly enveloping that black blob of a console.</p> <p>Which promptly crashed.</p> <p>It <em>turns out</em>, early Xbox units had an issue where the graphics card could fault when the console reached a certain temperature. There was nothing you could do about it in software. The hardware issue was escalated, the game was cleared for release, and the Red Bull was swapped out for beer – or let’s be honest, probably whisky. Science one, Bug zero.</p> <hr /> <p>Back at Steamclock, our scrolling performance Bug was more straightforward. After some process of elimination and some teamwork, yesterday we tracked it down to our crash reporting SDK. A recent change in how the Buddybuild SDK interacts with iOS means that Buddybuild’s Instant Replay feature now causes severe performance degradation, which is very visible when scrolling. Feature disabled, problem solved.</p> <p>Next time you hit a Bug that defies all explanation, it’s worth trying to channel the persistent and systematic nature of the Bug Hunters that came before you. Whether the Bug is in your code, a 3rd party library, or the thermal expansion of prototype hardware in the morning sun, the only solution is science. And maybe a little whisky.</p> http://www.allenpike.com/2018/caravan-to-xcoders Caravan to Xcoders 2018-05-01T00:00:10-08:00 Allen Pike http://www.allenpike.com/ <p>In 2013, I decided it was time to start an iOS development meet up. I’d run <a href="http://www.vanjs.com/">VanJS</a> for many years, which was great. There was one thing about running VanJS that was <em>not</em> great though, and that was using Meetup.com.</p> <p><img style='max-width: 100%' src="http://www.allenpike.com/images/2018/meetup.png" /></p> <p>You see, Meetup is optimized for getting small groups assembled and running. A huge proportion of Meetup’s focus is around making it easier for groups to organize, socialize, and maintain their momentum. Meetup continually adds new ways for members of a group to post various things to the group. If your goal is to have as many Meetup™s as possible surviving and paying dues, optimizing your KPIs for optimal synergy, then this is great.</p> <p>This is not as great if your goal is to curate a high-quality speaker series for a group with 3000 members, watched hungrily by dozens of hungry recruiters and other loud individuals.</p> <p>For example, a few years back Meetup added comments to events. They also opted everybody into notifications, as is the style. This probably worked fine for groups with 10 attendees, but at that time we were organizing a large event with 200 attendees. Naturally, Rando Calrissian hops onto the event’s page to say “ill be 5 min late”, which caused Meetup to email 200 people the very important alert that Mr. Calrissian will be 5 minutes late.</p> <p>In another fun encounter, I started getting urgent messages asking things like “I can’t get in, where is everybody?” I briefly panicked. Had I organized an event but <em>forgotten to attend</em>? I checked my Meetup page, and although <em>I</em> hadn’t, somebody else had! Meetup had quietly added the ability for all attendees to “propose” events, and assign them dates and times, <em>and send announcement emails</em>, without necessarily ensuring that there was a venue or staff or anything associated with that event. So a bunch of people showed up at our usual auditorium, for a talk that somebody had offhandedly suggested. Unfortunately, the auditorium costs $500 and needs to be booked weeks in advance, so this resulted in some poor customer sat.</p> <h2 id="an-interesting-problem">An interesting problem</h2> <p>Given all that, I wasn’t keen to start another group on Meetup. And doing some research, chatting with other event organizers, I learned that many organizers weren’t keen on Meetup either. Complaints about the lack of control were plentiful. Multiple Meetup customers told me point blank, “I hate Meetup.”</p> <p>A lot of the complaints centered around the UI. Meetup’s interface is full of feature clutter and bloat. They even have a feature where you can print out a checklist of attendees to check them off as they arrive. Who <em>does</em> that? In part due to this chaos, Meetup’s look and was about a decade behind the times – not the best for events focused on cutting edge design and development.</p> <p>In contrast, I saw that <a href="http://cocoaheadsmtl.com/">CocoaHeads Montréal</a> had a simple custom site that presented Just the Facts: the events, where they were, and how to RSVP. It was a breath of fresh air, and far closer to what I wanted for our new group than could be configured on Meetup.</p> <p>So, <a href="http://www.steamclock.com/">we</a> spent a couple weeks prototyping a custom web app, which we called Caravan, for our new iOS meet up, which we called VanCocoa. We left out all of Meetup’s social features, spam, and list-printing doodads. Instead, we focused on what we saw as our key differentiator: Caravan pages would be simple, attractive, and easy to use. In particular, went to great lengths to not require attendees to create accounts – years <a href="https://marco.org/2018/04/27/overcast42">before it was cool</a>.</p> <p><img style='max-width: 100%' src="http://www.allenpike.com/images/2018/vancocoa.jpg" /></p> <p>That August we hosted <a href="http://vancocoa.caravan.io/events/5">the first VanCocoa</a> using Caravan. Although people liked the event and the site, I immediately started to receive a lot of support email about it. 90% of the issues stemmed from my brilliant idea of not requiring accounts:</p> <ul> <li>“Did I RSVP?”</li> <li>“How do I un-RSVP?”</li> <li>“I accidentally RSVPed twice with two different emails, what do?”</li> <li>“Do I have to re-type my email each time I RSVP?”</li> <li>“I accidentally RSVP’ed to another event using a different email address than I’d used before, and now I’m getting announcements to both emails, can I just merge them somehow, should you maybe just have some kind of account login?”</li> </ul> <p>We were able to fix some of these in a workable way without requiring people to create accounts, but the product design challenges around avoiding logins made it painfully clear why every product in the space requires account creation for attendees. Blech.</p> <p>Meanwhile, I started demoing our Caravan prototype to the Meetup organizers who told me they hated Meetup so much. Each organizer was excited to see our progress, but had a feature request. Which is great! Every person’s feature request was different than every other person’s. Which is not great.</p> <p>One organizer wanted to charge fees. One organizer wanted a message board. Another wanted a <em>poll for pizza preferences?</em></p> <p>My favourite was the Meetup organizer in New York whose venue had some security rules that required them to do paperwork for every event. Before attendees could enter the venue, a security guard needed to check them off a list – a list they generated with the very Meetup feature I’d been using as an example of needless bloat. Well, <em>f</em>.</p> <p>As if that wasn’t bad enough, we were soon offered a <a href="https://vfs.edu/">cool new venue</a> for VanCocoa. It was very nice and well located, but had a catch: they needed us to sign attendees in. From a list. Well, <em>ff</em>.</p> <p>Perhaps even worse, we saw first-hand the power of Meetup’s network effects. Where our VanJS group on Meetup naturally grows substantially every month just by being an active tech event in Vancouver, our Caravan-based group grew ever so slowly, despite having talks of the same caliber. The iOS development community here isn’t quite as big as the web development community, but it’s strong and healthy – something that just wasn’t reflected in the attendance for VanCocoa, despite glowing reviews from attendees.</p> <h2 id="what-do">What do</h2> <p>Given what we’d learned from our prototyping and interviews, we went back to the drawing board and did some design explorations for what a “Next Generation” Caravan could look like. The more we explored and the more we talked to organizers of large Meetup groups though, the less it looked like there was a clear niche for a simple, beautiful Meetup competitor focused on large tech groups. Around that time I saw that CocoaHeads Montréal, the group that inspired Caravan in the first place, started using Meetup themselves. 😕</p> <p>So, we decided to shelve further development on Caravan and focus on <a href="https://www.allenpike.com/2014/podcast-recording/">other</a> <a href="https://www.steamclock.com/blog/2016/11/apple-music-avfoundation/">products</a> <a href="https://www.allenpike.com/2017/two-spies/">we wanted</a> <a href="https://www.steamclock.com/blog/2017/06/bluejay-swift-bluetooth/">to explore</a>. We kept the site running for VanCocoa for some time, but this winter I sent out a survey of the group asking about future directions. Besides positive reviews, a big message that came back was that we should switch to Meetup – even though people liked Caravan better – because more people would find the group that way. Well, who am I to argue?</p> <p>So today, we finally ported VanCocoa to Meetup. We also took the opportunity to rebrand: <strong>next week will be the first ever <a href="https://www.meetup.com/Vancouver-Xcoders/">Vancouver Xcoders</a></strong>. We’ve always had a great relationship with Seattle’s Xcoders group, and it felt right to join forces. If you’re in Vancouver or visiting sometime, and you’re interested in giving a talk about development in the Apple ecosystem, let me know.</p> <p>Of course, Meetup is still bad. Very bad. But there’s nothing like trying to build your own thing to help you appreciate the bad thing you don’t need to maintain. And I’m pretty sure the network effect and new name will bring together more of the Vancouver iOS community, and – perhaps more importantly – a more diverse audience beyond the bubble of folks that knew about VanCocoa.</p> <p>Of course, if you do end up beating Meetup at their own game, please drop me a line.</p> http://www.allenpike.com/2018/canadian-cheese-cartel The Canadian Cheese Cartel 2018-04-02T10:15:00-08:00 Allen Pike http://www.allenpike.com/ <p>Canada has a cheese problem.</p> <p>Back in 1970, dairy farmers convinced the Canadian government to enact something called the “National Milk Marketing Plan”. This regime greatly restricted the production and sale of dairy, creating <a href="http://nationalpost.com/opinion/national-post-editorial-board-its-time-to-end-the-milk-cartel">a cartel</a>. The goal was to increase dairy prices in Canada – because who doesn’t like high prices? However, in the 48 years since the cartel was established, something strange has occurred: we’ve had high prices.</p> <p><img style='max-width: 100%' src="http://www.allenpike.com/images/2018/cheeses.jpg" /></p> <p>How expensive is cheese in Canada, you ask? Well let’s consider pizza, everybody’s favourite cheese delivery mechanism. There’s a big nationwide chain here called Boston Pizza that sells, among other things, pizza. A large pepperoni pizza at Boston Pizza is $30.28.</p> <p>At Pizza Hut in the US, they currently have a deal on where you get a large pepperoni pizza for $7.99. That is less than $30.28 – even after health insurance premiums.</p> <p>Now, would I recommend paying $8 for a Pizza Hut pizza? No. Would I recommend paying $30 for a Boston Pizza Pizza? Also no. Should we be eating pizza in the first place? Well, yes, pizza is delicious. As is cheese – but it’s slightly <em>less</em> delicious in Canada, because it’s god damned expensive.</p> <p>It’s so expensive, in fact, that pizza places estimate their daily food cost with a single number: Cheese Cost. My brother used to work at a pizza place here, and the manager only wanted to know two numbers at the end of the day: revenue, and Cheese Cost. Meat? A rounding error. Vegetables? Irrelevant. Show Me The Cheese.</p> <p>With these cheese prices, you would expect a healthy import market. Except not, because dairy tariffs in Canada are <a href="http://www.international.gc.ca/controls-controles/prod/agri/dairy-laitiers/notices-avis/890.aspx?lang=eng">245.5%</a>. That is to say, if you want to import a $4.99 block of sharp <a href="https://www.tillamook.com/products/cheese.html">Tillamook Cheddar</a>, it will cost you $17.27 after import taxes. Surprisingly, there ain’t a lot of sharp Tillamook Cheddar here.</p> <p>Still, the cheese must flow. A few years back, some clever pizzeria owners noticed a loophole in Canadian customs laws. While there is an obscene tax on importing cheese, there was no such tariff on importing “food preparation items”. In 2011, the border lines swelled with restauranteurs filling their cars with Costco pizza kits. These kits contained highly sought-after mozzarella cheese, conveniently packaged with pepperoni and pizza crust. Once these bootleg kits got into Canada, the cheese went right onto freshly made pizzas, and the pepperoni and crust went right into the trash. Restaurant margins 1, a functional system of trade 0.</p> <p>While that loophole has <a href="http://www.betterfarming.com/online-news/feds-close-pizza-cheese-import-loophole-54153">since been closed</a>, our whackadoodle dairy controls continue to cause grief of an international scale. Canada’s largest trading partner, for example, was recently taken over by a feral animal of some sort, resulting in a dispute over free trade. While most of the “reasons” for the trade dispute amount to flaily nonsense, our dairy cartel is rightly a point of contention.</p> <p>America, Europe, and the world all want to offer Canadians great cheese at reasonable prices. Yet 12,965 dairy farmers are holding our nation hostage. They have created a world where a milk production quota costs $35,000 <em>per cow</em>, and a restaurant can charge $30 for mediocre pizza and stay in business. They have used lobbying and <a href="https://www.youtube.com/watch?v=MvH7Fn2fjOg">cheesy cheese ads</a> to lull us into submission, and as a result, have reduced the availability of cheese in this otherwise fine country.</p> <p>This cartel causes there to be less cheese, and that is a crime.</p> <p>Or at least, it should be.</p> <hr /> <p><em>Update, April 3</em></p> <p>Folks have been writing in to inform me that comparing Boston Pizza’s regular prices with Pizza Hut’s sale prices is not an equivalent comparison – which is true, since Pizza Hut is garbage and Boston Pizza thinks it’s people. So as my apology to Canadian dairy farmers, here are some additional facts people have sent in:</p> <ul> <li>Milk in Canada costs 2x what milk across the border does, but due to labour costs the difference in bulk cheese is usually less dramatic.</li> <li>USD and CAD are different currencies which fluctuate over time, increasing or decreasing the on-paper price of goods and services.</li> <li>The Canadian dairy cartel generates substantially less profit than the Mexican drug cartel does.</li> <li>Pizza Hut also has sale pricing in Canada.</li> <li>Boston Pizza is attempting to expand to the United States under the brand Boston’s, where they also charge too much for pizza.</li> <li>Whiz is not one of the styles of cheese recognized by the Canadian Dairy Commission.</li> </ul> <p><em>Update 2, April 13</em></p> <p>I have received many additional messages informing me that sarcastically comparing the price of highly divergent pizza vendors does not constitute a scientific evaluation of the international cheese trade. Thank you for your heartfelt messages. Unfortunately I left my Cheese Economics diploma in the car, but if you’d like a more reputable source for the claim I actually make (that the Canadian cheese market is silly), I invite you to read <a href="https://www.theglobeandmail.com/opinion/editorials/globe-editorial-dear-donald-trump-please-milk-canadas-sacred-dairy-cow/article34785387/">this Globe editorial</a> on the cartel, or <a href="http://www.cbc.ca/news/canada/hamilton/niagara-cheese-smuggling-cop-resigns-1.4209945">this CBC article</a> on a police officer who was arrested for smuggling cheese – for use in pizza.</p>