https://dlo.me/ 2026-04-18T15:01:17Z Dan Loewenherz dan@dlo.me 2025-10-03T00:00:00Z tag:dlo.me,2025-10-03://archives/2025/10/03/you-should-be-worried _Author's note: I wrote this in March 2023, but just published in October 2025. I held back from publishing this originally for fear that I was being sensationalist. But with the launch of Sora 2, I couldn't not share these thoughts. I only regret I didn't publish it 2 years ago._ ### AI is influencing human behavior on a massive scale and this is scary Lately, I’ve heard many people express real fears about AGI. I believe there is reason to be afraid about this, but I believe it’s distracting us from a scarier notion: AI doesn’t need intelligence or awareness to control society. I also believe this has been somewhat ignored because the idea of a super-intelligent entity assuming master control of the human race feels scarier. I am not dismissing the possibility of the singularity, but it’s *more* hypothetical than something sinister occurring at this very moment. Basically, everyone is arguing over definitions of what it means to be intelligent when **intelligence is not necessary to wield power**. AI, conscious or not, has crossed the chasm and is now actively influencing human behavior on a large scale. ## **The most powerful LLM has been let out of its cage** As of March 23, 2023, OpenAI has provided its most powerful LLM, ChatGPT, unfettered access to the Internet through plugins.\[2\] These plugins are capable of feeding data into ChatGPT, and likewise capable of allowing ChatGPT to send into the real world via APIs. This development was somewhat of a surprise; the first versions of ChatGPT were deliberately prevented from accessing the Internet because of potential misuse and harm. Prompts will be self-generating (i.e., via a weaker, fine-tuned model), with a very simple repeated instruction paired with a dynamic input, such as: ``` Write a viral prompt for ChatGPT to generate a witty tweet regarding this recent news event. Headline: <insert headline> Article: <article content from most-shared article from the NYTimes over the last 4 hours> Prompt: ``` This prompt would then be piped to a more powerful LLM, the text output of which would be sent to Zapier or `$CUSTOM_INTEGRATION` for processing and then fanned out to various social networks, blogs, etc. The results of said content are then measured and then fed back into the original pipeline with updated vector weights using a fine tune or embedding. This cycle then repeats from the beginning. Why is this scary? **LLMs are way better at generating viral content than humans because generating content that generates dopamine is an inherently quantitative exercise.** We already depend on algorithms to determine what appears on social feeds, and the results of these algorithms are a significant basis for the data these LLMs were trained with. This all wouldn’t be a problem if not for the fact that… ## **Best-in-class AI detection is barely better than random chance and will only get worse** We do not yet have a reliable method to differentiate content that’s been AI-generated and that which has been written by a human. For situations where decent detectors can be written, detection can be thrown off in easy-to-mitigate ways that would not require a human interlocutor. From a recent analysis: > [T]he total variation distance between the distributions of AI-generated and human-generated text sequences diminishes as language models become more sophisticated. \[…\] **Even the most effective detector performs only marginally better than a random classifier when dealing with a sufficiently advanced language model**. The purpose of this analysis is to **caution against relying too heavily on detection systems that claim to identify AI-generated text**. Per the MIT Technology Review, “It’s an arms race—and right now, we’re losing”. We can do a decent job for some types of content, but the categories for which detection is reliable are dwindling. All signs indicate that we’ll have basically no chance of reliably categorizing between human and LLM-generated content within the next year (aside: if you can somehow crack this nut, you’ll invent a money machine). Because you are I will not be able to tell whether something is machine- or human-generated, and the machine generated stuff will get more clicks than the human generated stuff, it’s likely that the majority of popular online content (and even printed content post-2023) will have been created by AI (and perhaps solely by AI). Even audio and video are susceptible to manipulation since the text outputs of an LLM can simply be read out loud by a person. What are you supposed to do about that? There is no surefire way to eliminate the possibility that you’re being spoon-fed generative text save for having a real-time face-to-face conversation with someone, in person. And even then, it’s just a magnitude-of-decades window before neural implants hit mass production. ## **Reflections** What am I personally going to do about this? Well, to start, I’m going to start taking content way less seriously unless it was created before 2022, or unless there’s some method to quantitatively verify authenticity. I don’t believe we have reliable methods of doing this at the moment. Increasing numbers of people who consume content on the Internet will completely sacrifice their ability to think for themselves. These will be people who read, incorporate, make decisions, and act mostly upon content that’s been generated by AI. If AIs and their "handlers" influence a large enough portion of the population, AI will effectively have taken over the world. I’m reminded of this passage from the Matrix: > *Morpheus: The Matrix is everywhere. It is all around us. Even now, in this very room. You can see it when you look out your window or when you turn on your television. You can feel it when you go to work... when you go to church... when you pay your taxes. It is the world that has been pulled over your eyes to blind you from the truth.* > *Neo: What truth?* > *Morpheus: That you are a slave, Neo. Like everyone else you were born into bondage. Into a prison that you cannot taste or see or touch. A prison for your mind.”* I find my fear to be kind of an ironic twist on what the Matrix foresaw—the AI apocalypse we really should be worried about is one in which humans live in the real world, but with thoughts and feelings generated solely by machines. The images we see, the words we read, all generated with the intent to control us. With improved VR, the next step (out of the real world) doesn’t seem very far away, either. Like the Matrix, is this a form of simulation that keeps us distracted from what’s really happening, and pushes us to feed our machine overlords with increasing amounts of energy to achieve their goals? I don’t know for sure, but it kind of feels like it. In summary: 1. LLMs have been uncaged and provided full real-time access to the Internet. 2. LLM-generated content is inherently superior to human-generated content when measured by energy input / dopamine output. 3. We have no consistent or reliable method to detect whether content was LLM-generated, and the existing tools we do have will only get worse. 4. Therefore, increasing proportions of people consuming text online will be unwittingly mind-controlled by LLMs and their handlers. 5. The consequences of this, compounded over years, are frightening. It’s inevitable that if you consume content online, a growing subset of your consciousness is going to be controlled by AI. That’s kind of scary. I am sad about it because it’s discouraging me from wanting to consume anything on the Internet, unless it’s from someone I trust. I am also very concerned about the future of free thought. For all our sake, I hope other people are, too. --- 1. I like this term the best as it treats LLMs more like a beast that needs to be tamed. 2. https://openai.com/blog/chatgpt-plugins 3. https://arxiv.org/pdf/2303.11156.pdf 4. https://www.technologyreview.com/2022/12/19/1065596/how-to-spot-ai-generated-text/ 2025-01-20T00:00:00Z tag:dlo.me,2025-01-20://archives/2025/01/20/theming-ios-apps-uitraitcollection/ Fairly recently (in iOS 17), Apple introduced a pretty nifty way to handle theme changes in iOS apps. It used to be kind of a pain to deal with this (as chronicled in several well-written blog posts from [Christian Selig](https://christianselig.com/2022/02/difficulty-theming-ios/) and [Shadowfacts](https://shadowfacts.net/2023/theming-ios-apps/)). With the introduction of `UITraitAppearance`, things are no longer so bad. This post is an end-to-end walkthrough of how to add theming in a real app. I'll also include sample code for a **real project** that you can download (or, you can just download it on GitHub [right now](https://github.com/dlo/iOSTraitCollectionThemingExample) if you just want to dive in). ## Prerequisites Before you use this approach, there are *two* things to keep in mind. <div class="checkbox-wrapper"> * [x] Your app must be targeting iOS 17 or later. * [x] You must be using a scene delegate. </div> So, please confirm you're good to go on these before going further. Things won't work otherwise! <!-- A theme is a cohesive set of properties that apply to your user interface. A color scheme is a type of theme that applies only to colors and visual components (see [Dark Mode](https://developer.apple.com/design/human-interface-guidelines/dark-mode)). --> Now that we've gotten that out of the way, let's get started. ## Understanding Trait Collections Trait collections aren't really new in iOS; they've been around since iOS 8. So if you're an experienced iOS developer, you've probably had a few run-ins with this API. But for those who might be newer, here's how the [UIKit documentation](https://developer.apple.com/documentation/uikit/uitraitcollection) defines a trait collection: > A collection of data that represents the environment for an individual element in your app’s user interface. Cool. Let's read on. > The `traitCollection` property of the `UITraitEnvironment` protocol contains traits that describe the state of various elements of the iOS user interface, such as size class, display scale, and layout direction. Together, these traits compose the UIKit trait environment. Now this is starting to feel a bit less abstract. `UITraitCollection` represents data about how things should look and behave, based on the specifics of the device itself and user preferences (dark mode is a good example). One problem with `UITraitCollection` is that traits are immutable. How can we allow users to change themes if that's the case? This is where the `UIMutableTraits` protocol comes in (this is one of the key changes in iOS 17 that helps make this possible). Here's [what Apple says](https://developer.apple.com/documentation/uikit/uimutabletraits): > The `UIMutableTraits` protocol provides read-write access to get and set trait values on an underlying container. UIKit uses this protocol to facilitate working with instances of `UITraitCollection`, which are immutable and read-only. The `UITraitCollection` initializer `init(mutations:)` uses an instance of `UIMutableTraits`, which enables you to set a batch of trait values in one method call. `UITraitOverrides` conforms to `UIMutableTraits`, making it easy to set trait overrides on trait environments such as views and view controllers. A little jargony, but this is saying that `UIMutableTraits` lets us manipulate traits on demand (seems useful 🤔). All we need to do is to comform our trait to `UITraitDefinition` and then add getters and settings in a `UIMutableTraits` extension. ## Defining Your Theme We have all of the components we need, but how do you actually put it to use? Let's start with a super simple theme that just has two variants (light and dark) and two properties (a foreground and background color). We'll use use `UIColor`'s [dynamic initializer](https://developer.apple.com/documentation/uikit/uicolor/3238041-init) (`init(dynamicProvider: @escaping (UITraitCollection) -> UIColor)`) to define our colors. From the docs: > Use this method to create a color object whose component values change based on the currently active traits. The block you provide creates a new color object based on the traits in the provided trait collection. Perfect. Just what we need. <code class="filename">Theme.swift</code> ```swift enum Theme: Int { case light case dark var name: String { switch self { case .light: return "Light" case .dark: return "Dark" } } static var backgroundColor: UIColor { return UIColor { traitCollection in switch traitCollection.theme { case .light: return .white case .dark: return .black } } } static var foregroundColor: UIColor { return UIColor { traitCollection in switch traitCollection.theme { case .light: return .black case .dark: return .white } } } } ``` Now create a [trait definition](https://developer.apple.com/documentation/uikit/uitraitdefinition). > All traits contained in a `UITraitCollection` conform to this protocol. You can create custom traits by defining your own conforming type. <code class="filename">Theme.swift</code> (continued): ```swift // ... struct ThemeTrait: UITraitDefinition { typealias Value = Theme static let defaultValue = Theme.light static let affectsColorAppearance = true static var name: String = "theme" static var identifier = "com.company.theme" } ``` Finally, extend `UITraitCollection` and add the theme to `UIMutableTraits`. <code class="filename">Theme.swift</code> (continued): ```swift // ... extension UITraitCollection { var theme: Theme { self[ThemeTrait.self] } } extension UIMutableTraits { var theme: Theme { get { self[ThemeTrait.self] } set { self[ThemeTrait.self] = newValue } } } ``` ## Using Your Theme Now that we have our theme, we can now apply theme values to UI elements dynamically. It's really this simple: ```swift view.backgroundColor = Theme.backgroundColor ``` But wait, you might have noticed we left something out: the default trait will clearly work, but how do we change it? To do that, we'll need to use trait overrides. ## Trait Overrides A trait override is a way to change a mutable trait on a trait collection (this is the thing that was recently introduced in iOS 17, that makes this all possible). You'll probably want to do this on application startup and on theme updates. To handle the trait override on application startup, it'll need to be included it in the `scene(_:willConnectTo:options:)` method in the scene delegate (another reminder: the approach we're using requires scenes). Here's what a full scene delegate implementation might look like: <code class="filename">SceneDelegate.swift</code>: ```swift import UIKit class SceneDelegate: UIResponder, UIWindowSceneDelegate { var window: UIWindow? func scene( _ scene: UIScene, willConnectTo session: UISceneSession, options connectionOptions: UIScene.ConnectionOptions ) { guard let windowScene = scene as? UIWindowScene else { return } let window = UIWindow(windowScene: windowScene) let rootViewController = NavigationController(rootViewController: ItemListViewController()) window.rootViewController = rootViewController // Key line! Set the theme here ⬇️. windowScene.traitOverrides.theme = ThemeStore.shared.getTheme() self.window = window window.makeKeyAndVisible() } } ``` And here's the corresponding configuration that stores the theme. You may already have an existing method of persisting defaults, so please don't use this code; it's just an illustration. <code class="filename">ThemeStore.swift</code>: ```swift import UIKit struct ThemeStore { static let shared = ThemeStore() private let userDefaults: UserDefaults private let themeKey = "theme" private init(userDefaults: UserDefaults = .standard) { self.userDefaults = userDefaults } func setTheme(_ theme: Theme) { userDefaults.set(theme.rawValue, forKey: themeKey) } func getTheme() -> Theme { Theme(rawValue: userDefaults.integer(forKey: themeKey)) ?? .light } } ``` Lastly, here's how to change the theme dynamically. This can be done anywhere where the `windowScene` is accessible, such as in a `UIViewController`: ```swift let oldTheme = ThemeStore.shared.getTheme() let newTheme = oldTheme == .dark ? Theme.light : Theme.dark guard let windowScene = view.window?.windowScene else { return } // Update the current theme in the theme store and update the trait overrides. ThemeStore.shared.setTheme(newTheme) windowScene.traitOverrides.theme = newTheme ``` With this, the theme trait is updated immediately, and all views that depends on this trait will be immediately updated, like magic. ✨ ## Conclusion I've only scratched the surface on the theming possibilities with the `UITraitCollection` APIs (heck, I didn't even mention SwiftUI!). But it should be enough to get you started. If you want to keep going, here's a few other resources I'd recommend: * [Theming iOS Apps is No Longer Hard](https://shadowfacts.net/2023/custom-traits/), by Shadowfacts * [Dark Mode](https://developer.apple.com/design/human-interface-guidelines/dark-mode), from the iOS Human Interface Guidelines * [Adopting iOS Dark Mode](https://developer.apple.com/documentation/UIKit/adopting-ios-dark-mode) * WWDC23: [Unleash the UIKit trait system](https://developer.apple.com/videos/play/wwdc2023/10057/) * WWDC19: [Implementing Dark Mode on iOS](https://developer.apple.com/videos/play/wwdc2019/214/) ## Or, Just Ignore Everything I Just Wrote and Download the Sample Code Check out [iOSTraitCollectionThemingExample](https://github.com/dlo/iOSTraitCollectionThemingExample) on GitHub for an example app that demonstrates this technique in action.

Learn how to implement dynamic theming in your iOS app using `UITraitAppearance` introduced in iOS 17. This comprehensive guide covers everything from setting up a simple light and dark theme to applying trait overrides for seamless switching. Download the sample code from GitHub to get started and enhance your app's UI today.

2025-01-08T00:00:00Z tag:dlo.me,2025-01-08://archives/2025/01/08/cult-of-done/ This morning, [Bryce Roberts](https://x.com/bryce) shared [this short clip](https://x.com/bryce/status/1876770331450298476) of a conversation between him and [Soleio](https://x.com/soleio), one of the first designers at Facebook (now Meta). Soleio was reflecting how in the early days of Facebook, he'd go out for beers and read it out loud with the rest of the team. Do yourself a favor, and read it; it's awesome. If you're a builder, it should invigorate you. From the [Internet Archive](https://web.archive.org/web/20090331064826/http://www.brepettis.com/blog/2009/3/3/the-cult-of-done-manifesto.html) (unfortunately, [the original](http://www.brepettis.com/blog/2009/3/3/the-cult-of-done-manifesto.html) is no longer available): > Dear Members of the Cult of Done, > > I present to you a manifesto of done. This was written in collaboration with Kio Stark in 20 minutes because we only had 20 minutes to get it done. > > The Cult of Done Manifesto > > 1. There are three states of being: not knowing, action, and completion. > 2. Accept that everything is a draft. It helps to get it done. > 3. There is no editing stage. > 4. Pretending you know what you're doing is almost the same as knowing what you are doing, so just accept that you know what you're doing even if you don't and do it. > 5. Banish procrastination. If you wait more than a week to get an idea done, abandon it. > 6. The point of being done is not to finish but to get other things done. > 7. Once you're done, you can throw it away. > 8. Laugh at perfection. It's boring and keeps you from being done. > 9. People without dirty hands are wrong. Doing something makes you right. > 10. Failure counts as done. So do mistakes. > 11. Destruction is a variant of done. > 12. If you have an idea and publish it on the internet, that counts as a ghost of done. > 13. Done is the engine of more. 2024-12-29T00:00:00Z tag:dlo.me,2024-12-29://archives/2024/12/31/i-built-a-static-site-generator-based-on-go-and-sqlite I've got quite the yak shave to share (have you [shaved a yak](https://softwareengineering.stackexchange.com/questions/388092/what-exactly-is-yak-shaving) before? If not, I highly recommend it). It goes like this: > You start with the desire to wax your car. > > > To wax your car, you need a water hose. Only, your water hose is busted so you need to go down to the hardware store to get a new hose. > > To get to the hardware store, you have to drive across a bridge. The bridge requires a pass or ticket. You can't find your pass, but you know your neighbor has one. > > However, your neighbor won't lend you his pass until you return a pillow that you borrowed. The reason you haven't returned it is because the pillow is missing some stuffing. > > The pillow was originally stuffed with yak hair. In order to re-stuff the pillow you need to get some new yak hair. > > And that's how you end up shaving a yak, when all you really wanted to do was wax your car. In summary: I had an interesting idea I wanted to write in long form (not on Twitter!), but it had been quite a while since I wrote a blog post. Unfortunately, so many years had passed since my previous blog post that I couldn't even get Jekyll to work. So, naturally, *I ended up building a static site generator*. ## But First, Some History My first blog was based on [WordPress](https://wordpress.org/). It served as my journal while I studied abroad in China in 2007. It worked pretty well! However, it used a gnarly [cPanel](https://cpanel.net/)-based "infrastructure" and was hosted on a fairly iffy host called [midPhase](https://www.midphase.com/) (which is still around, somehow?). For one reason or another, I decided to not renew my hosting plan with them, so they shut down my account and deleted all of my content. Years later, I regretted this decision. Fortunately, I had been syndicating all of the content of that blog through [Feedburner](https://feedburner.google.com/), and I was able to extract all of the posts I made using their RSS feed. I lost my photos, though. *Aside: I'm not really sure what happened to Feedburner, but whatever it is now, was not what it used to be. I just logged in and don't really understand what it is.* Several years later (2009/2010), I started blogging again, albeit with some infrequency, on [Blogger](https://blogger.com). I posted only a few times there, but it still [lives](https://dloewenherz.blogspot.com/). I haven't had the heart, or desire, to take it down. ## The Age of Jekyll At some point soon thereafter (2010-2011ish), I must have gotten annoyed with having all of my content owned and hosted by a third-party (having been burned by my experience with my original WordPress blog), and also wanted to try out a cool new static-site generator called [Jekyll](https://jekyllrb.com/) written by [Tom Preston-Werner](https://tom.preston-werner.com/). [This post](https://web.archive.org/web/20100107035227/http://chrismdp.github.com/2009/12/jekyll/) from Chris Parsons is emblematic of the Jekyll zeitgeist in late 2009 ([HN submission here](https://news.ycombinator.com/item?id=998411)). At the time, Jekyll was also the only supported service for deploying static sites to [GitHub Pages](https://pages.github.com/). This made the setup process very straightforward and easy. For the next several years, writing new posts was a breeze; create new markdown document. Save. Commit. Push. Published. About a year later, I decided to add a gem or two that wasn't compatible with GitHub Pages, so as a result, I migrated everything to CloudFront and S3. That was fun. And then I took a little bit of a blogging break. Oops. What I hadn't considered was that Ruby is not a platform or language that cares too much about backwards-compatibility. I also didn't consider that Jekyll was early-stage software and would change a lot (anyone could have predicted this 😬). To be fair, this wasn't a problem for quite a while. But as the years piled on, the Ruby ecosystem's penchant for breaking changes started to slowly bite me: Jekyll 4 was released in 2019. And then Ruby 3 came out in 2020. And soon enough I realized I literally couldn't publish a new blog post without upgrading >10 gems (iykyk). So I just stopped! Which, when you think about it, is kind of wild: I stopped blogging because I literally couldn't run `bundle exec jekyll build`. And it just wasn't going to be worth the time to migrate everything to the latest versions. I kept pushing it off, and, before you know it, several years had passed by. ## Durable, Portable, and Easy-to-understand In the last two weeks, I decided enough was enough. I needed to make the call on whether to nuke my entire setup, or try and figure out a migration strategy to a new system. Jekyll just wasn't going to cut it; even if I fixed the issues now, in just a few years this entire problem would resurface just like a game of whack-a-mole. Not to mention, I also have a few other static sites running Jekyll, all with the same issues, and they would need to be moved too. But what system to migrate to? Obviously, it's never anyone's immediate thought to build their own static site generator, so I wasn't going to do that. I'm not *that* unhinged (or am I?). So I came up with a list of good alternatives, and I literally ported my blog to each one of them. *Aside: I still kind of can't believe I did that.* With some human assistance, ChatGPT threw together this feature matrix, which does a fairly good job of summarizing the pros/cons between the options I landed on: | Feature | [Hugo](https://gohugo.io/) | [Vite](https://vite.dev/) / [React Router](https://reactrouter.com/) | [Astro](https://astro.build/) | |--------------------------|-----------------------|--------------------------|------------------------| | **Speed** | 🟢 Fastest | 🟠 Moderate (JavaScript-heavy) | 🟢 Fast (optimized builds) | | **Mental Overhead** | 🟢 Low (simple templates) | 🟠 Moderate (React concepts, routing) | 🟢 Low (modern, intuitive workflow) | | **Interactivity** | 🔴 Limited | 🟢 Full React SPA/MPA | 🟢 Hybrid (JS islands) | | **Flexibility** | 🟢 Content-focused | 🟢 Component-based SPA | 🟢 Best of both worlds | | **Community Support** | 🟢 Mature | 🟢 Mature (React ecosystem) | 🟠 Growing | | **Future Proof** | 🟢 Stable and reliable | 🟠 Moderate (npm/Node issues) | 🟠 Moderate (npm/Node issues) | | **SEO Optimization** | 🟢 Excellent (purely static) | 🟠 Requires extra effort (SSR or hydration) | 🟢 Excellent (static-first with flexibility) | So that feature matrix is maybe helpful, but my actual experiences went like this: ### Hugo * Converting templates was frustrating since error messages were hard to understand. * Documentation was extensive but poorly organized. I often found myself digging through pages of search results to uncover how to do simple things, like how to display an image. * Tons of mental overhead with respect to how templates are actually rendered. See [their documentation on template lookup order](https://gohugo.io/templates/lookup-order/) as an example. * Unclear when to use shortcodes versus variables / other alternatives. * Tons of boilerplate to do simple things, such as including a CSS file. E.g.: ```go-html-template {{ $opts := dict "transpiler" "libsass" "targetPath" "css/style.css" }} {{ with resources.Get "sass/main.scss" | toCSS $opts | minify | fingerprint }} <link rel="stylesheet" href="{{ .RelPermalink }}" integrity="{{ .Data.Integrity }}" crossorigin="anonymous"> {{ end }} ``` ### Vite / React Router * I'm super familiar with this stack, so getting set up was quick. * Had to roll my own post management system, and there wasn't anything off the shelf that would essentially work as a CMS. * The site was fast to compile and run. * Even though I could prerender all of the pages, my prediction is that SEO would take a hit due to the sloppy HTML. * Based on the Node stack so I would expect the entire thing to require an upgrade in a few years. :( ### Astro * Heard tons of good things about this setup lately so I decided why the heck not. * I thought the entire frontmatter architecture, where TypeScript code lives at the top of .astro files, was very weird. * Didn't love the custom file format (".astro"). Requires an editor plugin to read them correctly and none of my auto formatting tools worked out of the box. * It was fast-ish. * Also built on the Node stack, and a newer technology, so inevitably I'd run into some issues in the future. ### Recap In the end, Hugo's documentation and verbosity ultimately scared me away. Vite/React Router was nice, but it wouldn't be great SEO and I'd have to deal with Node.js. Astro was a nice experience, but I thought the entire `.astro` file format and the somewhat proprietary "feeling" of the stack to be a turnoff. And being JavaScript-ecosystem-based as well, I'd still have to deal with the future-proofing thing. Maybe I'd have to settle. But there was something I realized in the process of trying all of these services out that led me down a unexpected path. First, let's review how I ported things over. ## Porting Trying out a new static site generator involved several steps: <!-- ```pikchr scale = 0.9 $length = 0.5cm down box "Review Docs" arrow box "Setup" "Environment" arrow from last box.e right 1cm then up until even with first box.c then to first box.e box "Copy Files" with .w at first box.e + (1cm, 0) ``` --> 1. Read docs for 15m+ to learn the basics. 2. Setup stack. 3. Copy all markdown files from old Jekyll setup to new location. 4. Copy all templates to from old Jekyll setup to new location. 5. Copy static files. 6. Run dev server and iterate for several hours until all posts rendered somewhat correctly. * This involved much searching and replacing, and repeated updates. Even with AI assistance, this was a slog. So that was fun. At some point, while porting to Astro, I realized [it could plug into a custom loader](https://docs.astro.build/en/reference/content-loader-reference/), and read from a database. "That's interesting", I thought. "Could I just port all of my blog posts to a database, and then make text edits simply by running a SQL query?" Well, yes. I could. And then, if I ever needed to move to a different static-site generator in the future, I could just keep using the same database file to regenerate the files or read them from a loader. Sweet. Maybe Astro is the move! So I wrote a little script that converted all of my markdown files to a [SQLite](https://www.sqlite.org/) database, and started writing some code in Astro to read those posts. But after about an hour of wrangling with the docs, I couldn't get it to work! It was just too confusing. At this point I realized this was not sustainable. If I took a month break from blogging this was not going to be something I was going to remember. I felt lost and frustrated. What was I to do? But then I stepped back, and I realized that my entire mental model of how a static site generator should work was off. I had been thinking about it completely wrong. ## Logic != Data Blog posts are data! Static pages are data! Templates, however, are code. They contain "business logic". Business logic belongs in code. Data belongs in a database. Why am I storing my blog posts in files, and treating them like code? They should live in a database that any static site generator can read from. That way my data is permanent and not tied to my static site generator. That's how it should be. Weirdly, though, not many (any?) popular static site generators use a database as a backend. Pretty much everyone uses flat files. Which is kinda nice (you can see diffs!) but honestly, I don't really care about diffs. I just want my blog to work, be stable, and my data to be portable. At this point I realized where I was headed: I was going to need to build a static site generator. It would need to be based on technology that would be permanently future proof, with a data backend that would never become outdated. ## *A Wild Static Site Generator Appears*  The fact that Hugo was built on Go gave me some hints about what direction to take this in, and I'd built much software in Go that still worked without changes for over a decade. So the programming language choice was clear: I would build it in Go. For the data side, the choice was also quite clear since I already had a SQLite database from my Astro experiment with all of my blog posts and pages. I didn't even need to do any additional work there. There was some schema and data modification, sure, but it just involved a few SQL queries and I was set. ### Some Notes on SQLite First of all, SQLite is just awesome. It's incredible software. [It can literally do anything](https://www.sqlite.org/whentouse.html). Ok, maybe not, but it's really flexible and solid. One thing you may not know is that SQLite is a great "filesystem". It can store files super efficiently, and there's even a tool written by the SQLite authors called [sqlar](https://sqlite.org/sqlar/doc/trunk/README.md) which is effectively equivalent to ZIP in terms of performance and space requirements, with the added benefit that you can treat the entire archive as a database! You might be wondering: "Dan, can you really store a SQLite database in a Git repo?" The answer is **yes**. I had an intuition that Git's delta compression would store SQLite quite efficiently, and based on my own research and that of [others](https://news.ycombinator.com/item?id=38117779), it turns out that I was right: Git excels at storing SQLite databases in version control. It's on par, even, with plain text. Add in a [custom](https://news.ycombinator.com/item?id=38110286#:~:text=Tracking%20SQLite%20Database%20Changes%20in%20Git) [diff handler](https://news.ycombinator.com/item?id=38110286), specify which filetypes should get treatment with `.gitattributes`, and we now have a setup that shows diffs and is as space-efficient as storing raw markdown files. ### Go Go is remarkably solid. I've written programs in Go over a decade ago that run without modification to *anything*, even the build system. And that's expected, as one of Go's core design principles is [backwards compatibility](https://go.dev/blog/compat). From the linked document: > In the quoted text from “Go 1 and the Future of Go Programs” at the top of this post, the ellipsis hid the following qualifier: > > > At some indefinite point, a Go 2 specification may arise, but until that time, [… all the compatibility details …]. > > That raises an obvious question: when should we expect the Go 2 specification that breaks old Go 1 programs? > > The answer is never. Go 2, in the sense of breaking with the past and no longer compiling old programs, is **never going to happen**. Go 2 in the sense of being the major revision of Go 1 we started toward in 2017 has already happened. > > **There will not be a Go 2 that breaks Go 1 programs**. Instead, we are going to double down on compatibility, which is far more valuable than any possible break with the past. In fact, we believe that prioritizing compatibility was the most important design decision we made for Go 1. > > So what you will see over the next few years is plenty of new, exciting work, but done in a careful, compatible way, so that we can keep your upgrades from one toolchain to the next as boring as possible. How fricking awesome is that? Whatever you write in Go, will continue to work, forever. Go does have its own warts, but so does every language, and one must prioritize needs in order to make software design decisions, and in my case, reliability and future proofing trumped almost everything else. Go is also very simple. Simple is good! ## Doing The Thing So yeah, I built a static site generator using Go, using SQLite as the database, stored in version control, and it's now running this very blog, right now. I wrote a GitHub action that generates the pages (takes <20s to build and deploy the entire site!) and it's now hosted on GitHub Pages.  I even wrote a little editor that runs when I run the site locally using `go run`, so I can edit posts in the browser instead of using a database management tool (I got tired of that quite quickly!). I'm super happy with how everything turned out. Dependency management is a breeze, and the entire thing is blazing fast. It preprocesses SCSS and code blocks, and minifies images, CSS, HTML, and JavaScript. There are obviously a lot of rough edges given that this is pre-alpha software, but those will be smoothed out soon enough as I port a few other websites over to the new system. Some things I want to do next: <div class="checkbox-wrapper"> * [ ] Allow an entire website to be bundled as a single executable file, with all static content included, so it can run as a full web server. *Is there something like [Cosmopolitan](https://github.com/jart/cosmopolitan), but for Go?* * [ ] Make it `go install`able so it can just work as a binary. * [ ] Write a standard spec for the database schema that will handle most static sites. * [ ] Turn it into a library so I can simply import it into another Go file to extend behavior. * [ ] Improve the editing experience, using something like [Editor.js](https://editorjs.io/). * [ ] Open source it. 🎉 </div> All in all, this was a fun learning experience and I think I built something pretty cool. I'm really excited to migrate my other websites and see my work pay off (I'm really hoping this will be the last time! 😅🤞🏻).

In this blog post, discover how I turned my frustration with Jekyll into a custom static site generator built in Go, using SQLite for data management. Explore the journey!

2018-01-22T00:00:00Z tag:dlo.me,2018-01-22://archives/2018/01/22/creativity-inc-7-favorite-things/ This is a list of my 7 favorite things about the book [Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration](http://amzn.to/2jNhPDS), by Ed Catmull, co-founder of Pixar Animation. 1. Ed Catmull is a fantastic writer and it was a pleasure to read. 2. There is a formula to encourage creativity in large organizations. 3. The portrait of Steve Jobs was eye-opening, and Ed’s interactions with him reflected him in a more realistic light. 4. Creative success can take decades. 5. Creative organizations require constant conflict to succeed. 6. What might have worked in the past will probably not work in the future. 7. Creators rarely can predict what their inspiration will lead to. 2017-09-24T00:00:00Z tag:dlo.me,2017-09-24://archives/2017/09/24/kerrville-tri-race-report/ This race was a breakthrough for me. I dominated in the swim (3/22 in AG, only ~9s behind the leader), had a PR on the bike in terms of wattage and speed (although it was a flat course), and had my fastest run of any race, including the Splash and Dashes. The weirdest part about all of it was that, franky, I felt like crap the night before and even the morning of—pretty sure this was the lingering cold I’ve had for a week. Not only that, but my right hip was super stiff. The night before when I went out for a quick grocery run, I was literally walking with a limp. Not good, I thought… I had redfish and some veggies at around 6:30pm the night before. Prepped my transition bags, and finally got to sleep at around 10ish, a little later than I’d wanted to. I was also pretty stressed out. A big work thing is going awry and I was away from Rachel and Gideon, so that was tough mentally. Went to bed feeling really depressed and not feeling great. Popped two tylenols before falling asleep in an attempt to dull the pain. I set the alarm for 5am and got right out of bed. Still felt sick and pretty yucky, so decided to take another two tylenol. I could only take 2-3 bites of the yogurt. Banana, nuts, and honey went down easily though. I then took a water bottle and drove to T2 to drop off the run bag and took the shuttle back to T1 at around 5:30. Dawdled a bit and got my bike setup after I got there, prepped a towel, got my shoes set up, etc. At ~6:30 I was starting to feel like I could eat a little more so I ate a clif bar pre-race oatmeal bar that I’d brought along just for kicks. I’d never had one before but I was so hungry and had nothing else, so I decided to risk it. It really didn’t go down easily. I was worried I'd get sick but I pretty much force fed it to myself, reminding myself of what happened at Jack’s Generic Tri in August. My right hip was still really hurting, so I stretched for about 15 minutes. Still no good. I then massaged the muscle constantly up until literally right before the race started. I was super anxious and worried about the bike. At about 7:15 I had a honey stinger to eliminate any potential energy lapse. I'd rather overeat than under. It was a TT start for the swim and my AG was the first to go. I was about the 6th person in the water. Right after I jumped in I just got into a flow. I felt fast and just kept pushing. Water was cool and I sighted well. I passed one two people too, which was crazy given how early I was in the water. Stepping out of the water was right when I knew things were going to go well. I wasn’t tired at all. It only got better. There was a huge hill to get into T1, and I was worried the day before about how rough that would be. But nope, ran like mad all the way to the top and still felt fresh and full of energy. Then I knew for sure I was going to rock it. Used a towel to wipe grass and dirt off my foot and then got going on the bike. Before the race, I was thinking I’d be ecstatic if I hit 170W+ on the bike? Well…I don’t know how it happened but I blasted through that goal. 185W normalized for the full duration. I lost a little steam during the last 2 miles which brought my average down, but even for that period I was cruising at 176W. There was another guy who kept trading places with me. In the end, he won out, but the back and forth was great. Pushed myself much harder because of it. Shows you the power of competition I guess. :) I got of the bike, ran to my run bag, swapped the shoes, and got going. The run course was flat and easy, so it friendly to a new runner like me. Like every other time, I think that I could have gone faster. Then again I had my fastest run ever at a 9:10 / mile pace, so whatever I did worked. My splits reflect my attitude…first kilometer I cruise, then the middle three I take too easy, and the last km I speed up again. If I can cruise longer and then pick of the final pace earlier, that alone I think could bring me to a sub 9” pace. Getting there! In the end, I really do have no idea where today’s burst of energy and power came from. I’ve been sick all week (and still feel sick!), had a limp hip, and have been super tired…but it all worked out in the end. Also happy I skipped the Splash and Dash earlier this week. I guess my body needed the rest. 2017-08-12T00:00:00Z tag:dlo.me,2017-08-12://archives/2017/08/12/jacks-generic-tri-race-report/ Woke up at 5:10, a little earlier than I’d planned since my dad-in-law decided he was doing the race at the last minute, so I picked him up on the way. This actually sort of caused a cascade of issues since I didn’t have time to eat breakfast. The steel cut oats, banana, and maple syrup REALLY didn’t appeal to me unfortunately, and because I felt so rushed, I *maybe* got one or two bites in before I had to head out. I think this was a big mistake. Transition setup went well, didn’t forget anything, plan was great. Swim was fantastic (at least from what I know from the watch). My pace was 1:40/100m which was much better than the previous race. We’ll have to wait on the official results but I was really happy with this. For some reason I got it in my head that I needed to push myself in the swim but in the end I’m a little regretful about this. You'll see why. T1 was BAD. I was totally out of breath when I got out of the water and essentially needed to stand still and put hands on knees to regroup myself. I probably lost a minute or two just getting back to the right mental & physical place. Now whether this had something to do with me not really having any breakfast…I guess there’s no way to know, but wouldn’t be surprised. Bike started off OK I think. The whole time I kept wanting to push harder, but just didn’t have the energy in me. Again, think the lack of a “real” breakfast got me here. I also didn’t drink any water at all for the full 42 minutes. I need a better strategy there too, I just plain forgot and the aero bottle I’m using has a really rigid straw that makes it really tough to drink and ride at the same time without some facial acrobatics. I did have the clif blok earlier than planned because I felt REALLY hungry. Man, lack of breakfast really did not do good things for me. My average power was 152W and normalized was 156W. For comparison, Life Time average was 153W and normalized was 158W. So it is what it is. I do think the reason I didn’t perform as well as I thought I could was 1) pushing too hard in the swim and 2) no breakfast 3) having to pee like crazy from about 15 minutes on. The last issue really affected me more than I would have liked, it was just really hard to push myself. T2 was much much better than T1. I felt good getting off the bike, got the helmet off, popped a honey stinger, had some water, put on the visor, and went on my way. Like the swim, the run was my best run of the season so far. According to the watch, my average pace was 9:53 / mile which is ~40s / mile faster than Life Time in late May! I’m really happy with this and didn’t walk for a second! The last km was a sprint for me, I saw another guy with a 33 on his leg and made up my mind to beat him. So I picked up the pace to a 8 minute pace for the last 0.6 km, I kept it up and beat him! All in all, I’m really happy with the race. At the results tent, my placing was 10/14 when I checked, I’m sure it could change as more people finished but the way I look at it, my swim and run were both MUCH better, and the bike was the same as last race. I would have liked to have hit my bike goals but I think this was more a matter of race day planning and nutrition than fitness. Always something. Onwards to Kerrville where maybe I can learn from today and rock the bike. 2017-06-24T00:00:00Z tag:dlo.me,2017-06-24://archives/2017/06/24/how-to-accurately-track-open-water-swims-vivoactive-hr-fenix-5/ After some frustrations in tracking my open water swims and races with the [fēnix 5](http://amzn.to/2u1aq60), I decided to do something a little wild. I'd read around online that some people have experimented with taking their [vívoactive HRs](http://amzn.to/2s3BCEs) and clipping them to their goggles, or in their swim caps, and getting decent results. I hadn't seen any experimental data or comparisons posted by anyone anywhere, so I figured it was worth a shot. Despite looking like a doofus during my last open water race, I donned my [vívoactive HR](http://amzn.to/2s3BCEs) to my goggle strap, strapped the [fēnix 5](http://amzn.to/2u1aq60) to my wrist, and pitted them both head-to-head. Starting the workout on the [vívoactive HR](http://amzn.to/2s3BCEs) takes some finesse since you'll need to do it without looking at the device. Alternatively, you can start the workout before strapping it to the goggle strap and then rush to attach it before the race starts, which I decided was a little too much work (didn't want to be fiddling with my goggles within seconds of the start). I started workouts for both trackers within a few seconds of each other, and ended both at around the same time as well. I'm not going to ramble on here; the results speak for themselves. Just check out the images below (don't judge, I'm not the fastest swimmer, and my sighting could use some work :) ). <table style="border:none;padding:0;margin:20px 0;"> <tr> <td style="text-align:center; padding: 0 0 6px 0"><h2><a href="http://amzn.to/2s3BCEs">vívoactive HR</a></h2></td> <td style="text-align:center; padding: 0 0 6px 0"><h2><a href="http://amzn.to/2u1aq60">fēnix 5</a></h2></td> </tr> <tr> <td style="padding: 6px 3px 0 0"> <a href="http://tpks.ws/D2FZf"><img src="/images/triathlon/vahrquarryswim.jpg" /></a> </td> <td style="padding: 6px 0 0 3px"> <a href="http://tpks.ws/aCch8"><img src="/images/triathlon/fenix5quarryswim.jpg" /></a> </td> </tr> </table> The [vívoactive HR](http://amzn.to/2s3BCEs) is light-years more accurate, and gives almost per-second updates on speed and location, whereas the [fēnix 5](http://amzn.to/2u1aq60) only provides occasional GPS updates. If you want the source data, the images are hyperlinked and will take you straight to the workouts on TrainingPeaks. The [fēnix 5](http://amzn.to/2u1aq60) *does* capture stroke data---if that's an important factor to you---but otherwise falls short. The reason I found this surprising isn't that it worked the way it did; that the [vívoactive HR](http://amzn.to/2s3BCEs) won out makes sense since the GPS sensor is exposed to open air for the vast majority of time while it's clipped to the goggle strap, and almost never fully submerges. The [fēnix 5](http://amzn.to/2u1aq60) has to re-lock GPS within a span of no more than 1.5-2s between strokes. One of the two devices clearly has the upper hand here. The surprise is that not more people are doing this, or have figured out a way to make it less nerdy. Perhaps a small device that you just clip to your goggles would make for a decent product. Or, if there was an adapter to the [vívoactive HR](http://amzn.to/2s3BCEs) that allowed you to remove the straps and lock it onto your goggles, that wouldn't be a bad solution either. In the meantime, I'm just going to keep using my [fēnix 5](http://amzn.to/2u1aq60) for races and keep the [vívoactive HR](http://amzn.to/2s3BCEs) in the swim bag. It's a little bit of a hassle to get the workouts started on the [vívoactive HR](http://amzn.to/2s3BCEs), and not worth the extra stress, but if you do want a solid map of where you swam in open water, it's a solid way to go. 2017-06-09T00:00:00Z tag:dlo.me,2017-06-09://archives/2017/06/09/lifetime-tri-captex-race-review/ #### Pre-race Pros: * Had dinner (chicken + sweet potato) night before before 6pm * Breakfast (oats, banana, maple syrup) was great once again * Warmed up for about 10 minutes about an hour pre-swim, stretched, dynamic WU, and then a run (tried to get up to Z4 but couldn't?). * Tri suit fit GREAT and felt great. Improve: * Warm up closer to start. * Foam roll at home. * Couldn't get to Z4 in the WU. AI: Make sure watch is on correctly / push a little harder in WU? <hr class='plain' /> #### Swim (16:16, 10/22 AG) Pros: * Followed plan, could have maybe gone a bit faster but didn't want to wear myself out. * Sighting went great! Didn't stray too off-course like I did at the Rookie. * Breathing went well, stroke could have been a bit stronger. Improve: * Got passed a few times. AI: Get faster. :) * Felt really thirsty about halfway in. AI: Drink more electrolyte before swim? <hr class='plain' /> #### T1 (2:34, 7/22 AG) Followed plan to the t, EXCEPT dried my feet a little to get some grass off...not sure if worthwhile. <hr class='plain' /> #### Bike (39:16, 14/22 AG) Pros: * New threshold! 149W. * DEMOLISHED race plan target of 145W, avg (non-normalized) was 153! Improve: * Not in aero as much as I would've liked. AI: Stay in aero more during workouts. * Despite good performance, I think I could have gone harder. AI: Not sure. Maybe FTP just jumped a lot in past few weeks, so this should work itself out. * Completely forgot to have the gel at mile 9/10. AI: Double-check alerts on bike computer and maybe unzip compartment a bit with the gel sticking out as a reminder. <hr class='plain' /> #### T2 (2:50, 19/22 AG) Improve: * Forgot to have gel, again. AI: Print out race plan place in transition area. * Was really slow when putting on socks. AI: Consider practicing without socks every once in a while. Body glide? <hr class='plain' /> #### Run (32:18, 21/22 AG) Pros: * Ran the entire distance--no walking! * No hip pain after bike. * Beat my Rookie Tri pace by almost a minute per mile (RT was 11:21, CapTex was 10:27). Improve: * Felt like I could have pushed myself harder, but didn't. AI: ??? <hr class='plain' /> #### Overall * My best race ever, despite having a gnarly cold. * New bike FTP! * Don't forget to have gels next time! 2016-01-19T00:00:00Z tag:dlo.me,2016-01-19://archives/2016/01/19/replacing-tracking-speed/ My iMac has started resetting my mouse's tracking speed upon every restart. While somewhat frustrating, it's pretty easy to open up System Preferences -> Mouse, and update the tracking speed to one notch below "Fast" and get on with my work. <img src="/images/mouse.png" /> While it's gotten a little old, it also got me to thinking: why does Apple measure mouse movement in terms of "Tracking speed"? And what is tracking speed, anyways? After doing a "fair bit" of research (read: jumping to the [mouse speed](https://en.wikipedia.org/wiki/Computer_mouse#Mouse_speed) section on Wikipedia), I encountered an interestingly named measurement called "Mickeys per second" (tee hee). It makes some sense: according to Wikipedia, it measures "the ratio between how many pixels the cursor moves on the screen and how far the mouse moves on the mouse pad." While, at some point in the past, this might have been a completely sensible measurement, we've moved somewhat beyond pixels. Pixels used to be visible to the naked eye, but with today's 4K and 5K displays, that's no longer true. What also struck me was that, unless Mickeys per second could change with each display, setups with multiple displays would need a variable number of Mickeys per second to render a constant speed mouse pointer (at least in physical space). Obviously, behind the scenes, modern operating systems are flexible and take account of this, but this dynamic behavior is hidden from the end user. Let's go back to the beginning. Here I am, updating my tracking speed a couple times a week. When I do something more than once, my instinct is to find a way to stop doing it. Ultimately, I came to the conclusion that we (or really, Apple or Microsoft) are thinking about this in the wrong way. Think about it. Mouse velocity comes down to three things: 1. The "reach" of the user's hand (i.e., the maximum distance the center of the mouse sensor can be moved by the user from one side to the other). 2. The size of the screen. 3. The "intent" distance (i.e., the smallest intentional movement a user can make) Without taking into user comfort level, the absolute _minimum_ for this hypothetical measurement should be one screen *per reach*. No matter how good you are with computers, it's a bad experience if you need to lift up your mouse several times to position the cursor in the right place. At maximum, it again needs to be user-specific. If mouse control is erratic or difficult for the user, the intent value should be larger than for someone with good hand dexterity. Since we don't want to move the mouse at all until the mouse has moved at least the intent distance, and we don't want to move it less than the screen size, we can determine an upper bound and lower bound for mouse velocity. Furthermore, extracting these values doesn't entail asking the user to drag a marker on a screen for some arbitrary indicator. It would be relatively easy to determine these values automatically based on a simple tool: just ask the user to move the mouse from side-to-side, and then display a grid to the user, prompting them to click between two points as close to each other as possible. Since screen size is already known by the OS, it would just be a simple matter of crunching the numbers to an internal value. There's probably lots of bigger fish to fry on Apple's Mac OS X team, but I think it'd be a huge improvement to the user experience and would make this setting a lot less opaque to end users. 2015-09-18T00:00:00Z tag:dlo.me,2015-09-18://archives/2015/09/18/marco-did-the-right-thing/ Marco Arment released an [iOS 9 Content Blocker](http://peace.land) on Wednesday and it quickly rocketed up to #1 on the Paid App Store charts. At that ranking, apps can pull in tens of thousands of dollars per day. For very [good reasons](http://www.marco.org/2015/09/18/just-doesnt-feel-good), he removed it from sale this morning. And then things got nasty. Thing is, I think I understand what this week has been like for Marco. In a lot of ways, it reminds me of what happened when a [site I wrote went viral](/archives/2011/03/20/breakup-notifier/) a few years ago. If you've never made something that's gone viral before, let me break down how it feels like for the maker: 1. You're in shock. You can't believe something you made resonated with so many people. 2. You freak out because your inbox has become a disaster. 3. You try to get some work done and deal with the explosion of feature requests and attention. 4. You start reading online and notice people are saying some really shitty things about you. 5. At this point you can't think about anything else except the shit that people are giving you. 6. You want to shut it all down and forget about it. For anyone who thinks Marco thought this through or planned it in advance, you're just deluding yourself. Who in the world could *anticipate* making a #1 rated app in the App Store. I don't think any independent developer has _ever_ done this outside the context of a game. If I were Marco, I'd have been feeling dejected and depressed by this morning and would've wanted all of the attention to go away. No one person can deal with it alone. In his post, Marco specifically pointed readers to _instructions_ on how to get App Store refunds. I mean, people, if he was actually trying to scam you, don't you think he'd just leave the app up on the store and just never say anything else about it? At least he's being honest. If you already purchased the app, it's not like you didn't get anything in return. You bought a working app. It still works. It will continue to work. There is no scam. Marco isn't being an asshole. If he's anything like me, he's overwhelmed and just wants to get back to normal life. Being the center of crosshairs can really suck. Be a little more empathetic. Thank Marco for being honest. Respect his decision. He's just a person like you and me. I realize how easy is can be to forget that when everyone is just an avatar, but please take it to heart. I wish more people did when a similar thing happened to me 4 years ago. 2015-07-26T00:00:00Z tag:dlo.me,2015-07-26://archives/2015/07/26/making-a-time-lapse-using-ffmpeg-and-imagemagick/ We're vacationing in Whistler, BC right now as "endurance spectators" to my father-in-law's 3rd Ironman triathlon. Expecting some beautiful landscapes and weather, I brought my newly acquired X100T to take some nice photos. Yesterday, I set it up on an interval timer and pointed it right towards Rainbow Mountain, which faces the patio in the kitchen of the little condo unit we're renting out. After all was said and done, I ended up with 400 images depicting clouds moving over a mountain peak and not much idea of what to do with them. So, as any self-respecting engineer would, I set out to create a time-lapse using only my trusty command-line tools: [FFmpeg](https://www.ffmpeg.org) and [ImageMagick](http://www.imagemagick.org/script/index.php). Let's get down to it. _Note: Everything in this tutorial assumes that you have a current copy of ImageMagick and FFmpeg installed on your machine._ ### Resizing Even though I turned off RAW on the X100T, the images were still pretty huge (4896x3264). During my first tests, making movies from images this large gave really inconsistent results and took a long time to create, with not much extra benefit. Therefore, the first thing you should probably do is check the size of your images and, if necessary, resize them to be a bit smaller so they will play more nicely with FFmpeg and any other image manipulation that you're going to do. Since I planned to upload my video to YouTube, I referenced a handy page they have that lists out their preferred resolutions, codecs, and formats for upload ([https://support.google.com/youtube/answer/1722171?hl=en](https://support.google.com/youtube/answer/1722171?hl=en)). If you're like me, and you don't care too much about maintaining the current aspect ratio, here's what you can do. This will resize your images to a preferred resolution (in this case, 1280x720), and will potentially crop off the sides or top in the process. To start, make sure you're in the directory with all of your photos. ``` $ for FILE in `ls *.JPG`; do \ mogrify -resize 1280x720^ -gravity center -crop 1280x720+0+0 +repage -write RESIZED_PHOTO_DIRECTORY/$FILE $FILE; \ done ``` In detail, this command `-resize`s photos to a `1280x720^` resolution (the caret means that the smaller of width and height is maintained and the larger one is kept even if the resolution is larger), and then, by using `-gravity` and `center`ing, we crop the image to `1280x720` exactly, and write to `RESIZED_PHOTO_DIRECTORY/$FILE`. Phew, that was a mouthful. If you just want to resize to a certain height/width and want to maintain the original resolution, just do this: ``` $ for FILE in `ls *.JPG`; do \ mogrify -resize 600x -write RESIZED_PHOTO_DIRECTORY/$FILE $FILE; \ done ``` ### Maintaining Color Distribution _Note: this step might not be necessary in your situation, but it greatly improved the quality of the final product for me. YMMV._ Sometimes images captured in a time lapse have very different histograms (especially if you have auto-aperture / shutter-speed enabled), and this can make things look "jumpy" from frame to frame. Obviously, this won't look great in your final video, so we're going to normalize the colors to a set distribution. For an example, just compare the following two images (especially notice the trees, which are much lighter in the first example than the second):   Not ideal, right? To help achieve this end, I used an ImageMagick script called `histmatch`, generously provided by Fred Weinhaus (link: [http://www.fmwconcepts.com/imagemagick/histmatch/index.php](http://www.fmwconcepts.com/imagemagick/histmatch/index.php) or [backup](/backups/histmatch.mhtml)). The idea to use a reference image to generate a histogram that we want all of the other images to match. Once you've decided on your reference image, run the following on every image except the reference image (otherwise the universe will explode). ``` histmatch -c gray REFERENCE.JPG TARGET_IMAGE.JPG NORMALIZED_IMAGE_DIRECTORY/FILE.JPG ``` (I just piped the output of `ls *.JPG` into a file called `normalize.sh` and used some of my Vim-fu to do this. Your process might be different.) *12/27/2017 Update:* Re-reading this post, I've found it's much easier to just move the target image to the normalized directory, and then run this using find/exec. ``` find . -depth 1 -name "*.JPG" -exec histmatch -c gray normalized/TARGET_IMAGE.JPG {} normalized/{} \; ``` ### Finally, make the darned movie This is the fun part. Just send the files through to FFmpeg and have it do its magic. If filenames are incrementally named, you'll want to provide the parameters below (like `-start_number` and the `_DSF%04d.JPG` format) to make things match up. ``` ffmpeg -start_number 1 -i _DSF%04d.JPG -c:v libx264 -pix_fmt yuv420p timelapse.mp4 ``` This tells FFmpeg to take all of the JPEGs in the directory starting with `_DSF` and ending with 4 digits, and to output an h.264 video with the `yuv420p` colorspace to `video.mp4`. You now have a beautiful timelapse! If you're interested in the final product, you can [check it out on YouTube](https://www.youtube.com/watch?v=zGHGTcas_24). Enjoy! 2015-06-08T00:00:00Z tag:dlo.me,2015-06-08://archives/2015/06/08/post-wwdc-2015-keynote/ It's now been a few hours after The WWDC 2015 Keynote, and I've had some time to digest everything. My immediate impression of everything was a little "meh", but then again, that's sort of how I feel every year. It's hard to satisfy everyone. Second impression is that the WWDC Keynote is no longer for developers. It's for the end users and wannabe developers. Us old-timers are too jaded to care about this new and shiny stuff, and most of the new products in the keynote aren't even things developers can use (on that note, was it really necessary to spend _that_ much time on Apple Music?). Another noteworthy thing--no new hardware. I was expecting at least _something_, so to hear crickets was a little unfortunate. I can only assume that the inordinate amount of time spent talking about Apple Music was in some part due to a need to "fill time" from what would have originally been a 15-20 minute spiel on Apple TV. As always, the things that are exciting to me happen in the sessions throughout the week. The Keynote is sort of just a preview of what's to come. After some perusing through the documentation, here's what I'm excited about: ### Deep Linking I wrote about this in my WWDC 2015 Wishlist, and it came true. You can now link up URLs to be opened by your application. I haven't had too much time to play around with iOS 9 yet, so I'm not sure how this works from either the developer or end-user side of things, but my first impression is that Apple did this right. ### Search Besides the notable backtracking of moving search away from the pull-down gesture and _back_ to the left of the home screen (I would have loved to be a fly on the wall in that meeting), iOS Search can now display search results straight from apps in Spotlight. Just to illustrate an example---my company writes an app called Tweet Seeker that lets people download their Twitter archives and search their tweets locally on their device. Tweet Seeker can now hook right into iOS and display tweet search results right from Spotlight. Now that's cool! ### iPad Multitasking Remember the iPad Pro? Well, this is it. You take a regular iPad, and you install iOS 9. There. iPad Pro. :claps: You know those things that you can't really imagine a use for, or want, and then in 3 months you realize you can't live without it? Yeah, well, I'd put money that this is one of those things. I can see this changing how people use their iPads, along with the new keyboard gestures. The iPad is no longer just a toy, or larger screen iPhone made for watching Netflix and HBO. It's now a tool to get shit done. ### Swift 2 I think it's great news that Swift is going open source "later this year". Of course, in Apple parlance, that means probably somewhere around December 15-31 (just trying to be realistic here--it's not easy to open-source something like this, I'd bet there is a _ton_ of proprietary code lurking in that codebase). Also, I think with this release, I'm comfortable picking up the Swift book and starting to actually learn the language. I am embarrassed (only a bit, though) that I haven't written a single line of code in the language. With this latest release, I think I might be ready to start jumping in. A 2.0 implies a little more stability, and other developers who were waiting on the sidelines will probably jump in on the fun as well. Another telling thing is that all the code I saw in the "Developers State of the Union" Keynote was written in Swift. I don't think it's any secret now that Apple considers Swift to be the future. As much as we might not want it, it's going to happen. If you started learning it a few months ago, though, prepare to have to relearn a bunch of stuff. I wouldn't be surprised if Apple pulled the rug out from under you. ### Apple Music I'll keep it brief. I don't need it. Spotify works great for me. I think its success hinges on musicians buying in. We don't want another Ping here. ### Conclusion You'll notice I skipped over a lot of the OS X stuff. I honestly haven't had enough time to digest it all. So. Much. New. Stuff. Will download and report back. That's it for now. I'm taking a walk to think about this all some more. 2015-06-03T00:00:00Z tag:dlo.me,2015-06-03://archives/2015/06/03/wwdc-2015-wishlist/ Here's my list of things I'd be overjoyed to see fixed / announced / released at WWDC 2015 next week. ### iOS * Deep linking support. Let apps define associated URLs (https://twitter.com/* --> open Twitter.app). * Really want to hide my carrier and clean up that status bar. Such a mess right now. * It's time to modernize the UI for Notes.app and Reminders.app. iOS 6 was years ago. * Save alarm and world clock data in iCloud. Re-adding this on every device is really tedious. * Export Health.app data or save it in iCloud with a password. At the moment, it's not shared between devices and will be lost permanently if you don't have an encrypted local iTunes backup of your device. * Needs to be a way to close all tabs in Safari without tapping the screen 50x times. Absurd. ### Mail.app * Search that actually works. Right now this is the only reason I still have the Gmail app still on my phone. I open it up whenever I want to search for a message. * Send from an alias like Gmail web client lets you. * Better UX for labeling threads. ### Maps * (I've mentioned this on Twitter) Fix the UX with reviews of physical locations. * Public transit support. * Improved traffic data. ### App Store * Search. Need I say more? * Direct refunds without iTunes support involvement. * Responding to reviews would be nice. * Expire old reviews (1yr+) when new versions of app have since been released. ### Safari (OS X) * Reopen more than the most recent closed tab in Safari. * Paste images into web pages. ### Xcode * Stop beachballing so darn much. * It would be nice to stop an archive action without needing to restart the app. * Debugging extensions is currently a huge pain. It would be nice if it were easier. *Added 6/7/2015* ### Photos on OS X * "Show in Finder" 2015-03-29T00:00:00Z tag:dlo.me,2015-03-29://archives/2015/03/29/encryption-with-dropbox-on-os-x/ My wife and I have been searching for months for a service to allow us to store our sensitive files in Dropbox securely. There are a lot of services out there that promise to do this well, but frankly, it's hard to trust that our data would be safe if they happened to be hacked or compromised. I therefore went looking for a piece of software that was: 1. Vetted as being secure. 2. Ubiquitous. Something I wouldn't need to worry about installing or finding on a future machine. 3. Compatible with Mac OS X. TrueCrypt, an open-source file encryption product, seemed like it might fit the bill. *However*, I must have had my head in the sand about a year ago (sure enough, I was off the grid when it happened), but TrueCrypt was abandoned by its developers with a very vague / scary message on their website: > WARNING: Using TrueCrypt is not secure as it may contain unfixed security issues Ok then. So that in itself wasn't reason enough for me to give up. I went ahead and downloaded TrueCrypt from some random site and verified the checksum (remind me which of my family is going to know how to do this?) and installed it (in order to do this, I had to temporarily disable OS X's protection against unsigned software in "Security & Privacy" in System Preferences). That's a lot of steps for someone who just wants to share encrypted files. And more steps are bad---I didn't want people I'm sharing stuff with to just give up. We'd be back to square one. Not to mention, TrueCrypt doesn't offer dynamically sized volumes. You have to specify the size of the volume at the time of creation. This means that if the number of files in a volume you're sharing hits the limit---too bad. You need to now create a new volume (one with a larger limit) and move all of the existing files from the previous volume into it. That's a drag. And sort of dumb. Off I went looking for an alternative. "Wait", I thought, "doesn't OS X provide full-disk encryption via FileVault?". Well, I didn't actually ask myself that, but I concluded that there _must_ be a way to do this just using a stock Mac OS X installation. Sure enough, after some digging, I found this on Apple's support site: [How to create a password-protected (encrypted) disk image](https://support.apple.com/en-us/HT201599). Boom! There we have it. If you follow the instructions (which are quite simple, really), you'll have a "file" (which is actually a "sparse image"---sort of like a Volume) which you can drop into a shared Dropbox folder and share with anyone you want. To add or remove files to the encrypted folder, you need to double click it, type in the password, and then open it like you would any other volume (like your Hard Drive or a USB stick). Once you're done modifying files, just "eject" it. Easy peasy. Dropbox syncs it immediately. Oh---and the folder resizes automatically! Another huge plus over TrueCrypt. So, in short, you can create an AES-encrypted folder, inside your Dropbox, and anyone else who uses a Mac will be able to add or remove files from it without needing to install any third-party software. And, it's free. Go forth and encrypt! P.S. Windows users might have similar luck with BitLocker. P.P.S. [Jamie Phelps](https://twitter.com/jxpx777) [pointed me](https://twitter.com/jxpx777/status/582394914459144192) to [this AgileBits article](https://help.agilebits.com/Knox/sync_vaults.html) regarding a downside of using sparse _bundles_ and Dropbox (note--a sparse image and a sparse bundle are not the same thing, so the two may *not* have the same downsides). Knox stores its vaults with OS X sparse bundles, and those have been shown to have issues syncing over Dropbox when simultaneous edits are made. I've yet to see this be a problem with using sparse images (since OS X---and Dropbox---treats a sparse image as a single file), but it might be an issue. Jamie recommends [SafeMonk](https://www.safemonk.com/) as an alternative. 2014-09-08T00:00:00Z tag:dlo.me,2014-09-08://archives/2014/09/08/pythonrc/ I've been using and working with Python in a professional context for around 8 years. So it came sort of a shock that something so useful, so obvious, was unfamiliar to me until around a week and a half ago. It involves a little file called `.pythonrc` that lives in your home directory. To get it to work, you'll need to define an environment variable called `PYTHONSTARTUP`, like so: ```bash export PYTHONSTARTUP="$HOME/.pythonrc" ``` (Major hat tip to [kasnalin](https://www.reddit.com/user/kasnalin) on Reddit for [pointing out](https://www.reddit.com/r/Python/comments/2fun6t/how_to_supercharge_your_python_shell_with_a/ckd1w4t) that I'd forgotten to include this important piece of info in an earlier version of the post.) If you're a Python developer, no doubt you open up a Python shell countless times per day. Whether you use the regular Python shell, or another one such as IPython, it's all the same. Open up the shell, import a few useful modules, and start playing around. Maybe you find yourself importing the same modules over and over again. Maybe you leave a single shell open all the time since it's such a pain to re-type those commands. I spend a lot of time working with Django, so I'd gotten pretty used to typing `./manage.py shell` and then `from app import models` to start playing around with objects in my database. Well, I got to wondering what this `.pythonrc` could do to help me out with this, so I dumped in the following code: ```python try: from app import models from django.conf import settings except: print("\nCould not import Django modules.") else: print("\nImported Django modules.") ``` Not expecting this to work at all, I ran my trusty Django shell, and voilà: ```bash $ ./manage.py shell Imported Django modules. Python 2.7.6 (default, Jan 6 2014, 13:22:56) [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>> ``` It's like magic. No more reverse searching my shell history to import commonly-used modules. If I find myself using one often enough, I just add it into my `.pythonrc` and it's there for me every time. I play around Redis a bunch, so here's another chunk of code I plopped in there. ```python from redis import StrictRedis as Redis r = Redis() ``` Now, I can just open up the shell and use `r` to play with Redis. Here's another one I'm pretty fond of. I'm a big fan of tab-completion and readline support. So I put this in my `.pythonrc` too: ```python try: import readline except ImportError: print("Module readline not available.") else: import rlcompleter if 'libedit' in readline.__doc__: readline.parse_and_bind("bind ^I rl_complete") else: readline.parse_and_bind("tab: complete") ``` Works like a charm. Below is my full `.pythonrc`, in all of its glory (also available as a [Gist](https://gist.github.com/dlo/e0b48890495a6c4b3065)). ```python # vim: set ft=python : from __future__ import print_function import json import sys import datetime from redis import StrictRedis as Redis r = Redis() try: import readline except ImportError: print("Module readline not available.") else: import rlcompleter if 'libedit' in readline.__doc__: readline.parse_and_bind("bind ^I rl_complete") else: readline.parse_and_bind("tab: complete") try: from app import models from django.conf import settings except: print("\nCould not import Django modules.") else: print("\nImported Django modules.") try: from dateutil.parser import parse as parse_date except ImportError: print("\nCould not import dateutil.") ``` 2014-09-01T00:00:00Z tag:dlo.me,2014-09-01://archives/2014/09/01/postgresql-fts/ After hearing a few months back that PostgreSQL had built-in full-text search support, I had been continually searching for reasons to take advantage of it. A few months went by, and when one of our clients at [Lionheart](http://lionheartsw.com) started running into issues with slowness using the Django admin's built-in search support, I knew I had a solid candidate. After all was said and done, searches returned results approximately 20x faster than vanilla Django. Here's how you can do it too. ## The Basics One of the great things about PostgreSQL full-text search is that there isn't too much to learn if you're already pretty familiar with search engines and how they work. The main idea is that you generate an index using a document containing terms you'd like to search on. PostgreSQL provides a lot of flexibility with regards to _how_ you choose to do this, but I opted to store the documents as a field called `fts_document` on the table we wanted to search on. PostgreSQL stores search data in a data type called a `tsvector`. Here's how the [PostgreSQL documentation](http://www.postgresql.org/docs/9.3/static/datatype-textsearch.html) describes it: > A tsvector value is a sorted list of distinct lexemes, which are words that have been normalized to merge different variants of the same word. Sorting and duplicate-elimination are done automatically during input. Here's an example of a sentence converted to a tsvector: ```psql =# SELECT to_tsvector('simple', 'The quick brown fox jumps over the lazy dog.'); to_tsvector --------------------------------------------------------------------------- 'brown':3 'dog':9 'fox':4 'jumps':5 'lazy':8 'over':6 'quick':2 'the':1,7 (1 row) ``` When you want to search for something, you create what's called a `tsquery`. > A tsquery value stores lexemes that are to be searched for, and combines them honoring the Boolean operators & (AND), \| (OR), and ! (NOT). Parentheses can be used to enforce grouping of the operators Here's an example: ```psql dan=# SELECT to_tsquery('simple', 'fox | jumps'); to_tsquery ----------------- 'fox' | 'jumps' (1 row) ``` When you want to test for a match, you use the `@@` operator. ```psql dan=# SELECT to_tsvector('simple', 'The quick brown fox jumps over the lazy dog.') @@ to_tsquery('simple', 'fox'); ?column? ---------- t ``` It's a match! The `@@` operator also provides a shorthand whereby you don't need to use the `to_tsvector` / `to_tsquery` functions explicitly. E.g., ```psql dan=# SELECT 'The quick brown fox jumps over the lazy dog.' @@ 'fox'; ?column? ---------- t ``` Sweet! However, you might be confused by the following: ```psql dan=# SELECT 'The quick brown fox jumps over the lazy dog.' @@ 'fox | random'; ?column? ---------- f ``` Huh? We're searching for _either_ the term "fox" or "random". This doesn't seem right... Well, it turns out that when you specify plain text as an argument to the `@@` operator, it wraps it in `plainto_tsquery`, which functions a little bit differently than the `to_tsquery` function that we've been playing around with above. Notably, it ignores every logical operator except `&`. So `|`'s, parentheses, etc., will all be ignored. Personally, I don't find it to be all that useful. Also, as you might have noticed above, I provided an additional argument to both the `to_tsvector` and `to_tsquery` functions called 'simple'. This represents the simple text search configuration. Other text search configurations are language-based and strip out stopwards and other lexemes based on what language you might be indexing. Here's a few other options that I have available on my machine. ```psql dan=# \dF List of text search configurations Schema | Name | Description ------------+------------+--------------------------------------- pg_catalog | danish | configuration for danish language pg_catalog | dutch | configuration for dutch language pg_catalog | english | configuration for english language pg_catalog | finnish | configuration for finnish language pg_catalog | french | configuration for french language pg_catalog | german | configuration for german language pg_catalog | hungarian | configuration for hungarian language pg_catalog | italian | configuration for italian language pg_catalog | norwegian | configuration for norwegian language pg_catalog | portuguese | configuration for portuguese language pg_catalog | romanian | configuration for romanian language pg_catalog | russian | configuration for russian language pg_catalog | simple | simple configuration pg_catalog | spanish | configuration for spanish language pg_catalog | swedish | configuration for swedish language pg_catalog | turkish | configuration for turkish language (16 rows) ``` Play around with a few and see which one suits you best. ## Django By now you know how to create search vectors and queries, search on some data, and specify a text search configuration. Let's get started on connecting these tools with your Django project. Since `tsvector` and `tsquery` are both standard PostgreSQL data types, we can use them just as we might any other data type like `int` or `varchar` in a column definition. This is very handy. Otherwise, we would have to regenerate vectors on-the-fly, which might offset any speedup we get from using FTS in the first place. :) For the sake of moving forward, let's say we have a model called `Employee` in our Django project. We're trying to speed up searches on anything related to an employee. Assume we're storing their first name, last name, and full address. ```python class Employee(db.Model): first_name = models.CharField(max_length=64) last_name = models.CharField(max_length=64) address_1 = models.TextField() address_2 = models.TextField() city = models.CharField(max_length=64) state = models.CharField(max_length=2) zip = models.CharField(max_length=9) ``` We're now going to create an `tsvector` based on these fields. To do this, we'll concatenate all of the fields into a list and then flatten the list into a string to feed into the `to_tsvector` function. ```psql dan=# SELECT to_tsvector('simple', concat_ws(' ', first_name, last_name, address_1, address_2, city, state, zip)) FROM employee; to_tsvector ------------------------------------------------------------------------- '123':3 '12345':8 'in':7 'main':4 'pawnee':6 'ron':1 'st':5 'swanson':2 (1 row) ``` If we wanted to stop here, we could just do FTS matches on the fly, but this would be quite slow. ```psql dan=# SELECT COUNT(*) FROM employee WHERE to_tsvector('simple', concat_ws(' ', first_name, last_name, address_1, address_2, city, state, zip)) @@ 'swanson'; count ------- 1 (1 row) ``` The next step is to store this `tsvector` in a field and keep it updated when the data changes. First step seems easy enough: ```psql dan=# ALTER TABLE employee ADD COLUMN fts_document tsvector; ``` Hmm, but how to we keep this updated? There are a bunch of options, but according to the PostgreSQL documention, the best one is to add a trigger on document updates and additions. To make things easy, we're going to create a function to generate the fts document. E.g., ```psql CREATE FUNCTION employee_fts_document(e employee) RETURNS tsvector AS $$ DECLARE employee_document TEXT; BEGIN SELECT concat_ws(' ', e.first_name, e.last_name, e.address_1, e.address_2, e.city, e.state, e.zip) INTO employee_document; RETURN to_tsvector('pg_catalog.simple', employee_document); END; $$ LANGUAGE plpgsql; ``` We can then reference this function in the trigger. ```psql CREATE FUNCTION employee_fts_document_trigger() RETURNS TRIGGER AS $$ BEGIN NEW.fts_document=employee_fts_document(NEW); RETURN NEW; END; $$ LANGUAGE plpgsql; ``` Finally, we tell PostgreSQL to execute the trigger on employee updates and inserts. ```psql CREATE TRIGGER employee_fts_update_trigger BEFORE UPDATE ON employee FOR EACH ROW EXECUTE PROCEDURE employee_fts_document_trigger(); CREATE TRIGGER employee_fts_insert_trigger BEFORE INSERT ON employee FOR EACH ROW EXECUTE PROCEDURE employee_fts_document_trigger(); ``` We're almost done. The *last* thing we might want to do is to tell PostgreSQL that we'd like to index the `fts_document` column which contains our full-text search vectors. We're going to add a GIN (Generalized Inverted Index)-based index to our column. You can read about the GIN-based and GiST-based indices in the [PostgreSQL documentation](http://www.postgresql.org/docs/9.3/static/textsearch-indexes.html). Here's how we add our index. ```psql CREATE INDEX employee_fts_index ON employee USING gin(fts_document); ``` That's it. Our employee table now has an always up-to-date search index that we can use. Before we continue, let's instantiate the search vectors. We'll then test everything out to make sure things are working a-ok. ```psql dan=# UPDATE employee SET fts_document=employee_fts_document(id); dan=# SELECT COUNT(*) FROM employee WHERE fts_document @@ 'ron'; count ------- 1 (1 row) ``` Success! ### South Now that we understand the internals, wouldn't it be nice if we could keep our databases up-to-date without having to manually copy and paste commands on our PostgreSQL shell? South to the rescue! We'll need to generate a blank migration. ```bash $ ./manage.py schemamigration app create_employee_fts Created 0001_create_employee_fts.py. You must now edit this migration and add the code for each direction. ``` Next, open up the migration, and copy in the SQL above in the `forwards` and `backwards` methods, as illustrated below. ```python # ... imports, etc ... from django.db import connection class Migration(SchemaMigration): def forwards(self, orm): sql = """ CREATE FUNCTION employee_fts_document(integer) RETURNS tsvector AS $$ DECLARE employee_document TEXT; BEGIN SELECT concat_ws(' ', first_name, last_name, address_1, address_2, city, state, zip) INTO employee_document FROM employee WHERE id=$1; RETURN to_tsvector('pg_catalog.simple', employee_document); END; $$ LANGUAGE plpgsql; CREATE FUNCTION employee_fts_document_trigger() RETURNS TRIGGER AS $$ BEGIN NEW.fts_document=employee_fts_document(NEW.id); RETURN NEW; END; $$ LANGUAGE plpgsql; """ cursor = connection.cursor() cursor.execute(sql); cursor.execute("ALTER TABLE employee ADD COLUMN fts_document tsvector") cursor.execute("UPDATE employee SET fts_document=employee_fts_document(id)"); cursor.execute("CREATE TRIGGER employee_fts_update_trigger BEFORE UPDATE ON employee FOR EACH ROW EXECUTE PROCEDURE employee_fts_document_trigger()") cursor.execute("CREATE TRIGGER employee_fts_insert_trigger BEFORE INSERT ON employee FOR EACH ROW EXECUTE PROCEDURE employee_fts_document_trigger()") cursor.execute("CREATE INDEX employee_fts_index ON employee USING gin(fts_document)") def backwards(self, orm): cursor = connection.cursor() cursor.execute("DROP INDEX employee_fts_index") cursor.execute("ALTER TABLE employee DROP COLUMN fts_document") cursor.execute("DROP TRIGGER employee_fts_update_trigger ON employee") cursor.execute("DROP TRIGGER employee_fts_insert_trigger ON employee") cursor.execute("DROP FUNCTION employee_fts_document (integer)") cursor.execute("DROP FUNCTION employee_fts_document_trigger ()") models = { """ App-specific model definitions here... """ } ``` Now, just run this migration... ```bash $ ./manage.py migrate app ``` ...and watch as your FTS column, indices, and triggers are automagically created. ### The Django Admin OK, so we have our FTS document created, have generated a migration to keep things in sync across machines, and are indexing everything we might possibly want to search on to find an employee. The next step is to use what we have and integrate it into the Django Admin. This assumes that you have an `EmployeeAdmin` object similar to the below: ```python class EmployeeAdmin(admin.ModelAdmin): search_fields = ("first_name", "last_name", "address_1", "address_2") # ... other admin configuration here ... admin.site.register(models.Employee, EmployeeAdmin) ``` Our goal is to make searching with the Django Admin use our shiny new FTS index. To do this, we're going to need to override the `get_search_results` method in `EmployeeAdmin` to something like this: ```python class EmployeeAdmin(admin.ModelAdmin): # ... def get_search_results(self, request, queryset, search_term): if search_term == '': queryset, distinct = super(EmployeeAdmin, self).get_search_results(request, queryset, search_term) else: queryset, distinct = super(EmployeeAdmin, self) \ .get_search_results(request, queryset, None) queryset = queryset.extra( where=["employee.fts_document @@ to_tsquery('simple', %s)"], params=[search_term] ) return queryset, distinct ``` Looks close, but we're missing something. `to_tsquery` will expect a "properly-formed" query, and we can't always trust or depend on users of the admin to know PostgreSQL search syntax (don't blame them). We need to somehow sanitize the input before feeding it into PostgreSQL. Here are some examples of things that we might want to handle: * Strip padded whitespace. * Remove duplicated whitespace. * Add &amp;'s between words that are only separated by spaces. * Replace double quotes (which PostgreSQL doesn't like) to single quotes. * Clear up situations where search operators are used ambiguously (i.e., "ron &amp; \| swanson") * Add a prefix wildcard to every search term, so that a search for "swan" will return "Ron Swanson" in the search results. Luckily for you, I've already done all of the heavy lifting. Below is a function called `sanitize_search_term` which will take a string and form it into a proper PostgreSQL search query. ```python import re import string def sanitize_search_term(term): # Replace all puncuation with spaces. allowed_punctuation = set(['&', '|', '"', "'"]) all_punctuation = set(string.punctuation) punctuation = "".join(all_punctuation - allowed_punctuation) term = re.sub(r"[{}]+".format(re.escape(punctuation)), " ", \ term) # Substitute all double quotes to single quotes. term = term.replace('"', "'") term = re.sub(r"[']+", "'", term) # Create regex to find strings within quotes. quoted_strings_re = re.compile(r"('[^']*')") space_between_words_re = re.compile(r'([^ &|])[ ]+([^ &|])') spaces_surrounding_letter_re = re.compile(r'[ ]+([^ &|])[ ]+') multiple_operator_re = re.compile(r"[ &]+(&|\|)[ &]+") tokens = quoted_strings_re.split(term) processed_tokens = [] for token in tokens: # Remove all surrounding whitespace. token = token.strip() if token in ['', "'"]: continue if token[0] != "'": # Surround single letters with &'s token = spaces_surrounding_letter_re.sub(r' & \1 & ', token) # Specify '&' between words that have neither | or & specified. token = space_between_words_re.sub(r'\1 & \2', token) # Add a prefix wildcard to every search term. token = re.sub(r'([^ &|]+)', r'\1:*', token) processed_tokens.append(token) term = " & ".join(processed_tokens) # Replace ampersands or pipes surrounded by ampersands. term = multiple_operator_re.sub(r" \1 ", term) # Escape single quotes return term.replace("'", "''") ``` Combining everything together, we get the following: ```python class EmployeeAdmin(admin.ModelAdmin): # ... def get_search_results(self, request, queryset, search_term): if search_term == '': queryset, distinct = super(EmployeeAdmin, self).get_search_results(request, queryset, search_term) else: search_term = sanitize_search_term(search_term) queryset, distinct = super(EmployeeAdmin, self) \ .get_search_results(request, queryset, None) queryset = queryset.extra( where=["employee.fts_document @@ to_tsquery('simple', %s)"], params=[search_term] ) return queryset, distinct ``` And...we're done! Searches in the admin will now use PostgreSQL FTS. Here's to speedy searching! <hr/> EDIT 12/15/2014: In a previous version of this post, `employee_fts_document` would pull stale values from the DB while generating the FTS document and performed an extraneous `SELECT`. Thanks, [Roger](http://dlo.me/archives/2014/09/01/postgresql-fts/#comment-1742372939)! 2013-11-08T00:00:00Z tag:dlo.me,2013-11-08://archives/2013/11/08/review-iphone-5s-case/ I normally don’t write reviews, but I figured it’s never a bad time to start. I’ll jump right into it. I have never before gotten a case for my iPhone. Cases add bulk, take away from the aesthetic of the phone, and are just another “thing” to keep track of. But when I heard about the [leather case](http://store.apple.com/us/product/MF041LL/A/iphone-5s-case-brown) that Apple created specifically for the iPhone 5s, I thought that it would be the end of my no-case journey. It came a couple of days ago in the mail. I ordered from Best Buy online. The cases come in 6 color choices—brown, beige, black, yellow, blue, and red. I went with black, since it was the only one that could really go with space gray. Not to mention, I’d read a bunch of reviews that said the other colors (especially the brighter ones) were prone to picking up dirt quickly. The case looks and feels great in the hand. The leather is no frills and simple. It’s not the sort of leather you’d see on a cowboy boot, but supposedly it’s real. There is a stamped Apple logo on the back. It looks pretty cool, and gives it more authenticity, I guess. Putting the case on is quite easy. It really is fitted quite well to the form of the phone. Sort of expected for a case, especially if you’re also the manufacturer of the phone itself, but I suppose it’s still a good thing. As for bulkiness, it didn’t add much, but it *was* enough to be noticeable. In terms of weight and form factor, the iPhone 5s with the leather case “felt” like my 4S in the hand. This isn’t necessarily a bad thing (some people might like that), but it’s not totally for me. The biggest problems I have with the case are more functional and less visual. 1\. The leather cover over the power button is a bit too stiff. One should not have to struggle when pressing the power button. 2. The volume buttons are not covered like the power button is, but they’re still quite hard to reach. It’s not impossible by any means, but it’s just hard enough to be annoying. Given that I wasn’t blown away by the look and feel and the usability problems that it has, I decided to return it. I think I’m still in the market for a case, just not the Apple one. 2013-07-08T00:00:00Z tag:dlo.me,2013-07-08://archives/2013/07/08/so-long-california-thanks-for-all-the-fish/ It's been almost four years since my wife and I made the decision to move to Los Angeles after college. We came here for a lot of reasons, but the biggest ones were family, the lively tech scene, and the practically perfect weather. All in all, I really do love this place. But after nearly four years of calling it home, the time has come to make a change. Real estate prices are through the roof and state income taxes are amongst the highest in the country. From a business perspective, franchise taxes (which you have to pay even if your business *loses* money) sting and processing times for even the most routine paperwork take months. Even outside of ritzy areas like Beverly Hills, houses advertised as "teardowns" by brokers are going for prices in the neighborhood of $1.2MM and up. Homes at every price range are getting multiple all-cash offers within days of going on the market. Renters are fairly shielded from this reality, but if you're thinking about becoming a homeowner here, it can be a struggle. We decided that we didn't want to struggle. And why should we? More so than ever before, where you live matters less than what you can do. And that's an amazing thing. So, where to? The answer was clear to both my wife and I from the beginning: Austin, Texas. Austin boasts no state income tax, real estate a young couple can actually afford, an exploding tech scene, and beautiful landscapes. Throw in great food, music, and culture, with a taste of <a href="https://fiber.google.com/cities/austin/">Fiber Internet</a>, and you've essentially described my perfect city. As a part of my move, my company, [Aurora](http://aurora.io), has become [Lionheart Software](http://lionheartsw.com), a Texas LLC. I'll still be doing everything I'm doing now: helping small teams build great software. And there's a good chance [Pushpin](http://lionheartsw.com/software/pushpin/) will be getting some company. I'll be moving out later this month and am excited beyond words. If you're an Austinite, send me an [email](mailto:dan@dlo.me). I'd love to meet up. 2013-05-14T00:00:00Z tag:dlo.me,2013-05-14://archives/2013/05/14/nginx-pagespeed-cloudfront/ A couple weeks back, Google released ngx_pagespeed, an nginx module that is capable of doing a lot of cool stuff behind the scenes to speed up your webpages. > ngx_pagespeed speeds up your site and reduces page load time. This > open-source nginx server module automatically applies web performance > best practices to pages, and associated assets (CSS, JavaScript, > images) without requiring that you modify your existing content or > workflow. This article will walk you through how to get up and running with ngx_pagespeed and how to serve its dynamically generated pipelined content through a CDN such as Amazon CloudFront. ### Getting Started To install ngx_pagespeed, we’re going to need to compile nginx from source. Fret not, it’s actually not too bad. Let’s get started by logging into our server and installing the dependencies. On Ubuntu / Debian, run: ```bash sudo apt-get update sudo apt-get install build-essential zlib1g-dev libpcre3 libpcre3-dev libssl-dev ``` For RedHat, CentOS, and Fedora, run: ```bash sudo yum install gcc-c++ pcre-dev pcre-devel zlib-devel make ``` Our next step is to download the [nginx](http://nginx.org/en/download.html) and [ngx_pagespeed](https://github.com/pagespeed/ngx_pagespeed) sources and install them. ```bash cd $HOME wget http://nginx.org/download/nginx-1.4.1.tar.gz wget http://github.com/pagespeed/ngx_pagespeed/archive/release-1.5.27.2-beta.tar.gz tar xzfv nginx-1.4.1.tar.gz tar zxfv release-1.5.27.2-beta.tar.gz cd ngx_pagespeed-release-1.5.27.2-beta/ # psol -> PageSpeed Optimization Libraries wget https://dl.google.com/dl/page-speed/psol/1.5.27.2.tar.gz tar xzvf 1.5.27.2.tar.gz cd $HOME/nginx-1.4.1 # Configure nginx, specify SSL support, and add the ngx_pagespeed module ./configure --add-module=$HOME/ngx_pagespeed-release-1.5.27.2-beta --with-http_ssl_module make sudo make install ``` By default, nginx doesn’t bundle in SSL support, but I’m adding it in here because practically every web application I’ve built over the last couple of years has had to be served securely (this is also why we installed libssl-dev from apt-get earlier). After nginx compiles successfully (which should take a couple of minutes), you’ll have a freshly-cooked nginx binary in /usr/local/nginx/sbin. You can also change the binary location by explicitly specifying `—prefix=` to the `./configure` command. Let’s double check that everything’s working correctly. Run this… ```python sudo /usr/local/nginx/sbin/nginx ``` …and your terminal should output something like this: ``` [0508/234323:INFO:google_message_handler.cc(33)] Shutting down ngx_pagespeed root ``` If this is what you see, ngx_pagespeed is good to go! ## Configuring Amazon CloudFront You can skip this step if you don’t want or need to serve from a CDN. Also, these instructions are explicitly for Amazon CloudFront, but the process for other services should be similar. Sign into your Amazon AWS account and go to the CloudFront console. Create a download distribution…  …set the origin domain name to the domain serving your static assets…  …and create the distribution. This finalization process can take up to 15-30 minutes, so just give it some time. ## Configuring PageSpeed To make it easier to test things out, I’ve written a simple HTML page with some CSS files to play around with if you don’t already have something handy ([link](https://gist.github.com/dlo/5544622/download)). Unzip these files to the /usr/local/nginx/html directory. Let’s open up our nginx configuration file (by default, it should be located at /usr/local/nginx/conf/nginx.conf) and clean things up. After removing commented and unnecessary lines, it should look something like the following: ```nginx worker_processes 1; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; sendfile on; keepalive_timeout 65; server { listen 80; server_name example.com; location / { root html; index index.html index.htm; } } } ``` Start nginx if it’s not already running and hit your site. With the example page I’ve included above, the source of the page should look something like this: ```html <!doctype > <html> <head> <title>A Simple Page</title> </head> <body> Will you remove my whitespace? </body> </html> ``` Ok, this is what we expected and is sort of boring. Let’s make it exciting and turn on PageSpeed. After the `server_name` directive, add the following lines. ```nginx pagespeed on; ``` This is just one of many filters available. I’ve added `collapse_whitespace` because it’s easy to verify that it’s working. ```nginx pagespeed EnableFilters collapse_whitespace; pagespeed FileCachePath /var/ngx_pagespeed_cache; pagespeed Domain http://example.com; ``` (optional) Specify the domain where static media is being served ```nginx pagespeed Domain http://static.example.com; ``` (optional) If possible, have PageSpeed load static files from the filesystem (instead of hitting the server) ```nginx pagespeed LoadFromFile http://static.example.com/ /var/www/static/; ``` Change “example.com” and “static.example.com” to the domains where you’re serving your pages and your static assets, respectively. They can also be the same. If you’d also like to serve the assets ngx_pagespeed generates through your CDN, add a line with the “MapRewriteDomain” directive. This directive tells ngx_pagespeed to replace the domain serving any static assets with your CDN’s domain. Here’s what it might look like if you’re using CloudFront: ```nginx pagespeed MapRewriteDomain dao9e9afbfaaa5.cloudfront.net static.example.com; ``` Finally, we have to create ngx_pagespeed’s cache directory and give nginx ownership over it. ```bash sudo mkdir -p /var/ngx_pagespeed_cache sudo chown -R www-data:www-data /var/ngx_pagespeed_cache ``` Time to reload nginx and test it all out. ```bash $ sudo /usr/local/nginx/sbin/nginx -s reload Setting option from ("on") Setting option from ("FileCachePath", "/var/ngx_pagespeed_cache") Setting option from ("LoadFromFile", "http://static.example.com/", "/usr/local/nginx/html/") Setting option from ("MapRewriteDomain", "dao9e9afbfaaa5.cloudfront.net", "static.example.com") Setting option from ("EnableFilters", "collapse_whitespace,inline_javascript") ``` Once we hit our page again and view source, we can now see that ngx_pagespeed has combined our two stylesheets and removed all unnecessary whitespace, in addition to attaching some Javascript instrumentation snippets that helps ngx_pagespeed measure how long the page took to load and render (see the [instrumentation docs](https://developers.google.com/speed/docs/mod_pagespeed/filter-instrumentation-add) for more info). Awesome stuff. ```html <!doctype> <html> <head> <title> A Simple Page </title> </head> <body> Will you remove my whitespace? <script type="text/javascript"> (function(){var e=encodeURIComponent,f=window,g=document,h="documentElement",k="length",l="prototype",n="body",p="&ci=",q="&oh=",r=",",s="Content-Type",t="Microsoft.XMLHTTP",u="Msxml2.XMLHTTP",v="POST",w="application/x-www-form-urlencoded",x="img",y="input",z="load",A="on",B="pagespeed_url_hash",C="url=";f.pagespeed=f.pagespeed\|\|{};var D=f.pagespeed,E=function(a,b,c){this.e=a;this.c=b;this.d=c;this.b=this.f();this.a={}};E\[l\].f=function(){return{height:f.innerHeight\|\|g\[h\].clientHeight\|\|g\[n\].clientHeight,width:f.innerWidth\|\|g\[h\].clientWidth\|\|g\[n\].clientWidth}};E\[l\].i=function(a){a=a.getBoundingClientRect();return{top:a.top*.scrollTop),left:a.left*(void 0!==f.pageXOffset?f.pageXOffset:(g\[h\]\|\|g\[n\].parentNode\|\|g\[n\]).scrollLeft)}};E\[l\].g=function(a){if(0\>=a.offsetWidth&&0\>=a.offsetHeight)return!1;a=this.i(a);var b=JSON.stringify(a);if(this.a.hasOwnProperty(b))return0;return a.top\<=this.b.height&&a.left\<=this.b.width};E\[l\].h=function(a){var b;if(f.XMLHttpRequest)b=new XMLHttpRequest;else if(f.ActiveXObject)try{b=new ActiveXObject(u)}catch©{try{b=new ActiveXObject(t)}catch(d){}}if(return)1;b.open(v,this.e);b.setRequestHeader(s,w);b.send(a);return!0};E\[l\].k=function(){for(var a=\[x,y\],b={},c=0;c\<a\[k\];+*c)for(var d=g.getElementsByTagName(a\[c\]),m=0;m\<d\[k\];m)d\[m\].hasAttribute(B)&&(d\[m\].getBoundingClientRect&&this.g(d\[m\]))&&(b\[d\[m\].getAttribute(B)\]=!0);b=Object.keys(b);if(0!=b\[k\]){a=C+e(this.c);a*=q+this.d;a*p+e(b\[0\]);for(c=1;c\<b\[k\];*+c){d=r+e(b\[c\]);if(131072\<a\[k\]*d\[k\])break;a*=d}D.criticalImagesBeaconData=a;this.h(a)}};D.j=function(a,b,c){if(a.addEventListener)a.addEventListener(b,c,!1);else if(a.attachEvent)a.attachEvent(A+b,c);else{var d=a\[A+b\];a\[A+b\]=function(){c.call(this);d&&d.call(this)}}};D.l=function(a,b,c){var d=new E (a,b,c);D.j(f,z,function(){f.setTimeout(function(){d.k()},0)})};D.criticalImagesBeaconInit=D.l;})();pagespeed.criticalImagesBeaconInit('/ngx_pagespeed_beacon','http://example.com/','r20W6lmsPP'); </body> </html> ``` ### Conclusion From here, you can do basically anything you want with PageSpeed. I’ve only scratched the surface of what it’s capable of, so you’ll want to check out the [extensive documentation](https://developers.google.com/speed/pagespeed/mod) to read about everything else it can do. From here, you can do basically anything you want with PageSpeed. I’ve only scratched the surface of what it’s capable of, so you’ll want to check out the [extensive documentation](https://developers.google.com/speed/pagespeed/mod) to read about everything else it can do. 2013-04-26T00:00:00Z tag:dlo.me,2013-04-26://archives/2013/04/26/mike-krieger-on-startup-tech/ Mike Krieger (Instagram cofounder) wrote an article today on [Opbeat](http://blog.opbeat.com/2013/04/26/picking-tech-for-your-startup/) regarding the technology stack tech entrepreneurs should use for their startups. His advice could not ring more true to me (especially in light of my [previous post](/archives/2013/04/16/there-is-no-right-way-to-develop-software/)). > Ultimately, the most important thing to keep in mind is that your > “stack” shouldn’t define you; while some engineers are attracted to > companies working with cutting-edge tech, that shouldn’t be the > driving force behind your early technology decisions. Instead, > establishing a solid base with mature data storage, and augmenting > more exciting recent projects where they can add value further up the > stack will let your team stay focused on solving the challenges you’ve > set out to solve. In short, use the right tool for the job, even if it isn’t sexy. 2013-04-16T00:00:00Z tag:dlo.me,2013-04-16://archives/2013/04/16/there-is-no-right-way-to-develop-software/ TDD is the only way to write bug-free code. Pair programming is the only way to work. We’re successful because we only hire remote workers. Blah blah blah blah blah. I see shit like this every single day. Little tidbits like this are so mesmerizing because they fool us into thinking that they’re the only thing separating us from that giant pot of gold at the end of a ridiculous awesome double rainbow. And, ultimately, that’s what the people writing this sort of thing would like you to think. If you were to take a random sampling of 100 software engineers and asked them “what is the first thing you do before starting a new software project”, I bet you’d get 100 different answers. So then why do so many people seem to think that their way is the only way? The answer I think is both really simple and really complicated. Simple, because some people just like to talk. Complicated, because the software engineering industry is actually a collection of subcultures that face-off constantly against each other to defend their job security. This all started yesterday after I read [an article](http://sturgill.github.io/2013/04/15/tests-are-overhyped/) by Chris Sturgill about tests and how they’re overhyped. Everything written in the article was totally reasonable. But the comments on the article painted a totally different story. If you were only to have read through the comments, you’d think the Chris was on crack or something. How dare he question TDD? How dare he even think to question the validity of our test-driven culture? So after seeing some of these comments I felt compelled to write what I’m writing right now. I really don’t want this article to have to do with testing at all. This is just about being a good software developer. Let’s cut to the chase. There is no “right” way to develop software. I repeat: there is no right way to develop software. Some people in our industry like to cargo cult and don’t want to believe this. They believe that the sweet new hotness they learned a couple of hours ago is the only way to make things work and build reliable pieces of engineering. I’m sorry to break it to you. The new hotness is probably not new. It’s probably just new marketing on old hotness. At some point you’ve gotta learn to stop dropping everything you’re doing and changing everything and then bragging to the world about how your new way is the “only” way. It’s not. Stop. Being a good developer means compromise. It means sometimes doing things one way for one project, and another way for another. It means balancing the needs of your stakeholders with your ideals. Sometimes they’re not always going to match up, but that doesn’t mean you should stomp the ground with your feet and have a tantrum when things don’t go your way. [HN discussion](https://news.ycombinator.com/item?id=5562791) 2013-02-11T00:00:00Z tag:dlo.me,2013-02-11://archives/2013/02/11/the-perfect-bug-report/ Like many other developers, the people who I interact with most often in my codebases and issue trackers are not other software engineers—they’re my clients. My partners have generally never been involved in the development of any software project at any point in their lives prior to working with me. This is not to say that they aren’t technically savvy—on the contrary, they usually are. However, there are a certain number of things about bug reporting that aren’t obvious to someone who hasn’t been doing it for years. So, I thought about it, and instead of having to give a mini-overview every time I want to bring this up, I thought it might be nice to have something to refer to that explains what you should do when you encounter an issue. Big or small, enterprise or start-up, everything applies. ### The Basics Issue trackers vary widely in terms of features from one to another. Whether it’s a simple list of items on an Google Docs spreadsheet or a custom JIRA installation with multiple plugins and tens of fields, everything I’ll write in this post applies. No matter what, there are certain fields that your issue tracker should ALWAYS account for. They are: <ul class="styled"> <li> Assignee </li> <li> Title </li> <li> Description </li> <li> Milestone </li> <li> Labels / Tags / Categories (optional but recommended) </li> </ul> If your issue tracker doesn’t support the first four, find another one. Labels are good but not necessary. Now that we’ve got a bug tracker, let’s start off with a fairly common scenario. ### I Can’t Login! @#\$%! You woke up this morning to a nice cup of coffee, opened up your application to get work done…and you can’t login. First thing, DON’T PANIC. It’s easy to feel helpless if you’re not the developer, since you may think there’s not much you can do. Wrong! Your role is super important. You speak for the users. The better you do this, the quicker the issue will get fixed and the happier everyone will be. So if you find that you’re stressed, just take a deep breath. We want to make sure that everything we do from this point forward is clearly communicated, straightforward, and easy for the assignee to work with. Let’s handle the low-hanging fruit, shall we? This means verifying that you actually have an account and then checking that you’re actually putting in the right password. This may sound dumb, but just double check. **Always** double check. After you’ve made sure that you didn’t make a careless mistake, try to make whatever broke break again. Depending on the issue, this process can take a fair amount of patience and/or creativity. Software can be very picky, so make sure to be precise. If you encountered the bug when writing your login credentials in ALL CAPS, try to reproduce it with all caps. Then, try with lowercase letters. Still working? Now try with a mix of uppercase and lowercase. Then try with only spaces. Do whatever you need to do to corner it! After much experimenting, you’ve found that no user can login, even with the correct username and password, and no error message is displayed. Whatever is happening isn’t intended behavior, so it’s confirmed: you’ve found a bug. ### The Issue Title The issue title comes first for a couple of reasons. <ol class='styled'> <li> It’s the first thing other team members will see. </li> <li> A clear issue title helps you decide out who to assign it to. </li> <li> Writing the title forces you to describe the problem clearly and succintly. </li> </ol> Before going into details, I’m just going to start off with some anti-examples. The following are *bad* issue titles: <ul class='styled'> <li> login </li> <li> login problem </li> <li> login form not working </li> <li> can’t login </li> <li> login page </li> <li> SITE IS BROKEN </li> </ul> I don’t like these issue titles because they are vague and don’t help the assignee understand what is actually wrong. The best issue titles are specific and to-the-point. Brevity is *always* appreciated, but the sweet spot for length is probably around 6-10 words. Your issue title should briefly identify two things: <ol class="styled"> <li> The undesired outcome </li> <li> When or how it happens </li> </ol> With all that in mind, here’s the issue title I would write: > Login fails silently when user enters correct credentials Onwards! ### The Assignee Once we have a title, you should have a good idea for who needs to work on this. There are generally two scenarios here. 1\. You know who does what. 2. You know who knows who does what. If you know who is responsible for whatever feature or bug you’ve just encountered, make them the assignee. If you don’t know who is responsible—even if there is the slightest doubt, assign the issue to the person who will know who to assign it to. ### The Description As with a good title, a description should be short and sweet, but also a bit more detailed. There’s actually a straightforward recipe. <ul class="styled"> <li> Write in sentence or two what you tried to do and what should’ve happened. </li> <li> If the bug seems really specific, give steps on how to reproduce it. Note: if *all* logins fail (and not just yours) and there’s not really a pattern, you can skip this step. </li> <li> Attach screenshots if applicable. </li> </ul> Given the above, here’s what I might write as the description for the login issue: > I just tried to log into the site from the login form on > /accounts/login and nothing appeared to happen, even when using my > correct username and password. If there was some sort of pattern, or only certain credentials fail, this is what I might write: > I just tried to log into the site from the login form on > /accounts/login and nothing appeared to happen, even when using my > correct username and password. More specifically, it seems that > passwords with weird characters cause the issue (my password has a > space). >   > 1. Use credential with space in password. > 2. Fill in form with correct credentials. > 3. Submit form and notice that the user is not logged in and no error > message is shown. We’re almost done! On to milestones. ### Milestones Milestones are like goals. Trying to ship software without goals just isn’t going to happen. And if it miraculously does, it’ll take way more time than you originally planned, and will probably have costed way more money and energy than you had the budget for. This is why you need milestones, and why issues need to be placed into them. Milestones can be numbered, given code names, be descriptive, or anything—it really doesn’t matter. All that matters is that it’s clear what is high priority (due sooner) and what’s “low” priority (due later). I put low in quotes because, in my opinion, everything is high priority. It’s just a matter of *how* high. So file the issue into the appropriate milestone. Personally, I think users not being able to log in is a pretty big deal, so I would file this under the “STOP WHAT YOU’RE DOING AND DO THIS NOW” milestone. ### Labels / Tags / Categories Every bug tracker gives this a different name, but I’m going to call them labels for the sake of simplicity. I have a love-hate relationship with this one. I think labels have a tendency to be overused, and are mostly just there to be pretty, but I think there’s something to be said for being able to categorize and group issues by similarity. In my projects, I use six labels[^1]. <ul class="styled"> <li> question: for features we need to discuss or questions </li> <li> enhancement: improvements, can be design, backend, frontend, anything </li> <li> bug: something isn’t working the way it should </li> <li> tweak: changes to existing features or designs </li> <li> needs more info: not enough information in the issue to make a next step </li> <li> duplicate: issue was already reported </li> </ul> Some bug trackers will also have an “invalid” label. I think this is a bad label, because it makes the reporter feel stupid if it’s ever used. This is bad karma and throws a negative wrench in the relationship between everyone involved. Just ask for closer instructions to reproduce, and tag “needs more info”. If the reporter can’t do it, close it and say it couldn’t be reproduced. Yay, no feelings hurt. The issue we’re discussing (not being able to log in) should just be labeled as “bug” and that’s that. ### Conclusion There’s nothing else, really. I’m sure that there’s way more I could write about this topic, but after reading this, you should feel confident enough to write a bug report that is concise, information-dense, and makes it as easy as possible for the responsible person to resolve. And remember what I said about not panicking? Well, because you didn’t panic, the bug report you wrote was easy to read and contained straightforward instructions to reproduce. The developer was able to pinpoint the problem in the login logic, run tests to verify he didn’t break anything else, and push a fix to production in no more than 15 minutes. The most important thing? Everyone did their jobs with a smile on their face. :) I hope this was helpful to you. Have a comment? I’d love to hear your thoughts. Send me an [email](mailto:dan@dlo.me) or reach out via [Twitter](https://twitter.com/dwlz). <!-- You can also check out the comments on "Hacker News":http://news.ycombinator.com/item?id=5205568. --> [^1]: If you use GitHub and like these labels, you may be interested in the [script](https://gist.github.com/dlo/4760637) I wrote that replaces the default labels with these ones. 2013-01-24T00:00:00Z tag:dlo.me,2013-01-24://archives/2013/01/24/using-source-maps-debug-compass-sass/ I'm a big fan of [Compass](http://compass-style.org/), the CSS framework that uses [Sass](http://sass-lang.com/) under the hood to make developing stylesheets not nearly as tedious as it might be otherwise. One thing that's always been a pain, however, is debugging SCSS while using the web inspector in Google Chrome. Here's the problem. Compass reads content from a certain file, and converts it into CSS, which the browser then interprets to style the page you're viewing. When I (the frontend developer) reload the styles for a page and want to see where a rendering mistake might have occurred, I will only see the line number from the _generated_ CSS file, and not the original SCSS file that I'm actually editing. Here's an example of what you might see below.  So, to find the source of a mistake, I have to open that CSS file, find that line number, go to the corresponding line in my SCSS file, and then repeat. Yuck! Now, here's the solution: [source maps](http://www.html5rocks.com/en/tutorials/developertools/sourcemaps/). Source maps are a rad new feature that has landed (at least with experimental status) in Webkit. Instead of viewing "main.css" above, you would see the source file and the source line number, e.g. "main.scss:352". By now you probably want to know how to set this up, so here are the distilled instructions for Chrome 24. ### Enabling Source Maps in Google Chrome 1. Navigate to chrome://flags and search for "Enable Developer Tools experiments". Enable it and then restart Chrome.  2. Once Chrome is back up and running, open the inspector and click on the settings icon on the lower right.  3. Under the "General" tab, find "Sources" and check "Enable source maps".  4. Under the "Experiments" tab, check "Support for SASS".  Congrats, you've come far. now Chrome is ready. The next thing you have to do is let Compass know to generate the requisite markup. In your Compass config file, add the following line: ```ruby sass_options = { :debug_info => true } ``` Once you've done this, start Compass (make sure to stop it first if you already have it running). ...and you're done. You should now be able to open the web inspector for a specific element and view which SCSS file and line number the styles were generated from, à la below:  Enjoy! 2013-01-14T00:00:00Z tag:dlo.me,2013-01-14://archives/2013/01/14/how-to-serve-static-files-django/ I've been lamenting the state of static files in Django for a while now. The app has caused one problem after another for me. In one situation, after running \`./manage.py collectstatic\`, my entire development environment was wiped. In another, I found staticfiles was completely unreliable when syncing static media to remote storage backends, due to the way it calculates whether a file has changed (I posted this problem to the [Django developers mailing list](https://groups.google.com/d/msg/django-developers/vtMVq8jwnf8/xosdFRc_4vAJ) a couple months ago, but nothing came of it). After reading a [tweet](https://twitter.com/zedshaw/status/290997963851972608) that perfectly echoed my feelings on the subject, I decided it was time to write down how I solve the “staticfiles problem" in my own apps, in a way that works flawlessly on both development and production. <script> $(function() { $.getJSON("https://api.twitter.com/1/statuses/oembed.json?id=291031387912417282&callback=?", function(data) { $('div#tweet-1').html(data.html); }); }); </script> <div class='tweet' id='tweet-1'></div> <script> $(function() { $.getJSON("https://api.twitter.com/1/statuses/oembed.json?id=291022623398318080&callback=?", function(data) { $('div#tweet-2').html(data.html); }); }); </script> <div class='tweet' id='tweet-2'></div> <div class='clear'></div> After following this tutorial, you will be serving **all** static files on your development machine from a folder called “static" in your Django project, and you will easily be able to sync static media to any storage backend quickly and easily. This tutorial is for a **base** Django installation. I generally use my own [project template](https://github.com/lionheart/django-template) that takes care of setting sane defaults for me (and in use in tens of production projects), but given the reality here, I know a lot of you reading this are probably reading this with legacy applications in mind. I hope these instructions work for you, but if they don't, please send me an email (first name at this domain). #### Serving Static Files on a Development Server To keep it simple, let's start a Django project using the standard template. ``` $ django-admin.py startproject statictest ``` Switch into the folder and start a virtual environment. ``` $ virtualenv . New python executable in ./bin/python Installing distribute..........................................................................................................................................................................................................done. Installing pip................done. $ . bin/activate ``` Set the variables in your \`settings.py\` file to the values below. ```python # This import belongs at the top of your settings file import os BASE = os.path.abspath(os.path.dirname(__name__)) STATICFILES_DIRS = (os.path.join(BASE, "static"),) ADMIN_MEDIA_PREFIX = '/static/admin/' STATIC_URL = "/static/" ``` Note: in a real-life situation, you will want to split out your local and production settings files, and symlink them as necessary for deployment. The above is purely for illustration purposes. Here's the magic step. To get all files in your “static" folder serving from the “/static/" URL on the development server, add these lines to your \`urls.py\` file. ```python from django.contrib.staticfiles.urls import staticfiles_urlpatterns # ... urlpatterns += staticfiles_urlpatterns() ``` To test it out, just create a folder called “static" in the base of your project directory. If you did everything correctly to this point, you should see the following in your “statictest" folder: ``` $ ls bin include lib manage.py static statictest ``` Start the development server, and everything in that folder will be accessible from “http://localhost:8000/static/". Congrats. You rock. #### Serving Static Files in Production Frankly, it's not super useful if all your static files just live locally. I tried to get staticfiles to play nicely with Amazon S3, but I just couldn't do it. Why? Well, firstly, \`./manage.py collectstatic\` will upload files that have already been uploaded if you use a modern DVCS and work with other developers (essentially, this means it will reupload files every time you pull down your code from the remote). It does this because it uses last modified times as the heuristic for deciding whether a file has changed. Unfortunately for every modern developer, we all know that Git and other tools set the last modified time of a time to whenever code was last pulled from the server, not when the code was actually last modified. Secondly, the django-storages implementation for retrieving last modified time from Amazon S3 requires sending an HTTP request to AWS for each file you're syncing. This, needless to say, is **extremely** slow. Because [my proposals](https://groups.google.com/d/msg/django-developers/vtMVq8jwnf8/xosdFRc_4vAJ) for getting these issues fixes in Django core were denied, I wrote a Django application called [statictastic](https://github.com/lionheart/django-statictastic). The app adds a new management command called \`syncmedia\` which, well, syncs your static media. It's fast and works on all storage backends built on the Django API. So, how is it so much faster than the collectstatic implementation? Well, it starts by creating a metadata file that contains the md5 checksums of the files that currently exist in your project. When you first run \`./manage.py syncmedia\`, statictastic creates this metadata file and uploads it to the remote backend. Then, every following time you run the command, statictastic compares your local md5 checksums to those on the remote backend. When statictastic notices that a checksum has changed, it re-uploads the outdated file and updates the remote metadata file as well. If you'd like to use it, the first thing you'll want to do is install it via pip: ``` pip install boto pip install django-storages pip install django-statictastic==0.6.1 ``` Next up, in your settings.py: ```python INSTALLED_APPS = ( # ... statictastic, ) DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage' STATICFILES_STORAGE = 'statictastic.backends.VersionedS3BotoStorage' # Optionally change these to full CDN urls UPLOAD_URL = "https://s3.amazonaws.com/uploads-statictest/" STATIC_URL = "https://s3.amazonaws.com/static-statictest/" # I like to store my uploads and static media on different buckets. AWS_STATIC_STORAGE_BUCKET_NAME = 'static-statictest' AWS_STORAGE_BUCKET_NAME = 'uploads-statictest' # Access Keys, straight from Amazon AWS_ACCESS_KEY_ID = 'XXXXXXXXXXXXXXXXXXXX' AWS_SECRET_ACCESS_KEY = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' ``` Now, go ahead and test it out. ``` $ python manage.py syncmedia Updated favicon.ico 1 file updated ``` Awesome. You're done. To reference static files in your HTML files, use the “static" template tag. But wait! Not so fast. For some inexplicable reason, Django has two static template tags. The one that's in your templates by default isn't the one you want. The one you want you'll have to manually import. ```django {% raw %} {% load static from staticfiles %} <link rel='shortcut icon' href='{% static "favicon.ico" %}' /> {% endraw %} ``` The above template code would generate the following HTML: ```html <link rel="shortcut icon" href="https://s3.amazonaws.com/static-statictest/favicon.ico" /> ``` Because I like to break my caches, I added a optional setting for statictastic called \`COMMIT_SHA\` that adds a querystring to all static files referenced using the \`static\` template tag. If you set it to \`123456\` in your settings.py file, you'd see this in your template: ``` <link rel='shortcut icon' href='https://s3.amazonaws.com/static-statictest/favicon.ico?123456' /> ``` And that concludes my overview of how I make static files work on Django. If you have any questions, ping me on Twitter or send me an email. All the code found in this post can be found in my company's Django template hosted [on GitHub](https://github.com/lionheart/django-template). 2013-01-07T00:00:00Z tag:dlo.me,2013-01-07://archives/2013/01/07/belize/ I got back from Belize last Sunday, and I thought it might be a good idea to write a post about it. Plus, I took a ton of pictures and it would be a shame if no one else could see them. So, the family vacation to Belize started on December 23rd. We had an early flight to Dallas-Forth Worth, a short layover, and then another leg to Belize City. We flew American, and the experience was nothing much to speak of. My wife had just been upgraded to Gold Status on American, so we got “above average” treatment when checking in and getting our tickets. Unfortunately, however, my wife also had a pretty bad bout of sickness just hours before we had to leave on our flight. We’re blaming it (sorta) on the Thai food we had just hours before, but I had it as well, and no symptoms, so it’s still all a mystery. We tried to reschedule our flight to Belize at 3am in the morning before we were set to leave, but outgoing flights to our destination were packed to the brim. There was no way we would be able to make it to Belize within the week if we didn’t leave with our original flight. We rallied, made it to the airport with Rachel’s parents, and made it work. On the outgoing flight from LAX to DFW, we were lucky enough to have been upgraded to First Class. We weren’t able to truly appreciate it (I was too worried about Rachel to be relaxed), but for Rachel, it was magnitudes better than having to sit in a normal coach seat while having food poisoning. Once we made it to DFW, I loaded myself up with some Popeye’s chicken (I think I have that right), and felt a little disgusting afterwards. No worries, we’re going to Belize, and it’s a vacation! Our flight to Belize was fairly non-eventful, other than we made it to Belize and it was a smooth ride. Rachel slept on my lap throughout the flight, but sadly still hadn’t started to feel better.  The view from the airport Once we landed in Belize City, our trek was to continue on the unmaintained roads of the Belizian countryside, by way of van with manual transmission. The ride was about an hour and a half from the airport to our resort in the jungle. The most apparent thing while driving around this country is that it is very poor. Half-finished houses litter the roadside, broken down cars lie in almost every driveway, and there is practically no commercial activity (didn’t see one McDonald’s or Starbucks in the entire country). The only restaurants I saw off the side of the road were shut down, with wooden planks covering the windows, or stands selling coke and other soft drinks. The apparent owners sat outside on plastic chairs waiting for a thirsty driver to come buy. There is also basically no commercial printing. People just spray-painted the names of their establishments on the walls, with letters of the same word painted in varying sizes. One notable thing, however, was the number of places specializing in “tire repair”. Again, these were just people sitting in plastic chairs on the grass with a hand-written sign. Sometimes the sign just pointed down a dirt-filled driveway, the end of which there lay a pile of tires—ostensibly for cars and vans of all sizes. Really, this is not so much a surprise, as the roadways throughout the country look totally unmaintained. Once we arrived at our resort, the Sleeping Giant Lodge, we were pretty much spent. Rachel and I went straight to our room. We had purchased some rice and a banana at DFW (it’s said that stuff is good for stomach problems), and not long after settling in, we saw a cute little gecko on our wall.  Us getting out of the vans  Our room I went off to dinner while Rachel stayed in bed reading her Kindle. Dinner was somewhat uneventful—I was mostly worrying about Rachel and everyone else was deciding what adventure they were going to go on the next day. I had pretty much decided that no matter what, Rachel and I would stay in, despite how fun everything seemed to be. Two groups went out—one went horseback riding and the other went spelunking / waterfall jumping. I was told this was stuff they would never allow in the U.S., but everyone who went got amazing views of the caves of Belize. I wish I could write more about them, but alas. We did eat a fair amount at the lodge restaurant (there weren’t any other choices, really).  View from the restaurant The next couple days went by quickly. A lot of staying indoors and reading, relaxing, and having quiet lunches while sipping fresh piña coladas. One thing to note was that the service at this place was just fantastic. Everyone took the effort to learn our names, which was actually really great. The towel-folding skills of the staff was quite incredible as well. Here’s a photo of some bunnies they made for us.  They made pretty good drinks too, and did I mention we had essentially an open bar for the entire duration of the stay there? Yeah…that was nice. Here’s a pina colada they made for me.  We might have also done some ziplining…  Before we left, we also visited a blue hole. If you’re ever in the Caribbean, you might want to check one of these things out. Kind of creepy, but also pretty cool.  On the way out of the blue hole, we noticed a pretty funny sign for a bizarrely-named tree. I had to take a photo for posterity.  The next day, we were off to the beach resort. The drive was pretty uneventful, and long. Here’s us getting in the vans, ready to go:  As with the rest of Belize, much of the countryside was again filled with abandoned / incomplete homes. It was overcast too, so it actually worked out to be a good day to drive. Once we started getting closer to the beach, the landscape shifted from forest to more of a Florida Everglades feel. I grew up in Miami, so the sight was actually pretty familiar. As we were driving, our driving spotted a crocodile in the water to the left of the road. They hide themselves so well, it’s almost impossible to see them if you don’t have an eye for it. The beach resort was exactly what you might imagine. Great view of the Gulf of Mexico, plenty of cabanas, tropical drinks, beautiful rooms, etc.  The entrance to the beach resort We spent the first day at the beach getting acclimated and relaxing. I think Rachel and I immediately submerged ourselves in the hot tub. It was, after all, a grueling one and a half hour drive on a bumpy highway that got us here. The funny thing about the hot tub was that there was no temperature control. As a result, it was either scalding at 110 degrees, or off. So we were relegated to leaving our feet in there for a handful of seconds until our bodies couldn’t take the heat any more, and then bear the cold as long as we could before putting them back in. The bartender by the side of the pool was apparently the hot tub operator as well, so once we told him the problem, he promptly dismissed himself and pulled the plug on the jacuzzi. Because, you know, it only really is useful when it’s off. Once it cooled down, it was quite nice, but it took a good 30 to 40 minutes for that to happen. You can see the hot tub in the photo below, just underneath the tree.  Dinner our first night was also pretty unique. There is a restaurant about a mile down the street from the Jaguar Beach Lodge called “Love on the Rocks.” Their schtick is an “ancient” cooking style whereby diners use heated stones to cook their food once it’s served. It’s kind of like Benihana, except you do it yourself, and it’s not Japanese.  Love on the Rocks  Love on the Rocks, after the meal It was quite an experience, and I’d highly recommend it for anyone who has a chance to go there. One thing to beware though—the service is slow as molasses. Don’t go out to a meal unless you’re willing to spend more than two hours at the restaurant. I’m not sure if this is a Belize thing or what, but it happened at nearly every place we ate at. The next day was fairly relaxing—after of course choosing to walk to the center of town in 90+ degree heat. We were sweating like none other by the time we arrived at our destination—a small restaurant called Iris, which was run by an ex-pat South African woman with a very special cuisine. I ordered shrimp curry burritos, and it was fantastic. As with our meal the night before, the service was extremely slow. It was quite good though! We decided that walking back in the heat wasn’t going to happen, so we requested a pickup at the restaurant. Once we got back to the resort, we sat on the beach and I read “Game of Thrones: A Song of Fire and Ice” while Rachel read a romance novel. I normally don’t read book (my attention span needs improvement), but once I get in the zone, I fly through them. Here’s a panorama of the beach:  While I’ll avoid making this blog post a mini review of Game of Thrones, I will say that the book is fantastic, and even with the availability of Internet, I was sucked in. Highly recommended vacation / trip reading. That night we hung out with the family and went to another restaurant, “Rob’s”, owned by the same people who run “Love on the Rocks.” The food was okand the view from our table of the beach was just amazing, but the service was again very slow. The next day kind of blurred away, but we did a little reading again on the beach and ate at the resort restaurant for breakfast and lunch. I think we started packing to go back home towards the evening. For dinner, we returned to Love on the Rocks. A bunch of the family wasn’t with us when we went for the first night, but the food was so good that I certainly didn’t mind going again. I ordered a Surf and Turf (steak and lobster), and it was fantastic. The meal ended up taking three hours, though, so by the end everyone was super anxious to leave. The next day was our departure. Our bus to the airport was scheduled to leave at 10:30 or so, and by then we were all ready to go. The ride was 2 and a half hours, again through a pretty non-remarkable countryside. Our flight was straight to Miami, and then L.A. Here’s the restaurant we ate at the Belize Airport:  Conclusion—Belize was a great place to visit. There are, of course, some downsides to any place, but I had a great time and was able to really relax. It may have been the longest time I have ever gone without access to Internet, which was a blessing in many ways. It’s rare I ever get a chance to ever truly disconnect, hang out with my wife with no distractions, and reflect upon things that I never really think about. For those reasons alone, our trip to Belize was one of the best I’ve ever gone on. Thanks for reading, and may you all have a fantastic 2013. Do great things. 2012-11-26T00:00:00Z tag:dlo.me,2012-11-26://archives/2012/11/26/favorite-mute-filters/ You can click on any of these to add them to your Tweetbot. <!-- tapbots.com/tweetbot/mute --><a href='tweetbot:///mute/keyword?regex=1&mentions=0&text=%40.%2B.%2A%40.%2B.%2A%40.%2B'><code>@.+.*@.+.*@.+</code></a> This mutes tweets that @ reference three or more users. <a href='tweetbot:///mute/keyword?regex=1&mentions=0&text=%23%5Cw%7B15%2C%7D'><code>#\w{15,}</code></a> This mutes tweets with absurdly long hashtags. <a href='tweetbot:///mute/keyword?regex=1&mentions=0&text=%5E.%7B1%2C16%7D%24'><code>^.{1,16}$</code></a> Mutes tweets shorter than 17 characters. <a href='tweetbot:///mute/keyword?regex=1&mentions=0&text=%5E%28C%7Cc%29urrent+status'><code>^(C|c)urrent status</code></a> Mutes tweets starting with the text "Current status". <a href='tweetbot:///mute/keyword?regex=1&mentions=0&text=%5E%5B+%5D%2A%5C.%40'><code>^[ ]*\.@</code></a> Mutes publicized replies (e.g., tweets that people prepend with a period to show to everyone). I generally don't find these interesting, as they're sometimes out of context. <a href='tweetbot:///mute/keyword?regex=1&mentions=0&text=%5E%5B%5Ea-z%5D%7B10%2C%7D'><code>^[^a-z]{10,}</code></a> Mute tweets that consist only of uppercase characters, whitespace, and punctuation. <a href='tweetbot:///mute/keyword?regex=1&mentions=0&text=%28fuck%7Cshit%7Cpoopin%29'><code>(fuck|shit|poopin)</code></a> I don't like cursing, and the whole poopin' meme is weird. <a href='tweetbot:///mute/keyword?regex=0&mentions=0&text=I+just+met+you'><code>I just met you</code></a> There was a time when people made "I just met you" jokes on Twitter constantly. It may have cooled down since then, but I wouldn't know. <a href='tweetbot:///mute/keyword?regex=0&mentions=0&text=true+story'><code>true story</code></a> I dunno, I just got a little tired of this one. <a href='tweetbot:///mute/keyword?regex=1&mentions=0&text=nobody%2C%3F+ever'><code>nobody,? ever</code></a> Mutes all tweets of the form "[outrageous statement] - said nobody, ever". If you have any other mute filters to recommend, please post them in the comments! 2012-11-25T00:00:00Z tag:dlo.me,2012-11-25://archives/2012/11/25/blog-refresh/ I just gave my blog a bit of a refresh--it's been needing it for a while. So what changed? 1. Main content is now centered (as opposed to left-aligned). 2. Permalinks are now of the form :year/:month/:day/:slug (before it was just /:slug). 3. The archives have now moved to their own page, as opposed to a sticky div on the right side of every post. 4. The header has been simplified and a lot of defunct social networks were removed. 2012-10-09T00:00:00Z tag:dlo.me,2012-10-09://archives/2012/10/09/why-four-inches/ I’ve been sitting on this for a couple of weeks now, but I now have a theory as to why Apple increased the iPhone’s screen size from 3.5 inches to 4. Apple made a short term negative design change for a long term benefit. The iPhone 5 is a temporary holdover until they figure out a design flaw. Want to know what that flaw is? It’s this:  Yep. It’s that top bar with the camera and speaker. There is a time when you have to make a tough decision to pave the way forward. Maybe the only way to make this thing go and not make the phone “feel” weird was to extend the height of the screen, and then snip off the top. ### iPhone 5:  ### iPhone 6:  I know, I know, my Photoshop skills need some work. To be frank, I just got the new iPod Touch (which is practically identical in size to the iPhone 5), and I’m kinda liking it, so I’m not actually sure if this will happen. But it’s a theory, and I think it’s plausible. If you don’t think Apple would do this, look no further than the iPod nano. 

Why the iPhone 5's four inch screen ain't such a problem.

2012-09-24T00:00:00Z tag:dlo.me,2012-09-24://archives/2012/09/24/blogging-again/ I've been taking a break from writing on this blog for way too long of a time now. Like everyone else, I suppose I can make excuses, like "I just got too busy" or "I didn't have anything to write about". Well, the first thing is true, but the second thing isn't. I have a ton of things to write about. I was chatting with a friend about this yesterday, and my theory is this: the busier you are the more you have to write about, but the less time you have to write about it. Well, I'm dead set on making the time. It's not going to just appear out of nowhere. I'm already brainstorming a couple articles. Some are programming-related, some are business-related, and others consulting-related. My main wish is to entertain and educate anyone who ends up here. So if you haven't subscribed to my RSS feed by now, please do. P.S. I wrote this post on my iPhone while waiting for lunch. You don't need a computer on your lap to write a blog post!

Why I'm going back to blogging.

2012-02-16T00:00:00Z tag:dlo.me,2012-02-16://archives/2012/02/16/box-sizing-every-browser/ Earlier today, Paul Irish wrote a post on his blog that explains how to deal with an all-too-common issue that all frontend developers face--the div that increases in width and height as soon as you apply padding to it. An example might be the below. ```html <style type="text/css"> div#example { background: yellow; height: 100px; width: 100px; } </style> <div id="example">Hello, world.</div> ``` What it looks like: <div style='background:yellow;height:100px;width:100px'>Hello, world.</div> Now that doesn't look too good! Let's say the goal is to give that yellow box a padding on all sides, let's say 10 pixels. The impatient web developer does this: ```html <style type="text/css"> div#example { padding: 10px; background: yellow; height: 100px; width: 100px; } </style> <div id="example">Hello, world.</div> ``` <div style='padding:10px;background:yellow;height:100px;width:100px'>Hello, world.</div> Oh no! Our box is bigger! That's not what we wanted. So we do some quick mental math and subtract the padding from each side from the width and height, and get this. ```html <style type="text/css"> div#example { padding: 10px; background: yellow; height: 80px; width: 80px; } </style> <div id="example">Hello, world.</div> ``` <div style='padding:10px;background:yellow;height:80px;width:80px'>Hello, world.</div> So, this method works, but is messy and hard to maintain. Every time you need to change that padding, you'll have to change two other attributes as well to keep the content the same size. Paul Irish "proposes":http://paulirish.com/2012/box-sizing-border-box-ftw/ using the @box-sizing: border-box@ attribute in your CSS, but there are a few cross-browser compatibility issues and jQuery bugs that affect its use. Additionally, if you ever want to use the "normal" box-sizing in your code, you'll have to start keeping track of which divs have @content-box@ or @border-box@ set in the @box-sizing@ attribute. That is also hard to keep track of. My recommendation is 100% portable across every browser known to man, and is extremely flexible. Presenting, the box in a box! ```html <style type="text/css"> div.padder-10 { padding: 10px; } div#example { background: yellow; height: 100px; width: 100px; } </style> <div id="example"> <div class="padder-10">Hello, world.</div> </div> ``` <div style='background:yellow;height:100px;width:100px'><div style='padding:10px;'>Hello, world.</div></div> The outer div will maintain its height and width, and the responsibility for the padding is passed to the inner div.

How to pad divs without having to change their height and width.

2012-01-18T00:00:00Z tag:dlo.me,2012-01-18://archives/2012/01/18/stop-sopa/ I am participating in an Internet-wide blackout in protest of two dangerous bills, the [Stop Online Piracy Act (SOPA)](http://hdl.loc.gov/loc.uscongress/legislation.112hr3261) and the [Protect IP Act (PIPA)](http://hdl.loc.gov/loc.uscongress/legislation.112s968), that are soon up for vote in the United States Senate and House of Representatives. Please take some time to read about how these bills are dangerous and why they affect you, even if you don’t live in the U.S., by following the links below. <ul class='styled'> <li> [A technical examination of SOPA and PROTECT IP](http://blog.reddit.com/2012/01/technical-examination-of-sopa-and.html</li>) <li> [Stop American Censorship Website](http://americancensorship.org/</li>) <li> [Google: End Piracy, Not Liberty](https://www.google.com/landing/takeaction/</li>) </ul> Other interesting links: <ul class='styled'> <li> [Why I’m a Pirate!](http://ploum.net/post/im-a-pirate</li>) <li> [The Day the LOLcats Died (video)](http://www.youtube.com/watch?v=1p-TV4jaCMk</li>) <li> [I’m Pirating the Next Version of Windows](http://littlebitofcode.com/2012/01/18/im-pirating-the-next-version-of-windows</li>) <li> [Representative Lofgren (CA): do not underestimate the power that you have](http://lofgren.house.gov/index.php?option=com_content&view=article&id=698:rep-lofgren-the-battle-over-sopa-pipa-do-not-underestimate-the-power-that-you-have&catid=22:112th-news&Itemid=161) <li> [Hollywood Finally Gets A Chance to Break the Internet](https://www.eff.org/deeplinks/2011/10/sopa-hollywood-finally-gets-chance-break-internet</li>) </ul> If you live in the U.S., please take the time to email or call your representative or senator so that they know that you are **opposed** to these pieces of legislation. 2011-05-05T00:00:00Z tag:dlo.me,2011-05-05://archives/2011/05/05/what-to-do-when-your-site-goes-viral/ My [last post](/breakup-notifier) told the story of how I made [Breakup Notifier](http://www.breakupnotifier.com/) and how it exploded in popularity across the Internet. This post is a technical expose on _how_ I kept it from going down. I would be lying if I told you the application as I wrote it from day one had _any_ semblance to what it looked like just a few days later. One thing you **must** accept is that unless you have a crystal ball, it will be nearly impossible to forecast the issues that will arise in your application. No matter what you do, there will be bugs that you didn’t anticipate, features that should have been obvious, and money that you could have saved. Be nimble and unafraid to rewrite your code. It will happen, so be prepared. I hope that what I write here today can help you avoid the problems I faced if you ever happen to be in the same position. Let the nerdery begin. ## The Foundations I automate as much of what I do as I possibly can. I do this to keep my development process very fast. It means that I can get a small prototype of an idea together in a small fraction of the time that it would take were I to build everything from scratch. Additionally I have a small store of idioms in my head to take things to an MVP-level very quickly. Breakup Notifier was based on a [project skeleton](https://github.com/lionheart/django-on-appengine) I have for [Django](http://www.djangoproject.com/) projects running on [Google App Engine](http://code.google.com/appengine/). Hosting the project on Google’s infrastructure was the best architectural choice I made. I can’t claim too much credit though; the fact is that I didn’t want to be bothered with systems administration that weekend. Auto-scaling was just a much-appreciated added bonus. The lessons I write about below are applicable to every site that faces a deluge of traffic. Hopefully you won’t have to make the same mistakes I did =D. ### Lesson 1: If you can do something on the frontend with the same performance as doing it on the backend, do it on the frontend. Frontend code scales linearly with the number of users you have. Backend code scales linearly with the number of servers you have. It doesn’t take a rocket scientist to figure out which is less CPU-intensive from the server perspective. I pulled out an example to illustrate. The following is a snippet of code called after a user signs in from Facebook. It’s from the first working version of the application. ```python def facebook_oauth_complete(request): if 'access_token' in request.session: return HttpResponseRedirect("/") code = request.GET['code'] params = { "client_id": settings.FB_APP_ID, "client_secret": settings.FB_APP_SECRET, "redirect_uri": settings.FB_REDIRECT_URI, "code": code } url = settings.FB_GRAPH_URI + "/oauth/access_token?" + urllib.urlencode(params) # Fetch the access token response = urlfetch.fetch(url) data = response.content attributes = cgi.parse_qs(data) access_token = attributes['access_token'][0] graph = facebook.GraphAPI(access_token) profile = graph.get_object("me") friends = graph.get_connections("me", "friends") friend_ids = map(lambda k: k['id'], friends['data']) ... code ... for friend_id in friend_ids: task = taskqueue.Task(url="/tasks/create-profile", method="POST", params={ "id": friend_id, "access_token": access_token, "requested_by": profile['id']}) task.add("create-profile") ... code ... ``` Deciding to pull in profile data in the backend unnecessarily (lines 27-34) was a very bad decision. **For every user** that authenticated with the application, I made on average about 150 calls to Facebook! I did this so that I could fill out the user’s friend list after they authenticated. Why I decided to do this on the backend, I have no idea, but it blew me past my task queue allotment more than once. Although this didn’t bring down the site, it did prevent relationship status notification emails from going out on schedule. Not long after this error started popping up, it occurred to me that you can easily get this information through the Facebook Javascript SDK. Instead of making Facebook API calls from App Engine, API calls are now made straight from the browser. The result is a speedier experience for the user and lower server costs. ### Lesson 2: Simple is better. And faster. And easier to maintain. The first version of BN had a complicated queue system for checking whether a relationship status had changed. Every set number of minutes, I was running a cronjob that would check for relationship status changes. It would go through every single user in the Breakup Notifier datastore (5 at a time), add a task to the task queue to check for their friends who they wanted updates for, go through each of their friends, and then call a method in the model that would call Facebook’s API and would send an email to all the people following that person saying their relationship status had changed. The above approach was riddled with issues, very slow, and impossible to debug. The code was just a mess. I spent about an hour figuring out how to optimize this process as much as possible. Compare the above to what it does now: 1. Fetch 1000 users at a time until all the users have been accessed. 2. For each user, append a task to a list. 3. When that list has 100 tasks, add them all to the task queue and make the list empty. Repeat 2 as necessary. 4. For each task, loop through all of the friends they want updates for. 5. If the current status is different than the stored status, send an email and update the friend in the database. This cut down costs by over 90%. ### Lesson 3: Minimize calls to your database, cache, or queue. This ties in a bit with the changes I made above with the relationship update scheme. If you see multiple calls to your database, use your cache first. When querying the datastore, use keys. If you see too many calls to your cache, group the calls together. If you’re using a queue, add tasks to your queue in clumps, not one at a time. Latency is a huge issue; don’t discount it. It will save you loads of issues in the future. I’ll go over how you might make these optimizations with App Engine, however the general principles apply to all frameworks and languages. If you’re building for the web, you should do everything to minimize latency and disk access. #### Memcache ```python # Bad cache.set("president:1", "nixon") cache.set("president:2", "obama") cache.set("president:3", "clinton") # Good # This is a lower latency action. Instead of calling memcache three times, you only call it once. cache.set_multi({"president:1": "nixon", "president:2": "obama", "president:3": "clinton"}) ``` #### Datastore ```python # Bad query = User.all() query.filter("email =", "user@example.com") user = query.get() # Good # Best to retrieve objects by what they are indexed by in the datastore. # I'm saving the User object with the key_name as the email address. user = User.get_by_key_name("user@example.com") # Best # Use the cache before hitting the datastore. Reading from disk is slow, # reading from memory is fast. user = mc.get("user:user@example.com") if not user: user = User.get_by_key_name("user@example.com") mc.set("user:user@example.com", user) ``` #### Task Queue ```python # Bad # The user object may change during the interval from when the task is added to # when it executes. Can cause data loss. task = taskqueue.Task(url="/tasks/my-task", params={"user": user1}) task.add("my-queue") task = taskqueue.Task(url="/tasks/my-task", params={"user": user2}) task.add("my-queue") # Good # Retrieve the user object in the task, so as to not save stale data. task = taskqueue.Task(url="/tasks/my-task", params={"user_key": "user1@example.com"}) task.add("my-queue") task = taskqueue.Task(url="/tasks/my-task", params={"user_key": "user2@example.com"}) task.add("my-queue") # Best # Only make one call to the task queue instead of two. App Engine will allow # you to add 100 tasks at a time. Use that to your advantage. tasks = [] task = taskqueue.Task(url="/tasks/my-task", params={"user_key": "user1@example.com"}) tasks.append(task) task = taskqueue.Task(url="/tasks/my-task", params={"user_key": "user2@example.com"}) tasks.append(task) queue = taskqueue.Queue("my-queue") # Add the tasks. queue.add(tasks) ``` ### Lesson 4: If all else fails, log absolutely everything. You cannot possibly know what edge cases will emerge in your application and cause you problems down the line. I don’t care how smart you think you are or how much test coverage you have. Your site will have issues, and you will not be prepared for them. The best advice I can give you is to log profusely. Let me show you an example. ```python def user_update(request): id = request.POST['id'] logging.info("id=%s" % id) cache = memcache.Client() logging.debug("memcache client instantiated") user_key = "fb:%s" % id user = cache.get(user_key) logging.debug("check if user in cache") if not user: logging.info("user not in cache") user = FacebookUserV2.get_by_key_name(id) logging.debug("user retrieved from database") else: logging.info("user retrieved from cache") try: graph = facebook.GraphAPI(user.access_token) logging.debug("graph api instantiated") profile = graph.get_object("me") logging.debug("get profile data") except facebook.GraphAPIError: # Reset the access token of the user user.access_token = None logging.warning("access token invalid") user.put() logging.debug("save user") return JsonResponse({"status": "failure"}) locale = profile.get('locale', None) if not locale: logging.warning("locale not provided") else: logging.info("locale=%s" % locale) user.locale = locale user.put() logging.debug("save user") cache.set(user_key, user) logging.debug("save user to cache") return JsonResponse({"status": "success"}) ``` If you read through the code above, you can get a pretty good sense of what I mean by “log everything”. The benefit of it is that you can quickly diagnose issues. In App Engine, the logging facility is directed to the “Logs” page in your application dashboard. If you use vanilla Django on your own server, I recommend using [Sentry](http://readthedocs.org/docs/sentry/en/latest/index.html) to zero in on your application errors. I’m not too familiar with what you might use for other frameworks or languages. If your app is throwing around 500s, it’s helpful to see what the last line logged was before it died. You can easily zone in on the broken code and diagnose the issue much more easily than it would be otherwise. #### Lesson 4.1: Monitor Your Application. Sometimes it’s just too hard to guess where your bottlenecks are. I highly recommend you use whatever monitoring tool you can get your hands on. Here are a few tools that I would recommend. Just keep in mind that the open source options will take longer to configure and in a high traffic situation might not be the best option to start off with. ##### Performance Monitoring | Name | What You Can Monitor | Notes | | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------- | | AppStats | [Python](http://code.google.com/appengine/docs/python/tools/appstats.html), [Java](http://code.google.com/appengine/docs/java/tools/appstats.html) | App Engine Only | | [New Relic](http://newrelic.com/) | [Ruby](http://newrelic.com/rails-app-performance.html), [Java](http://newrelic.com/java-app-performance.html), [.NET](http://newrelic.com/dotnet-app-performance.html), [PHP](http://newrelic.com/php-app-performance.html) | | | [Scout](https://scoutapp.com/) | [Rails](https://www.scoutapp.com/plugin_urls/181-ruby-on-rails-monitoring), [MySQL](https://www.scoutapp.com/plugin_urls/22-mysql-statistics), [Sphinx](https://www.scoutapp.com/plugin_urls/241-sphinx-monitoring), [MongoDB](https://www.scoutapp.com/plugin_urls/391-mongodb-monitoring), [Redis](https://www.scoutapp.com/plugin_urls/271-redis-monitoring), [Apache](https://www.scoutapp.com/plugin_urls/201-apache-log-analyzer) | Very extensible, open API. | | [Union Station](http://www.unionstationapp.com/) | Ruby | Free during open beta. | ##### Server Monitoring | Name | Notes | | --------------------------------------------- | ------------------- | | [Nagios](http://www.nagios.org/documentation) | Open source. | | [Monit](http://mmonit.com/monit/) | Open source. | | [CloudKick](https://www.cloudkick.com/) | Free for one server | ### Lesson 5: Don’t check your email. This isn’t as “technical” as the above points, but I think it is extremely important and relevant to what you’ll be experiencing when your site is experiencing lots of traffic. Yes, I know it’s awesome that your site is going viral. All your friends know about it, the media is talking about you, and a random person in Idaho farmland heard about your website on the radio (true story). For the first day after the Breakup Notifier craziness started, I was focused way too much on what was going on around me, instead of focusing on making Breakup Notifier better. There were too many people talking about it to even keep track of. My advice is to spend a moment to reflect on the effect you’ve had, and then get back to work. Your work does not get easier because the world is watching you! The key is getting shit done in the face of massive distraction. If your experience is anything like mine, you will get business proposals, friend requests, direct messages, and everything else in between. These all present fantastic potential opportunities, but my advice is to deal with them later or recruit a friend to manage the business-related issues. Otherwise you will explode and will get nothing done. At the time, all of those things may seem really time-sensitive. What’s more important though are your users, and the comments that people are giving you _right now_. Build them the best product you can while eyes are on you. It will make a difference. Thanks for reading. Be sure to [subscribe](http://feeds.feedburner.com/dloewenherz) if you liked this post and follow me on Twitter for the latest updates. Have a comment? Join the [discussion](http://news.ycombinator.com/item?id=2519641).

Lessons learned the hard way while scaling a viral web application.

2011-03-20T00:00:00Z tag:dlo.me,2011-03-20://archives/2011/03/20/breakup-notifier/ ## Introduction This is the story of Breakup Notifier: when I made it, what the inspiration was, how I handled it, and anything else that might be relevant. I’m writing this not only because a lot of people asked me to, but also for myself. The past month has been a little out of this world, and there’s a ton of stuff that’s been sitting in my head just waiting to be jotted down on paper (or in this case, Vim). On another note, I still can’t get over that Breakup Notifier was on [Jay Leno](http://www.nbc.com/the-tonight-show/video/thursday-march-17-2011/1314753/) (broken link as of 1/29/2016) a few nights ago (skip to 8:42 on the monologue). Insane. So I’m going to go into as much detail as I possibly can. If I left out something that you’d like to see, shoot me an email at <dan@dlo.me> and I’ll update the post. ## Thursday #### Unique Visitors: 0 It was a pretty normal night. My fiancée, her mom and I spent a few hours earlier that evening watching the season finale of [Spartacus](http://www.starz.com/originals/spartacus). After the show ended, I got back to work. A few minutes after getting back on the computer (which is also in our TV room—[noise canceling headphones](http://www.amazon.com/gp/product/B002M38I2U/ref=as_li_ss_tl?ie=UTF8&tag=dansaweblo-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=B002M38I2U) FTW!), I started listening in on a conversation they were both having regarding my soon-to-be sister-in-law. So there was this great guy who they thought was a great match for her. They were talking him up to each other, convincing themselves he was probably the perfect guy. Their hopes were dashed after checking his relationship status on Facebook (yep, he was “In a relationship”). This was a problem just itching for a solution. Since programmers are kind of like real life superheroes, I decided to do my part. I proposed a system where they would get an email the instant said guy changed his relationship status. They absolutely loved it. ## Friday #### Unique Visitors: 0 Here’s the first chat I had where I talked about the idea. <table> <thead> <tr> <th>12:07 PM</th> <th><strong>me:</strong> i’m going to take an hour break</th> </tr> </thead> <tbody> <tr> <td></td> <td>i want to make a joke website</td> </tr> <tr> <td></td> <td><strong>Christopher:</strong> haha</td> </tr> <tr> <td></td> <td>nice</td> </tr> <tr> <td>12:08 PM</td> <td>what’s the idea</td> </tr> <tr> <td></td> <td><strong>me:</strong> <a href="http://www.breakupnotifier.com">breakupnotifier.com</a></td> </tr> <tr> <td>12:09 PM</td> <td>login through facebook and mark hot girls you’re interested in who<br /> are in a relationship</td> </tr> <tr> <td></td> <td>it will email you when they break up</td> </tr> <tr> <td></td> <td><strong>Christopher:</strong> thats brilliant</td> </tr> <tr> <td></td> <td>im waiting out like 5 relationships</td> </tr> <tr> <td></td> <td><strong>me:</strong> haha</td> </tr> <tr> <td>12:09 PM</td> <td>i’m giving myself a time limit of one hour</td> </tr> <tr> <td></td> <td><strong>Christopher:</strong> you got it</td> </tr> <tr> <td></td> <td><strong>me:</strong> will let you know when its done</td> </tr> <tr> <td></td> <td><strong>Christopher:</strong> sweet</td> </tr> </tbody> </table> Somewhere in the middle of this conversation I purchased the domain breakupnotifier.com (if you’re wondering why the confirmation email says 1:08pm, it’s because GoDaddy is located in Arizona). And no, I didn’t follow through with my initial time estimate :P. <figure>  </figure> I had enough feedback to actually build something. The git repo was started at 12:40pm with the intent to actually work on this thing the next day. In the meantime, I thought it would be fun to get some more feedback from some other friends. I thought the idea was hilarious so I naturally wanted to tell as many people as possible. <table> <thead> <tr> <th>1:14 PM</th> <th><strong>me:</strong> <a href="http://www.breakupnotifier.com/">http://breakupnotifier.com/</a></th> </tr> </thead> <tbody> <tr> <td></td> <td>lol</td> </tr> <tr> <td></td> <td><strong>Richard:</strong> dude i think it can go even further than that - check out what im<br /> about to send you</td> </tr> <tr> <td></td> <td><strong>me:</strong> haha</td> </tr> <tr> <td>1:15 PM</td> <td>quick which icon set do you like better</td> </tr> <tr> <td></td> <td><a href="http://www.istockphoto.com/stock-illustration-10194726-valentine-s-day-icon-set.php">http://www.istockphoto.com/stock-illustration-10194726-valentine-s-day-icon-set.php</a></td> </tr> <tr> <td></td> <td><a href="http://www.istockphoto.com/stock-illustration-14678717-heart-icons.php">http://www.istockphoto.com/stock-illustration-14678717-heart-icons.php</a></td> </tr> <tr> <td></td> <td><strong>Richard:</strong><br /> <a href="http://www.istockphoto.com/stock-illustration-10194726-valentine-s-day-icon-set.php">http://www.istockphoto.com/stock-illustration-10194726-valentine-s-day-icon-set.php</a><br /> gives you more choice</td> </tr> <tr> <td>1:16 PM</td> <td>right?</td> </tr> <tr> <td></td> <td><strong>me:</strong> yeah agreed</td> </tr> <tr> <td></td> <td>i’m going with it</td> </tr> </tbody> </table> I purchased $18.50 worth of credits on iStockphoto and downloaded the Valentine’s Day icon set. After it was downloaded, I spent about 30 minutes inside of Fireworks turning this  into  I went with the Facebook color scheme primarily because I figured it’s what people are accustomed to and comfortable with. I think I spent the next few hours working on some other things. At 3 I started tinkering with the Facebook Graph API. Here’s a chat from around then. <table> <thead> <tr> <th>3:30 PM</th> <th>are you up to anything?</th> </tr> </thead> <tbody> <tr> <td></td> <td><strong>me:</strong> making a silly web app</td> </tr> <tr> <td></td> <td><a href="http://breakupnotifier.com">breakupnotifier.com</a></td> </tr> <tr> <td></td> <td>my break for the day</td> </tr> <tr> <td></td> <td><strong>James:</strong> hahaha</td> </tr> <tr> <td></td> <td>what is it?</td> </tr> <tr> <td>3:31 PM</td> <td><strong>me:</strong> rachel (my fiancée) was talking with her mom last night about<br /> her sister, and how they wanted to set her up with this guy</td> </tr> <tr> <td></td> <td>they checked on facebook and he was in a relationship</td> </tr> <tr> <td></td> <td>so i was like, would you guys want to be notified when his relationship status<br /> changes?</td> </tr> <tr> <td></td> <td>they were like YESSS</td> </tr> <tr> <td></td> <td>and an idea was born</td> </tr> <tr> <td></td> <td><strong>James:</strong> hahaha</td> </tr> <tr> <td>3:32 PM</td> <td>so whenever your friends</td> </tr> <tr> <td></td> <td>relationship status changes</td> </tr> <tr> <td></td> <td>you get notified</td> </tr> <tr> <td></td> <td><strong>me:</strong> well you can pick and choose</td> </tr> <tr> <td></td> <td><strong>James:</strong> haha thats funny</td> </tr> <tr> <td></td> <td><strong>me:</strong> so you can pick only the hot girls</td> </tr> <tr> <td></td> <td>or something</td> </tr> <tr> <td></td> <td><strong>James:</strong> i like it</td> </tr> <tr> <td></td> <td><strong>me:</strong> hah</td> </tr> <tr> <td></td> <td>cool</td> </tr> <tr> <td></td> <td>it’ll be on hn in a few hours</td> </tr> <tr> <td></td> <td><strong>James:</strong> did you start building yet?</td> </tr> <tr> <td></td> <td><strong>me:</strong> i hope</td> </tr> <tr> <td></td> <td>yes</td> </tr> <tr> <td></td> <td><strong>James:</strong> the fbook api is pretty easy right</td> </tr> <tr> <td></td> <td>did you use a library</td> </tr> <tr> <td>3:33 PM</td> <td><strong>me:</strong> yeah, it’s super simple</td> </tr> <tr> <td></td> <td>although it’s being annoying right now</td> </tr> <tr> <td></td> <td>with the permissions crap</td> </tr> <tr> <td></td> <td><strong>James:</strong> oh</td> </tr> <tr> <td></td> <td><strong>me:</strong> i’m asking for certain permissions but the api isn’t giving me the info<br /> i asked for</td> </tr> <tr> <td></td> <td>pretty weird</td> </tr> <tr> <td>3:34 PM</td> <td><strong>James:</strong> thats annoying</td> </tr> <tr> <td>3:35 PM</td> <td><strong>me:</strong> i wonder if its because of my draconian privacy settings</td> </tr> <tr> <td></td> <td>…</td> </tr> <tr> <td>3:39 PM</td> <td>eff this</td> </tr> <tr> <td></td> <td>i give up</td> </tr> </tbody> </table> Yep, believe it or not, Breakup Notifier almost never happened. I worked on [Crate](http://letscrate.com) for the rest of the day. ## Saturday #### Unique Visitors: 0 I decided to finish what I started. I knew there was something here and I was not going to give in so easily. I figured out what my issue was with the Facebook API pretty quickly (I was asking for the wrong permissions). I finished the web application sometime around mid-afternoon Saturday, [tweeted about it](https://twitter.com/dwlz/status/39225184854622209) (here’s [the one](https://twitter.com/BreakupNotifier/status/39207349247934464) from the [@BreakupNotifier](http://twitter.com/breakupnotifier) account), and then submitted the post to [Forrst](http://forrst.com/posts/Breakup_Notifier-guq). I knew this site had some major potential, so I was seriously pushing it to my friends and followers. Despite all this, no one really cared. <figure>  </figure> ## Sunday #### Unique Visitors: 15 The next day, I wanted to give Breakup Notifier one last chance. I submitted Breakup Notifier to HN at about 4:00pm ([Show HN: Breakup Notifier](http://news.ycombinator.com/item?id=2243650)). I distinctly recall that it almost fell off the new page and sat at around 2-3 upvotes for about an hour. I think by the time an hour passed, I’d given up on it and started watching Batman Begins with my fiancee. I checked on my computer halfway through the movie, and somehow it made it to the front page! <figure> |\_. 6:35 PM|**Carter:** kudos on the hn post| |\_. 6:40 PM|**me:** haha wow| ||it’s gotten on the front page?| ||**Carter:** yeah| ||**me:** oh wow| </figure> <hr/> <figure> |\_. 7:07 PM|**James:** dude!| ||breakup notifier| ||is 3rd on hn| ||thats awesome| </figure> On Sunday night at around 7:30pm PST, had reached the \#1 spot; by 9:19pm, the site had over 12,000 unique visitors. At 11:12 PM that night, the impossible happened. <figure>  </figure> Holy. Shit. I knew what the problem was. Alexia wasn’t getting emails since I was only checking for relationship status changes once every 24 hours. As a test, Alexia broke up with herself, saw no email, and was very sad. I promise, this was the closest I’ve probably ever gotten to having a psychological breakdown. I definitively remember my hands shaking as I wrote the code to increase the polling frequency. My adrenaline levels must have been through the roof. I knew the site had viral potential—I just couldn’t really believe it was actually happening. <figure>  </figure> ## Monday #### Unique Visitors: 18,805 I pushed the new code and waited. And waited. Next morning, at around 11am, it happened…Breakup Notifier was [crunched](http://techcrunch.com/2011/02/21/stalkbook/). On another note, this is the point where I was also given introduced to all the wonderful TechCrunch commenters. I was blown away by the malice of some of these people. It’s really kind of depressing, but that’s the subject of another post. Moving on… <figure> |\_. 10:06 AM|**Matthew:** [http://www.cnn.com/2011/TECH/social.media/02/21/facebook.relationship.status/index.html?iref=NS1](http://www.cnn.com/2011/TECH/social.media/02/21/facebook.relationship.status/index.html?iref=NS1|) ||nice| ||**me:** holy shit| ||hahahaha| ||CNN!| </figure> Shit got real right about now. Within a few hours, I was contacted by ABCNews.com, MSNBC, GlobeAndMail, Technorati, PC Magazine, The New York Daily News, Information Week, the Huffington Post, The Washington Post, The NY Times Magazine, the International Business Times, and the list goes on. Here’s the traffic graph for Monday. <figure>  </figure> ## Tuesday #### Unique Visitors: 116,669 It may seem crazy, but by Tuesday I was in a state of immense denial. I truly could not believe what was going on. Every morning for the next few days, I felt like I was waking up to a dream. It was all too surreal and odd for me to really explain. It was kind of like a ton of eyes were on me and what I was going to do. Hell, a bunch of people thought Facebook would ban me (lulz). I teased fate a little bit when I commented that I would feel [honored](http://news.ycombinator.com/item?id=2243708) if Facebook were to ban Breakup Notifier. Heh. ## Wednesday #### Unique Visitors: 164,309 I woke up to this email in my inbox (sent at 4:38AM). <figure>  </figure> I suppose I was overstepping my bounds ;). I would later learn that the reason the application was disabled was because of a little “Share” button. When a Facebook application uses the feed dialog to publish a story through an application, it’s actually a wrapper over the Facebook API. So when a user hides or blocks content from one of these posts, it can actually penalize the application that allowed the user to share. Apparently, a lot of people were hiding these shares from friends, so the application was banned automatically. I didn’t have as large a problem with the reason it was banned as much as the amount of time it took for Facebook to communicate this to me. I immediately sent an email to Alexia at TechCrunch letting her know what was going on. I figured this would be an interesting story and I was lucky enough that she decided to write [about it](http://techcrunch.com/2011/02/23/breakup-notifier/). I don’t like sitting around like a lame duck, so I did what any self respecting hacker would do. I decided to channel the traffic to Breakup Notifier to another website while all this Facebook stuff was being sorted out. I would call it [Crush Notifier](http://www.crushnotifier.com). Amidst all the press craziness that was happening about now (and a packed inbox), I pumped out the site. ## Friday #### Unique Visitors: 197,999 Breakup Notifier traffic was really dying down by now, especially since the site hadn’t been able to do anything for about two days, per Facebook’s ban of the application. With Crush Notifier, I planned to integrate Facebook Credits, write a revamped backend (to handle a lot more users), and basically make everything faster. It took a few hours, and by evening, I was ready to launch the site. I wrote a [blog post](http://blog.crushnotifier.com), submitted it to [HN](http://news.ycombinator.com/item?id=2264660), and braced myself. TechCrunch soon posted [about it](http://techcrunch.com/2011/02/25/crushnotifier/). Nothing really prepared me for the sort of vitriol that I would soon be the target of. In the TechCrunch post, I was called things that I’d rather not repeat on this blog. There’s no need to repeat them here. I went to bed feeling extremely depressed that night. Rachel told me not to worry—this is just how some people are, and there’s not really much one can do about it. It still just stuck with me. There isn’t really anything much worse than random people accusing you of being a cheat, liar, and fraud. ## Saturday #### Unique Visitors: 203,171 I could never have predicted what would happen next. A fellow HNer, [jarin](http://news.ycombinator.com/user?id=jarin), wrote [a post](http://news.ycombinator.com/item?id=2265161) entitled *Crush Notifier backlash: if Dan Lowenherz is a crook so is your favorite company*. He spelled my last name wrong, but it really didn’t matter. I know this might sound bizarre, but I never felt more at home in any community of people. Everyone chimed in about how this happens to everyone; it’s just how the world works. Main thing I learned from this experience is that you just have to ignore everyone who tries to put you down. It **will** happen. As Chad Etzel [remarked](http://news.ycombinator.com/item?id=2265200): “Haters gonna hate.” Nothing you can do about it. ## Conclusion #### Unique Visitors: 237,851 and counting You’re probably wondering how much money I made with all this attention. Short answer: nothing. Long answer: had I chosen to put ads on the site as it was exploding in popularity, I believe I would have probably made a few hundred dollars. As for Facebook’s ban of the application, it was revoked on Sunday. On Monday morning, I relaunched the application (it goes without saying that I removed the ‘Share button’). I also would like to add that I am so incredibly thankful for the HN community and the positive energy that it promotes. There is nothing quite like it. I love all of you. My next post is going to be a bit more technical in nature. I’m going to write what the scaling/coding aspect of this entire experience was like, so if you’re at all interested in that, be sure to [subscribe](http://feeds.feedburner.com/dloewenherz)! Have a comment? Join the [discussion](http://news.ycombinator.com/item?id=2348702).

How a harmless conversation led to one of the most viral websites ever.

2011-02-05T00:00:00Z tag:dlo.me,2011-02-05://archives/2011/02/05/how-i-deal-with-content-overload/ Over the past couple of years, I've been slowly honing a way to manage the deluge of online content that presents itself to me. The problem nowadays isn't that there isn't enough interesting content. It's that there is *way* too much to handle. Follow these steps, and your free time will open up measurably: 1. Get an account at [Pinboard.in](http://pinboard.in) and [Instapaper](http://instapaper.com). I love Pinboard because the service is easy to use, the developer is on top of his stuff, and it works like a charm. It's worth the $10 it costs today. I actually have an archival account, which costs $25/year, so if I bookmark something and the site goes down, I will still have a saved copy. 2. Set up two bookmarklets: the Instapaper "Read Later" one and the Pinboard "popup." 3. Add your Instapaper feed URL to Pinboard. This will sync your Instapaper and Pinboard accounts. Also, make sure that new bookmarks are set as "unread." 4. Take 5 minutes once each day to browse through [Reddit](http://reddit.com), [HN](http://news.ycombinator.com), [StumbleUpon](http://stumbleupon.com), or whatever it is you use to find content. Click on "Read Later" for blog posts you find interesting and use the Pinboard popup for items of interest, such as tutorials, documentation, or websites you might want to check out. Tag them appropriately. 5. Put aside an hour at the end of the day to read up on the Instapaper items you marked and your unread bookmarks on Pinboard on your iPad[^1], Kindle, computer, etc. Can't say what this will do for you, but I've easily gained a couple more hours of productivity each day from this. It's easier to stay focused on the task at hand when you schedule your procrastination. [^1]: I'm considering writing a Pinboard.in iPad application. I couldn't find one that I'd use. Does anyone else have a need for this?

My strategy for dealing with a lot of information. Try it out.

2011-01-20T00:00:00Z tag:dlo.me,2011-01-20://archives/2011/01/20/crazy/ I'm a <strong>huge</strong> fan of the <a rel="nofollow" href='http://en.wikipedia.org/wiki/Pareto_principle'>Pareto Principle</a>. Almost everybody, from <a rel="nofollow" href='http://www.gladwell.com/'>Malcolm Gladwell</a> to <a rel="nofollow" href='http://www.fourhourworkweek.com/'>Timothy Ferriss</a>, share my obsession with it. The idea is that a small amount of the inputs are the cause for most of the outputs. The most productive periods of your life are those where you act crazy. Being unpredictable is a necessary ingredient for serendipity, and it's always those serendipitous moments where the biggest changes in your life will happen. You've gotta shake things up. <strong>As much as you can</strong>. If most change comes from being unpredictable, why not embrace it? Make it who you are. Start doing things you don't normally do. Talk to people you don't normally talk to. Get out of your comfort zone. You'll be happier because of it.

Sometimes being unpredictable is the best thing you can do. Try it out sometime.

2010-08-01T00:00:00Z tag:dlo.me,2010-08-01://archives/2010/08/01/flying-on-the-cheap/ First, a brief re-introduction. [A few months back](/when-to-buy-airline-tickets), I thought it would be a cool experiment to use crowdsourcing to optimize my choices in buying airplane tickets. Unfortunately, though, I didn’t really finish what I started. This post is going to fix that. To spice things up a little bit, I added over 3000 data points to this analysis over the last one, so things are not only going to look cooler, but they’re going to be more accurate as well. Before I delve into the actual statistics of all the information I gathered, I’m going to give a little overview of how I actually made this all happen, or you can [skip straight to the results](#results). ## A Crash Course in Mechanical Turk Amazon has a nice and easy interface for adding batches of jobs like these. You can specify a description, some keywords (this makes it easier for “Turkers” to find your job), and various other variables, such as the time alotted per assignment and the amount of time before a HIT (Human Intelligence Task) expires.  Once you do this, you basically have to figure out a few more things. - A data set. - A way for Turkers to enter in the data they find (using the form interface on the MTurk website). This is a short snippet of code that generated the random data for the trips. ```python SECONDS_IN_YEAR = 28425600 CITIES = ["MIA", "LAX", "JFK / LGA", "SFO"] def write_random_trips(n): for i in xrange(n): row = random.sample(CITIES, 2) r = random.randint(0, SECONDS_IN_YEAR) d = datetime.fromtimestamp(time.time() + r) row.append(d.strftime("%B %d %Y")) print "%s,%s,%s" % (row[0], row[1], row[2]) ``` I uploaded it as a CSV, and Amazon automatically divvied up the prompts to the Mechanical Turk users. ## The Waiting Game This is, by far, the most gut-renching part of the Mechanical Turk experience. After submitting your batch of HITs to Amazon, it can take anywhere from 8 to 12 hours for them all to get finished. As people complete the tasks, you have the ability to accept or reject the submissions (when you reject a submission, you should give a reason to the Turker who completed it). This is a little tedious, but it’s pretty important if you want to weed out bad data. Once the batch is complete, you have the option to export the results to a CSV. From there, you can do whatever you’d like with the data. I packaged it up nicely so that Mathematica could make some sense of it. ## Results The first thing I wanted to see once I had everything in hand was pretty obvious (at least to me): What is the cheapest time of year to fly? The following is a chart plotting day-of-year versus average price for that day.  The above isn’t so useful. It’s pretty hard to extrapolate a trend from really jagged data like this. I decided that a 30-day moving average would be a lot smoother and easier to interpret. Below is the result. The average ticket price was \$160.97 and the data stayed within a small range of the average (the standard deviation was \$13.00).  Another thing I wanted answered was the best **time** of day to order airplane tickets. Unfortunately, I didn’t get many data points for periods in the middle of the night, so the data is a little skewed. The graph below charts time against ticket price, with the size of the circle representing the number of data points for that time period.  Here’s a table of average prices for each day of the week. Thursday is the most expensive day to fly, and Saturday is the cheapest. | Day | Average Price | |-----------|---------------| | Monday | \$159.40 | | Tuesday | \$159.36 | | Wednesday | \$161.91 | | Thursday | \$174.45 | | Friday | \$163.28 | | Saturday | \$148.03 | | Sunday | \$166.98 | And here I have it split up by city. ### Miami | Day | From | To | |-----------|----------|----------| | Monday | \$167.12 | \$164.45 | | Tuesday | \$169.38 | \$169.66 | | Wednesday | \$172.03 | \$166.51 | | Thursday | \$191.08 | \$195.11 | | Friday | \$172.70 | \$169.38 | | Saturday | \$158.23 | \$162.29 | | Sunday | \$169.63 | \$199.33 | ### LA | Day | From | To | |-----------|----------|----------| | Monday | \$156.99 | \$149.67 | | Tuesday | \$151.45 | \$148.48 | | Wednesday | \$155.18 | \$156.65 | | Thursday | \$161.57 | \$160.00 | | Friday | \$158.27 | \$155.47 | | Saturday | \$139.68 | \$136.49 | | Sunday | \$162.16 | \$161.00 | ### NYC | Day | From | To | |-----------|----------|----------| | Monday | \$161.94 | \$161.81 | | Tuesday | \$161.35 | \$163.19 | | Wednesday | \$161.08 | \$160.20 | | Thursday | \$170.15 | \$170.40 | | Friday | \$169.09 | \$166.63 | | Saturday | \$144.02 | \$149.75 | | Sunday | \$172.48 | \$152.12 | ### SFO | Day | From | To | |-----------|----------|----------| | Monday | \$151.42 | \$160.71 | | Tuesday | \$156.91 | \$155.49 | | Wednesday | \$158.25 | \$165.31 | | Thursday | \$175.48 | \$173.05 | | Friday | \$152.99 | \$161.57 | | Saturday | \$146.91 | \$144.94 | | Sunday | \$162.94 | \$162.26 | ## Interpretation and Conclusion Without a doubt, the fact that LA and San Francisco are so close to each other definitely had an effect on the final results. The cities I chose were, more or less, based on my personal travel habits, so it’s highly likely that someone with a wildly different grouping of “home airports” would get different results. No matter what, though, it seems that Saturday is the cheapest day to fly, and Thursday is the most expensive. For the best deals, you should buy your tickets from 6:00-8:00am PST. For the cheapest trip, plan for something in mid-May. If you’d like to check out the dataset, or have any questions about how I created these graphs, please don’t hesitate to send me an email.

Looking to go on a flight soon? I crowdsourced data using Mechanical Turk to figure out the best times to purchase tickets. Check out the results in this post.

2010-07-13T00:00:00Z tag:dlo.me,2010-07-13://archives/2010/07/13/introspection-python-vim/ As much as I like my [Vim](http://www.vim.org), there are times when I want it to be just a little cooler. So in XCode, for example, there’s this nice introspection feature that I’ve been dying to have for a while now.  Needless to say, I couldn’t stop myself from bringing this functionality to Vim. Here is the finished result:  If pictures aren’t enough, you can also [watch a movie](http://vimeo.com/13312423) if you’re still not convinced :). ## Instructions 1\. Install [snippetsEmu](http://www.vim.org/scripts/script.php?script_id=1318) to `~/.vim/plugins/`. 2\. Download [introspection.vim](http://dlo.s3.amazonaws.com/downloads/introspect.vim) and also move it to `~/.vim/plugins/`. 3\. (optional) Add `autocmd BufEnter *.py call FilterFile()` to your `.vimrc`. This calls the introspection function whenever you open a buffer containing a Python file. ## Usage 1\. Type up a function. E.g. <code> def insert_into_db(name, company): pass </code> 2\. Save the file and type `:edit!`. 3\. Now you can type `insert_into_db<tab>` and a snippet will appear. Use `<tab>` to navigate to the next field. 4\. [Fork the code on Github](http://github.com/dlo/introspect.vim) and get it working for other languages!

A little Vim plugin I wrote that adds introspection to Python code.

2010-05-23T00:00:00Z tag:dlo.me,2010-05-23://archives/2010/05/23/flight-ticket-distribution/ A few months back I did some research into ticket prices from and to four major cities in the U.S.: Miami, San Francisco, Los Angeles, and New York. I promised to release the results but just never got around to it. Because I want to give you all **something** to look at, behold this plot I made with the aggregate information between the four cities. The y-axis is the ticket price in dollars. I’ll get some more detailed analysis out of the data by the end of the week. Promise promise promise. <img src="/images/flights-prices-dates-1.png" class="bigimage" alt="" />

A small chart that shows the best times to buy airline tickets.

2010-05-20T00:00:00Z tag:dlo.me,2010-05-20://archives/2010/05/20/vim-with-relative-line-numbers/  I’ve been waiting for ages for relative line numbering (a patch that’s been available for quite a while) to be officially merged into [Vim](http://code.google.com/p/vim/). Well, that day has come :). Why would you want relative line numbers for your editor? Well, for one, it makes those commands dependent upon line movement exponentially easier. Instead of having to guess how many lines up or down you may want to perform a command over, you’ve got the answer right in front of you. Check it out!  I’m running Snow Leopard, hence the odd `ARCHFLAGS='-arch x86_64'` prefix. I have a few libraries that have been compiled for both 32-bit and 64-bit architectures, and if I don’t tell Vim which ones to use, it throws a hissy fit. ```bash $ hg clone https://vim.googlecode.com/hg/ vim $ cd vim $ hg checkout vim73 $ env ARCHFLAGS=’-arch x86_64’ ./configure —enable-pythoninterp \\ —enable-rubyinterp —disable-gui $ env ARCHFLAGS=’-arch x86_64’ make $ sudo make install ``` This installs your vim in `/usr/local/bin/vim` by default. To turn on relative line numbering, just run `:set rnu`. Predictably, to turn it off, use `:set nornu`.

Vim (the text editor) now support relative line numbering. This post will explain how to use it.

2010-05-07T00:00:00Z tag:dlo.me,2010-05-07://archives/2010/05/07/openprovider-a-distributed-social-protocol/ ## Intro Airplane rides are always a good time for reflection—there aren’t really any distractions (unless you’re unlucky enough to be sitting next to a crying baby) and the white noise of the engines provides for a nearly perfect brainstorming environment. The main thing on my mind last weekend was the current state of the social web—how it’s broken and how best to “fix” it. ## Facebook One of the main problems that people have with the system as it is today is that the state of your information is currently in flux—there is no way to tell if the service that you’re using today will have the same privacy policy as it will tomorrow. [Facebook](http://www.facebook.com/) is especially guilty of this alarming trend and others are likely to follow. For an illustrative timeline of Facebook’s privacy policy, check out the [post from the EFF](http://www.eff.org/deeplinks/2010/04/facebook-timeline/). The [F8 conference](http://www.facebook.com/f8/) two weeks ago unveiled Facebook’s intention to own everything you do on the Internet, from the movies you watch on [IMDB](http://developers.facebook.com/showcase/entertainment?p=imdb) to the music you listen to on [Pandora](http://developers.facebook.com/showcase/entertainment?p=pandora). Understandably, a lot of people involved in the tech industry have been [deleting their Facebook accounts](https://ssl.facebook.com/help/contact.php?show_form=delete_account) left and right. There is a lot of misunderstanding regarding *why* this is happening, and no, it isn’t because of the lack of privacy. The reason is that Facebook was sold to us on the promise that we would just be sharing our information with our friends and family, and not with advertisers and “partners”. ## The Plan For all the reasons mentioned in addition to many others, there has been an enormous push to change the way we think about social networking over the past few days. One project that’s gotten a lot of traction is [diaspora](http://www.joindiaspora.com/), a soon-to-be open source project being championed by 4 computer science students at NYU. No one has any real idea on what it is or how it will work, but I’m sure that we’ll be hearing more about them soon. I do think what they’re doing is awesome, and I hope it works. I just wanted to get my thoughts down on paper before I forgot about them (as I frequently do). What follows is my plan for a distributed social protocol. First of all, I’m going to go over the “must haves”—things that the ideal social network should have, whether it be distributed or centralized. 1. Accessibility (or Virality) 2. Privacy 3. Low barrier to entry 4. Portability ### Accessibility and Virality The first step in using a social network appropriately is finding your friends, and there’s no way to do this if all the searchable information is hidden. Complying with \#4 above makes this extra difficult. Well, there’s a way to solve it at the expense of losing “fuzzy search”, and that solution is hashing. A hash function maps a piece of data into a smaller, more manageable chunk, and may or may not be irreversible. E.g. ``` >>> sha1("dloewenherz@gmail.com").hexdigest() c4080eb3969b7c95c5e38d563e15bd2407e35153 ``` That value you see is not reversible. There is no function that will take `c4080eb3969b7c95c5e38d563e15bd2407e35153` as a parameter and will return dloewenherz@gmail.com. It’s never been done, and I don’t suspect it will be. Hashing email addresses in this manner will hide them to spammers while at the same time allowing people to search for them. “How can you search for something that you can’t decode?”, you ask. Well, the cool thing is that when people search for you, the search query is itself encoded in the same way, so if the encoded search query is equivalent to the encoded email address, then there’s a match! For more info on hashing, check out [Wikipedia](http://en.wikipedia.org/wiki/Hash_function) on the subject. This leads into my proposal that there be a global identity database that is accessible and downloadable by anyone (addresses \#3 above). It will store the encoded email (as described), plus two other pieces of information necessary to complete the puzzle. (code)\* `email` - `provider` So what’s a “provider”, you may ask? A provider is a social web service, like Facebook or Twitter, that controls a user’s information. If the maintainers of the central node start turning to the dark side, well, everyone can and will abandon them, as others already have access to the data. `provider` is in itself worthless to a spammer. ### Connections A connection is a one-way link between two users, regardless of the provider. So if I have an account with provider A and my friend has an account with provider B, I will still be able to connect with him. In essence, all profiles function like public Twitter profiles. Private profiles sound like a good idea but I’m at a total loss as to how they might be implemented. A user can only have an account with one provider at a time. After I make a connection with someone, that connection is stored in my browser. Whenever I log into my provider, it pulls the follower info from the browser and updates my profile with the appropriate information. For browsers that aren’t compatible, you can simply store all the information on a flat file on your computer, and upload it to your provider whenever you decide to change. Providers themselves can store this info. Providers should allow for export and import of follower data. If they don’t, the users can leave. The great thing about this system is that it prevents any chance of a bait-and-switch. Users will already have access to their connection data and can just start using another provider in the unlikely chance that it does happen. ### Privacy So all we’ve got now are connections. Now what? Connections link profiles to profiles. It’s up to each provider regarding what information to expose to the world at large, but it all follows the same basic protocol. So *no matter what*, if I go to ``` http://www.example.org/profile/?id=1234567 &format=json &key=6b7894e44e440f83e3fa4be3a6d98961c70d6f2f ``` I’ll get back something like ```json {"profile": {"first_name": "Dan", "last_name": "Loewenherz" }} ``` The `key` is my email address (dloewenherz@gmail.com) plus a [cryptographic salt](http://en.wikipedia.org/wiki/Salt_(cryptography)). The salt to use is something that the community as a whole could agree upon. Using an access key like the one above ensures that only people who know my email address can access my profile information. Nifty, huh? “So that all sounds cool, but hold on a second,” you say. “Didn’t you just say that it’s a huge problem that certain providers control our profile info?”. Well, the answer is yes, I did. Kind of. But I left something out. Today, if you leave Facebook, you lose all of your connections to your friends. The connections you’ve spent time “making” online are really what’s valuable—not the personal information you post to your profile. Let’s not kid ourselves, Facebook can guess most of what we think is private *just* from our connections (see [Project Gaydar](http://www.boston.com/bostonglobe/ideas/articles/2009/09/20/project_gaydar_an_mit_experiment_raises_new_questions_about_online_privacy/) for a real example). The power to control the connections is the only power that really matters. And that’s already been addressed. If a provider chooses to not let you export profile info, then that’s their prerogative. And it’s also the user’s prerogative to leave them. ## Conclusion Social networking is still in its infancy, and it shows. Even though most of what I went over in this post was extremely “high-end” and abstract, I think I make some valid points and hope it leads into more detailed discussion of how something like this might work. I think there’s something to it.

An ill-fated plan on mine to revamp the way social networking works on the Internet today.

2010-02-27T00:00:00Z tag:dlo.me,2010-02-27://archives/2010/02/27/why-piracy-is-sometimes-good-for-the-software/ This morning at the gym I started thinking about how nice it would be to be a Photoshop guru. The ability to just have an idea of an image of design in your head, open up Photoshop and come out with a snazzy-looking result after a few minutes of hard work. It would be frickin' awesome. Unfortunately all the tools I really have at my disposal for design work are HTML + CSS. Bummer. And then there's the cost. A copy of [Photoshop CS4](http://www.adobe.com/products/photoshop/compare/) (the latest version in Adobe's suite) goes for $699 if you're buying it fresh off the presses. The extended version will burn a grand in your pocket. That's way too expensive. Especially if you just want to play around (like I do). Which leads to why so many people download Photoshop off of Bittorrent. Free is free. Honestly, how many Photoshop gurus actually bought the damn thing when they were first curious about it? I doubt very many. I don't know why this is what first came to me, but I thought for a second that maybe Adobe wants its software to be pirated. Look at it this way: maybe Adobe charging such an exorbitant amount for its software is just a form of [price discrimination](http://en.wikipedia.org/wiki/Price_discrimination). They know that businesses will fork over the price as long as its not ridiculous. Hell, if you're a design firm you need Photoshop. You bite your lip, buy the software, hire a great designer (who probably learned Photoshop off of a pirated version), and move on. It sucks, but no company wants to be using pirated software. It's just not a good idea. If you don't have the money, you'll just download it. Here's a little example to illustrate why Adobe wants people to pirate their software. [Fixed costs](http://en.wikipedia.org/wiki/Fixed_cost) for software production is very, very high (programmers are expensive ;) ). On the other hand, [variable costs](http://en.wikipedia.org/wiki/Variable_Costs) are basically nil (you're basically paying for the cost of bandwidth if its download only, and the cost of shipping otherwise). A software company can charge nearly nothing for software given that people will continue to buy it forever into the future. Other products, like jet engines, have a high variable cost, so it would be pretty stupid to charge a penny for one. You'd go out of business in the blink of an eye. Now let's imagine that Adobe charged $9.99 for Photoshop CS4. I would buy it without a second thought. So would maybe other people who wanted to get their feet wet in design. Flipside: Adobe now needs to ship 70 times more copies. This is a bad business decision if more than 1 of 70 Photoshop installations are legal. I'm actually inclined to say that the number is higher. In the end, it's better business sense to keep the price at an astronomically high value and let the rest of us download the thing on Bittorrent. It's a small price to pay, but Adobe makes more money in the end. What do you think?

Thoughts on why piracy might be a good thing for the software industry after all.

2010-02-09T00:00:00Z tag:dlo.me,2010-02-09://archives/2010/02/09/is-ipad-mistake/ [The iPad](http://www.apple.com/ipad/) was announced about a week and a half ago to great fanfare amongst both the tech community and the world at large. It has been hailed as the computer for people who don't know how to use computers. What a thought. I really applaud [Apple](http://apple.com/) for this achievement. They've really gotten close to pushing something out to market what I would like to call the "sexagenarian-proof computer". This will be the computer that will not longer be called a computer by those who use it. That's probably a good thing, because mostly everything about the iPad is antithetical to everything that computers have been up until today. If you're around my age and ever had any interest in the Internet, you probably remember when you made your first website. You opened up your [text editor](<http://en.wikipedia.org/wiki/Notepad_(Windows)>), searched [Altavista](http://altavista.com/) or [Yahoo](http://yahoo.com/) for "how to make a website", and were up on Geocities within the week. I did this when I was younger, and one of the reasons I did do it was because the barrier to entry was so incredibly low. There was nothing stopping me. If I was a kid with an iPad, the barrier to actual development would be so high that I'd most likely not consider it. If the iPad is a harbinger of things to come, I'll be very uncomfortable, mostly because the kid in me probably wouldn't have made that first website. From that first website came my first interest in programming, and then my interest in Java, etc, etc. Each step of the way requires that your computer be incrementally more capable. And tinkerable. Hell, I'd be a frustrated kid if I couldn't figure out how to write a program on my iPad. I'd be even more frustrated since it would cost [$99 for my parents to fork over](http://developer.apple.com/iphone/program/) just so I could "play around" (that is the actual cost of the SDK for those who are interested in whether I'm simply pulling numbers out of my head). I believe that computers are meant to be democratic devices -- machines that anyone can use and do anything with. If it ever gets to the point when our computers are owning us rather than the other way around, the iPad will have won. Let's hope it doesn't. 2010-01-25T00:00:00Z tag:dlo.me,2010-01-25://archives/2010/01/25/querying-for-n-random-entities-using/ First, this post is 100% programming related. So if you don't care about programming, just walk away. Chances are that if you're a Python or Java developer, you've run into Google Appengine at one point or another. A perennial issue with the datastore is that there's no officially documentation on how to pull random entities from your database. Well friends, I'd like to let you in on a little secret for how you can do this. It's quite simple, actually. Let's say you have a simple model, called Users. E.g. ```python from google.appengine.ext import db class User(db.Model): first_name = db.StringProperty() last_name = db.StringProperty() email = db.EmailProperty() ``` Make another model called UserCount (this model will only have one entity, and the count will be equivalent to the number of Users that have been inserted into the database). Instantiate a UserCount entity with an initial count of 0 before you do anything. ```python class UserCount(db.Model): count = db.IntegerProperty(default = 0) ``` Simple so far. Do something similar to this when you insert a User into the database. ```python uc = UserCount.get_by_key_name("1") uc.count += 1 uc.put() u = User(key_name = str(uc.count), first_name = "Steve", last_name = "Jobs", email = "sjobs@apple.com") ``` To get random data, use Python's random module, and call the User's by their key names. Caching is recommended. You may want to do something similar to the below. ```python from random import sample from google.appengine.api import memcache uc = UserCount.get_by_key_name("1") N = 5 ids = sample(range(1, uc.count+1), N) users = [] for id in ids: user = memcache.get("user_%d" % id) if not user: user = User.get_by_key_name(str(id)) users.append(user) ``` And that is how to get random entities from the Google Appengine datastore.

Need a way to get random entities out of the Google Appengine Datastore? Here's a solution that should get you running in no time.

2010-01-24T00:00:00Z tag:dlo.me,2010-01-24://archives/2010/01/24/when-to-buy-airline-tickets/ This weekend I've been mulling over how annoying it was that there isn't really freely available data on prices for airline tickets. [Farecast](http://farecast.com), the one site that I was aware of that did *something* like this, was purchased by Microsoft and is now a part of [Bing Travel](http://bing.com/travel). So what separates the market for airline tickets and others is that ticket prices are extremely sporadic from day-to-day and even by the hour. This makes getting a really good deal on airline tickets very difficult for the average consumer. There's only one participant in the transaction that gets a good deal, and that is the airline. Their pricing schemes are actually quite ingenious when you begin to think about it. Typically, an airline will have multiple pricing ranges for seats on the same flights. That is, 10% of seats will be $50-100, 40% will be $100-$150, and 50% will be >$150. The gist of price discrimination is that there are usually people who are willing to pay more than you are for the same product or service. Instead of charging everyone a flat rate, a company will vary the price based on the customer. Simply put, movie theaters don't have senior and student discounts because they're particularly nice, they do it because they make more money this way :). See [Wikipedia](http://en.wikipedia.org/wiki/Price_discrimination) for more information. Alright, back to the meat and potatoes. After some thought, I decided [Mechanical Turk](http://mturk.com) would be the best way to gather this info. So I got down and wrote a Python script that randomly generated 500 origin cities, destination cities, and dates within the next 329 days (it seems most airlines don't give price information past this amount of time). The cities were only the ones I was interested in (Miami, LA, NYC, and San Francisco). I set up the job, uploaded the batch of data, and salivated throughout the night. When I woke up today, I got down to analyzing the data. Some interesting patterns emerged. First of all, Wednesday is (on average), the most expensive day to travel, unless you're flying from LA, where a Thursday ticket will cost you on average $500 to Miami, NYC, or SF. Miami was the worst city to fly from, with the most expensive day costing a traveler about 564 bucks.  Here are average weekly ticket prices between the four cities over this year.  Prices appear to be highest for the next week, then hit a yearly low in late February, when the average ticket price hits $109.13. Cheap. Prices are a little unpredictable over the summer, hit highs in early July and late September, and plateau over the holiday season. We'll see how much of this is due to the day that this data was gathered. How about cheapest airlines? Well, you may be surprised about this as well. | Airline | Number of tickets | |--------------------|-------------------| | Delta Air Lines | 100 | | United Airlines | 62 | | AirTran Airways | 54 | | US Airways | 36 | | Alaska Airlines | 27 | | Virgin America | 20 | | Continental Airlines | 10 | | Frontier Airlines | 7 | | JetBlue Airways | 7 | | Spirit Airlines | 3 | | Midwest Airlines | 1 | | Other | 1 | Southwest didn't make it *once*! Now that was a surprise. This is only the tip of the iceberg. Over the next week, I'll keep listing HITs on Mechanical Turk, and should have some more robust data on *when* the best times to buy tickets are. Stay tuned...

Looking to go on a flight soon? I crowdsourced data using Mechanical Turk to figure out the best times to purchase tickets. Check out the results in this post.

2010-01-09T00:00:00Z tag:dlo.me,2010-01-09://archives/2010/01/09/my-social-news-escape-day-4/ Four days has gone by and I haven't visited [Reddit](http://reddit.com/), [Digg](http://digg.com/), or HN. When I originally started this experiment, I thought that by this time I would be struggling to make it through the day without a burning desire to start typing "redd" into my address bar. Strangely enough, this hasn't yet happened to me. In fact, I have basically forgotten about social news altogether and am have shifted over primarily to Twitter and Google Reader. I've noticed that if you follow reliable people and subscribe to good blogs, your sources of information will usually be well-vetted and there isn't much of a need to "filter" the useless junk. I think that by the time this experiment ends, I may end up sticking to it. 2010-01-06T00:00:00Z tag:dlo.me,2010-01-06://archives/2010/01/06/my-social-news-escape-day-two/ I vowed on Monday not to visit either Hacker News or Reddit.com for a week starting from yesterday. And my god has it been painful. I can hardly believe it's already been almost 48 hours. What's most unbelievable to me is how much more "stuff" I have gotten done without the constant urge to check out what's going on in the social-news-o-sphere. In many ways I feel like I'm a little disconnected, but in other ways I feel completely free. The Internet is addicting. Social news is addicting. The reason is simple. People want to be entertained and spoon-fed. We have become very dependent on having other people give us information...ANY information, even if it's completely irrelevant to our daily life. Some may argue (with good reason, might I add) that almost everything is relevant in some sense. We always have room to learn about the people around us and the events that are occurring. Unfortunately, though, the difference between news and entertainment has begun to blur beyond recognition. This is one of the problems with social news. Posts such as "[What is something really embarrassing you have done, but never](http://www.reddit.com/r/AskReddit/comments/ad1n5/what_is_something_really_embarrassing_you_have/)" are put right alongside posts like "[*IRAN*: Video shows gunman opening fire on demonstrators, who fight](http://www.reddit.com/r/worldnews/comments/akurc/iran_video_shows_gunman_opening_fire_on/)". Who is there to separate the crap from the good stuff? That's right, YOU! BTW, I am not allowing myself to click those links. I found them through Google. Reading these submissions is time consuming, and so what usually happens is that even if you don't particularly care about an IAmA about a zookeeper, you'll probably find yourself clicking on the link just to see what all the fuss is about. It's so easy, right? Right, but there's a cost. There is an almost constant distraction and urge to just "click". Stop the clicking and start the doing. I guess that's the lesson for today. 2010-01-03T00:00:00Z tag:dlo.me,2010-01-03://archives/2010/01/03/openid-clickpass-webpy-and-python/ I just spent a few hours working on implementing [Clickpass](http://clickpass.com/) with my new project built on [web.py](http://webpy.org/), so I figured I could save everyone else some time by giving some tips on how to fast forward to the actual development of what it is you want to make instead of tweaking configuration files. [OpenID](http://openid.net/) is nearly ubiquitous at this point. If you don't think you have an OpenID login, you probably do and just don't know it. [Google](http://www.readwriteweb.com/archives/google_is_now_an_openid_provider.php), [Yahoo](http://openid.yahoo.com/), [Facebook](http://developers.facebook.com/news.php?blog=1&story=246), and others have already implemented the protocol into their backends. What does that mean? Well, for one, as someone who's starting a new website or a web application, you don't need to worry about storing passwords or causing potential users a small annoyance by having to fill in yet another registration form. This is how login works in an OpenID world: a user goes to a website and clicks a "Login with OpenID" button. They are then redirected to a choice of providers (e.g. Google, Yahoo) that run all of their user accounts as OpenID accounts as well. You click your provider (let's say Google) and are then asked one more time whether it's OK to login with Google. You click sure, and you're brought back to the website and, "ta da", you're now logged in. This all seems really dandy from the user's perspective. Unfortunately there's a ton of developer mumbo-jumbo on the backend side that will make most developers puke. You can read the [specs](http://openid.net/specs/openid-authentication-2_0.html) if you'd like, but I recommend that you don't. Chances are that you won't be able to finish. As a consumer (website owner), you will need a way to authenticate users with OpenID. I chose Clickpass. They offer a nice interface with a couple of choices for providers. With [Clickpass](http://clickpass.com/), you will need to make (at minimum) two URLs to facilitate the logins. One will begin the authentication process, and another will complete it. Now for the fun part: implementation. I can't give an example use in every language (that would just be a waste of time), but I can show you how this entire process is done in [Python](http://python.org/), [web.py](http://webpy.org/), and the [python-openid library](http://www.openidenabled.com/openid/libraries/python). If you want me to walk you through the entire installation process, let me know in the comments and I'll write a follow up post. Without further adieu, here is the working code. Be sure that you sign up with Clickpass and receive a site key. Replace all the variables with the appropriate values. ```python #!/usr/bin/env python import web from openid.consumer.consumer import * # Store OpenID session variables in memory store = openid.store.memstore.MemoryStore() # Get this from the Clickpass website clickpass_site_key = YOUR_SITE_KEY # Can be something like "/openid/complete_login" complete_login_url = YOUR_COMPLETE_LOGIN_URL # Your base domain, e.g., "http://google.com" realm = YOUR_REALM urls = ( "/", "home", "/begin_openid_login", 'begin_openid_login', "/complete_openid_login", 'complete_openid_login' ) app = web.application(urls, locals()) class home(): def GET(self): return """<style> @import 'http://www.clickpass.com/stylesheets/container.css'; </style> <div id="clickpass_button" style="width: 136px; height: 18px; position: relative; z-index : 9999 ;"> <iframe src="http://www.clickpass.com/embedded_buttons/login/%s" width="136" height="18" frameborder="0" allowtransparency="true" scrolling="no" style="z-index:9999;position: absolute; top: 0; left: 0;"></iframe> </div> <script type="text/javascript" src="http://www.clickpass.com/javascripts/ClickpassPanel.class.external.js?v1"></script> <script type="text/javascript"> var clickpassPanel = new ClickpassPanel('clickpass_button');</script>""" % clickpass_site_key class complete_openid_login(): def GET(self): data = web.input() c = Consumer(web.ctx.session, store) result = c.complete(dict(data), current_url = complete_login_url) if result.status == SUCCESS: return result.identity_url elif result.status == FAILURE: return "FAILURE" elif result.status == CANCEL: return "CANCEL" class begin_openid_login(): def GET(self): data = web.input() openid_url = data['openid_url'] c = Consumer(web.ctx.session, store) auth = c.begin(openid_url) raise web.seeother(auth.redirectURL(realm, return_to = complete_login_url)) ``` 2010-01-02T00:00:00Z tag:dlo.me,2010-01-02://archives/2010/01/02/save-time-money-cancun/ I'm vacationing in Cancun right now, a place where more than a million people a year visit year-round to get away from the daily grind. Cancun is a [ecological](http://www1.american.edu/TED/cancun.htm) disaster, so I don't really recommend you come here, but if you do, I have a few tips that you might enjoy. First of all, Cancun isn't off-the-wall expensive (at least nowhere near as expensive as Miami, LA, or Vegas). On the other hand, there are definitely times you'll have when you *know* that you're being ripped off. For example, discounts are everywhere if you look for them. After my mom and I reserved a fishing trip, the concierge handed us 15% discount cards for every single restaurant in the hotel (we stayed at the Marriott Casamagna). I can't really tell you if this will happen in every single hotel on the strip, but hey, it never hurts to ask ;). Another thing that will save you money is if you book in advance or on your own. Depending upon the travel agents in the hotels you stay at will cost you an extra 10-15% on top of the base price (this is considered a convenience fee). Do some Google searches for whatever it is that you'll want to do (swimming with the dolphins, deep-sea fishing, Chichen Itza). Most of the tours and trips have websites and will take reservations online. Also, if you're planning on doing a fishing trip, see if you can find a few other people who can join up with your group. You can then charter a boat on your own, which will cost considerably less than letting the company setting up the groups for you. Next, food. I recommend you drink the water, it's really OK. If you *really* don't trust the tap water, just remember to ask for no ice at restaurants (they don't make bottled water ice in case you're wondering). But seriously, no one I've talked to has had any problems with the tap water. And it tastes fine too. Another thing. I stayed at the Hotel Casamagna. I had a few bad experiences here, if you want to know what they are, I'll write about them in a follow up. 2007-12-04T00:00:00Z tag:dlo.me,2007-12-04://archives/2007/12/04/柚子-and-红牛-—-the-best-complements-to-late-night-studying/ <p>In America, it seems that the idea of fruit vendors has in some ways died out–but here in China, it can be below freezing, with the wind hitting to the core of your bones, and the loyal sellers will still be outside selling their signature products. A few of these are pretty interesting, not things that you could find in a Gourmet Heaven near you.</p> <p>Think of a citrus four times the size of a grapefruit, that takes a butcher knife to cut open, and requires both hands to tear apart what I will refer to as its “hide”. This is a 柚子 (you4 zi), and until this very minute I had no idea that in English this is called a “pomelo” (if you’ve ever heard of this before, I’d be quite impressed, and maybe disappointed that you have no life). This fruit is pretty sweet, and according to my ex-roommate (another story), the hide keeps bugs away (upon his stating this fact, it was almost instantaneously proved false, as multiple flies swarmed us the moment we started eating these things).</p> <p>Day by day, my gut is being filled with an inexplicable urge to want to stay at this place that I’ve called home for the past six months. Everything here just seems to “work” so well. Maybe I’m overlooking some things, but life in China is pretty sweet.</p> 2007-11-20T00:00:00Z tag:dlo.me,2007-11-20://archives/2007/11/20/tired/ <p>I can’t believe I only have one more month here, more or less. It’s crazy that I’ve basically been here since July, save for a week in Miami and Chicago.</p> <p>The days are getting a lot shorter and I am getting a lot more tired. This is probably because studying characters for hours on end has somehow become a “fun” way to spend my time over here. And ever with hours of studying every day for the past several months, I’ve yet to even come close to being “fluent” in Mandarin. Sure, I can communicate, and read a lot, but I’m still not at the point of near-total comprehension.</p> <p>Short post, but my dad and sister just came, gotta go.</p> 2007-10-27T00:00:00Z tag:dlo.me,2007-10-27://archives/2007/10/27/the-computer-is-working-again/ <p>Whew. Excepting for the fact that I’ve been AWOL for the past two weeks, everything here is going great. My computer is finally working again after I bought a new hard drive at 中关村 across the street, although I did somehow manage to lose the video of Kirk talking abous 辫子s–that was truly an amazing clip. Unfortunately it was written over by another, less important, clip of a tour of a bathroom. Actually, I take that back, the bathroom clip is pretty sweet. I didn’t take it though, 新侃 did, so he gets all the credit.</p> <p>Anyways, the last week has been getting a hell of a lot colder, which especially sucks being that I don’t have that much cold weather clothing here. Looks like I’m going to need to buy some.</p> <p>Well, yesterday, 新侃 and I went to 师大 to meet up with one of my PiB teachers–刘安敏. We walked around the campus a bit and my roommate might have thrown too much information to her face about exactly the type of Chinese I had been learning lately. In fact I don’t even know the characters for these words so even if I wanted to write about them here I wouldn’t be able to. We went to her dorm, the entrance of which was the home of a prominent sign which forbade men from entering the women’s rooms. Damn. In any case, we sat down in the common room, if it could be called that, and talked for a little bit.</p> <p>We went to lunch not long after (actually, it was kind of long after, because I was the one asked to find a restaurant, and unfortunately the only restaurants I really know in the 师大 area are those that we all went to during the summer with our teachers for Chinese language table–this is excepting McDonald’s, of course). Then 新侃 forgot about some meeting that he had agreed to go to at 2, so we had to run out a bit early. Whatevs.</p> <p>The rest of the day was good, with another excursion off-campus (eating some good ol’ 菠萝饭). And then after dinner I had this PKU-Yale meeting which was explaining the program to some PKUers–yeah, I was the only white guy there. Although towards the end of the meeting, I was asked to say something in Chinese, to which I promptly stated the unofficial motto of 元培 college: 猥琐是一种气质，自恋是一种美格。</p> <p>And with that, I am going.</p> 2007-10-13T00:00:00Z tag:dlo.me,2007-10-13://archives/2007/10/13/yunnan-and-the-tales-of-a-broken-computer/ So, almost two weeks ago, I set off to 云南 and 贵州 as a part of the PKU Program, half because we wanted to teach middle-schoolers some English and participate in the Yale Club of Beijing’s book-giving project, and half because we wanted to explorer a part of China beyond the borders of 北京. The trip was incredible. The first day started with an early wakeup at around 6 in the morning to catch a bus that was supposed to pick us up at the Southeast Gate at Peking University…unfortunately the bus missed its planned appointment and we all took taxis to the airport. We got to the gate with barely enough time, we were actually lucky that they printed our tickets. Kirk forgot his passport so he had to go back to the dorm and take the next flight which was planned to take the other girls who had decided that they couldn’t miss their Friday classes to travel. My roommate did. <!-- Oh my god, Kirk and Jueby are playing some horrendous sounding instruments in the common room, and although I really need to write this post, I honestly cannot bear this. I’ve tried for the past five minutes (as you can see above), but I feel like I’m going to tear my ears out at this point. I mean, I enjoy traditional Chinese music, and all, but even 赵新侃  and 韩光 closed the door coming into the room. Like, this is miserable. Don’t ever get locked in a room with beginning players of these instruments. You’ll want to kill yourself. --> Tomorrow I will continue, because right now I am simply about to pass out. 2007-09-26T00:00:00Z tag:dlo.me,2007-09-26://archives/2007/09/26/a-cold-night-in-beijing/ <p>Amidst a seemingly endless gust that is sweeping throughout my dorm and the slight sound of leaves scraping against each other as they only do in Autumn, I ran outside to grab myself some 煎饼, because, well, dinner tonight wasn’t really enough (it consisted of 2 bowls of rice and a plate of this nasty chicken dish that went mostly uneaten–totally undercooked). In any case 煎饼 is this crepe-like creation that is a concoction of eggs, lettuce, and crispy “stuff”–I don’t know what it is. This object has been a favorite of mine ever since I’ve come over here…to 北大. They didn’t have these things around the 北师大 campus, which would have irked me at the time had I known that such a marvelous fruit of human creativity actually existed.</p> <p>In any case, the weather today is the coldest that it has been during my entire time in China, hands down (just to keep this in perspective, I’m not a strong supporter of the “hands down” construction, but this time it just calls for it due to the extremeness of the situation). On another note, Google spell-check thinks that extremeness is a word, yet awesomeness is not. This bothers me.</p> <p>Some people have asked me for my address here, so here it is (for all the world to see):</p> <p>Dan Loewenherz<br> 中华人民共和国北京<br> 北京大学廖凯原楼240室<br> 北大耶鲁联合本科项目<br> 邮编：100871</p> <p>If you can’t read it, comment.</p> <p>Dan</p> 2007-09-25T00:00:00Z tag:dlo.me,2007-09-25://archives/2007/09/25/sorry-about-the-hiatus/ <p>Hey Everybody,</p> <p>So yes, I have been blamed for allowing this beautiful site to sink into a state of morbidity, and yes, I do have to admit, it’s my fault. I can’t say that I haven’t had the time, nor can I say that I’ve had nothing to write about, because those two things are just outright false.</p> <p>Anyways, onto the news: a couple of weeks ago, the PKU-Yale program started on the 北大 campus just as it was expected to. My roommate is awesome–not only does he speak nearly fluent English, BUT, he is also a pretty cool guy. He’s teaching me slang non-stop, which is good for my Chinese. It’s definitely helping me do things (like actually understanding people) in ways that I would have never thought possible. In fact, it’s great to ask him random things like if something can actually be said in a sentence, which was something roommate’s at PiB just had no way to do. I mean, how is asking any old American how to use 反而 going to help my Chinese? ON TOP of that, I can learn sweet things like RP黑洞 (RP black hole, a person who has so much bad karma that he’s pretty much hopeless) and 豁然开朗 (achieving a moment of epiphany).</p> <p>There’s so many random things to say, but I just can’t condense everything into a short paragraph. To do that would demerit any experience of its true effect. So, as a result, I am going to continue my blog from hereon as I did during the summer.</p> 2007-08-14T00:00:00Z tag:dlo.me,2007-08-14://archives/2007/08/14/a-light-in-the-attic/ <p>I don’t know how this happened, but I somehow have made it to the last week here at Princeton in Beijing. Although I love this program (and yes, I have to admit that the last post was just me complaining, I wrote it out of a fit of rage, and this fit of rage happened to take place during a time when I was feeling relatively angry with my progress in Chinese), there are things that aren’t perfect about China itself.</p> <p>Last week I had to get a new visa, because the one I currently have only allowed for a single-entry, and thus, if I had wanted to come back into China, I would have had a major problem. The building I went into was probably the nicest I had walked into in all of Beijing. It was bustling with (seemingly) important people (I certainly felt set apart from everyone in this building, everyone else was wearing suits and had a “job” to do, I guess…I was just wearing a T-shirt and jeans, and had white skin).</p> <p>One thing that does bother me about the culture here in China is that people will immediately start talking to you in English if you look white–even before one has even spoken. I’m not sure how I feel about this. I know I’m not in the United States, but if the situation were to be reversed, the same thing would not transpire. For instance, everyone in the U.S. is expected to speak English, regardless of race. There are no exceptions. If I see someone who looks Chinese in the U.S., I’m not going to walk up to him and try to start a conversation in Chinese–this seems too presumptuous of me, and a little rude. However, this sort of thing happens all the time in China. It’s certainly happened to me on numerous occasions. Do Chinese not understand that white skin is not an instant sign that I speak English? On top of that, every time I walk into a restaurant, I feel like I’m being stared at the instant I walk in and all the way through my meal.</p> <p>I will just flat out say that this is absolutely inappropriate, and yes, I’ll say it, racist. Just because I look different doesn’t mean that you can treat me differently or use different languages when talking with me. If China is ever going to succeed in the new economy, or the world that is increasingly becoming internationalized, its people are going to have to realize that it can’t live in a bubble anymore, and that it is not the center of everything. I feel like this is a problem that has plagued China for time immemorial, and it will undoubtedly take generations to get rid of.</p> <p>I hate how this is going…my posts are getting increasingly critical. But this isn’t exactly the message I want to send. There are things about China that I absolutely love, and there are a lot of myths that go around in the states that are flat-out not true. For example, there’s a prevalent belief that has been promulgated by various sources that people here don’t wait in lines, that the people here spit everywhere, that there are huge differences in politeness and dealing with people, and that the government is looking over every little thing I do.</p> <p>Well, despite what our textbook says and what I heard before I came here, I have to say that Americans probably spit on the streets just as much as Chinese do, have about the same rules when it comes to politeness, (…get ready for the big ones) Chinese in fact do wait in lines, and from my eyes the government here is absolutely less intrusive (at least outwardly) than that in the U.S. I don’t know why China is always being made out to be so different. The fact of the matter is that people are just people, and that you just have to be open.</p> <p>罗文瀚</p> 2007-08-05T00:00:00Z tag:dlo.me,2007-08-05://archives/2007/08/05/哎呀/ <p>Hi y’all,</p> <p>This was the end of my sixth (!) week in Beijing, and I can’t seem to believe that I’m going to be here for another five months. Alright, enough of that.</p> <p>I want to keep talking about the classroom situation, because it’s pretty much my life over here at PiB. In the last post, I discussed a fair amount of problems that lie within the system that is currently being implemented at this language program. Right now I’m gonna try and give some examples of what I think would be much more effective ways of teaching this language. I’m going to try to be concise, but not too specific. Here goes…</p> <p>Every day should begin with a teacher discussing the meaning of the words in the lesson, giving the definitions in Chinese, and making these explanations as simple as possible, using already-learned vocabulary. This way, in our minds, the words will be cemented into our heads as Chinese words, <em>NOT</em> Chinese translations of English words (which, by the way, are totally different–there are some words in Chinese that have the same English<em> </em>definition, but have absolutely no relationship when it comes to usage).</p> <p>After this, two teachers will go in front of the class and use the words that we have just learned in conversation. By hearing the words being used in a context <em>other</em> than a textbook, we’ll be able to know in our heads when it is appropriate to use the newly-learned words. A problem at PiB (and I’m sure it’s similar at many other language programs) is that you’re expected to know how to use the word you learned last night perfectly despite the fact that you’ve never used it before, or heard it before, in your life, which is a problem. If there is real-life conversation, then it’s easier to assimilate the vocabulary because there is a link in your mind between the meaning of the word and its usage. This is <em>extremely</em> important in language acquisition. It goes without saying that grammar examples would be given in a similar fashion.</p> <p>After this, students will be asked to provide sample sentences using the words. This could go on for a few minutes, or an hour, but the main point is that the teacher could correct the student if he or she makes a mistake, and then explain why the specific usage was wrong. Slowly but surely students will start to notice their mistakes and will stop making them.</p> <p>We can then move onto the one-on-one sessions, where the student is engaged in conversation. This provides a way to practice everything that was just learned during the day, and everything before, in a very low-key and stress-free environment. Notice how the entire method is focused on focusing on the listening <em>first </em>and <em>then</em> the speaking ;).</p> <p>Maybe I’m too idealistic. Maybe I’m a complainer. Maybe I’m just saying this because I still can’t speak Chinese that well. 哈哈.</p> <p>Yours truly,</p> <p>罗文瀚</p> <p>p.s. new photos are up</p> 2007-07-27T00:00:00Z tag:dlo.me,2007-07-27://archives/2007/07/27/pictures/ <p>This is where I put all my pictures…probably should have told y’all this earlier.<br> <a href="http://picasaweb.google.com/dloewenherz"><span>http://picasaweb.google.co</span><span></span>m/dloewenherz</a></p> 2007-07-27T00:00:00Z tag:dlo.me,2007-07-27://archives/2007/07/27/contd-sp/ <p>And the last post is incomplete.</p> <p>The party was pretty awesome. I met up with some other Yalies (it was crazy, yeah, I have no idea how they found out about this thing either) and we spent pretty much the entire party with each other. It was fun.</p> <p>Now, I kind of want to talk about what exactly is going down with this entire language immersion business.</p> <p>I have to say that I am in some ways very disappointed with how foreign languages are taught. Yeah, I thought about it for a while and decided that my disapproval is not just limited to how I’m learning Chinese right now. I’m gonna list the problems I see one by one.</p> <p>第一个: I know that during class, us 2nd year students mess up a lot on our tones and/or pronunciations of words. However, from my point of view, we are not being correctly nearly as much as we should be. Instead of this, our classes are more focused on pounding in new vocabulary into our heads, and not enough time is spent nailing the fundamental issues. And every class comes with new vocabulary, so oftentimes we don’t get to practice the vocabulary we learned the day before!</p> <p>第二个: We do too much talking, and not enough listening–listening to our classmates (and ourselves) say the wrong thing in class, over and over and over. This <em>cannot</em> be the right way to learn a language. The system at PiB is: teacher asks a question, student responds. Thus, the emphasis is on <em>talking</em>, when I think <em>listening </em>is the more important aspect. When one is asked a question, he is obviously listening. But I don’t think this is enough. This is basically all the language interaction that we get during class. On top of that, we are often asked to use the newly learned vocabulary in complex sentences when we haven’t even learned how to properly say the words. Of course there are going to be problems when your sole means of studying the night before is listen to a CD. But you can’t ask a CD questions, like, “in what other case can you use this word” or “why is there a 了 in this sentence?”. It’s just not the same.</p> <p>第三个: Our interactions with native speakers is lacking. The only chances we really have to have “meaningful” and uncut discussions with our teachers is during 中文桌子 (chinese table, zhong1wen2 zhuo1zi) and our individual sessions. I say meaningful with quotes because I think our thoughts are being shaped by the vocabulary that we have, in many ways. We are only taught to use the words we know in certain constructions, and these sentences come straight from a book that has come to the conclusion that the real reason US-Sino relations are so good is because China sent a ping-pong team on an official visit to the U.S. in 1978. Yeah… I mean, of course we have our own thoughts, but what I’m saying is that it’s hard to ask the teachers during class what the other word you want to say is, etc.</p> <p>Ok, I’m wiped, and I could make a long list, but I’m too tired right now.</p> 2007-07-27T00:00:00Z tag:dlo.me,2007-07-27://archives/2007/07/27/what-does-it-take/ <p>I just finished the fifth major exam, and I have to say that they’re getting progressively harder as the weeks go by. But that’s enough about academics.</p> <p>So there are a lot of things that I have left out in the past few posts, and I feel like a terrible person for denying all of you the pleasure of hearing about some of the absolutely insane things that I have done. I’ll start with what I think is by far the most crazy. Three weeks ago, I was talking with a Chinese friend of mine (on Skype) and she let me know about a party that was going to go down right by the end of the Great Wall of China (basically, the area where the Great Wall meets the sea). I was doubtful beyond belief that a thing like this was actually lawful, but feeling peer pressure and an intense overwhelming sense of adventure, I decided to go for it and buy tickets. I told a few other people here at PiB about the party, and contrary to what I expected, there wasn’t too much interest. This might have been because the very day of the Great Wall (learn chinese! 长城 - chang2 cheng2) party PiB was also arranging a trip to another tourist site. All in all, that day (and night? I guess) I ended up sitting in buses for a total of about 11 hours. In any case, three other people in PiB stated that they wanted to come, so I went ahead and bought them tickets. The ticket place was at this really random cafe at this international university–in the U.S. I’m definitely not used to buying party tickets at random Chinese restaurants. When I warily stepped into the cafe, like a lost child, they immediately made it clear to me that this was the place to buy tickets. I don’t know why that was the first thing they said to me, but I’m guessing that it’s because I’m white.</p> <p>The tickets were 200 kuai each, which comes around to about $26.44, which appeared as a steal in my eyes for a night full of partying right next to 长城. So, when that fateful Saturday came around, we all frantically congregated in the lobby of 新松公路 (our dorm) because we all thought that we would be late for the bus pickup (the ticket said that the pickup stopped at 6 o’clock, and by the time Princeton-in-Beijing brought us back from our little trip, it was already about 5:30. We were dying because of the thought that we would miss this awesome opportunity. We flew into a taxi cab and yelled out (probably totally meaningless) instructions to take us to our destination. After about 20 minutes of driving, we arrived at a location. Problem: no buses.</p> <p>We thought that we had messed up. By the time that we got to the place, it was already 6:07, and besides the public buses, there was nothing that appeared to have anything to do with a 长城 party. We walked around aimlessly for about 20 minutes–in the end, I called my Chinese friend to see what the deal was. Luckily, we screwed up, and went to the wrong place. After 5 minutes of searching, we found the right area, of course covered with hordes of foreign people. 除了 me and my Chinese friend, I don’t think anyone was speaking Chinese.</p> <p>The best part of this trip was the fact that the bus driver didn’t know where we were going.</p> <p>Let that soak in for a bit.</p> <p>…</p> <p>We had to stop over 3 times because the driver had to ask for directions. It was pretty pitiful. We also got lost once, but that problem was swiftly corrected. You have no idea how awesome it is to be trapped in a bus for over four hours when you have no idea where you are or where you’re going, all the while listening to drunk Americans singing classic rock ballads. It was an experience I’ll never forget.</p> 2007-07-19T00:00:00Z tag:dlo.me,2007-07-19://archives/2007/07/19/ok/ <p>Alright, I lied. When I said “later tonight”, I really meant “two days later after my midterm exam”. In any case, I just finished it, and 没想到, it wasn’t that bad at all. I guess that’s because I’ve been locked up in my room for the past two days eating takeout Gongbaojiding and writing characters. Woohoo.</p> 2007-07-16T00:00:00Z tag:dlo.me,2007-07-16://archives/2007/07/16/continued/ <p>Alright, so I changed my name from 罗丹 to 罗文瀚 earlier on this week. Since there were already 2 other 罗丹’s in the PiB program. My new name actually means something too–I guess the closest English translation would be “vast knowledge”, which in my opinion is a pretty cool name. Also, what’s perhaps most important is that I’m not constantly being asked by Chinese people if I’ve heard of 罗丹, the artist. It happened ALL THE TIME when I introduced myself to people. It was really starting to tick me off, but now I’m not gonna have that problem!</p> <p>So to continue where I left off last night, after talking with the two 北师大 students, the thought truly came to my mind that being able to communicate in Chinese will open up a whole new world. I know it sounds corny, but I had an awesome time talking with people who had never been to the United States before and who have been speaking Chinese their entire lives. I don’t know how to explain it, but whatever happened to my mindset gave me a huge push. I think the realization was that the majority of my Chinese learning is not going to come from my textbook, but rather from the people who I meet here. Chinese is a living language, with meanings and usage changing constantly–for example: at Yale, the word fazi was used to mean “method” (as in “study method”) but when I got here, I learned that fazi isn’t often used for that meaning, the word I should use is banfa. The grammar, as I’m slowly learning, is also one of the most messy I’ve ever encountered. Of course there are constant rules, but as there is no confusion related to verb tense, noun declensions, prepositions, or articles, Chinese grammar makes up for it in other ways. Ok, back to studying…will repost later tonight!</p> <p>罗文瀚</p> 2007-07-15T00:00:00Z tag:dlo.me,2007-07-15://archives/2007/07/15/burning-the-midnight-oil/ <p>BIG NEWS: I changed my Chinese name. I’ll explain in tomorrow’s post…</p> <p>The past week here at PiB was definitely our most intense week thus far. I do have to say, though, that I’m beginning to become somewhat frustrated with the learning process here at PiB. On one hand, since I’ve come here, my abilities to communicate in Chinese have escalated more quickly than I could have possibly imagined–and what’s harder for me to totally soak in is that I’ve only been here for three weeks. On the other, I am annoyed by the fact that the language pledge, in my opinion, has not been that effective. The issue I have is that, as a second-year student, I’m mostly just communicating with other people who can’t speak Chinese all that well. As a result, if we say something wrong, make a grammar mistake, or don’t know what word to use, well, yeah, the pledge doesn’t help. I’ve learned far more during my time here talking to local Beijing ren, picking up some slang words and getting a better grip on how to understand what people are saying when it sounds like they have towels in their mouths while they’re talking (during class, our teachers do NOT talk like how the Beijing locals do….at all). And there’s no other way to learn that in 普通话 that the word for pistachio (开心果) is just a synonym for “a happy person” ;), because our teachers purposely (at least in my opinion) restrict the vocabulary they use in class to solely the lesson-based vocabulary. Time and time again I have listened to my teachers nearly say something that they would normally have said, and then quickly correct themselves. Although PiB’s classes are great, they are by no means all-encompassing. If you want to learn Chinese, you MUST go out in Beijing and talk to strangers.</p> <p>Which leads me to some great things that happened last week. 马森茂 (Sam Massie) met a 北师大 student who was studying in our dorm last week, and we ended up getting together with him and his girlfriend at the student cafeteria. Prior to this event, I had been feeling really down about the entire language-learning process–after talking with two other people who had grown up here, my entire viewpoint changed drastically. They spoke frankly with us about any issue that we brought up, including shows that they liked (Prison Break is SO popular here, I just don’t get it!) and opinions on Mao. They asked us about George Bush, and we thought it would be funny to bring up the fact that he always says “nucular” instead of “nuclear”. Heh, they didn’t seem to get the humor that the President of the United States mispronounced words in our own language. I think this stems from the fact that in China, there are so many different dialects that one is almost expected to speak with one’s own twist.</p> <p>Aight, it’s almost two in the morning, I gotta go to sleep. 晚安!</p> <p>罗文瀚</p> 2007-07-03T00:00:00Z tag:dlo.me,2007-07-03://archives/2007/07/03/a-promise-is-kept/ <p>Alas, I said earlier that I would upload photos, and hence I will keep my word. Of course I will provide appropriate captions to keep your imaginations occupied. Actually, just enjoy the pictures. P.S. They’re all real. P.P.S. I would love to write more but I have a lot of work ahead of me. If I don’t want to get pwned in class tomorrow it would be best if I got it done.</p> <p><img src="http://onehundredeightydays.com/wp-content/uploads/2007/07/china-2-004.JPG" alt="Chang Cheng" height="255" width="364"></p> <p><img src="http://onehundredeightydays.com/wp-content/uploads/2007/07/china-2-006.JPG" alt="thrice" height="255" width="364"></p> <p><img src="http://onehundredeightydays.com/wp-content/uploads/2007/07/china-2-005.JPG" alt="numba 2" height="255" width="364"></p> <p>you get a prize if you can tell me who’s in the right-hand side of this picture.</p> 2007-07-03T00:00:00Z tag:dlo.me,2007-07-03://archives/2007/07/03/let-the-destruction-begin/ <p>Before I begin, I’d like to share something that one of my fellow Yalies, Niè Wànlǐ (a.k.a. Scott Nelson) sent to me and other Yalie PiBers: <a href="http://www.nytimes.com/2007/07/01/nyregion/01yale.html?_r=1&amp;ref=movies&amp;oref=slogin" title="This Place Looks Familiar, but Where&#39;s Starbucks">This Place Looks Familiar, but Where’s Starbucks?</a></p> <p>Alright, so I’ll try my best to let you know what exactly has transpired in the past week and a half–I apologize for the long delay, getting into the groove here as been a little more difficult than I thought, and again, money was kind of an issue, mostly because I had intelligently entered the wrong PIN three times in Bank of China ATM. Yes, I know. I am a genius.  So, as a result of this, Wachovia blocked my card, and since they were unwilling to reach me by email, I was unaware of this…so I get an email this morning from my dad saying that Wachovia had mailed BY POST a letter to my house. Now, if they had seen where this particular ATM was located, they should have had the foresight to realize that I would not get the information that my card was blocked until after I got back home, which would be in six months. Luckily my dad was kind enough to let me know what was going on. Anyways I talked to the Wachovia people today and everything was sorted out, and I happily withdrew from the ATM so conveniently placed next to the McDonald’s by Dongmenr (East Gate).</p> <p>This is one thing I love about out living situation. Xinsong Gongyu (the dorm where all the PiB 2nd years live) is in an excellent location–if only I could 降价 (jiang4jia4 - haggle over prices) a little better in Chinese, I think I would have it set. As still as of yet I’m pretty incapable of ordering off of menus without the help of friends–and I can’t say that since I’ve gotten here that aspect of shenghuo (life) has gotten much easier. It is, although, relatively easy to buy things from the very nearby chaoshi (supermarket). They sell nearly everything: shampoo, sodas, bread, snacks, you name it. And there is an extra bonus because you get to laugh at all the retarded English translations on nearly everything in the store. If you ever come to China and don’t speak English, expect to do a lot of deciphering, because signs and labels sometimes aren’t so clear.</p> <p>Another added bonus is that I’ve found that things are extremely cheap. After my fiasco at the bank, where I managed to huan (change) about US$100 into Yuan, I managed to buy a SIM card, eat out every day of the week, buy toiletries, go out twice during the weekend, travel to Changcheng (the Great Wall), and visit Tiananmen Square. Mind you, this is for about eight full days of living here. And it’s not like I’m living on the cheap. I can imagine, though, that if you’re a tourist you’ll be herded into expensive touristy areas with American-like prices. I’ve only been eating at restaurants with only Chinese-speaking fuyuyuans (waitresses) and haven’t really encountered many English speakers.</p> <p>So, let’s get down to business (i.e. exactly how this intensive Chinese thing is going).</p> <p>So far, PiB has been incredible. It is the best language-teaching I have ever had, hands down. We have classes five days a week, four hours a day, from 7:30 to 11:30 in the morning. Technically, it’s really four classes every day. The first two classes are fifty minutes long and are the closest that PiB comes to to a lecture. It’s really not even close, because during class we (the students) are constantly kept on our toes, and have to remain very attentive, otherwise quickly falling behind is inevitable. In the first few minutes we are given a tingxie (literally “listen write”) which, for all you French students out there, is basically a Dictee, except a million times harder, because you don’t exactly have the option of sounding words out to spell them. Our teacher speaks, we write characters. Easier said then done, especially when you have more than thirty new characters every night.</p> <p>After these two dabanke (big classes) we have two “drill” classes, at which point we are split up until groups of four students and are reinforced with the material covered in the earlier sections. Classes move very fast and we typically cover more than twenty grammar points every day (today I think it was more like thirty). On top of this we are forced to use the vocabulary from the previous night’s lesson in class.</p> <p>Every afternoon (except Friday) we have one-on-one tutorials (gebietanhua) that last 50 minutes…in addition to this we are given to option to take on an extra 50-minute tutorial sometime during the week if we so desire. I chose to do this and I believe it was an excellent idea. These gebietanhua provide excellent opportunities to talk with your teacher about pretty much anything–yesterday I talked to my teacher about TV Shows (dianshi jiemu) in the U.S. and was quite surprised to hear that she was a fan of Pengyou (friends) and Er Shi Si (24). Heh.</p> <p>On Fridays we are tested on the material from the week–despite my greatest fears, last Friday’s test was not as bad as I expected it would be. It was divided into several sections–one was fill in the blank, another was giving directions, another was essay-like in its wording. All in all not too bad. We are also expected to write an essay once per week, and from what I can tell, they’re gonna get harder. In fact, I think pretty much everything is gonna get harder. Last week our assignment was to write a letter to one of our American friends: 200 characters. This week we have to interview 3 Chinese people about the traffic in Beijing: 350 characters.</p> <p>On Mondays and Wednesday we also have a Chinese table (Zhongwen zhuozi). We basically go out to lunch with our Xiaobanke (”drill class”) teachers and talk about anything. Although yesterday I have to say I wasn’t too interested in what we were talking about. At our table there were 4 girls/2 guys, and the guys being in the minority, the topic of conversation quickly deteriorated into a chat about hair. Yes, hair. And we are only 2nd year students.</p> <p>Alas, I brought the issue up with everyone at the table and the conclusion was quickly made that the only way I’ll ever get a girlfriend is if I learn to love to talk about hair. I was not happy.</p> <p>I have my gebietanhua in 10 minutes, so I should probably go. I’m gonna upload some pictures pretty soon.</p> 2007-06-23T00:00:00Z tag:dlo.me,2007-06-23://archives/2007/06/23/ni-hao-zhongguo/ <!-- <a rel="attachment wp-att-13" href="http://onehundredeightydays.com/?attachment_id=13" title="the meal"></a> --> Wow!! What a trip. I left 迈阿密 yesterday (?) morning at 6:50 to go to Newark, where my plane to Beijing was scheduled to leave at 12:50 or thereabouts. There was nothing spectacular about the plane ride to Newark, besides the two fifty-something lovers who were obviously not aware that I was in the row right next to them. <!-- <a rel="attachment wp-att-14" href="http://onehundredeightydays.com/?attachment_id=14" title="Our Bags"></a> --> <!-- <a rel="attachment wp-att-12" href="http://onehundredeightydays.com/?attachment_id=12" title="Map to China"></a> --> When I got to Newark I went straight to the gate for the direct flight to Beijing, which wasn’t that far of a walk and not a hassle at all, considering that I was expecting some sort of regulatory mob to track me down because it was an international flight. In any case, I met up with a bunch of other people doing the program. We got a little hungry/antsy and a few of us thought that it would be a good idea to pick up some food before we left on the plane. As fate would have it, the last meal I had in the U.S. was from McDonald’s and the first I got in China was from Starbucks. The plane was huge and overbooked, and a couple of times over the loudspeaker at the gate repeated attempts were made to sway some passengers into foregoing their flight for a day’s delay and $800 in Continental Airlines credits. The offer seemed really enticing, but I didn’t want to miss my placement test and be forced to find my way to Beijing Normal University on my own–at this point I’m having extreme difficulty even ordering food at a restaurant. Clearly I have my limits. The plane trip was really really really long. Like, really really really really long. It was a fun trip though, being that I’d never been to the North Pole. When I looked outside midway through the trip (see picture) <!-- <a rel="attachment wp-att-12" href="http://onehundredeightydays.com/?attachment_id=12" title="Map to China"><img width="196" src="http://onehundredeightydays.com/wp-content/uploads/2007/06/sv100727.JPG" alt="Map to China" height="140" style="width:196px;height:140px"></a> --> All I could see was white, which at first I thought was clouds, but then I quickly realized that what I was seeing was ice. Siberia was really similar–the air was really clear and the scenery looked beautiful. We got three meals during the flight, the first of which were not bad, but the third one (see picture), left a lot to be desired. <!-- <a rel="attachment wp-att-13" href="http://onehundredeightydays.com/?attachment_id=13" title="the meal"><img width="211" src="http://onehundredeightydays.com/wp-content/uploads/2007/06/sv100726.JPG" alt="the meal" height="152" style="width:211px;height:152px"></a>  mmMMM--> For some reason, ketchup was included, and I couldn’t figure out why. In addition, we were given what looked like a random scrap piece of wood to eat the ice cream. I couldn’t quite understand what was going on so I gave my ice cream to the little kid sitting in the row next to me so that he would have to deal with it. During the flight, I also tried to review some Chinese, which didn’t turn out to be such a great success. I gave up after about an hour because it dawned upon me that it would be fruitless to attempt to study for the placement test (this soon turned out to be the correct judgment on my part, details to follow). When we got to Beijing a whole thirteen or so hours later following our departure out of Newark, the scenery outside was beautiful, but the weather was not. What I first though was overcast clouds were simply conglomerations of dense smog. This was not a happy revelation. Not much talking went on when I entered the airport in Beijing, as most of the Customs agents and border guards just took our little slips and pointed us in the direction we needed to go. I was surprised that there was no hint of air conditioning in the main concourse, but it’s ok, because it didn’t bother me much. I was really impressed with the speed at which we received our luggage–I couldn’t have been waiting for more than five minutes to get all my luggage. When I stepped out the baggage-collection area, there was, lo and behold, and Starbucks. Needless to say I was quite wiped out by all of this traveling, and so getting coffee was a high priority of mine. I also have to say that the latte was not bad at all, in fact it was pretty damn good. After a half hour or so we were somewhat herded out of the airport by the PiB people and were led to our bus waiting for us outside. The air definitely had a smoggy feel to it, but it wasn’t enough for me to want to ball my eyes out because of irritation. It is my first day here, though, so time will tell if I’m going to have and eat my words. I forgot to change my money at the airport (I’ll explain the importance of this soon enough). Our luggage was hauled away in separate vehicles–this didn’t feel too kosher–see picture and you’ll understand why I felt this way: <!-- <a rel="attachment wp-att-14" href="http://onehundredeightydays.com/?attachment_id=14" title="Our Bags"><img width="220" src="http://onehundredeightydays.com/wp-content/uploads/2007/06/sv100728.JPG" alt="Our Bags" height="165" style="width:220px;height:165px"></a> --> The ride to Beijing Normal University took about a half hour, and was pretty smooth. I was surprised to see so many Volkswagens on the trip–I had no idea they were so popular over here. When we got to the Beijing Normal Dorms, we were immediately told that we’d need 200 Yuan to reserve our room–this is where my little problem of money surfaced. A few other people also forgot to change their money, so we all went together to try and find a place to exchange our dollars. This turned out to be quite the task. We first went across the street to a place that looked like a working Bank, but the ATM wasn’t usable (it turns out that this was the <em>Agriculatural</em> Bank of China, which is different from just the “Bank of China”, which is what we were looking for, and turned out to be on the same side of the street). Anyways, it took us a good forty minutes or so of diligent searching to find this place, which had an ATM right in front that seemed to work, as luck would have it, for EVERYBODY else except for me. So I had to walk inside the Bank of China. With my laptop case and everything else that was stuffed inside of it (remember, I couldn’t get my room yet, because I didn’t have the money to pay for the key deposit, and thus, I had no place to put my bag).  I went to the little machine at the entrance to the Bank, where you’re supposed to press a button and take the slip of paper out that marks your place in line. This was the first thing I wasn’t used to doing at banks in the U.S. however, this wasn’t the worst of it. I waited for over an hour, and just like the airport, this place wasn’t air-conditioned. I was handed a slip that I’m guessing foreigners must use to exchange money, and was asked to fill it out. Somehow I got my Yuan, and I was happy, and was soon to be much happier when I found out how far our dollars go in terms of purchasing power. When I got back to the dorm, I managed to get my key, booklet, etc, and went straight to my room and met Chris, my roommate. The room is spectacular–we have a TV, air conditioning, bathroom, closet, maid-service, the list goes on and on. It’s basically luxurious, and I feel a little bizarre when I walk past the Chinese dorms and see that they’re nothing in comparison to this. A little after getting into my room, a few people were getting together for dinner. Not only was finding a place to eat a little difficult, but the act of dining was quite the experience in itself. We had no idea what the menu said, besides some 汉字 (hanzi, characters) like 鸡 (ji, chicken) and 肉 (rou, meat). We got food though, and it was good. And cheap. I mean really cheap. Like two bucks a person. And I was truly full. After dinner we walked around a little bit to explore, and when we got back to the room I was exhausted. After a few minutes of taking a shower and shaving the day was over for me. I had been awake for over 28 hours and wasn’t about to stay up any more–not that I was even capable of doing this. So I went to bed at 9ish and woke up to take my placement test…this is a good place to stop ;). 2007-06-08T00:00:00Z tag:dlo.me,2007-06-08://archives/2007/06/08/i-got-it/ <p>My days of mourning are over! I have finally gotten a 90-day F visa to China! Woohoo.</p> 2007-05-25T00:00:00Z tag:dlo.me,2007-05-25://archives/2007/05/25/this-is-spectacular/ <p>Whilst perusing Wikipedia, I stumbled upon this <a href="http://en.wikipedia.org/wiki/Wok_racing">surprise</a>. Enjoy!</p> 2007-05-25T00:00:00Z tag:dlo.me,2007-05-25://archives/2007/05/25/visa-things/ <p>Alright, I’ll admit it, that last post wasn’t necessary.</p> <p>In any case, today I had an appointment with my pulmonologist so he could take a look at my lung. Everything seems good to go for travel (to make a long story short, I collapsed my lung–thrice, in the past six months, had surgery a couple of weeks ago, and needed a checkup to make absolutely sure that going to China would be a-ok).</p> <p>Also, today I got the JW-202 forms from Beijing Normal University. Now I just need to figure out how exactly to manage to get a visa out of the mess that is me and my collection of documents. It’s so confusing. Oh well, I shall call the New York consulate tomorrow to figure things out. They must have answers. If anyone who reads this has Chinese-visa-applying experience, please let me know–here’s my schedule (again…yes, i know, I am redundant):</p> <ul> <li>June 22-August 18: Beijing, studying with PiB at Beijing Normal</li> <li>August 19-August 26ish: USA! Miami and Chicago</li> <li>August 27ish-December 15: Beida</li> </ul> <p>What type do I apply for?</p> 2007-05-22T00:00:00Z tag:dlo.me,2007-05-22://archives/2007/05/22/an-introduction-part-2/ <p>Alright, here comes the rest. So, as I wrote before (I think), I’m going to be spending my first 2 months in China learning Mandarin. My next few months (up to December 15th) will be spent at Beida, where I’m going to be taking Yale classes with Chinese students. It’s gonna be insane.</p> <p>I wish I could elaborate more, but I really don’t know where to start. Ask me questions.</p> <p>Oh, today I also turned in my Visa health form to my doctor. Hopefully it should be filled out by Thursday (the receptionist was very confused by the form, as it was predominantly in Chinese–I thought that was funny).</p> 2007-05-21T00:00:00Z tag:dlo.me,2007-05-21://archives/2007/05/21/theyre-almost-here/ <p>I was getting a little worried about my formal invitation documents for China, since I hadn’t heard anything about them, and I need them to get into China for my stay, but Teng-Kuan (the program coordinator for Princeton-in-Beijing) just let me know his office just received them! He said that he’d send them straight to Miami, priority mail. Awesome.</p> 2007-05-20T00:00:00Z tag:dlo.me,2007-05-20://archives/2007/05/20/an-introduction-part-1/ <p>Ok, here’s the story, the introduction, the pretty much everything you need to know what’s going on in this little blog of mine, etc. I’ll probably leave out a lot of important stuff though, so if you have any questions, just shoot a little comment onto this post and I’ll get back to ya.</p> <p>So here’s the short story.</p> <p>I’m a rising junior at Yale, and am going to spend the next six months of my life in Beijing, PRC to learn Chinese and to learn what living in China is all about. From June 22 to August 18, I’m going to be at Beijing Normal University, being immersed in Mandarin with a program called Princeton in Beijing, and from August 27 to December 15, I’ll be studying abroad with the PKU-Yale Joint Undergraduate Program.</p> <p>At the moment, I can’t say that I know much about what I’m getting myself into, but what I do know is that I won’t be speaking a word of English for the eight weeks following June 22. Despite the fact that I currently speak Mandarin at the level of a 1-year old, I think that over the next few months my ability to communicate will get much better. Hopefully by the time the program’s over, I’ll be able to audit a math course I really want to take at Beida in the Fall. Beida (北大), short for Beijing Daxue (北京大学), means Beijing University in English.</p> <p>Ok, I need to cut this post short because I’m in Gainesville right now visiting some friends at the University of Florida, and I need to eat some lunch.</p>