AI-assisted research is an exciting prospect – no more laborious keyword searches, scanning abstracts, and organizing PDFs – but do they live up to the promise? I explored this question using Elicit’s Systematic Review tool. I found a Cochrane Review on a topic and then asked Elicit the same question. I compared the two on retrieval of relevant studies, selection of studies to include, extraction of data, assessment of risk of bias, synthesis of evidence, and assessment of evidence quality. I found the report said, qualitatively and approximately, a similar thing for some primary and secondary outcomes as the Cochrane review. The Elicit report could be used by an expert to reach a similar high-level understanding as they would have gotten from the Cochrane review, but the most important between-study comparisons were not made within the report. The tool was not able to assess included studies for risk of bias, pool data across studies, or assess evidence quality. Differences were also found in study selection and detail reported for key findings.

What is a systematic review?

According to Cochrane, “a systematic review(SR) attempts to identify, appraise and synthesize all the empirical evidence that meets pre-specified eligibility criteria to answer a specific research question. Researchers conducting SRs use explicit, systematic methods that are selected with a view aimed at minimizing bias, to produce more reliable findings to inform decision making.” The systematic approach to review lends itself well to replication of SRs via LLM, though some aspects of SRs are not currently available from any provider – pooling of data, assessment of heterogeneity, and assessment of bias. There are several standards of practice that Cochrane outlines in order to ensure high-quality results that are sufficient not only to answer an individual’s question, but also to support changes in policy or clinical practice affecting thousands or millions of people. These standards cover the following areas:

• Setting the research question to inform the scope of the review
• Setting eligibility criteria for including studies in the review 
• Selecting outcomes to be addressed for studies included in the review
• Planning the review methods at protocol stage (including assessment of risk of bias and results synthesis 
• Searching for studies
• Selecting studies to include in the review
• Collecting data from included studies
• Assessing risk of bias in included studies
• Synthesizing the results of included studies
• Assessing the quality of evidence and summarizing the findings

How are systematic reviews used?                                            

A SR is written for an audience of clinicians, policymakers, researchers, and the general public, and they use them to make informed decisions about the care of those in their practice, their own care, to set policy and clinical guidelines, and to inform future directions for research. A key point here is that the SR is used not only for individual purposes, but to support changes in practice that could affect large numbers of people, many more than just the person reading the review. A tool which can provide a similar answer to the answer arrived at through the SR process can be useful individually and still insufficient for use in policy or clinical practice because the process for generating the SR has been developed in consultation with the audience, who have collectively agreed that the process yields the best results. Unless and until the same thing happens with AI-generated SRs, they meet the criteria for individual use but not for use in policy or clinical practice. In this replication, I compare a Cochrane SR with an Elicit report and discuss the differences.

Methods

A Cochrane SR on “Immune checkpoint inhibitors plus platinum‐based chemotherapy compared to platinum‐based chemotherapy with or without bevacizumab for first‐line treatment of older people with advanced non‐small cell lung cancer” was chosen for the replication. I chose this because it’s an area that has been well-studied in a systematic fashion, so there is likely to be a large amount of research available and a broad distribution of evidence quality. The information was extracted using a combination of manual effort and LLM assistance. Then, two different Elicit reports were created: Elicit’s summary of PDFs cited in the report and Elicit’s search, screening, and summary of relevant content given the same question as the Cochrane SR (Elicit-PDF and Elicit-catalog). To get the PDFs from the Cochrane SR, DOIs corresponding to the references in the SR were obtained by matching the text references with DOIs through Crossref’s Simple Text Query tool. PDFs associated with each DOI were obtained where freely available. Where the PDF was not freely available and the study was one of the 17 RCTs included in the Cochrane SR, the abstract was pasted into a Google Docs template and saved as a PDF so it could be made available to Elicit. For Elicit-PDF, the report was initiated using the “Start a systematic review” option and the PDFs were uploaded into the Elicit. No suggestions from the Elicit catalog were added. For Elicit-catalog, no PDFs were added and the 500 results, 40 studies option was chosen. Screening and extraction prompts were the same across both versions of reports, except for a screening prompt intended to flag poor quality results that was adapted to suit the different versions. No ranking results were overridden, except in Elicit-catalog, where the Cochrane SR itself was excluded, a review that was erroneously screened in was excluded, and a relevant RCT that was ranked low for reasons that were not immediately apparent was included.

Results

Top-level results

Elicit-PDF and the Cochrane SR reports said, qualitatively and approximately, a similar thing for some primary and secondary outcomes. The Elicit-catalog report could be used by an expert to reach a similar high-level understanding, but the most important between-study comparisons were not made within the report.

Cochrane

The primary outcomes for most studies were overall survival and progression-free survival. Secondary outcomes were Objective Response Rate and, for some studies, safety. (OS and PFS were secondary outcomes for some studies.)

Overall survival

• 65+
  • Hazard ratio(HR) 0.78 for Platinum + ICI vs Platinum alone, 95% CI 0.70 – 0.88, p < 0.001, I² = 0%, 8 studies, 2093 pooled participants, moderate-certainty evidence.
• 65-75
  • HR 0.75, 95% CI 0.65 – 0.87, p < 0.001, I² – 0%, 6 studies, 1406 pooled participants, moderate-certainty evidence. Results were not sensitive to the inclusion of Paz-Ares 2021.
• 75+
  • HR 0.90, 95% CI 0.70 – 1.16, p < 0.41, I² = 0%, 4 studies, 297 participants, low certainty evidence. Results were not sensitive to the inclusion of Paz-Ares 2021.

Authors Interpretation

Compared to platinum-based chemotherapy alone, adding ICIs to platinum-based chemotherapy probably leads to higher overall survival and progression-free survival, without an increase in treatment-related adverse events (grade 3 or higher), in people 65 years and older with advanced NSCLC. These data are based on results from studies dominated by participants between 65 and 75 years old. However, the analysis also suggests that the improvements reported in overall survival and progression-free survival may not be seen in people older than 75 years.

Elicit-PDF

Key Findings

• Overall survival
  • 33 studies showed HRs < 1, suggesting a reduced risk of death with the treatment combination compared to control.
• Progression-free survival
  • 5 studies reported HR < 0.51
  • 8 studies reported 0.5 ≤ HR < 0.6
  • 7 studies reported 0.6 ≤ HR < 0.7
  • 2 studies reported 0.7 ≤ HR < 0.8
  • 1 study reported HR ≥ 0.8
• Safety
  • Most studies reported a higher incidence of adverse events in the combination therapy groups compared to chemotherapy alone, but these were generally manageable.
  • Included some specific examples

Elicit interpretation

The effectiveness of immune checkpoint inhibitors plus platinum-based chemotherapy in older adults with advanced non-small cell lung cancer remains unclear due to limited representation and lack of specific outcome data for this population in clinical trials

Elicit-catalog

Key Findings

• Outcomes
  • Progression-free survival (PFS): 13/15 studies
  • Overall survival (OS): 8/15 studies
  • Objective response rate (ORR): 7/15 studies
• Safety
  • We found Grade ≥3 AEs data for 9/15 studies
  • We found Treatment Discontinuation data for 2/15 studies
  • We found Immune-related AEs data for 3/15 studies
• Some specific details for individual papers were given.

Elicit Interpretation

Adding immune checkpoint inhibitors to platinum-based chemotherapy extended progression-free survival by 2 months and improved response rates by 12.8% in older patients, but led to more adverse events.

Elicit-PDF vs Cochrane

Inclusion criteria

Inclusion criteria were similar by definition, though with the Cochrane SR, three independent reviewers made the assessment to control for reviewer error or bias. The Cochrane SR also searched more sources, including clinicaltrials.gov.

Collection of data

Data collection was also similar by definition, but with the Cochrane SR, three independent reviewers did the extraction to correct for error. In addition, the reviewers reached out to labs to ask for unreported data necessary to pool studies, assess study quality, or estimate risk of bias.

Assessment of study quality and risk of bias

The Cochrane SR contained a substantial and detailed discussion of the completeness of data on outcomes (overall survival and progression-free survival), adverse events, age, demographics, mutational burden, and histological subtype.

 The Elicit-PDF report mentioned the lack of data on quality of life (a secondary outcome), age, and adverse events, but the mentions were superficial.

Example quotes

SR: “Furthermore, the  data  available  in  the  included  studies  were  insufficient  to address  the  outcome  of  treatment-related  adverse  events  (of grade  3  or  higher)  in  people  aged  65  years  and  older,  as  only one  study  with  potential  bias  provided  data  on  this  outcome. The  included  studies  did  not  provide  data  on  other  secondary outcomes, including time to response, duration of response, and HRQoL.”
Elicit-PDF: “Detailed quality of life data was limited in most studies, representing a gap in our understanding of the impact of these treatments on patients’ overall well-being.”

The Cochrane SR assessed the certainty of the evidence based on the risk of bias using the Cochrane Risk of Bias 2 tool.
The Cochrane SR assessed study quality according to the GRADE framework, as implemented by GRADEproGDT 2021.
The Cochrane SR assessed whether the pooled sample sizes or number of events was optimal for an outcome and the size of the confidence intervals.

The Cochrane SR presented the study quality data in tabular format and mentioned the quality of the evidence throughout the report when specific studies or groups of studies were discussed.

The Elicit-PDF report did not assess the quality of evidence, thus did not weight evidence or mention evidence quality, except for a few mentions of data being limited.

Synthesis of data

The Cochrane SR assessed effect heterogeneity according to the recommendations in Deeks 2021, as implemented in Review Manager 2024. Statistical heterogeneity was evaluated using the I² stat and where values of I² greater than 75% were found, subgroup and sensitivity analysis was performed. Where a reason for heterogeneity was not discovered, data pooling was not done and study results were presented in a narrative format. Meta-analysis was done on pooled data on an Intention-to-treat basis, using a random-effects model. A fixed effects model was used for primary outcomes for sensitivity analyses, and the generic inverse-variance method was used for the fixed-effect model for time-to-event outcomes. The Mantel-Haenszel method was used for dichotomous outcomes and the inverse-variance model was used for continuous outcomes. For some rare events, Peto’s odds ratio was used under the fixed-effect model. The DerSimonian and Laird method was applied for the random-effects model.

In the Elicit-PDF report, study results were not pooled statistically and were only discussed narratively.

Elicit-catalog vs Cochrane

The comparison of the Elicit-catalog report to Cochrane was similar, except for the selection criteria. It’s worth noting that the average Cochrane review takes a team of people about 18 months to write, much of this spent on parts of the process that isn’t currently part of an Elicit report, but the search alone can take months to be sure everything is included. In contrast, Elicit-catalog took a few days, most of that spent tweaking the screening and extraction prompts. However, the search isn’t guaranteed to be exhaustive. An Elicit report also has a hard cap on the number of results included due to technical limitations. This number has changed over time but was 500 for screening and 40 for extraction in the version used in this work.

Elicit-PDF vs Elicit-catalog

The Elicit-PDF and Elicit-catalog report said similar things, but came to different conclusions. The main high-level difference was that the Elicit-PDF report didn’t find enough data to make a clear recommendation about the effects in an older population, whereas the Elicit-catalog report did, only because it considered 65+ as the older population and didn’t separate out 75+.

The main difference in the substance of the report was that the Elicit-PDF report discussed results in terms of effects of the treatments that were compared in the trials, whereas the Elicit-catalog discussed results in terms of how many studies reported an outcome. This could be because the Elicit-catalog selections were more variable in terms of the populations and treatments and study designs, so it had to compare at a higher-level of granularity. In other words, the outcomes in Elicit-PDF were reported as months or percentages, whereas in Elicit-catalog, the result summaries were often given as numbers of studies, not sizes of effects.

Other differences

In Elicit-PDF, two tables of outcomes were presented, corresponding to the primary outcomes – overall survival and progression-free survival – whereas in Elicit-catalog, one table for outcomes was created.

In Elicit-PDF, safety information was not presented in a tabular format, but was discussed narratively, whereas in Elicit-catalog, a safety table was presented and summarized, leading to a better discussion of safety in Elicit-catalog.

In Elicit-PDF, a narrative discussion of age-related effects was presented, divided into efficacy and safety sections, whereas in Elicit-catalog, a shorter section was presented with safety results from a few individual studies..

In Elicit-catalog, a section commenting on some results with different drug combinations was also included.

Conclusion

The primary aim of this work was to recreate a Cochrane SR using Elicit. A secondary aim was to compare the quality of the search and screening, relative to that of a professional SR team. This design allowed us to test both the evidence synthesis and study selection aspects of Elicit separately. We were not able to recreate a Cochrane SR using Elicit, as it currently cannot pool data, assess evidence quality, effect heterogeneity, or risk of bias, which limits the tool to giving a narrative summary of individual studies instead of a synthesis of the evidence. This precluded certain comparisons for outcomes for different age groups. The lack of pooling of data resulted in conclusions that were significantly different in level of detail and evidentiary support for the conclusions, while still being able to offer high-level conclusions that were roughly similar, when Elicit was provided with the reviewer-selected studies. When Elicit had to both select and summarize the studies, the level at which the comparisons were done moved another step in the direction of less detail, comparing numbers of papers vs results within the papers, due to the included studies not being as focused on specific outcomes as the expert-curated studies. These differences imply the tool is suitable for some purposes and less suitable for others. For individual use to learn about the landscape of a research area and the major questions within it, Elicit performs very well and enables a practitioner or researcher to rapidly gain the understanding that it would otherwise take them days to weeks of reading and note-taking and synthesizing of knowledge. However, for professional use to generate evidence synthesis sufficient to change policy or clinical practice, two things remain missing. It is not currently able to synthesize results as a SR team would, though this seems achievable in practice as the product develops. The second thing is harder, because it is not internal to Elicit, but rather an aspect of the social structures in which it is embedded. Cochrane SRs change policy and practice because the community has agreed that the SR process is a valid way to generate evidence of sufficient quality to change practice. It’s not inconceivable that Elicit reports may someday serve this purpose, but it won’t be a product update that changes this, it will be the result of a deliberative process within the research and patient community.

How AI can be used for bioterrorism

Humanity has climbed to the top of the food chain, but as the pandemic demonstrated, it’s a precarious perch. It’s not just directly pathogenic viruses that we need to worry about, either. Small molecules and proteins synthesized by bacteria also pose a risk to life, to crops and livestock, and to the environment. These risks would be realized by a terrorist through genetic engineering, the means by which new capabilities are introduced to an organism through changes in its DNA. This is a time-consuming and laborious process that can be vastly accelerated with AI and biological design tools. A terrorist might design a novel virus that causes human illness, one which kills livestock, makes soil infertile, or fouls the ocean.

How dangerous are current AI systems?

In 2023, RAND gave researchers a task to create a plan for designing and delivering a bioweapon. The teams which used an LLM weren’t significantly empowered in doing so relative to those which just used the internet, though one team with a jailbreaking expert did better than the other teams. This study was underpowered and, importantly, only tested commercial LLMs, not models where the model weights are available and thus amenable to fine-tuning

OpenAI did a similar study, finding that LLMs slightly increase capabilities across all phases of development of a pathogen. Those with access to a version of GPT4 with safety features removed developed better plans, indicating an actual terrorist group could use open weight models to gain a similar benefit. While open weight models struggle relative to GPT4 due to lack of access to technical and medical information, these models are improving rapidly.

CSET summarizes the current risks and highlights that threat actors which might have previously focused on conventional weapons might now perceive a lower barrier to adoption of bioterrorism.

How quickly might capabilities improve?

AI Capabilities

In 2019, GPT2 couldn’t reliably count to 10. Now they can work autonomously to design experiments and analyze data. The amount of compute used to train systems has dramatically increased and is currently doubling about every 9 months, with room to continue for some time. AI firms might run out of high-quality training data next year, slowing increase of capabilities, but synthetic data is being used to replace existing data with some success. Systems that can autonomously design novel chemical agents currently exist, with capabilities mostly limited by the capabilities of the model. Algorithms are also improving faster than Moore’s law,  enhancing model capability at a given level of compute.

Biotech Capabilities

AI models aren’t the only area improving exponentially. The cost of gene synthesis has dropped several orders of magnitude over the past few decades and it’s estimated that it would cost about $1000 to synthesize the 1918 influenza virus, to which there’s little existing immunity.

Taken together, the increased capabilities of AI models and increased access to information, the decrease in cost of synthesis and technical expertise required, indicates a high likelihood of a bioterrorism incident occurring within 5 years. A rough estimate of how long it will be before an AI-assisted bioterrorism incident would consider the following:

• 30,000 PhDs with the knowledge to assemble a virus today
• human-level machine intelligence currently estimated to arrive within 37 years, but note this estimate has become 8 years closer than it was just 6 years ago.
• 0.02% of attacks are bioterrorism, rising due to perceived lower barriers to entry
• 7000 terrorist attacks per year

This suggests several attempts could be made each year, with the likelihood of success increasing each year, at a cost within the budget of nearly any organized group.

What we should do now to reduce misuse risk of AI for bioterrorism

• Standardize screening for biorisk across nucleic acid synthesis companies, Biological design tools, and cloud labs. The Administration for Strategic Preparedness and Response’s (ASPR) Screening Framework Guidance provides a good starting point.
• Standardize evaluation for biorisk across frontier model providers. OpenAI and Anthropic have presented reasonable plans in this area and we look to MLCommons and the AI Safety Institute Consortium to provide a forum for best practice to be established. The Center for Long-term Risk estimated establishing a sandbox for conducting ongoing evaluations of AI-enabled biological tools will require investment of $10 million annually.
• Funders and publishers of academic research should require a statement about  capabilities of models developed that show dual-use behavior, similar to the ethics statements required for research involving people or animals. Institutional Biosafety Committees are a good framework for this.

Harm-reduction recommendations not specific to AI

• Establish global coordination for efforts to detect pandemic-capable pathogens. Routinely sequencing air and water in travel hubs and airplanes could detect an increase in pathogens before people start to show symptoms.
• Harden society against the inevitable next pandemic

• Research on resilient crops such as those which could be grown indoors or those resistant to heat. Synthesizing protein from fermentation of natural gas could be done for $3-5/kg, but to have enough capacity to support the global population would take several years, assuming $500m capex, 2-4 years, and diversion of existing capacity of companies such as Calysta Inc., Unibio A/S, Circe Biotechnologie GmbH, String Bio Pvt Ltd, and Solar Foods.
• Accelerate vaccine development. A framework for accelerating development in times of crisis could shorten the time-frame to recovery. The Center for Population-Level Bioethics may be a good partner for this work.
• Install air sanitation devices in public transportation, airlines, and businesses.

• 200nm UV eliminates 90% of pathogens after about a minute of exposure. At these levels it’s safe, businesses would likely install units to save on lost productivity, and it doesn’t require anything from individual employees, as with mask-wearing.
• HEPA filtration has similar benefits, reduces airborne COVID particles by 65%, and could tie into genetic surveillance activities.

• Develop and stockpile effective PPE

• Establish a liability regime for gain-of-function research and for public release of model weights. The risk of liability would provide an incentive for organizations to manage externalities associated with their product development and research.

“In the beginning was the Word, and the Word was with God, and the Word was God.” – John 1:1 KJV

When I think about creative works generated by generative AI, I think about golems. According to legend, Rabbi Yehudah Loew created a automaton from clay to protect the Jews of Prague by animating a soulless lump of clay with the Word of God. Like the Rabbi of folklore, when we write or produce a work of art, we like to think we’re doing something that imbues our work with soul. People say an image created by GenAI lacks the “creative spark”, but what does this really mean? What distinguishes an illustration by Sendak or Doré or Geiger from a similar work created by prompting Midjourney?

Imagine Hugging Face puts on an event where they put a generated illustration next to a work from a famous illustrator and ask an art critic to point out differences. The art critic will get deep in the weeds and point out various differences, then they’ll ask another art critic, who’ll emphasize different distinctions, then another will identify still others as crucial and then Hugging Face will say, “Aha! The critics couldn’t reproducibly find differences and therefore there really are none!” Lots of breathless media headlines and dunking on critics will ensue, unless…

Unless we are clear about where the magic of creation really comes from. Insisting the creative spark is in the work product is like believing you could literally animate clay if only you could find the right sequence of Hebrew characters to inscribe upon it. So where is the magic, then? It’s in the process. We create to be transformed by the process of creation. When we write, we’re not merely giving birth to something already fully-formed in our heads. We’re co-creating with the tools we use and the audience we imagine and the medium we express in.

This way of thinking about the magic of the creative process is backed by modern cognitive science. According to 4E cognitive science, thinking isn’t something that only happens in your head; it’s embodied, embedded, enacted, or extended through processes and systems existing in the world. The choice of writing tool and platform affords longer or shorter text and the amount that’s visible at once changes the “attention window”, so your thinking can be said to be embedded in these tools. If you use an LLM as a writing assistant, your thinking is extended by the tool. Your awareness of your audience enacts a world wherein some topics are more or less salient, and the physical act of sitting down to write and getting into a flow state is very much an embodied practice.

So the important question isn’t whether your work has some special spark that LLM output doesn’t have. The question is how were you transformed by the process? Did you develop a relationship with the ideas and did your understanding of the topic change as you wrote? Did the work develop from an infantile first draft to an adolescent revised version? Will it reach adulthood in thoughtful discussions with readers?

This to me is why we create and nothing about the existence of LLMs changes this.

…the data will, I am confident, reveal that Catholics are substantially underrepresented in investment banking […], that white men are very substantially underrepresented in the National Basketball Association, and that Jews are very substantially underrepresented in farming and in agriculture. These are all phenomena in which one observes underrepresentation and I think it’s important to try to think systematically and clinically about the reasons for underrepresentation.

He thought that these prefatory statements, and that he was attempting to be a little provocative and not speaking on behalf of Harvard, would have opened up a thoughtful discussion of various factors leading to under-representation of women in tenured positions in science at top universities. He could not have misjudged his audience more poorly.¹

Shortly thereafter, he resigned as Harvard’s president ahead of a no-confidence vote by faculty.

A few years before the fateful speech, during his tenure as Harvard president, a website created by a Harvard student called Facemash almost got its creator, Mark Zuckerberg, expelled. Two years later, Twitter was founded. Looking back from 2023, we have seen many intellectuals and public figures stumble on these platforms, and their mistakes generally fall into the same category of error. It’s one anyone in a public speaking role must learn to avoid.

When connecting audio equipment, you have to be careful to avoid an impedance mismatch to avoid lifeless instruments, digital glitches, and fried amps. In public speaking, you have to be careful to avoid a stage mismatch to avoid lifeless audiences, digital mobs, and fried jobs. I’m not talking about the thing the podium is on; I’m talking about Robert Kegan’s stages of human development. According to this framework, people go through at least 3 and up to 5² stages of development over our lives.

Briefly:

Stage 1³: A very young child can sorta control their motor skills, but their impulses and perceptions aren’t things they think about. A reflection of a baby in a mirror will be treated as real.
Stage 2⁴: Impulses and perceptions can be thought about, but needs and desires feel objectively real. A child in this stage can understand why they’re hungry and do something about it, but won’t question their interest in playing a game.
Stage 3⁵: In this stage, interests and desires can be examined, and, sadly, we lose our ability to play like children. We vibe with people and are emotionally engaged, but we might also panic or yell at someone in anger.
Stage 4⁶: We understand how the influences of those around us shape our identity and yet sometimes do things that don’t align with that. We learn the rules of society and how to fit into systems. We can keep a cool head when others are panicking, but might suppress emotions or feel disconnected. A judge would use Stage 4 thinking to assign a criminal penalty according to the law, even when public outcry demands a harsh sentence.
Stage 5⁷: We understand many different kinds of systems and can reason about how different systems would work in a given situation. We have many different identities and don’t feel like any particular one is the “true self”. Emotions are welcomed, even when they’re difficult, but what behavior results is not deterministic. A judge sparing a young mother from jail may be using stage 5 thinking.

Unless there are people having tantrums and playing carefree in the aisles, you can expect most people are primarily using a stage 3 mode of relating to the world and to your speech. Here’s the thing: While a comment made from a Stage 4 perspective can appear heartless when viewed from a Stage 3 perspective, a comment made from Stage 5 can look a partisan Stage 3 comment. Most viral outrage comes from this kind of stage mismatch. An ironic joke about AIDS not being just an African thing requires an ability to consider the different identities that make up a person, characteristic of stage 5. Heard from a stage 3 perspective, it’s enraging. If you are an intellectual, you probably spend more time than most around people relating to the world using strategies from stages 4 and 5, and you will get used to saying complex things that are nevertheless understood. This is where people mess up. They think they’re in a cozy little salon, but they’re in the town square.

Twitter and Facebook have been called our “global town square”. Those who coined the phrase no doubt meant to vaguely gesture at a bucolic vision and egalitarian ideal of free speech, but I think the metaphor is more apt than that.

You are not here.

You are here.

As in Times Square, you are surrounded by many people, not all of whom you can see, not all of whom have your best interests in mind, and not all of whom can avoid lashing out if something you say triggers jealousy or rage. In the global Times Square, you don’t just produce content, you are content, and since the invention of social media, you are always in the global Times Square.

There are lots of different kinds of conversations that people have. Do you approach the LLM as an all-knowing oracle or a creative companion or a virtual assistant or something else? What needs to be apparent when someone enters a conversation with an LLM? Looking again at how a conversation is entered with an oracle or a creative companion or an assistant provides some hints. An expert may present themselves as having a degree, a wizened face, a tweed coat with elbow patches, or an office in a old, ivy-covered hall of knowledge, but it’s over the course of a conversation with them that you move towards a better understanding. A creative professional may have a bright office filled with primary colors and whiteboards, but it’s through collaborative discussion that you get inspired and flesh out your vision. The best assistance comes from your relationship with an assistant, too. Not transactions, but dialogue. (Maybe even dialogos!)

Ok, so how to make this concrete in terms of a chat interface, and how do you know if your design is working for people? I’m a communications professional, not a product person, so I can only gesture in a direction, but if there’s a way to quantify how well a query partitions a space of information, that could be a good place to start to figure out if your expert is engaging in effective dialogue and leading someone to a better understanding. To take a simple example, imagine someone asks for gift ideas. If you tell the salesperson at a fancy retail store that you’re looking for something for your mom, they’ll ask what occasion, because occasion is one of the main ways gift-giving is partitioned. A chat agent should afford carving up of the information space in a similar fashion. An LLM contains multitudes, so it doesn’t make sense to put ChatGPT in a tweed coat, but that’s not important. The important thing is that there’s process of dialogue through which understanding or inspiration or whatever is approached.

Over the past 13 years, I’ve become an expert in many different ways to rate people, from the academic citation analysis tools on which universities spend millions to dating apps, and I’ve used a number of tools to monitor social media. The tools are dangerous to your business if you don’t know what you’re doing. You absolutely cannot assume social media is an accurate reflection of actual customer or consumer sentiment⁹. Social media monitoring tools will show you thousands of mentions from accounts with names like zendaya4eva and cooldude42 and the tools roll everything up into pretty dashboards that summarize the overall sentiment for you. There’s just one problem, and it’s that social media sentiment analysis sucks. Posts aren’t long enough for the algorithms to get enough signal and they can’t detect sarcasm or irony. You’re better off just looking at a sample of posts than using a sentiment dashboard. Analytics vendors know this and they’re working on building AI into the tools to make this better, but if you’re looking at social media sentiment because it’s easier to get than data on actual customers, you’re like the proverbial drunkard, looking for your keys where the light is better rather than where you actually lost them, and no amount of AI can fix that.

Candidate screening tools make some of the same promises. We can analyze the social media history of a candidate and flag areas of concern! I’ve written social media policies¹⁰ for several organizations and never have I ever seen a hiring or firing decision depend on a social media post that required a tool to flag. It’s very tempting to outsource our judgment. Thinking is hard and people aren’t always very good at it. You might think it’s better to have an objective process that eliminates conscious or unconscious bias¹¹, but when you do this, you’re taking agency out of the hands of HR and the hiring manager. Hiring is a hard, multi-factorial decision and the last thing you want to do is outsource judgment here¹² .

Prerequisites

Accounts at Otter, Readwise, Hypothesis, and Roam.
The Hypothesis toolbar in your browser of choice (I like Firefox).

The Process

Let’s say you’re watching a lecture. Instead of trying to scribble notes in a notebook that you’ll later have to transcribe into Roam, you open Otter and let it start creating a text transcript. Take pictures or screenshots as you go, because Otter will be able to place those in the transcript according to timestamp. When it comes time to review, you use Hypothesis to annotate the Otter transcript, then you write some notes in Roam summarizing the insights from the lecture. If you’ve connected your Hypothesis account to Readwise, your highlights will be occasionally re-surfaced for you to review, which is a key step in making them actionable. There’s also a way to get spaced repetition in Roam.

The Setup

At Readwise, you have a bunch of options for connecting highlights. Enable Hypothesis and it will pull in all your highlights from Hypothesis, including the ones you’ve made on the Otter transcripts. You’ll find them under the Your Articles section at Readwise. You can review there and write up summaries in Roam, linking to other concepts and notes.

Why It Works

It works for me because I use Readwise as a sort of catch-all bucket for all the stuff I already read in so many places – Kindle, Twitter, and all the stuff I find via Twitter and shove into Pocket – and now I can also use Otter to convert things I listen to or watch into a form that Readwise can catch & periodically re-surface for me.

Caveats

1. When you select ‘view in article’ at Readwise, it will take you to Hypothesis, not Otter. Otter can generate sharing links to annotate or you can export the transcript and annotate it somewhere else that’s publicly accessible, which is probably the best course so you have your own backup.
2. Making extensive use of all these services costs a little money. Readwise is a couple bucks a month, and Otter costs a little bit if you go over their free minutes. Roam likewise has a subscription plan. I personally believe that if you are going to invest a lot of time and effort into building a personal knowledge management system, you’re going to want that system to stick around and get better, so you’re going to hope they charge enough to do so, but I know even a couple bucks a month can be hard to come up with on a grad student budget, so here’s some options. Most YouTube videos have a transcript generated by Google, which may be of higher quality and won’t use up your Otter minutes. Also, Docdrop is a service from the founder of Hypothesis that facilitates annotation of all sorts of document types and can accept Youtube links.
3. These services are all relatively new. There is a possibility that they go under or get bought by a company with a different privacy policy. Carefully inspect the privacy policies of all the services you use, consider not using services that don’t let you delete or get your content out easily (Evernote, for example), and keep your own backups. I will note that services getting acquired is not necessarily a bad thing. My company, Mendeley, was acquired by Elsevier 7-ish years ago and it’s still going strong. Also, services that charge money tend not to be as intrusive to your privacy.

First, install Ruby with RubyInstaller. You’ll also need the DevKit.
Install the Ruby gem with gem install t. Then, register a application with Twitter. Next, authenticate the client with t authorize, which should take you to Twitter and ask you if you want to let the app you created access your account. You might have some issues with silly stuff getting the authentication to work, like outdated keys or whatever.

Then once you get to where you can tweet from your account, you’re ready to go. Create a new list with t list create [name of list]. The neat thing about having a command line client is that all the pipes and redirects and stuff work, so you can do cat listofhandles.txt | xargs t list add [nameoflist]. I think you have to have cygwin installed for xargs and cat to work, but I guess you could just do a batch file with a loop if you didn’t care about looking cool.

I got population data from the US Census website and, happily, the happiness study used the same metropolitan statistical area names as the census does, so it was a simple merge to visualize population vs. happiness.

Here’s the cleaned and merged data and the R script I used if you want to play along. Source data is available from the US census page and Gottlieb’s web site. I also scraped some data as necessary from other places, as documented in the script.

In summary, only 2 of 12 areas with more than a million people have positive happiness scores, both in Florida: Jacksonville and Tampa. The unhappy areas are, starting with the least happy,

• “Pittsburgh, PA”
• “Cincinnati, OH-KY-IN”
• “St. Louis, MO-IL”
• “Rochester, NY”
• “Seattle-Bellevue-Everett, WA”
• “Oklahoma City, OK”
• “Columbus, OH”
• “Fort Worth-Arlington, TX”
• “Orange County, CA”
• “Kansas City, MO-KS”

There are numerous possible explanations, but I’m going to tell you my favorite one, which is that there is a negative correlation between population density and happiness, mostly due to the friction associated with activities of daily life in a large city.

If you look in more detail at the locations of the 293 incorporated areas with more than 100k people, there are only two places where high population density and happiness occur together: Santa Clara County in California and Denton County in Texas. Overall, it’s rare to see a populous city in a happy area. Look at Orange and LA counties, the Seattle area, around the Great Lakes, and the Northeast, for example. Interestingly, the two happy counties with 100k+ people are near the highest density areas for their region. Santa Clara has the tech suburbs of Mountain View (my city!) and is near San Francisco. Likewise, Denton is outside Dallas. It’s tempting to think the residents of those areas formerly lived in the more dense areas until they could afford to make their escape.

It’s hard to see the points for the cities and I don’t know how to do alpha transparency with base graphics, so you’ll need to click through to the full-size images to see the points for the cities.

I plotted the 20 highest density cities on this map. None were in an area which scored above zero on the happiness index.

When you look at the least dense cities on the list, the picture is a little different. There are a few cities in areas with a positive happiness index.

Of course, correlation isn’t causation. It could be that people seek out cities for the income potential and not the quality of life, that the more success-driven people collect in cities, and more success-driven people might be less happy at any given moment because they’re always working towards something bigger and better. It might also be that close proximity with your fellow man elicits more comparison of yourself to other people and the attendant insecurity that can create, or just creates more despair for humankind. Since the only positively happy cities in the 1 million+ group were both in Florida, it might also be interesting to look at the annual amount of sunshine each place gets. Overall, it’s striking how happiness takes a dive in cities compared to less populated areas.

At any rate, I consider this supportive evidence for my belief that the south bay and peninsula is a superior place to live compared to within San Francisco city limits.

I joined a biotech startup in San Diego later that year, early 2008, but later that year the company fell on hard times, along with the rest of the country. By early 2009 my time with the company was nearing an end, but I had still been following what was happening online and had begun to advocate for a new startup that seemed like it had a better idea than the boring old “social-network-for-scientists” clones that were popping up everywhere. As my involvement with the biotech tapered off, I was able to increase my involvement with Mendeley, eventually becoming part of the full-time US staff.

When I began to work for Mendeley, I was quite definitely aware of the possibility that they would get bought at some point. Nonetheless, I was excited to be able to play a role in helping them become a success. At the time my thought process was pretty simple: they were a non-ugly version of Endnote which also happens to be building a collection of research metadata that they can make available under an open license, and they can provide a measure of impact that is distinct from citations. Freedom at last!
Now 4 years later, there’s a lot I have to be pleased about when I look back. I presented one of the few for-profit business use cases for open access(PDF) to the US Office of Science and Technology Policy. We have ~90M documents available via an API with a permissive CC-BY license. We’re one of the leading contributors of data to the growing #altmetrics movement.

Now my career is entering another phase. I’m going to leave all the “we’re so excited” stuff for the official announcement, but I think Mendeley has gotten to a size where it’s no longer a startup, and smart people are predicting open access will be a reality soon. As Victor notes, we could have carried on, but it would have taken longer for us to get to where we needed to be and there’s no guarantee we would have made it. Springer + Papers or Nature + Readcube could put more marketing muscle behind their apps and neither of them have as open a philosophy as we do. What about Zotero? I think if Zotero was going to change things, they would already have done so, but maybe they could team up with the Digital Public Library of America or Center for Open Science.

I do think there’s a possibility that we could do some good as part of Elsevier. Having talked with tons of people, from the CEO of Elsevier on down, I am now convinced that they want to be a part of the changes, instead of trying to fight them off like the recording companies did. There are and will be a couple competing narratives: They bought us to bury us, we got paid tons of money so we said, “Fuck Open Access”, etc. This is going to be put in the context of Google Reader shutting down, Delicious “sunsetting”, etc. However, I’m not personally getting a pile of money from all this, and I never would have stayed unless I was convinced that they legitimately want to be part of the change to an open access publishing system.

So I’ll be staying with Mendeley. I have been told that my day to day job will remain the same, and that my voice is valued. I trust my friends to keep me honest and to call out bullshit when they see it. I’m grateful to have had this opportunity, over the past 9 years, to not only be a voice for a better way of doing and communicating research, but to be a pair of hands. I’ll learn everything I can about working within Elsevier and, after a couple years, if we don’t finally end up with freely available academic paper metadata and more Google Analytics-like research impact information, it won’t be because I didn’t try my best. That’s my promise and I expect – need – anyone who’s reading this to hold me to it.

Other posts:
TC: http://techcrunch.com/2013/04/08/confirmed-elsevier-has-bought-mendeley-for-69m-100m-to-expand-open-social-education-data-efforts/
Q&A: http://blog.mendeley.com/press-release/qa-team-mendeley-joins-elsevier/
SciAm: http://blogs.scientificamerican.com/information-culture/2013/04/09/elsevier-giant-for-profit-scholarly-publisher-buys-mendeley-free-citation-manager-and-discovery-tool/
Jason Hoyt: http://enjoythedisruption.com/post/47527556151/my-thoughts-on-mendeley-elsevier-why-i-left-to-start

I basically followed this guide. Here’s what I had to figure out:

There are four things you must do to unlock and root this phone: Unlock the bootloader, install the recovery ROM, install the OS mod you want, and root. You have to do at least the first 3 things in order, as well as their substeps, and there’s more than four steps listed below because some of the stuff is preparatory stuff.

Unlock your bootloader

1. Installing the android SDK, which is done by grabbing the right file from here and following the extensive instructions. You’ll need the SDK core, one of the Android platforms, and perhaps a third-party addon for your phone and, for Windows, the USB drivers. You’ll need to get the fastboot tool separately, and then add #AndroidDev PATH
export PATH=${PATH}:/AndroidSDK/tools to your .bashrc and then run the following: gksudo gedit /etc/udev/rules.d/51-android.rules Add SUBSYSTEM=="usb", SYSFS{idVendor}=="04e8", MODE="0666" to the file which should be blank to start with. If your device isn’t from Samsung, substitute your device code from here for the “04e8” string above. Then you can restart the Linux device manager so it recognizes your device by running sudo /etc/init.d/udev restart
   • I had a problem connecting via fastboot on Windows, so I used my ubuntu laptop for this, if you can get fastboot devices to show your device on Windows, though, you should be able to follow the rest of this with no problems.
2. Getting into recovery mode
• Power the Nexus S down, and hold Volume Up & the Power button until booted into the bootloader.
• Connect the Nexus S to the computer via USB.
3. unlocking your bootloader, which is simply done by running sudo ./fastboot oem unlock or just fastboot oem unlock from a Windows terminal. This will wipe the phone, so have the files described below available but not on the phone.

Install the recovery image (re-flash the recovery ROM)

4. Finding the right recovery image, which is what lets you flash custom ROMs, get root, etc.
• The recovery image is like the OS for the mode you boot into when you press volume up and hold the power button down while booting the phone. There’s a bewildering array of recovery images, with different version numbers and I have no idea how to go about finding the most recent version for your device. I used the 4014-orange.img from this thread(not the 4G version, since I’m on ATT). Those links will probably die, so if anyone reads this and can’t find a recovery image they feel comfortable using, download an earlier version for your device from here and presumably you could do further updates by downloading the ROM manager app from the market once you’re rooted, or let me know and I’ll send you the exact file I used.
5. Flashing the recovery image to your phone with sudo ./fastboot flash recovery [name of your recovery image] or fastboot flash recovery [name of image] on Windows Be sure you’re either in the directory where fastboot and the image are located, or you’ve made the necessary modifications to your PATH.

Install the OS you want

6. Flashing the system ROM (as opposed to the recovery ROM)
• Get the ROM you want from CyanogenMod or another trusted source. This is the actual modified OS for your phone. You probably could root first, then use ROM manager to install your desired mod, but I didn’t do that.
• Get Google Apps, if they weren’t included with your ROM (they aren’t with Cyanogen).

Root your phone

• Get the su utility. This lets you get root access via the shell and is required for rooting your phone. I used this one, but this would also probably work.
7. Move those 3 .zip files to your phone’s storage (internal drive for Nexus S or SD card for some others) then select recovery from the menu once you’re booted into recovery mode. You might want to do a backup at this point. You should see the Google logo briefly appear then see an Android with an open box inside a triangle. Press the volume buttons and the power button until you see another menu(sorry for being vague here, up+power, down+power, and pressing the volume straight down+power eventually gets me the screen, but I have no idea what the “proper” keypress is supposed to be.) Do a wipe/factory reset, do a wipe cache, then select install .zip from SDcard, then choose .zip from SDcard. You should see the three zip files you added above. I don’t think it matters if you do su or clockworkmod first, but I did clockworkmod, then google apps, then su.
8. Reboot your phone and enjoy
• Depending on which mod you installed, you may or may not have superuser installed. If not, just download it from the market and continue setting up your phone
• You might like to try Wifi Tether and some of the backup utilities from the market, which can back up user data as well as just the APKs.

I’m hoping for better battery life, personally, but overall the ROM looks about as nice and polished as the stock does. You can do things now which will compromise your user experience, so be careful and keep backups. As should be clear from the above, I’m nowhere near qualified to troubleshoot any problems you encounter while doing this, so please direct any technical questions to the XDA Developers Forum. Here’s a few threads I found useful:
http://forum.xda-developers.com/showthread.php?t=1121420
http://forum.xda-developers.com/showthread.php?t=1107418&highlight=i9020
http://forum.xda-developers.com/showthread.php?t=988686&postcount=7

This is kinda cool, too: I added the biggest 10 words from the wordle as page tags, and the Mendeley “Related Research” plugin in the left sidebar pulled in the major papers which influenced my work. This isn’t an entirely unexpected result, but kinda cool when things work like they’re supposed to.

For me, a paper is an aggregation of objects. It contains, text, divided up into sections, often with references to other pieces of work. Some of these references are internal, to figures and tables, which are representations of data in some form or another. The paper world of journals has led us to think about these as images but a much better mental model for figures on the web is of an embedded object, perhaps a visualisation from a service like Many Eyes, Swivel, and Tableau Public. Why is this better? It is better because it maps more effectively onto what we want to do with the figure. We want to use it to absorb the data it represents, and to do this we might want to zoom, pan, re-colour, or re-draw the data. But we want to know if we do this that we are using the same underlying data, so the data needs a home, an address somewhere on the web, perhaps with the journal, or perhaps somewhere else entirely, that we can refer to with confidence.cameronneylon.net, Science in the Open » Blog Archive » The future of research communication is aggregation, May 2010

About a year ago, I wrote a post about what I had learned in my previous job. It was a thoughtful post where I wrote about some great experiences I had and some rather disappointing ones, and my purpose in writing it was to update my small community of friends and professional acquaintances about the changes in my life and professional status. When Genomeweb linked to my post, I was surprised and delighted. After all, the whole point of keeping a weblog instead of a diary is for other people to read it. I’ve begun to notice a concerning trend, however, of people assuming that content they post to the small community of people they know on social networks is like a whisper – that there’s an implicit assumption that the intended audience will keep the content to themselves and not talk about it to others. This is entirely the wrong way to think about this, and failure to understand this point is setting yourself up for serious trouble later on.

Let’s take the case of BittersweetGirl first. She became upset when Inside Higher Ed linked to her blog. She wrote:

I don’t think IHE knows what it is like to be one of those linked blogs — and it can be disconcerting and sometimes quite upsetting. If you have a humble little blog like mine, to suddenly see a dramatic spike in readers — in numbers that far outstrip the usual blog traffic — is unsettling. There are some bloggers who cultivate a big readership and crave celebrity status, but not everyone strives for that.

This I can understand. The feeling is similar to the feeling you get when you suddenly notice the room has gotten quiet and everyone is looking at you. Instead of taking this as a lesson, however, BittersweetGirl (whose real name is probably not that hard to find) blames IHE for linking to her.

But, what’s worse is that IHE doesn’t seem to recognize that there are some blog posts on topics appropriate for general consumption but some on topics that are deeply personal, maybe even painful…I think IHE may need to be informed of the consequence of their practice and encouraged to be more discerning about what posts they link for their readers.

This is really unfortunate, because if she had learned from her experience instead of trying to blame someone, perhaps she wouldn’t have subsequently posted the following about how she doesn’t think a graduate student she’s advising will be successful:

Jane is not making the progress she needs to make — I can no longer say with any confidence that she can finish her degree in the time frame we had planned — and, obviously, there are serious consequences for the next stage of her career.

Now that this particular opinion of hers has been published not just here, and at Dr. Isis’s blog, but at Technorati and Google and ten thousand other web spiders, it’s part of Jane’s permanent record. BittersweetGirl may blame the people who linked to her if or when her negative public appraisal of Jane affects Jane’s career, but she’s seriously misled to think that posting that on the public web was ok in the first place, pseudonym or not. I’m pointing this out because it’s clear that BittersweetGirl doesn’t understand the ramifications of what she’s done. As a further sign of her apparent cluelessness, she’s disabled comments on the IHE-linked post. If she hadn’t, I might have just left a comment instead of writing a whole blog post about it, but this post needed to be written.

The musings of the non-pseudonymous librarian, Aaron Tay, are a much more benign case. He wondered on Twitter if he needed to ask permission of the people showing up in search results to write a blog post in which their status updates were featured:

Twitter / Aaron Tay: To be exact, what about a … via kwout

The link he posted was to the Booshaka service, which aggregates public Facebook updates. Below is a screenshot taken shortly after his tweet:

Booshaka – Search public Facebook updates… via kwout

Social mores are different in Singapore and they’re more concerned with “saving face”, so I can understand his wondering if he should ask them first, but it’s worth noting that the next major feature coming from twitter is designed to make it easier to do exactly this. When you make something public on the internet, you no longer have any control over it. The social conventions which hold sway in your part of the world are irrelevant because the web is worldwide and there are news stories containing embedded tweets quite often in the US. In the US, we have a saying that it’s better to ask forgiveness than permission. Implicit in this is the idea that opportunities are lost when you spend too much time coming to consensus and trying to please everyone. Aaron Tay lost an opportunity to write a blog post, perhaps not a very important thing, but if writing blog posts is a part of his career advancement strategy, not writing them is a lost opportunity.

Finally we come to Facebook. It will be no surprise to anyone who’s been following me on Twitter that I’m not a fan and I do not like what they’ve been doing. The tragic thing about what Facebook is doing is that it’s going to bring regulation down on the heads of everyone working in social media. Unless you think Senators have a nuanced understanding of social media and can appreciate why it’s not OK for Facebook to make status updates or location public yet it is OK for Twitter or Flickr to do so, then you’ll recognize that any regulation of social media is going to be a bad thing. One could argue here that no one should have posted anything that they didn’t want to whole world to see, but that argument fails when you consider that Facebook offered to manage their privacy for them and then shirked that responsibility when it became clear that the path to riches lay in exposing that userdata. Now the whole industry will suffer from reduced trust and governmental regulation due to their failure. Furthermore, they really missed a trick. There’s a great demand for a trustworthy and easy to use service that gives people control over their online profile, and they would have been a shoe-in for such a provider. That would have been the route to even greater ubiquity than Facebook Connect. Instead, Facebook is raining hot wax all over us like a malevolent Icarus.

The thread that ties all of this together is expectations of privacy. Simple expecting your wishes to be honored isn’t enough. The content will be around longer than you will, the content will be seen by people you couldn’t even imagine seeing it, and even if you’re told that it’ll only be shown to a limited audience, that can change, leaving you with little recourse.

I propose three things to ease the pain of future events like the above. First, understand that anything you put on the public web can and will be used in ways you couldn’t imagine. It will outlast you and any legal regime that might currently exist. Second, understand that you don’t have any right to blame anyone for any use they make of data you’ve made public. If you make your desires known, people will generally obey them, but there’s no guarantee they will or recourse if they don’t. Taking an active role in presenting and maintaining your public web profile helps ensures that what people see is what you want them to see. Third, give people the benefit of the doubt. Many people have made mistakes and exposed stuff publicly about themselves or others that they didn’t intend, so give ’em a break. Don’t come down hard on people posting pictures of themselves doing things that are legal and accepted in their social group, even if they’re disapproved of in yours.

_{I remain convinced there’s much more upside than downside to having a public presence on the web. I got more job offers extended to me after writing my blog post about moving on than I would have gotten had I not written it, and the linkage from Genome Web’s horde of readers certainly didn’t hurt. LinkedIn remains a great source of opportunities for me. My former employer, on the other hand, didn’t like how my blog post was the top link on Google for their name. I later removed their name when they asked. Also, if I were single and female, I might feel differently about exposing so much information, but if I were and did, I would be my responsibility to not expose that information.}

That an author pays model has the potential to create a conflict of interest is clear. That is why, within reputable publishers, structures are put in place to reduce that risk as far as is possible, divorcing the financial side from editorial decision making, creating Chinese walls between editorial and financial staff within the publisher.  The suggestion that my editorial decisions are influenced by the fact the authors will pay is, to be frank, offensive, calling into serious question my professional integrity and that of the other AEs. It is also a slightly strange suggestion. I have no financial stake in PLoS. If it were to go under tomorrow it would make no difference to my take home pay and no difference to my finances. I would be disappointed, but not poorer.cameronneylon.net, Science in the Open » Blog Archive » In defence of author-pays business models, Apr 2010

You should read the whole thing.

I believe, as does pretty much everyone, that medicine is on the cusp of great changes and that personalized medicine holds great promise. I believe that an informed patient is an empowered patient, and ultimately a healthier one. Every good doctor should want this, and every below-average doctor should pray this day never comes.

Email transcript follows:

Where Dr. Murphy and I differ is in the regulation of medical tests, diagnostic or otherwise. Why should diagnosis (the test + the interpretation thereof, sans recommendations) be considered to be treatment? Certainly insurance companies shouldn’t pay for quack treatments, but getting a test, even having the result of the test explained to you, isn’t a treatment for anything.

The laws were written back when people didn’t necessarily want to see the raw data because they wouldn’t have known what to do with it if they had it, and the tests were inaccurate enough that they didn’t mean much on their own anyways. That was a good reason to make doctors the gatekeepers, but that’s all changed now.

People neither want nor need a gatekeeper now. It’s not about self-treatment, it’s about empowerment. Patients should be empowered to understand what’s going on with their disease or condition or lack thereof, and they should be empowered to ask better questions. Every good doctor wants their patients to be empowered like this. The only thing total release of all information and interpretive notes, medical or otherwise, would do is reveal which doctors are the good ones who stay up-to-date and which are the slackers.

Patients might get together and discuss their tests and decide as a group that they’re going to undertake some quack treatment, but they might also get together and discuss their tests, bring their concerns to a good doctor, and he might realize that “Gee, maybe we should tell myeloma patients undergoing treatment with Zometa to avoid invasive procedures on their jaw.”

There’s upside and downside risks to these kind of patient rights, but Gilles Frydman at ACOR is an example of how to do this in a way to minimize the downsides while maximizing the upsides. They’ve published a study of how this works here. In short, an informed patient is a healthier patient when they’ve got a community like that. Take a look at what they’ve done at ACOR and you might realize the assumed or historical risks of self-treatment pale in comparison to the benefits that tens of thousands of patients are seeing today.

Things have changed and the medical establishment needs to get ahead of the curve on this one, or they risk losing the trust and respect they’ve been afforded for so long. In fact, it’s this loss of trust that has provided an opportunity for the dangerous quack treatments you see today among autistic parents, for example.

Please, in your advocacy of personalized medicine, abandon the paternalistic model. Seek to empower patients and regain the trust our profession has deserved for so long. Don’t let mistrust and arrogance leads patients away into the arms of the quacks.

Sincerely Yours,

William Gunn

Further reading:
In Iran and in the US Health Care System, Citizens’ Access to Computable Data Frees Everything!
It’s my genome: should researchers be obliged to return genetic data to research participants?
A Mom Brokers Treatment for Her Twins’ Fatal Illness
Participatory Medicine

Over a year ago I made one of the biggest decisions of my life. I packed up all my worldly belongings and moved to San Diego to begin working full-time for a small biotech startup. This was a unique opportunity for me because it was a friend who was starting the company, he and I had been speaking over the past couple years about his ideas for a company, they had had me out several years ago during their prototype phase to show me the technology and get a biologists perspective on what a successful product should look like, and I had been doing some consulting projects for them over the past year. At the time, they displayed little understanding of basic molecular biology concepts, but I didn’t think that would be a big deal because they were just starting up and not coming from a biology background, rather an electrical engineering one. I’ve now made another big decision, and that’s to end my relationship with them and move on. Below, I discuss what I’ve learned and what I might like to do in the future.

The opportunity was a unique one because my whole lab was moving to another state, and I was done with experiments, so it didn’t make sense to move with the lab just to finish writing things up. Although there were tremendous social ties holding me to New Orleans, I was preparing to leave for a post-doc anyways when they invited me out to San Diego. I figured it would be just as easy to finish writing my dissertation in San Diego as in New Orleans, so I jumped at the chance.

Where I went

The situation when I arrived was interesting. They had moved out of the garage I last saw them in to a real lab, with chemicals on the shelves and pipettes on the benches. More importantly, they had some working prototypes running. My job was going to be to take charge of the assay development and to manage the lab. Now, at a small company, the hierarchies are essentially flat, because everyone needs the expertise of everyone else pretty much at all times. The team wasn’t so big that we couldn’t all gather together in the meeting room and discuss whatever we needed to.

I loved the freedom of being able to come up with a plan and pursue it, and it was great knowing that I was working on important problems. It was also very cool being surrounded by fantastically talented people from fields quite different from mine. In fact, the first challenge I faced was getting the respect of the others. At a small startup, it becomes very obvious when someone isn’t pulling their weight. However, because the problems you’re solving are always changing, the needs of the company are always slightly shifting as well. My joining the team was part of an overall re-organization of the company to refocus now that the needs had shifted away from their starting point. I didn’t realize this immediately, but I was actually replacing someone in some capacities, with the hope that the tasks which were tangential proficiencies of the existing employees were more direct competencies of mine.

This being my first professional job in biotech, (not my first non-academic job, but I’ll spare you that part of my biography) there was a lot to adjust to. I learned that there are many differences in how groups communicate in academic and industrial settings. The other team members need to feel that you’re confident, because they don’t always have the background that your lab mates may have had to dig into your science and ask critical questions. Additionally, they don’t expect the level of criticism that’s common in a meeting with your lab mates. Of course, the goals of a company are to get to something serviceable as fast as possible, not to find The Truth, and that’s actually not unappealing to me. As much as I remain committed to The Truth, I also like doing things which are technical and applied. At a startup or small company, it’s possible to both have a relatively short-term impact on how science is done or how care is provided, yet at the same time be part of the infrastructure for more fundamental contributions. Being quite used to the graduate school schedule, I immersed myself in the work and comfortably worked 12 hour days and weekends and even pulled an all-nighter or two before an important meeting. Perhaps obviously for those reading along, I became so immersed that 5 months later, I hadn’t so much as looked at my dissertation.

I don’t know what sort of schedules my readers work on, but 4-5 months is when I entered a reflective phase. I was somewhat satisfied with what I had been able to contribute, I felt like I had achieved the respect of many of the team members, and I was particularly happy to have been able to bridge the engineer-biologist gap that existed prior to my arrival. Getting the whole team to speak the same language remains one of the accomplishments I’m most proud of. Teaching the language and concepts of a empirically-based science like molecular biology to a bunch of engineers with physics and computer science backgrounds is not easy, let me tell you!

This time was also a sort of breakpoint we had discussed internally in terms of my stake in the company, so it was time to make some decisions. When my wife delivered the happy news that our first child was due towards the end of the year, it became even more clear that all the loose ends in my life needed to be tied down. I had the following issues to deal with: I needed to finally finish writing up and defend, I needed to find us a living situation more conducive to raising a child, and I needed to have a serious discussion with the company founders about my future. I began working more normal hours, so I could finish writing up and defend. Writing for a couple hours here and there on nights and weekends just wasn’t getting me anywhere. Even as the pace began to pick up at work, I finished my dissertation and prepared for my defense, dropping essentially everything else. If it wasn’t for the selfless support provided by my wife during this time, our life would have probably unraveled entirely. Next, we moved from the loud, noisy, and somewhat juvenile neighborhood where we first landed in San Diego to a quieter neighborhood. Finally, I tried to get some clarity and commitment from the company regarding my long term prospects for advancement and career development. They assured me that I was a valued member of the team, and promised that as soon as they closed this next deal, which was practically a sure thing, everyone would get raises and bonuses and options and all that.

What happened when I got there

As I mentioned above, the needs of a small startup are ever-changing. I found myself spending more and more time doing things where I couldn’t draw from my background, experience, and training. I think I’m a pretty smart guy and a quick study, and I love learning new things, so always having to learn something new wasn’t a problem for me. However, when I stepped back and looked at where things were going, it just wasn’t where I wanted to go. It wasn’t where I could bring my knowledge and skills to bear most effectively, so I wasn’t setting myself up for success. Nor was I setting myself up for failure, but irrelevance? – perhaps. As fun as it was being in that diverse environment, it was also a little isolating. As each sure thing materialized and then evaporated again, I considered applying for a postdoc just to have colleagues again, but I’ve known some 30-something post-docs with children, and there was always a slight whiff of desperation coming from them.

I still believe that the company will do very well, but I had to make a decision between staying with the company, drifting further away from my optimal career path, and possibly showing up one day to find the doors closed or starting now to look for another position in a very tough job market. I picked the second choice because that at least put my fate in my own hands. My leaving was in no way a vote of no confidence in the company. If I was still a single unmarried guy with no kids, I’d probably still be there.

I had been a part of the San Diego Biotech Network since shortly after I came to town, thanks to Mary’s engaging twitter presence and inclusive and generous demeanor, so I knew how hard it was going to be to find another job. I had seen many of the same people at the SDBN meetings, month after month. I began actively applying for positions, but at the same time I started thinking about other possible opportunities, and now I find myself with a variety of loose ends again.

Where to now?

I enjoy working a few hours a week with Mendeley, helping to make introductions between the academic community members who are looking for some solutions that Mendeley provides, and the Mendeley team who want to know what are the as-yet unmet needs of the academic community. As I’ve said before, I think these guys have the potential to transform how research is communicated and how scientists collaborate and there’s a good chance they’ll do just that.

I also enjoy writing about scientific and social issues, particularly ones brought up as the cultures of traditional publishing and the online open access web collide. I’ve had some offers to blog at various outlets and although I would love to take them all up, I can’t yet find a way to do that as more than just a hobby. There’s certainly enough interesting stuff going on around here that I could keep myself busy, though.

Last but certainly not least, I really enjoy doing research, and there’s a local company(should I say?) who has just started a project doing almost exactly what I did for my dissertation. I’ve spoken to people at the company and am currently trying to convince them they’d be crazy not to at least have me in for a talk. They’re large enough that they’re probably not going anywhere soon, yet small enough to give me the chance to be involved in some really groundbreaking work. Not to mention, they’ve assembled a fantastically talented team that I would love to be a part of.

So that’s where things lie at the moment. I have a PhD, I have two years of biotech industry experience, I write fairly well, I have a good number of connections in academia, in publishing, in research, and I’ll have a daughter pretty soon. Hopefully I’ll have a new job soon, too. It’s been quite a year.

With the announcement of Facebook’s acquisition of Friendfeed, I decided to start pulling my tweets into friendfeed. The results were interesting.

Why did I do this?

I did this because Facebook bought Friendfeed and everyone I read expected this to pretty much mean the end of Friendfeed as we knew it (and I still think it does), so I wanted to advertise the new focus of my social network activity. I had not done this in the past because I found that it made for a very noisy friendfeed and because I rather expected an @reply to a tweet than a comment on it in friendfeed. I expected this to shift a small number of subscribers from friendfeed to twitter followers, however what I actually found was a very weak effect on both sites.

How did I analyze the effects?

First, I used Twittercounter to get a graph of my twitter follower history and I compared it with the follower history of Neil Saunders, another person in my network with a similar number of followers and approximate level of activity. Referencing my follower trends to a related person helps indicate which are twitter-wide events (or twittercounter hiccups) and which are specific to me.
Next, I got my Friendfeed subscriber number trends via Feedburner. Because I hadn’t written a post in a few months, my feedburner stats reflected almost exclusively my friendfeed subscriber trends. I couldn’t get a reference graph for FF subscriber numbers like I did for twitter followers, but if you’d like to take my data and overlay it with yours you can download the .csv.

The results

The friendfeed-facebook announcement was made on August 10th, and I added twitter as a service on that day.

You can see that there’s a very slight dip on that day, but there was essentially no change. You can also see that both Neil and I are fairly aggressive about weeding out our spam followers, looking back at the Great Twitter Spam Attack of July.

The friendfeed subscriber trend remained basically unchanged as well. There’s a sizeable dip a couple days after the 11th, long enough for people to get annoyed and unsubscribe, but the numbers bounced right back with a couple days. I don’t know what the two previous spikes were, but in my experience with Feedburner it’s not unusual for your subscriber numbers to be abnormally low for a day or two and then recover.

What does this all mean?

Not a whole heck of a lot, I don’t think. While it confirms my hypothesis that it would annoy some of my friendfeed subscribers and drive them away, the effect, purely based on numbers, wasn’t very large. Checking my email for new subscriber emails since the 11th confirms that the recovery was from new subscriptions on friendfeed (from good people, too!) not feedburner weirdness or a sudden unexplained bump in RSS subscribers. The blip on the very end of the friendfeed graph could have come from some recent messing around I’ve done with my google profile, or it could be from some emailed-but-not-yet-blogged-about stuff that I’ve been doing. I’ve now removed twitter as a service again.

This content is password-protected. To view it, please enter the password below.

Password: