Apparently, Mr Kidd has published original research on his blog for more than ten years. His research includes reconstructing medieval manuscripts that have been broken and pages sold individually. He and others in his field use, among other things, sale catalogs and images from collector sites, libraries, museums, etc. Sale catalogs also apparently have original research and scholarly writing as part of their item descriptions (I am not of this world and am open to correction, for sure).

Auction catalogues often contain original research, and often contain descriptions and images of manuscripts that are otherwise completely unpublished. Their value  to scholarship is immense. Prof. Rossi appears not to know this.

Peter Kidd: https://mssprovenance.blogspot.com/2022/12/the-receptio-rossi-affair-part-v.html

A book was recently published open access, which was funded by a European national funding body, that used original images and text from the blog without attribution. Some text was copied word for word and some was paraphrased. Images the author had gotten from manuscript owners were copied.

Mr. Kidd contacted the publisher because he couldn’t find any direct e-mail for the author. A secretary replied with a couple pretty mean e-mails. Later the author posted documents with some other fascinating statements.

• It’s CC-BY-NC-ND 4.0 licensed and downloads are “forbidden”
• “Every image of the De Roucy manuscript was obtained through the WayBack method, from collectors and dealers.” [is the WayBack method seriously using Archive.org and assuming whatever you find there or on Google images is fair game? because really?]
• “I regret to inform you that blogs are not scientific texts, published by academic publishers, so their value is nil!” [doesn’t mean you don’t cite it, though, if you use it!]
• Some online people looked and there are stock photos used for some of the members of the publishing house staff, the address isn’t legit, and the pictures are not of that address. If you’re going to fabricate an identity, you need to be more thorough in today’s world.
• The cost for humanities scholars to publish a book OA: 20K?
• The book was reportedly peer reviewed – if this was done by knowledgeable scholars in the field… how?
• “Also, perhaps worth remembering, blog posts do not have a DOI number and it can happen that the information provided is scientifically unreliable.” [having a DOI doesn’t make the content any more or less reliable]

More here: https://twitter.com/mssprovenance/status/1606653720174944257

It’s this last that has me writing today. August 31 Wiley pulled about 1,300 books from library collections. Reportedly on third party aggregator sites like those from Ebsco and ProQuest (a part of Clarivate).

Those of you who work in academic settings probably get how this date is a huge problem, at least in the northern hemisphere. Right, so the books that were removed were heavily used by classes and access to them was already woven through many syllabi. AAAACK! It turns out, that Wiley likely did inform Ebsco and ProQuest well in advance, and some libraries are reporting that they did know. Unfortunately, many of the people who needed to know didn’t (do you know how hard it is for libraries to tell all the instructors, adjuncts, TAs, professors to look at a list of 1,300 books to see if any of them will go poof as they are prepping their classes? It’s not going to happen. What happens, is the poof, and then the scream, and then the pitchforks and torches.) Also libraries that were informed found no online alternative and either had to buy the print book or do without and these were heavily used. Presumably Wiley removed them because they weren’t getting enough money for them due to how they were licensed.

That’s just one case, however, and there’s another one that I think is not getting enough attention. Medical libraries buy ebooks, too, for the basic science and research, for education, and for reference in clinical settings, and likely lots of other reasons. For clinical settings, you want to make sure you have the very latest, up-to-date, and best you can have. To do otherwise can lead to poor patient outcomes and even possibly tragedies. On the other hand, for science and history of medicine, and rare/under-researched situations, you do need to go back. The famous case, for example, of a clinical trials participant who died from an interaction that was well documented in the literature, if some of it was pre-1966.

When libraries buy an ebook collection for clinical support, they expect the vendor to curate the collection (at least I think they do? please correct me if not). Some vendors do have a separate backfile- key is that when you’re in a hurry, you get presented the most recent information.

More background: bioterrorism and small pox. There was only a little ongoing research into small pox and a low if steady amount into bioterrorism prior to the post-9/11 anthrax attacks but then the research went wild. Suddenly there were a ton of books, handbooks, etc. And now we’re back to a much lower level of research (below graphs are quick and dirty- and I really should look at published books, but they illustrate – books probably shift to the right a little since they take longer in most cases)

And so here we are with a monkeypox outbreak (hopefully declining just now) and hey, all of our ebook collections have removed “old” books on smallpox! How helpful! Has all the information from the spike in the 2000s been incorporated into the current volumes of books? No idea. Hopefully? (once again, glad to hear from someone who knows the answer).

We had a researcher let us know about a broken link to a bioterrorism book from ~2006. Yep, it has been removed from the ebook platform. This platform maintains archival access to 1 previous edition of textbooks (this book not included). Yep, we got rid of our (my lab’s) physical collection in 2009. ¯_(ツ)_/¯ . We will ask other libraries to borrow – for a time – but aack.

In the end, when you buy a print book in the US, you buy it and you have it (first sale doctrine, too). When you license some use of an ebook… you may not actually get it, and it may not be there when you need it and you have to keep and keep paying to continue to have it and it may not be in a useful format for what you need.

Update: https://www.infodocket.com/2022/10/05/wiley-plans-to-restore-access-to-approximately-1380-e-book-titles-removed-from-proquest-academic-complete-collection-titles-will-return-as-soon-as-possible/

What steps can you take now to avoid this? (If you have full text access to WSJ – either individually or through your local library, these articles might be useful and you can skip below: https://www.wsj.com/articles/how-to-pass-on-your-passwords-when-you-die-11656253211 , https://www.wsj.com/articles/what-to-do-before-you-die-a-tech-checklist-facebook-google-amazon-twitter-11608310486 ) RedCross has an infographic here that’s sourced from National Association of Attorneys General that’s really useful.

Google – inactive account manager https://support.google.com/accounts/answer/3036546?hl=en , https://myaccount.google.com/inactive?pli=1 . Set an amount of time after which they will take action. They can first try to contact you by multiple methods, and then they will notify anyone you list and provide them access. You can say whether you want your account to be deleted and if you want to set an auto reply on your gmail.

Facebook ( Insta has some different requirements, weirdly) – here https://www.facebook.com/help/1506822589577997 . You can choose to have a legacy account manager or to have the whole business deleted. I know this is tempting, but make sure this isn’t the only place there are pictures of you and your family.

Instagram – https://help.instagram.com/231764660354188

Twitter – looks like they delete. You may decide to let this one just stay, empty. https://help.twitter.com/en/rules-and-policies/contact-twitter-about-a-deceased-family-members-account

Apple – legacy contacts – https://support.apple.com/en-us/HT212360

Password managers – check to see if there’s an emergency kit (sometimes with qr code or other way) or if you can designate a legacy contact.

Dani Bassett – Science as a Branched Flow

Profile page. Flow through irregular media – wave dynamics scattering… branches and makes a tree-like pattern. Non-uniform and non-obvious paths

Citation trails like academic bricks (Ahmed, 2017 – likely this) to build careers, to build fields.

Papers with white authors in first authorship overperform by 8% articles by AoC undercited by 17.2%… This trend is getting stronger. (can’t read my hastily jotted notes. They cited a number of different studies showing women were cited less than men.

We can be part of the change we want to see by checking our reference lists – are we citing the same old guys from our grad school days? Are we citing only men?

(see their slide: https://twitter.com/ICSSI2022/status/1534888861477781504)

Rebecca Eisenberg Science of Patents

(I felt like there was a lot of Patent 101 here which I certainly don’t need. I didn’t take many notes. Disclaimer: the one question I tried to ask was sort of scoffed at? So that colors my memory of the session)

Main points:

• Patents unlike publications because inventors get to decide what to patent, if it meets basic requirements (novel, non-obvious, described … )(and yet, not really? don’t we all work in organizations that own our IP? patenting takes money? patenting takes know-how?)
• Difference in who counts as an inventor vs who counts as an author
• Studies in SoS misuse patent information – mistaking terms applied by inventor for those applied by examiner (see this study: https://www.nber.org/papers/w25742) to quote from that:

We argue that there is a fundamental problem in interpreting front page citations as knowledge flows. These citations are not simply a list of earlier patents, academic articles, and other documents which were relevant to the invention. Rather, they derive from a legal “duty of disclosure” requiring inventors to list documents material to the patentability of their claims. Material to patentability means there is no need to cite tools that help build the invention, information used to avoid unpromising paths, basic facts that focus effort, or research suggesting a technological hole or a market need

Joseph Staudt – Specialized Human Capital and STEM Career Outcomes

His page. Best of markets for some fields but worst for others. Is the US producing too many or too few STEM PhDs?

He looked at “quality” of job placements as faculty, postdocs, industry using UMETRICS, ProQuest Dissertations and Theses, IRS W-2 (!), LEHD (about).

Faculty placement – assume “good” position. In some fields like Chem, Geo, lowest fraction of faculty placements.

Post Docs – may indicate weak demand or demand but position requires additional skills after PhD. So look at if there is higher wage growth after postdoc. He found fields with higher % in post docs had lower earnings (d’oh!)

Industry – depends on quality of placement. (his way of measuring this is fancy and huh). Pay + dissertation abstract vs firm portfolio patent classification. He used the Census Business Register to see if the person’s dissertation was in the same general field as the company patented. (so yeah, i mean like if you have a Boeing or something or gosh, my place of work. I have a good placement but ?)

1 s.d. higher relevance 5% higher earnings.

In the end, it depends on field.

Julia Barnett Intersectional Inequalities in the Impact of Online Visibility on Citations

Her page. Looked at Altmetric % researchers listed by field

Ratio of women on the team (pub 2013-2019). Ethnicity by Blau’s index.

Women are less visible online and work visible online is more cited. Intersectional Penalty

Mengyi Sun Effects of Peer Review Format on Bias

Profile. (preprint but looks like it also made it to JASIST) Does blinding reviews reduce prestige bias, increase accuracy and reliability? A natural experiment of a large computer science conference that changed from single-blinded review to double-blinded in 2018. Prestigious scholars got lower scores but did not generally change acceptance status (threshold was lower). Papers rejected under single-blind (and then appearing elsewhere I guess) were more highly cited than those rejected under double-blind so better acceptance decisions.

Sarah Rajtmajer Synthetic Market for Estimating Confidence in Published Work

Part of larger DARPA SCORE project described by Nosek earlier. Paper > features > trader bots . Trained with output from major replication projects. Some papers are too dissimilar from training so no bots participated (they know what they don’t know). Next, multiple claims and hybrid markets.

Dakota Murray – Understanding Disagreement in Science

publications. Looking to quantify disagreement in science. Retrieved citation context for articles. Looked for cues of disagreement. Found the most in social sciences and humanities, and health policy/health economics, and Paleontology/geology.

Sadamori Kojaku Who Wrote This Paper

page Disambiguating authors at scale. Usually use author traits, but when you try to do that with standard information retrieval methods – like string similarity or edit distances – you get horrible accuracy (most affiliations include University, for example). He used DisAmBert (built on BERT) which outperformed. Most important feature turned out to be article title, which is typically omitted from string matching methods.

Binglu Wang Dynamics of Innovation Abandonment

Using MAG, USPTO, commercial products. Abandonment probability changes over time. Are their structural/social effects? Preferential abandonment. Collapse of network with removed nodes – as happens with sudden disappearance of things like MySpace.

Panel – Innovation and Inequality

Donna Ginther (Kansas) – reviewing her important work 11 years ago in disparate funding at NIH (and NSF) for race/ethnicity, and gender. In the US, the gender gap has closed. Race/ethnicity persists;however, there is a paucity of data on race/ethnicity in the US and none in Europe. Women’s application rates are up and their success rate is higher than mens (31%-28%). Women are also more likely to apply for NSF and be successful. Gap is larger at NSF than NIH. Cites Nakamura et al – attributed to “halo effect” for white investigators instead of bias against black/Hispanic investigators.

Rem Koning (HBS) – who do we invent for? Labor market has bias and gaps, but how about the product market? Patents by women 1976-2010 – USPTO NBER. Their data set is available on Harvard dataverse. Fed text in, spit out MESH terms. Female management of teams more likely to patent in areas of interest to females (like say ovarian cancer).

Innovation platforms like Kickstarter – response on there is taken to VC companies for funding. But majority of users there are men (75-90%). 30% of female products have a female inventor. These products grow 45% less than neutral or male tech

***Marcella Alsan – Excellent talk. Clinical trials… Life expectancy gap. Black less likely to participate in clinical trials and less likely to be prescribed new medicine.

Regulators and interest groups pressuring to diversify trials but market penalized Moderna when slowed recruitment to get a more diverse base for COVID-19 vaccine.

Does lack of racial diversity in the innovative process affect how Black patients benefit from new technologies? Efficacy and diversity both matter in doctor’s prescribing intentions. Diversity matters more for doctors who treat Black patients. If the trial is diverse, prescribing gap depends on efficacy. Same efficacy, diverse trial no different for white but 0.78 s.d. for black patients.

In responding to audience questions, Dr Alsan spoke with the importance of working together with doctors of the community – how that impacted recruitment, study design, and understanding the results.

The first invited speaker for the afternoon got stuck in his room isolating after testing positive for COVID-19. They explicitly required in-person attendance, but I would have appreciated if he could have Zoomed in if he felt like it. (maybe he didn’t feel up to it? or maybe they couldn’t – but there was good wifi so… probably a policy thing). Disappointed because I’ve always been interested in his research. ¯\_(ツ)_/¯

Sadamor Kojaku – Mapping Science Foraging

Compared frog vs bird metaphor: work in a well, in one place vs. foraging more widely. How can we operationalize the concept of foraging in order to study it?

Trying a bag of topics approach, we can show a changing topics over time for authors. Relationships between scientists. But this approach pretends topics are discrete. Need a continuous representation.

He moved to node2vec and 64 dimensional space. It predicts PACs classifications.

Most jumps in topics are small but long jumps are not rare. Scientists don’t go back to previous topics. Longest jump can happen any time in career.

Sam Zhang – Labor Advantages Drive Greater Productivity at Elite Institutions

His web page. Relevant pre-print.

Similar scientists have different productivity based on their institution. Funded student and postdoc labor is a driver to productivity. Used the literature to classify fields by collaboration norms. 262 PhD degree-granting institutions, NSF survey counts of funded/unfunded. If you have the same address as a faculty member but aren’t on the list of faculty > group member.

More prestigious institutions have more funded students across disciplines. In fields with collaboration norms, productivity goes up with more labor.

Government Funding Panel

Daniel Correa, CEO FAS (former OSTP and other places)- Endless Frontier Act is lauded, but it’s authorization, not appropriation. The time is now for engagement. You don’t need ambitious policy from Congress to get things done. Agencies have leeway and are looking for good ideas. The challenge – we don’t know enough about who. The costs of getting funded are huge. After you complete work, need to follow up with a proposal to fix the problem. Find empowered public servants who are the connective tissues between science and government. Bring them action plans. FAS Impetus Institute – improve efficacy of federal R&D investment

Arthur Lupia – UMich, formerly gov’t. The average federal grant is 2 times the US median household income. The grant is instead of funding housing, roads, or whatever other things government has to take care of. It must return a substantial distinct public value. When you go in to meeting with government, be ready to answer the question: whey should we fund you instead of children’s cancer research? Whose quality of life will be improved. What scientific opportunities are empowered?

Dewey Murdick – CSET at Georgetown, formerly gov’t

Federal funding no longer dominant. US Federal Funding not dominant in global science. ( I wrote this down but not sure the context: https://www.sciencedirect.com/journal/technological-forecasting-and-social-change/vol/146/suppl/C#article-56)

Tara Schwetz – NIH

They have a lot of different programs in place to assess if/how NIH is achieving (? I can’t think of the word) their mission.

RCR – relative citation ratio

search analytics – word2vec cluster visualization

person disambiguation

Status of ARPA-H: transformative high risk/high reward research. Speed health breakthroughs. Data collection and assessment. Embracing evidence-based diction. Active program management. Continuous process improvement. Transparent, collaborative, and diverse. Funded by Omibus 3/22. New Deputy Director named: Adam Russell (formerly of ARLIS, IARPA, DARPA)

Clara Boothby – Turnover Replacement and Career Age Distribution

Related pre-print. This generation has a postdoc apocalypse. Attrition rates from academic settings are high. Why are universities still recruiting doctoral students when the career market is so bad? They looked at publications 1980-2019 for authors with 80% of their publications coming from a US address. Biology has a 2.8% per year growth rate. New researchers are 10% of the workforce each year and researchers leaving about the same. Turnover is higher in early career positions. Chemistry turnover may be to industry.

Kevin Kniffin Walking the Talk of Disciplinarily

His profile page. Are people more interdisciplinary? How interdisciplinary work is doesn’t depend on gender, citizenship, race. Interdisciplinary researchers get paid less and are more likely to postdoc but it does depend on field Computer Science not penalized

Xiang Zheng Parenthood contributes to gender gaps in academia

Relevant pre-print. Combined a survey of 7,764 researchers with information on their publications from WoS they found that the established gender gap in academia is primarily a parenthood gap. In their survey, women were less likely to have children and likely to have fewer. Decisions to have children are career related (60%), Women were more likely to report negative impact of children on career.

Katie Spoon Explaining gendered retention patterns in academia

Conflicting evidence of gendered differences in retention. Existing papers are about small groups or institutions or fields. She used data from Academic Analytics Research Center – 279,419.

Women were at higher risk of leaving throughout their career. Survey of people from longitudinal study: workplace culture, harassment reasons to leave. Switch institutions due to pull, leave academia due to push

Timothy Errington – Replicability of preclinical cancer biology

Article: https://elifesciences.org/articles/71601 . Reports on repeating experiments from papers. About 46% they were able to replicate. Generally smaller effect sizes than the original report.

Christian Chacua – Migration and Research Interest

profile page: https://cchacua.github.io/christianchacua/ . Studied Colombian researchers who worked overseas for some part of their publishing career. Even ones who had topics tied to geography (volcano) or ecology, changed subjects when they moved. (this was a really interesting talk but too short and also, toward the end of a really long day)

Hao Peng Gender Gap in Scholarly Self-Promotion on Social Media

pre-print Looked at 7M tweets about articles by 1.3M authors from 2018. Women less likely to self-promote. Magnitude of the gender gap has more to do with journal prestige than other factors they looked at. When women do tweet, their papers get more attention.

Louis Shekhtman Mapping Philanthropic Grants to Science

This was pretty cool and different from everyone else’s. Looked at a ton of tax returns from philanthropic organizations

Emma Zajaela In person vs Virtual Conferences

Conferences promote collaborations. Small group interactions more so. Virtual conferences still promote interactions, but usually set events and not informal mingling. pre-print here

https://twitter.com/ICSSI2022/status/1534643208440905728

note: this is edited – I left off the panel because incorrectly remembered them being in the afternoon

Brian Nosek – Objective Human and Machine Assessments of Research Claims

Brian Nosek from the Center for Open Science (page) did the opening keynote for the second day. He’s on the DARPA SCORE program as a performer and he talked about their methods and outcomes so far.

How do we assess credibility of research claims (* in the quantitative behavioral and social science literature)? There are research programs and quick heuristics. They looked to see if things were reproducible (**given: question was a good one, instrument was appropriate, data was good…). They looked at 62 journals, 50 papers per year, downselected to 5 papers per year per journal, manually extracted claims from the abstract (***statistical claim) – picked one claim, for 200 papers entire sequence of claims or “claim tree.” They failed replication if data were not provided. Human and machine judgements roughly correlated but messy. They (want to? have already?) made a prototype researchers can run their papers through. But, bias in AI, can also use that to study the biases in science.

Janet Vertesi – Thinking Organizationally About the Science of Science

Her Princeton page. She is a sociologist who conducted a 15-year ethnography with two big science teams that work on planetary missions. Read her books!

She talked about how culture impacts knowledge outputs and outcomes. In flatter organizations, culture substitutes for formal structure – it has to do with control, communications, and legitimacy. Going back to Vaughan’s 1999 Challenger study – the organizations impact outcomes.

Spacecraft teams are special because they include lots of different disciplines. They are high performing – you can see their results. They last a long time (like more than a decade in many cases). But each is unique in local micropolitical processes to decide what to do and how to allocate scarce resources and each has a “mission personality.”

Two different teams. One is a matrix organization with 12 different instrument sub-teams and 5 cross-cutting teams on interests (something to study using the spacecraft or rover’s instruments). “Poly vocality” Emphasis on fairness – everyone “equally unhappy”. Talk of conflict with alliances. Instrumental metonymy – the instrument is a person.

Second team takes a collectivist approach. One team with everyone equal. Everybody has to agree on the plan. Each person has to answer “are you happy” at the end of each meeting (the bit about fairness should probably be next to this team – but this is the order of my notes.. hmmm). Shared ownership of the plan. The reports out (in the literature) are mostly co-authored by all. Integrated use of all sensors vs. instrument by instrument. Gendered stereotype threat for team members.

Data and Equipment Panel

This was a real who’s who of people providing data in this space (outside of the 800lb gorillas, that is)

Julia Lane – Has over many years worked on IRIS, PatentsView, NORC, Starmetrics, etc. Discussed the difficulty of funding infrastructure vs the huge impact. Great example was funding the development of Jupyter notebooks. Recently working with THE NATIONAL ARTIFICIAL INTELLIGENCE RESEARCH RESOURCE TASK FORCE (NAIRRTF) (I’M NOT YELLING I JUST COPIED FROM THEIR PAGE) on computing, data, and workforce capacity.

In response to the Evidence Act, her group at NYU was asked to create a data repository [aside: not clear how this differs from a million others – she said it was supposed to be like Amazon, but ????]. They discovered, quel surprise, that more metadata was needed to make the data really usable for future projects. Even more, people wanted rich context in which the data had been gathered, used [aside: yeah, i can find like maybe 20 articles without any effort at all saying this but they discovered it, too]. So they had a competition for organizations to use NLP and other methods to create rich context for the datasets. They describe all this in their book which is available as an ebook open access. There’s also a more balanced description on their site: https://coleridgeinitiative.org/projects-and-research/show-us-the-data/ (snark aside: building a knowledge community around government data does sound like a good idea as well as more provenance and linking between datasets/funding/research products is definitely good. I do firmly reject the idea that there was no way to find books before Amazon or that Amazon is the best way to find books now.)

Dario Taraborelli – The missing Citation Graph

Dario is now at the Chan Zuckerberg Initiative. He has already archived his slides: https://zenodo.org/record/6629285

We are gaining the benefits of open sharing of research results (papers), but the many software packages and dependencies that made this research possible are not tracked. The SoftCite dataset (10.5281/zenodo.4444074) tracks mentions of software packages in PMC and econ open access articles. You can see NumPy has >1M dependencies but has never received government funding (despite trying)

Euan Adie – Overton.io

Euan founded this company to provide a service that helps identify how scholarly articles are used in policy documents worldwide. They have about 5.9M documents from 1,580 sources, 188 countries. Data are robust to 2012-2015. Policy includes government (including US Congress) and think tank

What are policy documents linking to? Mostly social science. Often things published in the language of the policy organization. There is no shared citation culture – even within government.

Free for research, incorporated into Elsevier Plum Metrics, and for pay for governments, policy orgs, etc.

Jason Priem – Our Research

Open Alex – open transparent, updated. API, download a snapshot, and coming this summer, web GUI. It’s self-sustaining and will remain open. Started with Microsoft Academic Graph. Machine learning has matured a lot and they use it to help with metadata. There is a PID graph of DOI, ORCID, ISSN, ROAR

Coverage is broader than Scopus and WoS but narrower than Google Scholar

Curated for to ensure good quality data, but not to exclude low impact or international publications. It has limited coverage of things like software, data, and books

30M works were added this week

Between the AM and the PM there was a “poster slam” in which each poster presenter has just a brief time to give an overview of their work. This is really effective if you can get the slides to work.

Yian Yin: Tipping Point in Failure Dynamics

Dr Yin’s page is here. So this was post lunch, and I have very few notes. I believe he reviewed his earlier work that looked at NIH grants and VC. His current work looks at data science challenges on Kaggle, Top Coder, MathWorks (and probably one more?). The improvement in solutions was punctuated – there were quick improvements to start, and then plateaus with incremental improvements and then more periods of greater improvement.

Panel: Funding Science

This panel included representatives from funders and people who had worked for funders but have since moved on.

Alexander Berger – Open Philanthropy: A consumer of science of science, but reads our literature with confusion and disappointment. Is skeptical that everything is best measured using citation data from afar and maybe we need more case studies (yes!). Need societal impact more than just citation impact.

Rikke Christiansen – Novo Nordisk : World’s 3rd largest foundation. Founded as a condition of getting the original license to sell insulin in Scandinavia. They have traditionally been very conservative in funding but are looking at new ways to experiment and also ways of analyzing the data they have gathered to assess their grantmaking.

Theodore Hodapp – Gordon and Betty Moore Foundation: They are experimenting with a new program to fund people (vs projects) in experimental physics.

Daniel Goroff (moderator) – Alfred P Sloan Foundation

Contributed Paper: Robert Ward on Interdisciplinarity

Here’s his profile. We say interdisciplinarity is good but we penalize people in measurement. Interdisciplinarity as recombination or integration – we say we want one but we get the other. His study looks as papers as a sequence of ideas each sourced from one or more disciplines. Local paper paper matrix of ideas. Global (WoS) matrix of ideas that occur together (hmmm). Integration of multiple fields lowers citation.

Christopher Esposito – Knowledge Obsolescence

Here’s his page. Corporate patenting is going up, age great invention is going up, team size is going up. … translating my notes: as you gain experience in your career, you accumulate knowledge but some of your knowledge becomes obsolete the longer time in career. He used age of introduction of a patent subclass code as a proxy for field growth of knowledge (hmmmm). Working in a team means losing less ground

Erin Leahy – What types of Novelty Are Most Disruptive

Profile. Previously novelty extent, amount; type of novelty: result, theory, method; impact: consolidating, disruptive

Hypothesis is that new theory or result consolidating; new method disruptive

Looked at papers described in Citation Classics (this is brilliant! what a great resource to find out how the authors viewed their contributions – in retrospect). Notes are a bit fuzzy but I think her hypotheses were supported.

Alexander Furnas – Partisan Differences in the use of Science in Policy

His page. He used overton (yay!) and publications from dimensions. Reliance on science in policy in the US is going up. Most of that increase is due to think tanks – gov’t is stable. They disproportionately rely on high impact science. Left in the us uses more science than right but that doesn’t hold across countries.

Science here refers to anything published in a journal

Tara Sowirirajan – Innovation Glass Ceiling

Her profile. Innovation as day to day vs. blue sky. Patents are atypical or innovative by CPC codes used. Women get fewer patents, women don’t appeal negative patent decisions (missed a bunch here)

Jiaxin Pei – Do Journalists Overstate Science

Profile here. This work is described here. They have a certainty estimator python package.

Daniel Gross – Effects of WW2 Research Efforts on Post War

Pulled records for all 2,200 OSRD contracts to look at long lasting effect on extent, location

Misha Teplitskiy – Effect of Conference Presentations on Diffusion

Profile here. This was a pretty neat study. They used conference apps and looked at sessions people starred as intention to attend. If they starred multiple for a slot, that was fractionalized. They then looked for future citations.

Dashun Wang opened the conference with an introduction. The purpose of bringing together and creating a useful dialog between the producers and users of science of science & innovation research. Standing on the shoulders of (white male) giants of Kuhn, Garfield, Merton (actually may have provided Zuckerman, too so not entirely male). Using a diversity of approaches.

Panel: Innovation and Growth

Moderator Heidi Williams

Rush Holt : Choosing measures of success in evaluation is important and shouldn’t be solely on the scientists’ terms nor should it be entirely on economics terms. Rather should incorporate returns to society and should incorporate society’s view. He edited a Princeton Univ Press re-release of Vannevar Bush’s Science: The Endless Frontier (in libraries, original version– freely available). Bush’s view of the bargain of science. Scientists get the funding, do evaluation, and have autonomy. Society gets the outputs and new products. Science education to fill the pipeline [gosh my handwriting is bad – missed the next point]. A cadre of well-trained scientists is no substitute for a well-informed public [I think the part my notes missed is that this autonomy with limited oversight may not direct research to the problems most important to society. The just trust us view, turns out to not be adequate in this age of mis/disinformation if the public is not well-informed] Scientists shouldn’t be telling the public they can’t handle it but instead do a better job enabling the public to learn and ask for evidence.

Benjamin Jones : 2.8% of US GDP for R&D, both public and private. Private is going up as a percentage and government funding is going down (since the 1960s). Funding of basic science is less attractive for private funders. Research shows $1 in R&D funding returns $5 in social benefits but the group making the investment isn’t the group getting the benefit. The US could go to 5x or 6x this amount and it would still pay for itself. We have seen a productivity slow down with the decrease in R&D funding. How to fund science, though? Should it go to NSF, NIH? Which is not important, they each provide returns. There is a problem, however, with herding. It is not efficient to be like children’s soccer to have everyone working on the same problem. The other problem with funding increases like this is the pipeline [aside – my blogging colleague Drug Monkey has lots to say about employment in his and related fields]. You have to have enough people who can perform the research when you increase the funding quickly.

Marcia McNutt : Innovation and Climate Change. [presented issue of climate change and discussed the relative importance of adaptation and mitigation. Teach science as exploration, not facts at all levels.

Bhaven Sampat: Crises and funding of science. What were the long run effects of the WW2 shock to funding [for one thing, MPOW wouldn’t exist, lol]. What do we know about innovation policy during crises. What are the lessons we can take for normal times. In 6 months prior to the formation of OSRD, 2k contracts with universities and funding levels that were way higher. Bottom-up priority setting with military to set goals. Coordination so there were no holes and no herding. The research was explicitly applied – basic research done before the crisis. Speed. [the authors are also summarizing this research here]. Turning to vaccine research – the mRNA research, sequencing, etc., was all well known but the Manhattan Project – like Warp Speed funding brought this to production very quickly. BUT the aerosol research and the safety of schools trailed. There was a TON of herding (for ex, how much money and time was wasted in studying therapeutics that were universally found not to be useful?) – there was a need for air traffic control. In non-crisis times, there needs to be better support for disruptive or breakthrough efforts

Brian Uzzi: Conditions favoring good ideas in science

See a list of his research here. [Extra caveat – I was so into his talk and listening super closely that I didn’t actually write much and I’m not seeing this in his pre-prints or papers to check details ]

There is robust evidence that diverse team composition leads to more innovative and impactful work.

AI accurately identifies dominant [unreadable maybe patents or tech?] prior to the market not on actual capability but on network of dependencies [oh, i think this was patents?].

Using the author gender estimator (Santamaria et al 2018) – which is of course not inclusive of all genders but does have some face validity when compared to self-report. … Big teams dominate, gender diversity increases, and in medical subfields the same…

Using data and doc2vec over granted vs denied patents from US, UK, Canada. free text to neural networks. separate analysis for enumerated fields such as cpc, authors, organizations. Bag of words model works better than for enumerated. Ground truth they used the results of the appeals board. Compare if machine would grant patent vs board and original decision. Machine would have correctly granted 47% of originally rejected (good), but also 31% of correctly rejected (not as good – but he says to err on the grant side.. i’m not so sure).

Audience questions were quite good. Q1) Matilda effect – do women gain the benefits of the impact of gender diversity A) they looked at team leadership – first or last author – did not have effect. Q2) gender diverse teams and gender diverse fields. A)didn’t find a strong signal but the smaller the field the smaller the effects

Karim Lakhani: Studying Science While Doing Science

Dr Lakhani is from Harvard Business School coming from studying open innovation and open source software. They are working to do field work and experiments to understand the science of science and innovation. This is to get around the myopia from looking only at outputs or citations of outputs or tech transfer of IP.

How do collaborations form? They did an experiment to change search costs for finding collaboration partners. They had an event for which attendance was mandatory to get funding. Randomized breakout rooms increased collaboration by 75% (more on this in their paper). Speed dating for scientists as powerful as co-location – accelerates water cooler time.

Bias in evaluation. Inter-rater reliability in VC, SME judgement, junior SME judgement isn’t great – closer the researcher is to the proposal, the more they will punish it (more picky). Novelty gets punished.

RCTs are required to do science (in some fields) but not to manage science. Portfolio of curiosity based science vs novel/incremental science. How do we train scientists to be managers in the scientific enterprise (not at all or not well?). Corporate R&D needs to be studied as well as academic.

It’s been a loooooooooong time since I attended a conference in person, but when I saw the speakers for this one and noted it had free registration and is local-ish to me… I was in! And I made the right choice. A bunch of my favorite science of science types were there – folks I hadn’t seen in forever. The talks were all very interesting (if biased – which I will discuss later) and there were plenty of opportunities to chat with other attendees. A couple speakers swooped in, talked, and then left but unlike ASIST, the vast majority of attendees stayed until the very end.

The conference was sponsored by a major information science vendor that starts with a C but also AFOSR, Sloan Foundation, National Academies, and the Northwestern Kellogg School of Management. AFOSR has been an active sponsor in this space for a while and it’s good they are continuing to work in this area.

So what was the conference about? All but one of the presentations used quantitative measures and models to study what science is being done, what patents are being applied for and granted, who is doing the science, who is working together, how science can be/is funded, who is being funded to do science, what can we learn from WW2 (and post-) to understand other crisis science funding and performance, is novelty rewarded, what is the interaction with gender/race/ethnicity… And then a bunch of meta panels – what does government need to hear, who is funding and what are they funding (like from an apply here this way point of view), what new datasets are available.

I will say it was disappointing that there was nearly no actual talking to people used in any of these studies with the notable exception of Vertesi’s 15-year ethnography of NASA teams. One interesting change was the addition of more business researchers. Typically, even when some of the speakers are at business schools, they’ve still employed the same science modeling using citations we usually see but that seemed to change this time. The other thing that was new was the inclusion of economists. They of course are using models and datasets from their world which aren’t typical. So that was good. We also had a couple law professors but I’m not sure their inclusion was actually very helpful. It had a ton of potential but more on that when I get to the individual posts.

The conference is at https://www.icssi.org/ but unfortunately it just says “invited” or “contributed” so I’m going to have to rely entirely on my hand-scribbled notes and tweets O_o . The hashtag was #icssi but i was tweeting from my phone so not terribly prolific. Hopefully now that the conference is over, they’ll put more detailed info up.

Someone asked me about computer programs for qualitative research (CAQDAS). These are super spendy but can really help. But anyway I said if it’s only one project you can totally use paper and use displays from that one book.*

You know, the one book. It’s gray and it’s paperback, and it’s pretty heavy? Probably has qualitative in the title? … if only I could brain today…

So I search WorldCat, Amazon, Google for qualitative – analysis? gray ? use image search to look for the cover… no, no, no… it’s not new at all

I started thinking. This thing was assigned in my class. In fact, this thing is probably assigned in a lot of classes and has been for years!

There’s an online database called Open Syllabus. It does what it says on the package. Better than just being a pile of them, you can search, rank, and even explore a graph:

So I typed qualitative and there it was

Funny thing – there’s a 2014 and 2020 edition that are not in fact gray, but here’s the cover of my edition:

*This originally appeared on Twitter in a thread